2024-03-19T01:30:24.052 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-19T01:30:24.052 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_pdonnell@teuthology --name pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi --archive /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609330 --description fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} --verbose -- /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609330/orig.config.yaml 2024-03-19T01:30:24.102 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-19T01:30:24.204 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609330 branch: wip-batrick-testing-20240318.181317 description: fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} email: pdonnell@redhat.com first_in_suite: false job_id: '7609330' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-batrick-testing-20240318.181317 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mds beacon grace: 60 mds heartbeat grace: 60 osd heartbeat grace: 60 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds valgrind exit: true mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 mon osd crush smoke test: false osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd fast shutdown: false osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - PG_AVAILABILITY - PG_DEGRADED - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow request - SLOW_OPS - MON_DOWN sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 valgrind: exit_on_first_error: false mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes watchdog: daemon_restart: normal ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: valgrind: - --tool=memcheck - --leak-check=full - --show-reachable=yes check-counter: counters: mds: - mds.exported - mds.imported install: ceph: debuginfo: true flavor: default sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240318.181317 sha1: e48b1f6c0608b2615e1890cc46286c3877d6e910 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 3005 sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 sleep_before_teardown: 0 subset: 433/512 suite: fs suite_branch: wip-batrick-testing-20240318.181317 suite_path: /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: e48b1f6c0608b2615e1890cc46286c3877d6e910 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi055.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKlRIQTGkVhqf+6pMMXumtETSjuIj1Q6n2/pZ7EW7JggKQFzxhOFNzD60SE8WCRzx7rVm3nQcyVq4i5pk+nPOnQ= smithi084.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBlNX356useTeVfcKcLCf9iAQL/T/oUnh9iHG7sjZ48DuT6vicuCSMWgw8zihf0RmaAtR/cuXslFxmaySc5msVc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-18_21:05:49 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-19T01:30:24.204 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa; will attempt to use it 2024-03-19T01:30:24.205 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks 2024-03-19T01:30:24.205 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-19T01:30:24.207 INFO:teuthology.task.internal:Checking packages... 2024-03-19T01:30:24.231 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'b6ae24918b03b1b8a19c9263a50655bba80bb810' 2024-03-19T01:30:24.231 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-19T01:30:24.231 INFO:teuthology.packaging:ref: None 2024-03-19T01:30:24.231 INFO:teuthology.packaging:tag: None 2024-03-19T01:30:24.231 INFO:teuthology.packaging:branch: wip-batrick-testing-20240318.181317 2024-03-19T01:30:24.231 INFO:teuthology.packaging:sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T01:30:24.232 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-batrick-testing-20240318.181317 2024-03-19T01:30:24.446 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2206-gb6ae2491-1jammy 2024-03-19T01:30:24.447 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-19T01:30:24.454 INFO:teuthology.task.internal:no buildpackages task found 2024-03-19T01:30:24.454 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-19T01:30:24.461 INFO:teuthology.task.internal:Saving configuration 2024-03-19T01:30:24.480 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-19T01:30:24.489 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T01:30:24.519 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609330', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 01:24:59.424129', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPfYyu2DRh0J3Z7NHgabqzvxaq1JyfSmFMDAIHscQsgE'} 2024-03-19T01:30:24.543 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609330', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 01:24:59.426172', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAV/QmBV1VYran48xMwQswlRdINB2X9zRROqKkXaJAF/'} 2024-03-19T01:30:24.543 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-19T01:30:24.549 INFO:teuthology.task.internal:roles: ubuntu@smithi055.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-19T01:30:24.549 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-03-19T01:30:24.549 INFO:teuthology.run_tasks:Running task console_log... 2024-03-19T01:30:24.624 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f04a421a4c0>, signals=[15]) 2024-03-19T01:30:24.625 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-19T01:30:24.631 INFO:teuthology.task.internal:Opening connections... 2024-03-19T01:30:24.631 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi055.front.sepia.ceph.com 2024-03-19T01:30:24.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:30:24.716 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi084.front.sepia.ceph.com 2024-03-19T01:30:24.717 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:30:24.796 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-19T01:30:24.803 DEBUG:teuthology.orchestra.run.smithi055:> uname -m 2024-03-19T01:30:24.808 INFO:teuthology.orchestra.run.smithi055.stdout:x86_64 2024-03-19T01:30:24.808 DEBUG:teuthology.orchestra.run.smithi055:> cat /etc/os-release 2024-03-19T01:30:24.856 INFO:teuthology.orchestra.run.smithi055.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-19T01:30:24.856 INFO:teuthology.orchestra.run.smithi055.stdout:NAME="Ubuntu" 2024-03-19T01:30:24.856 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION_ID="22.04" 2024-03-19T01:30:24.857 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-19T01:30:24.857 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION_CODENAME=jammy 2024-03-19T01:30:24.857 INFO:teuthology.orchestra.run.smithi055.stdout:ID=ubuntu 2024-03-19T01:30:24.857 INFO:teuthology.orchestra.run.smithi055.stdout:ID_LIKE=debian 2024-03-19T01:30:24.857 INFO:teuthology.orchestra.run.smithi055.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-19T01:30:24.857 INFO:teuthology.orchestra.run.smithi055.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-19T01:30:24.857 INFO:teuthology.orchestra.run.smithi055.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-19T01:30:24.857 INFO:teuthology.orchestra.run.smithi055.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-19T01:30:24.857 INFO:teuthology.orchestra.run.smithi055.stdout:UBUNTU_CODENAME=jammy 2024-03-19T01:30:24.858 INFO:teuthology.lock.ops:Updating smithi055.front.sepia.ceph.com on lock server 2024-03-19T01:30:24.883 DEBUG:teuthology.orchestra.run.smithi084:> uname -m 2024-03-19T01:30:24.888 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2024-03-19T01:30:24.888 DEBUG:teuthology.orchestra.run.smithi084:> cat /etc/os-release 2024-03-19T01:30:24.936 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-19T01:30:24.937 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="Ubuntu" 2024-03-19T01:30:24.937 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="22.04" 2024-03-19T01:30:24.937 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-19T01:30:24.937 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_CODENAME=jammy 2024-03-19T01:30:24.937 INFO:teuthology.orchestra.run.smithi084.stdout:ID=ubuntu 2024-03-19T01:30:24.937 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE=debian 2024-03-19T01:30:24.937 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-19T01:30:24.937 INFO:teuthology.orchestra.run.smithi084.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-19T01:30:24.937 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-19T01:30:24.937 INFO:teuthology.orchestra.run.smithi084.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-19T01:30:24.937 INFO:teuthology.orchestra.run.smithi084.stdout:UBUNTU_CODENAME=jammy 2024-03-19T01:30:24.938 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2024-03-19T01:30:24.958 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-19T01:30:24.967 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-19T01:30:24.972 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-19T01:30:24.973 DEBUG:teuthology.orchestra.run.smithi055:> test '!' -e /home/ubuntu/cephtest 2024-03-19T01:30:24.975 DEBUG:teuthology.orchestra.run.smithi084:> test '!' -e /home/ubuntu/cephtest 2024-03-19T01:30:24.984 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-19T01:30:24.990 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-19T01:30:24.990 DEBUG:teuthology.orchestra.run.smithi055:> test -z $(ls -A /var/lib/ceph) 2024-03-19T01:30:25.022 DEBUG:teuthology.orchestra.run.smithi084:> test -z $(ls -A /var/lib/ceph) 2024-03-19T01:30:25.038 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-19T01:30:25.095 INFO:teuthology.run_tasks:Running task kernel... 2024-03-19T01:30:25.110 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-19T01:30:25.110 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-19T01:30:25.111 DEBUG:teuthology.orchestra.run.smithi055:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T01:30:25.111 DEBUG:teuthology.orchestra.run.smithi084:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T01:30:25.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:30:25.116 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2024-03-19T01:30:25.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:30:25.117 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-03-19T01:30:25.165 INFO:teuthology.orchestra.run.smithi055.stdout:5.15.0-56-generic 2024-03-19T01:30:25.165 INFO:teuthology.task.kernel:Running kernel on smithi055: 5.15.0-56-generic 2024-03-19T01:30:25.166 DEBUG:teuthology.orchestra.run.smithi055:> sudo apt-get clean 2024-03-19T01:30:25.167 INFO:teuthology.orchestra.run.smithi084.stdout:5.15.0-56-generic 2024-03-19T01:30:25.167 INFO:teuthology.task.kernel:Running kernel on smithi084: 5.15.0-56-generic 2024-03-19T01:30:25.167 DEBUG:teuthology.orchestra.run.smithi084:> sudo apt-get clean 2024-03-19T01:30:25.295 DEBUG:teuthology.orchestra.run.smithi055:> sudo apt-get update 2024-03-19T01:30:25.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo apt-get update 2024-03-19T01:30:25.428 INFO:teuthology.orchestra.run.smithi055.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-19T01:30:25.460 INFO:teuthology.orchestra.run.smithi055.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-19T01:30:25.465 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-19T01:30:25.466 INFO:teuthology.orchestra.run.smithi084.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-19T01:30:25.466 INFO:teuthology.orchestra.run.smithi084.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-19T01:30:25.531 INFO:teuthology.orchestra.run.smithi055.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-19T01:30:25.537 INFO:teuthology.orchestra.run.smithi055.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-19T01:30:25.550 INFO:teuthology.orchestra.run.smithi084.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-19T01:30:25.664 INFO:teuthology.orchestra.run.smithi055.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [592 kB] 2024-03-19T01:30:25.713 INFO:teuthology.orchestra.run.smithi084.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [421 kB] 2024-03-19T01:30:25.746 INFO:teuthology.orchestra.run.smithi055.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,490 kB] 2024-03-19T01:30:25.783 INFO:teuthology.orchestra.run.smithi055.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [288 kB] 2024-03-19T01:30:25.784 INFO:teuthology.orchestra.run.smithi084.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [592 kB] 2024-03-19T01:30:25.787 INFO:teuthology.orchestra.run.smithi055.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-19T01:30:25.788 INFO:teuthology.orchestra.run.smithi055.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-19T01:30:25.788 INFO:teuthology.orchestra.run.smithi055.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,605 kB] 2024-03-19T01:30:25.791 INFO:teuthology.orchestra.run.smithi084.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,246 kB] 2024-03-19T01:30:25.807 INFO:teuthology.orchestra.run.smithi055.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [268 kB] 2024-03-19T01:30:25.810 INFO:teuthology.orchestra.run.smithi055.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-19T01:30:25.810 INFO:teuthology.orchestra.run.smithi055.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [696 kB] 2024-03-19T01:30:25.817 INFO:teuthology.orchestra.run.smithi055.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,058 kB] 2024-03-19T01:30:25.830 INFO:teuthology.orchestra.run.smithi055.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [239 kB] 2024-03-19T01:30:25.833 INFO:teuthology.orchestra.run.smithi055.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-19T01:30:25.837 INFO:teuthology.orchestra.run.smithi084.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [225 kB] 2024-03-19T01:30:25.843 INFO:teuthology.orchestra.run.smithi084.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-19T01:30:25.843 INFO:teuthology.orchestra.run.smithi084.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,531 kB] 2024-03-19T01:30:25.857 INFO:teuthology.orchestra.run.smithi055.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-03-19T01:30:25.857 INFO:teuthology.orchestra.run.smithi055.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,160 B] 2024-03-19T01:30:25.857 INFO:teuthology.orchestra.run.smithi055.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-03-19T01:30:25.857 INFO:teuthology.orchestra.run.smithi055.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-19T01:30:25.857 INFO:teuthology.orchestra.run.smithi055.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-03-19T01:30:25.858 INFO:teuthology.orchestra.run.smithi055.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-03-19T01:30:25.858 INFO:teuthology.orchestra.run.smithi055.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-03-19T01:30:25.858 INFO:teuthology.orchestra.run.smithi055.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-19T01:30:25.858 INFO:teuthology.orchestra.run.smithi055.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-03-19T01:30:25.858 INFO:teuthology.orchestra.run.smithi055.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-03-19T01:30:25.865 INFO:teuthology.orchestra.run.smithi084.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,490 kB] 2024-03-19T01:30:25.870 INFO:teuthology.orchestra.run.smithi084.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-19T01:30:25.871 INFO:teuthology.orchestra.run.smithi084.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [255 kB] 2024-03-19T01:30:25.874 INFO:teuthology.orchestra.run.smithi084.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-19T01:30:25.874 INFO:teuthology.orchestra.run.smithi084.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [851 kB] 2024-03-19T01:30:25.884 INFO:teuthology.orchestra.run.smithi084.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-03-19T01:30:25.886 INFO:teuthology.orchestra.run.smithi055.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-03-19T01:30:25.886 INFO:teuthology.orchestra.run.smithi055.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-19T01:30:25.890 INFO:teuthology.orchestra.run.smithi084.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-03-19T01:30:25.920 INFO:teuthology.orchestra.run.smithi084.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-19T01:30:25.920 INFO:teuthology.orchestra.run.smithi084.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [288 kB] 2024-03-19T01:30:25.920 INFO:teuthology.orchestra.run.smithi084.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-19T01:30:25.920 INFO:teuthology.orchestra.run.smithi084.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-19T01:30:25.920 INFO:teuthology.orchestra.run.smithi084.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-19T01:30:25.921 INFO:teuthology.orchestra.run.smithi084.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-19T01:30:25.924 INFO:teuthology.orchestra.run.smithi084.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-19T01:30:25.925 INFO:teuthology.orchestra.run.smithi084.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-19T01:30:25.926 INFO:teuthology.orchestra.run.smithi084.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,605 kB] 2024-03-19T01:30:25.957 INFO:teuthology.orchestra.run.smithi084.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [268 kB] 2024-03-19T01:30:25.968 INFO:teuthology.orchestra.run.smithi055.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [421 kB] 2024-03-19T01:30:25.969 INFO:teuthology.orchestra.run.smithi084.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-19T01:30:25.970 INFO:teuthology.orchestra.run.smithi084.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,058 kB] 2024-03-19T01:30:25.983 INFO:teuthology.orchestra.run.smithi084.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [696 kB] 2024-03-19T01:30:25.992 INFO:teuthology.orchestra.run.smithi084.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [239 kB] 2024-03-19T01:30:25.996 INFO:teuthology.orchestra.run.smithi084.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-19T01:30:25.996 INFO:teuthology.orchestra.run.smithi084.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-03-19T01:30:25.997 INFO:teuthology.orchestra.run.smithi084.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,160 B] 2024-03-19T01:30:25.997 INFO:teuthology.orchestra.run.smithi084.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-03-19T01:30:25.997 INFO:teuthology.orchestra.run.smithi084.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-19T01:30:26.007 INFO:teuthology.orchestra.run.smithi084.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-03-19T01:30:26.008 INFO:teuthology.orchestra.run.smithi084.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-03-19T01:30:26.019 INFO:teuthology.orchestra.run.smithi084.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-03-19T01:30:26.019 INFO:teuthology.orchestra.run.smithi084.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-19T01:30:26.020 INFO:teuthology.orchestra.run.smithi084.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-03-19T01:30:26.020 INFO:teuthology.orchestra.run.smithi084.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-03-19T01:30:26.020 INFO:teuthology.orchestra.run.smithi084.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-03-19T01:30:26.027 INFO:teuthology.orchestra.run.smithi084.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-19T01:30:26.246 INFO:teuthology.orchestra.run.smithi055.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,246 kB] 2024-03-19T01:30:26.411 INFO:teuthology.orchestra.run.smithi055.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [225 kB] 2024-03-19T01:30:26.423 INFO:teuthology.orchestra.run.smithi055.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-19T01:30:26.424 INFO:teuthology.orchestra.run.smithi055.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,531 kB] 2024-03-19T01:30:26.505 INFO:teuthology.orchestra.run.smithi055.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-19T01:30:26.506 INFO:teuthology.orchestra.run.smithi055.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [255 kB] 2024-03-19T01:30:26.514 INFO:teuthology.orchestra.run.smithi055.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-19T01:30:26.514 INFO:teuthology.orchestra.run.smithi055.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-03-19T01:30:26.531 INFO:teuthology.orchestra.run.smithi055.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [851 kB] 2024-03-19T01:30:26.716 INFO:teuthology.orchestra.run.smithi055.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-03-19T01:30:27.112 INFO:teuthology.orchestra.run.smithi055.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-19T01:30:27.118 INFO:teuthology.orchestra.run.smithi055.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-19T01:30:27.119 INFO:teuthology.orchestra.run.smithi055.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-19T01:30:27.133 INFO:teuthology.orchestra.run.smithi055.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-19T01:30:27.135 INFO:teuthology.orchestra.run.smithi055.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-19T01:30:31.625 INFO:teuthology.orchestra.run.smithi084.stdout:Fetched 12.3 MB in 2s (7,063 kB/s) 2024-03-19T01:30:31.770 INFO:teuthology.orchestra.run.smithi055.stdout:Fetched 12.3 MB in 2s (6,912 kB/s) 2024-03-19T01:30:33.053 INFO:teuthology.orchestra.run.smithi084.stdout:Reading package lists... 2024-03-19T01:30:33.071 DEBUG:teuthology.orchestra.run.smithi084:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-19T01:30:33.125 INFO:teuthology.orchestra.run.smithi084.stdout:Reading package lists... 2024-03-19T01:30:33.149 INFO:teuthology.orchestra.run.smithi055.stdout:Reading package lists... 2024-03-19T01:30:33.168 DEBUG:teuthology.orchestra.run.smithi055:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-19T01:30:33.234 INFO:teuthology.orchestra.run.smithi055.stdout:Reading package lists... 2024-03-19T01:30:33.317 INFO:teuthology.orchestra.run.smithi084.stdout:Building dependency tree... 2024-03-19T01:30:33.318 INFO:teuthology.orchestra.run.smithi084.stdout:Reading state information... 2024-03-19T01:30:33.441 INFO:teuthology.orchestra.run.smithi055.stdout:Building dependency tree... 2024-03-19T01:30:33.441 INFO:teuthology.orchestra.run.smithi055.stdout:Reading state information... 2024-03-19T01:30:33.470 INFO:teuthology.orchestra.run.smithi084.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T01:30:33.470 INFO:teuthology.orchestra.run.smithi084.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T01:30:33.470 INFO:teuthology.orchestra.run.smithi084.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T01:30:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T01:30:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-19T01:30:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: python2.7-minimal 2024-03-19T01:30:33.471 INFO:teuthology.orchestra.run.smithi084.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T01:30:33.472 INFO:teuthology.orchestra.run.smithi084.stdout:The following additional packages will be installed: 2024-03-19T01:30:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-19T01:30:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-03-19T01:30:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-03-19T01:30:33.474 INFO:teuthology.orchestra.run.smithi084.stdout:Suggested packages: 2024-03-19T01:30:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-19T01:30:33.474 INFO:teuthology.orchestra.run.smithi084.stdout:Recommended packages: 2024-03-19T01:30:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: thermald 2024-03-19T01:30:33.516 INFO:teuthology.orchestra.run.smithi084.stdout:The following NEW packages will be installed: 2024-03-19T01:30:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-19T01:30:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-03-19T01:30:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: linux-modules-extra-5.15.0-101-generic 2024-03-19T01:30:33.517 INFO:teuthology.orchestra.run.smithi084.stdout:The following packages will be upgraded: 2024-03-19T01:30:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-19T01:30:33.619 INFO:teuthology.orchestra.run.smithi055.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T01:30:33.619 INFO:teuthology.orchestra.run.smithi055.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T01:30:33.619 INFO:teuthology.orchestra.run.smithi055.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T01:30:33.620 INFO:teuthology.orchestra.run.smithi055.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T01:30:33.620 INFO:teuthology.orchestra.run.smithi055.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-19T01:30:33.620 INFO:teuthology.orchestra.run.smithi055.stdout: python2.7-minimal 2024-03-19T01:30:33.620 INFO:teuthology.orchestra.run.smithi055.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T01:30:33.621 INFO:teuthology.orchestra.run.smithi055.stdout:The following additional packages will be installed: 2024-03-19T01:30:33.621 INFO:teuthology.orchestra.run.smithi055.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-19T01:30:33.621 INFO:teuthology.orchestra.run.smithi055.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-03-19T01:30:33.622 INFO:teuthology.orchestra.run.smithi055.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-03-19T01:30:33.623 INFO:teuthology.orchestra.run.smithi055.stdout:Suggested packages: 2024-03-19T01:30:33.623 INFO:teuthology.orchestra.run.smithi055.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-19T01:30:33.623 INFO:teuthology.orchestra.run.smithi055.stdout:Recommended packages: 2024-03-19T01:30:33.623 INFO:teuthology.orchestra.run.smithi055.stdout: thermald 2024-03-19T01:30:33.674 INFO:teuthology.orchestra.run.smithi055.stdout:The following NEW packages will be installed: 2024-03-19T01:30:33.675 INFO:teuthology.orchestra.run.smithi055.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-19T01:30:33.675 INFO:teuthology.orchestra.run.smithi055.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-03-19T01:30:33.675 INFO:teuthology.orchestra.run.smithi055.stdout: linux-modules-extra-5.15.0-101-generic 2024-03-19T01:30:33.676 INFO:teuthology.orchestra.run.smithi055.stdout:The following packages will be upgraded: 2024-03-19T01:30:33.677 INFO:teuthology.orchestra.run.smithi055.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-19T01:30:33.696 INFO:teuthology.orchestra.run.smithi084.stdout:3 upgraded, 5 newly installed, 0 to remove and 298 not upgraded. 2024-03-19T01:30:33.696 INFO:teuthology.orchestra.run.smithi084.stdout:Need to get 113 MB of archives. 2024-03-19T01:30:33.696 INFO:teuthology.orchestra.run.smithi084.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-19T01:30:33.696 INFO:teuthology.orchestra.run.smithi084.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-03-19T01:30:33.735 INFO:teuthology.orchestra.run.smithi055.stdout:3 upgraded, 5 newly installed, 0 to remove and 298 not upgraded. 2024-03-19T01:30:33.735 INFO:teuthology.orchestra.run.smithi055.stdout:Need to get 113 MB of archives. 2024-03-19T01:30:33.735 INFO:teuthology.orchestra.run.smithi055.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-19T01:30:33.735 INFO:teuthology.orchestra.run.smithi055.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-03-19T01:30:34.238 INFO:teuthology.orchestra.run.smithi055.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-03-19T01:30:34.368 INFO:teuthology.orchestra.run.smithi055.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-03-19T01:30:35.086 INFO:teuthology.orchestra.run.smithi084.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-03-19T01:30:35.473 INFO:teuthology.orchestra.run.smithi084.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-03-19T01:30:35.490 INFO:teuthology.orchestra.run.smithi055.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-03-19T01:30:35.507 INFO:teuthology.orchestra.run.smithi055.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-03-19T01:30:35.511 INFO:teuthology.orchestra.run.smithi055.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-03-19T01:30:35.705 INFO:teuthology.orchestra.run.smithi055.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-03-19T01:30:35.752 INFO:teuthology.orchestra.run.smithi055.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-03-19T01:30:36.155 INFO:teuthology.orchestra.run.smithi055.stdout:Fetched 113 MB in 2s (54.7 MB/s) 2024-03-19T01:30:36.319 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-03-19T01:30:37.649 INFO:teuthology.orchestra.run.smithi084.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-03-19T01:30:37.649 INFO:teuthology.orchestra.run.smithi084.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-03-19T01:30:37.649 INFO:teuthology.orchestra.run.smithi084.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-03-19T01:30:38.356 INFO:teuthology.orchestra.run.smithi055.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-19T01:30:38.360 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T01:30:38.433 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:30:38.545 INFO:teuthology.orchestra.run.smithi084.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-03-19T01:30:38.679 INFO:teuthology.orchestra.run.smithi084.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-03-19T01:30:39.082 INFO:teuthology.orchestra.run.smithi084.stdout:Fetched 113 MB in 5s (22.0 MB/s) 2024-03-19T01:30:39.238 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-03-19T01:30:40.851 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-03-19T01:30:40.872 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T01:30:40.959 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:30:41.259 INFO:teuthology.orchestra.run.smithi084.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-19T01:30:41.264 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T01:30:41.421 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-03-19T01:30:41.442 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T01:30:41.469 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:30:41.479 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:30:43.587 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-03-19T01:30:43.608 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T01:30:43.693 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:30:44.182 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-03-19T01:30:44.202 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T01:30:44.239 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:30:48.345 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-03-19T01:30:48.472 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-19T01:30:48.788 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-03-19T01:30:48.909 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-19T01:30:49.186 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-03-19T01:30:49.209 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-03-19T01:30:49.253 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-19T01:30:51.105 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-03-19T01:30:51.229 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-19T01:30:51.587 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-03-19T01:30:51.724 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-19T01:30:52.044 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-03-19T01:30:52.067 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-03-19T01:30:52.111 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-19T01:30:54.653 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-03-19T01:30:54.689 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T01:30:54.728 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:30:57.909 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-03-19T01:30:58.030 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-19T01:30:58.372 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-19T01:30:58.444 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-03-19T01:30:58.482 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T01:30:58.493 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:30:58.511 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:30:58.619 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-03-19T01:30:58.748 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:31:00.087 INFO:teuthology.orchestra.run.smithi055.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-19T01:31:00.087 INFO:teuthology.orchestra.run.smithi055.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-19T01:31:00.087 INFO:teuthology.orchestra.run.smithi055.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-03-19T01:31:00.087 INFO:teuthology.orchestra.run.smithi055.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-03-19T01:31:00.234 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:31:01.813 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-03-19T01:31:01.938 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-19T01:31:02.026 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:31:02.297 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-19T01:31:02.410 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:31:02.511 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-03-19T01:31:02.665 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:31:03.453 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-03-19T01:31:03.578 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-03-19T01:31:03.705 INFO:teuthology.orchestra.run.smithi055.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:31:03.777 INFO:teuthology.orchestra.run.smithi055.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-19T01:31:03.777 INFO:teuthology.orchestra.run.smithi055.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-03-19T01:31:03.965 INFO:teuthology.orchestra.run.smithi084.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-19T01:31:03.965 INFO:teuthology.orchestra.run.smithi084.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-19T01:31:03.965 INFO:teuthology.orchestra.run.smithi084.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-03-19T01:31:03.965 INFO:teuthology.orchestra.run.smithi084.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-03-19T01:31:04.142 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:31:05.526 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:31:07.453 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-03-19T01:31:07.578 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-03-19T01:31:07.730 INFO:teuthology.orchestra.run.smithi084.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T01:31:07.796 INFO:teuthology.orchestra.run.smithi084.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-19T01:31:07.796 INFO:teuthology.orchestra.run.smithi084.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-03-19T01:31:18.220 INFO:teuthology.orchestra.run.smithi055.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-19T01:31:18.220 INFO:teuthology.orchestra.run.smithi055.stdout:Sourcing file `/etc/default/grub' 2024-03-19T01:31:18.234 INFO:teuthology.orchestra.run.smithi055.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-19T01:31:18.250 INFO:teuthology.orchestra.run.smithi055.stdout:Generating grub configuration file ... 2024-03-19T01:31:18.529 INFO:teuthology.orchestra.run.smithi055.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-19T01:31:18.550 INFO:teuthology.orchestra.run.smithi055.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-19T01:31:18.853 INFO:teuthology.orchestra.run.smithi055.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-19T01:31:18.859 INFO:teuthology.orchestra.run.smithi055.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-19T01:31:18.929 INFO:teuthology.orchestra.run.smithi055.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-19T01:31:18.934 INFO:teuthology.orchestra.run.smithi055.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-19T01:31:19.108 INFO:teuthology.orchestra.run.smithi055.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-19T01:31:19.108 INFO:teuthology.orchestra.run.smithi055.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-19T01:31:19.108 INFO:teuthology.orchestra.run.smithi055.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-19T01:31:19.156 INFO:teuthology.orchestra.run.smithi055.stdout:done 2024-03-19T01:31:19.620 DEBUG:teuthology.orchestra.run.smithi055:> dpkg -s linux-image-generic 2024-03-19T01:31:19.636 INFO:teuthology.orchestra.run.smithi055.stdout:Package: linux-image-generic 2024-03-19T01:31:19.637 INFO:teuthology.orchestra.run.smithi055.stdout:Status: install ok installed 2024-03-19T01:31:19.637 INFO:teuthology.orchestra.run.smithi055.stdout:Priority: optional 2024-03-19T01:31:19.637 INFO:teuthology.orchestra.run.smithi055.stdout:Section: kernel 2024-03-19T01:31:19.637 INFO:teuthology.orchestra.run.smithi055.stdout:Installed-Size: 21 2024-03-19T01:31:19.637 INFO:teuthology.orchestra.run.smithi055.stdout:Maintainer: Ubuntu Kernel Team 2024-03-19T01:31:19.637 INFO:teuthology.orchestra.run.smithi055.stdout:Architecture: amd64 2024-03-19T01:31:19.637 INFO:teuthology.orchestra.run.smithi055.stdout:Source: linux-meta 2024-03-19T01:31:19.637 INFO:teuthology.orchestra.run.smithi055.stdout:Version: 5.15.0.101.98 2024-03-19T01:31:19.637 INFO:teuthology.orchestra.run.smithi055.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-19T01:31:19.637 INFO:teuthology.orchestra.run.smithi055.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-19T01:31:19.637 INFO:teuthology.orchestra.run.smithi055.stdout:Recommends: thermald 2024-03-19T01:31:19.637 INFO:teuthology.orchestra.run.smithi055.stdout:Description: Generic Linux kernel image 2024-03-19T01:31:19.637 INFO:teuthology.orchestra.run.smithi055.stdout: This package will always depend on the latest generic kernel image 2024-03-19T01:31:19.637 INFO:teuthology.orchestra.run.smithi055.stdout: available. 2024-03-19T01:31:19.638 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-03-19T01:31:19.638 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-19T01:31:19.638 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-19T01:31:19.638 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-19T01:31:19.638 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi055.front.sepia.ceph.com, path=None, version=distro) 2024-03-19T01:31:19.638 DEBUG:teuthology.orchestra.run.smithi055:> sudo apt-get clean 2024-03-19T01:31:19.936 DEBUG:teuthology.orchestra.run.smithi055:> sudo apt-get update 2024-03-19T01:31:20.102 INFO:teuthology.orchestra.run.smithi055.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-19T01:31:20.102 INFO:teuthology.orchestra.run.smithi055.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-19T01:31:20.105 INFO:teuthology.orchestra.run.smithi055.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-19T01:31:20.126 INFO:teuthology.orchestra.run.smithi055.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-19T01:31:21.509 INFO:teuthology.orchestra.run.smithi055.stdout:Reading package lists... 2024-03-19T01:31:21.527 DEBUG:teuthology.orchestra.run.smithi055:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-19T01:31:21.580 INFO:teuthology.orchestra.run.smithi055.stdout:Reading package lists... 2024-03-19T01:31:21.758 INFO:teuthology.orchestra.run.smithi055.stdout:Building dependency tree... 2024-03-19T01:31:21.759 INFO:teuthology.orchestra.run.smithi055.stdout:Reading state information... 2024-03-19T01:31:21.790 INFO:teuthology.orchestra.run.smithi084.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-19T01:31:21.790 INFO:teuthology.orchestra.run.smithi084.stdout:Sourcing file `/etc/default/grub' 2024-03-19T01:31:21.819 INFO:teuthology.orchestra.run.smithi084.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-19T01:31:21.839 INFO:teuthology.orchestra.run.smithi084.stdout:Generating grub configuration file ... 2024-03-19T01:31:21.918 INFO:teuthology.orchestra.run.smithi055.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-03-19T01:31:21.918 INFO:teuthology.orchestra.run.smithi055.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T01:31:21.918 INFO:teuthology.orchestra.run.smithi055.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T01:31:21.918 INFO:teuthology.orchestra.run.smithi055.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T01:31:21.918 INFO:teuthology.orchestra.run.smithi055.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T01:31:21.918 INFO:teuthology.orchestra.run.smithi055.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-19T01:31:21.918 INFO:teuthology.orchestra.run.smithi055.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-19T01:31:21.919 INFO:teuthology.orchestra.run.smithi055.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T01:31:21.957 INFO:teuthology.orchestra.run.smithi055.stdout:0 upgraded, 0 newly installed, 0 to remove and 298 not upgraded. 2024-03-19T01:31:21.960 DEBUG:teuthology.orchestra.run.smithi055:> dpkg -s linux-image-generic 2024-03-19T01:31:22.019 INFO:teuthology.orchestra.run.smithi055.stdout:Package: linux-image-generic 2024-03-19T01:31:22.019 INFO:teuthology.orchestra.run.smithi055.stdout:Status: install ok installed 2024-03-19T01:31:22.019 INFO:teuthology.orchestra.run.smithi055.stdout:Priority: optional 2024-03-19T01:31:22.019 INFO:teuthology.orchestra.run.smithi055.stdout:Section: kernel 2024-03-19T01:31:22.019 INFO:teuthology.orchestra.run.smithi055.stdout:Installed-Size: 21 2024-03-19T01:31:22.020 INFO:teuthology.orchestra.run.smithi055.stdout:Maintainer: Ubuntu Kernel Team 2024-03-19T01:31:22.020 INFO:teuthology.orchestra.run.smithi055.stdout:Architecture: amd64 2024-03-19T01:31:22.020 INFO:teuthology.orchestra.run.smithi055.stdout:Source: linux-meta 2024-03-19T01:31:22.020 INFO:teuthology.orchestra.run.smithi055.stdout:Version: 5.15.0.101.98 2024-03-19T01:31:22.020 INFO:teuthology.orchestra.run.smithi055.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-19T01:31:22.020 INFO:teuthology.orchestra.run.smithi055.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-19T01:31:22.020 INFO:teuthology.orchestra.run.smithi055.stdout:Recommends: thermald 2024-03-19T01:31:22.020 INFO:teuthology.orchestra.run.smithi055.stdout:Description: Generic Linux kernel image 2024-03-19T01:31:22.020 INFO:teuthology.orchestra.run.smithi055.stdout: This package will always depend on the latest generic kernel image 2024-03-19T01:31:22.020 INFO:teuthology.orchestra.run.smithi055.stdout: available. 2024-03-19T01:31:22.021 DEBUG:teuthology.orchestra.run.smithi055:> mktemp 2024-03-19T01:31:22.068 INFO:teuthology.orchestra.run.smithi055.stdout:/tmp/tmp.ovJUaVbYPR 2024-03-19T01:31:22.069 DEBUG:teuthology.orchestra.run.smithi055:> sudo cp /boot/grub/grub.cfg /tmp/tmp.ovJUaVbYPR 2024-03-19T01:31:22.117 INFO:teuthology.orchestra.run.smithi084.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-19T01:31:22.123 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0666 /tmp/tmp.ovJUaVbYPR 2024-03-19T01:31:22.141 INFO:teuthology.orchestra.run.smithi084.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-19T01:31:22.236 DEBUG:teuthology.orchestra.remote:smithi055:/tmp/tmp.ovJUaVbYPR is 10KB 2024-03-19T01:31:22.250 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.ovJUaVbYPR 2024-03-19T01:31:22.255 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-19T01:31:22.308 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:31:22.308 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-19T01:31:22.308 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-19T01:31:22.379 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-03-19T01:31:22.379 DEBUG:teuthology.orchestra.run.smithi055:> sudo update-grub 2024-03-19T01:31:22.513 INFO:teuthology.orchestra.run.smithi084.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-19T01:31:22.519 INFO:teuthology.orchestra.run.smithi084.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-19T01:31:22.595 INFO:teuthology.orchestra.run.smithi084.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-19T01:31:22.600 INFO:teuthology.orchestra.run.smithi084.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-19T01:31:22.778 INFO:teuthology.orchestra.run.smithi084.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-19T01:31:22.778 INFO:teuthology.orchestra.run.smithi084.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-19T01:31:22.778 INFO:teuthology.orchestra.run.smithi084.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-19T01:31:22.809 INFO:teuthology.orchestra.run.smithi084.stdout:done 2024-03-19T01:31:23.257 DEBUG:teuthology.orchestra.run.smithi084:> dpkg -s linux-image-generic 2024-03-19T01:31:23.277 INFO:teuthology.orchestra.run.smithi084.stdout:Package: linux-image-generic 2024-03-19T01:31:23.277 INFO:teuthology.orchestra.run.smithi084.stdout:Status: install ok installed 2024-03-19T01:31:23.277 INFO:teuthology.orchestra.run.smithi084.stdout:Priority: optional 2024-03-19T01:31:23.277 INFO:teuthology.orchestra.run.smithi084.stdout:Section: kernel 2024-03-19T01:31:23.277 INFO:teuthology.orchestra.run.smithi084.stdout:Installed-Size: 21 2024-03-19T01:31:23.277 INFO:teuthology.orchestra.run.smithi084.stdout:Maintainer: Ubuntu Kernel Team 2024-03-19T01:31:23.277 INFO:teuthology.orchestra.run.smithi084.stdout:Architecture: amd64 2024-03-19T01:31:23.278 INFO:teuthology.orchestra.run.smithi084.stdout:Source: linux-meta 2024-03-19T01:31:23.278 INFO:teuthology.orchestra.run.smithi084.stdout:Version: 5.15.0.101.98 2024-03-19T01:31:23.278 INFO:teuthology.orchestra.run.smithi084.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-19T01:31:23.278 INFO:teuthology.orchestra.run.smithi084.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-19T01:31:23.278 INFO:teuthology.orchestra.run.smithi084.stdout:Recommends: thermald 2024-03-19T01:31:23.278 INFO:teuthology.orchestra.run.smithi084.stdout:Description: Generic Linux kernel image 2024-03-19T01:31:23.278 INFO:teuthology.orchestra.run.smithi084.stdout: This package will always depend on the latest generic kernel image 2024-03-19T01:31:23.278 INFO:teuthology.orchestra.run.smithi084.stdout: available. 2024-03-19T01:31:23.278 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-03-19T01:31:23.278 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-19T01:31:23.278 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-19T01:31:23.279 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-19T01:31:23.279 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi084.front.sepia.ceph.com, path=None, version=distro) 2024-03-19T01:31:23.279 DEBUG:teuthology.orchestra.run.smithi084:> sudo apt-get clean 2024-03-19T01:31:23.443 DEBUG:teuthology.orchestra.run.smithi084:> sudo apt-get update 2024-03-19T01:31:23.552 INFO:teuthology.orchestra.run.smithi055.stderr:Sourcing file `/etc/default/grub' 2024-03-19T01:31:23.554 INFO:teuthology.orchestra.run.smithi055.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-19T01:31:23.556 INFO:teuthology.orchestra.run.smithi055.stderr:Generating grub configuration file ... 2024-03-19T01:31:23.566 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-19T01:31:23.682 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-19T01:31:23.763 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-19T01:31:23.793 INFO:teuthology.orchestra.run.smithi055.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-19T01:31:23.800 INFO:teuthology.orchestra.run.smithi055.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-19T01:31:23.846 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-19T01:31:24.032 INFO:teuthology.orchestra.run.smithi055.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-19T01:31:24.037 INFO:teuthology.orchestra.run.smithi055.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-19T01:31:24.082 INFO:teuthology.orchestra.run.smithi055.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-19T01:31:24.087 INFO:teuthology.orchestra.run.smithi055.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-19T01:31:24.211 INFO:teuthology.orchestra.run.smithi055.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-19T01:31:24.211 INFO:teuthology.orchestra.run.smithi055.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-19T01:31:24.211 INFO:teuthology.orchestra.run.smithi055.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-19T01:31:24.232 INFO:teuthology.orchestra.run.smithi055.stderr:done 2024-03-19T01:31:24.241 DEBUG:teuthology.orchestra.run.smithi055:> sudo shutdown -r now 2024-03-19T01:31:24.935 INFO:teuthology.orchestra.run.smithi084.stdout:Reading package lists... 2024-03-19T01:31:24.954 DEBUG:teuthology.orchestra.run.smithi084:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-19T01:31:25.014 INFO:teuthology.orchestra.run.smithi084.stdout:Reading package lists... 2024-03-19T01:31:25.210 INFO:teuthology.orchestra.run.smithi084.stdout:Building dependency tree... 2024-03-19T01:31:25.211 INFO:teuthology.orchestra.run.smithi084.stdout:Reading state information... 2024-03-19T01:31:25.382 INFO:teuthology.orchestra.run.smithi084.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-03-19T01:31:25.382 INFO:teuthology.orchestra.run.smithi084.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T01:31:25.382 INFO:teuthology.orchestra.run.smithi084.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T01:31:25.383 INFO:teuthology.orchestra.run.smithi084.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T01:31:25.383 INFO:teuthology.orchestra.run.smithi084.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T01:31:25.383 INFO:teuthology.orchestra.run.smithi084.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-19T01:31:25.383 INFO:teuthology.orchestra.run.smithi084.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-19T01:31:25.383 INFO:teuthology.orchestra.run.smithi084.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T01:31:25.423 INFO:teuthology.orchestra.run.smithi084.stdout:0 upgraded, 0 newly installed, 0 to remove and 298 not upgraded. 2024-03-19T01:31:25.425 DEBUG:teuthology.orchestra.run.smithi084:> dpkg -s linux-image-generic 2024-03-19T01:31:25.440 INFO:teuthology.orchestra.run.smithi084.stdout:Package: linux-image-generic 2024-03-19T01:31:25.441 INFO:teuthology.orchestra.run.smithi084.stdout:Status: install ok installed 2024-03-19T01:31:25.441 INFO:teuthology.orchestra.run.smithi084.stdout:Priority: optional 2024-03-19T01:31:25.441 INFO:teuthology.orchestra.run.smithi084.stdout:Section: kernel 2024-03-19T01:31:25.441 INFO:teuthology.orchestra.run.smithi084.stdout:Installed-Size: 21 2024-03-19T01:31:25.441 INFO:teuthology.orchestra.run.smithi084.stdout:Maintainer: Ubuntu Kernel Team 2024-03-19T01:31:25.441 INFO:teuthology.orchestra.run.smithi084.stdout:Architecture: amd64 2024-03-19T01:31:25.441 INFO:teuthology.orchestra.run.smithi084.stdout:Source: linux-meta 2024-03-19T01:31:25.441 INFO:teuthology.orchestra.run.smithi084.stdout:Version: 5.15.0.101.98 2024-03-19T01:31:25.441 INFO:teuthology.orchestra.run.smithi084.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-19T01:31:25.441 INFO:teuthology.orchestra.run.smithi084.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-19T01:31:25.441 INFO:teuthology.orchestra.run.smithi084.stdout:Recommends: thermald 2024-03-19T01:31:25.441 INFO:teuthology.orchestra.run.smithi084.stdout:Description: Generic Linux kernel image 2024-03-19T01:31:25.441 INFO:teuthology.orchestra.run.smithi084.stdout: This package will always depend on the latest generic kernel image 2024-03-19T01:31:25.441 INFO:teuthology.orchestra.run.smithi084.stdout: available. 2024-03-19T01:31:25.442 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2024-03-19T01:31:25.488 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.CgLNdJ8EDI 2024-03-19T01:31:25.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo cp /boot/grub/grub.cfg /tmp/tmp.CgLNdJ8EDI 2024-03-19T01:31:25.543 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.CgLNdJ8EDI 2024-03-19T01:31:25.651 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.CgLNdJ8EDI is 10KB 2024-03-19T01:31:25.663 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.CgLNdJ8EDI 2024-03-19T01:31:25.669 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-19T01:31:25.723 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:31:25.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-19T01:31:25.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-19T01:31:25.799 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-03-19T01:31:25.799 DEBUG:teuthology.orchestra.run.smithi084:> sudo update-grub 2024-03-19T01:31:27.147 INFO:teuthology.orchestra.run.smithi084.stderr:Sourcing file `/etc/default/grub' 2024-03-19T01:31:27.148 INFO:teuthology.orchestra.run.smithi084.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-19T01:31:27.151 INFO:teuthology.orchestra.run.smithi084.stderr:Generating grub configuration file ... 2024-03-19T01:31:27.333 INFO:teuthology.orchestra.run.smithi084.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-19T01:31:27.340 INFO:teuthology.orchestra.run.smithi084.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-19T01:31:27.602 INFO:teuthology.orchestra.run.smithi084.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-19T01:31:27.607 INFO:teuthology.orchestra.run.smithi084.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-19T01:31:27.648 INFO:teuthology.orchestra.run.smithi084.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-19T01:31:27.653 INFO:teuthology.orchestra.run.smithi084.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-19T01:31:27.772 INFO:teuthology.orchestra.run.smithi084.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-19T01:31:27.772 INFO:teuthology.orchestra.run.smithi084.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-19T01:31:27.773 INFO:teuthology.orchestra.run.smithi084.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-19T01:31:27.784 INFO:teuthology.orchestra.run.smithi084.stderr:done 2024-03-19T01:31:27.791 DEBUG:teuthology.orchestra.run.smithi084:> sudo shutdown -r now 2024-03-19T01:31:54.266 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-19T01:31:54.267 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-19T01:31:54.268 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:31:57.797 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-19T01:31:57.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-19T01:31:57.798 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:32:12.702 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.55 2024-03-19T01:32:16.286 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-03-19T01:32:21.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-19T01:32:21.707 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:32:25.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-19T01:32:25.291 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:32:28.354 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-03-19T01:32:40.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-19T01:32:40.367 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:32:58.782 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-03-19T01:33:13.794 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-19T01:33:13.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:33:14.226 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-03-19T01:33:14.842 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-19T01:33:14.842 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-101-generic"... 2024-03-19T01:33:14.843 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-03-19T01:33:14.888 INFO:teuthology.orchestra.run.smithi084.stdout:5.15.0-101-generic 2024-03-19T01:33:14.889 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-03-19T01:33:14.889 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-19T01:33:14.889 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-19T01:33:15.890 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-19T01:33:15.891 DEBUG:teuthology.orchestra.run.smithi084:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-19T01:33:15.951 INFO:teuthology.orchestra.run.smithi084.stdout:ttyS1 2024-03-19T01:33:15.970 DEBUG:teuthology.parallel:result is None 2024-03-19T01:33:21.709 DEBUG:teuthology.orchestra.remote:timed out 2024-03-19T01:33:33.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-19T01:33:33.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:33:33.941 DEBUG:teuthology.orchestra.run.smithi055:> true 2024-03-19T01:33:34.535 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-19T01:33:34.535 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-101-generic"... 2024-03-19T01:33:34.535 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2024-03-19T01:33:34.580 INFO:teuthology.orchestra.run.smithi055.stdout:5.15.0-101-generic 2024-03-19T01:33:34.581 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-03-19T01:33:34.581 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-19T01:33:34.581 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-19T01:33:35.582 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-19T01:33:35.582 DEBUG:teuthology.orchestra.run.smithi055:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-19T01:33:35.654 INFO:teuthology.orchestra.run.smithi055.stdout:ttyS1 2024-03-19T01:33:35.675 DEBUG:teuthology.parallel:result is None 2024-03-19T01:33:35.676 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-19T01:33:35.684 INFO:teuthology.task.internal:Creating test directory... 2024-03-19T01:33:35.684 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T01:33:35.686 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T01:33:35.693 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-19T01:33:35.737 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-19T01:33:35.745 INFO:teuthology.task.internal:Creating archive directory... 2024-03-19T01:33:35.745 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T01:33:35.748 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T01:33:35.856 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-19T01:33:35.863 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-19T01:33:35.863 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T01:33:35.866 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T01:33:35.882 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T01:33:35.884 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T01:33:35.890 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T01:33:35.892 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T01:33:35.894 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-19T01:33:35.901 INFO:teuthology.task.internal:Configuring sudo... 2024-03-19T01:33:35.901 DEBUG:teuthology.orchestra.run.smithi055:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T01:33:35.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T01:33:35.955 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-19T01:33:35.964 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-19T01:33:35.964 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T01:33:35.998 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T01:33:36.003 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T01:33:36.053 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T01:33:36.101 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:33:36.101 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T01:33:36.163 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T01:33:36.169 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T01:33:36.217 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:33:36.217 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T01:33:36.291 DEBUG:teuthology.orchestra.run.smithi055:> sudo service rsyslog restart 2024-03-19T01:33:36.293 DEBUG:teuthology.orchestra.run.smithi084:> sudo service rsyslog restart 2024-03-19T01:33:36.366 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-19T01:33:36.374 INFO:teuthology.task.internal:Starting timer... 2024-03-19T01:33:36.374 INFO:teuthology.run_tasks:Running task pcp... 2024-03-19T01:33:36.390 INFO:teuthology.run_tasks:Running task selinux... 2024-03-19T01:33:36.400 DEBUG:teuthology.task.selinux:Excluding smithi055: OS 'ubuntu' does not support SELinux 2024-03-19T01:33:36.400 DEBUG:teuthology.task.selinux:Excluding smithi084: OS 'ubuntu' does not support SELinux 2024-03-19T01:33:36.401 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-19T01:33:36.401 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-03-19T01:33:36.401 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-19T01:33:36.401 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-19T01:33:36.411 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-19T01:33:36.428 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-19T01:33:36.429 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi055.front.sepia.ceph.com,smithi084.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-19T01:40:37.566 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi055.front.sepia.ceph.com'), Remote(name='ubuntu@smithi084.front.sepia.ceph.com')] 2024-03-19T01:40:37.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-19T01:40:37.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:40:37.654 DEBUG:teuthology.orchestra.run.smithi055:> true 2024-03-19T01:40:37.734 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-19T01:40:37.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-19T01:40:37.735 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:40:37.826 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-03-19T01:40:37.902 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-19T01:40:37.903 INFO:teuthology.run_tasks:Running task clock... 2024-03-19T01:40:37.915 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-19T01:40:37.915 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T01:40:37.915 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T01:40:37.918 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T01:40:37.918 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T01:40:37.948 INFO:teuthology.orchestra.run.smithi055.stdout:19 Mar 01:40:37 ntpd[17053]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-19T01:40:37.949 INFO:teuthology.orchestra.run.smithi055.stdout:19 Mar 01:40:37 ntpd[17053]: Command line: ntpd -gq 2024-03-19T01:40:37.949 INFO:teuthology.orchestra.run.smithi055.stdout:19 Mar 01:40:37 ntpd[17053]: ---------------------------------------------------- 2024-03-19T01:40:37.949 INFO:teuthology.orchestra.run.smithi055.stdout:19 Mar 01:40:37 ntpd[17053]: ntp-4 is maintained by Network Time Foundation, 2024-03-19T01:40:37.949 INFO:teuthology.orchestra.run.smithi055.stdout:19 Mar 01:40:37 ntpd[17053]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-19T01:40:37.949 INFO:teuthology.orchestra.run.smithi055.stdout:19 Mar 01:40:37 ntpd[17053]: corporation. Support and training for ntp-4 are 2024-03-19T01:40:37.949 INFO:teuthology.orchestra.run.smithi055.stdout:19 Mar 01:40:37 ntpd[17053]: available at https://www.nwtime.org/support 2024-03-19T01:40:37.949 INFO:teuthology.orchestra.run.smithi055.stdout:19 Mar 01:40:37 ntpd[17053]: ---------------------------------------------------- 2024-03-19T01:40:37.950 INFO:teuthology.orchestra.run.smithi055.stdout:19 Mar 01:40:37 ntpd[17053]: proto: precision = 0.063 usec (-24) 2024-03-19T01:40:37.950 INFO:teuthology.orchestra.run.smithi055.stdout:19 Mar 01:40:37 ntpd[17053]: basedate set to 2022-02-04 2024-03-19T01:40:37.950 INFO:teuthology.orchestra.run.smithi055.stdout:19 Mar 01:40:37 ntpd[17053]: gps base set to 2022-02-06 (week 2196) 2024-03-19T01:40:37.950 INFO:teuthology.orchestra.run.smithi055.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-19T01:40:37.951 INFO:teuthology.orchestra.run.smithi055.stdout:19 Mar 01:40:37 ntpd[17053]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-19T01:40:37.951 INFO:teuthology.orchestra.run.smithi055.stdout:19 Mar 01:40:37 ntpd[17053]: restrict ::: KOD does nothing without LIMITED. 2024-03-19T01:40:37.951 INFO:teuthology.orchestra.run.smithi055.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-19T01:40:37.952 INFO:teuthology.orchestra.run.smithi055.stdout:19 Mar 01:40:37 ntpd[17053]: Listen and drop on 0 v6wildcard [::]:123 2024-03-19T01:40:37.952 INFO:teuthology.orchestra.run.smithi055.stdout:19 Mar 01:40:37 ntpd[17053]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-19T01:40:37.952 INFO:teuthology.orchestra.run.smithi055.stdout:19 Mar 01:40:37 ntpd[17053]: Listen normally on 2 lo 127.0.0.1:123 2024-03-19T01:40:37.953 INFO:teuthology.orchestra.run.smithi055.stdout:19 Mar 01:40:37 ntpd[17053]: Listen normally on 3 ens1f0 172.21.15.55:123 2024-03-19T01:40:37.953 INFO:teuthology.orchestra.run.smithi055.stdout:19 Mar 01:40:37 ntpd[17053]: Listen normally on 4 lo [::1]:123 2024-03-19T01:40:37.953 INFO:teuthology.orchestra.run.smithi055.stdout:19 Mar 01:40:37 ntpd[17053]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fef8:2606%3]:123 2024-03-19T01:40:37.953 INFO:teuthology.orchestra.run.smithi055.stdout:19 Mar 01:40:37 ntpd[17053]: Listening on routing socket on fd #22 for interface updates 2024-03-19T01:40:37.970 INFO:teuthology.orchestra.run.smithi084.stdout:19 Mar 01:40:37 ntpd[17173]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-19T01:40:37.971 INFO:teuthology.orchestra.run.smithi084.stdout:19 Mar 01:40:37 ntpd[17173]: Command line: ntpd -gq 2024-03-19T01:40:37.971 INFO:teuthology.orchestra.run.smithi084.stdout:19 Mar 01:40:37 ntpd[17173]: ---------------------------------------------------- 2024-03-19T01:40:37.971 INFO:teuthology.orchestra.run.smithi084.stdout:19 Mar 01:40:37 ntpd[17173]: ntp-4 is maintained by Network Time Foundation, 2024-03-19T01:40:37.971 INFO:teuthology.orchestra.run.smithi084.stdout:19 Mar 01:40:37 ntpd[17173]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-19T01:40:37.971 INFO:teuthology.orchestra.run.smithi084.stdout:19 Mar 01:40:37 ntpd[17173]: corporation. Support and training for ntp-4 are 2024-03-19T01:40:37.971 INFO:teuthology.orchestra.run.smithi084.stdout:19 Mar 01:40:37 ntpd[17173]: available at https://www.nwtime.org/support 2024-03-19T01:40:37.971 INFO:teuthology.orchestra.run.smithi084.stdout:19 Mar 01:40:37 ntpd[17173]: ---------------------------------------------------- 2024-03-19T01:40:37.972 INFO:teuthology.orchestra.run.smithi084.stdout:19 Mar 01:40:37 ntpd[17173]: proto: precision = 0.043 usec (-24) 2024-03-19T01:40:37.972 INFO:teuthology.orchestra.run.smithi084.stdout:19 Mar 01:40:37 ntpd[17173]: basedate set to 2022-02-04 2024-03-19T01:40:37.972 INFO:teuthology.orchestra.run.smithi084.stdout:19 Mar 01:40:37 ntpd[17173]: gps base set to 2022-02-06 (week 2196) 2024-03-19T01:40:37.972 INFO:teuthology.orchestra.run.smithi084.stdout:19 Mar 01:40:37 ntpd[17173]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-19T01:40:37.972 INFO:teuthology.orchestra.run.smithi084.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-19T01:40:37.972 INFO:teuthology.orchestra.run.smithi084.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-19T01:40:37.973 INFO:teuthology.orchestra.run.smithi084.stdout:19 Mar 01:40:37 ntpd[17173]: restrict ::: KOD does nothing without LIMITED. 2024-03-19T01:40:37.973 INFO:teuthology.orchestra.run.smithi084.stdout:19 Mar 01:40:37 ntpd[17173]: Listen and drop on 0 v6wildcard [::]:123 2024-03-19T01:40:37.974 INFO:teuthology.orchestra.run.smithi084.stdout:19 Mar 01:40:37 ntpd[17173]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-19T01:40:37.974 INFO:teuthology.orchestra.run.smithi084.stdout:19 Mar 01:40:37 ntpd[17173]: Listen normally on 2 lo 127.0.0.1:123 2024-03-19T01:40:37.974 INFO:teuthology.orchestra.run.smithi084.stdout:19 Mar 01:40:37 ntpd[17173]: Listen normally on 3 enp3s0f1 172.21.15.84:123 2024-03-19T01:40:37.974 INFO:teuthology.orchestra.run.smithi084.stdout:19 Mar 01:40:37 ntpd[17173]: Listen normally on 4 lo [::1]:123 2024-03-19T01:40:37.974 INFO:teuthology.orchestra.run.smithi084.stdout:19 Mar 01:40:37 ntpd[17173]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d751%5]:123 2024-03-19T01:40:37.974 INFO:teuthology.orchestra.run.smithi084.stdout:19 Mar 01:40:37 ntpd[17173]: Listening on routing socket on fd #22 for interface updates 2024-03-19T01:40:38.949 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:38 ntpd[17053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:38.949 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:38 ntpd[17053]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:38.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:38 ntpd[17053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:38.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:38 ntpd[17053]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:38.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:38 ntpd[17053]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:40:38.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:38 ntpd[17053]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:40:38.971 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:38 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:38.971 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:38 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:38.971 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:38 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:38.971 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:38 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:38.971 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:38 ntpd[17173]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:40:38.971 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:38 ntpd[17173]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:40:39.949 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:39 ntpd[17053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:39.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:39 ntpd[17053]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:39.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:39 ntpd[17053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:39.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:39 ntpd[17053]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:39.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:39 ntpd[17053]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:40:39.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:39 ntpd[17053]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:40:40.949 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:40 ntpd[17053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:40.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:40 ntpd[17053]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:40.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:40 ntpd[17053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:40.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:40 ntpd[17053]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:40.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:40 ntpd[17053]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:40:40.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:40 ntpd[17053]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:40:40.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:40 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:40.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:40 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:40.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:40 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:40.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:40 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:40.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:40 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:40.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:40 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:40.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:40 ntpd[17173]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:40:40.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:40 ntpd[17173]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:40:40.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:40 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:40.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:40 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:40.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:40 ntpd[17173]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:40:40.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:40 ntpd[17173]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:40:41.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:41 ntpd[17053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:41.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:41 ntpd[17053]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:41.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:41 ntpd[17053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:41.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:41 ntpd[17053]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:41.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:41 ntpd[17053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:41.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:41 ntpd[17053]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:41.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:41 ntpd[17053]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:40:41.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:41 ntpd[17053]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:40:41.971 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:41 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:41.971 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:41 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:41.971 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:41 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:41.971 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:41 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:41.971 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:41 ntpd[17173]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:40:41.971 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:41 ntpd[17173]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:40:42.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:42 ntpd[17053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:42.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:42 ntpd[17053]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:42.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:42 ntpd[17053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:42.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:42 ntpd[17053]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:42.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:42 ntpd[17053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:42.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:42 ntpd[17053]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:42.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:42 ntpd[17053]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:40:42.951 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:42 ntpd[17053]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:40:42.951 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:42 ntpd[17053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:42.951 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:42 ntpd[17053]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:42.951 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:42 ntpd[17053]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:40:42.951 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:42 ntpd[17053]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:40:42.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:42 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:42.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:42 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:42.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:42 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:42.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:42 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:42.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:42 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:42.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:42 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:42.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:42 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:42.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:42 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:42.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:42 ntpd[17173]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:40:42.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:42 ntpd[17173]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:40:42.973 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:42 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:42.973 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:42 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:42.973 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:42 ntpd[17173]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:40:42.973 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:42 ntpd[17173]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:40:43.949 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:43 ntpd[17053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:43.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:43 ntpd[17053]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:43.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:43 ntpd[17053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:43.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:43 ntpd[17053]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:43.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:43 ntpd[17053]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:40:43.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:43 ntpd[17053]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:40:43.971 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:43 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:43.971 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:43 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:43.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:43 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:43.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:43 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:43.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:43 ntpd[17173]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:40:43.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:43 ntpd[17173]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:40:44.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:44 ntpd[17053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:44.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:44 ntpd[17053]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:44.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:44 ntpd[17053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:44.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:44 ntpd[17053]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:44.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:44 ntpd[17053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:44.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:44 ntpd[17053]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:44.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:44 ntpd[17053]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:40:44.950 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:44 ntpd[17053]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:40:44.951 INFO:teuthology.orchestra.run.smithi055.stdout:19 Mar 01:40:44 ntpd[17053]: ntpd: time slew +0.001231 s 2024-03-19T01:40:44.951 INFO:teuthology.orchestra.run.smithi055.stdout:ntpd: time slew +0.001231s 2024-03-19T01:40:44.951 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:44 ntpd[17053]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-19T01:40:44.951 INFO:teuthology.orchestra.run.smithi055.stderr:19 Mar 01:40:44 ntpd[17053]: can't open /var/log/ntpstats/loopstats.20240319: Permission denied 2024-03-19T01:40:44.971 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:44 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:44.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:44 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:44.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:44 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:44.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:44 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:44.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:44 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T01:40:44.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:44 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T01:40:44.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:44 ntpd[17173]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T01:40:44.972 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:44 ntpd[17173]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T01:40:44.972 INFO:teuthology.orchestra.run.smithi084.stdout:19 Mar 01:40:44 ntpd[17173]: ntpd: time slew -0.002093 s 2024-03-19T01:40:44.972 INFO:teuthology.orchestra.run.smithi084.stdout:ntpd: time slew -0.002093s 2024-03-19T01:40:44.973 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:44 ntpd[17173]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-19T01:40:44.973 INFO:teuthology.orchestra.run.smithi084.stderr:19 Mar 01:40:44 ntpd[17173]: can't open /var/log/ntpstats/loopstats.20240319: Permission denied 2024-03-19T01:40:44.999 INFO:teuthology.orchestra.run.smithi055.stdout: remote refid st t when poll reach delay offset jitter 2024-03-19T01:40:44.999 INFO:teuthology.orchestra.run.smithi055.stdout:============================================================================== 2024-03-19T01:40:44.999 INFO:teuthology.orchestra.run.smithi055.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T01:40:44.999 INFO:teuthology.orchestra.run.smithi055.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T01:40:44.999 INFO:teuthology.orchestra.run.smithi055.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T01:40:44.999 INFO:teuthology.orchestra.run.smithi055.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T01:40:45.021 INFO:teuthology.orchestra.run.smithi084.stdout: remote refid st t when poll reach delay offset jitter 2024-03-19T01:40:45.021 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================== 2024-03-19T01:40:45.022 INFO:teuthology.orchestra.run.smithi084.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T01:40:45.022 INFO:teuthology.orchestra.run.smithi084.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T01:40:45.022 INFO:teuthology.orchestra.run.smithi084.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T01:40:45.022 INFO:teuthology.orchestra.run.smithi084.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T01:40:45.023 INFO:teuthology.run_tasks:Running task install... 2024-03-19T01:40:45.031 DEBUG:teuthology.task.install:project ceph 2024-03-19T01:40:45.031 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'}} 2024-03-19T01:40:45.031 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'debuginfo': True, 'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'} 2024-03-19T01:40:45.031 INFO:teuthology.task.install:Using flavor: default 2024-03-19T01:40:45.037 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-19T01:40:45.037 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-19T01:40:45.038 DEBUG:teuthology.orchestra.run.smithi055:> sudo apt-key list | grep Ceph 2024-03-19T01:40:45.038 DEBUG:teuthology.orchestra.run.smithi084:> sudo apt-key list | grep Ceph 2024-03-19T01:40:45.098 INFO:teuthology.orchestra.run.smithi055.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-19T01:40:45.116 INFO:teuthology.orchestra.run.smithi084.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-19T01:40:45.124 INFO:teuthology.orchestra.run.smithi055.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-19T01:40:45.124 INFO:teuthology.orchestra.run.smithi055.stdout:uid [ unknown] Ceph.com (release key) 2024-03-19T01:40:45.124 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, ceph-common-dbg, ceph-fuse-dbg, ceph-mds-dbg, ceph-mgr-dbg, ceph-mon-dbg, ceph-osd-dbg, ceph-test-dbg, libcephfs2-dbg, librados2-dbg, libradosstriper1-dbg, librbd1-dbg, librgw2-dbg, radosgw-dbg, rbd-fuse-dbg, rbd-mirror-dbg, rbd-nbd-dbg, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-03-19T01:40:45.125 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-03-19T01:40:45.125 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T01:40:45.144 INFO:teuthology.orchestra.run.smithi084.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-19T01:40:45.144 INFO:teuthology.orchestra.run.smithi084.stdout:uid [ unknown] Ceph.com (release key) 2024-03-19T01:40:45.144 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, ceph-common-dbg, ceph-fuse-dbg, ceph-mds-dbg, ceph-mgr-dbg, ceph-mon-dbg, ceph-osd-dbg, ceph-test-dbg, libcephfs2-dbg, librados2-dbg, libradosstriper1-dbg, librbd1-dbg, librgw2-dbg, radosgw-dbg, rbd-fuse-dbg, rbd-mirror-dbg, rbd-nbd-dbg, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-03-19T01:40:45.144 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-03-19T01:40:45.145 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T01:40:45.270 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default/ 2024-03-19T01:40:45.270 INFO:teuthology.task.install.deb:Package version is 19.0.0-2206-gb6ae2491-1jammy 2024-03-19T01:40:45.358 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default/ 2024-03-19T01:40:45.358 INFO:teuthology.task.install.deb:Package version is 19.0.0-2206-gb6ae2491-1jammy 2024-03-19T01:40:45.383 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:40:45.383 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-19T01:40:45.398 DEBUG:teuthology.orchestra.run.smithi055:> sudo apt-get update 2024-03-19T01:40:45.529 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:40:45.529 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-19T01:40:45.543 DEBUG:teuthology.orchestra.run.smithi084:> sudo apt-get update 2024-03-19T01:40:45.573 INFO:teuthology.orchestra.run.smithi055.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-19T01:40:45.585 INFO:teuthology.orchestra.run.smithi055.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-19T01:40:45.606 INFO:teuthology.orchestra.run.smithi055.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-19T01:40:45.683 INFO:teuthology.orchestra.run.smithi055.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy InRelease 2024-03-19T01:40:45.688 INFO:teuthology.orchestra.run.smithi055.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-19T01:40:45.715 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-19T01:40:45.716 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-19T01:40:45.728 INFO:teuthology.orchestra.run.smithi055.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2024-03-19T01:40:45.746 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-19T01:40:45.763 INFO:teuthology.orchestra.run.smithi084.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy InRelease 2024-03-19T01:40:45.776 INFO:teuthology.orchestra.run.smithi055.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-19T01:40:45.777 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-19T01:40:45.784 INFO:teuthology.orchestra.run.smithi084.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2024-03-19T01:40:45.805 INFO:teuthology.orchestra.run.smithi084.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-19T01:40:45.813 INFO:teuthology.orchestra.run.smithi055.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-19T01:40:45.826 INFO:teuthology.orchestra.run.smithi084.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,412 B] 2024-03-19T01:40:45.853 INFO:teuthology.orchestra.run.smithi084.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-19T01:40:45.855 INFO:teuthology.orchestra.run.smithi055.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,412 B] 2024-03-19T01:40:45.955 INFO:teuthology.orchestra.run.smithi055.stdout:Fetched 28.2 kB in 0s (74.7 kB/s) 2024-03-19T01:40:46.077 INFO:teuthology.orchestra.run.smithi084.stdout:Fetched 28.2 kB in 0s (78.8 kB/s) 2024-03-19T01:40:46.924 INFO:teuthology.orchestra.run.smithi055.stdout:Reading package lists... 2024-03-19T01:40:46.941 DEBUG:teuthology.orchestra.run.smithi055:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2206-gb6ae2491-1jammy cephadm=19.0.0-2206-gb6ae2491-1jammy ceph-mds=19.0.0-2206-gb6ae2491-1jammy ceph-mgr=19.0.0-2206-gb6ae2491-1jammy ceph-common=19.0.0-2206-gb6ae2491-1jammy ceph-fuse=19.0.0-2206-gb6ae2491-1jammy ceph-test=19.0.0-2206-gb6ae2491-1jammy ceph-volume=19.0.0-2206-gb6ae2491-1jammy radosgw=19.0.0-2206-gb6ae2491-1jammy python3-rados=19.0.0-2206-gb6ae2491-1jammy python3-rgw=19.0.0-2206-gb6ae2491-1jammy python3-cephfs=19.0.0-2206-gb6ae2491-1jammy python3-rbd=19.0.0-2206-gb6ae2491-1jammy libcephfs2=19.0.0-2206-gb6ae2491-1jammy libcephfs-dev=19.0.0-2206-gb6ae2491-1jammy librados2=19.0.0-2206-gb6ae2491-1jammy librbd1=19.0.0-2206-gb6ae2491-1jammy rbd-fuse=19.0.0-2206-gb6ae2491-1jammy ceph-common-dbg=19.0.0-2206-gb6ae2491-1jammy ceph-fuse-dbg=19.0.0-2206-gb6ae2491-1jammy ceph-mds-dbg=19.0.0-2206-gb6ae2491-1jammy ceph-mgr-dbg=19.0.0-2206-gb6ae2491-1jammy ceph-mon-dbg=19.0.0-2206-gb6ae2491-1jammy ceph-osd-dbg=19.0.0-2206-gb6ae2491-1jammy ceph-test-dbg=19.0.0-2206-gb6ae2491-1jammy libcephfs2-dbg=19.0.0-2206-gb6ae2491-1jammy librados2-dbg=19.0.0-2206-gb6ae2491-1jammy libradosstriper1-dbg=19.0.0-2206-gb6ae2491-1jammy librbd1-dbg=19.0.0-2206-gb6ae2491-1jammy librgw2-dbg=19.0.0-2206-gb6ae2491-1jammy radosgw-dbg=19.0.0-2206-gb6ae2491-1jammy rbd-fuse-dbg=19.0.0-2206-gb6ae2491-1jammy rbd-mirror-dbg=19.0.0-2206-gb6ae2491-1jammy rbd-nbd-dbg=19.0.0-2206-gb6ae2491-1jammy python3-cephfs=19.0.0-2206-gb6ae2491-1jammy cephfs-shell=19.0.0-2206-gb6ae2491-1jammy cephfs-top=19.0.0-2206-gb6ae2491-1jammy cephfs-mirror=19.0.0-2206-gb6ae2491-1jammy 2024-03-19T01:40:46.969 INFO:teuthology.orchestra.run.smithi084.stdout:Reading package lists... 2024-03-19T01:40:46.986 DEBUG:teuthology.orchestra.run.smithi084:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2206-gb6ae2491-1jammy cephadm=19.0.0-2206-gb6ae2491-1jammy ceph-mds=19.0.0-2206-gb6ae2491-1jammy ceph-mgr=19.0.0-2206-gb6ae2491-1jammy ceph-common=19.0.0-2206-gb6ae2491-1jammy ceph-fuse=19.0.0-2206-gb6ae2491-1jammy ceph-test=19.0.0-2206-gb6ae2491-1jammy ceph-volume=19.0.0-2206-gb6ae2491-1jammy radosgw=19.0.0-2206-gb6ae2491-1jammy python3-rados=19.0.0-2206-gb6ae2491-1jammy python3-rgw=19.0.0-2206-gb6ae2491-1jammy python3-cephfs=19.0.0-2206-gb6ae2491-1jammy python3-rbd=19.0.0-2206-gb6ae2491-1jammy libcephfs2=19.0.0-2206-gb6ae2491-1jammy libcephfs-dev=19.0.0-2206-gb6ae2491-1jammy librados2=19.0.0-2206-gb6ae2491-1jammy librbd1=19.0.0-2206-gb6ae2491-1jammy rbd-fuse=19.0.0-2206-gb6ae2491-1jammy ceph-common-dbg=19.0.0-2206-gb6ae2491-1jammy ceph-fuse-dbg=19.0.0-2206-gb6ae2491-1jammy ceph-mds-dbg=19.0.0-2206-gb6ae2491-1jammy ceph-mgr-dbg=19.0.0-2206-gb6ae2491-1jammy ceph-mon-dbg=19.0.0-2206-gb6ae2491-1jammy ceph-osd-dbg=19.0.0-2206-gb6ae2491-1jammy ceph-test-dbg=19.0.0-2206-gb6ae2491-1jammy libcephfs2-dbg=19.0.0-2206-gb6ae2491-1jammy librados2-dbg=19.0.0-2206-gb6ae2491-1jammy libradosstriper1-dbg=19.0.0-2206-gb6ae2491-1jammy librbd1-dbg=19.0.0-2206-gb6ae2491-1jammy librgw2-dbg=19.0.0-2206-gb6ae2491-1jammy radosgw-dbg=19.0.0-2206-gb6ae2491-1jammy rbd-fuse-dbg=19.0.0-2206-gb6ae2491-1jammy rbd-mirror-dbg=19.0.0-2206-gb6ae2491-1jammy rbd-nbd-dbg=19.0.0-2206-gb6ae2491-1jammy python3-cephfs=19.0.0-2206-gb6ae2491-1jammy cephfs-shell=19.0.0-2206-gb6ae2491-1jammy cephfs-top=19.0.0-2206-gb6ae2491-1jammy cephfs-mirror=19.0.0-2206-gb6ae2491-1jammy 2024-03-19T01:40:47.003 INFO:teuthology.orchestra.run.smithi055.stdout:Reading package lists... 2024-03-19T01:40:47.047 INFO:teuthology.orchestra.run.smithi084.stdout:Reading package lists... 2024-03-19T01:40:47.203 INFO:teuthology.orchestra.run.smithi055.stdout:Building dependency tree... 2024-03-19T01:40:47.203 INFO:teuthology.orchestra.run.smithi055.stdout:Reading state information... 2024-03-19T01:40:47.250 INFO:teuthology.orchestra.run.smithi084.stdout:Building dependency tree... 2024-03-19T01:40:47.250 INFO:teuthology.orchestra.run.smithi084.stdout:Reading state information... 2024-03-19T01:40:47.388 INFO:teuthology.orchestra.run.smithi055.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T01:40:47.388 INFO:teuthology.orchestra.run.smithi055.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T01:40:47.388 INFO:teuthology.orchestra.run.smithi055.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T01:40:47.388 INFO:teuthology.orchestra.run.smithi055.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T01:40:47.389 INFO:teuthology.orchestra.run.smithi055.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-19T01:40:47.389 INFO:teuthology.orchestra.run.smithi055.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-19T01:40:47.389 INFO:teuthology.orchestra.run.smithi055.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T01:40:47.390 INFO:teuthology.orchestra.run.smithi055.stdout:The following additional packages will be installed: 2024-03-19T01:40:47.390 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-19T01:40:47.390 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-19T01:40:47.390 INFO:teuthology.orchestra.run.smithi055.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-19T01:40:47.390 INFO:teuthology.orchestra.run.smithi055.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-19T01:40:47.391 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-19T01:40:47.391 INFO:teuthology.orchestra.run.smithi055.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-19T01:40:47.391 INFO:teuthology.orchestra.run.smithi055.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-19T01:40:47.391 INFO:teuthology.orchestra.run.smithi055.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-19T01:40:47.391 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-19T01:40:47.391 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-03-19T01:40:47.392 INFO:teuthology.orchestra.run.smithi055.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-03-19T01:40:47.392 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-03-19T01:40:47.392 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-03-19T01:40:47.392 INFO:teuthology.orchestra.run.smithi055.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-03-19T01:40:47.392 INFO:teuthology.orchestra.run.smithi055.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-03-19T01:40:47.392 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-03-19T01:40:47.392 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-03-19T01:40:47.392 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-03-19T01:40:47.392 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-03-19T01:40:47.392 INFO:teuthology.orchestra.run.smithi055.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-03-19T01:40:47.392 INFO:teuthology.orchestra.run.smithi055.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-03-19T01:40:47.392 INFO:teuthology.orchestra.run.smithi055.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-03-19T01:40:47.392 INFO:teuthology.orchestra.run.smithi055.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-03-19T01:40:47.392 INFO:teuthology.orchestra.run.smithi055.stdout: qttranslations5-l10n rbd-mirror rbd-nbd socat unzip xmlstarlet zip 2024-03-19T01:40:47.393 INFO:teuthology.orchestra.run.smithi055.stdout:Suggested packages: 2024-03-19T01:40:47.393 INFO:teuthology.orchestra.run.smithi055.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-19T01:40:47.393 INFO:teuthology.orchestra.run.smithi055.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-19T01:40:47.393 INFO:teuthology.orchestra.run.smithi055.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-19T01:40:47.393 INFO:teuthology.orchestra.run.smithi055.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-19T01:40:47.393 INFO:teuthology.orchestra.run.smithi055.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-19T01:40:47.393 INFO:teuthology.orchestra.run.smithi055.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-19T01:40:47.393 INFO:teuthology.orchestra.run.smithi055.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-19T01:40:47.393 INFO:teuthology.orchestra.run.smithi055.stdout:Recommended packages: 2024-03-19T01:40:47.393 INFO:teuthology.orchestra.run.smithi055.stdout: btrfs-tools 2024-03-19T01:40:47.430 INFO:teuthology.orchestra.run.smithi084.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T01:40:47.430 INFO:teuthology.orchestra.run.smithi084.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T01:40:47.430 INFO:teuthology.orchestra.run.smithi084.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T01:40:47.430 INFO:teuthology.orchestra.run.smithi084.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T01:40:47.431 INFO:teuthology.orchestra.run.smithi084.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-19T01:40:47.431 INFO:teuthology.orchestra.run.smithi084.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-19T01:40:47.431 INFO:teuthology.orchestra.run.smithi084.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T01:40:47.432 INFO:teuthology.orchestra.run.smithi084.stdout:The following additional packages will be installed: 2024-03-19T01:40:47.432 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-19T01:40:47.432 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-19T01:40:47.432 INFO:teuthology.orchestra.run.smithi084.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-19T01:40:47.432 INFO:teuthology.orchestra.run.smithi084.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-19T01:40:47.433 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-19T01:40:47.433 INFO:teuthology.orchestra.run.smithi084.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-19T01:40:47.433 INFO:teuthology.orchestra.run.smithi084.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-19T01:40:47.434 INFO:teuthology.orchestra.run.smithi084.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-19T01:40:47.434 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-19T01:40:47.434 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-03-19T01:40:47.434 INFO:teuthology.orchestra.run.smithi084.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-03-19T01:40:47.434 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-03-19T01:40:47.434 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-03-19T01:40:47.434 INFO:teuthology.orchestra.run.smithi084.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-03-19T01:40:47.434 INFO:teuthology.orchestra.run.smithi084.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-03-19T01:40:47.435 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-03-19T01:40:47.435 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-03-19T01:40:47.435 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-03-19T01:40:47.435 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-03-19T01:40:47.435 INFO:teuthology.orchestra.run.smithi084.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-03-19T01:40:47.435 INFO:teuthology.orchestra.run.smithi084.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-03-19T01:40:47.436 INFO:teuthology.orchestra.run.smithi084.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-03-19T01:40:47.436 INFO:teuthology.orchestra.run.smithi084.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-03-19T01:40:47.436 INFO:teuthology.orchestra.run.smithi084.stdout: qttranslations5-l10n rbd-mirror rbd-nbd socat unzip xmlstarlet zip 2024-03-19T01:40:47.437 INFO:teuthology.orchestra.run.smithi084.stdout:Suggested packages: 2024-03-19T01:40:47.437 INFO:teuthology.orchestra.run.smithi084.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-19T01:40:47.437 INFO:teuthology.orchestra.run.smithi084.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-19T01:40:47.437 INFO:teuthology.orchestra.run.smithi084.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-19T01:40:47.437 INFO:teuthology.orchestra.run.smithi084.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-19T01:40:47.437 INFO:teuthology.orchestra.run.smithi084.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-19T01:40:47.437 INFO:teuthology.orchestra.run.smithi084.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-19T01:40:47.437 INFO:teuthology.orchestra.run.smithi084.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-19T01:40:47.437 INFO:teuthology.orchestra.run.smithi084.stdout:Recommended packages: 2024-03-19T01:40:47.437 INFO:teuthology.orchestra.run.smithi084.stdout: btrfs-tools 2024-03-19T01:40:47.533 INFO:teuthology.orchestra.run.smithi055.stdout:The following NEW packages will be installed: 2024-03-19T01:40:47.533 INFO:teuthology.orchestra.run.smithi055.stdout: ceph ceph-base ceph-common ceph-common-dbg ceph-fuse ceph-fuse-dbg ceph-mds 2024-03-19T01:40:47.533 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds-dbg ceph-mgr ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-dbg 2024-03-19T01:40:47.533 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2024-03-19T01:40:47.533 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon ceph-mon-dbg ceph-osd ceph-osd-dbg ceph-test ceph-test-dbg 2024-03-19T01:40:47.533 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-volume cephadm cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev 2024-03-19T01:40:47.533 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2 libcephfs2-dbg libdouble-conversion3 libjq1 liblttng-ust1 2024-03-19T01:40:47.533 INFO:teuthology.orchestra.run.smithi055.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-03-19T01:40:47.533 INFO:teuthology.orchestra.run.smithi055.stdout: libqt5network5 librados2 librados2-dbg libradosstriper1 libradosstriper1-dbg 2024-03-19T01:40:47.534 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1 librbd1-dbg librdkafka1 libreadline-dev librgw2 librgw2-dbg 2024-03-19T01:40:47.534 INFO:teuthology.orchestra.run.smithi055.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-03-19T01:40:47.534 INFO:teuthology.orchestra.run.smithi055.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-03-19T01:40:47.534 INFO:teuthology.orchestra.run.smithi055.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-03-19T01:40:47.534 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-03-19T01:40:47.534 INFO:teuthology.orchestra.run.smithi055.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-03-19T01:40:47.534 INFO:teuthology.orchestra.run.smithi055.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-03-19T01:40:47.534 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-19T01:40:47.534 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-19T01:40:47.534 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-19T01:40:47.534 INFO:teuthology.orchestra.run.smithi055.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-19T01:40:47.534 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-19T01:40:47.534 INFO:teuthology.orchestra.run.smithi055.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-19T01:40:47.534 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-03-19T01:40:47.535 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-03-19T01:40:47.535 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-03-19T01:40:47.535 INFO:teuthology.orchestra.run.smithi055.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-19T01:40:47.535 INFO:teuthology.orchestra.run.smithi055.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-19T01:40:47.535 INFO:teuthology.orchestra.run.smithi055.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-19T01:40:47.535 INFO:teuthology.orchestra.run.smithi055.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-19T01:40:47.535 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw 2024-03-19T01:40:47.535 INFO:teuthology.orchestra.run.smithi055.stdout: radosgw-dbg rbd-fuse rbd-fuse-dbg rbd-mirror rbd-mirror-dbg rbd-nbd 2024-03-19T01:40:47.535 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd-dbg socat unzip xmlstarlet zip 2024-03-19T01:40:47.568 INFO:teuthology.orchestra.run.smithi084.stdout:The following NEW packages will be installed: 2024-03-19T01:40:47.568 INFO:teuthology.orchestra.run.smithi084.stdout: ceph ceph-base ceph-common ceph-common-dbg ceph-fuse ceph-fuse-dbg ceph-mds 2024-03-19T01:40:47.568 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds-dbg ceph-mgr ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-dbg 2024-03-19T01:40:47.568 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2024-03-19T01:40:47.568 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon ceph-mon-dbg ceph-osd ceph-osd-dbg ceph-test ceph-test-dbg 2024-03-19T01:40:47.568 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-volume cephadm cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev 2024-03-19T01:40:47.568 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2 libcephfs2-dbg libdouble-conversion3 libjq1 liblttng-ust1 2024-03-19T01:40:47.568 INFO:teuthology.orchestra.run.smithi084.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-03-19T01:40:47.568 INFO:teuthology.orchestra.run.smithi084.stdout: libqt5network5 librados2 librados2-dbg libradosstriper1 libradosstriper1-dbg 2024-03-19T01:40:47.568 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1 librbd1-dbg librdkafka1 libreadline-dev librgw2 librgw2-dbg 2024-03-19T01:40:47.568 INFO:teuthology.orchestra.run.smithi084.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-03-19T01:40:47.568 INFO:teuthology.orchestra.run.smithi084.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-03-19T01:40:47.569 INFO:teuthology.orchestra.run.smithi084.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-03-19T01:40:47.569 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-03-19T01:40:47.569 INFO:teuthology.orchestra.run.smithi084.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-03-19T01:40:47.569 INFO:teuthology.orchestra.run.smithi084.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-03-19T01:40:47.569 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-19T01:40:47.569 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-19T01:40:47.569 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-19T01:40:47.569 INFO:teuthology.orchestra.run.smithi084.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-19T01:40:47.569 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-19T01:40:47.569 INFO:teuthology.orchestra.run.smithi084.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-19T01:40:47.569 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-03-19T01:40:47.569 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-03-19T01:40:47.569 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-03-19T01:40:47.569 INFO:teuthology.orchestra.run.smithi084.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-19T01:40:47.570 INFO:teuthology.orchestra.run.smithi084.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-19T01:40:47.570 INFO:teuthology.orchestra.run.smithi084.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-19T01:40:47.570 INFO:teuthology.orchestra.run.smithi084.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-19T01:40:47.570 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw 2024-03-19T01:40:47.570 INFO:teuthology.orchestra.run.smithi084.stdout: radosgw-dbg rbd-fuse rbd-fuse-dbg rbd-mirror rbd-mirror-dbg rbd-nbd 2024-03-19T01:40:47.570 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd-dbg socat unzip xmlstarlet zip 2024-03-19T01:40:47.599 INFO:teuthology.orchestra.run.smithi055.stdout:0 upgraded, 142 newly installed, 0 to remove and 285 not upgraded. 2024-03-19T01:40:47.599 INFO:teuthology.orchestra.run.smithi055.stdout:Need to get 3,820 MB of archives. 2024-03-19T01:40:47.599 INFO:teuthology.orchestra.run.smithi055.stdout:After this operation, 4,633 MB of additional disk space will be used. 2024-03-19T01:40:47.600 INFO:teuthology.orchestra.run.smithi055.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-03-19T01:40:47.620 INFO:teuthology.orchestra.run.smithi055.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-19T01:40:47.632 INFO:teuthology.orchestra.run.smithi055.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-03-19T01:40:47.633 INFO:teuthology.orchestra.run.smithi084.stdout:0 upgraded, 142 newly installed, 0 to remove and 285 not upgraded. 2024-03-19T01:40:47.633 INFO:teuthology.orchestra.run.smithi084.stdout:Need to get 3,820 MB of archives. 2024-03-19T01:40:47.633 INFO:teuthology.orchestra.run.smithi084.stdout:After this operation, 4,633 MB of additional disk space will be used. 2024-03-19T01:40:47.633 INFO:teuthology.orchestra.run.smithi084.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-03-19T01:40:47.653 INFO:teuthology.orchestra.run.smithi084.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-19T01:40:47.665 INFO:teuthology.orchestra.run.smithi084.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-03-19T01:40:47.681 INFO:teuthology.orchestra.run.smithi055.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-19T01:40:47.706 INFO:teuthology.orchestra.run.smithi055.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-19T01:40:47.708 INFO:teuthology.orchestra.run.smithi084.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2206-gb6ae2491-1jammy [3,546 kB] 2024-03-19T01:40:47.710 INFO:teuthology.orchestra.run.smithi055.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-19T01:40:47.711 INFO:teuthology.orchestra.run.smithi084.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-19T01:40:47.723 INFO:teuthology.orchestra.run.smithi055.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-19T01:40:47.735 INFO:teuthology.orchestra.run.smithi084.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-19T01:40:47.738 INFO:teuthology.orchestra.run.smithi084.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-19T01:40:47.752 INFO:teuthology.orchestra.run.smithi084.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-19T01:40:47.752 INFO:teuthology.orchestra.run.smithi055.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2206-gb6ae2491-1jammy [3,546 kB] 2024-03-19T01:40:47.776 INFO:teuthology.orchestra.run.smithi055.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-19T01:40:47.780 INFO:teuthology.orchestra.run.smithi055.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-19T01:40:47.792 INFO:teuthology.orchestra.run.smithi055.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-19T01:40:47.796 INFO:teuthology.orchestra.run.smithi055.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-19T01:40:47.796 INFO:teuthology.orchestra.run.smithi055.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-19T01:40:47.805 INFO:teuthology.orchestra.run.smithi055.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-19T01:40:47.807 INFO:teuthology.orchestra.run.smithi055.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-19T01:40:47.810 INFO:teuthology.orchestra.run.smithi055.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-19T01:40:47.811 INFO:teuthology.orchestra.run.smithi055.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-19T01:40:47.811 INFO:teuthology.orchestra.run.smithi055.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-19T01:40:47.812 INFO:teuthology.orchestra.run.smithi084.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-19T01:40:47.818 INFO:teuthology.orchestra.run.smithi084.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-19T01:40:47.826 INFO:teuthology.orchestra.run.smithi055.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-19T01:40:47.828 INFO:teuthology.orchestra.run.smithi055.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-19T01:40:47.830 INFO:teuthology.orchestra.run.smithi084.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-19T01:40:47.830 INFO:teuthology.orchestra.run.smithi055.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-19T01:40:47.831 INFO:teuthology.orchestra.run.smithi055.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-19T01:40:47.834 INFO:teuthology.orchestra.run.smithi084.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-19T01:40:47.835 INFO:teuthology.orchestra.run.smithi084.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-19T01:40:47.843 INFO:teuthology.orchestra.run.smithi084.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-19T01:40:47.845 INFO:teuthology.orchestra.run.smithi084.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-19T01:40:47.847 INFO:teuthology.orchestra.run.smithi055.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-19T01:40:47.847 INFO:teuthology.orchestra.run.smithi084.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-19T01:40:47.848 INFO:teuthology.orchestra.run.smithi055.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-19T01:40:47.849 INFO:teuthology.orchestra.run.smithi055.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-19T01:40:47.849 INFO:teuthology.orchestra.run.smithi055.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-19T01:40:47.849 INFO:teuthology.orchestra.run.smithi084.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-19T01:40:47.850 INFO:teuthology.orchestra.run.smithi084.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-19T01:40:47.850 INFO:teuthology.orchestra.run.smithi055.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-19T01:40:47.850 INFO:teuthology.orchestra.run.smithi055.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-19T01:40:47.856 INFO:teuthology.orchestra.run.smithi084.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-19T01:40:47.858 INFO:teuthology.orchestra.run.smithi084.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-19T01:40:47.860 INFO:teuthology.orchestra.run.smithi084.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-19T01:40:47.875 INFO:teuthology.orchestra.run.smithi084.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-19T01:40:47.876 INFO:teuthology.orchestra.run.smithi084.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-19T01:40:47.877 INFO:teuthology.orchestra.run.smithi084.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-19T01:40:47.877 INFO:teuthology.orchestra.run.smithi084.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-19T01:40:47.877 INFO:teuthology.orchestra.run.smithi084.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-19T01:40:47.878 INFO:teuthology.orchestra.run.smithi084.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-19T01:40:47.878 INFO:teuthology.orchestra.run.smithi084.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-19T01:40:47.882 INFO:teuthology.orchestra.run.smithi055.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-19T01:40:47.883 INFO:teuthology.orchestra.run.smithi055.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-19T01:40:47.885 INFO:teuthology.orchestra.run.smithi055.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-19T01:40:47.886 INFO:teuthology.orchestra.run.smithi055.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-19T01:40:47.895 INFO:teuthology.orchestra.run.smithi084.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-19T01:40:47.895 INFO:teuthology.orchestra.run.smithi084.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-19T01:40:47.898 INFO:teuthology.orchestra.run.smithi084.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-19T01:40:47.907 INFO:teuthology.orchestra.run.smithi055.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-19T01:40:47.907 INFO:teuthology.orchestra.run.smithi055.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-19T01:40:47.907 INFO:teuthology.orchestra.run.smithi055.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-19T01:40:47.907 INFO:teuthology.orchestra.run.smithi055.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-19T01:40:47.908 INFO:teuthology.orchestra.run.smithi055.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-19T01:40:47.908 INFO:teuthology.orchestra.run.smithi055.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-19T01:40:47.914 INFO:teuthology.orchestra.run.smithi084.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-19T01:40:47.914 INFO:teuthology.orchestra.run.smithi084.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-19T01:40:47.915 INFO:teuthology.orchestra.run.smithi084.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-19T01:40:47.916 INFO:teuthology.orchestra.run.smithi084.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-19T01:40:47.916 INFO:teuthology.orchestra.run.smithi084.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-19T01:40:47.917 INFO:teuthology.orchestra.run.smithi084.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-19T01:40:47.918 INFO:teuthology.orchestra.run.smithi084.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-19T01:40:47.924 INFO:teuthology.orchestra.run.smithi055.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-19T01:40:47.925 INFO:teuthology.orchestra.run.smithi055.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-19T01:40:47.930 INFO:teuthology.orchestra.run.smithi055.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-19T01:40:47.930 INFO:teuthology.orchestra.run.smithi055.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-19T01:40:47.933 INFO:teuthology.orchestra.run.smithi084.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-19T01:40:47.934 INFO:teuthology.orchestra.run.smithi084.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-19T01:40:47.939 INFO:teuthology.orchestra.run.smithi084.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-19T01:40:47.945 INFO:teuthology.orchestra.run.smithi055.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-19T01:40:47.945 INFO:teuthology.orchestra.run.smithi055.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-19T01:40:47.946 INFO:teuthology.orchestra.run.smithi055.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-19T01:40:47.949 INFO:teuthology.orchestra.run.smithi055.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-19T01:40:47.949 INFO:teuthology.orchestra.run.smithi055.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-19T01:40:47.950 INFO:teuthology.orchestra.run.smithi055.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-19T01:40:47.953 INFO:teuthology.orchestra.run.smithi084.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-19T01:40:47.953 INFO:teuthology.orchestra.run.smithi084.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-19T01:40:47.954 INFO:teuthology.orchestra.run.smithi084.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-19T01:40:47.956 INFO:teuthology.orchestra.run.smithi084.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-19T01:40:47.957 INFO:teuthology.orchestra.run.smithi084.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-19T01:40:47.958 INFO:teuthology.orchestra.run.smithi084.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-19T01:40:47.959 INFO:teuthology.orchestra.run.smithi084.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-19T01:40:47.966 INFO:teuthology.orchestra.run.smithi055.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-19T01:40:47.967 INFO:teuthology.orchestra.run.smithi055.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-19T01:40:47.968 INFO:teuthology.orchestra.run.smithi055.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-19T01:40:47.969 INFO:teuthology.orchestra.run.smithi055.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-19T01:40:47.972 INFO:teuthology.orchestra.run.smithi084.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-19T01:40:47.974 INFO:teuthology.orchestra.run.smithi084.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-19T01:40:47.975 INFO:teuthology.orchestra.run.smithi084.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-19T01:40:47.986 INFO:teuthology.orchestra.run.smithi055.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-19T01:40:47.997 INFO:teuthology.orchestra.run.smithi084.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-19T01:40:48.000 INFO:teuthology.orchestra.run.smithi084.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-19T01:40:48.067 INFO:teuthology.orchestra.run.smithi055.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-19T01:40:48.068 INFO:teuthology.orchestra.run.smithi055.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-19T01:40:48.070 INFO:teuthology.orchestra.run.smithi055.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-19T01:40:48.070 INFO:teuthology.orchestra.run.smithi055.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-19T01:40:48.071 INFO:teuthology.orchestra.run.smithi055.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-19T01:40:48.087 INFO:teuthology.orchestra.run.smithi084.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-19T01:40:48.089 INFO:teuthology.orchestra.run.smithi084.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-19T01:40:48.091 INFO:teuthology.orchestra.run.smithi084.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-19T01:40:48.091 INFO:teuthology.orchestra.run.smithi084.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-19T01:40:48.093 INFO:teuthology.orchestra.run.smithi084.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-19T01:40:48.109 INFO:teuthology.orchestra.run.smithi055.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-19T01:40:48.111 INFO:teuthology.orchestra.run.smithi055.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-19T01:40:48.112 INFO:teuthology.orchestra.run.smithi055.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-19T01:40:48.129 INFO:teuthology.orchestra.run.smithi084.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-19T01:40:48.131 INFO:teuthology.orchestra.run.smithi084.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-19T01:40:48.132 INFO:teuthology.orchestra.run.smithi084.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-19T01:40:48.148 INFO:teuthology.orchestra.run.smithi055.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-19T01:40:48.148 INFO:teuthology.orchestra.run.smithi055.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-19T01:40:48.158 INFO:teuthology.orchestra.run.smithi055.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-19T01:40:48.158 INFO:teuthology.orchestra.run.smithi084.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-19T01:40:48.158 INFO:teuthology.orchestra.run.smithi084.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-19T01:40:48.158 INFO:teuthology.orchestra.run.smithi084.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-19T01:40:48.159 INFO:teuthology.orchestra.run.smithi084.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-19T01:40:48.159 INFO:teuthology.orchestra.run.smithi084.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-19T01:40:48.159 INFO:teuthology.orchestra.run.smithi055.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-19T01:40:48.159 INFO:teuthology.orchestra.run.smithi055.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-19T01:40:48.160 INFO:teuthology.orchestra.run.smithi055.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-19T01:40:48.160 INFO:teuthology.orchestra.run.smithi055.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-19T01:40:48.160 INFO:teuthology.orchestra.run.smithi084.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-19T01:40:48.160 INFO:teuthology.orchestra.run.smithi055.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-19T01:40:48.165 INFO:teuthology.orchestra.run.smithi055.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-19T01:40:48.166 INFO:teuthology.orchestra.run.smithi084.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-19T01:40:48.167 INFO:teuthology.orchestra.run.smithi084.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-19T01:40:48.171 INFO:teuthology.orchestra.run.smithi084.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-19T01:40:48.177 INFO:teuthology.orchestra.run.smithi055.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-19T01:40:48.179 INFO:teuthology.orchestra.run.smithi055.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-19T01:40:48.180 INFO:teuthology.orchestra.run.smithi055.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-19T01:40:48.185 INFO:teuthology.orchestra.run.smithi084.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-19T01:40:48.187 INFO:teuthology.orchestra.run.smithi084.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-19T01:40:48.188 INFO:teuthology.orchestra.run.smithi084.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-19T01:40:48.192 INFO:teuthology.orchestra.run.smithi084.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-19T01:40:48.195 INFO:teuthology.orchestra.run.smithi084.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-03-19T01:40:48.195 INFO:teuthology.orchestra.run.smithi084.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-19T01:40:48.196 INFO:teuthology.orchestra.run.smithi084.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-19T01:40:48.197 INFO:teuthology.orchestra.run.smithi055.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-19T01:40:48.201 INFO:teuthology.orchestra.run.smithi055.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-03-19T01:40:48.201 INFO:teuthology.orchestra.run.smithi055.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-19T01:40:48.203 INFO:teuthology.orchestra.run.smithi055.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-19T01:40:48.203 INFO:teuthology.orchestra.run.smithi055.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-19T01:40:48.209 INFO:teuthology.orchestra.run.smithi084.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-19T01:40:48.210 INFO:teuthology.orchestra.run.smithi055.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-19T01:40:48.211 INFO:teuthology.orchestra.run.smithi055.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-19T01:40:48.214 INFO:teuthology.orchestra.run.smithi084.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-19T01:40:48.215 INFO:teuthology.orchestra.run.smithi084.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-19T01:40:48.218 INFO:teuthology.orchestra.run.smithi055.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-19T01:40:48.218 INFO:teuthology.orchestra.run.smithi055.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-19T01:40:48.219 INFO:teuthology.orchestra.run.smithi055.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-19T01:40:48.228 INFO:teuthology.orchestra.run.smithi084.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-19T01:40:48.229 INFO:teuthology.orchestra.run.smithi084.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-19T01:40:48.230 INFO:teuthology.orchestra.run.smithi084.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-19T01:40:48.230 INFO:teuthology.orchestra.run.smithi084.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-19T01:40:48.232 INFO:teuthology.orchestra.run.smithi084.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-19T01:40:48.233 INFO:teuthology.orchestra.run.smithi084.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-19T01:40:48.239 INFO:teuthology.orchestra.run.smithi055.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-19T01:40:48.241 INFO:teuthology.orchestra.run.smithi084.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-19T01:40:48.242 INFO:teuthology.orchestra.run.smithi055.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-19T01:40:48.244 INFO:teuthology.orchestra.run.smithi055.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-19T01:40:48.251 INFO:teuthology.orchestra.run.smithi084.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-19T01:40:48.251 INFO:teuthology.orchestra.run.smithi084.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-19T01:40:48.254 INFO:teuthology.orchestra.run.smithi084.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-19T01:40:48.255 INFO:teuthology.orchestra.run.smithi055.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-19T01:40:48.256 INFO:teuthology.orchestra.run.smithi055.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-19T01:40:48.256 INFO:teuthology.orchestra.run.smithi055.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-19T01:40:48.260 INFO:teuthology.orchestra.run.smithi055.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-19T01:40:48.261 INFO:teuthology.orchestra.run.smithi055.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-19T01:40:48.277 INFO:teuthology.orchestra.run.smithi084.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-19T01:40:48.368 INFO:teuthology.orchestra.run.smithi084.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2206-gb6ae2491-1jammy [3,225 kB] 2024-03-19T01:40:48.595 INFO:teuthology.orchestra.run.smithi084.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2206-gb6ae2491-1jammy [712 kB] 2024-03-19T01:40:48.622 INFO:teuthology.orchestra.run.smithi084.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2206-gb6ae2491-1jammy [356 kB] 2024-03-19T01:40:48.641 INFO:teuthology.orchestra.run.smithi084.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2206-gb6ae2491-1jammy [32.8 kB] 2024-03-19T01:40:48.642 INFO:teuthology.orchestra.run.smithi084.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2206-gb6ae2491-1jammy [180 kB] 2024-03-19T01:40:48.658 INFO:teuthology.orchestra.run.smithi084.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2206-gb6ae2491-1jammy [64.6 kB] 2024-03-19T01:40:48.659 INFO:teuthology.orchestra.run.smithi084.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2206-gb6ae2491-1jammy [330 kB] 2024-03-19T01:40:48.676 INFO:teuthology.orchestra.run.smithi084.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2206-gb6ae2491-1jammy [6,651 kB] 2024-03-19T01:40:48.931 INFO:teuthology.orchestra.run.smithi084.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2206-gb6ae2491-1jammy [112 kB] 2024-03-19T01:40:48.934 INFO:teuthology.orchestra.run.smithi084.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2206-gb6ae2491-1jammy [466 kB] 2024-03-19T01:40:48.952 INFO:teuthology.orchestra.run.smithi084.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2206-gb6ae2491-1jammy [25.3 MB] 2024-03-19T01:40:49.046 INFO:teuthology.orchestra.run.smithi055.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2206-gb6ae2491-1jammy [3,225 kB] 2024-03-19T01:40:49.746 INFO:teuthology.orchestra.run.smithi055.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2206-gb6ae2491-1jammy [712 kB] 2024-03-19T01:40:49.895 INFO:teuthology.orchestra.run.smithi055.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2206-gb6ae2491-1jammy [356 kB] 2024-03-19T01:40:49.968 INFO:teuthology.orchestra.run.smithi055.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2206-gb6ae2491-1jammy [32.8 kB] 2024-03-19T01:40:49.980 INFO:teuthology.orchestra.run.smithi055.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2206-gb6ae2491-1jammy [180 kB] 2024-03-19T01:40:50.017 INFO:teuthology.orchestra.run.smithi055.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2206-gb6ae2491-1jammy [64.6 kB] 2024-03-19T01:40:50.019 INFO:teuthology.orchestra.run.smithi055.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2206-gb6ae2491-1jammy [330 kB] 2024-03-19T01:40:50.022 INFO:teuthology.orchestra.run.smithi084.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2206-gb6ae2491-1jammy [5,085 kB] 2024-03-19T01:40:50.093 INFO:teuthology.orchestra.run.smithi055.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2206-gb6ae2491-1jammy [6,651 kB] 2024-03-19T01:40:50.133 INFO:teuthology.orchestra.run.smithi084.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2206-gb6ae2491-1jammy [244 kB] 2024-03-19T01:40:50.142 INFO:teuthology.orchestra.run.smithi084.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2206-gb6ae2491-1jammy [124 kB] 2024-03-19T01:40:50.146 INFO:teuthology.orchestra.run.smithi084.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2206-gb6ae2491-1jammy [1,526 kB] 2024-03-19T01:40:50.207 INFO:teuthology.orchestra.run.smithi084.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2206-gb6ae2491-1jammy [6,258 kB] 2024-03-19T01:40:50.457 INFO:teuthology.orchestra.run.smithi084.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2206-gb6ae2491-1jammy [22.8 MB] 2024-03-19T01:40:51.378 INFO:teuthology.orchestra.run.smithi055.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2206-gb6ae2491-1jammy [112 kB] 2024-03-19T01:40:51.390 INFO:teuthology.orchestra.run.smithi055.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2206-gb6ae2491-1jammy [466 kB] 2024-03-19T01:40:51.445 INFO:teuthology.orchestra.run.smithi084.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2206-gb6ae2491-1jammy [14.0 kB] 2024-03-19T01:40:51.446 INFO:teuthology.orchestra.run.smithi084.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2206-gb6ae2491-1jammy [870 kB] 2024-03-19T01:40:51.461 INFO:teuthology.orchestra.run.smithi055.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2206-gb6ae2491-1jammy [25.3 MB] 2024-03-19T01:40:51.462 INFO:teuthology.orchestra.run.smithi084.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2206-gb6ae2491-1jammy [2,434 kB] 2024-03-19T01:40:51.503 INFO:teuthology.orchestra.run.smithi084.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2206-gb6ae2491-1jammy [777 kB] 2024-03-19T01:40:51.513 INFO:teuthology.orchestra.run.smithi084.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2206-gb6ae2491-1jammy [144 kB] 2024-03-19T01:40:51.520 INFO:teuthology.orchestra.run.smithi084.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2206-gb6ae2491-1jammy [1,936 kB] 2024-03-19T01:40:51.594 INFO:teuthology.orchestra.run.smithi084.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2206-gb6ae2491-1jammy [8,625 kB] 2024-03-19T01:40:51.927 INFO:teuthology.orchestra.run.smithi084.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2206-gb6ae2491-1jammy [14.2 kB] 2024-03-19T01:40:51.928 INFO:teuthology.orchestra.run.smithi084.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2206-gb6ae2491-1jammy [50.2 MB] 2024-03-19T01:40:54.434 INFO:teuthology.orchestra.run.smithi084.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2206-gb6ae2491-1jammy [121 kB] 2024-03-19T01:40:54.436 INFO:teuthology.orchestra.run.smithi084.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-2206-gb6ae2491-1jammy [229 kB] 2024-03-19T01:40:54.441 INFO:teuthology.orchestra.run.smithi084.stdout:Get:120 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-2206-gb6ae2491-1jammy [34.0 kB] 2024-03-19T01:40:54.443 INFO:teuthology.orchestra.run.smithi084.stdout:Get:121 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-2206-gb6ae2491-1jammy [23.4 kB] 2024-03-19T01:40:54.444 INFO:teuthology.orchestra.run.smithi084.stdout:Get:122 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2206-gb6ae2491-1jammy [38.9 kB] 2024-03-19T01:40:54.445 INFO:teuthology.orchestra.run.smithi084.stdout:Get:123 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2206-gb6ae2491-1jammy [13.2 MB] 2024-03-19T01:40:54.957 INFO:teuthology.orchestra.run.smithi084.stdout:Get:124 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2206-gb6ae2491-1jammy [91.4 kB] 2024-03-19T01:40:54.960 INFO:teuthology.orchestra.run.smithi084.stdout:Get:125 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.0.0-2206-gb6ae2491-1jammy [3,122 kB] 2024-03-19T01:40:55.140 INFO:teuthology.orchestra.run.smithi084.stdout:Get:126 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 rbd-nbd amd64 19.0.0-2206-gb6ae2491-1jammy [179 kB] 2024-03-19T01:40:55.149 INFO:teuthology.orchestra.run.smithi084.stdout:Get:127 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [763 MB] 2024-03-19T01:40:56.399 INFO:teuthology.orchestra.run.smithi055.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2206-gb6ae2491-1jammy [5,085 kB] 2024-03-19T01:40:57.476 INFO:teuthology.orchestra.run.smithi055.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2206-gb6ae2491-1jammy [244 kB] 2024-03-19T01:40:57.508 INFO:teuthology.orchestra.run.smithi055.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2206-gb6ae2491-1jammy [124 kB] 2024-03-19T01:40:57.535 INFO:teuthology.orchestra.run.smithi055.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2206-gb6ae2491-1jammy [1,526 kB] 2024-03-19T01:40:57.969 INFO:teuthology.orchestra.run.smithi055.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2206-gb6ae2491-1jammy [6,258 kB] 2024-03-19T01:40:59.388 INFO:teuthology.orchestra.run.smithi055.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2206-gb6ae2491-1jammy [22.8 MB] 2024-03-19T01:41:03.719 INFO:teuthology.orchestra.run.smithi055.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2206-gb6ae2491-1jammy [14.0 kB] 2024-03-19T01:41:03.719 INFO:teuthology.orchestra.run.smithi055.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2206-gb6ae2491-1jammy [870 kB] 2024-03-19T01:41:03.828 INFO:teuthology.orchestra.run.smithi055.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2206-gb6ae2491-1jammy [2,434 kB] 2024-03-19T01:41:04.330 INFO:teuthology.orchestra.run.smithi055.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2206-gb6ae2491-1jammy [777 kB] 2024-03-19T01:41:04.471 INFO:teuthology.orchestra.run.smithi055.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2206-gb6ae2491-1jammy [144 kB] 2024-03-19T01:41:04.510 INFO:teuthology.orchestra.run.smithi055.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2206-gb6ae2491-1jammy [1,936 kB] 2024-03-19T01:41:04.955 INFO:teuthology.orchestra.run.smithi055.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2206-gb6ae2491-1jammy [8,625 kB] 2024-03-19T01:41:06.857 INFO:teuthology.orchestra.run.smithi055.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2206-gb6ae2491-1jammy [14.2 kB] 2024-03-19T01:41:06.858 INFO:teuthology.orchestra.run.smithi055.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2206-gb6ae2491-1jammy [50.2 MB] 2024-03-19T01:41:15.533 INFO:teuthology.orchestra.run.smithi055.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2206-gb6ae2491-1jammy [121 kB] 2024-03-19T01:41:15.535 INFO:teuthology.orchestra.run.smithi055.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-2206-gb6ae2491-1jammy [229 kB] 2024-03-19T01:41:15.540 INFO:teuthology.orchestra.run.smithi055.stdout:Get:120 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-2206-gb6ae2491-1jammy [34.0 kB] 2024-03-19T01:41:15.541 INFO:teuthology.orchestra.run.smithi055.stdout:Get:121 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-2206-gb6ae2491-1jammy [23.4 kB] 2024-03-19T01:41:15.542 INFO:teuthology.orchestra.run.smithi055.stdout:Get:122 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2206-gb6ae2491-1jammy [38.9 kB] 2024-03-19T01:41:15.543 INFO:teuthology.orchestra.run.smithi055.stdout:Get:123 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2206-gb6ae2491-1jammy [13.2 MB] 2024-03-19T01:41:19.101 INFO:teuthology.orchestra.run.smithi055.stdout:Get:124 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2206-gb6ae2491-1jammy [91.4 kB] 2024-03-19T01:41:19.103 INFO:teuthology.orchestra.run.smithi055.stdout:Get:125 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.0.0-2206-gb6ae2491-1jammy [3,122 kB] 2024-03-19T01:41:19.923 INFO:teuthology.orchestra.run.smithi055.stdout:Get:126 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 rbd-nbd amd64 19.0.0-2206-gb6ae2491-1jammy [179 kB] 2024-03-19T01:41:19.970 INFO:teuthology.orchestra.run.smithi055.stdout:Get:127 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [763 MB] 2024-03-19T01:41:34.618 INFO:teuthology.orchestra.run.smithi084.stdout:Get:128 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [19.0 MB] 2024-03-19T01:41:36.108 INFO:teuthology.orchestra.run.smithi084.stdout:Get:129 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [86.7 MB] 2024-03-19T01:41:42.141 INFO:teuthology.orchestra.run.smithi084.stdout:Get:130 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [42.8 MB] 2024-03-19T01:41:45.540 INFO:teuthology.orchestra.run.smithi084.stdout:Get:131 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [159 MB] 2024-03-19T01:41:55.735 INFO:teuthology.orchestra.run.smithi084.stdout:Get:132 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [542 MB] 2024-03-19T01:42:27.485 INFO:teuthology.orchestra.run.smithi055.stdout:Get:128 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [19.0 MB] 2024-03-19T01:42:28.295 INFO:teuthology.orchestra.run.smithi055.stdout:Get:129 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [86.7 MB] 2024-03-19T01:42:29.365 INFO:teuthology.orchestra.run.smithi084.stdout:Get:133 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [1,193 MB] 2024-03-19T01:42:34.105 INFO:teuthology.orchestra.run.smithi055.stdout:Get:130 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [42.8 MB] 2024-03-19T01:42:37.202 INFO:teuthology.orchestra.run.smithi055.stdout:Get:131 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [159 MB] 2024-03-19T01:42:48.468 INFO:teuthology.orchestra.run.smithi055.stdout:Get:132 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [542 MB] 2024-03-19T01:43:38.237 INFO:teuthology.orchestra.run.smithi055.stdout:Get:133 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [1,193 MB] 2024-03-19T01:43:40.209 INFO:teuthology.orchestra.run.smithi084.stdout:Get:134 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [17.1 MB] 2024-03-19T01:43:41.019 INFO:teuthology.orchestra.run.smithi084.stdout:Get:135 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 librados2-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [83.3 MB] 2024-03-19T01:43:45.365 INFO:teuthology.orchestra.run.smithi084.stdout:Get:136 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [11.4 MB] 2024-03-19T01:43:45.778 INFO:teuthology.orchestra.run.smithi084.stdout:Get:137 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 librbd1-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [104 MB] 2024-03-19T01:43:51.461 INFO:teuthology.orchestra.run.smithi084.stdout:Get:138 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 librgw2-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [175 MB] 2024-03-19T01:44:00.998 INFO:teuthology.orchestra.run.smithi084.stdout:Get:139 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 radosgw-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [339 MB] 2024-03-19T01:44:22.049 INFO:teuthology.orchestra.run.smithi084.stdout:Get:140 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [1,794 kB] 2024-03-19T01:44:22.082 INFO:teuthology.orchestra.run.smithi084.stdout:Get:141 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [97.7 MB] 2024-03-19T01:44:27.725 INFO:teuthology.orchestra.run.smithi084.stdout:Get:142 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 rbd-nbd-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [3,146 kB] 2024-03-19T01:44:28.394 INFO:teuthology.orchestra.run.smithi084.stdout:Fetched 3,820 MB in 3min 40s (17.3 MB/s) 2024-03-19T01:44:28.536 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-pyperclip. 2024-03-19T01:44:28.635 INFO:teuthology.orchestra.run.smithi084.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-19T01:44:28.641 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-03-19T01:44:28.684 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-03-19T01:44:28.937 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-19T01:44:28.963 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-19T01:44:28.995 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-19T01:44:29.323 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-cmd2. 2024-03-19T01:44:29.349 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-03-19T01:44:29.389 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-03-19T01:44:29.734 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-19T01:44:29.760 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-19T01:44:29.842 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-19T01:44:30.196 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-19T01:44:30.222 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-19T01:44:30.263 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-19T01:44:30.615 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-19T01:44:30.642 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-19T01:44:30.682 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-19T01:44:31.154 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-19T01:44:31.180 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-19T01:44:31.237 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:44:31.657 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-19T01:44:31.683 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-19T01:44:31.723 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:44:32.077 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-19T01:44:32.102 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-19T01:44:32.143 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:44:32.563 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-19T01:44:32.589 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-19T01:44:32.638 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-19T01:44:33.016 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package librados2. 2024-03-19T01:44:33.042 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../010-librados2_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:44:33.082 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking librados2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:44:33.535 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package librbd1. 2024-03-19T01:44:33.554 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../011-librbd1_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:44:33.593 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking librbd1 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:44:34.080 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libcephfs2. 2024-03-19T01:44:34.106 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:44:34.154 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libcephfs2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:44:34.490 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-rados. 2024-03-19T01:44:34.517 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../013-python3-rados_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:44:34.565 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-rados (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:44:34.918 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-19T01:44:34.945 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:44:34.984 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-ceph-argparse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:44:35.237 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-cephfs. 2024-03-19T01:44:35.264 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:44:35.304 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-cephfs (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:44:35.666 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-19T01:44:35.689 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:44:35.740 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-ceph-common (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:44:36.051 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-prettytable. 2024-03-19T01:44:36.077 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-03-19T01:44:36.126 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-19T01:44:36.445 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-rbd. 2024-03-19T01:44:36.472 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:44:36.520 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-rbd (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:44:36.906 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-19T01:44:36.932 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-19T01:44:36.973 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-19T01:44:37.259 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-19T01:44:37.274 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-19T01:44:37.317 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-19T01:44:37.628 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-19T01:44:37.655 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-19T01:44:37.728 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-19T01:44:38.132 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package lua5.1. 2024-03-19T01:44:38.158 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-19T01:44:38.206 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-19T01:44:38.534 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package lua-any. 2024-03-19T01:44:38.561 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-03-19T01:44:38.601 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-19T01:44:38.862 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package zip. 2024-03-19T01:44:38.877 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-03-19T01:44:38.920 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking zip (3.0-12build2) ... 2024-03-19T01:44:39.282 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package unzip. 2024-03-19T01:44:39.308 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-19T01:44:39.348 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-19T01:44:39.819 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package luarocks. 2024-03-19T01:44:39.843 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-19T01:44:39.893 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-19T01:44:40.285 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package librgw2. 2024-03-19T01:44:40.311 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../027-librgw2_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:44:40.351 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking librgw2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:44:40.887 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-rgw. 2024-03-19T01:44:40.902 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:44:40.945 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-rgw (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:44:41.248 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-19T01:44:41.263 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-19T01:44:41.314 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-19T01:44:41.634 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libradosstriper1. 2024-03-19T01:44:41.660 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:44:41.700 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libradosstriper1 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:44:42.129 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-common. 2024-03-19T01:44:42.155 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../031-ceph-common_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:44:42.195 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-common (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:44:43.492 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-base. 2024-03-19T01:44:43.518 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../032-ceph-base_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:44:43.559 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-base (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:44:44.088 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-19T01:44:44.103 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-19T01:44:44.146 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-19T01:44:44.450 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-cheroot. 2024-03-19T01:44:44.475 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-19T01:44:44.515 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-19T01:44:44.862 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-19T01:44:44.889 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-19T01:44:44.935 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-19T01:44:45.229 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-19T01:44:45.244 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-19T01:44:45.279 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-19T01:44:45.524 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-19T01:44:45.550 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-19T01:44:45.590 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-19T01:44:45.893 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-tempora. 2024-03-19T01:44:45.919 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-03-19T01:44:45.960 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-19T01:44:46.263 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-portend. 2024-03-19T01:44:46.289 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-03-19T01:44:46.329 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-19T01:44:46.582 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-19T01:44:46.608 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-19T01:44:46.640 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-19T01:44:46.927 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-19T01:44:46.953 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-19T01:44:46.993 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-19T01:44:47.330 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-natsort. 2024-03-19T01:44:47.356 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-03-19T01:44:47.396 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-19T01:44:47.658 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-logutils. 2024-03-19T01:44:47.685 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-03-19T01:44:47.724 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-19T01:44:48.027 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-19T01:44:48.054 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-19T01:44:48.093 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-19T01:44:48.405 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-mako. 2024-03-19T01:44:48.432 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-19T01:44:48.496 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-19T01:44:48.825 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-19T01:44:48.852 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-19T01:44:48.891 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-19T01:44:49.160 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-19T01:44:49.187 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-19T01:44:49.218 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-19T01:44:49.505 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-webob. 2024-03-19T01:44:49.531 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-19T01:44:49.571 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-19T01:44:49.924 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-waitress. 2024-03-19T01:44:49.951 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-19T01:44:49.994 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-19T01:44:50.586 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-tempita. 2024-03-19T01:44:50.612 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-19T01:44:50.652 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-19T01:44:50.955 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-paste. 2024-03-19T01:44:50.981 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-19T01:44:51.021 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-19T01:44:51.316 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-19T01:44:51.343 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-19T01:44:51.374 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-19T01:44:51.383 INFO:teuthology.orchestra.run.smithi055.stdout:Get:134 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [17.1 MB] 2024-03-19T01:44:51.661 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-19T01:44:51.687 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-19T01:44:51.727 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-19T01:44:52.020 INFO:teuthology.orchestra.run.smithi055.stdout:Get:135 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 librados2-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [83.3 MB] 2024-03-19T01:44:52.072 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-webtest. 2024-03-19T01:44:52.098 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-03-19T01:44:52.138 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-19T01:44:52.424 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-pecan. 2024-03-19T01:44:52.439 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-19T01:44:52.474 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-19T01:44:52.804 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-certifi. 2024-03-19T01:44:52.833 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-03-19T01:44:52.870 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-19T01:44:53.198 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-idna. 2024-03-19T01:44:53.225 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-03-19T01:44:53.265 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-19T01:44:53.534 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-urllib3. 2024-03-19T01:44:53.549 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-19T01:44:53.592 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-19T01:44:53.888 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-requests. 2024-03-19T01:44:53.914 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-19T01:44:53.954 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-19T01:44:54.265 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-19T01:44:54.291 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-19T01:44:54.331 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-19T01:44:54.576 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-19T01:44:54.591 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:44:54.634 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:44:55.039 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-19T01:44:55.065 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:44:55.106 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:44:55.367 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-19T01:44:55.393 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-19T01:44:55.425 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-19T01:44:55.728 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-openssl. 2024-03-19T01:44:55.754 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-03-19T01:44:55.794 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-19T01:44:56.148 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-mgr. 2024-03-19T01:44:56.174 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:44:56.214 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-mgr (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:44:56.219 INFO:teuthology.orchestra.run.smithi055.stdout:Get:136 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [11.4 MB] 2024-03-19T01:44:56.584 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-mon. 2024-03-19T01:44:56.610 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:44:56.650 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-mon (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:44:56.681 INFO:teuthology.orchestra.run.smithi055.stdout:Get:137 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 librbd1-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [104 MB] 2024-03-19T01:44:57.203 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-osd. 2024-03-19T01:44:57.230 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:44:57.262 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-osd (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:44:58.406 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph. 2024-03-19T01:44:58.432 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../068-ceph_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:44:58.473 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:44:58.792 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-fuse. 2024-03-19T01:44:58.819 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:44:58.858 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-fuse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:44:59.261 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-mds. 2024-03-19T01:44:59.288 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:44:59.336 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-mds (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:44:59.739 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package cephadm. 2024-03-19T01:44:59.766 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../071-cephadm_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:44:59.806 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking cephadm (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:00.125 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-19T01:45:00.152 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-19T01:45:00.191 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-19T01:45:00.527 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-19T01:45:00.543 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-19T01:45:00.586 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-19T01:45:01.649 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-babel. 2024-03-19T01:45:01.677 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-19T01:45:01.707 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-19T01:45:02.027 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-jinja2. 2024-03-19T01:45:02.053 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-19T01:45:02.093 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-19T01:45:02.396 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-19T01:45:02.423 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:45:02.454 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:02.785 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-19T01:45:02.812 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-03-19T01:45:02.849 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-19T01:45:03.135 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-routes. 2024-03-19T01:45:03.162 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-19T01:45:03.201 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-19T01:45:03.522 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-19T01:45:03.548 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:45:03.588 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:04.024 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-19T01:45:04.040 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-19T01:45:04.083 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-19T01:45:04.561 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-joblib. 2024-03-19T01:45:04.588 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-19T01:45:04.636 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-19T01:45:04.998 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-19T01:45:05.025 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-19T01:45:05.158 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-19T01:45:05.411 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-sklearn. 2024-03-19T01:45:05.437 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-19T01:45:05.469 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-19T01:45:06.186 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-19T01:45:06.214 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:45:06.261 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:07.264 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-cachetools. 2024-03-19T01:45:07.292 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-03-19T01:45:07.331 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-19T01:45:07.592 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-19T01:45:07.619 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-19T01:45:07.650 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-19T01:45:07.947 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-19T01:45:07.975 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-19T01:45:08.014 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-19T01:45:08.224 INFO:teuthology.orchestra.run.smithi055.stdout:Get:138 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 librgw2-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [175 MB] 2024-03-19T01:45:08.325 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-rsa. 2024-03-19T01:45:08.354 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-03-19T01:45:08.391 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-19T01:45:08.720 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-google-auth. 2024-03-19T01:45:08.748 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-03-19T01:45:08.786 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-19T01:45:09.082 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-19T01:45:09.108 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-19T01:45:09.148 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-19T01:45:09.451 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-websocket. 2024-03-19T01:45:09.478 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-03-19T01:45:09.517 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-19T01:45:09.812 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-19T01:45:09.840 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-19T01:45:09.879 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-19T01:45:10.542 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-19T01:45:10.559 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:45:10.601 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:10.920 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-19T01:45:10.947 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-19T01:45:10.986 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-19T01:45:11.332 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-19T01:45:11.359 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-19T01:45:11.398 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-19T01:45:11.634 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package jq. 2024-03-19T01:45:11.650 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-19T01:45:11.684 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-19T01:45:11.903 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package socat. 2024-03-19T01:45:11.919 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-19T01:45:11.961 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-19T01:45:12.281 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package xmlstarlet. 2024-03-19T01:45:12.309 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-19T01:45:12.348 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-19T01:45:12.677 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-test. 2024-03-19T01:45:12.705 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../099-ceph-test_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:45:12.743 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-test (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:15.098 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-volume. 2024-03-19T01:45:15.125 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:45:15.164 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-volume (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:15.543 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package cephfs-mirror. 2024-03-19T01:45:15.571 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:45:15.609 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking cephfs-mirror (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:15.929 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-colorama. 2024-03-19T01:45:15.956 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-03-19T01:45:15.995 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-03-19T01:45:16.265 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package cephfs-shell. 2024-03-19T01:45:16.292 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:45:16.331 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking cephfs-shell (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:16.550 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package cephfs-top. 2024-03-19T01:45:16.566 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:45:16.600 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking cephfs-top (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:16.852 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-19T01:45:16.880 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:45:16.919 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libcephfs-dev (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:17.297 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-19T01:45:17.325 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-19T01:45:17.363 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-19T01:45:17.701 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-19T01:45:17.729 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-03-19T01:45:17.767 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-19T01:45:18.096 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package nvme-cli. 2024-03-19T01:45:18.124 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-19T01:45:18.171 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-19T01:45:18.568 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package pkg-config. 2024-03-19T01:45:18.595 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-19T01:45:18.634 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-19T01:45:18.895 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-19T01:45:18.911 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-19T01:45:18.953 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-19T01:45:19.382 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-19T01:45:19.409 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-19T01:45:19.448 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-19T01:45:19.751 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-pastescript. 2024-03-19T01:45:19.778 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-03-19T01:45:19.817 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-19T01:45:20.121 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-pluggy. 2024-03-19T01:45:20.148 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-19T01:45:20.179 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-19T01:45:20.474 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-psutil. 2024-03-19T01:45:20.501 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-19T01:45:20.540 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-19T01:45:20.878 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-py. 2024-03-19T01:45:20.904 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-03-19T01:45:20.934 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-19T01:45:21.337 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-pygments. 2024-03-19T01:45:21.366 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-19T01:45:21.537 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-19T01:45:22.008 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-19T01:45:22.036 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-19T01:45:22.075 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-19T01:45:22.394 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-toml. 2024-03-19T01:45:22.422 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-03-19T01:45:22.460 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-19T01:45:22.730 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-pytest. 2024-03-19T01:45:22.746 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-19T01:45:22.780 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-19T01:45:23.125 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-simplejson. 2024-03-19T01:45:23.152 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-19T01:45:23.191 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-19T01:45:23.469 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-19T01:45:23.497 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-19T01:45:23.535 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-19T01:45:24.231 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package radosgw. 2024-03-19T01:45:24.259 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../122-radosgw_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:45:24.314 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking radosgw (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:25.176 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package rbd-fuse. 2024-03-19T01:45:25.203 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:45:25.242 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking rbd-fuse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:25.589 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package rbd-mirror. 2024-03-19T01:45:25.614 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../124-rbd-mirror_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:45:25.653 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking rbd-mirror (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:26.064 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package rbd-nbd. 2024-03-19T01:45:26.080 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../125-rbd-nbd_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:45:26.130 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking rbd-nbd (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:26.366 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-common-dbg. 2024-03-19T01:45:26.394 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../126-ceph-common-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:45:26.433 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-common-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:31.183 INFO:teuthology.orchestra.run.smithi055.stdout:Get:139 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 radosgw-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [339 MB] 2024-03-19T01:45:33.986 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-fuse-dbg. 2024-03-19T01:45:34.013 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../127-ceph-fuse-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:45:34.052 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-fuse-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:34.530 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-mds-dbg. 2024-03-19T01:45:34.545 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../128-ceph-mds-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:45:34.579 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-mds-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:35.632 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-mgr-dbg. 2024-03-19T01:45:35.648 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../129-ceph-mgr-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:45:35.681 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-mgr-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:36.334 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-mon-dbg. 2024-03-19T01:45:36.350 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../130-ceph-mon-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:45:36.392 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-mon-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:38.102 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-osd-dbg. 2024-03-19T01:45:38.118 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../131-ceph-osd-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:45:38.169 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-osd-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:43.479 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-test-dbg. 2024-03-19T01:45:43.495 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../132-ceph-test-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:45:43.529 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-test-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:55.162 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libcephfs2-dbg. 2024-03-19T01:45:55.178 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../133-libcephfs2-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:45:55.219 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libcephfs2-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:55.680 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package librados2-dbg. 2024-03-19T01:45:55.696 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../134-librados2-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:45:55.746 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking librados2-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:55.962 INFO:teuthology.orchestra.run.smithi055.stdout:Get:140 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [1,794 kB] 2024-03-19T01:45:55.995 INFO:teuthology.orchestra.run.smithi055.stdout:Get:141 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [97.7 MB] 2024-03-19T01:45:56.883 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libradosstriper1-dbg. 2024-03-19T01:45:56.899 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../135-libradosstriper1-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:45:56.949 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libradosstriper1-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:57.393 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package librbd1-dbg. 2024-03-19T01:45:57.421 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../136-librbd1-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:45:57.460 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking librbd1-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:45:58.662 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package librgw2-dbg. 2024-03-19T01:45:58.689 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../137-librgw2-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:45:58.729 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking librgw2-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:00.573 INFO:teuthology.orchestra.run.smithi055.stdout:Get:142 https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/ubuntu/jammy/flavors/default jammy/main amd64 rbd-nbd-dbg amd64 19.0.0-2206-gb6ae2491-1jammy [3,146 kB] 2024-03-19T01:46:00.664 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package radosgw-dbg. 2024-03-19T01:46:00.692 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../138-radosgw-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:00.730 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking radosgw-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:01.274 INFO:teuthology.orchestra.run.smithi055.stdout:Fetched 3,820 MB in 5min 13s (12.2 MB/s) 2024-03-19T01:46:01.409 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-pyperclip. 2024-03-19T01:46:01.511 INFO:teuthology.orchestra.run.smithi055.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-19T01:46:01.517 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-03-19T01:46:01.557 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-03-19T01:46:01.868 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-19T01:46:01.895 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-19T01:46:01.935 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-19T01:46:02.230 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-cmd2. 2024-03-19T01:46:02.256 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-03-19T01:46:02.296 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-03-19T01:46:02.624 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-19T01:46:02.651 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-19T01:46:02.716 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-19T01:46:03.053 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-19T01:46:03.080 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-19T01:46:03.120 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-19T01:46:03.456 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-19T01:46:03.483 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-19T01:46:03.614 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-19T01:46:03.976 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-19T01:46:04.003 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-19T01:46:04.043 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:46:04.150 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package rbd-fuse-dbg. 2024-03-19T01:46:04.166 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../139-rbd-fuse-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:04.223 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking rbd-fuse-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:04.502 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package rbd-mirror-dbg. 2024-03-19T01:46:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-19T01:46:04.530 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../140-rbd-mirror-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:04.539 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-19T01:46:04.568 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking rbd-mirror-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:04.579 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:46:04.915 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-19T01:46:04.942 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-19T01:46:04.982 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:46:05.268 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-19T01:46:05.284 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-19T01:46:05.318 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-19T01:46:05.646 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package librados2. 2024-03-19T01:46:05.673 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../010-librados2_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:05.713 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking librados2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:05.737 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package rbd-nbd-dbg. 2024-03-19T01:46:05.754 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../141-rbd-nbd-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:05.795 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking rbd-nbd-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:06.161 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-19T01:46:06.165 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package librbd1. 2024-03-19T01:46:06.192 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../011-librbd1_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:06.232 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking librbd1 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:06.400 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-19T01:46:06.526 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-19T01:46:06.627 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libcephfs2. 2024-03-19T01:46:06.653 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:06.693 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libcephfs2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:06.790 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-19T01:46:06.979 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-rados. 2024-03-19T01:46:07.006 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../013-python3-rados_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:07.046 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-rados (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:07.133 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-19T01:46:07.373 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-19T01:46:07.400 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:46:07.440 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-ceph-argparse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:07.565 INFO:teuthology.orchestra.run.smithi084.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-19T01:46:07.636 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up cephadm (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:07.676 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-cephfs. 2024-03-19T01:46:07.691 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:07.734 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-cephfs (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:08.004 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-19T01:46:08.030 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:46:08.091 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-ceph-common (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:08.162 INFO:teuthology.orchestra.run.smithi084.stdout:Adding system user cephadm....done 2024-03-19T01:46:08.310 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-19T01:46:08.406 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-prettytable. 2024-03-19T01:46:08.433 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-03-19T01:46:08.473 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-19T01:46:08.586 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-19T01:46:08.734 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-rbd. 2024-03-19T01:46:08.761 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:08.800 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-rbd (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:08.853 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-19T01:46:08.979 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-19T01:46:09.154 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-19T01:46:09.181 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-19T01:46:09.220 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-19T01:46:09.248 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-19T01:46:09.481 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-19T01:46:09.490 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-19T01:46:09.497 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-19T01:46:09.531 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-19T01:46:09.616 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-19T01:46:09.867 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-19T01:46:09.894 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-19T01:46:09.918 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-19T01:46:09.942 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-19T01:46:10.278 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-19T01:46:10.312 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package lua5.1. 2024-03-19T01:46:10.340 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-19T01:46:10.379 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-19T01:46:10.546 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-03-19T01:46:10.715 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package lua-any. 2024-03-19T01:46:10.731 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-03-19T01:46:10.774 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-19T01:46:10.797 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-19T01:46:11.048 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-19T01:46:11.076 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package zip. 2024-03-19T01:46:11.103 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-03-19T01:46:11.151 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking zip (3.0-12build2) ... 2024-03-19T01:46:11.291 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-19T01:46:11.538 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package unzip. 2024-03-19T01:46:11.565 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-19T01:46:11.567 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-19T01:46:11.613 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-19T01:46:11.835 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-19T01:46:12.050 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package luarocks. 2024-03-19T01:46:12.077 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-19T01:46:12.116 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-19T01:46:12.186 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-ceph-argparse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:12.445 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-19T01:46:12.524 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package librgw2. 2024-03-19T01:46:12.551 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../027-librgw2_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:12.571 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-19T01:46:12.599 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking librgw2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:12.689 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-03-19T01:46:12.932 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-19T01:46:13.058 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-19T01:46:13.177 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-rgw. 2024-03-19T01:46:13.204 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:13.252 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-rgw (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:13.317 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-19T01:46:13.621 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-19T01:46:13.648 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-19T01:46:13.688 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-19T01:46:13.724 INFO:teuthology.orchestra.run.smithi084.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-19T01:46:13.757 INFO:teuthology.orchestra.run.smithi084.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-19T01:46:13.820 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-19T01:46:13.947 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-19T01:46:13.999 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libradosstriper1. 2024-03-19T01:46:14.026 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:14.066 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libradosstriper1 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:14.298 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-19T01:46:14.399 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-19T01:46:14.586 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-common. 2024-03-19T01:46:14.612 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../031-ceph-common_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:14.658 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-19T01:46:14.660 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-common (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:14.934 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-19T01:46:15.268 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up zip (3.0-12build2) ... 2024-03-19T01:46:15.395 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-19T01:46:16.055 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-19T01:46:16.183 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-base. 2024-03-19T01:46:16.210 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../032-ceph-base_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:16.259 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-base (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:16.298 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-19T01:46:16.399 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-19T01:46:16.516 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-19T01:46:16.809 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-19T01:46:16.870 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-19T01:46:16.897 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-19T01:46:16.945 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-19T01:46:17.185 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-19T01:46:17.315 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-cheroot. 2024-03-19T01:46:17.342 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-19T01:46:17.390 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-19T01:46:17.436 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-19T01:46:17.784 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-19T01:46:17.811 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-19T01:46:17.829 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-19T01:46:17.851 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-19T01:46:18.131 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-03-19T01:46:18.154 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-19T01:46:18.181 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-19T01:46:18.220 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-19T01:46:18.440 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-19T01:46:18.574 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-19T01:46:18.601 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-19T01:46:18.648 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-19T01:46:18.783 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-19T01:46:19.018 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-tempora. 2024-03-19T01:46:19.034 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-19T01:46:19.045 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-03-19T01:46:19.093 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-19T01:46:19.160 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-ceph-common (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:19.413 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-portend. 2024-03-19T01:46:19.439 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-03-19T01:46:19.453 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-idna (3.3-1) ... 2024-03-19T01:46:19.479 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-19T01:46:19.754 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-19T01:46:19.840 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-19T01:46:19.867 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-19T01:46:19.915 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-19T01:46:20.285 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-19T01:46:20.312 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-19T01:46:20.360 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-19T01:46:20.688 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-natsort. 2024-03-19T01:46:20.704 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-03-19T01:46:20.747 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-19T01:46:20.993 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-19T01:46:21.058 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-logutils. 2024-03-19T01:46:21.085 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-03-19T01:46:21.125 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-19T01:46:21.236 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:46:21.345 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-19T01:46:21.428 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-19T01:46:21.454 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-19T01:46:21.494 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-19T01:46:21.597 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-19T01:46:21.706 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-19T01:46:21.815 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-19T01:46:21.822 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-mako. 2024-03-19T01:46:21.849 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-19T01:46:21.889 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-19T01:46:22.116 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-19T01:46:22.192 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-19T01:46:22.219 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-19T01:46:22.250 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-19T01:46:22.393 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-19T01:46:22.545 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-19T01:46:22.571 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-19T01:46:22.611 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-19T01:46:22.643 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:46:22.745 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-19T01:46:22.914 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-webob. 2024-03-19T01:46:22.941 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-19T01:46:22.980 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-19T01:46:23.046 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-19T01:46:23.289 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-19T01:46:23.317 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-waitress. 2024-03-19T01:46:23.344 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-19T01:46:23.387 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-19T01:46:23.540 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-19T01:46:23.686 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-tempita. 2024-03-19T01:46:23.702 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-19T01:46:23.736 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-19T01:46:23.799 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-19T01:46:23.998 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-paste. 2024-03-19T01:46:24.025 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-19T01:46:24.050 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-19T01:46:24.064 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-19T01:46:24.318 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-19T01:46:24.367 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-19T01:46:24.383 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-19T01:46:24.425 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-19T01:46:24.435 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-19T01:46:24.728 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-19T01:46:24.729 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-19T01:46:24.755 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-19T01:46:24.795 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-19T01:46:24.848 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-19T01:46:25.090 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-webtest. 2024-03-19T01:46:25.117 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-03-19T01:46:25.149 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-19T01:46:25.156 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-19T01:46:25.442 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-pecan. 2024-03-19T01:46:25.458 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-19T01:46:25.459 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-19T01:46:25.492 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-19T01:46:25.743 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-19T01:46:25.839 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-certifi. 2024-03-19T01:46:25.860 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-19T01:46:25.866 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-03-19T01:46:25.905 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-19T01:46:25.978 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-19T01:46:26.050 INFO:teuthology.orchestra.run.smithi084.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-19T01:46:26.208 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-idna. 2024-03-19T01:46:26.235 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-03-19T01:46:26.275 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-19T01:46:26.279 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-19T01:46:26.389 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-19T01:46:26.578 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-urllib3. 2024-03-19T01:46:26.604 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-19T01:46:26.644 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-19T01:46:26.782 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-19T01:46:26.964 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-requests. 2024-03-19T01:46:26.991 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-19T01:46:27.030 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-19T01:46:27.033 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-19T01:46:27.159 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-19T01:46:27.359 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-19T01:46:27.386 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-19T01:46:27.425 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-19T01:46:27.435 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:46:27.544 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-19T01:46:27.720 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-19T01:46:27.747 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:46:27.786 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:27.853 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-19T01:46:28.155 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-19T01:46:28.183 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-19T01:46:28.210 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:28.249 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:28.510 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-19T01:46:28.526 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-19T01:46:28.560 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-19T01:46:28.639 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-19T01:46:28.834 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-19T01:46:28.863 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-openssl. 2024-03-19T01:46:28.890 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-03-19T01:46:28.930 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-19T01:46:29.101 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-19T01:46:29.275 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-mgr. 2024-03-19T01:46:29.302 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:29.341 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-mgr (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:29.369 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-19T01:46:29.637 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-19T01:46:29.728 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-mon. 2024-03-19T01:46:29.755 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:29.794 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-mon (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:30.006 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-19T01:46:30.274 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-19T01:46:30.383 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-osd. 2024-03-19T01:46:30.410 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:30.449 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-osd (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:30.634 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-19T01:46:30.760 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up librados2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:30.886 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:31.012 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-19T01:46:31.332 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libcephfs2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:31.458 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libradosstriper1 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:31.584 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-19T01:46:31.694 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph. 2024-03-19T01:46:31.722 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../068-ceph_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:31.761 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:31.868 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up librados2-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:31.994 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up librbd1 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:32.080 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-fuse. 2024-03-19T01:46:32.107 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:32.120 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libcephfs2-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:32.238 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-mgr-modules-core (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:32.288 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-fuse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:32.356 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-fuse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:32.553 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-19T01:46:32.554 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-19T01:46:32.717 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-mds. 2024-03-19T01:46:32.744 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:32.783 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-mds (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:33.032 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libradosstriper1-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:33.158 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libcephfs-dev (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:33.169 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package cephadm. 2024-03-19T01:46:33.197 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../071-cephadm_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:33.236 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking cephadm (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:33.284 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-fuse-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:33.409 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-rados (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:33.535 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up librgw2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:33.555 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-19T01:46:33.583 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-19T01:46:33.622 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-19T01:46:33.661 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up cephfs-top (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:33.787 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-19T01:46:33.975 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-19T01:46:34.002 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-19T01:46:34.041 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-19T01:46:34.982 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up librbd1-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:35.013 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-babel. 2024-03-19T01:46:35.043 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-19T01:46:35.071 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-19T01:46:35.108 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-rbd (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:35.234 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up librgw2-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:35.360 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up rbd-fuse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:35.374 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-jinja2. 2024-03-19T01:46:35.390 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-19T01:46:35.432 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-19T01:46:35.486 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up rbd-nbd (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:35.612 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-rgw (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:35.727 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-19T01:46:35.729 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-cephfs (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:35.755 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:46:35.785 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:35.847 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up rbd-nbd-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:35.948 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up cephfs-shell (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:36.074 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up rbd-fuse-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:36.122 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-19T01:46:36.149 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-03-19T01:46:36.188 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-19T01:46:36.200 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-common (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:36.499 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-routes. 2024-03-19T01:46:36.527 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-19T01:46:36.566 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-19T01:46:36.768 INFO:teuthology.orchestra.run.smithi084.stdout:Adding group ceph....done 2024-03-19T01:46:36.911 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-19T01:46:36.939 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:46:36.977 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:37.144 INFO:teuthology.orchestra.run.smithi084.stdout:Adding system user ceph....done 2024-03-19T01:46:37.312 INFO:teuthology.orchestra.run.smithi084.stdout:Setting system user ceph properties....done 2024-03-19T01:46:37.329 INFO:teuthology.orchestra.run.smithi084.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-19T01:46:37.406 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-19T01:46:37.433 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-19T01:46:37.470 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-19T01:46:37.472 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-19T01:46:37.811 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-19T01:46:37.968 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-joblib. 2024-03-19T01:46:37.995 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-19T01:46:38.034 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-19T01:46:38.292 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up cephfs-mirror (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:38.371 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-19T01:46:38.398 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-19T01:46:38.501 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-19T01:46:38.501 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-19T01:46:38.523 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-19T01:46:38.851 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-sklearn. 2024-03-19T01:46:38.879 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-19T01:46:38.917 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-19T01:46:38.977 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-test (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:39.103 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up radosgw (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:39.567 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-19T01:46:39.567 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-19T01:46:39.602 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-19T01:46:39.618 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:46:39.652 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:40.079 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-base (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:40.411 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-19T01:46:40.605 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-cachetools. 2024-03-19T01:46:40.632 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-03-19T01:46:40.662 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-19T01:46:40.864 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-mds (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:40.941 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-19T01:46:40.969 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-19T01:46:41.016 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-19T01:46:41.080 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-19T01:46:41.080 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-19T01:46:41.386 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-19T01:46:41.413 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-19T01:46:41.460 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-19T01:46:41.533 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-common-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:41.659 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up rbd-mirror (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:41.822 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-rsa. 2024-03-19T01:46:41.850 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-03-19T01:46:41.856 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T01:46:41.856 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T01:46:41.896 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-19T01:46:42.276 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-google-auth. 2024-03-19T01:46:42.304 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-03-19T01:46:42.335 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-mgr (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:42.350 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-19T01:46:42.554 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-19T01:46:42.555 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-19T01:46:42.729 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-19T01:46:42.757 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-19T01:46:42.795 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-19T01:46:43.027 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-mds-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:43.090 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-websocket. 2024-03-19T01:46:43.118 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-03-19T01:46:43.153 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up radosgw-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:43.156 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-19T01:46:43.288 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-test-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:43.422 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-osd (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:43.518 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-19T01:46:43.546 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-19T01:46:43.593 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-19T01:46:43.694 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-19T01:46:43.694 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-19T01:46:44.199 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-mgr-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:44.299 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-19T01:46:44.315 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:46:44.317 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-osd-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:44.357 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:44.426 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up rbd-mirror-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:44.527 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:44.618 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-19T01:46:44.645 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:44.646 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-19T01:46:44.676 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-19T01:46:44.796 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-mon (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:45.021 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-19T01:46:45.022 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-19T01:46:45.022 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-19T01:46:45.049 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-19T01:46:45.087 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-19T01:46:45.331 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package jq. 2024-03-19T01:46:45.348 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-19T01:46:45.381 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-19T01:46:45.514 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-mgr-cephadm (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:45.601 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package socat. 2024-03-19T01:46:45.617 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-19T01:46:45.659 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-19T01:46:45.673 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:45.799 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-mgr-dashboard (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:45.950 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-mon-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:45.987 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package xmlstarlet. 2024-03-19T01:46:46.004 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-19T01:46:46.045 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-19T01:46:46.059 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-volume (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:46.400 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-test. 2024-03-19T01:46:46.403 INFO:teuthology.orchestra.run.smithi084.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-19T01:46:46.428 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../099-ceph-test_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:46.466 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-test (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:46.512 INFO:teuthology.orchestra.run.smithi084.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-19T01:46:46.665 INFO:teuthology.orchestra.run.smithi084.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-19T01:46:47.882 INFO:teuthology.orchestra.run.smithi084.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-19T01:46:48.584 INFO:teuthology.orchestra.run.smithi084.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-19T01:46:48.588 DEBUG:teuthology.orchestra.run.smithi084:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-03-19T01:46:48.642 INFO:teuthology.orchestra.run.smithi084.stdout:Reading package lists... 2024-03-19T01:46:48.787 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-volume. 2024-03-19T01:46:48.815 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:46:48.839 INFO:teuthology.orchestra.run.smithi084.stdout:Building dependency tree... 2024-03-19T01:46:48.839 INFO:teuthology.orchestra.run.smithi084.stdout:Reading state information... 2024-03-19T01:46:48.853 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-volume (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:49.019 INFO:teuthology.orchestra.run.smithi084.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-03-19T01:46:49.019 INFO:teuthology.orchestra.run.smithi084.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-03-19T01:46:49.019 INFO:teuthology.orchestra.run.smithi084.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2024-03-19T01:46:49.019 INFO:teuthology.orchestra.run.smithi084.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2024-03-19T01:46:49.020 INFO:teuthology.orchestra.run.smithi084.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-03-19T01:46:49.020 INFO:teuthology.orchestra.run.smithi084.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T01:46:49.020 INFO:teuthology.orchestra.run.smithi084.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T01:46:49.020 INFO:teuthology.orchestra.run.smithi084.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T01:46:49.020 INFO:teuthology.orchestra.run.smithi084.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T01:46:49.020 INFO:teuthology.orchestra.run.smithi084.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-19T01:46:49.020 INFO:teuthology.orchestra.run.smithi084.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-19T01:46:49.020 INFO:teuthology.orchestra.run.smithi084.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T01:46:49.021 INFO:teuthology.orchestra.run.smithi084.stdout:The following additional packages will be installed: 2024-03-19T01:46:49.021 INFO:teuthology.orchestra.run.smithi084.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-03-19T01:46:49.021 INFO:teuthology.orchestra.run.smithi084.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-03-19T01:46:49.021 INFO:teuthology.orchestra.run.smithi084.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2024-03-19T01:46:49.021 INFO:teuthology.orchestra.run.smithi084.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-03-19T01:46:49.021 INFO:teuthology.orchestra.run.smithi084.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-19T01:46:49.022 INFO:teuthology.orchestra.run.smithi084.stdout:Suggested packages: 2024-03-19T01:46:49.022 INFO:teuthology.orchestra.run.smithi084.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-03-19T01:46:49.022 INFO:teuthology.orchestra.run.smithi084.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2024-03-19T01:46:49.022 INFO:teuthology.orchestra.run.smithi084.stdout: libengine-pkcs11-openssl 2024-03-19T01:46:49.059 INFO:teuthology.orchestra.run.smithi084.stdout:The following NEW packages will be installed: 2024-03-19T01:46:49.060 INFO:teuthology.orchestra.run.smithi084.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-03-19T01:46:49.060 INFO:teuthology.orchestra.run.smithi084.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-03-19T01:46:49.060 INFO:teuthology.orchestra.run.smithi084.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-03-19T01:46:49.061 INFO:teuthology.orchestra.run.smithi084.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-03-19T01:46:49.061 INFO:teuthology.orchestra.run.smithi084.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-03-19T01:46:49.061 INFO:teuthology.orchestra.run.smithi084.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-03-19T01:46:49.061 INFO:teuthology.orchestra.run.smithi084.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-19T01:46:49.062 INFO:teuthology.orchestra.run.smithi084.stdout:The following packages will be upgraded: 2024-03-19T01:46:49.062 INFO:teuthology.orchestra.run.smithi084.stdout: libssl3 2024-03-19T01:46:49.120 INFO:teuthology.orchestra.run.smithi084.stdout:1 upgraded, 39 newly installed, 0 to remove and 284 not upgraded. 2024-03-19T01:46:49.120 INFO:teuthology.orchestra.run.smithi084.stdout:Need to get 54.6 MB of archives. 2024-03-19T01:46:49.120 INFO:teuthology.orchestra.run.smithi084.stdout:After this operation, 202 MB of additional disk space will be used. 2024-03-19T01:46:49.120 INFO:teuthology.orchestra.run.smithi084.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-03-19T01:46:49.233 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package cephfs-mirror. 2024-03-19T01:46:49.242 INFO:teuthology.orchestra.run.smithi084.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-03-19T01:46:49.261 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:49.299 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking cephfs-mirror (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:49.313 INFO:teuthology.orchestra.run.smithi084.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-03-19T01:46:49.330 INFO:teuthology.orchestra.run.smithi084.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-03-19T01:46:49.332 INFO:teuthology.orchestra.run.smithi084.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-03-19T01:46:49.335 INFO:teuthology.orchestra.run.smithi084.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-03-19T01:46:49.336 INFO:teuthology.orchestra.run.smithi084.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-03-19T01:46:49.340 INFO:teuthology.orchestra.run.smithi084.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-03-19T01:46:49.340 INFO:teuthology.orchestra.run.smithi084.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-03-19T01:46:49.341 INFO:teuthology.orchestra.run.smithi084.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-03-19T01:46:49.341 INFO:teuthology.orchestra.run.smithi084.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-03-19T01:46:49.341 INFO:teuthology.orchestra.run.smithi084.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-03-19T01:46:49.360 INFO:teuthology.orchestra.run.smithi084.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-03-19T01:46:49.602 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-colorama. 2024-03-19T01:46:49.630 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-03-19T01:46:49.660 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-03-19T01:46:49.863 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package cephfs-shell. 2024-03-19T01:46:49.879 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:46:49.902 INFO:teuthology.orchestra.run.smithi084.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-03-19T01:46:49.906 INFO:teuthology.orchestra.run.smithi084.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-03-19T01:46:49.907 INFO:teuthology.orchestra.run.smithi084.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-03-19T01:46:49.913 INFO:teuthology.orchestra.run.smithi084.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-03-19T01:46:49.913 INFO:teuthology.orchestra.run.smithi084.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-03-19T01:46:49.921 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking cephfs-shell (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:49.923 INFO:teuthology.orchestra.run.smithi084.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-03-19T01:46:49.927 INFO:teuthology.orchestra.run.smithi084.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-03-19T01:46:49.943 INFO:teuthology.orchestra.run.smithi084.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-03-19T01:46:49.943 INFO:teuthology.orchestra.run.smithi084.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-03-19T01:46:49.978 INFO:teuthology.orchestra.run.smithi084.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-03-19T01:46:49.978 INFO:teuthology.orchestra.run.smithi084.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-03-19T01:46:50.011 INFO:teuthology.orchestra.run.smithi084.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-03-19T01:46:50.122 INFO:teuthology.orchestra.run.smithi084.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-03-19T01:46:50.141 INFO:teuthology.orchestra.run.smithi084.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-03-19T01:46:50.143 INFO:teuthology.orchestra.run.smithi084.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-03-19T01:46:50.144 INFO:teuthology.orchestra.run.smithi084.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-03-19T01:46:50.145 INFO:teuthology.orchestra.run.smithi084.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-03-19T01:46:50.173 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package cephfs-top. 2024-03-19T01:46:50.190 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-2206-gb6ae2491-1jammy_all.deb ... 2024-03-19T01:46:50.204 INFO:teuthology.orchestra.run.smithi084.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-03-19T01:46:50.209 INFO:teuthology.orchestra.run.smithi084.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-03-19T01:46:50.232 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking cephfs-top (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:50.509 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-19T01:46:50.526 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:50.568 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libcephfs-dev (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:50.660 INFO:teuthology.orchestra.run.smithi084.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-03-19T01:46:50.660 INFO:teuthology.orchestra.run.smithi084.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-03-19T01:46:50.660 INFO:teuthology.orchestra.run.smithi084.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-03-19T01:46:50.660 INFO:teuthology.orchestra.run.smithi084.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-03-19T01:46:50.661 INFO:teuthology.orchestra.run.smithi084.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-03-19T01:46:50.661 INFO:teuthology.orchestra.run.smithi084.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-03-19T01:46:50.676 INFO:teuthology.orchestra.run.smithi084.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-03-19T01:46:50.676 INFO:teuthology.orchestra.run.smithi084.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-03-19T01:46:50.895 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-19T01:46:50.923 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-19T01:46:50.962 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-19T01:46:51.081 INFO:teuthology.orchestra.run.smithi084.stdout:Preconfiguring packages ... 2024-03-19T01:46:51.233 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-19T01:46:51.245 INFO:teuthology.orchestra.run.smithi084.stdout:Fetched 54.6 MB in 2s (34.0 MB/s) 2024-03-19T01:46:51.249 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-03-19T01:46:51.283 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-19T01:46:51.355 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package flex. 2024-03-19T01:46:51.445 INFO:teuthology.orchestra.run.smithi084.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173943 files and directories currently installed.) 2024-03-19T01:46:51.450 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-03-19T01:46:51.495 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking flex (2.6.4-8build2) ... 2024-03-19T01:46:51.679 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package nvme-cli. 2024-03-19T01:46:51.707 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-19T01:46:51.745 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-19T01:46:52.079 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-19T01:46:52.150 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package pkg-config. 2024-03-19T01:46:52.178 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-19T01:46:52.216 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-19T01:46:52.251 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-03-19T01:46:52.478 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-19T01:46:52.494 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-19T01:46:52.536 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-19T01:46:52.694 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-19T01:46:52.972 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-19T01:46:53.000 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-19T01:46:53.039 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-19T01:46:53.235 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package bison. 2024-03-19T01:46:53.317 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-pastescript. 2024-03-19T01:46:53.339 INFO:teuthology.orchestra.run.smithi084.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174016 files and directories currently installed.) 2024-03-19T01:46:53.345 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-03-19T01:46:53.346 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-03-19T01:46:53.375 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-19T01:46:53.395 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-03-19T01:46:53.670 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-pluggy. 2024-03-19T01:46:53.698 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-19T01:46:53.736 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-19T01:46:53.829 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-03-19T01:46:53.857 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-03-19T01:46:53.896 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-03-19T01:46:54.056 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-psutil. 2024-03-19T01:46:54.085 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-19T01:46:54.122 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-19T01:46:54.350 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package dump. 2024-03-19T01:46:54.378 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-03-19T01:46:54.441 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking dump (0.4b47-3) ... 2024-03-19T01:46:54.476 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-py. 2024-03-19T01:46:54.504 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-03-19T01:46:54.542 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-19T01:46:54.728 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package indent. 2024-03-19T01:46:54.743 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:46:54.786 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-19T01:46:54.871 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-pygments. 2024-03-19T01:46:54.899 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-19T01:46:54.937 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-19T01:46:55.055 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-03-19T01:46:55.071 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-03-19T01:46:55.113 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-19T01:46:55.408 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-03-19T01:46:55.424 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-19T01:46:55.436 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-03-19T01:46:55.453 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-19T01:46:55.467 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-19T01:46:55.491 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-19T01:46:55.711 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-03-19T01:46:55.726 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-03-19T01:46:55.769 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-03-19T01:46:55.802 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-toml. 2024-03-19T01:46:55.831 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-03-19T01:46:55.869 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-19T01:46:56.114 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libfl2:amd64. 2024-03-19T01:46:56.129 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-pytest. 2024-03-19T01:46:56.141 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-03-19T01:46:56.146 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-19T01:46:56.179 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-19T01:46:56.180 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-03-19T01:46:56.416 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-03-19T01:46:56.432 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-03-19T01:46:56.474 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-19T01:46:56.525 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-simplejson. 2024-03-19T01:46:56.553 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-19T01:46:56.591 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-19T01:46:56.718 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libjson-perl. 2024-03-19T01:46:56.734 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-03-19T01:46:56.768 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-03-19T01:46:56.869 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-19T01:46:56.898 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-19T01:46:56.935 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-19T01:46:57.071 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libllvm14:amd64. 2024-03-19T01:46:57.099 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-03-19T01:46:57.138 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-19T01:46:57.681 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package radosgw. 2024-03-19T01:46:57.709 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../122-radosgw_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:57.747 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking radosgw (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:58.676 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package rbd-fuse. 2024-03-19T01:46:58.693 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:58.725 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking rbd-fuse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:58.865 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-03-19T01:46:58.893 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-19T01:46:58.931 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-19T01:46:59.070 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package rbd-mirror. 2024-03-19T01:46:59.099 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../124-rbd-mirror_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:59.136 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking rbd-mirror (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:59.184 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libmbim-proxy. 2024-03-19T01:46:59.200 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-19T01:46:59.259 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-19T01:46:59.539 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package rbd-nbd. 2024-03-19T01:46:59.559 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../125-rbd-nbd_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:59.597 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking rbd-nbd (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:59.866 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-common-dbg. 2024-03-19T01:46:59.883 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../126-ceph-common-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:46:59.925 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-common-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:46:59.970 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-03-19T01:46:59.998 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-19T01:47:00.029 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-19T01:47:00.381 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libndp0:amd64. 2024-03-19T01:47:00.409 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-03-19T01:47:00.448 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-19T01:47:00.784 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libnm0:amd64. 2024-03-19T01:47:00.811 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-03-19T01:47:00.851 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-19T01:47:01.194 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libpq5:amd64. 2024-03-19T01:47:01.222 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:47:01.261 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:47:01.630 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-03-19T01:47:01.658 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:47:01.697 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-19T01:47:01.999 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libqmi-proxy. 2024-03-19T01:47:02.027 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:47:02.066 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-19T01:47:02.310 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-03-19T01:47:02.325 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-19T01:47:02.368 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-19T01:47:02.855 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-03-19T01:47:02.882 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-03-19T01:47:02.921 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-03-19T01:47:03.257 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package modemmanager. 2024-03-19T01:47:03.285 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-19T01:47:03.324 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-19T01:47:03.963 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package network-manager. 2024-03-19T01:47:03.990 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-03-19T01:47:04.037 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-03-19T01:47:04.644 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ppp. 2024-03-19T01:47:04.672 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-03-19T01:47:04.939 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-03-19T01:47:05.300 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package pptp-linux. 2024-03-19T01:47:05.316 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-03-19T01:47:05.350 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-03-19T01:47:05.602 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package network-manager-pptp. 2024-03-19T01:47:05.630 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-03-19T01:47:05.669 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-03-19T01:47:05.964 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package postgresql-client-common. 2024-03-19T01:47:05.992 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-03-19T01:47:06.030 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking postgresql-client-common (238) ... 2024-03-19T01:47:06.334 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package postgresql-client-14. 2024-03-19T01:47:06.362 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:47:06.400 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:47:06.947 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package postgresql-common. 2024-03-19T01:47:06.975 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-03-19T01:47:07.024 INFO:teuthology.orchestra.run.smithi084.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-03-19T01:47:07.052 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking postgresql-common (238) ... 2024-03-19T01:47:07.512 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package postgresql-14. 2024-03-19T01:47:07.540 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:47:07.581 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:47:07.920 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-fuse-dbg. 2024-03-19T01:47:07.937 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../127-ceph-fuse-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:47:07.970 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-fuse-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:47:08.431 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-mds-dbg. 2024-03-19T01:47:08.459 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../128-ceph-mds-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:47:08.497 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-mds-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:47:08.997 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package postgresql. 2024-03-19T01:47:09.014 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-03-19T01:47:09.055 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking postgresql (14+238) ... 2024-03-19T01:47:09.282 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package postgresql-client. 2024-03-19T01:47:09.299 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-03-19T01:47:09.332 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking postgresql-client (14+238) ... 2024-03-19T01:47:09.577 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package postgresql-contrib. 2024-03-19T01:47:09.604 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-03-19T01:47:09.625 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-mgr-dbg. 2024-03-19T01:47:09.642 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../129-ceph-mgr-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:47:09.646 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking postgresql-contrib (14+238) ... 2024-03-19T01:47:09.683 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-mgr-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:47:09.971 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package tcl8.6. 2024-03-19T01:47:09.999 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-03-19T01:47:10.046 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-19T01:47:10.323 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package tcl. 2024-03-19T01:47:10.351 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-03-19T01:47:10.394 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-03-19T01:47:10.402 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-mon-dbg. 2024-03-19T01:47:10.419 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../130-ceph-mon-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:47:10.460 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-mon-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:47:10.692 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package wpasupplicant. 2024-03-19T01:47:10.720 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-03-19T01:47:10.750 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-19T01:47:11.131 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-03-19T01:47:11.148 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-03-19T01:47:11.190 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-03-19T01:47:11.559 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package usb-modeswitch. 2024-03-19T01:47:11.588 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-03-19T01:47:11.634 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-19T01:47:11.921 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up postgresql-client-common (238) ... 2024-03-19T01:47:12.140 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up flex (2.6.4-8build2) ... 2024-03-19T01:47:12.229 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-osd-dbg. 2024-03-19T01:47:12.257 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../131-ceph-osd-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:47:12.266 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-19T01:47:12.295 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-osd-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:47:12.408 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:47:12.535 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-03-19T01:47:12.685 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-19T01:47:12.836 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:47:13.346 INFO:teuthology.orchestra.run.smithi084.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-03-19T01:47:13.442 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-19T01:47:13.846 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-19T01:47:13.846 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-19T01:47:14.319 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-03-19T01:47:14.470 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-19T01:47:14.621 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-03-19T01:47:14.747 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-03-19T01:47:14.873 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-03-19T01:47:14.999 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-19T01:47:15.134 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-03-19T01:47:15.922 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-19T01:47:16.157 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up dump (0.4b47-3) ... 2024-03-19T01:47:16.250 INFO:teuthology.orchestra.run.smithi084.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-03-19T01:47:16.325 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-19T01:47:16.451 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-19T01:47:16.593 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up tcl (8.6.11+1build2) ... 2024-03-19T01:47:16.744 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libjson-perl (4.04000-1) ... 2024-03-19T01:47:16.895 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-03-19T01:47:17.002 INFO:teuthology.orchestra.run.smithi084.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-03-19T01:47:17.080 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-19T01:47:17.215 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-19T01:47:17.391 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-19T01:47:17.517 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-19T01:47:17.643 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-03-19T01:47:17.665 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-test-dbg. 2024-03-19T01:47:17.682 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../132-ceph-test-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:47:17.732 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-test-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:47:17.828 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up postgresql-client (14+238) ... 2024-03-19T01:47:17.978 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-03-19T01:47:18.485 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:47:18.485 INFO:teuthology.orchestra.run.smithi084.stdout:The following network interfaces were found in /etc/network/interfaces 2024-03-19T01:47:18.486 INFO:teuthology.orchestra.run.smithi084.stdout:which means they are currently configured by ifupdown: 2024-03-19T01:47:18.486 INFO:teuthology.orchestra.run.smithi084.stdout:- enp3s0f1 2024-03-19T01:47:18.486 INFO:teuthology.orchestra.run.smithi084.stdout:If you want to manage those interfaces with NetworkManager instead 2024-03-19T01:47:18.486 INFO:teuthology.orchestra.run.smithi084.stdout:remove their configuration from /etc/network/interfaces. 2024-03-19T01:47:18.486 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:47:18.607 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-03-19T01:47:18.937 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-03-19T01:47:19.269 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-03-19T01:47:19.781 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-19T01:47:19.932 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-19T01:47:20.058 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-03-19T01:47:20.176 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up postgresql-common (238) ... 2024-03-19T01:47:21.477 INFO:teuthology.orchestra.run.smithi084.stdout:Adding user postgres to group ssl-cert 2024-03-19T01:47:21.726 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:47:21.726 INFO:teuthology.orchestra.run.smithi084.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-03-19T01:47:21.903 INFO:teuthology.orchestra.run.smithi084.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-03-19T01:47:21.904 INFO:teuthology.orchestra.run.smithi084.stdout:Removing obsolete dictionary files: 2024-03-19T01:47:22.462 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-03-19T01:47:22.836 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:47:23.181 INFO:teuthology.orchestra.run.smithi084.stdout:Creating new PostgreSQL cluster 14/main ... 2024-03-19T01:47:23.203 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-03-19T01:47:23.238 INFO:teuthology.orchestra.run.smithi084.stdout:The files belonging to this database system will be owned by user "postgres". 2024-03-19T01:47:23.238 INFO:teuthology.orchestra.run.smithi084.stdout:This user must also own the server process. 2024-03-19T01:47:23.238 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:47:23.238 INFO:teuthology.orchestra.run.smithi084.stdout:The database cluster will be initialized with locale "en_US". 2024-03-19T01:47:23.239 INFO:teuthology.orchestra.run.smithi084.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-03-19T01:47:23.239 INFO:teuthology.orchestra.run.smithi084.stdout:The default text search configuration will be set to "english". 2024-03-19T01:47:23.239 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:47:23.239 INFO:teuthology.orchestra.run.smithi084.stdout:Data page checksums are disabled. 2024-03-19T01:47:23.239 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:47:23.239 INFO:teuthology.orchestra.run.smithi084.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-03-19T01:47:23.239 INFO:teuthology.orchestra.run.smithi084.stdout:creating subdirectories ... ok 2024-03-19T01:47:23.240 INFO:teuthology.orchestra.run.smithi084.stdout:selecting dynamic shared memory implementation ... posix 2024-03-19T01:47:23.250 INFO:teuthology.orchestra.run.smithi084.stdout:selecting default max_connections ... 100 2024-03-19T01:47:23.267 INFO:teuthology.orchestra.run.smithi084.stdout:selecting default shared_buffers ... 128MB 2024-03-19T01:47:23.277 INFO:teuthology.orchestra.run.smithi084.stdout:selecting default time zone ... Etc/UTC 2024-03-19T01:47:23.278 INFO:teuthology.orchestra.run.smithi084.stdout:creating configuration files ... ok 2024-03-19T01:47:23.381 INFO:teuthology.orchestra.run.smithi084.stdout:running bootstrap script ... ok 2024-03-19T01:47:24.026 INFO:teuthology.orchestra.run.smithi084.stdout:performing post-bootstrap initialization ... ok 2024-03-19T01:47:24.640 INFO:teuthology.orchestra.run.smithi084.stdout:syncing data to disk ... ok 2024-03-19T01:47:25.043 INFO:teuthology.orchestra.run.smithi084.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-03-19T01:47:27.800 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-19T01:47:27.943 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up postgresql-contrib (14+238) ... 2024-03-19T01:47:28.093 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up postgresql (14+238) ... 2024-03-19T01:47:28.245 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-19T01:47:28.396 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-19T01:47:28.662 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-03-19T01:47:28.662 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-03-19T01:47:29.231 INFO:teuthology.orchestra.run.smithi084.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-03-19T01:47:29.315 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libcephfs2-dbg. 2024-03-19T01:47:29.344 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../133-libcephfs2-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:47:29.381 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libcephfs2-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:47:29.525 INFO:teuthology.orchestra.run.smithi084.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-19T01:47:29.685 INFO:teuthology.orchestra.run.smithi084.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-19T01:47:29.792 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package librados2-dbg. 2024-03-19T01:47:29.809 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../134-librados2-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:47:29.842 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking librados2-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:47:30.979 INFO:teuthology.orchestra.run.smithi084.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-03-19T01:47:31.053 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libradosstriper1-dbg. 2024-03-19T01:47:31.069 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../135-libradosstriper1-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:47:31.087 INFO:teuthology.orchestra.run.smithi084.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-19T01:47:31.111 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libradosstriper1-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:47:31.514 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package librbd1-dbg. 2024-03-19T01:47:31.542 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../136-librbd1-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:47:31.580 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking librbd1-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:47:31.622 INFO:teuthology.orchestra.run.smithi084.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-19T01:47:31.628 DEBUG:teuthology.parallel:result is None 2024-03-19T01:47:32.858 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package librgw2-dbg. 2024-03-19T01:47:32.887 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../137-librgw2-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:47:32.924 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking librgw2-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:47:34.843 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package radosgw-dbg. 2024-03-19T01:47:34.872 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../138-radosgw-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:47:34.909 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking radosgw-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:47:38.429 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package rbd-fuse-dbg. 2024-03-19T01:47:38.458 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../139-rbd-fuse-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:47:38.495 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking rbd-fuse-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:47:38.790 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package rbd-mirror-dbg. 2024-03-19T01:47:38.819 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../140-rbd-mirror-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:47:38.856 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking rbd-mirror-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:47:40.084 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package rbd-nbd-dbg. 2024-03-19T01:47:40.112 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../141-rbd-nbd-dbg_19.0.0-2206-gb6ae2491-1jammy_amd64.deb ... 2024-03-19T01:47:40.142 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking rbd-nbd-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:47:40.514 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-19T01:47:40.747 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-19T01:47:40.873 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-19T01:47:41.138 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-19T01:47:41.474 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-19T01:47:41.884 INFO:teuthology.orchestra.run.smithi055.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-19T01:47:41.959 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up cephadm (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:47:42.501 INFO:teuthology.orchestra.run.smithi055.stdout:Adding system user cephadm....done 2024-03-19T01:47:42.623 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-19T01:47:42.891 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-19T01:47:43.151 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-19T01:47:43.277 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-19T01:47:43.553 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-19T01:47:43.821 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-19T01:47:43.955 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-19T01:47:44.257 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-19T01:47:44.617 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-19T01:47:44.885 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-03-19T01:47:45.153 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-19T01:47:45.429 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-19T01:47:45.689 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-19T01:47:45.948 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-19T01:47:46.191 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-19T01:47:46.451 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-ceph-argparse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:47:46.718 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-19T01:47:46.844 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-19T01:47:46.962 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-03-19T01:47:47.222 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-19T01:47:47.348 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-19T01:47:47.582 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-19T01:47:47.976 INFO:teuthology.orchestra.run.smithi055.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-19T01:47:48.031 INFO:teuthology.orchestra.run.smithi055.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-19T01:47:48.094 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-19T01:47:48.220 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-19T01:47:48.588 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-19T01:47:48.708 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-19T01:47:48.993 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-19T01:47:49.252 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-19T01:47:49.570 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up zip (3.0-12build2) ... 2024-03-19T01:47:49.696 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-19T01:47:50.365 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-19T01:47:50.625 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-19T01:47:50.751 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-19T01:47:50.877 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-19T01:47:51.195 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-19T01:47:51.588 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-19T01:47:51.823 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-19T01:47:52.200 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-19T01:47:52.493 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-03-19T01:47:52.811 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-19T01:47:53.162 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-19T01:47:53.414 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-19T01:47:53.515 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-ceph-common (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:47:53.825 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-idna (3.3-1) ... 2024-03-19T01:47:54.143 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-19T01:47:55.333 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-19T01:47:55.585 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:47:55.736 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-19T01:47:56.004 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-19T01:47:56.122 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-19T01:47:56.223 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-19T01:47:56.516 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-19T01:47:56.776 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-19T01:47:57.044 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:47:57.170 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-19T01:47:57.480 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-19T01:47:57.723 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-19T01:47:57.966 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-19T01:47:58.225 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-19T01:47:58.494 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-19T01:47:58.761 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-19T01:47:58.887 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-19T01:47:59.189 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-19T01:47:59.298 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-19T01:47:59.583 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-19T01:47:59.893 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-19T01:48:00.155 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-19T01:48:00.295 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-19T01:48:00.421 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-19T01:48:00.501 INFO:teuthology.orchestra.run.smithi055.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-19T01:48:00.748 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-19T01:48:00.858 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-19T01:48:01.268 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-19T01:48:01.788 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-19T01:48:02.003 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-19T01:48:02.271 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T01:48:02.389 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-19T01:48:02.707 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-19T01:48:03.025 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-19T01:48:03.335 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-19T01:48:03.555 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-19T01:48:03.806 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-19T01:48:04.075 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-19T01:48:04.343 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-19T01:48:04.728 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-19T01:48:05.030 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-19T01:48:05.399 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-19T01:48:05.525 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up librados2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:05.652 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:05.778 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-19T01:48:06.090 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libcephfs2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:06.191 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libradosstriper1 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:06.325 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-19T01:48:06.611 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up librados2-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:06.736 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up librbd1 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:06.863 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libcephfs2-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:06.989 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-mgr-modules-core (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:07.107 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-fuse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:07.334 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-19T01:48:07.334 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-19T01:48:07.800 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libradosstriper1-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:07.926 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libcephfs-dev (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:08.052 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-fuse-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:08.178 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-rados (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:08.304 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up librgw2 (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:08.422 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up cephfs-top (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:08.523 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-19T01:48:09.803 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up librbd1-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:09.929 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-rbd (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:10.055 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up librgw2-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:10.173 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up rbd-fuse (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:10.274 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up rbd-nbd (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:10.400 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-rgw (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:10.526 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-cephfs (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:10.652 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up rbd-nbd-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:10.779 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up cephfs-shell (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:10.905 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up rbd-fuse-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:11.048 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-common (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:11.617 INFO:teuthology.orchestra.run.smithi055.stdout:Adding group ceph....done 2024-03-19T01:48:11.997 INFO:teuthology.orchestra.run.smithi055.stdout:Adding system user ceph....done 2024-03-19T01:48:12.279 INFO:teuthology.orchestra.run.smithi055.stdout:Setting system user ceph properties....done 2024-03-19T01:48:12.294 INFO:teuthology.orchestra.run.smithi055.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-19T01:48:12.441 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-19T01:48:12.782 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-19T01:48:13.510 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up cephfs-mirror (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:13.727 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-19T01:48:13.727 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-19T01:48:14.195 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-test (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:14.321 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up radosgw (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:14.796 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-19T01:48:14.796 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-19T01:48:15.248 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-base (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:15.600 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-19T01:48:16.059 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-mds (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:16.269 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-19T01:48:16.269 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-19T01:48:16.711 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-common-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:16.854 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up rbd-mirror (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:17.059 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T01:48:17.059 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T01:48:17.838 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-mgr (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:18.142 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-19T01:48:18.142 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-19T01:48:18.661 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-mds-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:18.840 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up radosgw-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:18.967 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-test-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:19.093 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-osd (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:19.381 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-19T01:48:19.381 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-19T01:48:19.853 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-mgr-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:19.954 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-osd-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:20.072 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up rbd-mirror-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:20.198 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:20.324 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:20.484 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-mon (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:20.703 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-19T01:48:20.703 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-19T01:48:21.152 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-mgr-cephadm (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:21.303 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:21.429 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-mgr-dashboard (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:21.555 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-mon-dbg (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:21.656 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-volume (19.0.0-2206-gb6ae2491-1jammy) ... 2024-03-19T01:48:22.026 INFO:teuthology.orchestra.run.smithi055.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-19T01:48:22.127 INFO:teuthology.orchestra.run.smithi055.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-19T01:48:22.272 INFO:teuthology.orchestra.run.smithi055.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-19T01:48:23.231 INFO:teuthology.orchestra.run.smithi055.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-19T01:48:23.910 INFO:teuthology.orchestra.run.smithi055.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-19T01:48:23.913 DEBUG:teuthology.orchestra.run.smithi055:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-03-19T01:48:23.967 INFO:teuthology.orchestra.run.smithi055.stdout:Reading package lists... 2024-03-19T01:48:24.168 INFO:teuthology.orchestra.run.smithi055.stdout:Building dependency tree... 2024-03-19T01:48:24.169 INFO:teuthology.orchestra.run.smithi055.stdout:Reading state information... 2024-03-19T01:48:24.351 INFO:teuthology.orchestra.run.smithi055.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-03-19T01:48:24.352 INFO:teuthology.orchestra.run.smithi055.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-03-19T01:48:24.352 INFO:teuthology.orchestra.run.smithi055.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2024-03-19T01:48:24.352 INFO:teuthology.orchestra.run.smithi055.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2024-03-19T01:48:24.352 INFO:teuthology.orchestra.run.smithi055.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-03-19T01:48:24.352 INFO:teuthology.orchestra.run.smithi055.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T01:48:24.352 INFO:teuthology.orchestra.run.smithi055.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T01:48:24.352 INFO:teuthology.orchestra.run.smithi055.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T01:48:24.352 INFO:teuthology.orchestra.run.smithi055.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T01:48:24.353 INFO:teuthology.orchestra.run.smithi055.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-19T01:48:24.353 INFO:teuthology.orchestra.run.smithi055.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-19T01:48:24.353 INFO:teuthology.orchestra.run.smithi055.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T01:48:24.353 INFO:teuthology.orchestra.run.smithi055.stdout:The following additional packages will be installed: 2024-03-19T01:48:24.354 INFO:teuthology.orchestra.run.smithi055.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-03-19T01:48:24.354 INFO:teuthology.orchestra.run.smithi055.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-03-19T01:48:24.354 INFO:teuthology.orchestra.run.smithi055.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2024-03-19T01:48:24.354 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-03-19T01:48:24.354 INFO:teuthology.orchestra.run.smithi055.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-19T01:48:24.355 INFO:teuthology.orchestra.run.smithi055.stdout:Suggested packages: 2024-03-19T01:48:24.355 INFO:teuthology.orchestra.run.smithi055.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-03-19T01:48:24.355 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2024-03-19T01:48:24.356 INFO:teuthology.orchestra.run.smithi055.stdout: libengine-pkcs11-openssl 2024-03-19T01:48:24.396 INFO:teuthology.orchestra.run.smithi055.stdout:The following NEW packages will be installed: 2024-03-19T01:48:24.397 INFO:teuthology.orchestra.run.smithi055.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-03-19T01:48:24.397 INFO:teuthology.orchestra.run.smithi055.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-03-19T01:48:24.397 INFO:teuthology.orchestra.run.smithi055.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-03-19T01:48:24.397 INFO:teuthology.orchestra.run.smithi055.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-03-19T01:48:24.398 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-03-19T01:48:24.398 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-03-19T01:48:24.398 INFO:teuthology.orchestra.run.smithi055.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-19T01:48:24.399 INFO:teuthology.orchestra.run.smithi055.stdout:The following packages will be upgraded: 2024-03-19T01:48:24.400 INFO:teuthology.orchestra.run.smithi055.stdout: libssl3 2024-03-19T01:48:24.503 INFO:teuthology.orchestra.run.smithi055.stdout:1 upgraded, 39 newly installed, 0 to remove and 284 not upgraded. 2024-03-19T01:48:24.503 INFO:teuthology.orchestra.run.smithi055.stdout:Need to get 54.6 MB of archives. 2024-03-19T01:48:24.503 INFO:teuthology.orchestra.run.smithi055.stdout:After this operation, 202 MB of additional disk space will be used. 2024-03-19T01:48:24.503 INFO:teuthology.orchestra.run.smithi055.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-03-19T01:48:24.711 INFO:teuthology.orchestra.run.smithi055.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-03-19T01:48:24.795 INFO:teuthology.orchestra.run.smithi055.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-03-19T01:48:24.813 INFO:teuthology.orchestra.run.smithi055.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-03-19T01:48:24.814 INFO:teuthology.orchestra.run.smithi055.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-03-19T01:48:24.818 INFO:teuthology.orchestra.run.smithi055.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-03-19T01:48:24.820 INFO:teuthology.orchestra.run.smithi055.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-03-19T01:48:24.820 INFO:teuthology.orchestra.run.smithi055.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-03-19T01:48:24.822 INFO:teuthology.orchestra.run.smithi055.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-03-19T01:48:24.823 INFO:teuthology.orchestra.run.smithi055.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-03-19T01:48:24.824 INFO:teuthology.orchestra.run.smithi055.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-03-19T01:48:24.829 INFO:teuthology.orchestra.run.smithi055.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-03-19T01:48:24.845 INFO:teuthology.orchestra.run.smithi055.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-03-19T01:48:25.254 INFO:teuthology.orchestra.run.smithi055.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-03-19T01:48:25.256 INFO:teuthology.orchestra.run.smithi055.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-03-19T01:48:25.256 INFO:teuthology.orchestra.run.smithi055.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-03-19T01:48:25.259 INFO:teuthology.orchestra.run.smithi055.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-03-19T01:48:25.311 INFO:teuthology.orchestra.run.smithi055.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-03-19T01:48:25.318 INFO:teuthology.orchestra.run.smithi055.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-03-19T01:48:25.320 INFO:teuthology.orchestra.run.smithi055.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-03-19T01:48:25.329 INFO:teuthology.orchestra.run.smithi055.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-03-19T01:48:25.329 INFO:teuthology.orchestra.run.smithi055.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-03-19T01:48:25.357 INFO:teuthology.orchestra.run.smithi055.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-03-19T01:48:25.357 INFO:teuthology.orchestra.run.smithi055.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-03-19T01:48:25.370 INFO:teuthology.orchestra.run.smithi055.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-03-19T01:48:25.392 INFO:teuthology.orchestra.run.smithi055.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-03-19T01:48:25.396 INFO:teuthology.orchestra.run.smithi055.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-03-19T01:48:25.397 INFO:teuthology.orchestra.run.smithi055.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-03-19T01:48:25.397 INFO:teuthology.orchestra.run.smithi055.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-03-19T01:48:25.398 INFO:teuthology.orchestra.run.smithi055.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-03-19T01:48:25.415 INFO:teuthology.orchestra.run.smithi055.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-03-19T01:48:25.417 INFO:teuthology.orchestra.run.smithi055.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-03-19T01:48:25.703 INFO:teuthology.orchestra.run.smithi055.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-03-19T01:48:25.704 INFO:teuthology.orchestra.run.smithi055.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-03-19T01:48:25.704 INFO:teuthology.orchestra.run.smithi055.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-03-19T01:48:25.704 INFO:teuthology.orchestra.run.smithi055.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-03-19T01:48:25.704 INFO:teuthology.orchestra.run.smithi055.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-03-19T01:48:25.704 INFO:teuthology.orchestra.run.smithi055.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-03-19T01:48:25.717 INFO:teuthology.orchestra.run.smithi055.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-03-19T01:48:25.718 INFO:teuthology.orchestra.run.smithi055.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-03-19T01:48:26.146 INFO:teuthology.orchestra.run.smithi055.stdout:Preconfiguring packages ... 2024-03-19T01:48:26.311 INFO:teuthology.orchestra.run.smithi055.stdout:Fetched 54.6 MB in 1s (41.7 MB/s) 2024-03-19T01:48:26.456 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package flex. 2024-03-19T01:48:26.561 INFO:teuthology.orchestra.run.smithi055.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173943 files and directories currently installed.) 2024-03-19T01:48:26.567 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-03-19T01:48:26.612 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking flex (2.6.4-8build2) ... 2024-03-19T01:48:27.197 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-19T01:48:27.369 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-03-19T01:48:27.806 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-19T01:48:28.372 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package bison. 2024-03-19T01:48:28.479 INFO:teuthology.orchestra.run.smithi055.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174016 files and directories currently installed.) 2024-03-19T01:48:28.485 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-03-19T01:48:28.532 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-03-19T01:48:29.000 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-03-19T01:48:29.029 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-03-19T01:48:29.058 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-03-19T01:48:29.487 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package dump. 2024-03-19T01:48:29.516 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-03-19T01:48:29.554 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking dump (0.4b47-3) ... 2024-03-19T01:48:29.866 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package indent. 2024-03-19T01:48:29.894 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:48:29.932 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-19T01:48:30.219 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-03-19T01:48:30.247 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-03-19T01:48:30.285 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-19T01:48:30.614 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-03-19T01:48:30.643 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-03-19T01:48:30.681 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-19T01:48:30.933 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-03-19T01:48:30.962 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-03-19T01:48:31.000 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-03-19T01:48:31.321 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libfl2:amd64. 2024-03-19T01:48:31.349 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-03-19T01:48:31.387 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-03-19T01:48:31.665 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-03-19T01:48:31.693 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-03-19T01:48:31.740 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-19T01:48:32.026 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libjson-perl. 2024-03-19T01:48:32.054 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-03-19T01:48:32.103 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-03-19T01:48:32.414 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libllvm14:amd64. 2024-03-19T01:48:32.443 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-03-19T01:48:32.481 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-19T01:48:34.325 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-03-19T01:48:34.354 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-19T01:48:34.400 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-19T01:48:34.720 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libmbim-proxy. 2024-03-19T01:48:34.748 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-19T01:48:34.794 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-19T01:48:35.131 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-03-19T01:48:35.159 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-19T01:48:35.197 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-19T01:48:35.592 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libndp0:amd64. 2024-03-19T01:48:35.620 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-03-19T01:48:35.666 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-19T01:48:36.069 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libnm0:amd64. 2024-03-19T01:48:36.098 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-03-19T01:48:36.144 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-19T01:48:36.514 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libpq5:amd64. 2024-03-19T01:48:36.542 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:48:36.580 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:48:36.933 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-03-19T01:48:36.962 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:48:37.000 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-19T01:48:37.345 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libqmi-proxy. 2024-03-19T01:48:37.372 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:48:37.419 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-19T01:48:37.713 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-03-19T01:48:37.741 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-19T01:48:37.788 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-19T01:48:38.308 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-03-19T01:48:38.336 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-03-19T01:48:38.374 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-03-19T01:48:38.727 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package modemmanager. 2024-03-19T01:48:38.756 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-19T01:48:38.794 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-19T01:48:39.450 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package network-manager. 2024-03-19T01:48:39.478 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-03-19T01:48:39.525 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-03-19T01:48:40.132 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ppp. 2024-03-19T01:48:40.160 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-03-19T01:48:40.426 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-03-19T01:48:40.728 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package pptp-linux. 2024-03-19T01:48:40.745 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-03-19T01:48:40.786 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-03-19T01:48:41.047 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package network-manager-pptp. 2024-03-19T01:48:41.076 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-03-19T01:48:41.114 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-03-19T01:48:41.417 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package postgresql-client-common. 2024-03-19T01:48:41.445 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-03-19T01:48:41.483 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking postgresql-client-common (238) ... 2024-03-19T01:48:41.770 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package postgresql-client-14. 2024-03-19T01:48:41.799 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:48:41.828 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:48:42.433 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package postgresql-common. 2024-03-19T01:48:42.462 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-03-19T01:48:42.519 INFO:teuthology.orchestra.run.smithi055.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-03-19T01:48:42.548 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking postgresql-common (238) ... 2024-03-19T01:48:43.007 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package postgresql-14. 2024-03-19T01:48:43.036 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-19T01:48:43.077 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:48:44.452 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package postgresql. 2024-03-19T01:48:44.469 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-03-19T01:48:44.501 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking postgresql (14+238) ... 2024-03-19T01:48:44.773 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package postgresql-client. 2024-03-19T01:48:44.802 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-03-19T01:48:44.845 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking postgresql-client (14+238) ... 2024-03-19T01:48:45.089 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package postgresql-contrib. 2024-03-19T01:48:45.118 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-03-19T01:48:45.156 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking postgresql-contrib (14+238) ... 2024-03-19T01:48:45.475 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package tcl8.6. 2024-03-19T01:48:45.504 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-03-19T01:48:45.558 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-19T01:48:45.811 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package tcl. 2024-03-19T01:48:45.828 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-03-19T01:48:45.874 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-03-19T01:48:46.222 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package wpasupplicant. 2024-03-19T01:48:46.251 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-03-19T01:48:46.289 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-19T01:48:46.651 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-03-19T01:48:46.680 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-03-19T01:48:46.718 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-03-19T01:48:47.096 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package usb-modeswitch. 2024-03-19T01:48:47.124 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-03-19T01:48:47.171 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-19T01:48:47.451 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up postgresql-client-common (238) ... 2024-03-19T01:48:47.627 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up flex (2.6.4-8build2) ... 2024-03-19T01:48:47.744 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-19T01:48:47.870 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:48:48.005 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-03-19T01:48:48.408 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-19T01:48:48.698 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:48:49.230 INFO:teuthology.orchestra.run.smithi055.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-03-19T01:48:49.329 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-19T01:48:49.758 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-19T01:48:49.759 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-19T01:48:50.249 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-03-19T01:48:50.375 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-19T01:48:50.484 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-03-19T01:48:50.585 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-03-19T01:48:50.686 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-03-19T01:48:50.812 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-19T01:48:50.938 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-03-19T01:48:51.743 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-19T01:48:51.844 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up dump (0.4b47-3) ... 2024-03-19T01:48:51.938 INFO:teuthology.orchestra.run.smithi055.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-03-19T01:48:52.004 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-19T01:48:52.139 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-19T01:48:52.264 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up tcl (8.6.11+1build2) ... 2024-03-19T01:48:52.390 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libjson-perl (4.04000-1) ... 2024-03-19T01:48:52.516 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-03-19T01:48:52.606 INFO:teuthology.orchestra.run.smithi055.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-03-19T01:48:52.668 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-19T01:48:52.769 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-19T01:48:52.912 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-19T01:48:53.038 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-19T01:48:53.172 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-03-19T01:48:53.374 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up postgresql-client (14+238) ... 2024-03-19T01:48:53.533 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-03-19T01:48:53.986 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:48:53.987 INFO:teuthology.orchestra.run.smithi055.stdout:The following network interfaces were found in /etc/network/interfaces 2024-03-19T01:48:53.987 INFO:teuthology.orchestra.run.smithi055.stdout:which means they are currently configured by ifupdown: 2024-03-19T01:48:53.987 INFO:teuthology.orchestra.run.smithi055.stdout:- ens1f0 2024-03-19T01:48:53.987 INFO:teuthology.orchestra.run.smithi055.stdout:If you want to manage those interfaces with NetworkManager instead 2024-03-19T01:48:53.987 INFO:teuthology.orchestra.run.smithi055.stdout:remove their configuration from /etc/network/interfaces. 2024-03-19T01:48:53.987 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:48:54.108 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-03-19T01:48:54.433 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-03-19T01:48:54.762 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-03-19T01:48:55.244 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-19T01:48:55.387 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-19T01:48:55.513 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-03-19T01:48:55.639 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up postgresql-common (238) ... 2024-03-19T01:48:56.484 INFO:teuthology.orchestra.run.smithi055.stdout:Adding user postgres to group ssl-cert 2024-03-19T01:48:56.724 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:48:56.724 INFO:teuthology.orchestra.run.smithi055.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-03-19T01:48:56.900 INFO:teuthology.orchestra.run.smithi055.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-03-19T01:48:56.900 INFO:teuthology.orchestra.run.smithi055.stdout:Removing obsolete dictionary files: 2024-03-19T01:48:57.433 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-03-19T01:48:57.807 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-19T01:48:58.140 INFO:teuthology.orchestra.run.smithi055.stdout:Creating new PostgreSQL cluster 14/main ... 2024-03-19T01:48:58.160 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-03-19T01:48:58.208 INFO:teuthology.orchestra.run.smithi055.stdout:The files belonging to this database system will be owned by user "postgres". 2024-03-19T01:48:58.208 INFO:teuthology.orchestra.run.smithi055.stdout:This user must also own the server process. 2024-03-19T01:48:58.208 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:48:58.208 INFO:teuthology.orchestra.run.smithi055.stdout:The database cluster will be initialized with locale "en_US". 2024-03-19T01:48:58.208 INFO:teuthology.orchestra.run.smithi055.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-03-19T01:48:58.208 INFO:teuthology.orchestra.run.smithi055.stdout:The default text search configuration will be set to "english". 2024-03-19T01:48:58.208 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:48:58.208 INFO:teuthology.orchestra.run.smithi055.stdout:Data page checksums are disabled. 2024-03-19T01:48:58.209 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:48:58.209 INFO:teuthology.orchestra.run.smithi055.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-03-19T01:48:58.209 INFO:teuthology.orchestra.run.smithi055.stdout:creating subdirectories ... ok 2024-03-19T01:48:58.209 INFO:teuthology.orchestra.run.smithi055.stdout:selecting dynamic shared memory implementation ... posix 2024-03-19T01:48:58.221 INFO:teuthology.orchestra.run.smithi055.stdout:selecting default max_connections ... 100 2024-03-19T01:48:58.242 INFO:teuthology.orchestra.run.smithi055.stdout:selecting default shared_buffers ... 128MB 2024-03-19T01:48:58.253 INFO:teuthology.orchestra.run.smithi055.stdout:selecting default time zone ... Etc/UTC 2024-03-19T01:48:58.253 INFO:teuthology.orchestra.run.smithi055.stdout:creating configuration files ... ok 2024-03-19T01:48:58.372 INFO:teuthology.orchestra.run.smithi055.stdout:running bootstrap script ... ok 2024-03-19T01:48:59.022 INFO:teuthology.orchestra.run.smithi055.stdout:performing post-bootstrap initialization ... ok 2024-03-19T01:48:59.827 INFO:teuthology.orchestra.run.smithi055.stdout:syncing data to disk ... ok 2024-03-19T01:49:00.212 INFO:teuthology.orchestra.run.smithi055.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-03-19T01:49:02.956 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-19T01:49:03.057 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up postgresql-contrib (14+238) ... 2024-03-19T01:49:03.183 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up postgresql (14+238) ... 2024-03-19T01:49:03.309 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-19T01:49:03.435 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-19T01:49:03.695 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-03-19T01:49:03.695 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-03-19T01:49:04.246 INFO:teuthology.orchestra.run.smithi055.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-03-19T01:49:04.565 INFO:teuthology.orchestra.run.smithi055.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-19T01:49:04.733 INFO:teuthology.orchestra.run.smithi055.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-19T01:49:06.054 INFO:teuthology.orchestra.run.smithi055.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-03-19T01:49:06.161 INFO:teuthology.orchestra.run.smithi055.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-19T01:49:06.685 INFO:teuthology.orchestra.run.smithi055.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-19T01:49:06.691 DEBUG:teuthology.parallel:result is None 2024-03-19T01:49:06.692 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T01:49:06.915 DEBUG:teuthology.orchestra.run.smithi055:> dpkg-query -W -f '${Version}' ceph 2024-03-19T01:49:06.935 INFO:teuthology.orchestra.run.smithi055.stdout:19.0.0-2206-gb6ae2491-1jammy 2024-03-19T01:49:06.936 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2206-gb6ae2491-1jammy 2024-03-19T01:49:06.936 INFO:teuthology.task.install:The correct ceph version 19.0.0-2206-gb6ae2491-1jammy is installed. 2024-03-19T01:49:06.938 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T01:49:07.075 DEBUG:teuthology.orchestra.run.smithi084:> dpkg-query -W -f '${Version}' ceph 2024-03-19T01:49:07.095 INFO:teuthology.orchestra.run.smithi084.stdout:19.0.0-2206-gb6ae2491-1jammy 2024-03-19T01:49:07.096 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2206-gb6ae2491-1jammy 2024-03-19T01:49:07.096 INFO:teuthology.task.install:The correct ceph version 19.0.0-2206-gb6ae2491-1jammy is installed. 2024-03-19T01:49:07.098 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-19T01:49:07.098 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:49:07.098 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T01:49:07.113 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:49:07.113 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T01:49:07.152 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-19T01:49:07.153 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:49:07.153 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T01:49:07.167 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T01:49:07.223 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:49:07.223 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T01:49:07.236 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T01:49:07.296 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-19T01:49:07.297 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:49:07.297 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T01:49:07.310 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T01:49:07.366 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:49:07.366 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T01:49:07.380 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T01:49:07.436 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-19T01:49:07.436 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:49:07.436 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T01:49:07.449 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T01:49:07.502 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:49:07.502 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T01:49:07.515 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T01:49:07.570 INFO:teuthology.run_tasks:Running task ceph... 2024-03-19T01:49:07.663 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-19T01:49:07.663 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 777 /var/log/ceph 2024-03-19T01:49:07.666 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 777 /var/log/ceph 2024-03-19T01:49:07.679 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-19T01:49:07.679 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-19T01:49:07.719 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-19T01:49:07.735 INFO:tasks.ceph:Creating extra log directories... 2024-03-19T01:49:07.735 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-19T01:49:07.775 DEBUG:teuthology.orchestra.run.smithi084:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-19T01:49:07.796 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-19T01:49:07.797 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-19T01:49:07.797 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-19T01:49:07.797 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:49:07.797 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-19T01:49:07.835 DEBUG:teuthology.orchestra.run.smithi055:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-19T01:49:07.911 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:49:07.911 DEBUG:teuthology.orchestra.run.smithi084:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-19T01:49:07.917 DEBUG:teuthology.orchestra.run.smithi084:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-19T01:49:07.999 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-19T01:49:07.999 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mds beacon grace': 60, 'mds heartbeat grace': 60, 'osd heartbeat grace': 60}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'PG_AVAILABILITY', 'PG_DEGRADED', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow request', 'SLOW_OPS', 'MON_DOWN'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-19T01:49:07.999 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609330', 'branch': 'wip-batrick-testing-20240318.181317', 'description': 'fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7609330', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-batrick-testing-20240318.181317'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mds beacon grace': 60, 'mds heartbeat grace': 60, 'osd heartbeat grace': 60}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'PG_AVAILABILITY', 'PG_DEGRADED', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow request', 'SLOW_OPS', 'MON_DOWN'], 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'valgrind': {'exit_on_first_error': False, 'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes']}, 'watchdog': {'daemon_restart': 'normal'}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'ceph-fuse': {'client.0': {'valgrind': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes']}}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-batrick-testing-20240318.181317', 'sha1': 'e48b1f6c0608b2615e1890cc46286c3877d6e910'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 3005, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'sleep_before_teardown': 0, 'subset': '433/512', 'suite': 'fs', 'suite_branch': 'wip-batrick-testing-20240318.181317', 'suite_path': '/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/batrick/ceph', 'suite_sha1': 'e48b1f6c0608b2615e1890cc46286c3877d6e910', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi055.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKlRIQTGkVhqf+6pMMXumtETSjuIj1Q6n2/pZ7EW7JggKQFzxhOFNzD60SE8WCRzx7rVm3nQcyVq4i5pk+nPOnQ=', 'smithi084.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBlNX356useTeVfcKcLCf9iAQL/T/oUnh9iHG7sjZ48DuT6vicuCSMWgw8zihf0RmaAtR/cuXslFxmaySc5msVc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'debuginfo': True, 'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mds beacon grace': 60, 'mds heartbeat grace': 60, 'osd heartbeat grace': 60}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'PG_AVAILABILITY', 'PG_DEGRADED', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow request', 'SLOW_OPS', 'MON_DOWN'], 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'valgrind': {'exit_on_first_error': False, 'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes']}, 'watchdog': {'daemon_restart': 'normal'}, 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'workunit': {'clients': {'all': ['suites/dbench.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-18_21:05:49', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-19T01:49:08.000 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-19T01:49:08.002 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-19T01:49:08.040 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -d -m0777 -- /var/run/ceph 2024-03-19T01:49:08.051 DEBUG:teuthology.orchestra.run.smithi084:> sudo install -d -m0777 -- /var/run/ceph 2024-03-19T01:49:08.096 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:49:08.096 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T01:49:08.111 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T01:49:08.111 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_1 2024-03-19T01:49:08.159 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T01:49:08.159 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:49:08.159 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 867 Links: 1 2024-03-19T01:49:08.159 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:49:08.159 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-19 01:37:18.970331071 +0000 2024-03-19T01:49:08.159 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-19 01:37:18.730333215 +0000 2024-03-19T01:49:08.159 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-19 01:37:18.730333215 +0000 2024-03-19T01:49:08.159 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: - 2024-03-19T01:49:08.159 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T01:49:08.213 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-19T01:49:08.213 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-19T01:49:08.214 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000562232 s, 911 kB/s 2024-03-19T01:49:08.215 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T01:49:08.264 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_2 2024-03-19T01:49:08.311 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T01:49:08.311 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:49:08.311 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 878 Links: 1 2024-03-19T01:49:08.311 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:49:08.311 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-19 01:37:19.482326499 +0000 2024-03-19T01:49:08.312 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-19 01:37:19.214328892 +0000 2024-03-19T01:49:08.312 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-19 01:37:19.214328892 +0000 2024-03-19T01:49:08.312 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: - 2024-03-19T01:49:08.312 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T01:49:08.366 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-19T01:49:08.366 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-19T01:49:08.366 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000492973 s, 1.0 MB/s 2024-03-19T01:49:08.368 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T01:49:08.416 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_3 2024-03-19T01:49:08.462 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T01:49:08.462 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:49:08.463 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 895 Links: 1 2024-03-19T01:49:08.463 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:49:08.463 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-19 01:37:19.966322175 +0000 2024-03-19T01:49:08.463 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-19 01:37:19.722324355 +0000 2024-03-19T01:49:08.463 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-19 01:37:19.722324355 +0000 2024-03-19T01:49:08.463 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: - 2024-03-19T01:49:08.463 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T01:49:08.518 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-19T01:49:08.518 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-19T01:49:08.518 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000397844 s, 1.3 MB/s 2024-03-19T01:49:08.519 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T01:49:08.569 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_4 2024-03-19T01:49:08.618 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T01:49:08.618 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:49:08.618 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 919 Links: 1 2024-03-19T01:49:08.618 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:49:08.618 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-19 01:37:20.482317566 +0000 2024-03-19T01:49:08.618 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-19 01:37:20.226319853 +0000 2024-03-19T01:49:08.618 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-19 01:37:20.226319853 +0000 2024-03-19T01:49:08.619 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: - 2024-03-19T01:49:08.619 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T01:49:08.673 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-19T01:49:08.673 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-19T01:49:08.673 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000534485 s, 958 kB/s 2024-03-19T01:49:08.674 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T01:49:08.724 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-19T01:49:08.724 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:49:08.724 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T01:49:08.730 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T01:49:08.730 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_1 2024-03-19T01:49:08.776 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T01:49:08.776 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:49:08.776 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 886 Links: 1 2024-03-19T01:49:08.776 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:49:08.776 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-03-19 01:37:16.549198109 +0000 2024-03-19T01:49:08.777 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-03-19 01:37:16.265200642 +0000 2024-03-19T01:49:08.777 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-03-19 01:37:16.265200642 +0000 2024-03-19T01:49:08.777 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-03-19T01:49:08.777 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T01:49:08.830 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-03-19T01:49:08.831 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-03-19T01:49:08.831 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.0004602 s, 1.1 MB/s 2024-03-19T01:49:08.832 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T01:49:08.881 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_2 2024-03-19T01:49:08.930 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T01:49:08.930 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:49:08.930 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 904 Links: 1 2024-03-19T01:49:08.930 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:49:08.930 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-03-19 01:37:17.053193612 +0000 2024-03-19T01:49:08.930 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-03-19 01:37:16.809195789 +0000 2024-03-19T01:49:08.931 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-03-19 01:37:16.809195789 +0000 2024-03-19T01:49:08.931 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-03-19T01:49:08.931 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T01:49:08.982 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-03-19T01:49:08.982 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-03-19T01:49:08.982 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000373259 s, 1.4 MB/s 2024-03-19T01:49:08.983 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T01:49:09.033 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_3 2024-03-19T01:49:09.080 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T01:49:09.080 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:49:09.080 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-03-19T01:49:09.080 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:49:09.080 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-03-19 01:37:17.501189616 +0000 2024-03-19T01:49:09.080 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-03-19 01:37:17.281191579 +0000 2024-03-19T01:49:09.080 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-03-19 01:37:17.281191579 +0000 2024-03-19T01:49:09.080 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-03-19T01:49:09.081 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T01:49:09.134 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-03-19T01:49:09.134 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-03-19T01:49:09.134 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000425712 s, 1.2 MB/s 2024-03-19T01:49:09.135 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T01:49:09.185 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_4 2024-03-19T01:49:09.232 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T01:49:09.232 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:49:09.232 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 937 Links: 1 2024-03-19T01:49:09.232 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:49:09.232 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-03-19 01:37:17.997185191 +0000 2024-03-19T01:49:09.232 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-03-19 01:37:17.737187511 +0000 2024-03-19T01:49:09.232 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-03-19 01:37:17.737187511 +0000 2024-03-19T01:49:09.232 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-03-19T01:49:09.233 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T01:49:09.286 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-03-19T01:49:09.286 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-03-19T01:49:09.286 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.00040904 s, 1.3 MB/s 2024-03-19T01:49:09.287 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T01:49:09.338 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-19T01:49:09.338 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi055.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi084.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-19T01:49:09.338 INFO:tasks.ceph:Generating config... 2024-03-19T01:49:09.340 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-19T01:49:09.340 INFO:tasks.ceph:[client] debug client = 20 2024-03-19T01:49:09.340 INFO:tasks.ceph:[client] debug ms = 1 2024-03-19T01:49:09.340 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-19T01:49:09.340 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-19T01:49:09.340 INFO:tasks.ceph:[global] mds beacon grace = 60 2024-03-19T01:49:09.340 INFO:tasks.ceph:[global] mds heartbeat grace = 60 2024-03-19T01:49:09.340 INFO:tasks.ceph:[global] osd heartbeat grace = 60 2024-03-19T01:49:09.340 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-19T01:49:09.340 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-19T01:49:09.340 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-19T01:49:09.340 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-19T01:49:09.341 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-19T01:49:09.341 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-19T01:49:09.341 INFO:tasks.ceph:[mds] mds valgrind exit = True 2024-03-19T01:49:09.341 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-19T01:49:09.341 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-19T01:49:09.341 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-19T01:49:09.341 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-19T01:49:09.341 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-19T01:49:09.341 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-19T01:49:09.341 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-19T01:49:09.341 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-19T01:49:09.341 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-19T01:49:09.341 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-19T01:49:09.342 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-19T01:49:09.342 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-19T01:49:09.342 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-19T01:49:09.342 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-19T01:49:09.342 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-19T01:49:09.342 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-19T01:49:09.342 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-19T01:49:09.342 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-19T01:49:09.342 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-19T01:49:09.342 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-19T01:49:09.342 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-19T01:49:09.342 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-19T01:49:09.343 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-19T01:49:09.343 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-19T01:49:09.343 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-19T01:49:09.343 INFO:tasks.ceph:Setting up mon.a... 2024-03-19T01:49:09.343 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-19T01:49:09.379 INFO:teuthology.orchestra.run.smithi055.stdout:creating /etc/ceph/ceph.keyring 2024-03-19T01:49:09.390 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-19T01:49:09.471 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-19T01:49:09.526 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.55'), ('mon.b', '172.21.15.84'), ('mon.c', '[v2:172.21.15.84:3301,v1:172.21.15.84:6790]')] 2024-03-19T01:49:09.526 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.55,172.21.15.84,[v2:172.21.15.84:3301,v1:172.21.15.84:6790]', 'mds beacon grace': 60, 'mds heartbeat grace': 60, 'osd heartbeat grace': 60}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-19T01:49:09.527 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:49:09.527 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-19T01:49:09.575 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.55 --add b 172.21.15.84 --addv c '[v2:172.21.15.84:3301,v1:172.21.15.84:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:49:09.663 INFO:teuthology.orchestra.run.smithi055.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:49:09.663 INFO:teuthology.orchestra.run.smithi055.stdout:monmaptool: generated fsid def9d4e5-0bc7-449d-b893-2490cbd10e25 2024-03-19T01:49:09.663 INFO:teuthology.orchestra.run.smithi055.stdout:setting min_mon_release = quincy 2024-03-19T01:49:09.663 INFO:teuthology.orchestra.run.smithi055.stdout:epoch 0 2024-03-19T01:49:09.663 INFO:teuthology.orchestra.run.smithi055.stdout:fsid def9d4e5-0bc7-449d-b893-2490cbd10e25 2024-03-19T01:49:09.663 INFO:teuthology.orchestra.run.smithi055.stdout:last_changed 2024-03-19T01:49:09.665054+0000 2024-03-19T01:49:09.663 INFO:teuthology.orchestra.run.smithi055.stdout:created 2024-03-19T01:49:09.665054+0000 2024-03-19T01:49:09.663 INFO:teuthology.orchestra.run.smithi055.stdout:min_mon_release 17 (quincy) 2024-03-19T01:49:09.664 INFO:teuthology.orchestra.run.smithi055.stdout:election_strategy: 1 2024-03-19T01:49:09.664 INFO:teuthology.orchestra.run.smithi055.stdout:0: [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon.a 2024-03-19T01:49:09.664 INFO:teuthology.orchestra.run.smithi055.stdout:1: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.b 2024-03-19T01:49:09.664 INFO:teuthology.orchestra.run.smithi055.stdout:2: [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] mon.c 2024-03-19T01:49:09.664 INFO:teuthology.orchestra.run.smithi055.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-19T01:49:09.666 DEBUG:teuthology.orchestra.run.smithi055:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-19T01:49:09.671 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID def9d4e5-0bc7-449d-b893-2490cbd10e25... 2024-03-19T01:49:09.672 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-19T01:49:09.715 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-19T01:49:09.743 INFO:teuthology.orchestra.run.smithi055.stdout:[global] 2024-03-19T01:49:09.744 INFO:teuthology.orchestra.run.smithi055.stdout: chdir = "" 2024-03-19T01:49:09.744 INFO:teuthology.orchestra.run.smithi055.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-19T01:49:09.744 INFO:teuthology.orchestra.run.smithi055.stdout: auth supported = cephx 2024-03-19T01:49:09.744 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.744 INFO:teuthology.orchestra.run.smithi055.stdout: filestore xattr use omap = true 2024-03-19T01:49:09.744 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.744 INFO:teuthology.orchestra.run.smithi055.stdout: mon clock drift allowed = 1.000 2024-03-19T01:49:09.744 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.744 INFO:teuthology.orchestra.run.smithi055.stdout: osd crush chooseleaf type = 0 2024-03-19T01:49:09.744 INFO:teuthology.orchestra.run.smithi055.stdout: auth debug = true 2024-03-19T01:49:09.744 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.744 INFO:teuthology.orchestra.run.smithi055.stdout: ms die on old message = true 2024-03-19T01:49:09.745 INFO:teuthology.orchestra.run.smithi055.stdout: ms die on bug = true 2024-03-19T01:49:09.745 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.745 INFO:teuthology.orchestra.run.smithi055.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-19T01:49:09.745 INFO:teuthology.orchestra.run.smithi055.stdout: mon pg warn max object skew = 0 2024-03-19T01:49:09.745 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.745 INFO:teuthology.orchestra.run.smithi055.stdout: # disable pg_autoscaler by default for new pools 2024-03-19T01:49:09.745 INFO:teuthology.orchestra.run.smithi055.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-19T01:49:09.745 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.745 INFO:teuthology.orchestra.run.smithi055.stdout: osd pool default size = 2 2024-03-19T01:49:09.745 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.745 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd allow primary affinity = true 2024-03-19T01:49:09.745 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd allow pg remap = true 2024-03-19T01:49:09.745 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on legacy crush tunables = false 2024-03-19T01:49:09.745 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on crush straw calc version zero = false 2024-03-19T01:49:09.746 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on no sortbitwise = false 2024-03-19T01:49:09.746 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on osd down out interval zero = false 2024-03-19T01:49:09.746 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on too few osds = false 2024-03-19T01:49:09.746 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-19T01:49:09.746 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-19T01:49:09.746 INFO:teuthology.orchestra.run.smithi055.stdout: mon_allow_pool_size_one = true 2024-03-19T01:49:09.746 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.746 INFO:teuthology.orchestra.run.smithi055.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-19T01:49:09.746 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.746 INFO:teuthology.orchestra.run.smithi055.stdout: osd default data pool replay window = 5 2024-03-19T01:49:09.746 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.746 INFO:teuthology.orchestra.run.smithi055.stdout: mon allow pool delete = true 2024-03-19T01:49:09.746 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.746 INFO:teuthology.orchestra.run.smithi055.stdout: mon cluster log file level = debug 2024-03-19T01:49:09.747 INFO:teuthology.orchestra.run.smithi055.stdout: debug asserts on shutdown = true 2024-03-19T01:49:09.747 INFO:teuthology.orchestra.run.smithi055.stdout: mon health detail to clog = false 2024-03-19T01:49:09.747 INFO:teuthology.orchestra.run.smithi055.stdout: mon host = "172.21.15.55,172.21.15.84,[v2:172.21.15.84:3301,v1:172.21.15.84:6790]" 2024-03-19T01:49:09.747 INFO:teuthology.orchestra.run.smithi055.stdout: mds beacon grace = 60 2024-03-19T01:49:09.747 INFO:teuthology.orchestra.run.smithi055.stdout: mds heartbeat grace = 60 2024-03-19T01:49:09.747 INFO:teuthology.orchestra.run.smithi055.stdout: osd heartbeat grace = 60 2024-03-19T01:49:09.747 INFO:teuthology.orchestra.run.smithi055.stdout: fsid = def9d4e5-0bc7-449d-b893-2490cbd10e25 2024-03-19T01:49:09.747 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.747 INFO:teuthology.orchestra.run.smithi055.stdout:[osd] 2024-03-19T01:49:09.747 INFO:teuthology.orchestra.run.smithi055.stdout: osd journal size = 100 2024-03-19T01:49:09.747 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.747 INFO:teuthology.orchestra.run.smithi055.stdout: osd scrub load threshold = 5.0 2024-03-19T01:49:09.747 INFO:teuthology.orchestra.run.smithi055.stdout: osd scrub max interval = 600 2024-03-19T01:49:09.747 INFO:teuthology.orchestra.run.smithi055.stdout: osd mclock profile = high_recovery_ops 2024-03-19T01:49:09.749 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.749 INFO:teuthology.orchestra.run.smithi055.stdout: osd recover clone overlap = true 2024-03-19T01:49:09.749 INFO:teuthology.orchestra.run.smithi055.stdout: osd recovery max chunk = 1048576 2024-03-19T01:49:09.750 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.750 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug shutdown = true 2024-03-19T01:49:09.750 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug op order = true 2024-03-19T01:49:09.750 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug verify stray on activate = true 2024-03-19T01:49:09.750 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug trim objects = true 2024-03-19T01:49:09.750 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.750 INFO:teuthology.orchestra.run.smithi055.stdout: osd open classes on start = true 2024-03-19T01:49:09.750 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug pg log writeout = true 2024-03-19T01:49:09.750 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.750 INFO:teuthology.orchestra.run.smithi055.stdout: osd deep scrub update digest min age = 30 2024-03-19T01:49:09.750 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.750 INFO:teuthology.orchestra.run.smithi055.stdout: osd map max advance = 10 2024-03-19T01:49:09.750 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.750 INFO:teuthology.orchestra.run.smithi055.stdout: journal zero on create = true 2024-03-19T01:49:09.751 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.751 INFO:teuthology.orchestra.run.smithi055.stdout: filestore ondisk finisher threads = 3 2024-03-19T01:49:09.751 INFO:teuthology.orchestra.run.smithi055.stdout: filestore apply finisher threads = 3 2024-03-19T01:49:09.751 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.751 INFO:teuthology.orchestra.run.smithi055.stdout: bdev debug aio = true 2024-03-19T01:49:09.751 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug misdirected ops = true 2024-03-19T01:49:09.751 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore block size = 96636764160 2024-03-19T01:49:09.751 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore fsck on mount = True 2024-03-19T01:49:09.751 INFO:teuthology.orchestra.run.smithi055.stdout: debug bluefs = 20 2024-03-19T01:49:09.751 INFO:teuthology.orchestra.run.smithi055.stdout: debug bluestore = 20 2024-03-19T01:49:09.751 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2024-03-19T01:49:09.751 INFO:teuthology.orchestra.run.smithi055.stdout: debug osd = 20 2024-03-19T01:49:09.751 INFO:teuthology.orchestra.run.smithi055.stdout: debug rocksdb = 10 2024-03-19T01:49:09.751 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd backfillfull_ratio = 0.85 2024-03-19T01:49:09.751 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd full ratio = 0.9 2024-03-19T01:49:09.752 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd nearfull ratio = 0.8 2024-03-19T01:49:09.752 INFO:teuthology.orchestra.run.smithi055.stdout: osd failsafe full ratio = 0.95 2024-03-19T01:49:09.752 INFO:teuthology.orchestra.run.smithi055.stdout: osd fast shutdown = False 2024-03-19T01:49:09.752 INFO:teuthology.orchestra.run.smithi055.stdout: osd objectstore = bluestore 2024-03-19T01:49:09.752 INFO:teuthology.orchestra.run.smithi055.stdout: osd op complaint time = 180 2024-03-19T01:49:09.752 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:[mgr] 2024-03-19T01:49:09.752 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2024-03-19T01:49:09.752 INFO:teuthology.orchestra.run.smithi055.stdout: debug mgr = 20 2024-03-19T01:49:09.752 INFO:teuthology.orchestra.run.smithi055.stdout: debug mon = 20 2024-03-19T01:49:09.753 INFO:teuthology.orchestra.run.smithi055.stdout: debug auth = 20 2024-03-19T01:49:09.753 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min pgs per osd = 4 2024-03-19T01:49:09.753 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min bytes per osd = 10 2024-03-19T01:49:09.753 INFO:teuthology.orchestra.run.smithi055.stdout: mgr/telemetry/nag = false 2024-03-19T01:49:09.753 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.753 INFO:teuthology.orchestra.run.smithi055.stdout:[mon] 2024-03-19T01:49:09.753 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2024-03-19T01:49:09.753 INFO:teuthology.orchestra.run.smithi055.stdout: debug mon = 20 2024-03-19T01:49:09.753 INFO:teuthology.orchestra.run.smithi055.stdout: debug paxos = 20 2024-03-19T01:49:09.753 INFO:teuthology.orchestra.run.smithi055.stdout: debug auth = 20 2024-03-19T01:49:09.753 INFO:teuthology.orchestra.run.smithi055.stdout: mon data avail warn = 5 2024-03-19T01:49:09.753 INFO:teuthology.orchestra.run.smithi055.stdout: mon mgr mkfs grace = 240 2024-03-19T01:49:09.753 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min pgs per osd = 4 2024-03-19T01:49:09.753 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd reporter subtree level = osd 2024-03-19T01:49:09.754 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd prime pg temp = true 2024-03-19T01:49:09.754 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min bytes per osd = 10 2024-03-19T01:49:09.754 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.754 INFO:teuthology.orchestra.run.smithi055.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-19T01:49:09.754 INFO:teuthology.orchestra.run.smithi055.stdout: auth mon ticket ttl = 660 # 11m 2024-03-19T01:49:09.754 INFO:teuthology.orchestra.run.smithi055.stdout: auth service ticket ttl = 240 # 4m 2024-03-19T01:49:09.754 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.754 INFO:teuthology.orchestra.run.smithi055.stdout: # don't complain about insecure global_id in the test suite 2024-03-19T01:49:09.754 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-19T01:49:09.754 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-19T01:49:09.754 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.754 INFO:teuthology.orchestra.run.smithi055.stdout: # 1m isn't quite enough 2024-03-19T01:49:09.754 INFO:teuthology.orchestra.run.smithi055.stdout: mon_down_mkfs_grace = 2m 2024-03-19T01:49:09.754 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.755 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_filestore_osds = false 2024-03-19T01:49:09.755 INFO:teuthology.orchestra.run.smithi055.stdout: mon op complaint time = 120 2024-03-19T01:49:09.755 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd crush smoke test = False 2024-03-19T01:49:09.755 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:49:09.755 INFO:teuthology.orchestra.run.smithi055.stdout:[client] 2024-03-19T01:49:09.755 INFO:teuthology.orchestra.run.smithi055.stdout: rgw cache enabled = true 2024-03-19T01:49:09.755 INFO:teuthology.orchestra.run.smithi055.stdout: rgw enable ops log = true 2024-03-19T01:49:09.755 INFO:teuthology.orchestra.run.smithi055.stdout: rgw enable usage log = true 2024-03-19T01:49:09.755 INFO:teuthology.orchestra.run.smithi055.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-19T01:49:09.755 INFO:teuthology.orchestra.run.smithi055.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-19T01:49:09.755 INFO:teuthology.orchestra.run.smithi055.stdout: client mount timeout = 600 2024-03-19T01:49:09.755 INFO:teuthology.orchestra.run.smithi055.stdout: debug client = 20 2024-03-19T01:49:09.755 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2024-03-19T01:49:09.755 INFO:teuthology.orchestra.run.smithi055.stdout: rados mon op timeout = 900 2024-03-19T01:49:09.755 INFO:teuthology.orchestra.run.smithi055.stdout: rados osd op timeout = 900 2024-03-19T01:49:09.756 INFO:teuthology.orchestra.run.smithi055.stdout:[mon.a] 2024-03-19T01:49:09.756 INFO:teuthology.orchestra.run.smithi055.stdout:[mon.b] 2024-03-19T01:49:09.756 INFO:teuthology.orchestra.run.smithi055.stdout:[mon.c] 2024-03-19T01:49:09.756 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.a] 2024-03-19T01:49:09.756 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.c] 2024-03-19T01:49:09.756 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.e] 2024-03-19T01:49:09.756 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.b] 2024-03-19T01:49:09.756 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.d] 2024-03-19T01:49:09.756 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.f] 2024-03-19T01:49:09.756 INFO:teuthology.orchestra.run.smithi055.stdout:[mds] 2024-03-19T01:49:09.756 INFO:teuthology.orchestra.run.smithi055.stdout: debug mds = 20 2024-03-19T01:49:09.756 INFO:teuthology.orchestra.run.smithi055.stdout: debug mds balancer = 20 2024-03-19T01:49:09.756 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2024-03-19T01:49:09.756 INFO:teuthology.orchestra.run.smithi055.stdout: mds debug frag = True 2024-03-19T01:49:09.757 INFO:teuthology.orchestra.run.smithi055.stdout: mds debug scatterstat = True 2024-03-19T01:49:09.757 INFO:teuthology.orchestra.run.smithi055.stdout: mds op complaint time = 180 2024-03-19T01:49:09.757 INFO:teuthology.orchestra.run.smithi055.stdout: mds valgrind exit = True 2024-03-19T01:49:09.757 INFO:teuthology.orchestra.run.smithi055.stdout: mds verify scatter = True 2024-03-19T01:49:09.757 INFO:teuthology.orchestra.run.smithi055.stdout: osd op complaint time = 180 2024-03-19T01:49:09.757 INFO:teuthology.orchestra.run.smithi055.stdout: rados mon op timeout = 900 2024-03-19T01:49:09.757 INFO:teuthology.orchestra.run.smithi055.stdout: rados osd op timeout = 900 2024-03-19T01:49:09.759 INFO:teuthology.orchestra.run.smithi084.stdout:[global] 2024-03-19T01:49:09.759 INFO:teuthology.orchestra.run.smithi084.stdout: chdir = "" 2024-03-19T01:49:09.759 INFO:teuthology.orchestra.run.smithi084.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-19T01:49:09.759 INFO:teuthology.orchestra.run.smithi084.stdout: auth supported = cephx 2024-03-19T01:49:09.759 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.759 INFO:teuthology.orchestra.run.smithi084.stdout: filestore xattr use omap = true 2024-03-19T01:49:09.759 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.759 INFO:teuthology.orchestra.run.smithi084.stdout: mon clock drift allowed = 1.000 2024-03-19T01:49:09.759 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.759 INFO:teuthology.orchestra.run.smithi084.stdout: osd crush chooseleaf type = 0 2024-03-19T01:49:09.759 INFO:teuthology.orchestra.run.smithi084.stdout: auth debug = true 2024-03-19T01:49:09.759 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.760 INFO:teuthology.orchestra.run.smithi084.stdout: ms die on old message = true 2024-03-19T01:49:09.760 INFO:teuthology.orchestra.run.smithi084.stdout: ms die on bug = true 2024-03-19T01:49:09.760 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.760 INFO:teuthology.orchestra.run.smithi084.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-19T01:49:09.760 INFO:teuthology.orchestra.run.smithi084.stdout: mon pg warn max object skew = 0 2024-03-19T01:49:09.760 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.760 INFO:teuthology.orchestra.run.smithi084.stdout: # disable pg_autoscaler by default for new pools 2024-03-19T01:49:09.760 INFO:teuthology.orchestra.run.smithi084.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-19T01:49:09.760 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.760 INFO:teuthology.orchestra.run.smithi084.stdout: osd pool default size = 2 2024-03-19T01:49:09.760 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.760 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd allow primary affinity = true 2024-03-19T01:49:09.760 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd allow pg remap = true 2024-03-19T01:49:09.760 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on legacy crush tunables = false 2024-03-19T01:49:09.761 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on crush straw calc version zero = false 2024-03-19T01:49:09.761 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on no sortbitwise = false 2024-03-19T01:49:09.761 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on osd down out interval zero = false 2024-03-19T01:49:09.761 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on too few osds = false 2024-03-19T01:49:09.761 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-19T01:49:09.761 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-19T01:49:09.761 INFO:teuthology.orchestra.run.smithi084.stdout: mon_allow_pool_size_one = true 2024-03-19T01:49:09.761 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.761 INFO:teuthology.orchestra.run.smithi084.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-19T01:49:09.761 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.761 INFO:teuthology.orchestra.run.smithi084.stdout: osd default data pool replay window = 5 2024-03-19T01:49:09.761 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.761 INFO:teuthology.orchestra.run.smithi084.stdout: mon allow pool delete = true 2024-03-19T01:49:09.761 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: mon cluster log file level = debug 2024-03-19T01:49:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: debug asserts on shutdown = true 2024-03-19T01:49:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: mon health detail to clog = false 2024-03-19T01:49:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: mon host = "172.21.15.55,172.21.15.84,[v2:172.21.15.84:3301,v1:172.21.15.84:6790]" 2024-03-19T01:49:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: mds beacon grace = 60 2024-03-19T01:49:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: mds heartbeat grace = 60 2024-03-19T01:49:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: osd heartbeat grace = 60 2024-03-19T01:49:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: fsid = def9d4e5-0bc7-449d-b893-2490cbd10e25 2024-03-19T01:49:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.762 INFO:teuthology.orchestra.run.smithi084.stdout:[osd] 2024-03-19T01:49:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: osd journal size = 100 2024-03-19T01:49:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: osd scrub load threshold = 5.0 2024-03-19T01:49:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: osd scrub max interval = 600 2024-03-19T01:49:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: osd mclock profile = high_recovery_ops 2024-03-19T01:49:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: osd recover clone overlap = true 2024-03-19T01:49:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: osd recovery max chunk = 1048576 2024-03-19T01:49:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug shutdown = true 2024-03-19T01:49:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug op order = true 2024-03-19T01:49:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug verify stray on activate = true 2024-03-19T01:49:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug trim objects = true 2024-03-19T01:49:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: osd open classes on start = true 2024-03-19T01:49:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug pg log writeout = true 2024-03-19T01:49:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.763 INFO:teuthology.orchestra.run.smithi084.stdout: osd deep scrub update digest min age = 30 2024-03-19T01:49:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: osd map max advance = 10 2024-03-19T01:49:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: journal zero on create = true 2024-03-19T01:49:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: filestore ondisk finisher threads = 3 2024-03-19T01:49:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: filestore apply finisher threads = 3 2024-03-19T01:49:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: bdev debug aio = true 2024-03-19T01:49:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug misdirected ops = true 2024-03-19T01:49:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: bluestore block size = 96636764160 2024-03-19T01:49:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: bluestore fsck on mount = True 2024-03-19T01:49:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: debug bluefs = 20 2024-03-19T01:49:09.764 INFO:teuthology.orchestra.run.smithi084.stdout: debug bluestore = 20 2024-03-19T01:49:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2024-03-19T01:49:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: debug osd = 20 2024-03-19T01:49:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: debug rocksdb = 10 2024-03-19T01:49:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd backfillfull_ratio = 0.85 2024-03-19T01:49:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd full ratio = 0.9 2024-03-19T01:49:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd nearfull ratio = 0.8 2024-03-19T01:49:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: osd failsafe full ratio = 0.95 2024-03-19T01:49:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: osd fast shutdown = False 2024-03-19T01:49:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: osd objectstore = bluestore 2024-03-19T01:49:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: osd op complaint time = 180 2024-03-19T01:49:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.765 INFO:teuthology.orchestra.run.smithi084.stdout:[mgr] 2024-03-19T01:49:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2024-03-19T01:49:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: debug mgr = 20 2024-03-19T01:49:09.765 INFO:teuthology.orchestra.run.smithi084.stdout: debug mon = 20 2024-03-19T01:49:09.766 INFO:teuthology.orchestra.run.smithi084.stdout: debug auth = 20 2024-03-19T01:49:09.766 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min pgs per osd = 4 2024-03-19T01:49:09.766 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min bytes per osd = 10 2024-03-19T01:49:09.766 INFO:teuthology.orchestra.run.smithi084.stdout: mgr/telemetry/nag = false 2024-03-19T01:49:09.766 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.766 INFO:teuthology.orchestra.run.smithi084.stdout:[mon] 2024-03-19T01:49:09.766 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2024-03-19T01:49:09.766 INFO:teuthology.orchestra.run.smithi084.stdout: debug mon = 20 2024-03-19T01:49:09.766 INFO:teuthology.orchestra.run.smithi084.stdout: debug paxos = 20 2024-03-19T01:49:09.766 INFO:teuthology.orchestra.run.smithi084.stdout: debug auth = 20 2024-03-19T01:49:09.766 INFO:teuthology.orchestra.run.smithi084.stdout: mon data avail warn = 5 2024-03-19T01:49:09.766 INFO:teuthology.orchestra.run.smithi084.stdout: mon mgr mkfs grace = 240 2024-03-19T01:49:09.766 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min pgs per osd = 4 2024-03-19T01:49:09.766 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd reporter subtree level = osd 2024-03-19T01:49:09.767 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd prime pg temp = true 2024-03-19T01:49:09.767 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min bytes per osd = 10 2024-03-19T01:49:09.767 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.767 INFO:teuthology.orchestra.run.smithi084.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-19T01:49:09.767 INFO:teuthology.orchestra.run.smithi084.stdout: auth mon ticket ttl = 660 # 11m 2024-03-19T01:49:09.767 INFO:teuthology.orchestra.run.smithi084.stdout: auth service ticket ttl = 240 # 4m 2024-03-19T01:49:09.767 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.767 INFO:teuthology.orchestra.run.smithi084.stdout: # don't complain about insecure global_id in the test suite 2024-03-19T01:49:09.767 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-19T01:49:09.767 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-19T01:49:09.767 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.767 INFO:teuthology.orchestra.run.smithi084.stdout: # 1m isn't quite enough 2024-03-19T01:49:09.767 INFO:teuthology.orchestra.run.smithi084.stdout: mon_down_mkfs_grace = 2m 2024-03-19T01:49:09.767 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.768 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_filestore_osds = false 2024-03-19T01:49:09.768 INFO:teuthology.orchestra.run.smithi084.stdout: mon op complaint time = 120 2024-03-19T01:49:09.768 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd crush smoke test = False 2024-03-19T01:49:09.768 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-19T01:49:09.768 INFO:teuthology.orchestra.run.smithi084.stdout:[client] 2024-03-19T01:49:09.768 INFO:teuthology.orchestra.run.smithi084.stdout: rgw cache enabled = true 2024-03-19T01:49:09.768 INFO:teuthology.orchestra.run.smithi084.stdout: rgw enable ops log = true 2024-03-19T01:49:09.768 INFO:teuthology.orchestra.run.smithi084.stdout: rgw enable usage log = true 2024-03-19T01:49:09.768 INFO:teuthology.orchestra.run.smithi084.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-19T01:49:09.768 INFO:teuthology.orchestra.run.smithi084.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-19T01:49:09.768 INFO:teuthology.orchestra.run.smithi084.stdout: client mount timeout = 600 2024-03-19T01:49:09.768 INFO:teuthology.orchestra.run.smithi084.stdout: debug client = 20 2024-03-19T01:49:09.768 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2024-03-19T01:49:09.768 INFO:teuthology.orchestra.run.smithi084.stdout: rados mon op timeout = 900 2024-03-19T01:49:09.769 INFO:teuthology.orchestra.run.smithi084.stdout: rados osd op timeout = 900 2024-03-19T01:49:09.769 INFO:teuthology.orchestra.run.smithi084.stdout:[mon.a] 2024-03-19T01:49:09.769 INFO:teuthology.orchestra.run.smithi084.stdout:[mon.b] 2024-03-19T01:49:09.769 INFO:teuthology.orchestra.run.smithi084.stdout:[mon.c] 2024-03-19T01:49:09.769 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.a] 2024-03-19T01:49:09.769 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.c] 2024-03-19T01:49:09.769 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.e] 2024-03-19T01:49:09.769 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.b] 2024-03-19T01:49:09.769 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.d] 2024-03-19T01:49:09.769 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.f] 2024-03-19T01:49:09.769 INFO:teuthology.orchestra.run.smithi084.stdout:[mds] 2024-03-19T01:49:09.769 INFO:teuthology.orchestra.run.smithi084.stdout: debug mds = 20 2024-03-19T01:49:09.769 INFO:teuthology.orchestra.run.smithi084.stdout: debug mds balancer = 20 2024-03-19T01:49:09.769 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2024-03-19T01:49:09.769 INFO:teuthology.orchestra.run.smithi084.stdout: mds debug frag = True 2024-03-19T01:49:09.770 INFO:teuthology.orchestra.run.smithi084.stdout: mds debug scatterstat = True 2024-03-19T01:49:09.770 INFO:teuthology.orchestra.run.smithi084.stdout: mds op complaint time = 180 2024-03-19T01:49:09.770 INFO:teuthology.orchestra.run.smithi084.stdout: mds valgrind exit = True 2024-03-19T01:49:09.770 INFO:teuthology.orchestra.run.smithi084.stdout: mds verify scatter = True 2024-03-19T01:49:09.770 INFO:teuthology.orchestra.run.smithi084.stdout: osd op complaint time = 180 2024-03-19T01:49:09.770 INFO:teuthology.orchestra.run.smithi084.stdout: rados mon op timeout = 900 2024-03-19T01:49:09.770 INFO:teuthology.orchestra.run.smithi084.stdout: rados osd op timeout = 900 2024-03-19T01:49:09.770 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-19T01:49:09.770 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-19T01:49:09.848 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-19T01:49:09.848 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:49:09.848 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-19T01:49:09.854 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:49:09.854 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-19T01:49:09.902 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi055.front.sepia.ceph.com 2024-03-19T01:49:09.902 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:49:09.902 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-19T01:49:09.902 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-19T01:49:09.970 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:49:09.970 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:49:10.019 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi084.front.sepia.ceph.com 2024-03-19T01:49:10.019 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:49:10.019 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-19T01:49:10.019 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-19T01:49:10.043 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:49:10.043 DEBUG:teuthology.orchestra.run.smithi084:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:49:10.092 INFO:tasks.ceph:Setting up mon nodes... 2024-03-19T01:49:10.092 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-19T01:49:10.093 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-19T01:49:10.137 INFO:teuthology.orchestra.run.smithi055.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-19T01:49:10.140 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-19T01:49:10.184 INFO:teuthology.orchestra.run.smithi084.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-19T01:49:10.275 INFO:tasks.ceph:Setting up mds nodes... 2024-03-19T01:49:10.275 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-19T01:49:10.320 INFO:teuthology.orchestra.run.smithi055.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-19T01:49:10.323 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-19T01:49:10.379 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-19T01:49:10.464 INFO:teuthology.orchestra.run.smithi055.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-19T01:49:10.467 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-19T01:49:10.523 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-03-19T01:49:10.609 INFO:teuthology.orchestra.run.smithi055.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-03-19T01:49:10.611 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-03-19T01:49:10.666 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-19T01:49:10.712 INFO:teuthology.orchestra.run.smithi084.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-19T01:49:10.715 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-19T01:49:10.727 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-19T01:49:10.812 INFO:teuthology.orchestra.run.smithi084.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-19T01:49:10.814 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-19T01:49:10.871 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-03-19T01:49:10.952 INFO:teuthology.orchestra.run.smithi084.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-03-19T01:49:10.954 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-03-19T01:49:11.006 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-19T01:49:11.006 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-19T01:49:11.045 INFO:teuthology.orchestra.run.smithi055.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-19T01:49:11.056 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-19T01:49:11.056 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi055.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi084.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-19T01:49:11.056 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-19T01:49:11.110 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-19T01:49:11.110 INFO:tasks.ceph:role: osd.0 2024-03-19T01:49:11.110 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-19T01:49:11.110 DEBUG:teuthology.orchestra.run.smithi055:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-19T01:49:11.194 INFO:teuthology.orchestra.run.smithi055.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T01:49:11.195 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T01:49:11.195 INFO:teuthology.orchestra.run.smithi055.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T01:49:11.195 INFO:teuthology.orchestra.run.smithi055.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T01:49:11.195 INFO:teuthology.orchestra.run.smithi055.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T01:49:11.195 INFO:teuthology.orchestra.run.smithi055.stdout: = sunit=0 swidth=0 blks 2024-03-19T01:49:11.195 INFO:teuthology.orchestra.run.smithi055.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T01:49:11.195 INFO:teuthology.orchestra.run.smithi055.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T01:49:11.195 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T01:49:11.195 INFO:teuthology.orchestra.run.smithi055.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T01:49:12.486 INFO:teuthology.orchestra.run.smithi055.stdout:Discarding blocks...Done. 2024-03-19T01:49:12.489 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi055.front.sepia.ceph.com -o noatime 2024-03-19T01:49:12.489 DEBUG:teuthology.orchestra.run.smithi055:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-19T01:49:12.552 DEBUG:teuthology.orchestra.run.smithi055:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-19T01:49:12.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:49:12.603 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T01:49:12.603 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-19T01:49:12.656 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-19T01:49:12.656 INFO:tasks.ceph:role: osd.1 2024-03-19T01:49:12.656 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-19T01:49:12.657 DEBUG:teuthology.orchestra.run.smithi055:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-19T01:49:12.738 INFO:teuthology.orchestra.run.smithi055.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T01:49:12.739 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T01:49:12.739 INFO:teuthology.orchestra.run.smithi055.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T01:49:12.739 INFO:teuthology.orchestra.run.smithi055.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T01:49:12.739 INFO:teuthology.orchestra.run.smithi055.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T01:49:12.739 INFO:teuthology.orchestra.run.smithi055.stdout: = sunit=0 swidth=0 blks 2024-03-19T01:49:12.739 INFO:teuthology.orchestra.run.smithi055.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T01:49:12.739 INFO:teuthology.orchestra.run.smithi055.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T01:49:12.739 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T01:49:12.740 INFO:teuthology.orchestra.run.smithi055.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T01:49:14.030 INFO:teuthology.orchestra.run.smithi055.stdout:Discarding blocks...Done. 2024-03-19T01:49:14.033 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi055.front.sepia.ceph.com -o noatime 2024-03-19T01:49:14.033 DEBUG:teuthology.orchestra.run.smithi055:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-19T01:49:14.093 DEBUG:teuthology.orchestra.run.smithi055:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-19T01:49:14.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:49:14.143 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T01:49:14.143 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-19T01:49:14.196 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-19T01:49:14.196 INFO:tasks.ceph:role: osd.2 2024-03-19T01:49:14.196 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-19T01:49:14.196 DEBUG:teuthology.orchestra.run.smithi055:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-19T01:49:14.278 INFO:teuthology.orchestra.run.smithi055.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T01:49:14.278 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T01:49:14.278 INFO:teuthology.orchestra.run.smithi055.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T01:49:14.279 INFO:teuthology.orchestra.run.smithi055.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T01:49:14.279 INFO:teuthology.orchestra.run.smithi055.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T01:49:14.279 INFO:teuthology.orchestra.run.smithi055.stdout: = sunit=0 swidth=0 blks 2024-03-19T01:49:14.279 INFO:teuthology.orchestra.run.smithi055.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T01:49:14.279 INFO:teuthology.orchestra.run.smithi055.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T01:49:14.279 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T01:49:14.279 INFO:teuthology.orchestra.run.smithi055.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T01:49:15.595 INFO:teuthology.orchestra.run.smithi055.stdout:Discarding blocks...Done. 2024-03-19T01:49:15.597 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi055.front.sepia.ceph.com -o noatime 2024-03-19T01:49:15.598 DEBUG:teuthology.orchestra.run.smithi055:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-19T01:49:15.660 DEBUG:teuthology.orchestra.run.smithi055:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-19T01:49:15.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:49:15.711 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T01:49:15.711 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-19T01:49:15.763 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-19T01:49:15.763 INFO:tasks.ceph:role: osd.3 2024-03-19T01:49:15.764 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-19T01:49:15.764 DEBUG:teuthology.orchestra.run.smithi055:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-19T01:49:15.846 INFO:teuthology.orchestra.run.smithi055.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T01:49:15.847 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T01:49:15.847 INFO:teuthology.orchestra.run.smithi055.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T01:49:15.847 INFO:teuthology.orchestra.run.smithi055.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T01:49:15.847 INFO:teuthology.orchestra.run.smithi055.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T01:49:15.847 INFO:teuthology.orchestra.run.smithi055.stdout: = sunit=0 swidth=0 blks 2024-03-19T01:49:15.847 INFO:teuthology.orchestra.run.smithi055.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T01:49:15.847 INFO:teuthology.orchestra.run.smithi055.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T01:49:15.847 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T01:49:15.847 INFO:teuthology.orchestra.run.smithi055.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T01:49:17.169 INFO:teuthology.orchestra.run.smithi055.stdout:Discarding blocks...Done. 2024-03-19T01:49:17.171 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi055.front.sepia.ceph.com -o noatime 2024-03-19T01:49:17.172 DEBUG:teuthology.orchestra.run.smithi055:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-19T01:49:17.233 DEBUG:teuthology.orchestra.run.smithi055:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-19T01:49:17.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:49:17.283 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T01:49:17.283 DEBUG:teuthology.orchestra.run.smithi055:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:49:17.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:17.395+0000 7f66481467c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-19T01:49:17.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:17.395+0000 7f66481467c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-19T01:49:17.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:17.395+0000 7f66481467c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-19T01:49:17.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:17.395+0000 7f66481467c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-19T01:49:17.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:17.395+0000 7f66481467c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-19T01:49:17.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:17.395+0000 7f66481467c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-19T01:49:20.366 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-19T01:49:20.382 DEBUG:teuthology.orchestra.run.smithi055:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:49:20.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:20.467+0000 7f651f7307c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-19T01:49:20.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:20.467+0000 7f651f7307c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-19T01:49:20.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:20.467+0000 7f651f7307c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-19T01:49:20.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:20.467+0000 7f651f7307c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-19T01:49:20.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:20.467+0000 7f651f7307c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-19T01:49:20.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:20.467+0000 7f651f7307c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-19T01:49:23.458 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-19T01:49:23.515 DEBUG:teuthology.orchestra.run.smithi055:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:49:23.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:23.603+0000 7f65377ce7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-19T01:49:23.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:23.603+0000 7f65377ce7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-19T01:49:23.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:23.603+0000 7f65377ce7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-19T01:49:23.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:23.603+0000 7f65377ce7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-19T01:49:23.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:23.603+0000 7f65377ce7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-19T01:49:23.605 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:23.603+0000 7f65377ce7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-19T01:49:26.609 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-19T01:49:26.664 DEBUG:teuthology.orchestra.run.smithi055:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:49:26.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:26.747+0000 7faa8f3057c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-19T01:49:26.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:26.747+0000 7faa8f3057c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-19T01:49:26.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:26.747+0000 7faa8f3057c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-19T01:49:26.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:26.747+0000 7faa8f3057c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-19T01:49:26.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:26.747+0000 7faa8f3057c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-19T01:49:26.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:26.747+0000 7faa8f3057c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-19T01:49:29.734 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-19T01:49:29.791 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-19T01:49:29.806 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-19T01:49:29.806 INFO:tasks.ceph:role: osd.4 2024-03-19T01:49:29.807 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-19T01:49:29.807 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-19T01:49:29.871 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T01:49:29.871 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T01:49:29.871 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T01:49:29.871 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T01:49:29.872 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T01:49:29.872 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2024-03-19T01:49:29.872 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T01:49:29.872 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T01:49:29.872 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T01:49:29.872 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T01:49:30.318 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2024-03-19T01:49:30.321 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2024-03-19T01:49:30.321 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-19T01:49:30.417 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-19T01:49:30.427 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T01:49:30.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:49:30.428 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-19T01:49:30.482 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-19T01:49:30.482 INFO:tasks.ceph:role: osd.5 2024-03-19T01:49:30.483 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-19T01:49:30.483 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-19T01:49:30.543 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T01:49:30.543 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T01:49:30.543 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T01:49:30.543 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T01:49:30.543 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T01:49:30.543 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2024-03-19T01:49:30.543 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T01:49:30.543 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T01:49:30.544 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T01:49:30.544 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T01:49:31.013 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2024-03-19T01:49:31.015 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2024-03-19T01:49:31.015 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-19T01:49:31.126 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-19T01:49:31.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:49:31.137 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T01:49:31.137 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-19T01:49:31.191 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-19T01:49:31.191 INFO:tasks.ceph:role: osd.6 2024-03-19T01:49:31.191 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-19T01:49:31.191 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-19T01:49:31.254 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T01:49:31.254 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T01:49:31.255 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T01:49:31.255 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T01:49:31.255 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T01:49:31.255 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2024-03-19T01:49:31.255 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T01:49:31.255 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T01:49:31.255 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T01:49:31.255 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T01:49:31.729 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2024-03-19T01:49:31.731 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2024-03-19T01:49:31.731 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-19T01:49:31.860 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-19T01:49:31.870 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T01:49:31.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:49:31.871 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-19T01:49:31.928 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-19T01:49:31.928 INFO:tasks.ceph:role: osd.7 2024-03-19T01:49:31.928 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-19T01:49:31.929 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-19T01:49:31.990 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T01:49:31.990 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T01:49:31.990 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T01:49:31.991 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T01:49:31.991 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T01:49:31.991 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2024-03-19T01:49:31.991 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T01:49:31.991 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T01:49:31.991 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T01:49:31.991 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T01:49:32.430 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2024-03-19T01:49:32.431 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2024-03-19T01:49:32.431 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-19T01:49:32.509 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-19T01:49:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:49:32.561 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T01:49:32.561 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:49:32.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:32.667+0000 7fe04c0f67c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-19T01:49:32.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:32.671+0000 7fe04c0f67c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-19T01:49:32.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:32.671+0000 7fe04c0f67c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-19T01:49:32.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:32.671+0000 7fe04c0f67c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-19T01:49:32.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:32.671+0000 7fe04c0f67c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-19T01:49:32.677 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:32.675+0000 7fe04c0f67c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-19T01:49:35.651 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-19T01:49:35.665 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:49:35.750 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:35.747+0000 7f639e8fe7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-19T01:49:35.751 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:35.747+0000 7f639e8fe7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-19T01:49:35.751 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:35.747+0000 7f639e8fe7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-19T01:49:35.751 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:35.747+0000 7f639e8fe7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-19T01:49:35.751 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:35.747+0000 7f639e8fe7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-19T01:49:35.751 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:35.747+0000 7f639e8fe7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-19T01:49:38.002 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:49:38.005 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:49:38.722 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-19T01:49:38.734 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:49:38.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:38.811+0000 7f45ad26e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-19T01:49:38.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:38.811+0000 7f45ad26e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-19T01:49:38.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:38.811+0000 7f45ad26e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-19T01:49:38.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:38.811+0000 7f45ad26e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-19T01:49:38.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:38.811+0000 7f45ad26e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-19T01:49:38.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:38.811+0000 7f45ad26e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-19T01:49:41.806 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-19T01:49:41.860 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:49:41.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:41.939+0000 7f55e68637c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-19T01:49:41.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:41.939+0000 7f55e68637c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-19T01:49:41.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:41.939+0000 7f55e68637c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-19T01:49:41.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:41.939+0000 7f55e68637c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-19T01:49:41.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:41.939+0000 7f55e68637c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-19T01:49:41.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:49:41.939+0000 7f55e68637c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-19T01:49:44.906 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-19T01:49:44.959 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-19T01:49:44.959 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:49:44.959 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-19T01:49:44.974 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:49:44.974 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-19T01:49:45.030 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:49:45.030 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-19T01:49:45.085 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:49:45.085 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-03-19T01:49:45.141 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:49:45.141 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-19T01:49:45.198 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:49:45.198 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-19T01:49:45.254 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:49:45.254 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-19T01:49:45.309 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:49:45.310 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-19T01:49:45.365 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:49:45.365 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-19T01:49:45.379 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:49:45.379 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-19T01:49:45.436 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:49:45.436 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-19T01:49:45.491 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:49:45.491 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-03-19T01:49:45.547 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:49:45.547 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-19T01:49:45.603 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:49:45.603 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-19T01:49:45.658 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:49:45.658 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-19T01:49:45.710 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:49:45.710 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-19T01:49:45.763 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:49:45.763 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-19T01:49:45.769 INFO:tasks.ceph:Adding keys to all mons... 2024-03-19T01:49:45.769 DEBUG:teuthology.orchestra.run.smithi055:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-19T01:49:45.811 DEBUG:teuthology.orchestra.run.smithi084:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-19T01:49:45.821 INFO:teuthology.orchestra.run.smithi055.stdout:[mgr.y] 2024-03-19T01:49:45.821 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAW7/hlaXBTCBAAT7EYw1lbKdpvRoJjh/3SYA== 2024-03-19T01:49:45.821 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.a] 2024-03-19T01:49:45.821 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAW7/hleSo7ExAADjrBDkO2qHzvL/I07nqZzw== 2024-03-19T01:49:45.821 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.c] 2024-03-19T01:49:45.821 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAW7/hlDaTVGxAAg4I63P7lA396sAjoED/8fw== 2024-03-19T01:49:45.821 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.e] 2024-03-19T01:49:45.821 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAW7/hl0MBwJBAA3yy+AD6enp4icFowhzGnCA== 2024-03-19T01:49:45.821 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.0] 2024-03-19T01:49:45.821 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAd7/hlnAOfFxAAphP2NkLPtmOQ8UkcW0fq3w== 2024-03-19T01:49:45.821 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.1] 2024-03-19T01:49:45.822 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAg7/hlHUv1GxAA9HgzifSe8GqImrstzaVP/g== 2024-03-19T01:49:45.822 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.2] 2024-03-19T01:49:45.822 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAj7/hlxk4YJBAALNcwnLD/8SXuxxz3B59P+g== 2024-03-19T01:49:45.822 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.3] 2024-03-19T01:49:45.822 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAm7/hlzYWfLBAA12L6NTQLXsaZBnp9gX9MHQ== 2024-03-19T01:49:45.822 INFO:teuthology.orchestra.run.smithi055.stdout:[mgr.x] 2024-03-19T01:49:45.822 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAW7/hlpks4CxAAx/iy09AWE/fxjp4eNzS+Hw== 2024-03-19T01:49:45.822 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.b] 2024-03-19T01:49:45.822 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAW7/hlmPqyKhAAsX5grnF7kVVZDekCla7YIA== 2024-03-19T01:49:45.822 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.d] 2024-03-19T01:49:45.822 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAW7/hl66OiMBAAQKYmaadLsV1YF7Z54ra8vQ== 2024-03-19T01:49:45.822 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.f] 2024-03-19T01:49:45.822 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAW7/hl84j2OBAAwjHOvUxmSIWo+1EHzR3rNw== 2024-03-19T01:49:45.823 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.4] 2024-03-19T01:49:45.823 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAs7/hlH1kjKBAA8DHUPLawhQMQJ+WDOSWQvg== 2024-03-19T01:49:45.823 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.5] 2024-03-19T01:49:45.823 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAv7/hlWR3wLBAAwj1rkCfiOfs3ZNc835UGJw== 2024-03-19T01:49:45.823 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.6] 2024-03-19T01:49:45.823 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAy7/hlwXW1MBAAmJFqhLLEn5VhiMUyTM6fVQ== 2024-03-19T01:49:45.823 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.7] 2024-03-19T01:49:45.823 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQA17/hlXk1OOBAAk1c3p9ToHBhb9sa41CzRgw== 2024-03-19T01:49:45.823 INFO:teuthology.orchestra.run.smithi055.stdout:[client.0] 2024-03-19T01:49:45.823 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAX7/hluyrVAhAAseLYhL2N9UsVPQKJZ1K9jQ== 2024-03-19T01:49:45.825 INFO:teuthology.orchestra.run.smithi084.stdout:[mgr.y] 2024-03-19T01:49:45.825 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAW7/hlaXBTCBAAT7EYw1lbKdpvRoJjh/3SYA== 2024-03-19T01:49:45.825 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.a] 2024-03-19T01:49:45.825 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAW7/hleSo7ExAADjrBDkO2qHzvL/I07nqZzw== 2024-03-19T01:49:45.825 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.c] 2024-03-19T01:49:45.825 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAW7/hlDaTVGxAAg4I63P7lA396sAjoED/8fw== 2024-03-19T01:49:45.825 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.e] 2024-03-19T01:49:45.825 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAW7/hl0MBwJBAA3yy+AD6enp4icFowhzGnCA== 2024-03-19T01:49:45.825 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.0] 2024-03-19T01:49:45.825 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAd7/hlnAOfFxAAphP2NkLPtmOQ8UkcW0fq3w== 2024-03-19T01:49:45.825 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.1] 2024-03-19T01:49:45.825 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAg7/hlHUv1GxAA9HgzifSe8GqImrstzaVP/g== 2024-03-19T01:49:45.826 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.2] 2024-03-19T01:49:45.826 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAj7/hlxk4YJBAALNcwnLD/8SXuxxz3B59P+g== 2024-03-19T01:49:45.826 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.3] 2024-03-19T01:49:45.826 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAm7/hlzYWfLBAA12L6NTQLXsaZBnp9gX9MHQ== 2024-03-19T01:49:45.826 INFO:teuthology.orchestra.run.smithi084.stdout:[mgr.x] 2024-03-19T01:49:45.826 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAW7/hlpks4CxAAx/iy09AWE/fxjp4eNzS+Hw== 2024-03-19T01:49:45.826 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.b] 2024-03-19T01:49:45.826 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAW7/hlmPqyKhAAsX5grnF7kVVZDekCla7YIA== 2024-03-19T01:49:45.826 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.d] 2024-03-19T01:49:45.826 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAW7/hl66OiMBAAQKYmaadLsV1YF7Z54ra8vQ== 2024-03-19T01:49:45.826 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.f] 2024-03-19T01:49:45.826 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAW7/hl84j2OBAAwjHOvUxmSIWo+1EHzR3rNw== 2024-03-19T01:49:45.827 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.4] 2024-03-19T01:49:45.827 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAs7/hlH1kjKBAA8DHUPLawhQMQJ+WDOSWQvg== 2024-03-19T01:49:45.827 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.5] 2024-03-19T01:49:45.827 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAv7/hlWR3wLBAAwj1rkCfiOfs3ZNc835UGJw== 2024-03-19T01:49:45.827 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.6] 2024-03-19T01:49:45.827 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAy7/hlwXW1MBAAmJFqhLLEn5VhiMUyTM6fVQ== 2024-03-19T01:49:45.827 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.7] 2024-03-19T01:49:45.827 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQA17/hlXk1OOBAAk1c3p9ToHBhb9sa41CzRgw== 2024-03-19T01:49:45.827 INFO:teuthology.orchestra.run.smithi084.stdout:[client.0] 2024-03-19T01:49:45.827 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAX7/hluyrVAhAAseLYhL2N9UsVPQKJZ1K9jQ== 2024-03-19T01:49:45.827 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-19T01:49:45.869 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-19T01:49:45.915 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T01:49:45.955 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T01:49:46.007 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T01:49:46.010 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T01:49:46.054 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T01:49:46.056 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T01:49:46.140 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:49:46.142 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:49:46.185 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:49:46.231 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:49:46.274 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:49:46.315 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:49:46.361 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:49:46.399 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:49:46.447 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-19T01:49:46.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-19T01:49:46.536 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T01:49:46.567 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T01:49:46.612 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T01:49:46.614 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T01:49:46.698 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T01:49:46.701 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-19T01:49:46.789 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:49:46.791 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:49:46.876 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:49:46.878 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:49:46.968 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:49:46.971 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:49:47.012 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:49:47.055 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T01:49:47.095 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-19T01:49:47.139 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-19T01:49:47.183 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-19T01:49:47.183 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-19T01:49:47.243 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-19T01:49:47.348 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-19T01:49:47.358 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-19T01:49:47.372 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-19T01:49:47.466 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-19T01:49:47.476 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-19T01:49:47.529 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-19T01:49:47.614 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-19T01:49:47.669 DEBUG:teuthology.orchestra.run.smithi055:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:49:47.671 DEBUG:teuthology.orchestra.run.smithi084:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-19T01:49:47.715 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-19T01:49:47.715 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-19T01:49:47.715 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-19T01:49:47.715 DEBUG:teuthology.orchestra.run.smithi055:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-19T01:49:47.719 INFO:tasks.ceph.mon.a:Started 2024-03-19T01:49:47.719 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-19T01:49:47.719 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-19T01:49:47.719 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-19T01:49:47.760 INFO:tasks.ceph.mon.b:Started 2024-03-19T01:49:47.760 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-19T01:49:47.761 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-19T01:49:47.761 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-19T01:49:47.763 INFO:tasks.ceph.mon.c:Started 2024-03-19T01:49:47.763 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-19T01:49:47.763 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-19T01:49:47.763 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-19T01:49:47.766 INFO:tasks.ceph.mgr.y:Started 2024-03-19T01:49:47.766 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-19T01:49:47.766 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-19T01:49:47.769 INFO:tasks.ceph.mgr.x:Started 2024-03-19T01:49:47.769 INFO:tasks.ceph:Setting crush tunables to default 2024-03-19T01:49:47.769 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster ceph osd crush tunables default 2024-03-19T01:49:47.891 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:47.891+0000 7f689ba39640 1 Processor -- start 2024-03-19T01:49:47.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:47.891+0000 7f689ba39640 1 -- start start 2024-03-19T01:49:47.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:47.891+0000 7f689ba39640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f6894154340 0x7f689415e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:49:47.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:47.891+0000 7f689ba39640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6894154720 0x7f68940c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:49:47.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:47.891+0000 7f689ba39640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f689415aca0 0x7f6894072e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:49:47.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:47.891+0000 7f689ba39640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f6894073840 con 0x7f689415aca0 2024-03-19T01:49:47.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:47.891+0000 7f689ba39640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f689415df10 con 0x7f6894154340 2024-03-19T01:49:47.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:47.891+0000 7f689ba39640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f6894071280 con 0x7f6894154720 2024-03-19T01:49:47.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:47.891+0000 7f6898fd4640 1 -- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6894154720 msgr2=0x7f68940c73c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.84:3301/0 2024-03-19T01:49:47.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:47.891+0000 7f6899fd6640 1 -- >> v2:172.21.15.55:3300/0 conn(0x7f689415aca0 msgr2=0x7f6894072e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.55:3300/0 2024-03-19T01:49:47.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:47.891+0000 7f68997d5640 1 -- >> v2:172.21.15.84:3300/0 conn(0x7f6894154340 msgr2=0x7f689415e3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.84:3300/0 2024-03-19T01:49:47.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:47.891+0000 7f6898fd4640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6894154720 0x7f68940c73c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T01:49:47.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:47.891+0000 7f68997d5640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f6894154340 0x7f689415e3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T01:49:47.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:47.891+0000 7f6899fd6640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f689415aca0 0x7f6894072e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T01:49:48.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:48.095+0000 7f6899fd6640 1 -- >> v2:172.21.15.55:3300/0 conn(0x7f689415aca0 msgr2=0x7f6894072e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.55:3300/0 2024-03-19T01:49:48.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:48.095+0000 7f6899fd6640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f689415aca0 0x7f6894072e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T01:49:48.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:48.095+0000 7f68997d5640 1 -- >> v2:172.21.15.84:3300/0 conn(0x7f6894154340 msgr2=0x7f689415e3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.84:3300/0 2024-03-19T01:49:48.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:48.095+0000 7f6898fd4640 1 -- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6894154720 msgr2=0x7f68940c73c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.84:3301/0 2024-03-19T01:49:48.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:48.095+0000 7f68997d5640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f6894154340 0x7f689415e3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T01:49:48.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:48.095+0000 7f6898fd4640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6894154720 0x7f68940c73c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T01:49:48.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:48.499+0000 7f6899fd6640 1 -- >> v2:172.21.15.55:3300/0 conn(0x7f689415aca0 msgr2=0x7f6894072e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.55:3300/0 2024-03-19T01:49:48.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:48.499+0000 7f6899fd6640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f689415aca0 0x7f6894072e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T01:49:48.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:48.499+0000 7f68997d5640 1 -- >> v2:172.21.15.84:3300/0 conn(0x7f6894154340 msgr2=0x7f689415e3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.84:3300/0 2024-03-19T01:49:48.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:48.499+0000 7f6898fd4640 1 -- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6894154720 msgr2=0x7f68940c73c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.84:3301/0 2024-03-19T01:49:48.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:48.499+0000 7f68997d5640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f6894154340 0x7f689415e3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T01:49:48.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:48.499+0000 7f6898fd4640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6894154720 0x7f68940c73c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T01:49:49.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:49.303+0000 7f6899fd6640 1 -- >> v2:172.21.15.55:3300/0 conn(0x7f689415aca0 msgr2=0x7f6894072e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.55:3300/0 2024-03-19T01:49:49.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:49.303+0000 7f6899fd6640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f689415aca0 0x7f6894072e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T01:49:49.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:49.303+0000 7f68997d5640 1 -- >> v2:172.21.15.84:3300/0 conn(0x7f6894154340 msgr2=0x7f689415e3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.84:3300/0 2024-03-19T01:49:49.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:49.303+0000 7f6898fd4640 1 -- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6894154720 msgr2=0x7f68940c73c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.84:3301/0 2024-03-19T01:49:49.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:49.303+0000 7f68997d5640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f6894154340 0x7f689415e3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T01:49:49.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:49.303+0000 7f6898fd4640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6894154720 0x7f68940c73c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T01:49:50.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:50.891+0000 7f6886ffd640 1 -- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6894154720 msgr2=0x7f68940c73c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:49:50.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:50.891+0000 7f6886ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6894154720 0x7f68940c73c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:49:50.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:50.891+0000 7f6886ffd640 1 -- >> v2:172.21.15.84:3300/0 conn(0x7f6894154340 msgr2=0x7f689415e3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:49:50.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:50.891+0000 7f6886ffd640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f6894154340 0x7f689415e3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:49:50.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:50.891+0000 7f6886ffd640 1 -- >> v2:172.21.15.55:3300/0 conn(0x7f689415aca0 msgr2=0x7f6894072e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:49:50.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:50.891+0000 7f6886ffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f689415aca0 0x7f6894072e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:49:50.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:50.891+0000 7f6886ffd640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f689415d900 con 0x7f6878007370 2024-03-19T01:49:50.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:50.891+0000 7f6886ffd640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f687800c0e0 con 0x7f68780036c0 2024-03-19T01:49:50.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:50.891+0000 7f6898fd4640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f6878007370 legacy=0x7f6878009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6789/0 2024-03-19T01:49:50.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:50.891+0000 7f68997d5640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7f68780036c0 legacy=0x7f6878005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6789/0 2024-03-19T01:49:51.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:51.095+0000 7f6898fd4640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f6878007370 legacy=0x7f6878009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6789/0 2024-03-19T01:49:51.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:51.095+0000 7f68997d5640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7f68780036c0 legacy=0x7f6878005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6789/0 2024-03-19T01:49:51.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:51.499+0000 7f6898fd4640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f6878007370 legacy=0x7f6878009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6789/0 2024-03-19T01:49:51.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:51.499+0000 7f68997d5640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7f68780036c0 legacy=0x7f6878005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6789/0 2024-03-19T01:49:52.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:52.303+0000 7f6898fd4640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f6878007370 legacy=0x7f6878009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6789/0 2024-03-19T01:49:52.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:52.303+0000 7f68997d5640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7f68780036c0 legacy=0x7f6878005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6789/0 2024-03-19T01:49:53.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:53.891+0000 7f6886ffd640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7f68780036c0 legacy=0x7f6878005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:49:53.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:53.891+0000 7f6886ffd640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f6878007370 legacy=0x7f6878009830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:49:53.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:53.891+0000 7f6899fd6640 1 -- reap_dead start 2024-03-19T01:49:53.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:53.891+0000 7f6886ffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f687800c2e0 0x7f6878009830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:49:53.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:53.891+0000 7f6886ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f687800caf0 0x7f6878009f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:49:53.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:53.891+0000 7f6886ffd640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f687800d3e0 con 0x7f687800c6c0 2024-03-19T01:49:53.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:53.891+0000 7f6886ffd640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f689415df10 con 0x7f687800c2e0 2024-03-19T01:49:53.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:53.891+0000 7f6886ffd640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f6894071280 con 0x7f687800caf0 2024-03-19T01:49:53.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:53.891+0000 7f68997d5640 1 -- >> v2:172.21.15.55:3300/0 conn(0x7f687800c2e0 msgr2=0x7f6878009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.55:3300/0 2024-03-19T01:49:53.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:53.891+0000 7f68997d5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f687800c2e0 0x7f6878009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T01:49:53.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:53.891+0000 7f6898fd4640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7f687800c6c0 legacy=0x7f6878005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6789/0 2024-03-19T01:49:53.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:53.891+0000 7f6899fd6640 1 -- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f687800caf0 msgr2=0x7f6878009f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.84:3301/0 2024-03-19T01:49:53.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:53.891+0000 7f6899fd6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f687800caf0 0x7f6878009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T01:49:54.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:54.095+0000 7f68997d5640 1 -- >> v2:172.21.15.55:3300/0 conn(0x7f687800c2e0 msgr2=0x7f6878009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.55:3300/0 2024-03-19T01:49:54.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:54.095+0000 7f68997d5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f687800c2e0 0x7f6878009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T01:49:54.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:54.095+0000 7f6898fd4640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7f687800c6c0 legacy=0x7f6878005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6789/0 2024-03-19T01:49:54.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:54.095+0000 7f6899fd6640 1 -- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f687800caf0 msgr2=0x7f6878009f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.84:3301/0 2024-03-19T01:49:54.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:54.095+0000 7f6899fd6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f687800caf0 0x7f6878009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T01:49:54.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:54.499+0000 7f68997d5640 1 -- >> v2:172.21.15.55:3300/0 conn(0x7f687800c2e0 msgr2=0x7f6878009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.55:3300/0 2024-03-19T01:49:54.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:54.499+0000 7f68997d5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f687800c2e0 0x7f6878009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T01:49:54.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:54.499+0000 7f6899fd6640 1 -- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f687800caf0 msgr2=0x7f6878009f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.84:3301/0 2024-03-19T01:49:54.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:54.499+0000 7f6899fd6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f687800caf0 0x7f6878009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T01:49:54.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:54.499+0000 7f6898fd4640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7f687800c6c0 legacy=0x7f6878005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6789/0 2024-03-19T01:49:55.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:55.303+0000 7f68997d5640 1 -- >> v2:172.21.15.55:3300/0 conn(0x7f687800c2e0 msgr2=0x7f6878009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.55:3300/0 2024-03-19T01:49:55.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:55.303+0000 7f6899fd6640 1 -- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f687800caf0 msgr2=0x7f6878009f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.84:3301/0 2024-03-19T01:49:55.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:55.303+0000 7f6899fd6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f687800caf0 0x7f6878009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T01:49:55.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:55.303+0000 7f6898fd4640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7f687800c6c0 legacy=0x7f6878005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.84:6789/0 2024-03-19T01:49:55.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:55.303+0000 7f68997d5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f687800c2e0 0x7f6878009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T01:49:56.066 INFO:tasks.ceph.mon.b.smithi084.stderr:==00:00:00:08.162 27626== Warning: unimplemented fcntl command: 1036 2024-03-19T01:49:56.115 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:08.211 27627== Warning: unimplemented fcntl command: 1036 2024-03-19T01:49:56.394 INFO:tasks.ceph.mon.b.smithi084.stderr:==00:00:00:08.490 27626== Warning: unimplemented fcntl command: 1036 2024-03-19T01:49:56.432 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:00:08.590 27555== Warning: unimplemented fcntl command: 1036 2024-03-19T01:49:56.442 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:00:08.538 27627== Warning: unimplemented fcntl command: 1036 2024-03-19T01:49:56.777 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:00:08.935 27555== Warning: unimplemented fcntl command: 1036 2024-03-19T01:49:56.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:56.891+0000 7f6886ffd640 1 -- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f687800caf0 msgr2=0x7f6878009f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:49:56.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:56.891+0000 7f6886ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f687800caf0 0x7f6878009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:49:56.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:56.891+0000 7f6886ffd640 1 -- >> v2:172.21.15.55:3300/0 conn(0x7f687800c2e0 msgr2=0x7f6878009830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:49:56.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:56.891+0000 7f6886ffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f687800c2e0 0x7f6878009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:49:56.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:56.891+0000 7f6886ffd640 1 -- >> v1:172.21.15.84:6789/0 conn(0x7f687800c6c0 legacy=0x7f6878005ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:49:56.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:56.891+0000 7f6886ffd640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f687800a4a0 0x7f687800a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:49:56.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:56.895+0000 7f6886ffd640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f689415d900 con 0x7f6878006270 2024-03-19T01:49:56.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:56.895+0000 7f6886ffd640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f689415df10 con 0x7f687800a4a0 2024-03-19T01:49:56.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:56.895+0000 7f6898fd4640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f6878006270 legacy=0x7f6878006680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6789/0 2024-03-19T01:49:56.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:56.895+0000 7f68997d5640 1 -- >> v2:172.21.15.84:3300/0 conn(0x7f687800a4a0 msgr2=0x7f687800a8b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.84:3300/0 2024-03-19T01:49:56.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:56.895+0000 7f68997d5640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f687800a4a0 0x7f687800a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T01:49:57.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:57.099+0000 7f6898fd4640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f6878006270 legacy=0x7f6878006680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6789/0 2024-03-19T01:49:57.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:57.099+0000 7f68997d5640 1 -- >> v2:172.21.15.84:3300/0 conn(0x7f687800a4a0 msgr2=0x7f687800a8b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.84:3300/0 2024-03-19T01:49:57.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:57.099+0000 7f68997d5640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f687800a4a0 0x7f687800a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T01:49:57.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:57.503+0000 7f6898fd4640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f6878006270 legacy=0x7f6878006680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6789/0 2024-03-19T01:49:57.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:57.503+0000 7f68997d5640 1 -- >> v2:172.21.15.84:3300/0 conn(0x7f687800a4a0 msgr2=0x7f687800a8b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.84:3300/0 2024-03-19T01:49:57.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:57.503+0000 7f68997d5640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f687800a4a0 0x7f687800a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T01:49:58.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:58.431+0000 7f68997d5640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f687800a4a0 0x7f687800a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:49:58.439 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:58.439+0000 7f68997d5640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f687800a4a0 0x7f687800a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:42542/0 (socket says 172.21.15.55:42542) 2024-03-19T01:49:58.439 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:58.439+0000 7f68997d5640 1 -- 172.21.15.55:0/2514237336 learned_addr learned my addr 172.21.15.55:0/2514237336 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:49:58.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:58.647+0000 7f6898fd4640 1 -- 172.21.15.55:0/2514237336 >> v1:172.21.15.55:6789/0 conn(0x7f6878006270 legacy=0x7f6878006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-19T01:49:58.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:58.647+0000 7f6898fd4640 1 -- 172.21.15.55:0/2514237336 >> v1:172.21.15.55:6789/0 conn(0x7f6878006270 legacy=0x7f6878006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:49:58.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:58.647+0000 7f6898fd4640 1 --1- 172.21.15.55:0/2514237336 >> v1:172.21.15.55:6789/0 conn(0x7f6878006270 0x7f6878006680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify read banner and identify addresses failed 2024-03-19T01:49:58.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:58.955+0000 7f68997d5640 1 -- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f687800a4a0 msgr2=0x7f687800a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-19T01:49:58.956 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:58.955+0000 7f68997d5640 1 -- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f687800a4a0 msgr2=0x7f687800a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:49:58.956 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:58.955+0000 7f68997d5640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f687800a4a0 0x7f687800a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:49:58.956 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:58.955+0000 7f68997d5640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f687800a4a0 0x7f687800a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T01:49:59.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:59.895+0000 7f6886ffd640 1 -- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f687800a4a0 msgr2=0x7f687800a8b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:49:59.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:59.895+0000 7f6886ffd640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f687800a4a0 0x7f687800a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:49:59.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:59.895+0000 7f6886ffd640 1 -- 172.21.15.55:0/2514237336 >> v1:172.21.15.55:6789/0 conn(0x7f6878006270 legacy=0x7f6878006680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:49:59.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:59.895+0000 7f6899fd6640 1 -- 172.21.15.55:0/2514237336 reap_dead start 2024-03-19T01:49:59.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:59.895+0000 7f6886ffd640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 0x7f68780097b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:49:59.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:59.895+0000 7f6886ffd640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 0x7f6878009f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:49:59.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:59.895+0000 7f6886ffd640 1 -- 172.21.15.55:0/2514237336 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f687800d3e0 con 0x7f687800c6c0 2024-03-19T01:49:59.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:59.895+0000 7f6886ffd640 1 -- 172.21.15.55:0/2514237336 --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f689415df10 con 0x7f6878006650 2024-03-19T01:49:59.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:59.895+0000 7f6886ffd640 1 -- 172.21.15.55:0/2514237336 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f6894071280 con 0x7f6878006270 2024-03-19T01:49:59.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:59.895+0000 7f6898fd4640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 0x7f68780097b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:49:59.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:59.903+0000 7f6899fd6640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 0x7f6878009f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:49:59.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:59.907+0000 7f6899fd6640 1 -- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 msgr2=0x7f6878009f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-19T01:49:59.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:59.907+0000 7f6899fd6640 1 -- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 msgr2=0x7f6878009f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:49:59.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:59.907+0000 7f6899fd6640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 0x7f6878009f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:49:59.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:59.907+0000 7f6899fd6640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 0x7f6878009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T01:49:59.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:59.923+0000 7f6898fd4640 1 -- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 msgr2=0x7f68780097b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-19T01:49:59.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:59.923+0000 7f6898fd4640 1 -- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 msgr2=0x7f68780097b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:49:59.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:59.923+0000 7f6898fd4640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 0x7f68780097b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:49:59.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:49:59.923+0000 7f6898fd4640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 0x7f68780097b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T01:50:00.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:00.119+0000 7f6899fd6640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 0x7f6878009f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:00.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:00.123+0000 7f6899fd6640 1 -- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 msgr2=0x7f6878009f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-19T01:50:00.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:00.123+0000 7f6899fd6640 1 -- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 msgr2=0x7f6878009f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:50:00.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:00.123+0000 7f6899fd6640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 0x7f6878009f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:50:00.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:00.123+0000 7f6899fd6640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 0x7f6878009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T01:50:00.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:00.127+0000 7f6898fd4640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 0x7f68780097b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:00.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:00.127+0000 7f6898fd4640 1 -- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 msgr2=0x7f68780097b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-19T01:50:00.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:00.127+0000 7f6898fd4640 1 -- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 msgr2=0x7f68780097b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:50:00.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:00.127+0000 7f6898fd4640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 0x7f68780097b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:50:00.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:00.127+0000 7f6898fd4640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 0x7f68780097b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T01:50:00.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:00.527+0000 7f6899fd6640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 0x7f6878009f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:00.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:00.527+0000 7f6899fd6640 1 -- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 msgr2=0x7f6878009f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-19T01:50:00.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:00.527+0000 7f6899fd6640 1 -- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 msgr2=0x7f6878009f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:50:00.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:00.527+0000 7f6899fd6640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 0x7f6878009f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:50:00.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:00.527+0000 7f6899fd6640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 0x7f6878009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T01:50:00.622 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:00.623+0000 7f6898fd4640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 0x7f68780097b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:00.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:00.695+0000 7f6898fd4640 1 -- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 msgr2=0x7f68780097b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-19T01:50:00.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:00.695+0000 7f6898fd4640 1 -- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 msgr2=0x7f68780097b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:50:00.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:00.695+0000 7f6898fd4640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 0x7f68780097b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:50:00.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:00.695+0000 7f6898fd4640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 0x7f68780097b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T01:50:01.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:01.331+0000 7f6899fd6640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 0x7f6878009f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:01.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:01.335+0000 7f6899fd6640 1 -- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 msgr2=0x7f6878009f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-19T01:50:01.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:01.335+0000 7f6899fd6640 1 -- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 msgr2=0x7f6878009f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:50:01.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:01.335+0000 7f6899fd6640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 0x7f6878009f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:50:01.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:01.335+0000 7f6899fd6640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 0x7f6878009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T01:50:01.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:01.499+0000 7f6898fd4640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 0x7f68780097b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:01.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:01.499+0000 7f6898fd4640 1 -- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 msgr2=0x7f68780097b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-19T01:50:01.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:01.499+0000 7f6898fd4640 1 -- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 msgr2=0x7f68780097b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:50:01.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:01.499+0000 7f6898fd4640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 0x7f68780097b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:50:01.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:01.499+0000 7f6898fd4640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 0x7f68780097b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T01:50:02.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:02.895+0000 7f6886ffd640 1 -- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 msgr2=0x7f68780097b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:50:02.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:02.895+0000 7f6886ffd640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6878006270 0x7f68780097b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:02.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:02.895+0000 7f6886ffd640 1 -- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 msgr2=0x7f6878009f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:50:02.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:02.895+0000 7f6886ffd640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.84:3300/0 conn(0x7f6878006650 0x7f6878009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:02.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:02.895+0000 7f6886ffd640 1 -- 172.21.15.55:0/2514237336 >> v1:172.21.15.84:6789/0 conn(0x7f687800c6c0 legacy=0x7f6878005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:02.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:02.895+0000 7f6886ffd640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.55:3300/0 conn(0x7f687800a4a0 0x7f687800a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:02.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:02.895+0000 7f6886ffd640 1 -- 172.21.15.55:0/2514237336 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f689415d900 con 0x7f6878011730 2024-03-19T01:50:02.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:02.895+0000 7f6886ffd640 1 -- 172.21.15.55:0/2514237336 --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f689415df10 con 0x7f687800a4a0 2024-03-19T01:50:02.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:02.903+0000 7f68997d5640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.55:3300/0 conn(0x7f687800a4a0 0x7f687800a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:02.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:02.907+0000 7f68997d5640 1 -- 172.21.15.55:0/2514237336 >> v2:172.21.15.55:3300/0 conn(0x7f687800a4a0 msgr2=0x7f687800a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-19T01:50:02.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:02.907+0000 7f68997d5640 1 -- 172.21.15.55:0/2514237336 >> v2:172.21.15.55:3300/0 conn(0x7f687800a4a0 msgr2=0x7f687800a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:50:02.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:02.907+0000 7f68997d5640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.55:3300/0 conn(0x7f687800a4a0 0x7f687800a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:50:02.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:02.907+0000 7f68997d5640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.55:3300/0 conn(0x7f687800a4a0 0x7f687800a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T01:50:03.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:03.239+0000 7f68997d5640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.55:3300/0 conn(0x7f687800a4a0 0x7f687800a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:03.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:03.279+0000 7f68997d5640 1 -- 172.21.15.55:0/2514237336 >> v2:172.21.15.55:3300/0 conn(0x7f687800a4a0 msgr2=0x7f687800a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-19T01:50:03.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:03.279+0000 7f68997d5640 1 -- 172.21.15.55:0/2514237336 >> v2:172.21.15.55:3300/0 conn(0x7f687800a4a0 msgr2=0x7f687800a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:50:03.282 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:03.279+0000 7f68997d5640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.55:3300/0 conn(0x7f687800a4a0 0x7f687800a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:50:03.282 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:03.279+0000 7f68997d5640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.55:3300/0 conn(0x7f687800a4a0 0x7f687800a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T01:50:03.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:03.767+0000 7f68997d5640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.55:3300/0 conn(0x7f687800a4a0 0x7f687800a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:03.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:03.787+0000 7f68997d5640 1 -- 172.21.15.55:0/2514237336 >> v2:172.21.15.55:3300/0 conn(0x7f687800a4a0 msgr2=0x7f687800a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-19T01:50:03.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:03.787+0000 7f68997d5640 1 -- 172.21.15.55:0/2514237336 >> v2:172.21.15.55:3300/0 conn(0x7f687800a4a0 msgr2=0x7f687800a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-19T01:50:03.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:03.787+0000 7f68997d5640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.55:3300/0 conn(0x7f687800a4a0 0x7f687800a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:50:03.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:03.787+0000 7f68997d5640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.55:3300/0 conn(0x7f687800a4a0 0x7f687800a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T01:50:04.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.011+0000 7f6887fff640 1 -- 172.21.15.55:0/2514237336 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3632692244 0 0) 0x7f689415d900 con 0x7f6878011730 2024-03-19T01:50:04.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.011+0000 7f6887fff640 1 -- 172.21.15.55:0/2514237336 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6880003530 con 0x7f6878011730 2024-03-19T01:50:04.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.151+0000 7f6887fff640 1 -- 172.21.15.55:0/2514237336 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 445298476 0 0) 0x7f6880003530 con 0x7f6878011730 2024-03-19T01:50:04.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.151+0000 7f6887fff640 1 -- 172.21.15.55:0/2514237336 >> v2:172.21.15.55:3300/0 conn(0x7f687800a4a0 msgr2=0x7f687800a8b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:50:04.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.151+0000 7f6887fff640 1 --2- 172.21.15.55:0/2514237336 >> v2:172.21.15.55:3300/0 conn(0x7f687800a4a0 0x7f687800a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:04.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.151+0000 7f6887fff640 1 -- 172.21.15.55:0/2514237336 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f687800c0e0 con 0x7f6878011730 2024-03-19T01:50:04.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.151+0000 7f6887fff640 1 -- 172.21.15.55:0/2514237336 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f6888009470 con 0x7f6878011730 2024-03-19T01:50:04.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.151+0000 7f6887fff640 1 -- 172.21.15.55:0/2514237336 >> v1:172.21.15.55:6789/0 conn(0x7f6878011730 legacy=0x7f6878013b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:04.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.151+0000 7f6899fd6640 1 -- 172.21.15.55:0/2514237336 reap_dead start 2024-03-19T01:50:04.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.151+0000 7f6887fff640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68800039c0 0x7f6880003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:04.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.151+0000 7f6887fff640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6880004810 0x7f6880004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:04.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.151+0000 7f6887fff640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6880005220 0x7f688000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:04.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.151+0000 7f6887fff640 1 -- 172.21.15.55:0/2514237336 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f689415df10 con 0x7f6880005220 2024-03-19T01:50:04.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.151+0000 7f6887fff640 1 -- 172.21.15.55:0/2514237336 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f688000c150 con 0x7f6880004810 2024-03-19T01:50:04.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.151+0000 7f6887fff640 1 -- 172.21.15.55:0/2514237336 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f688000c350 con 0x7f68800039c0 2024-03-19T01:50:04.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.155+0000 7f68997d5640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68800039c0 0x7f6880003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:04.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.155+0000 7f6898fd4640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6880004810 0x7f6880004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:04.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.235+0000 7f6898fd4640 1 -- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68800039c0 msgr2=0x7f6880003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:04.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.235+0000 7f6898fd4640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68800039c0 0x7f6880003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:04.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.235+0000 7f6898fd4640 1 -- 172.21.15.55:0/2514237336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6880005220 msgr2=0x7f688000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:04.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.235+0000 7f6898fd4640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6880005220 0x7f688000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:04.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.235+0000 7f6898fd4640 1 -- 172.21.15.55:0/2514237336 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6880003530 con 0x7f6880004810 2024-03-19T01:50:04.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.243+0000 7f6898fd4640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6880004810 0x7f6880004c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6888003b30 tx=0x7f6888020300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64e15aa7fa89dc49 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:04.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.347+0000 7f6887fff640 1 -- 172.21.15.55:0/2514237336 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68880b5070 con 0x7f6880004810 2024-03-19T01:50:04.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.347+0000 7f6887fff640 1 -- 172.21.15.55:0/2514237336 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68880b0600 con 0x7f6880004810 2024-03-19T01:50:04.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.347+0000 7f6887fff640 1 -- 172.21.15.55:0/2514237336 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6888009c30 con 0x7f6880004810 2024-03-19T01:50:04.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.347+0000 7f689ba39640 1 -- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6880004810 msgr2=0x7f6880004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:04.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.347+0000 7f689ba39640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6880004810 0x7f6880004c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6888003b30 tx=0x7f6888020300 comp rx=0 tx=0).stop 2024-03-19T01:50:04.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.351+0000 7f689ba39640 1 -- 172.21.15.55:0/2514237336 shutdown_connections 2024-03-19T01:50:04.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.351+0000 7f689ba39640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6880005220 0x7f688000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:04.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.351+0000 7f689ba39640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6880004810 0x7f6880004c90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:04.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.351+0000 7f689ba39640 1 --2- 172.21.15.55:0/2514237336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68800039c0 0x7f6880003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:04.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.351+0000 7f689ba39640 1 -- 172.21.15.55:0/2514237336 >> 172.21.15.55:0/2514237336 conn(0x7f68940cc9c0 msgr2=0x7f68940ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:04.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.351+0000 7f689ba39640 1 -- 172.21.15.55:0/2514237336 shutdown_connections 2024-03-19T01:50:04.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.351+0000 7f689ba39640 1 -- 172.21.15.55:0/2514237336 wait complete. 2024-03-19T01:50:04.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.351+0000 7f689ba39640 1 Processor -- start 2024-03-19T01:50:04.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.351+0000 7f689ba39640 1 -- start start 2024-03-19T01:50:04.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.355+0000 7f689ba39640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68800039c0 0x7f68941efef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:04.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.355+0000 7f689ba39640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6880004810 0x7f689415c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:04.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.355+0000 7f689ba39640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6880005220 0x7f6894159730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:04.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.355+0000 7f689ba39640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f689415b100 con 0x7f6880004810 2024-03-19T01:50:04.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.355+0000 7f689ba39640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f689415dba0 con 0x7f68800039c0 2024-03-19T01:50:04.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.355+0000 7f689ba39640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f6894070c20 con 0x7f6880005220 2024-03-19T01:50:04.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.355+0000 7f6898fd4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6880004810 0x7f689415c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:04.356 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.355+0000 7f6898fd4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6880004810 0x7f689415c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47596/0 (socket says 172.21.15.55:47596) 2024-03-19T01:50:04.356 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.355+0000 7f6898fd4640 1 -- 172.21.15.55:0/3948819280 learned_addr learned my addr 172.21.15.55:0/3948819280 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:50:04.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.375+0000 7f6898fd4640 1 -- 172.21.15.55:0/3948819280 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6880005220 msgr2=0x7f6894159730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:04.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.375+0000 7f6898fd4640 1 --2- 172.21.15.55:0/3948819280 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6880005220 0x7f6894159730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:04.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.375+0000 7f6898fd4640 1 -- 172.21.15.55:0/3948819280 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68800039c0 msgr2=0x7f68941efef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:04.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.375+0000 7f6898fd4640 1 --2- 172.21.15.55:0/3948819280 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68800039c0 0x7f68941efef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:04.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.375+0000 7f6898fd4640 1 -- 172.21.15.55:0/3948819280 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f689415ba40 con 0x7f6880004810 2024-03-19T01:50:04.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.383+0000 7f6898fd4640 1 --2- 172.21.15.55:0/3948819280 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6880004810 0x7f689415c540 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6888020130 tx=0x7f688801d100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:04.415 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.415+0000 7f6885ffb640 1 -- 172.21.15.55:0/3948819280 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68880b5050 con 0x7f6880004810 2024-03-19T01:50:04.415 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.415+0000 7f6885ffb640 1 -- 172.21.15.55:0/3948819280 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68880c1070 con 0x7f6880004810 2024-03-19T01:50:04.415 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.415+0000 7f6885ffb640 1 -- 172.21.15.55:0/3948819280 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68880b07b0 con 0x7f6880004810 2024-03-19T01:50:04.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.415+0000 7f689ba39640 1 -- 172.21.15.55:0/3948819280 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68941f2c30 con 0x7f6880004810 2024-03-19T01:50:04.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.415+0000 7f689ba39640 1 -- 172.21.15.55:0/3948819280 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68941f31f0 con 0x7f6880004810 2024-03-19T01:50:04.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.415+0000 7f689ba39640 1 -- 172.21.15.55:0/3948819280 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6894068a40 con 0x7f6880004810 2024-03-19T01:50:04.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.451+0000 7f6885ffb640 1 -- 172.21.15.55:0/3948819280 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f68880b04c0 con 0x7f6880004810 2024-03-19T01:50:04.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.451+0000 7f6885ffb640 1 -- 172.21.15.55:0/3948819280 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f68880bc9f0 con 0x7f6880004810 2024-03-19T01:50:04.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.643+0000 7f6885ffb640 1 -- 172.21.15.55:0/3948819280 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f688801a070 con 0x7f6880004810 2024-03-19T01:50:04.717 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:04.715+0000 7f689ba39640 1 -- 172.21.15.55:0/3948819280 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f68940c3cf0 con 0x7f6880004810 2024-03-19T01:50:04.806 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:04.807+0000 7f2e680a7180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T01:50:04.822 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:04.823+0000 7fcb1fd6c180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T01:50:05.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:05.039+0000 7f6885ffb640 1 -- 172.21.15.55:0/3948819280 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f68880bc450 con 0x7f6880004810 2024-03-19T01:50:05.038 INFO:teuthology.orchestra.run.smithi055.stderr:adjusted tunables profile to default 2024-03-19T01:50:05.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:05.039+0000 7f689ba39640 1 -- 172.21.15.55:0/3948819280 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6880004810 msgr2=0x7f689415c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:05.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:05.039+0000 7f689ba39640 1 --2- 172.21.15.55:0/3948819280 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6880004810 0x7f689415c540 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6888020130 tx=0x7f688801d100 comp rx=0 tx=0).stop 2024-03-19T01:50:05.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:05.039+0000 7f689ba39640 1 -- 172.21.15.55:0/3948819280 shutdown_connections 2024-03-19T01:50:05.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:05.039+0000 7f689ba39640 1 --2- 172.21.15.55:0/3948819280 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6880005220 0x7f6894159730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:05.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:05.039+0000 7f689ba39640 1 --2- 172.21.15.55:0/3948819280 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6880004810 0x7f689415c540 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:05.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:05.039+0000 7f689ba39640 1 --2- 172.21.15.55:0/3948819280 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68800039c0 0x7f68941efef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:05.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:05.039+0000 7f689ba39640 1 -- 172.21.15.55:0/3948819280 >> 172.21.15.55:0/3948819280 conn(0x7f68940cc9c0 msgr2=0x7f68940c9710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:05.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:05.039+0000 7f689ba39640 1 -- 172.21.15.55:0/3948819280 shutdown_connections 2024-03-19T01:50:05.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:05.039+0000 7f689ba39640 1 -- 172.21.15.55:0/3948819280 wait complete. 2024-03-19T01:50:05.058 INFO:tasks.ceph:check_enable_crimson: False 2024-03-19T01:50:05.059 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-19T01:50:05.059 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:50:05.059 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-19T01:50:05.070 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:50:05.070 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-19T01:50:05.130 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:50:05.131 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-19T01:50:05.186 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:50:05.186 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-19T01:50:05.240 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:50:05.241 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-19T01:50:05.254 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:50:05.254 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-19T01:50:05.310 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:50:05.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-19T01:50:05.321 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:05.323+0000 7fcb1fd6c180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T01:50:05.332 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-19T01:50:05.332 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-19T01:50:05.349 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:05.347+0000 7f2e680a7180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T01:50:05.386 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd new 35d5e3c5-d089-4f38-8961-98d064188f80 0 2024-03-19T01:50:05.546 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.547+0000 7fd81c8f2640 1 Processor -- start 2024-03-19T01:50:05.546 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.547+0000 7fd81c8f2640 1 -- start start 2024-03-19T01:50:05.547 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.551+0000 7fd81c8f2640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd818152120 0x7fd8180c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:05.547 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:05.551+0000 7fcb1fd6c180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T01:50:05.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.551+0000 7fd81c8f2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd8180c9610 0x7fd818157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:05.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.551+0000 7fd81c8f2640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fd818070780 0x7fd818158510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:05.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.551+0000 7fd81c8f2640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fd818152500 con 0x7fd818152120 2024-03-19T01:50:05.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.551+0000 7fd81c8f2640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fd818070590 con 0x7fd818070780 2024-03-19T01:50:05.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.551+0000 7fd81c8f2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd8180c3530 con 0x7fd8180c9610 2024-03-19T01:50:05.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.551+0000 7fd815d9b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd8180c9610 0x7fd818157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:05.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.551+0000 7fd816d9d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fd818070780 0x7fd818158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:05.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.551+0000 7fd815d9b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd8180c9610 0x7fd818157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.84:46828/0 (socket says 172.21.15.84:46828) 2024-03-19T01:50:05.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.551+0000 7fd815d9b640 1 -- 172.21.15.84:0/3301303134 learned_addr learned my addr 172.21.15.84:0/3301303134 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-03-19T01:50:05.549 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:05.547+0000 7f2e680a7180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T01:50:05.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.551+0000 7fd81659c640 1 --2- 172.21.15.84:0/3301303134 >> v2:172.21.15.55:3300/0 conn(0x7fd818152120 0x7fd8180c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:05.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.551+0000 7fd815d9b640 1 -- 172.21.15.84:0/3301303134 >> v2:172.21.15.84:3300/0 conn(0x7fd818070780 msgr2=0x7fd818158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:05.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.551+0000 7fd815d9b640 1 --2- 172.21.15.84:0/3301303134 >> v2:172.21.15.84:3300/0 conn(0x7fd818070780 0x7fd818158510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:05.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.551+0000 7fd815d9b640 1 -- 172.21.15.84:0/3301303134 >> v2:172.21.15.55:3300/0 conn(0x7fd818152120 msgr2=0x7fd8180c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:05.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.551+0000 7fd815d9b640 1 --2- 172.21.15.84:0/3301303134 >> v2:172.21.15.55:3300/0 conn(0x7fd818152120 0x7fd8180c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:05.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.551+0000 7fd815d9b640 1 -- 172.21.15.84:0/3301303134 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd81806fff0 con 0x7fd8180c9610 2024-03-19T01:50:05.554 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.555+0000 7fd815d9b640 1 --2- 172.21.15.84:0/3301303134 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd8180c9610 0x7fd818157fd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd818071f20 tx=0x7fd80c01c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c64a8069714276e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:05.559 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.563+0000 7fd81559a640 1 -- 172.21.15.84:0/3301303134 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80c0b0070 con 0x7fd8180c9610 2024-03-19T01:50:05.559 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.563+0000 7fd81559a640 1 -- 172.21.15.84:0/3301303134 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd80c0ab600 con 0x7fd8180c9610 2024-03-19T01:50:05.559 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.563+0000 7fd81559a640 1 -- 172.21.15.84:0/3301303134 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80c0ab9d0 con 0x7fd8180c9610 2024-03-19T01:50:05.560 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.563+0000 7fd81c8f2640 1 -- 172.21.15.84:0/3301303134 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd8180c9610 msgr2=0x7fd818157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:05.560 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.563+0000 7fd81c8f2640 1 --2- 172.21.15.84:0/3301303134 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd8180c9610 0x7fd818157fd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd818071f20 tx=0x7fd80c01c520 comp rx=0 tx=0).stop 2024-03-19T01:50:05.560 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.563+0000 7fd81c8f2640 1 -- 172.21.15.84:0/3301303134 shutdown_connections 2024-03-19T01:50:05.560 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.563+0000 7fd81c8f2640 1 --2- 172.21.15.84:0/3301303134 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd8180c9610 0x7fd818157fd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:05.560 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.563+0000 7fd81c8f2640 1 --2- 172.21.15.84:0/3301303134 >> v2:172.21.15.84:3300/0 conn(0x7fd818070780 0x7fd818158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:05.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.563+0000 7fd81c8f2640 1 --2- 172.21.15.84:0/3301303134 >> v2:172.21.15.55:3300/0 conn(0x7fd818152120 0x7fd8180c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:05.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.563+0000 7fd81c8f2640 1 -- 172.21.15.84:0/3301303134 >> 172.21.15.84:0/3301303134 conn(0x7fd818067600 msgr2=0x7fd818071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:05.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.563+0000 7fd81c8f2640 1 -- 172.21.15.84:0/3301303134 shutdown_connections 2024-03-19T01:50:05.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.563+0000 7fd81c8f2640 1 -- 172.21.15.84:0/3301303134 wait complete. 2024-03-19T01:50:05.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.563+0000 7fd81c8f2640 1 Processor -- start 2024-03-19T01:50:05.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.563+0000 7fd81c8f2640 1 -- start start 2024-03-19T01:50:05.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.563+0000 7fd81c8f2640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd818070780 0x7fd81806cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:05.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.563+0000 7fd81c8f2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd8180c9610 0x7fd818156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:05.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.563+0000 7fd81c8f2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd818152120 0x7fd818142330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:05.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.563+0000 7fd81c8f2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd8180c31c0 con 0x7fd818152120 2024-03-19T01:50:05.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.563+0000 7fd81c8f2640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd8180c2ee0 con 0x7fd818070780 2024-03-19T01:50:05.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.563+0000 7fd81c8f2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd8180c86d0 con 0x7fd8180c9610 2024-03-19T01:50:05.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.563+0000 7fd815d9b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd8180c9610 0x7fd818156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:05.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.567+0000 7fd81659c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd818070780 0x7fd81806cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:05.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.567+0000 7fd815d9b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd8180c9610 0x7fd818156f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.84:46842/0 (socket says 172.21.15.84:46842) 2024-03-19T01:50:05.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.567+0000 7fd815d9b640 1 -- 172.21.15.84:0/3934073302 learned_addr learned my addr 172.21.15.84:0/3934073302 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-03-19T01:50:05.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.567+0000 7fd816d9d640 1 --2- 172.21.15.84:0/3934073302 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd818152120 0x7fd818142330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:05.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.567+0000 7fd815d9b640 1 -- 172.21.15.84:0/3934073302 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd818070780 msgr2=0x7fd81806cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:05.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.567+0000 7fd815d9b640 1 --2- 172.21.15.84:0/3934073302 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd818070780 0x7fd81806cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:05.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.567+0000 7fd815d9b640 1 -- 172.21.15.84:0/3934073302 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd818152120 msgr2=0x7fd818142330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:05.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.567+0000 7fd815d9b640 1 --2- 172.21.15.84:0/3934073302 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd818152120 0x7fd818142330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:05.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.567+0000 7fd815d9b640 1 -- 172.21.15.84:0/3934073302 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd81815a900 con 0x7fd8180c9610 2024-03-19T01:50:05.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.571+0000 7fd815d9b640 1 --2- 172.21.15.84:0/3934073302 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd8180c9610 0x7fd818156f60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd80c0097c0 tx=0x7fd80c0192c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:05.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.575+0000 7fd806ffd640 1 -- 172.21.15.84:0/3934073302 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80c0b0040 con 0x7fd8180c9610 2024-03-19T01:50:05.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.575+0000 7fd806ffd640 1 -- 172.21.15.84:0/3934073302 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd80c0b9070 con 0x7fd8180c9610 2024-03-19T01:50:05.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.575+0000 7fd81c8f2640 1 -- 172.21.15.84:0/3934073302 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd81815abe0 con 0x7fd8180c9610 2024-03-19T01:50:05.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.575+0000 7fd806ffd640 1 -- 172.21.15.84:0/3934073302 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80c0b4660 con 0x7fd8180c9610 2024-03-19T01:50:05.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.575+0000 7fd81c8f2640 1 -- 172.21.15.84:0/3934073302 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8181f33f0 con 0x7fd8180c9610 2024-03-19T01:50:05.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.575+0000 7fd81c8f2640 1 -- 172.21.15.84:0/3934073302 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd818068a40 con 0x7fd8180c9610 2024-03-19T01:50:05.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.783+0000 7fd806ffd640 1 -- 172.21.15.84:0/3934073302 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd80c0b4800 con 0x7fd8180c9610 2024-03-19T01:50:05.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.783+0000 7fd806ffd640 1 -- 172.21.15.84:0/3934073302 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fd80c0be7b0 con 0x7fd8180c9610 2024-03-19T01:50:05.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.783+0000 7fd806ffd640 1 -- 172.21.15.84:0/3934073302 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd80c016110 con 0x7fd8180c9610 2024-03-19T01:50:05.857 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:05.859+0000 7fd81c8f2640 1 -- 172.21.15.84:0/3934073302 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "osd new", "uuid": "35d5e3c5-d089-4f38-8961-98d064188f80", "id": 0} v 0) -- 0x7fd8180c84b0 con 0x7fd8180c9610 2024-03-19T01:50:06.127 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:06.131+0000 7fcb1fd6c180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T01:50:06.155 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:06.155+0000 7f2e680a7180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T01:50:06.186 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:06.187+0000 7fcb1fd6c180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T01:50:06.200 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.203+0000 7fd806ffd640 1 -- 172.21.15.84:0/3934073302 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "35d5e3c5-d089-4f38-8961-98d064188f80", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fd80c0b4cd0 con 0x7fd8180c9610 2024-03-19T01:50:06.201 INFO:teuthology.orchestra.run.smithi084.stdout:0 2024-03-19T01:50:06.202 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.203+0000 7fd81c8f2640 1 -- 172.21.15.84:0/3934073302 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd8180c9610 msgr2=0x7fd818156f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:06.202 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.203+0000 7fd81c8f2640 1 --2- 172.21.15.84:0/3934073302 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd8180c9610 0x7fd818156f60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd80c0097c0 tx=0x7fd80c0192c0 comp rx=0 tx=0).stop 2024-03-19T01:50:06.203 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.203+0000 7fd81c8f2640 1 -- 172.21.15.84:0/3934073302 shutdown_connections 2024-03-19T01:50:06.203 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.203+0000 7fd81c8f2640 1 --2- 172.21.15.84:0/3934073302 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd818152120 0x7fd818142330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:06.203 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.203+0000 7fd81c8f2640 1 --2- 172.21.15.84:0/3934073302 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd8180c9610 0x7fd818156f60 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:06.203 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.203+0000 7fd81c8f2640 1 --2- 172.21.15.84:0/3934073302 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd818070780 0x7fd81806cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:06.203 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.203+0000 7fd81c8f2640 1 -- 172.21.15.84:0/3934073302 >> 172.21.15.84:0/3934073302 conn(0x7fd818067600 msgr2=0x7fd818074540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:06.203 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.203+0000 7fd81c8f2640 1 -- 172.21.15.84:0/3934073302 shutdown_connections 2024-03-19T01:50:06.203 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.203+0000 7fd81c8f2640 1 -- 172.21.15.84:0/3934073302 wait complete. 2024-03-19T01:50:06.213 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:06.215+0000 7f2e680a7180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T01:50:06.224 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd new 8e58f791-001a-44d0-a464-ca68a479f70c 1 2024-03-19T01:50:06.250 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:06.251+0000 7fcb1fd6c180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T01:50:06.270 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:06.271+0000 7f2e680a7180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T01:50:06.334 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:06.335+0000 7fcb1fd6c180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T01:50:06.353 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:06.351+0000 7f2e680a7180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T01:50:06.354 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.355+0000 7f42aae9f640 1 Processor -- start 2024-03-19T01:50:06.354 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.355+0000 7f42aae9f640 1 -- start start 2024-03-19T01:50:06.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.359+0000 7f42aae9f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f42a40c61c0 0x7f42a40708a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:06.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.359+0000 7f42aae9f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f42a40c73c0 0x7f42a40c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:06.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.359+0000 7f42aae9f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42a40c3f60 con 0x7f42a41510f0 2024-03-19T01:50:06.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.359+0000 7f42aae9f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f42a40c6630 con 0x7f42a40c61c0 2024-03-19T01:50:06.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.359+0000 7f42aae9f640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f42a41514d0 con 0x7f42a40c73c0 2024-03-19T01:50:06.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.359+0000 7f42a943c640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f42a40c73c0 0x7f42a40c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:06.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.359+0000 7f42a8c3b640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f42a41510f0 0x7f42a415a410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:35640/0 (socket says 172.21.15.84:35640) 2024-03-19T01:50:06.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.359+0000 7f42a8c3b640 1 -- 172.21.15.84:0/4072540627 learned_addr learned my addr 172.21.15.84:0/4072540627 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-19T01:50:06.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.359+0000 7f42a3fff640 1 --2- 172.21.15.84:0/4072540627 >> v2:172.21.15.55:3300/0 conn(0x7f42a40c61c0 0x7f42a40708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:06.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.363+0000 7f42a943c640 1 -- 172.21.15.84:0/4072540627 >> v2:172.21.15.55:3300/0 conn(0x7f42a40c61c0 msgr2=0x7f42a40708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:06.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.363+0000 7f42a943c640 1 --2- 172.21.15.84:0/4072540627 >> v2:172.21.15.55:3300/0 conn(0x7f42a40c61c0 0x7f42a40708a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:06.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.363+0000 7f42a943c640 1 -- 172.21.15.84:0/4072540627 >> v1:172.21.15.84:6789/0 conn(0x7f42a41510f0 legacy=0x7f42a415a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:06.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.363+0000 7f42a943c640 1 -- 172.21.15.84:0/4072540627 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a40c6870 con 0x7f42a40c73c0 2024-03-19T01:50:06.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.367+0000 7f42a943c640 1 --2- 172.21.15.84:0/4072540627 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42a40c73c0 0x7f42a40c77f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f42a415b510 tx=0x7f429c01f110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4eacc60727907c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:06.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.371+0000 7f42a37fe640 1 -- 172.21.15.84:0/4072540627 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429c0b4070 con 0x7f42a40c73c0 2024-03-19T01:50:06.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.371+0000 7f42a37fe640 1 -- 172.21.15.84:0/4072540627 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f429c0af600 con 0x7f42a40c73c0 2024-03-19T01:50:06.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.371+0000 7f42a37fe640 1 -- 172.21.15.84:0/4072540627 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429c0af9d0 con 0x7f42a40c73c0 2024-03-19T01:50:06.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.375+0000 7f42aae9f640 1 -- 172.21.15.84:0/4072540627 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42a40c73c0 msgr2=0x7f42a40c77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:06.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.375+0000 7f42aae9f640 1 --2- 172.21.15.84:0/4072540627 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42a40c73c0 0x7f42a40c77f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f42a415b510 tx=0x7f429c01f110 comp rx=0 tx=0).stop 2024-03-19T01:50:06.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.375+0000 7f42aae9f640 1 -- 172.21.15.84:0/4072540627 shutdown_connections 2024-03-19T01:50:06.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.375+0000 7f42aae9f640 1 --2- 172.21.15.84:0/4072540627 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42a40c73c0 0x7f42a40c77f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:06.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.375+0000 7f42aae9f640 1 --2- 172.21.15.84:0/4072540627 >> v2:172.21.15.55:3300/0 conn(0x7f42a40c61c0 0x7f42a40708a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:06.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.375+0000 7f42aae9f640 1 -- 172.21.15.84:0/4072540627 >> 172.21.15.84:0/4072540627 conn(0x7f42a40cc9c0 msgr2=0x7f42a40ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:06.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.375+0000 7f42aae9f640 1 -- 172.21.15.84:0/4072540627 shutdown_connections 2024-03-19T01:50:06.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.375+0000 7f42aae9f640 1 -- 172.21.15.84:0/4072540627 wait complete. 2024-03-19T01:50:06.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.375+0000 7f42aae9f640 1 Processor -- start 2024-03-19T01:50:06.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.375+0000 7f42aae9f640 1 -- start start 2024-03-19T01:50:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.375+0000 7f42aae9f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42a40c61c0 0x7f42a41488e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.375+0000 7f42aae9f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42a40c73c0 0x7f42a40c34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.375+0000 7f42aae9f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f42a41510f0 0x7f42a41edb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.375+0000 7f42aae9f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f42a40c44e0 con 0x7f42a40c61c0 2024-03-19T01:50:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.375+0000 7f42aae9f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f42a40c4200 con 0x7f42a40c73c0 2024-03-19T01:50:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.375+0000 7f42aae9f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f42a40c6e30 con 0x7f42a41510f0 2024-03-19T01:50:06.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.375+0000 7f42a3fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42a40c73c0 0x7f42a40c34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:06.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.375+0000 7f42a943c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f42a41510f0 0x7f42a41edb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:06.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.375+0000 7f42a3fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42a40c73c0 0x7f42a40c34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:50946/0 (socket says 172.21.15.84:50946) 2024-03-19T01:50:06.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.375+0000 7f42a3fff640 1 -- 172.21.15.84:0/1228049400 learned_addr learned my addr 172.21.15.84:0/1228049400 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-03-19T01:50:06.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.375+0000 7f42a8c3b640 1 --2- 172.21.15.84:0/1228049400 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42a40c61c0 0x7f42a41488e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:06.377 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.379+0000 7f42a3fff640 1 -- 172.21.15.84:0/1228049400 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f42a41510f0 msgr2=0x7f42a41edb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:06.377 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.379+0000 7f42a3fff640 1 --2- 172.21.15.84:0/1228049400 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f42a41510f0 0x7f42a41edb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:06.377 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.379+0000 7f42a3fff640 1 -- 172.21.15.84:0/1228049400 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42a40c61c0 msgr2=0x7f42a41488e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:06.377 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.379+0000 7f42a3fff640 1 --2- 172.21.15.84:0/1228049400 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42a40c61c0 0x7f42a41488e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:06.377 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.379+0000 7f42a3fff640 1 -- 172.21.15.84:0/1228049400 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a41eea20 con 0x7f42a40c73c0 2024-03-19T01:50:06.377 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.379+0000 7f42a943c640 1 --2- 172.21.15.84:0/1228049400 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f42a41510f0 0x7f42a41edb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:50:06.380 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.383+0000 7f42a3fff640 1 --2- 172.21.15.84:0/1228049400 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42a40c73c0 0x7f42a40c34a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f429000e870 tx=0x7f429000ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:06.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.387+0000 7f42a17fa640 1 -- 172.21.15.84:0/1228049400 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4290017070 con 0x7f42a40c73c0 2024-03-19T01:50:06.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.387+0000 7f42a17fa640 1 -- 172.21.15.84:0/1228049400 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f429000c940 con 0x7f42a40c73c0 2024-03-19T01:50:06.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.387+0000 7f42a17fa640 1 -- 172.21.15.84:0/1228049400 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429000ccf0 con 0x7f42a40c73c0 2024-03-19T01:50:06.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.387+0000 7f42aae9f640 1 -- 172.21.15.84:0/1228049400 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42a41ee460 con 0x7f42a40c73c0 2024-03-19T01:50:06.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.387+0000 7f42aae9f640 1 -- 172.21.15.84:0/1228049400 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42a41f1170 con 0x7f42a40c73c0 2024-03-19T01:50:06.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.387+0000 7f42aae9f640 1 -- 172.21.15.84:0/1228049400 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42a4068a40 con 0x7f42a40c73c0 2024-03-19T01:50:06.394 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:06.395+0000 7fcb1fd6c180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T01:50:06.409 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:06.407+0000 7f2e680a7180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T01:50:06.472 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:06.475+0000 7fcb1fd6c180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T01:50:06.489 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:06.491+0000 7f2e680a7180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T01:50:06.526 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:06.527+0000 7fcb1fd6c180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T01:50:06.544 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:06.543+0000 7f2e680a7180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T01:50:06.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.591+0000 7f42a17fa640 1 -- 172.21.15.84:0/1228049400 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f429001a070 con 0x7f42a40c73c0 2024-03-19T01:50:06.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.591+0000 7f42a17fa640 1 -- 172.21.15.84:0/1228049400 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f42900064c0 con 0x7f42a40c73c0 2024-03-19T01:50:06.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.591+0000 7f42a17fa640 1 -- 172.21.15.84:0/1228049400 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4290006710 con 0x7f42a40c73c0 2024-03-19T01:50:06.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.663+0000 7f42aae9f640 1 -- 172.21.15.84:0/1228049400 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8e58f791-001a-44d0-a464-ca68a479f70c", "id": 1} v 0) -- 0x7f42a406b720 con 0x7f42a40c73c0 2024-03-19T01:50:06.676 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:06.679+0000 7fcb1fd6c180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T01:50:06.700 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:06.699+0000 7f2e680a7180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T01:50:06.738 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:06.739+0000 7fcb1fd6c180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T01:50:06.765 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:06.763+0000 7f2e680a7180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T01:50:06.952 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.955+0000 7f42a17fa640 1 -- 172.21.15.84:0/1228049400 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8e58f791-001a-44d0-a464-ca68a479f70c", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f42900059a0 con 0x7f42a40c73c0 2024-03-19T01:50:06.952 INFO:teuthology.orchestra.run.smithi084.stdout:1 2024-03-19T01:50:06.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.955+0000 7f42aae9f640 1 -- 172.21.15.84:0/1228049400 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42a40c73c0 msgr2=0x7f42a40c34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:06.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.955+0000 7f42aae9f640 1 --2- 172.21.15.84:0/1228049400 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42a40c73c0 0x7f42a40c34a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f429000e870 tx=0x7f429000ec70 comp rx=0 tx=0).stop 2024-03-19T01:50:06.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.955+0000 7f42aae9f640 1 -- 172.21.15.84:0/1228049400 shutdown_connections 2024-03-19T01:50:06.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.955+0000 7f42aae9f640 1 --2- 172.21.15.84:0/1228049400 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f42a41510f0 0x7f42a41edb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:06.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.955+0000 7f42aae9f640 1 --2- 172.21.15.84:0/1228049400 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42a40c73c0 0x7f42a40c34a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:06.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.955+0000 7f42aae9f640 1 --2- 172.21.15.84:0/1228049400 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42a40c61c0 0x7f42a41488e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:06.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.955+0000 7f42aae9f640 1 -- 172.21.15.84:0/1228049400 >> 172.21.15.84:0/1228049400 conn(0x7f42a40cc9c0 msgr2=0x7f42a406bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:06.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.955+0000 7f42aae9f640 1 -- 172.21.15.84:0/1228049400 shutdown_connections 2024-03-19T01:50:06.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:06.955+0000 7f42aae9f640 1 -- 172.21.15.84:0/1228049400 wait complete. 2024-03-19T01:50:06.974 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd new d44f38b8-67f4-43de-98a6-f3c520acaf10 2 2024-03-19T01:50:07.166 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.166+0000 7fbce2fa7640 1 Processor -- start 2024-03-19T01:50:07.166 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.166+0000 7fbce2fa7640 1 -- start start 2024-03-19T01:50:07.167 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.166+0000 7fbce2fa7640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fbcdc13de70 0x7fbcdc06e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:07.167 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.166+0000 7fbce2fa7640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fbcdc06f1b0 0x7fbcdc158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:07.167 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.166+0000 7fbce2fa7640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcdc15b740 con 0x7fbcdc06eb70 2024-03-19T01:50:07.167 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.166+0000 7fbce2fa7640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fbcdc13e250 con 0x7fbcdc13de70 2024-03-19T01:50:07.167 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.166+0000 7fbce2fa7640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fbcdc15ba20 con 0x7fbcdc06f1b0 2024-03-19T01:50:07.168 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.170+0000 7fbce1544640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fbcdc06f1b0 0x7fbcdc158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:07.168 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.170+0000 7fbce1544640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fbcdc06f1b0 0x7fbcdc158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:50954/0 (socket says 172.21.15.84:50954) 2024-03-19T01:50:07.168 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.170+0000 7fbce1544640 1 -- 172.21.15.84:0/290309732 learned_addr learned my addr 172.21.15.84:0/290309732 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-03-19T01:50:07.168 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.170+0000 7fbce0d43640 1 --2- 172.21.15.84:0/290309732 >> v2:172.21.15.55:3300/0 conn(0x7fbcdc13de70 0x7fbcdc06e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:07.169 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.170+0000 7fbce1544640 1 -- 172.21.15.84:0/290309732 >> v2:172.21.15.55:3300/0 conn(0x7fbcdc13de70 msgr2=0x7fbcdc06e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:07.170 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.170+0000 7fbce1544640 1 --2- 172.21.15.84:0/290309732 >> v2:172.21.15.55:3300/0 conn(0x7fbcdc13de70 0x7fbcdc06e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:07.170 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.170+0000 7fbce1544640 1 -- 172.21.15.84:0/290309732 >> v1:172.21.15.55:6789/0 conn(0x7fbcdc06eb70 legacy=0x7fbcdc0c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:07.170 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.170+0000 7fbce1544640 1 -- 172.21.15.84:0/290309732 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcdc15bd00 con 0x7fbcdc06f1b0 2024-03-19T01:50:07.173 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.174+0000 7fbce1544640 1 --2- 172.21.15.84:0/290309732 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbcdc06f1b0 0x7fbcdc158420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc141870 tx=0x7fbcd401f1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79286d383595405b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:07.177 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.178+0000 7fbcdb7fe640 1 -- 172.21.15.84:0/290309732 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd40b4070 con 0x7fbcdc06f1b0 2024-03-19T01:50:07.177 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.178+0000 7fbcdb7fe640 1 -- 172.21.15.84:0/290309732 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcd40af400 con 0x7fbcdc06f1b0 2024-03-19T01:50:07.177 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.178+0000 7fbcdb7fe640 1 -- 172.21.15.84:0/290309732 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd40af7d0 con 0x7fbcdc06f1b0 2024-03-19T01:50:07.178 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.178+0000 7fbce2fa7640 1 -- 172.21.15.84:0/290309732 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbcdc06f1b0 msgr2=0x7fbcdc158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:07.178 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.178+0000 7fbce2fa7640 1 --2- 172.21.15.84:0/290309732 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbcdc06f1b0 0x7fbcdc158420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc141870 tx=0x7fbcd401f1a0 comp rx=0 tx=0).stop 2024-03-19T01:50:07.178 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.178+0000 7fbce2fa7640 1 -- 172.21.15.84:0/290309732 shutdown_connections 2024-03-19T01:50:07.178 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.178+0000 7fbce2fa7640 1 --2- 172.21.15.84:0/290309732 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbcdc06f1b0 0x7fbcdc158420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:07.178 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.178+0000 7fbce2fa7640 1 --2- 172.21.15.84:0/290309732 >> v2:172.21.15.55:3300/0 conn(0x7fbcdc13de70 0x7fbcdc06e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:07.178 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.178+0000 7fbce2fa7640 1 -- 172.21.15.84:0/290309732 >> 172.21.15.84:0/290309732 conn(0x7fbcdc06fb40 msgr2=0x7fbcdc06ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:07.178 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.178+0000 7fbce2fa7640 1 -- 172.21.15.84:0/290309732 shutdown_connections 2024-03-19T01:50:07.179 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.178+0000 7fbce2fa7640 1 -- 172.21.15.84:0/290309732 wait complete. 2024-03-19T01:50:07.179 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.182+0000 7fbce2fa7640 1 Processor -- start 2024-03-19T01:50:07.179 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.182+0000 7fbce2fa7640 1 -- start start 2024-03-19T01:50:07.180 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.182+0000 7fbce2fa7640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbcdc06eb70 0x7fbcdc1e3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:07.180 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.182+0000 7fbce2fa7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbcdc06f1b0 0x7fbcdc14fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:07.180 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.182+0000 7fbce2fa7640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbcdc13de70 0x7fbcdc14ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:07.181 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.182+0000 7fbce2fa7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbcdc0c5f20 con 0x7fbcdc06f1b0 2024-03-19T01:50:07.181 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.182+0000 7fbce2fa7640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fbcdc15c2c0 con 0x7fbcdc06eb70 2024-03-19T01:50:07.181 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.182+0000 7fbce2fa7640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbcdc0c60a0 con 0x7fbcdc13de70 2024-03-19T01:50:07.181 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.182+0000 7fbce0d43640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbcdc06eb70 0x7fbcdc1e3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:07.181 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.182+0000 7fbce1544640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbcdc13de70 0x7fbcdc14ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:07.181 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.182+0000 7fbce0d43640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbcdc06eb70 0x7fbcdc1e3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:50962/0 (socket says 172.21.15.84:50962) 2024-03-19T01:50:07.181 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.182+0000 7fbce0d43640 1 -- 172.21.15.84:0/2207957927 learned_addr learned my addr 172.21.15.84:0/2207957927 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-03-19T01:50:07.184 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.186+0000 7fbce0d43640 1 -- 172.21.15.84:0/2207957927 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbcdc13de70 msgr2=0x7fbcdc14ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:07.184 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.186+0000 7fbce0d43640 1 --2- 172.21.15.84:0/2207957927 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbcdc13de70 0x7fbcdc14ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:07.184 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.186+0000 7fbce0d43640 1 -- 172.21.15.84:0/2207957927 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbcdc06f1b0 msgr2=0x7fbcdc14fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:07.184 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.186+0000 7fbce0d43640 1 --2- 172.21.15.84:0/2207957927 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbcdc06f1b0 0x7fbcdc14fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:07.184 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.186+0000 7fbce0d43640 1 -- 172.21.15.84:0/2207957927 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcdc1e13e0 con 0x7fbcdc06eb70 2024-03-19T01:50:07.186 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.186+0000 7fbce0d43640 1 --2- 172.21.15.84:0/2207957927 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbcdc06eb70 0x7fbcdc1e3100 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000e960 tx=0x7fbcd000ee20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:07.191 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.194+0000 7fbcd97fa640 1 -- 172.21.15.84:0/2207957927 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0017070 con 0x7fbcdc06eb70 2024-03-19T01:50:07.191 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.194+0000 7fbcd97fa640 1 -- 172.21.15.84:0/2207957927 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcd000ca00 con 0x7fbcdc06eb70 2024-03-19T01:50:07.191 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.194+0000 7fbcd97fa640 1 -- 172.21.15.84:0/2207957927 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd000cdb0 con 0x7fbcdc06eb70 2024-03-19T01:50:07.191 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.194+0000 7fbce2fa7640 1 -- 172.21.15.84:0/2207957927 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcdc1e16c0 con 0x7fbcdc06eb70 2024-03-19T01:50:07.191 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.194+0000 7fbce2fa7640 1 -- 172.21.15.84:0/2207957927 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcdc138300 con 0x7fbcdc06eb70 2024-03-19T01:50:07.192 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.194+0000 7fbce2fa7640 1 -- 172.21.15.84:0/2207957927 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcdc068a40 con 0x7fbcdc06eb70 2024-03-19T01:50:07.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.314+0000 7fbcd97fa640 1 -- 172.21.15.84:0/2207957927 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbcd001a070 con 0x7fbcdc06eb70 2024-03-19T01:50:07.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.314+0000 7fbcd97fa640 1 -- 172.21.15.84:0/2207957927 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fbcd0006650 con 0x7fbcdc06eb70 2024-03-19T01:50:07.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.314+0000 7fbcd97fa640 1 -- 172.21.15.84:0/2207957927 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fbcd0013890 con 0x7fbcdc06eb70 2024-03-19T01:50:07.326 INFO:tasks.ceph.mgr.x.smithi084.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-19T01:50:07.326 INFO:tasks.ceph.mgr.x.smithi084.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-19T01:50:07.326 INFO:tasks.ceph.mgr.x.smithi084.stderr: from numpy import show_config as show_numpy_config 2024-03-19T01:50:07.389 INFO:tasks.ceph.mgr.y.smithi055.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-19T01:50:07.390 INFO:tasks.ceph.mgr.y.smithi055.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-19T01:50:07.390 INFO:tasks.ceph.mgr.y.smithi055.stderr: from numpy import show_config as show_numpy_config 2024-03-19T01:50:07.391 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:07.390+0000 7fcb1fd6c180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T01:50:07.392 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.394+0000 7fbce2fa7640 1 -- 172.21.15.84:0/2207957927 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d44f38b8-67f4-43de-98a6-f3c520acaf10", "id": 2} v 0) -- 0x7fbcdc158f30 con 0x7fbcdc06eb70 2024-03-19T01:50:07.469 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:07.467+0000 7f2e680a7180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T01:50:07.547 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:07.550+0000 7fcb1fd6c180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T01:50:07.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.602+0000 7fbcd97fa640 1 -- 172.21.15.84:0/2207957927 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d44f38b8-67f4-43de-98a6-f3c520acaf10", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fbcd0006030 con 0x7fbcdc06eb70 2024-03-19T01:50:07.600 INFO:teuthology.orchestra.run.smithi084.stdout:2 2024-03-19T01:50:07.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.602+0000 7fbce2fa7640 1 -- 172.21.15.84:0/2207957927 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbcdc06eb70 msgr2=0x7fbcdc1e3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:07.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.602+0000 7fbce2fa7640 1 --2- 172.21.15.84:0/2207957927 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbcdc06eb70 0x7fbcdc1e3100 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000e960 tx=0x7fbcd000ee20 comp rx=0 tx=0).stop 2024-03-19T01:50:07.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.602+0000 7fbce2fa7640 1 -- 172.21.15.84:0/2207957927 shutdown_connections 2024-03-19T01:50:07.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.602+0000 7fbce2fa7640 1 --2- 172.21.15.84:0/2207957927 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbcdc13de70 0x7fbcdc14ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:07.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.602+0000 7fbce2fa7640 1 --2- 172.21.15.84:0/2207957927 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbcdc06f1b0 0x7fbcdc14fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:07.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.602+0000 7fbce2fa7640 1 --2- 172.21.15.84:0/2207957927 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbcdc06eb70 0x7fbcdc1e3100 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:07.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.602+0000 7fbce2fa7640 1 -- 172.21.15.84:0/2207957927 >> 172.21.15.84:0/2207957927 conn(0x7fbcdc06fb40 msgr2=0x7fbcdc139ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:07.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.602+0000 7fbce2fa7640 1 -- 172.21.15.84:0/2207957927 shutdown_connections 2024-03-19T01:50:07.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.602+0000 7fbce2fa7640 1 -- 172.21.15.84:0/2207957927 wait complete. 2024-03-19T01:50:07.621 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd new 850a7316-2d74-43f0-8649-b5f8d8f6692e 3 2024-03-19T01:50:07.656 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:07.655+0000 7f2e680a7180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T01:50:07.722 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:07.722+0000 7fcb1fd6c180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T01:50:07.742 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.742+0000 7f34a4e4c640 1 Processor -- start 2024-03-19T01:50:07.742 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.742+0000 7f34a4e4c640 1 -- start start 2024-03-19T01:50:07.743 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.746+0000 7f34a4e4c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f34a0152120 0x7f34a00c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:07.743 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.746+0000 7f34a4e4c640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34a006fff0 con 0x7f34a00c9610 2024-03-19T01:50:07.744 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.746+0000 7f34a4e4c640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34a00c39c0 con 0x7f34a0070780 2024-03-19T01:50:07.744 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.746+0000 7f34a4e4c640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f34a0152500 con 0x7f34a0152120 2024-03-19T01:50:07.744 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.746+0000 7f349ed9d640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f34a0070780 0x7f34a01586e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:35652/0 (socket says 172.21.15.84:35652) 2024-03-19T01:50:07.744 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.746+0000 7f349ed9d640 1 -- 172.21.15.84:0/1780603108 learned_addr learned my addr 172.21.15.84:0/1780603108 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-19T01:50:07.744 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.746+0000 7f349e59c640 1 --2- 172.21.15.84:0/1780603108 >> v2:172.21.15.55:3300/0 conn(0x7f34a0152120 0x7f34a00c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:07.746 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.746+0000 7f349d59a640 1 -- 172.21.15.84:0/1780603108 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4171509658 0 0) 0x7f34a00c39c0 con 0x7f34a0070780 2024-03-19T01:50:07.746 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.746+0000 7f349d59a640 1 -- 172.21.15.84:0/1780603108 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3488003530 con 0x7f34a0070780 2024-03-19T01:50:07.751 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.750+0000 7f349d59a640 1 -- 172.21.15.84:0/1780603108 <== mon.1 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1052566618 0 0) 0x7f3488003530 con 0x7f34a0070780 2024-03-19T01:50:07.751 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.750+0000 7f349d59a640 1 -- 172.21.15.84:0/1780603108 >> v2:172.21.15.55:3300/0 conn(0x7f34a0152120 msgr2=0x7f34a00c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:07.751 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.750+0000 7f349d59a640 1 --2- 172.21.15.84:0/1780603108 >> v2:172.21.15.55:3300/0 conn(0x7f34a0152120 0x7f34a00c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:07.751 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.750+0000 7f349d59a640 1 -- 172.21.15.84:0/1780603108 >> v1:172.21.15.55:6789/0 conn(0x7f34a00c9610 legacy=0x7f34a0157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:07.751 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.750+0000 7f349d59a640 1 -- 172.21.15.84:0/1780603108 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34a00702b0 con 0x7f34a0070780 2024-03-19T01:50:07.751 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.750+0000 7f349d59a640 1 -- 172.21.15.84:0/1780603108 <== mon.1 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f3498001a00 con 0x7f34a0070780 2024-03-19T01:50:07.751 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.750+0000 7f349d59a640 1 -- 172.21.15.84:0/1780603108 >> v1:172.21.15.84:6789/0 conn(0x7f34a0070780 legacy=0x7f34a01586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:07.751 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.750+0000 7f349d59a640 1 --2- 172.21.15.84:0/1780603108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3488003a10 0x7f3488003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:07.751 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.750+0000 7f349d59a640 1 --2- 172.21.15.84:0/1780603108 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3488004810 0x7f3488004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:07.751 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.750+0000 7f349d59a640 1 --2- 172.21.15.84:0/1780603108 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f34880051d0 0x7f348800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:07.751 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.750+0000 7f349d59a640 1 -- 172.21.15.84:0/1780603108 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f348800c0a0 con 0x7f3488003a10 2024-03-19T01:50:07.751 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.750+0000 7f349d59a640 1 -- 172.21.15.84:0/1780603108 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f348800c220 con 0x7f3488004810 2024-03-19T01:50:07.751 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.750+0000 7f349d59a640 1 -- 172.21.15.84:0/1780603108 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f348800c420 con 0x7f34880051d0 2024-03-19T01:50:07.752 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.754+0000 7f349ed9d640 1 --2- 172.21.15.84:0/1780603108 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f34880051d0 0x7f348800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:07.753 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.754+0000 7f349e59c640 1 --2- 172.21.15.84:0/1780603108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3488003a10 0x7f3488003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:07.753 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.754+0000 7f349dd9b640 1 --2- 172.21.15.84:0/1780603108 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3488004810 0x7f3488004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:07.754 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.754+0000 7f349ed9d640 1 -- 172.21.15.84:0/1780603108 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3488004810 msgr2=0x7f3488004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:07.754 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.754+0000 7f349ed9d640 1 --2- 172.21.15.84:0/1780603108 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3488004810 0x7f3488004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:07.754 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.754+0000 7f349ed9d640 1 -- 172.21.15.84:0/1780603108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3488003a10 msgr2=0x7f3488003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:07.754 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.754+0000 7f349ed9d640 1 --2- 172.21.15.84:0/1780603108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3488003a10 0x7f3488003e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:07.754 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.754+0000 7f349ed9d640 1 -- 172.21.15.84:0/1780603108 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3488003530 con 0x7f34880051d0 2024-03-19T01:50:07.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.758+0000 7f349ed9d640 1 -- 172.21.15.84:0/1780603108 reap_dead start 2024-03-19T01:50:07.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.758+0000 7f349ed9d640 1 --2- 172.21.15.84:0/1780603108 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f34880051d0 0x7f348800b9d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f34a0070c30 tx=0x7f349801d2a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd7ec45dc523fecd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:07.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.762+0000 7f349d59a640 1 -- 172.21.15.84:0/1780603108 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349800a3f0 con 0x7f34880051d0 2024-03-19T01:50:07.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.762+0000 7f349d59a640 1 -- 172.21.15.84:0/1780603108 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34980ad650 con 0x7f34880051d0 2024-03-19T01:50:07.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.762+0000 7f349d59a640 1 -- 172.21.15.84:0/1780603108 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34980ada40 con 0x7f34880051d0 2024-03-19T01:50:07.763 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.762+0000 7f34a4e4c640 1 -- 172.21.15.84:0/1780603108 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f34880051d0 msgr2=0x7f348800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:07.763 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.762+0000 7f34a4e4c640 1 --2- 172.21.15.84:0/1780603108 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f34880051d0 0x7f348800b9d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f34a0070c30 tx=0x7f349801d2a0 comp rx=0 tx=0).stop 2024-03-19T01:50:07.763 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.762+0000 7f34a4e4c640 1 -- 172.21.15.84:0/1780603108 shutdown_connections 2024-03-19T01:50:07.763 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.762+0000 7f34a4e4c640 1 --2- 172.21.15.84:0/1780603108 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f34880051d0 0x7f348800b9d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:07.763 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.762+0000 7f34a4e4c640 1 -- 172.21.15.84:0/1780603108 >> 172.21.15.84:0/1780603108 conn(0x7f34a0067600 msgr2=0x7f34a0071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:07.763 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.762+0000 7f34a4e4c640 1 -- 172.21.15.84:0/1780603108 shutdown_connections 2024-03-19T01:50:07.763 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.762+0000 7f34a4e4c640 1 -- 172.21.15.84:0/1780603108 wait complete. 2024-03-19T01:50:07.764 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.766+0000 7f34a4e4c640 1 Processor -- start 2024-03-19T01:50:07.764 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.766+0000 7f34a4e4c640 1 -- start start 2024-03-19T01:50:07.764 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.766+0000 7f34a4e4c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f34880051d0 0x7f34a01f08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:07.764 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.766+0000 7f34a4e4c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34a0154d50 0x7f34a0159190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:07.765 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.766+0000 7f34a4e4c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34a01557a0 0x7f34a015bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:07.765 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.766+0000 7f34a4e4c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f34a00c86d0 con 0x7f34a01557a0 2024-03-19T01:50:07.765 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.766+0000 7f34a4e4c640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f34a00c3530 con 0x7f34a0154d50 2024-03-19T01:50:07.765 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.766+0000 7f34a4e4c640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f34a00c90d0 con 0x7f34880051d0 2024-03-19T01:50:07.765 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.766+0000 7f349e59c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f34880051d0 0x7f34a01f08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:07.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.766+0000 7f349dd9b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34a0154d50 0x7f34a0159190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:07.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.766+0000 7f349e59c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f34880051d0 0x7f34a01f08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.84:46880/0 (socket says 172.21.15.84:46880) 2024-03-19T01:50:07.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.766+0000 7f349e59c640 1 -- 172.21.15.84:0/3887286871 learned_addr learned my addr 172.21.15.84:0/3887286871 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-03-19T01:50:07.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.766+0000 7f349ed9d640 1 --2- 172.21.15.84:0/3887286871 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34a01557a0 0x7f34a015bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:07.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.770+0000 7f349e59c640 1 -- 172.21.15.84:0/3887286871 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34a0154d50 msgr2=0x7f34a0159190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:07.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.770+0000 7f349e59c640 1 --2- 172.21.15.84:0/3887286871 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34a0154d50 0x7f34a0159190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:07.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.770+0000 7f349e59c640 1 -- 172.21.15.84:0/3887286871 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34a01557a0 msgr2=0x7f34a015bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:07.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.770+0000 7f349e59c640 1 --2- 172.21.15.84:0/3887286871 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34a01557a0 0x7f34a015bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:07.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.770+0000 7f349e59c640 1 -- 172.21.15.84:0/3887286871 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34a01539d0 con 0x7f34880051d0 2024-03-19T01:50:07.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.774+0000 7f349e59c640 1 --2- 172.21.15.84:0/3887286871 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f34880051d0 0x7f34a01f08c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f34900108c0 tx=0x7f3490010d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:07.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.778+0000 7f348effd640 1 -- 172.21.15.84:0/3887286871 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349000f040 con 0x7f34880051d0 2024-03-19T01:50:07.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.778+0000 7f348effd640 1 -- 172.21.15.84:0/3887286871 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f349000eb60 con 0x7f34880051d0 2024-03-19T01:50:07.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.778+0000 7f348effd640 1 -- 172.21.15.84:0/3887286871 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349001c400 con 0x7f34880051d0 2024-03-19T01:50:07.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.778+0000 7f34a4e4c640 1 -- 172.21.15.84:0/3887286871 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34a0153cb0 con 0x7f34880051d0 2024-03-19T01:50:07.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.778+0000 7f34a4e4c640 1 -- 172.21.15.84:0/3887286871 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f34a0154270 con 0x7f34880051d0 2024-03-19T01:50:07.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.778+0000 7f34a4e4c640 1 -- 172.21.15.84:0/3887286871 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34a0068a40 con 0x7f34880051d0 2024-03-19T01:50:07.778 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.778+0000 7f348effd640 1 -- 172.21.15.84:0/3887286871 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f34900059a0 con 0x7f34880051d0 2024-03-19T01:50:07.778 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.778+0000 7f348effd640 1 -- 172.21.15.84:0/3887286871 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f349001ca30 con 0x7f34880051d0 2024-03-19T01:50:07.792 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:07.794+0000 7fcb1fd6c180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T01:50:07.826 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:07.827+0000 7f2e680a7180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T01:50:07.892 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:07.891+0000 7f2e680a7180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T01:50:07.898 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.898+0000 7f348effd640 1 -- 172.21.15.84:0/3887286871 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3490018030 con 0x7f34880051d0 2024-03-19T01:50:07.926 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:07.926+0000 7fcb1fd6c180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T01:50:07.974 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:07.974+0000 7f34a4e4c640 1 -- 172.21.15.84:0/3887286871 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "osd new", "uuid": "850a7316-2d74-43f0-8649-b5f8d8f6692e", "id": 3} v 0) -- 0x7f34a00c84e0 con 0x7f34880051d0 2024-03-19T01:50:08.031 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:08.031+0000 7f2e680a7180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T01:50:08.134 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:50:08.137 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:50:08.169 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.170+0000 7f348effd640 1 -- 172.21.15.84:0/3887286871 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "850a7316-2d74-43f0-8649-b5f8d8f6692e", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f349001cc30 con 0x7f34880051d0 2024-03-19T01:50:08.170 INFO:teuthology.orchestra.run.smithi084.stdout:3 2024-03-19T01:50:08.171 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.170+0000 7f34a4e4c640 1 -- 172.21.15.84:0/3887286871 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f34880051d0 msgr2=0x7f34a01f08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:08.171 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.170+0000 7f34a4e4c640 1 --2- 172.21.15.84:0/3887286871 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f34880051d0 0x7f34a01f08c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f34900108c0 tx=0x7f3490010d80 comp rx=0 tx=0).stop 2024-03-19T01:50:08.171 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.170+0000 7f34a4e4c640 1 -- 172.21.15.84:0/3887286871 shutdown_connections 2024-03-19T01:50:08.171 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.170+0000 7f34a4e4c640 1 --2- 172.21.15.84:0/3887286871 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34a01557a0 0x7f34a015bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:08.171 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.170+0000 7f34a4e4c640 1 --2- 172.21.15.84:0/3887286871 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34a0154d50 0x7f34a0159190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:08.171 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.170+0000 7f34a4e4c640 1 --2- 172.21.15.84:0/3887286871 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f34880051d0 0x7f34a01f08c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:08.171 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.170+0000 7f34a4e4c640 1 -- 172.21.15.84:0/3887286871 >> 172.21.15.84:0/3887286871 conn(0x7f34a0067600 msgr2=0x7f34a006f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:08.171 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.170+0000 7f34a4e4c640 1 -- 172.21.15.84:0/3887286871 shutdown_connections 2024-03-19T01:50:08.171 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.170+0000 7f34a4e4c640 1 -- 172.21.15.84:0/3887286871 wait complete. 2024-03-19T01:50:08.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd new 4e6152ae-a77d-4bb6-a855-5782ad5411a5 4 2024-03-19T01:50:08.343 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.346+0000 7ff43f32a640 1 Processor -- start 2024-03-19T01:50:08.343 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.346+0000 7ff43f32a640 1 -- start start 2024-03-19T01:50:08.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.346+0000 7ff43f32a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff4380c6a20 0x7ff4380c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:08.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.346+0000 7ff43f32a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff43806c910 0x7ff4380c9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:08.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.346+0000 7ff43f32a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4380c8960 con 0x7ff438150ca0 2024-03-19T01:50:08.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.346+0000 7ff43f32a640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7ff43815acd0 con 0x7ff4380c6a20 2024-03-19T01:50:08.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.346+0000 7ff43f32a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff43815af90 con 0x7ff43806c910 2024-03-19T01:50:08.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.346+0000 7ff43d8c7640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff43806c910 0x7ff4380c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:08.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.346+0000 7ff43d0c6640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7ff438150ca0 0x7ff4380cbfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:35664/0 (socket says 172.21.15.84:35664) 2024-03-19T01:50:08.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.346+0000 7ff43d0c6640 1 -- 172.21.15.84:0/36454610 learned_addr learned my addr 172.21.15.84:0/36454610 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-19T01:50:08.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.346+0000 7ff43c8c5640 1 --2- 172.21.15.84:0/36454610 >> v2:172.21.15.84:3300/0 conn(0x7ff4380c6a20 0x7ff4380c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:08.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.350+0000 7ff43d8c7640 1 -- 172.21.15.84:0/36454610 >> v2:172.21.15.84:3300/0 conn(0x7ff4380c6a20 msgr2=0x7ff4380c8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:08.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.350+0000 7ff43d8c7640 1 --2- 172.21.15.84:0/36454610 >> v2:172.21.15.84:3300/0 conn(0x7ff4380c6a20 0x7ff4380c8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:08.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.350+0000 7ff43d8c7640 1 -- 172.21.15.84:0/36454610 >> v1:172.21.15.84:6789/0 conn(0x7ff438150ca0 legacy=0x7ff4380cbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:08.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.350+0000 7ff43d8c7640 1 -- 172.21.15.84:0/36454610 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43806b340 con 0x7ff43806c910 2024-03-19T01:50:08.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.354+0000 7ff43d8c7640 1 --2- 172.21.15.84:0/36454610 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff43806c910 0x7ff4380c9350 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff430009d90 tx=0x7ff43001f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dd4fe06fd905975 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:08.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.358+0000 7ff42ffff640 1 -- 172.21.15.84:0/36454610 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4300b4070 con 0x7ff43806c910 2024-03-19T01:50:08.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.358+0000 7ff42ffff640 1 -- 172.21.15.84:0/36454610 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4300af600 con 0x7ff43806c910 2024-03-19T01:50:08.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.358+0000 7ff42ffff640 1 -- 172.21.15.84:0/36454610 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4300af9d0 con 0x7ff43806c910 2024-03-19T01:50:08.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.358+0000 7ff43f32a640 1 -- 172.21.15.84:0/36454610 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff43806c910 msgr2=0x7ff4380c9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:08.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.358+0000 7ff43f32a640 1 --2- 172.21.15.84:0/36454610 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff43806c910 0x7ff4380c9350 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff430009d90 tx=0x7ff43001f150 comp rx=0 tx=0).stop 2024-03-19T01:50:08.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.358+0000 7ff43f32a640 1 -- 172.21.15.84:0/36454610 shutdown_connections 2024-03-19T01:50:08.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.358+0000 7ff43f32a640 1 --2- 172.21.15.84:0/36454610 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff43806c910 0x7ff4380c9350 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:08.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.358+0000 7ff43f32a640 1 --2- 172.21.15.84:0/36454610 >> v2:172.21.15.84:3300/0 conn(0x7ff4380c6a20 0x7ff4380c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:08.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.358+0000 7ff43f32a640 1 -- 172.21.15.84:0/36454610 >> 172.21.15.84:0/36454610 conn(0x7ff43806e5a0 msgr2=0x7ff438072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:08.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.358+0000 7ff43f32a640 1 -- 172.21.15.84:0/36454610 shutdown_connections 2024-03-19T01:50:08.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.358+0000 7ff43f32a640 1 -- 172.21.15.84:0/36454610 wait complete. 2024-03-19T01:50:08.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.362+0000 7ff43f32a640 1 Processor -- start 2024-03-19T01:50:08.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.362+0000 7ff43f32a640 1 -- start start 2024-03-19T01:50:08.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.362+0000 7ff43f32a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff43806c910 0x7ff438159470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:08.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.362+0000 7ff43f32a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff4380c6a20 0x7ff4380cb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:08.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.362+0000 7ff43f32a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff438150ca0 0x7ff4380cb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:08.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.362+0000 7ff43f32a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff43806b900 con 0x7ff43806c910 2024-03-19T01:50:08.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.362+0000 7ff43f32a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff4380670e0 con 0x7ff438150ca0 2024-03-19T01:50:08.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.362+0000 7ff43f32a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff4380cc6c0 con 0x7ff4380c6a20 2024-03-19T01:50:08.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.362+0000 7ff43d8c7640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff438150ca0 0x7ff4380cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:08.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.362+0000 7ff43d8c7640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff438150ca0 0x7ff4380cb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:50990/0 (socket says 172.21.15.84:50990) 2024-03-19T01:50:08.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.362+0000 7ff43d8c7640 1 -- 172.21.15.84:0/1243261951 learned_addr learned my addr 172.21.15.84:0/1243261951 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-03-19T01:50:08.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.362+0000 7ff43c8c5640 1 --2- 172.21.15.84:0/1243261951 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff4380c6a20 0x7ff4380cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:08.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.366+0000 7ff43d8c7640 1 -- 172.21.15.84:0/1243261951 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff4380c6a20 msgr2=0x7ff4380cb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:08.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.366+0000 7ff43d8c7640 1 --2- 172.21.15.84:0/1243261951 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff4380c6a20 0x7ff4380cb250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:08.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.366+0000 7ff43d8c7640 1 -- 172.21.15.84:0/1243261951 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff43806c910 msgr2=0x7ff438159470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:08.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.366+0000 7ff43d8c7640 1 --2- 172.21.15.84:0/1243261951 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff43806c910 0x7ff438159470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:08.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.366+0000 7ff43d8c7640 1 -- 172.21.15.84:0/1243261951 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43814a630 con 0x7ff438150ca0 2024-03-19T01:50:08.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.366+0000 7ff43d8c7640 1 --2- 172.21.15.84:0/1243261951 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff438150ca0 0x7ff4380cb790 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff4300b2040 tx=0x7ff43001bef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:08.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.370+0000 7ff42dffb640 1 -- 172.21.15.84:0/1243261951 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4300b4040 con 0x7ff438150ca0 2024-03-19T01:50:08.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.370+0000 7ff42dffb640 1 -- 172.21.15.84:0/1243261951 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4300c1070 con 0x7ff438150ca0 2024-03-19T01:50:08.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.370+0000 7ff43f32a640 1 -- 172.21.15.84:0/1243261951 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff43815b8e0 con 0x7ff438150ca0 2024-03-19T01:50:08.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.370+0000 7ff42dffb640 1 -- 172.21.15.84:0/1243261951 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff430009450 con 0x7ff438150ca0 2024-03-19T01:50:08.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.370+0000 7ff43f32a640 1 -- 172.21.15.84:0/1243261951 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4381f0b90 con 0x7ff438150ca0 2024-03-19T01:50:08.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.374+0000 7ff43f32a640 1 -- 172.21.15.84:0/1243261951 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff438068a40 con 0x7ff438150ca0 2024-03-19T01:50:08.482 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.482+0000 7ff42dffb640 1 -- 172.21.15.84:0/1243261951 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff4300ba6b0 con 0x7ff438150ca0 2024-03-19T01:50:08.482 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.482+0000 7ff42dffb640 1 -- 172.21.15.84:0/1243261951 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7ff4300c6b50 con 0x7ff438150ca0 2024-03-19T01:50:08.482 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.482+0000 7ff42dffb640 1 -- 172.21.15.84:0/1243261951 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff430018cb0 con 0x7ff438150ca0 2024-03-19T01:50:08.560 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.562+0000 7ff43f32a640 1 -- 172.21.15.84:0/1243261951 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4e6152ae-a77d-4bb6-a855-5782ad5411a5", "id": 4} v 0) -- 0x7ff438074d80 con 0x7ff438150ca0 2024-03-19T01:50:08.566 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:08.566+0000 7fcb1fd6c180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T01:50:08.643 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:08.642+0000 7fcb1fd6c180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T01:50:08.685 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:08.683+0000 7f2e680a7180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T01:50:08.765 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:08.763+0000 7f2e680a7180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T01:50:08.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.802+0000 7ff42dffb640 1 -- 172.21.15.84:0/1243261951 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4e6152ae-a77d-4bb6-a855-5782ad5411a5", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7ff4300c6640 con 0x7ff438150ca0 2024-03-19T01:50:08.802 INFO:teuthology.orchestra.run.smithi084.stdout:4 2024-03-19T01:50:08.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.806+0000 7ff43f32a640 1 -- 172.21.15.84:0/1243261951 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff438150ca0 msgr2=0x7ff4380cb790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:08.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.806+0000 7ff43f32a640 1 --2- 172.21.15.84:0/1243261951 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff438150ca0 0x7ff4380cb790 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff4300b2040 tx=0x7ff43001bef0 comp rx=0 tx=0).stop 2024-03-19T01:50:08.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.806+0000 7ff43f32a640 1 -- 172.21.15.84:0/1243261951 shutdown_connections 2024-03-19T01:50:08.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.806+0000 7ff43f32a640 1 --2- 172.21.15.84:0/1243261951 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff438150ca0 0x7ff4380cb790 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:08.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.806+0000 7ff43f32a640 1 --2- 172.21.15.84:0/1243261951 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff4380c6a20 0x7ff4380cb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:08.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.806+0000 7ff43f32a640 1 --2- 172.21.15.84:0/1243261951 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff43806c910 0x7ff438159470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:08.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.806+0000 7ff43f32a640 1 -- 172.21.15.84:0/1243261951 >> 172.21.15.84:0/1243261951 conn(0x7ff43806e5a0 msgr2=0x7ff43814f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:08.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.806+0000 7ff43f32a640 1 -- 172.21.15.84:0/1243261951 shutdown_connections 2024-03-19T01:50:08.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.806+0000 7ff43f32a640 1 -- 172.21.15.84:0/1243261951 wait complete. 2024-03-19T01:50:08.825 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd new 1217ceba-dd31-4615-96f1-dfaf57160f3e 5 2024-03-19T01:50:08.828 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:08.830+0000 7fcb1fd6c180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T01:50:08.903 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:08.906+0000 7fcb1fd6c180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T01:50:08.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.942+0000 7fa399cd5640 1 Processor -- start 2024-03-19T01:50:08.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.942+0000 7fa399cd5640 1 -- start start 2024-03-19T01:50:08.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.942+0000 7fa399cd5640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa3941510f0 0x7fa3940c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:08.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.942+0000 7fa399cd5640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa3940c73c0 0x7fa3940c7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:08.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.942+0000 7fa399cd5640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3940c3f60 con 0x7fa39415a410 2024-03-19T01:50:08.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.942+0000 7fa399cd5640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fa394070fb0 con 0x7fa3941510f0 2024-03-19T01:50:08.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.942+0000 7fa399cd5640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa3941514d0 con 0x7fa3940c73c0 2024-03-19T01:50:08.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.946+0000 7fa393fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa3940c73c0 0x7fa3940c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:08.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.946+0000 7fa3937fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa3941510f0 0x7fa3940c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:08.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.946+0000 7fa38bfff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fa39415a410 0x7fa3940708a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:35674/0 (socket says 172.21.15.84:35674) 2024-03-19T01:50:08.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.946+0000 7fa38bfff640 1 -- 172.21.15.84:0/1803324362 learned_addr learned my addr 172.21.15.84:0/1803324362 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-19T01:50:08.946 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.946+0000 7fa393fff640 1 -- 172.21.15.84:0/1803324362 >> v2:172.21.15.84:3300/0 conn(0x7fa3941510f0 msgr2=0x7fa3940c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:08.946 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.946+0000 7fa393fff640 1 --2- 172.21.15.84:0/1803324362 >> v2:172.21.15.84:3300/0 conn(0x7fa3941510f0 0x7fa3940c61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:08.946 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.946+0000 7fa393fff640 1 -- 172.21.15.84:0/1803324362 >> v1:172.21.15.84:6789/0 conn(0x7fa39415a410 legacy=0x7fa3940708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:08.947 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.946+0000 7fa393fff640 1 -- 172.21.15.84:0/1803324362 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3940c6870 con 0x7fa3940c73c0 2024-03-19T01:50:08.950 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.950+0000 7fa393fff640 1 --2- 172.21.15.84:0/1803324362 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa3940c73c0 0x7fa3940c7830 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa38c009d90 tx=0x7fa38c01f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8e0fbec31f5f4c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:08.954 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:08.955+0000 7f2e680a7180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T01:50:08.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.954+0000 7fa392ffd640 1 -- 172.21.15.84:0/1803324362 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa38c0b4070 con 0x7fa3940c73c0 2024-03-19T01:50:08.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa392ffd640 1 -- 172.21.15.84:0/1803324362 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa38c0af600 con 0x7fa3940c73c0 2024-03-19T01:50:08.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa392ffd640 1 -- 172.21.15.84:0/1803324362 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa38c0af9d0 con 0x7fa3940c73c0 2024-03-19T01:50:08.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa399cd5640 1 -- 172.21.15.84:0/1803324362 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa3940c73c0 msgr2=0x7fa3940c7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:08.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa399cd5640 1 --2- 172.21.15.84:0/1803324362 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa3940c73c0 0x7fa3940c7830 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa38c009d90 tx=0x7fa38c01f150 comp rx=0 tx=0).stop 2024-03-19T01:50:08.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa399cd5640 1 -- 172.21.15.84:0/1803324362 shutdown_connections 2024-03-19T01:50:08.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa399cd5640 1 --2- 172.21.15.84:0/1803324362 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa3940c73c0 0x7fa3940c7830 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:08.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa399cd5640 1 --2- 172.21.15.84:0/1803324362 >> v2:172.21.15.84:3300/0 conn(0x7fa3941510f0 0x7fa3940c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:08.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa399cd5640 1 -- 172.21.15.84:0/1803324362 >> 172.21.15.84:0/1803324362 conn(0x7fa3940cc9c0 msgr2=0x7fa3940ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:08.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa399cd5640 1 -- 172.21.15.84:0/1803324362 shutdown_connections 2024-03-19T01:50:08.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa399cd5640 1 -- 172.21.15.84:0/1803324362 wait complete. 2024-03-19T01:50:08.957 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa399cd5640 1 Processor -- start 2024-03-19T01:50:08.957 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa399cd5640 1 -- start start 2024-03-19T01:50:08.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa399cd5640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa3940c73c0 0x7fa3941f1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:08.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa399cd5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3941510f0 0x7fa3941edca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:08.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa399cd5640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa39415a410 0x7fa394142f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:08.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa399cd5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa3940c44e0 con 0x7fa3941510f0 2024-03-19T01:50:08.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa399cd5640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa3940c4200 con 0x7fa39415a410 2024-03-19T01:50:08.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa399cd5640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa3940c6e30 con 0x7fa3940c73c0 2024-03-19T01:50:08.959 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa393fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa39415a410 0x7fa394142f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:08.959 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa3937fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa3940c73c0 0x7fa3941f1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:08.959 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa393fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa39415a410 0x7fa394142f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:51008/0 (socket says 172.21.15.84:51008) 2024-03-19T01:50:08.959 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.958+0000 7fa393fff640 1 -- 172.21.15.84:0/128746145 learned_addr learned my addr 172.21.15.84:0/128746145 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-03-19T01:50:08.959 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.962+0000 7fa38bfff640 1 --2- 172.21.15.84:0/128746145 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3941510f0 0x7fa3941edca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:08.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.962+0000 7fa393fff640 1 -- 172.21.15.84:0/128746145 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa3940c73c0 msgr2=0x7fa3941f1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:08.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.962+0000 7fa393fff640 1 --2- 172.21.15.84:0/128746145 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa3940c73c0 0x7fa3941f1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:08.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.962+0000 7fa393fff640 1 -- 172.21.15.84:0/128746145 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3941510f0 msgr2=0x7fa3941edca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:08.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.962+0000 7fa393fff640 1 --2- 172.21.15.84:0/128746145 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3941510f0 0x7fa3941edca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:08.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.962+0000 7fa393fff640 1 -- 172.21.15.84:0/128746145 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3941434b0 con 0x7fa39415a410 2024-03-19T01:50:08.964 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.966+0000 7fa393fff640 1 --2- 172.21.15.84:0/128746145 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa39415a410 0x7fa394142f70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa38c009d90 tx=0x7fa38c01bef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:08.968 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.970+0000 7fa390ff9640 1 -- 172.21.15.84:0/128746145 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa38c0b4040 con 0x7fa39415a410 2024-03-19T01:50:08.968 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.970+0000 7fa390ff9640 1 -- 172.21.15.84:0/128746145 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa38c0c1070 con 0x7fa39415a410 2024-03-19T01:50:08.968 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.970+0000 7fa390ff9640 1 -- 172.21.15.84:0/128746145 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa38c009450 con 0x7fa39415a410 2024-03-19T01:50:08.969 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.970+0000 7fa399cd5640 1 -- 172.21.15.84:0/128746145 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3941f12b0 con 0x7fa39415a410 2024-03-19T01:50:08.969 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.970+0000 7fa399cd5640 1 -- 172.21.15.84:0/128746145 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3941f1550 con 0x7fa39415a410 2024-03-19T01:50:08.969 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.970+0000 7fa399cd5640 1 -- 172.21.15.84:0/128746145 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa394068a40 con 0x7fa39415a410 2024-03-19T01:50:08.972 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.974+0000 7fa390ff9640 1 -- 172.21.15.84:0/128746145 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa38c0ba6b0 con 0x7fa39415a410 2024-03-19T01:50:08.972 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:08.974+0000 7fa390ff9640 1 -- 172.21.15.84:0/128746145 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fa38c0c6d30 con 0x7fa39415a410 2024-03-19T01:50:09.031 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:09.031+0000 7f2e680a7180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T01:50:09.088 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.090+0000 7fa390ff9640 1 -- 172.21.15.84:0/128746145 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa3941434b0 con 0x7fa39415a410 2024-03-19T01:50:09.166 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.166+0000 7fa399cd5640 1 -- 172.21.15.84:0/128746145 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1217ceba-dd31-4615-96f1-dfaf57160f3e", "id": 5} v 0) -- 0x7fa39414a4a0 con 0x7fa39415a410 2024-03-19T01:50:09.208 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:09.210+0000 7fcb1fd6c180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T01:50:09.271 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:09.270+0000 7fcb1fd6c180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T01:50:09.338 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:09.339+0000 7f2e680a7180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T01:50:09.402 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-19T01:50:09.403+0000 7f2e680a7180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T01:50:09.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.434+0000 7fa390ff9640 1 -- 172.21.15.84:0/128746145 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1217ceba-dd31-4615-96f1-dfaf57160f3e", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fa38c018cb0 con 0x7fa39415a410 2024-03-19T01:50:09.434 INFO:teuthology.orchestra.run.smithi084.stdout:5 2024-03-19T01:50:09.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.434+0000 7fa399cd5640 1 -- 172.21.15.84:0/128746145 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa39415a410 msgr2=0x7fa394142f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:09.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.434+0000 7fa399cd5640 1 --2- 172.21.15.84:0/128746145 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa39415a410 0x7fa394142f70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa38c009d90 tx=0x7fa38c01bef0 comp rx=0 tx=0).stop 2024-03-19T01:50:09.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.434+0000 7fa399cd5640 1 -- 172.21.15.84:0/128746145 shutdown_connections 2024-03-19T01:50:09.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.434+0000 7fa399cd5640 1 --2- 172.21.15.84:0/128746145 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa39415a410 0x7fa394142f70 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:09.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.434+0000 7fa399cd5640 1 --2- 172.21.15.84:0/128746145 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3941510f0 0x7fa3941edca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:09.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.434+0000 7fa399cd5640 1 --2- 172.21.15.84:0/128746145 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa3940c73c0 0x7fa3941f1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:09.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.438+0000 7fa399cd5640 1 -- 172.21.15.84:0/128746145 >> 172.21.15.84:0/128746145 conn(0x7fa3940cc9c0 msgr2=0x7fa39415a880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:09.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.438+0000 7fa399cd5640 1 -- 172.21.15.84:0/128746145 shutdown_connections 2024-03-19T01:50:09.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.438+0000 7fa399cd5640 1 -- 172.21.15.84:0/128746145 wait complete. 2024-03-19T01:50:09.456 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd new 3d9ed78a-2284-4c28-a24a-70eb812c7836 6 2024-03-19T01:50:09.627 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.630+0000 7ff1b2304640 1 Processor -- start 2024-03-19T01:50:09.627 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.630+0000 7ff1b2304640 1 -- start start 2024-03-19T01:50:09.628 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.630+0000 7ff1b2304640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff1ac15a540 0x7ff1ac0c9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:09.628 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.630+0000 7ff1b2304640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1ac070bd0 con 0x7ff1ac15a920 2024-03-19T01:50:09.628 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.630+0000 7ff1b2304640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1ac070e90 con 0x7ff1ac15ade0 2024-03-19T01:50:09.628 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.630+0000 7ff1b2304640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7ff1ac06d440 con 0x7ff1ac15a540 2024-03-19T01:50:09.629 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.630+0000 7ff1b08a1640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7ff1ac15ade0 0x7ff1ac1431b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:35676/0 (socket says 172.21.15.84:35676) 2024-03-19T01:50:09.629 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.630+0000 7ff1b08a1640 1 -- 172.21.15.84:0/3308745263 learned_addr learned my addr 172.21.15.84:0/3308745263 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-19T01:50:09.629 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.630+0000 7ff1abfff640 1 --2- 172.21.15.84:0/3308745263 >> v2:172.21.15.84:3300/0 conn(0x7ff1ac15a540 0x7ff1ac0c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:09.638 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.638+0000 7ff1abfff640 1 -- 172.21.15.84:0/3308745263 >> v1:172.21.15.84:6789/0 conn(0x7ff1ac15ade0 legacy=0x7ff1ac1431b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:09.638 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.638+0000 7ff1abfff640 1 -- 172.21.15.84:0/3308745263 >> v1:172.21.15.55:6789/0 conn(0x7ff1ac15a920 legacy=0x7ff1ac06cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:09.638 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.638+0000 7ff1abfff640 1 -- 172.21.15.84:0/3308745263 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ac06f830 con 0x7ff1ac15a540 2024-03-19T01:50:09.645 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.646+0000 7ff1abfff640 1 --2- 172.21.15.84:0/3308745263 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff1ac15a540 0x7ff1ac0c9480 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff19c0097c0 tx=0x7ff19c01ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aeb10cb218af81d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:09.650 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.650+0000 7ff1aaffd640 1 -- 172.21.15.84:0/3308745263 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c0b5070 con 0x7ff1ac15a540 2024-03-19T01:50:09.651 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.650+0000 7ff1aaffd640 1 -- 172.21.15.84:0/3308745263 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff19c0a8950 con 0x7ff1ac15a540 2024-03-19T01:50:09.651 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.650+0000 7ff1aaffd640 1 -- 172.21.15.84:0/3308745263 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c0a8d20 con 0x7ff1ac15a540 2024-03-19T01:50:09.651 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.654+0000 7ff1b2304640 1 -- 172.21.15.84:0/3308745263 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff1ac15a540 msgr2=0x7ff1ac0c9480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:09.652 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.654+0000 7ff1b2304640 1 --2- 172.21.15.84:0/3308745263 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff1ac15a540 0x7ff1ac0c9480 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff19c0097c0 tx=0x7ff19c01ec50 comp rx=0 tx=0).stop 2024-03-19T01:50:09.652 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.654+0000 7ff1b2304640 1 -- 172.21.15.84:0/3308745263 shutdown_connections 2024-03-19T01:50:09.652 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.654+0000 7ff1b2304640 1 --2- 172.21.15.84:0/3308745263 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff1ac15a540 0x7ff1ac0c9480 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:09.652 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.654+0000 7ff1b2304640 1 -- 172.21.15.84:0/3308745263 >> 172.21.15.84:0/3308745263 conn(0x7ff1ac067600 msgr2=0x7ff1ac071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:09.652 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.654+0000 7ff1b2304640 1 -- 172.21.15.84:0/3308745263 shutdown_connections 2024-03-19T01:50:09.652 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.654+0000 7ff1b2304640 1 -- 172.21.15.84:0/3308745263 wait complete. 2024-03-19T01:50:09.652 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.654+0000 7ff1b2304640 1 Processor -- start 2024-03-19T01:50:09.653 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.654+0000 7ff1b2304640 1 -- start start 2024-03-19T01:50:09.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.654+0000 7ff1b2304640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff1ac15a540 0x7ff1ac157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:09.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.654+0000 7ff1b2304640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff1ac15a920 0x7ff1ac0c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:09.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.654+0000 7ff1b2304640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1ac15ade0 0x7ff1ac155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:09.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.654+0000 7ff1b2304640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac06fdf0 con 0x7ff1ac15ade0 2024-03-19T01:50:09.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.654+0000 7ff1b2304640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac06f5e0 con 0x7ff1ac15a540 2024-03-19T01:50:09.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.654+0000 7ff1b2304640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff1ac070860 con 0x7ff1ac15a920 2024-03-19T01:50:09.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.654+0000 7ff1abfff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff1ac15a540 0x7ff1ac157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:09.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.654+0000 7ff1ab7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff1ac15a920 0x7ff1ac0c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:09.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.654+0000 7ff1abfff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff1ac15a540 0x7ff1ac157600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:51022/0 (socket says 172.21.15.84:51022) 2024-03-19T01:50:09.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.654+0000 7ff1abfff640 1 -- 172.21.15.84:0/903488900 learned_addr learned my addr 172.21.15.84:0/903488900 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-03-19T01:50:09.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.658+0000 7ff1abfff640 1 -- 172.21.15.84:0/903488900 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff1ac15a920 msgr2=0x7ff1ac0c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:09.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.658+0000 7ff1abfff640 1 --2- 172.21.15.84:0/903488900 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff1ac15a920 0x7ff1ac0c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:09.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.658+0000 7ff1abfff640 1 -- 172.21.15.84:0/903488900 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1ac15ade0 msgr2=0x7ff1ac155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:09.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.658+0000 7ff1abfff640 1 --2- 172.21.15.84:0/903488900 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1ac15ade0 0x7ff1ac155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:09.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.658+0000 7ff1abfff640 1 -- 172.21.15.84:0/903488900 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ac1422d0 con 0x7ff1ac15a540 2024-03-19T01:50:09.660 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.662+0000 7ff1abfff640 1 --2- 172.21.15.84:0/903488900 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff1ac15a540 0x7ff1ac157600 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff19c01c880 tx=0x7ff19c01b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:09.664 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.666+0000 7ff1a8ff9640 1 -- 172.21.15.84:0/903488900 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c0b5040 con 0x7ff1ac15a540 2024-03-19T01:50:09.664 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.666+0000 7ff1a8ff9640 1 -- 172.21.15.84:0/903488900 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff19c0c0070 con 0x7ff1ac15a540 2024-03-19T01:50:09.665 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.666+0000 7ff1b2304640 1 -- 172.21.15.84:0/903488900 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1ac0cd400 con 0x7ff1ac15a540 2024-03-19T01:50:09.665 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.666+0000 7ff1a8ff9640 1 -- 172.21.15.84:0/903488900 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c0b1b10 con 0x7ff1ac15a540 2024-03-19T01:50:09.665 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.666+0000 7ff1b2304640 1 -- 172.21.15.84:0/903488900 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1ac1f3a20 con 0x7ff1ac15a540 2024-03-19T01:50:09.665 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.666+0000 7ff1b2304640 1 -- 172.21.15.84:0/903488900 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1ac068a40 con 0x7ff1ac15a540 2024-03-19T01:50:09.668 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.670+0000 7ff1a8ff9640 1 -- 172.21.15.84:0/903488900 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff19c0b1cb0 con 0x7ff1ac15a540 2024-03-19T01:50:09.668 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.670+0000 7ff1a8ff9640 1 -- 172.21.15.84:0/903488900 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7ff19c0c5dc0 con 0x7ff1ac15a540 2024-03-19T01:50:09.819 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.818+0000 7ff1a8ff9640 1 -- 172.21.15.84:0/903488900 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff19c0ca020 con 0x7ff1ac15a540 2024-03-19T01:50:09.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:09.894+0000 7ff1b2304640 1 -- 172.21.15.84:0/903488900 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3d9ed78a-2284-4c28-a24a-70eb812c7836", "id": 6} v 0) -- 0x7ff1ac141ff0 con 0x7ff1ac15a540 2024-03-19T01:50:10.328 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.330+0000 7ff1a8ff9640 1 -- 172.21.15.84:0/903488900 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mgrmap(e 2) ==== 43583+0+0 (secure 0 0 0) 0x7ff19c0c5460 con 0x7ff1ac15a540 2024-03-19T01:50:10.707 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.706+0000 7ff1a8ff9640 1 -- 172.21.15.84:0/903488900 <== mon.1 v2:172.21.15.84:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3d9ed78a-2284-4c28-a24a-70eb812c7836", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7ff19c0187b0 con 0x7ff1ac15a540 2024-03-19T01:50:10.707 INFO:teuthology.orchestra.run.smithi084.stdout:6 2024-03-19T01:50:10.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.710+0000 7ff1b2304640 1 -- 172.21.15.84:0/903488900 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff1ac15a540 msgr2=0x7ff1ac157600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:10.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.710+0000 7ff1b2304640 1 --2- 172.21.15.84:0/903488900 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff1ac15a540 0x7ff1ac157600 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff19c01c880 tx=0x7ff19c01b9f0 comp rx=0 tx=0).stop 2024-03-19T01:50:10.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.710+0000 7ff1b2304640 1 -- 172.21.15.84:0/903488900 shutdown_connections 2024-03-19T01:50:10.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.710+0000 7ff1b2304640 1 --2- 172.21.15.84:0/903488900 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1ac15ade0 0x7ff1ac155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:10.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.710+0000 7ff1b2304640 1 --2- 172.21.15.84:0/903488900 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff1ac15a920 0x7ff1ac0c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:10.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.710+0000 7ff1b2304640 1 --2- 172.21.15.84:0/903488900 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff1ac15a540 0x7ff1ac157600 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:10.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.710+0000 7ff1b2304640 1 -- 172.21.15.84:0/903488900 >> 172.21.15.84:0/903488900 conn(0x7ff1ac067600 msgr2=0x7ff1ac0cb970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:10.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.710+0000 7ff1b2304640 1 -- 172.21.15.84:0/903488900 shutdown_connections 2024-03-19T01:50:10.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.710+0000 7ff1b2304640 1 -- 172.21.15.84:0/903488900 wait complete. 2024-03-19T01:50:10.734 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd new c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc 7 2024-03-19T01:50:10.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.890+0000 7f25732a5640 1 Processor -- start 2024-03-19T01:50:10.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.890+0000 7f25732a5640 1 -- start start 2024-03-19T01:50:10.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.890+0000 7f25732a5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f256c152120 0x7f256c0c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:10.889 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.890+0000 7f25732a5640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f256c0c9610 0x7f256c157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:10.889 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.890+0000 7f25732a5640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f256c070780 0x7f256c158510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:10.889 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.890+0000 7f25732a5640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f256c152500 con 0x7f256c152120 2024-03-19T01:50:10.889 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.890+0000 7f25732a5640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f256c070590 con 0x7f256c0c9610 2024-03-19T01:50:10.889 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.890+0000 7f25732a5640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f256c0c3530 con 0x7f256c070780 2024-03-19T01:50:10.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.890+0000 7f2571842640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f256c070780 0x7f256c158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:10.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.890+0000 7f2570840640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f256c0c9610 0x7f256c157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:10.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.890+0000 7f2571842640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f256c070780 0x7f256c158510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.84:59732/0 (socket says 172.21.15.84:59732) 2024-03-19T01:50:10.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.890+0000 7f2571842640 1 -- 172.21.15.84:0/4169799696 learned_addr learned my addr 172.21.15.84:0/4169799696 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-03-19T01:50:10.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.894+0000 7f2571842640 1 -- 172.21.15.84:0/4169799696 >> v2:172.21.15.84:3300/0 conn(0x7f256c0c9610 msgr2=0x7f256c157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:10.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.894+0000 7f2571842640 1 --2- 172.21.15.84:0/4169799696 >> v2:172.21.15.84:3300/0 conn(0x7f256c0c9610 0x7f256c157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:10.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.894+0000 7f2571842640 1 -- 172.21.15.84:0/4169799696 >> v2:172.21.15.55:3300/0 conn(0x7f256c152120 msgr2=0x7f256c0c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:10.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.894+0000 7f2571842640 1 --2- 172.21.15.84:0/4169799696 >> v2:172.21.15.55:3300/0 conn(0x7f256c152120 0x7f256c0c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:10.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.894+0000 7f2571842640 1 -- 172.21.15.84:0/4169799696 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f256c06fff0 con 0x7f256c070780 2024-03-19T01:50:10.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.898+0000 7f2571842640 1 --2- 172.21.15.84:0/4169799696 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f256c070780 0x7f256c158510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f256c0cb130 tx=0x7f256801c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15fbe49b40cbffa7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:10.901 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.902+0000 7f2563fff640 1 -- 172.21.15.84:0/4169799696 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25680a5670 con 0x7f256c070780 2024-03-19T01:50:10.901 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.902+0000 7f2563fff640 1 -- 172.21.15.84:0/4169799696 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25680a5810 con 0x7f256c070780 2024-03-19T01:50:10.901 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.902+0000 7f2563fff640 1 -- 172.21.15.84:0/4169799696 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25680a5be0 con 0x7f256c070780 2024-03-19T01:50:10.902 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.902+0000 7f25732a5640 1 -- 172.21.15.84:0/4169799696 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f256c070780 msgr2=0x7f256c158510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:10.902 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.902+0000 7f25732a5640 1 --2- 172.21.15.84:0/4169799696 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f256c070780 0x7f256c158510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f256c0cb130 tx=0x7f256801c9f0 comp rx=0 tx=0).stop 2024-03-19T01:50:10.903 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.902+0000 7f25732a5640 1 -- 172.21.15.84:0/4169799696 shutdown_connections 2024-03-19T01:50:10.903 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.902+0000 7f25732a5640 1 --2- 172.21.15.84:0/4169799696 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f256c070780 0x7f256c158510 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:10.903 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.902+0000 7f25732a5640 1 --2- 172.21.15.84:0/4169799696 >> v2:172.21.15.84:3300/0 conn(0x7f256c0c9610 0x7f256c157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:10.903 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.902+0000 7f25732a5640 1 --2- 172.21.15.84:0/4169799696 >> v2:172.21.15.55:3300/0 conn(0x7f256c152120 0x7f256c0c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:10.903 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.902+0000 7f25732a5640 1 -- 172.21.15.84:0/4169799696 >> 172.21.15.84:0/4169799696 conn(0x7f256c067600 msgr2=0x7f256c071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:10.903 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.902+0000 7f25732a5640 1 -- 172.21.15.84:0/4169799696 shutdown_connections 2024-03-19T01:50:10.903 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.902+0000 7f25732a5640 1 -- 172.21.15.84:0/4169799696 wait complete. 2024-03-19T01:50:10.903 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.906+0000 7f25732a5640 1 Processor -- start 2024-03-19T01:50:10.904 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.906+0000 7f25732a5640 1 -- start start 2024-03-19T01:50:10.905 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.906+0000 7f25732a5640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f256c070780 0x7f256c0c6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:10.905 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.906+0000 7f25732a5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f256c0c9610 0x7f256c157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:10.905 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.906+0000 7f25732a5640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f256c152120 0x7f256c1f69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:10.905 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.906+0000 7f25732a5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f256c0c31c0 con 0x7f256c0c9610 2024-03-19T01:50:10.905 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.906+0000 7f25732a5640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f256c0c2ee0 con 0x7f256c070780 2024-03-19T01:50:10.905 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.906+0000 7f25732a5640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f256c0c86d0 con 0x7f256c152120 2024-03-19T01:50:10.905 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.906+0000 7f2571842640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f256c152120 0x7f256c1f69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:10.905 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.906+0000 7f2571041640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f256c070780 0x7f256c0c6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:10.906 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.906+0000 7f2571842640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f256c152120 0x7f256c1f69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.84:59740/0 (socket says 172.21.15.84:59740) 2024-03-19T01:50:10.906 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.906+0000 7f2571842640 1 -- 172.21.15.84:0/3115465715 learned_addr learned my addr 172.21.15.84:0/3115465715 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-03-19T01:50:10.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.910+0000 7f2571842640 1 -- 172.21.15.84:0/3115465715 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f256c070780 msgr2=0x7f256c0c6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:10.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.910+0000 7f2571842640 1 --2- 172.21.15.84:0/3115465715 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f256c070780 0x7f256c0c6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:10.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.910+0000 7f2571842640 1 -- 172.21.15.84:0/3115465715 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f256c0c9610 msgr2=0x7f256c157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:10.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.910+0000 7f2571842640 1 --2- 172.21.15.84:0/3115465715 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f256c0c9610 0x7f256c157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:10.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.910+0000 7f2571842640 1 -- 172.21.15.84:0/3115465715 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f256c0c50e0 con 0x7f256c152120 2024-03-19T01:50:10.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.910+0000 7f2571041640 1 --2- 172.21.15.84:0/3115465715 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f256c070780 0x7f256c0c6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:50:10.911 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.914+0000 7f2571842640 1 --2- 172.21.15.84:0/3115465715 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f256c152120 0x7f256c1f69c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f256801d290 tx=0x7f2568019790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:10.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.914+0000 7f2561ffb640 1 -- 172.21.15.84:0/3115465715 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25680bc020 con 0x7f256c152120 2024-03-19T01:50:10.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.914+0000 7f25732a5640 1 -- 172.21.15.84:0/3115465715 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f256c0c5380 con 0x7f256c152120 2024-03-19T01:50:10.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.914+0000 7f25732a5640 1 -- 172.21.15.84:0/3115465715 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f256c0c70a0 con 0x7f256c152120 2024-03-19T01:50:10.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.918+0000 7f25732a5640 1 -- 172.21.15.84:0/3115465715 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f256c144c40 con 0x7f256c152120 2024-03-19T01:50:10.915 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.918+0000 7f2561ffb640 1 -- 172.21.15.84:0/3115465715 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25680a5670 con 0x7f256c152120 2024-03-19T01:50:10.915 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:10.918+0000 7f2561ffb640 1 -- 172.21.15.84:0/3115465715 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25680a5820 con 0x7f256c152120 2024-03-19T01:50:11.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:11.046+0000 7f2561ffb640 1 -- 172.21.15.84:0/3115465715 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 2) ==== 43583+0+0 (secure 0 0 0) 0x7f25680bf020 con 0x7f256c152120 2024-03-19T01:50:11.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:11.062+0000 7f2561ffb640 1 -- 172.21.15.84:0/3115465715 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f25680ad5b0 con 0x7f256c152120 2024-03-19T01:50:11.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:11.062+0000 7f2561ffb640 1 -- 172.21.15.84:0/3115465715 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f25680c6030 con 0x7f256c152120 2024-03-19T01:50:11.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:11.162+0000 7f25732a5640 1 -- 172.21.15.84:0/3115465715 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc", "id": 7} v 0) -- 0x7f256c068a40 con 0x7f256c152120 2024-03-19T01:50:11.838 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:11.834+0000 7f2561ffb640 1 -- 172.21.15.84:0/3115465715 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mgrmap(e 3) ==== 86794+0+0 (secure 0 0 0) 0x7f25680ca430 con 0x7f256c152120 2024-03-19T01:50:12.593 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:12.594+0000 7f2561ffb640 1 -- 172.21.15.84:0/3115465715 <== mon.2 v2:172.21.15.84:3301/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f25680165e0 con 0x7f256c152120 2024-03-19T01:50:12.594 INFO:teuthology.orchestra.run.smithi084.stdout:7 2024-03-19T01:50:12.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:12.598+0000 7f25732a5640 1 -- 172.21.15.84:0/3115465715 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f256c152120 msgr2=0x7f256c1f69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:12.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:12.598+0000 7f25732a5640 1 --2- 172.21.15.84:0/3115465715 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f256c152120 0x7f256c1f69c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f256801d290 tx=0x7f2568019790 comp rx=0 tx=0).stop 2024-03-19T01:50:12.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:12.598+0000 7f25732a5640 1 -- 172.21.15.84:0/3115465715 shutdown_connections 2024-03-19T01:50:12.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:12.598+0000 7f25732a5640 1 --2- 172.21.15.84:0/3115465715 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f256c152120 0x7f256c1f69c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:12.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:12.598+0000 7f25732a5640 1 --2- 172.21.15.84:0/3115465715 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f256c0c9610 0x7f256c157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:12.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:12.598+0000 7f25732a5640 1 --2- 172.21.15.84:0/3115465715 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f256c070780 0x7f256c0c6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:12.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:12.598+0000 7f25732a5640 1 -- 172.21.15.84:0/3115465715 >> 172.21.15.84:0/3115465715 conn(0x7f256c067600 msgr2=0x7f256c14d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:12.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:12.598+0000 7f25732a5640 1 -- 172.21.15.84:0/3115465715 shutdown_connections 2024-03-19T01:50:12.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-19T01:50:12.598+0000 7f25732a5640 1 -- 172.21.15.84:0/3115465715 wait complete. 2024-03-19T01:50:12.616 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-19T01:50:12.616 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 0 2024-03-19T01:50:12.620 INFO:tasks.ceph.osd.0:Started 2024-03-19T01:50:12.620 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-19T01:50:12.620 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 1 2024-03-19T01:50:12.622 INFO:tasks.ceph.osd.1:Started 2024-03-19T01:50:12.623 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-19T01:50:12.623 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 2 2024-03-19T01:50:12.625 INFO:tasks.ceph.osd.2:Started 2024-03-19T01:50:12.626 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-19T01:50:12.626 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 3 2024-03-19T01:50:12.629 INFO:tasks.ceph.osd.3:Started 2024-03-19T01:50:12.629 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-19T01:50:12.629 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 4 2024-03-19T01:50:12.631 INFO:tasks.ceph.osd.4:Started 2024-03-19T01:50:12.631 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-19T01:50:12.631 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 5 2024-03-19T01:50:12.633 INFO:tasks.ceph.osd.5:Started 2024-03-19T01:50:12.634 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-19T01:50:12.634 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 6 2024-03-19T01:50:12.636 INFO:tasks.ceph.osd.6:Started 2024-03-19T01:50:12.636 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-19T01:50:12.636 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 7 2024-03-19T01:50:12.639 INFO:tasks.ceph.osd.7:Started 2024-03-19T01:50:12.639 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:50:12.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.791+0000 7f3e56075640 1 Processor -- start 2024-03-19T01:50:12.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.791+0000 7f3e56075640 1 -- start start 2024-03-19T01:50:12.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.791+0000 7f3e56075640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f3e5013dd00 0x7f3e500c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:12.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.791+0000 7f3e56075640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3e500c9780 0x7f3e500c8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:12.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.791+0000 7f3e56075640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e500c8f70 con 0x7f3e5015b550 2024-03-19T01:50:12.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.791+0000 7f3e56075640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f3e500c9250 con 0x7f3e5013dd00 2024-03-19T01:50:12.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.791+0000 7f3e56075640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3e501582b0 con 0x7f3e500c9780 2024-03-19T01:50:12.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.795+0000 7f3e4f7fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f3e5013dd00 0x7f3e500c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:12.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.795+0000 7f3e4f7fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f3e5013dd00 0x7f3e500c7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:56650/0 (socket says 172.21.15.55:56650) 2024-03-19T01:50:12.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.795+0000 7f3e4f7fe640 1 -- 172.21.15.55:0/4226993641 learned_addr learned my addr 172.21.15.55:0/4226993641 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:50:12.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.799+0000 7f3e4f7fe640 1 -- 172.21.15.55:0/4226993641 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3e500c9780 msgr2=0x7f3e500c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:12.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.799+0000 7f3e4f7fe640 1 --2- 172.21.15.55:0/4226993641 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3e500c9780 0x7f3e500c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:12.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.799+0000 7f3e4f7fe640 1 -- 172.21.15.55:0/4226993641 >> v1:172.21.15.84:6789/0 conn(0x7f3e5015b550 legacy=0x7f3e500c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:12.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.799+0000 7f3e4f7fe640 1 -- 172.21.15.55:0/4226993641 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e50158590 con 0x7f3e5013dd00 2024-03-19T01:50:12.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.799+0000 7f3e4f7fe640 1 --2- 172.21.15.55:0/4226993641 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3e5013dd00 0x7f3e500c7430 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3e40009870 tx=0x7f3e4001c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5c37cf873790726 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:12.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.803+0000 7f3e4effd640 1 -- 172.21.15.55:0/4226993641 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e400ab430 con 0x7f3e5013dd00 2024-03-19T01:50:12.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.803+0000 7f3e4effd640 1 -- 172.21.15.55:0/4226993641 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e400b0070 con 0x7f3e5013dd00 2024-03-19T01:50:12.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.803+0000 7f3e4effd640 1 -- 172.21.15.55:0/4226993641 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e400ab7f0 con 0x7f3e5013dd00 2024-03-19T01:50:12.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.803+0000 7f3e56075640 1 -- 172.21.15.55:0/4226993641 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3e5013dd00 msgr2=0x7f3e500c7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:12.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.803+0000 7f3e56075640 1 --2- 172.21.15.55:0/4226993641 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3e5013dd00 0x7f3e500c7430 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3e40009870 tx=0x7f3e4001c560 comp rx=0 tx=0).stop 2024-03-19T01:50:12.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.803+0000 7f3e56075640 1 -- 172.21.15.55:0/4226993641 shutdown_connections 2024-03-19T01:50:12.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.803+0000 7f3e56075640 1 --2- 172.21.15.55:0/4226993641 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3e500c9780 0x7f3e500c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:12.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.807+0000 7f3e56075640 1 --2- 172.21.15.55:0/4226993641 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3e5013dd00 0x7f3e500c7430 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:12.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.807+0000 7f3e56075640 1 -- 172.21.15.55:0/4226993641 >> 172.21.15.55:0/4226993641 conn(0x7f3e5006e5a0 msgr2=0x7f3e50072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:12.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.807+0000 7f3e56075640 1 -- 172.21.15.55:0/4226993641 shutdown_connections 2024-03-19T01:50:12.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.807+0000 7f3e56075640 1 -- 172.21.15.55:0/4226993641 wait complete. 2024-03-19T01:50:12.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.807+0000 7f3e56075640 1 Processor -- start 2024-03-19T01:50:12.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.807+0000 7f3e56075640 1 -- start start 2024-03-19T01:50:12.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.807+0000 7f3e56075640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3e500c9780 0x7f3e50159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:12.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.807+0000 7f3e56075640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e5013dd00 0x7f3e501338d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:12.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.807+0000 7f3e56075640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3e5015b550 0x7f3e5013e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:12.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.807+0000 7f3e56075640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3e50158f10 con 0x7f3e5013dd00 2024-03-19T01:50:12.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.807+0000 7f3e56075640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3e50158d90 con 0x7f3e500c9780 2024-03-19T01:50:12.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.807+0000 7f3e56075640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3e500c5e40 con 0x7f3e5015b550 2024-03-19T01:50:12.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.807+0000 7f3e47fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e5013dd00 0x7f3e501338d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:12.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.807+0000 7f3e4f7fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3e500c9780 0x7f3e50159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:12.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.807+0000 7f3e47fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e5013dd00 0x7f3e501338d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:54386/0 (socket says 172.21.15.55:54386) 2024-03-19T01:50:12.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.807+0000 7f3e47fff640 1 -- 172.21.15.55:0/4154256566 learned_addr learned my addr 172.21.15.55:0/4154256566 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:50:12.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.811+0000 7f3e4f7fe640 1 -- 172.21.15.55:0/4154256566 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3e5015b550 msgr2=0x7f3e5013e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:12.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.811+0000 7f3e4f7fe640 1 --2- 172.21.15.55:0/4154256566 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3e5015b550 0x7f3e5013e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:12.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.811+0000 7f3e4f7fe640 1 -- 172.21.15.55:0/4154256566 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e5013dd00 msgr2=0x7f3e501338d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:12.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.811+0000 7f3e4f7fe640 1 --2- 172.21.15.55:0/4154256566 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e5013dd00 0x7f3e501338d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:12.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.811+0000 7f3e4f7fe640 1 -- 172.21.15.55:0/4154256566 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e50141250 con 0x7f3e500c9780 2024-03-19T01:50:12.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.815+0000 7f3e4f7fe640 1 --2- 172.21.15.55:0/4154256566 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3e500c9780 0x7f3e50159120 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3e500cd0e0 tx=0x7f3e40019300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:12.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.819+0000 7f3e4cff9640 1 -- 172.21.15.55:0/4154256566 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e400b4420 con 0x7f3e500c9780 2024-03-19T01:50:12.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.819+0000 7f3e4cff9640 1 -- 172.21.15.55:0/4154256566 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e400b45e0 con 0x7f3e500c9780 2024-03-19T01:50:12.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.819+0000 7f3e4cff9640 1 -- 172.21.15.55:0/4154256566 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e400b47d0 con 0x7f3e500c9780 2024-03-19T01:50:12.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.819+0000 7f3e56075640 1 -- 172.21.15.55:0/4154256566 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e5015a120 con 0x7f3e500c9780 2024-03-19T01:50:12.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.819+0000 7f3e56075640 1 -- 172.21.15.55:0/4154256566 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e5015a6e0 con 0x7f3e500c9780 2024-03-19T01:50:12.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.819+0000 7f3e56075640 1 -- 172.21.15.55:0/4154256566 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e50068a40 con 0x7f3e500c9780 2024-03-19T01:50:12.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.835+0000 7f3e4cff9640 1 -- 172.21.15.55:0/4154256566 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 3) ==== 86794+0+0 (secure 0 0 0) 0x7f3e400bd030 con 0x7f3e500c9780 2024-03-19T01:50:12.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.839+0000 7f3e4cff9640 1 -- 172.21.15.55:0/4154256566 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f3e400ee5b0 con 0x7f3e500c9780 2024-03-19T01:50:12.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:12.951+0000 7f3e4cff9640 1 -- 172.21.15.55:0/4154256566 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3e400c4060 con 0x7f3e500c9780 2024-03-19T01:50:13.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:13.023+0000 7f3e56075640 1 -- 172.21.15.55:0/4154256566 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3e50157d40 con 0x7f3e500c9780 2024-03-19T01:50:13.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:13.243+0000 7f3e4cff9640 1 -- 172.21.15.55:0/4154256566 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mgrmap(e 4) ==== 87027+0+0 (secure 0 0 0) 0x7f3e400eb670 con 0x7f3e500c9780 2024-03-19T01:50:13.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:13.243+0000 7f3e4cff9640 1 --2- 172.21.15.55:0/4154256566 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3e3001d180 0x7f3e3006a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:13.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:13.243+0000 7f3e47fff640 1 --2- 172.21.15.55:0/4154256566 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3e3001d180 0x7f3e3006a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:13.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:13.243+0000 7f3e47fff640 1 --2- 172.21.15.55:0/4154256566 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3e3001d180 0x7f3e3006a7b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3e50159660 tx=0x7f3e38009110 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:13.259 INFO:tasks.ceph.osd.2.smithi055.stderr:2024-03-19T01:50:13.259+0000 7fe4ebbda7c0 -1 Falling back to public interface 2024-03-19T01:50:13.284 INFO:tasks.ceph.osd.5.smithi084.stderr:2024-03-19T01:50:13.286+0000 7fec483c87c0 -1 Falling back to public interface 2024-03-19T01:50:13.306 INFO:tasks.ceph.osd.3.smithi055.stderr:2024-03-19T01:50:13.307+0000 7faad15147c0 -1 Falling back to public interface 2024-03-19T01:50:13.316 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:13.318+0000 7fcac71b9640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-19T01:50:13.333 INFO:tasks.ceph.osd.7.smithi084.stderr:2024-03-19T01:50:13.334+0000 7f1e9c6317c0 -1 Falling back to public interface 2024-03-19T01:50:13.335 INFO:tasks.ceph.osd.4.smithi084.stderr:2024-03-19T01:50:13.338+0000 7ff7143fc7c0 -1 Falling back to public interface 2024-03-19T01:50:13.338 INFO:tasks.ceph.osd.1.smithi055.stderr:2024-03-19T01:50:13.339+0000 7f7f8700d7c0 -1 Falling back to public interface 2024-03-19T01:50:13.359 INFO:tasks.ceph.osd.6.smithi084.stderr:2024-03-19T01:50:13.362+0000 7f3c25cfe7c0 -1 Falling back to public interface 2024-03-19T01:50:13.374 INFO:tasks.ceph.osd.0.smithi055.stderr:2024-03-19T01:50:13.375+0000 7f810010c7c0 -1 Falling back to public interface 2024-03-19T01:50:13.398 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-03-19T01:50:13.398+0000 7fcac71b9640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-19T01:50:13.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:13.555+0000 7f3e4cff9640 1 -- 172.21.15.55:0/4154256566 <== mon.1 v2:172.21.15.84:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f3e40016150 con 0x7f3e500c9780 2024-03-19T01:50:13.556 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:50:13.556 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":10,"fsid":"def9d4e5-0bc7-449d-b893-2490cbd10e25","created":"2024-03-19T01:50:02.482954+0000","modified":"2024-03-19T01:50:12.168944+0000","last_up_change":"0.000000","last_in_change":"2024-03-19T01:50:12.168944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"35d5e3c5-d089-4f38-8961-98d064188f80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8e58f791-001a-44d0-a464-ca68a479f70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d44f38b8-67f4-43de-98a6-f3c520acaf10","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"850a7316-2d74-43f0-8649-b5f8d8f6692e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e6152ae-a77d-4bb6-a855-5782ad5411a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1217ceba-dd31-4615-96f1-dfaf57160f3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3d9ed78a-2284-4c28-a24a-70eb812c7836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:50:13.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:13.559+0000 7f3e56075640 1 -- 172.21.15.55:0/4154256566 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3e3001d180 msgr2=0x7f3e3006a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:13.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:13.559+0000 7f3e56075640 1 --2- 172.21.15.55:0/4154256566 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3e3001d180 0x7f3e3006a7b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3e50159660 tx=0x7f3e38009110 comp rx=0 tx=0).stop 2024-03-19T01:50:13.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:13.559+0000 7f3e56075640 1 -- 172.21.15.55:0/4154256566 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3e500c9780 msgr2=0x7f3e50159120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:13.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:13.559+0000 7f3e56075640 1 --2- 172.21.15.55:0/4154256566 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3e500c9780 0x7f3e50159120 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3e500cd0e0 tx=0x7f3e40019300 comp rx=0 tx=0).stop 2024-03-19T01:50:13.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:13.559+0000 7f3e56075640 1 -- 172.21.15.55:0/4154256566 shutdown_connections 2024-03-19T01:50:13.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:13.559+0000 7f3e56075640 1 --2- 172.21.15.55:0/4154256566 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3e3001d180 0x7f3e3006a7b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:13.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:13.559+0000 7f3e56075640 1 --2- 172.21.15.55:0/4154256566 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3e5015b550 0x7f3e5013e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:13.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:13.559+0000 7f3e56075640 1 --2- 172.21.15.55:0/4154256566 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e5013dd00 0x7f3e501338d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:13.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:13.559+0000 7f3e56075640 1 --2- 172.21.15.55:0/4154256566 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3e500c9780 0x7f3e50159120 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:13.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:13.559+0000 7f3e56075640 1 -- 172.21.15.55:0/4154256566 >> 172.21.15.55:0/4154256566 conn(0x7f3e5006e5a0 msgr2=0x7f3e5013bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:13.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:13.559+0000 7f3e56075640 1 -- 172.21.15.55:0/4154256566 shutdown_connections 2024-03-19T01:50:13.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:13.559+0000 7f3e56075640 1 -- 172.21.15.55:0/4154256566 wait complete. 2024-03-19T01:50:13.582 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-19T01:50:13.582 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-19T01:50:14.389 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-19T01:50:14.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.511+0000 7f8b70540640 1 Processor -- start 2024-03-19T01:50:14.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.515+0000 7f8b70540640 1 -- start start 2024-03-19T01:50:14.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.515+0000 7f8b70540640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8b6806ca30 0x7f8b680cbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:14.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.515+0000 7f8b70540640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f8b680cc4c0 0x7f8b680cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:14.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.515+0000 7f8b70540640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b680cce00 con 0x7f8b680cc8e0 2024-03-19T01:50:14.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.515+0000 7f8b70540640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f8b680cc280 con 0x7f8b680cc4c0 2024-03-19T01:50:14.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.515+0000 7f8b70540640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f8b68154cb0 con 0x7f8b6806ca30 2024-03-19T01:50:14.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.515+0000 7f8b6dadb640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f8b680cc4c0 0x7f8b680cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:14.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.515+0000 7f8b6dadb640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f8b680cc4c0 0x7f8b680cd8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:54388/0 (socket says 172.21.15.55:54388) 2024-03-19T01:50:14.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.515+0000 7f8b6dadb640 1 -- 172.21.15.55:0/1948668064 learned_addr learned my addr 172.21.15.55:0/1948668064 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:50:14.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.515+0000 7f8b6e2dc640 1 --2- 172.21.15.55:0/1948668064 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8b6806ca30 0x7f8b680cbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:14.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.519+0000 7f8b6dadb640 1 -- 172.21.15.55:0/1948668064 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8b6806ca30 msgr2=0x7f8b680cbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:14.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.519+0000 7f8b6dadb640 1 --2- 172.21.15.55:0/1948668064 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8b6806ca30 0x7f8b680cbd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:14.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.519+0000 7f8b6dadb640 1 -- 172.21.15.55:0/1948668064 >> v1:172.21.15.84:6789/0 conn(0x7f8b680cc8e0 legacy=0x7f8b680ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:14.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.519+0000 7f8b6dadb640 1 -- 172.21.15.55:0/1948668064 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b6815a700 con 0x7f8b680cc4c0 2024-03-19T01:50:14.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.523+0000 7f8b6dadb640 1 --2- 172.21.15.55:0/1948668064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b680cc4c0 0x7f8b680cd8c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c009e50 tx=0x7f8b5c01f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cadffcedc33005bd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:14.527 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.527+0000 7f8b6d2da640 1 -- 172.21.15.55:0/1948668064 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c0b4070 con 0x7f8b680cc4c0 2024-03-19T01:50:14.527 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.527+0000 7f8b6d2da640 1 -- 172.21.15.55:0/1948668064 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b5c0af400 con 0x7f8b680cc4c0 2024-03-19T01:50:14.527 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.527+0000 7f8b6d2da640 1 -- 172.21.15.55:0/1948668064 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c0af7d0 con 0x7f8b680cc4c0 2024-03-19T01:50:14.528 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.527+0000 7f8b70540640 1 -- 172.21.15.55:0/1948668064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b680cc4c0 msgr2=0x7f8b680cd8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:14.528 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.527+0000 7f8b70540640 1 --2- 172.21.15.55:0/1948668064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b680cc4c0 0x7f8b680cd8c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c009e50 tx=0x7f8b5c01f210 comp rx=0 tx=0).stop 2024-03-19T01:50:14.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.527+0000 7f8b70540640 1 -- 172.21.15.55:0/1948668064 shutdown_connections 2024-03-19T01:50:14.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.527+0000 7f8b70540640 1 --2- 172.21.15.55:0/1948668064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b680cc4c0 0x7f8b680cd8c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:14.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.527+0000 7f8b70540640 1 --2- 172.21.15.55:0/1948668064 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8b6806ca30 0x7f8b680cbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:14.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.527+0000 7f8b70540640 1 -- 172.21.15.55:0/1948668064 >> 172.21.15.55:0/1948668064 conn(0x7f8b680b1670 msgr2=0x7f8b680b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:14.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.527+0000 7f8b70540640 1 -- 172.21.15.55:0/1948668064 shutdown_connections 2024-03-19T01:50:14.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.527+0000 7f8b70540640 1 -- 172.21.15.55:0/1948668064 wait complete. 2024-03-19T01:50:14.530 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.531+0000 7f8b70540640 1 Processor -- start 2024-03-19T01:50:14.530 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.531+0000 7f8b70540640 1 -- start start 2024-03-19T01:50:14.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.531+0000 7f8b70540640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b6806ca30 0x7f8b6815d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:14.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.531+0000 7f8b70540640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8b680cc4c0 0x7f8b6815e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:14.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.531+0000 7f8b70540640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b680cc8e0 0x7f8b6814de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:14.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.531+0000 7f8b70540640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6815acc0 con 0x7f8b680cc8e0 2024-03-19T01:50:14.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.531+0000 7f8b70540640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6815b030 con 0x7f8b6806ca30 2024-03-19T01:50:14.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.531+0000 7f8b70540640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f8b6806ce10 con 0x7f8b680cc4c0 2024-03-19T01:50:14.532 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.531+0000 7f8b6eadd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b680cc8e0 0x7f8b6814de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:14.532 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.531+0000 7f8b6eadd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b680cc8e0 0x7f8b6814de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:54400/0 (socket says 172.21.15.55:54400) 2024-03-19T01:50:14.532 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.531+0000 7f8b6eadd640 1 -- 172.21.15.55:0/3900388007 learned_addr learned my addr 172.21.15.55:0/3900388007 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:50:14.532 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.531+0000 7f8b6dadb640 1 --2- 172.21.15.55:0/3900388007 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8b680cc4c0 0x7f8b6815e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:14.532 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.531+0000 7f8b6e2dc640 1 --2- 172.21.15.55:0/3900388007 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b6806ca30 0x7f8b6815d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:14.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.535+0000 7f8b6eadd640 1 -- 172.21.15.55:0/3900388007 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8b680cc4c0 msgr2=0x7f8b6815e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:14.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.535+0000 7f8b6eadd640 1 --2- 172.21.15.55:0/3900388007 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8b680cc4c0 0x7f8b6815e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:14.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.535+0000 7f8b6eadd640 1 -- 172.21.15.55:0/3900388007 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b6806ca30 msgr2=0x7f8b6815d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:14.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.535+0000 7f8b6eadd640 1 --2- 172.21.15.55:0/3900388007 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b6806ca30 0x7f8b6815d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:14.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.535+0000 7f8b6eadd640 1 -- 172.21.15.55:0/3900388007 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b680b0670 con 0x7f8b680cc8e0 2024-03-19T01:50:14.538 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.539+0000 7f8b6eadd640 1 --2- 172.21.15.55:0/3900388007 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b680cc8e0 0x7f8b6814de70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8b5800c430 tx=0x7f8b5800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:14.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.543+0000 7f8b66ffd640 1 -- 172.21.15.55:0/3900388007 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b58017070 con 0x7f8b680cc8e0 2024-03-19T01:50:14.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.543+0000 7f8b66ffd640 1 -- 172.21.15.55:0/3900388007 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b580052d0 con 0x7f8b680cc8e0 2024-03-19T01:50:14.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.543+0000 7f8b66ffd640 1 -- 172.21.15.55:0/3900388007 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b58003800 con 0x7f8b680cc8e0 2024-03-19T01:50:14.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.543+0000 7f8b70540640 1 -- 172.21.15.55:0/3900388007 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b68151550 con 0x7f8b680cc8e0 2024-03-19T01:50:14.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.543+0000 7f8b70540640 1 -- 172.21.15.55:0/3900388007 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b6814faf0 con 0x7f8b680cc8e0 2024-03-19T01:50:14.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.543+0000 7f8b70540640 1 -- 172.21.15.55:0/3900388007 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b6804f590 con 0x7f8b680cc8e0 2024-03-19T01:50:14.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.563+0000 7f8b66ffd640 1 -- 172.21.15.55:0/3900388007 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 4) ==== 87027+0+0 (secure 0 0 0) 0x7f8b5801a070 con 0x7f8b680cc8e0 2024-03-19T01:50:14.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.563+0000 7f8b66ffd640 1 --2- 172.21.15.55:0/3900388007 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8b54067620 0x7f8b54069ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:14.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.563+0000 7f8b6e2dc640 1 --2- 172.21.15.55:0/3900388007 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8b54067620 0x7f8b54069ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:14.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.563+0000 7f8b6e2dc640 1 --2- 172.21.15.55:0/3900388007 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8b54067620 0x7f8b54069ae0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8b60009ac0 tx=0x7f8b60009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:14.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.854+0000 7f8b66ffd640 1 -- 172.21.15.55:0/3900388007 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f8b580542a0 con 0x7f8b680cc8e0 2024-03-19T01:50:14.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:14.854+0000 7f8b66ffd640 1 -- 172.21.15.55:0/3900388007 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f8b58005480 con 0x7f8b680cc8e0 2024-03-19T01:50:15.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:15.038+0000 7f8b70540640 1 -- 172.21.15.55:0/3900388007 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b6815e770 con 0x7f8b680cc8e0 2024-03-19T01:50:15.128 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:15.126+0000 7f8b66ffd640 1 -- 172.21.15.55:0/3900388007 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f8b58055610 con 0x7f8b680cc8e0 2024-03-19T01:50:15.129 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-19T01:50:15.129 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"def9d4e5-0bc7-449d-b893-2490cbd10e25","created":"2024-03-19T01:50:02.482954+0000","modified":"2024-03-19T01:50:12.168944+0000","last_up_change":"0.000000","last_in_change":"2024-03-19T01:50:12.168944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"35d5e3c5-d089-4f38-8961-98d064188f80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8e58f791-001a-44d0-a464-ca68a479f70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d44f38b8-67f4-43de-98a6-f3c520acaf10","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"850a7316-2d74-43f0-8649-b5f8d8f6692e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e6152ae-a77d-4bb6-a855-5782ad5411a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1217ceba-dd31-4615-96f1-dfaf57160f3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3d9ed78a-2284-4c28-a24a-70eb812c7836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:50:15.131 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:15.130+0000 7f8b70540640 1 -- 172.21.15.55:0/3900388007 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8b54067620 msgr2=0x7f8b54069ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:15.131 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:15.130+0000 7f8b70540640 1 --2- 172.21.15.55:0/3900388007 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8b54067620 0x7f8b54069ae0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8b60009ac0 tx=0x7f8b60009190 comp rx=0 tx=0).stop 2024-03-19T01:50:15.131 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:15.130+0000 7f8b70540640 1 -- 172.21.15.55:0/3900388007 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b680cc8e0 msgr2=0x7f8b6814de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:15.131 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:15.130+0000 7f8b70540640 1 --2- 172.21.15.55:0/3900388007 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b680cc8e0 0x7f8b6814de70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8b5800c430 tx=0x7f8b5800c8f0 comp rx=0 tx=0).stop 2024-03-19T01:50:15.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:15.130+0000 7f8b70540640 1 -- 172.21.15.55:0/3900388007 shutdown_connections 2024-03-19T01:50:15.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:15.130+0000 7f8b70540640 1 --2- 172.21.15.55:0/3900388007 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8b54067620 0x7f8b54069ae0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:15.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:15.130+0000 7f8b70540640 1 --2- 172.21.15.55:0/3900388007 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b680cc8e0 0x7f8b6814de70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:15.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:15.130+0000 7f8b70540640 1 --2- 172.21.15.55:0/3900388007 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8b680cc4c0 0x7f8b6815e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:15.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:15.130+0000 7f8b70540640 1 --2- 172.21.15.55:0/3900388007 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b6806ca30 0x7f8b6815d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:15.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:15.130+0000 7f8b70540640 1 -- 172.21.15.55:0/3900388007 >> 172.21.15.55:0/3900388007 conn(0x7f8b680b1670 msgr2=0x7f8b680ca360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:15.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:15.130+0000 7f8b70540640 1 -- 172.21.15.55:0/3900388007 shutdown_connections 2024-03-19T01:50:15.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:15.130+0000 7f8b70540640 1 -- 172.21.15.55:0/3900388007 wait complete. 2024-03-19T01:50:15.152 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-19T01:50:17.027 INFO:tasks.ceph.osd.2.smithi055.stderr:2024-03-19T01:50:17.026+0000 7fe4ebbda7c0 -1 osd.2 0 log_to_monitors true 2024-03-19T01:50:17.050 INFO:tasks.ceph.osd.5.smithi084.stderr:2024-03-19T01:50:17.050+0000 7fec483c87c0 -1 osd.5 0 log_to_monitors true 2024-03-19T01:50:17.100 INFO:tasks.ceph.osd.3.smithi055.stderr:2024-03-19T01:50:17.098+0000 7faad15147c0 -1 osd.3 0 log_to_monitors true 2024-03-19T01:50:17.166 INFO:tasks.ceph.osd.1.smithi055.stderr:2024-03-19T01:50:17.166+0000 7f7f8700d7c0 -1 osd.1 0 log_to_monitors true 2024-03-19T01:50:17.200 INFO:tasks.ceph.osd.4.smithi084.stderr:2024-03-19T01:50:17.202+0000 7ff7143fc7c0 -1 osd.4 0 log_to_monitors true 2024-03-19T01:50:17.214 INFO:tasks.ceph.osd.7.smithi084.stderr:2024-03-19T01:50:17.214+0000 7f1e9c6317c0 -1 osd.7 0 log_to_monitors true 2024-03-19T01:50:17.229 INFO:tasks.ceph.osd.0.smithi055.stderr:2024-03-19T01:50:17.226+0000 7f810010c7c0 -1 osd.0 0 log_to_monitors true 2024-03-19T01:50:17.285 INFO:tasks.ceph.osd.6.smithi084.stderr:2024-03-19T01:50:17.286+0000 7f3c25cfe7c0 -1 osd.6 0 log_to_monitors true 2024-03-19T01:50:19.293 INFO:tasks.ceph.osd.5.smithi084.stderr:2024-03-19T01:50:19.294+0000 7fec44371640 -1 osd.5 0 waiting for initial osdmap 2024-03-19T01:50:19.451 INFO:tasks.ceph.osd.5.smithi084.stderr:2024-03-19T01:50:19.454+0000 7fec3c1a6640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:50:19.507 INFO:tasks.ceph.osd.2.smithi055.stderr:2024-03-19T01:50:19.506+0000 7fe4e8395640 -1 osd.2 0 waiting for initial osdmap 2024-03-19T01:50:19.552 INFO:tasks.ceph.osd.3.smithi055.stderr:2024-03-19T01:50:19.550+0000 7faacd4bd640 -1 osd.3 0 waiting for initial osdmap 2024-03-19T01:50:19.654 INFO:tasks.ceph.osd.3.smithi055.stderr:2024-03-19T01:50:19.654+0000 7faac52f2640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:50:19.655 INFO:tasks.ceph.osd.2.smithi055.stderr:2024-03-19T01:50:19.654+0000 7fe4df9b8640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:50:20.349 INFO:tasks.ceph.osd.1.smithi055.stderr:2024-03-19T01:50:20.346+0000 7f7f82fb6640 -1 osd.1 0 waiting for initial osdmap 2024-03-19T01:50:20.386 INFO:tasks.ceph.osd.1.smithi055.stderr:2024-03-19T01:50:20.386+0000 7f7f7adeb640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:50:20.386 INFO:tasks.ceph.osd.0.smithi055.stderr:2024-03-19T01:50:20.386+0000 7f80fc8c7640 -1 osd.0 0 waiting for initial osdmap 2024-03-19T01:50:20.422 INFO:tasks.ceph.osd.0.smithi055.stderr:2024-03-19T01:50:20.422+0000 7f80f3eea640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:50:21.208 INFO:tasks.ceph.osd.7.smithi084.stderr:2024-03-19T01:50:21.210+0000 7f1e985da640 -1 osd.7 0 waiting for initial osdmap 2024-03-19T01:50:21.210 INFO:tasks.ceph.osd.6.smithi084.stderr:2024-03-19T01:50:21.210+0000 7f3c224b9640 -1 osd.6 0 waiting for initial osdmap 2024-03-19T01:50:21.231 INFO:tasks.ceph.osd.4.smithi084.stderr:2024-03-19T01:50:21.234+0000 7ff7103a5640 -1 osd.4 0 waiting for initial osdmap 2024-03-19T01:50:21.263 INFO:tasks.ceph.osd.7.smithi084.stderr:2024-03-19T01:50:21.266+0000 7f1e9040f640 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:50:21.279 INFO:tasks.ceph.osd.4.smithi084.stderr:2024-03-19T01:50:21.282+0000 7ff7081da640 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:50:21.320 INFO:tasks.ceph.osd.6.smithi084.stderr:2024-03-19T01:50:21.322+0000 7f3c19adc640 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:50:21.959 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-19T01:50:22.085 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.086+0000 7f4a01ef9640 1 Processor -- start 2024-03-19T01:50:22.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.086+0000 7f4a01ef9640 1 -- start start 2024-03-19T01:50:22.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.086+0000 7f4a01ef9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49fc152050 0x7f49fc0746f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:22.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.086+0000 7f4a01ef9640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f49fc158dd0 0x7f49fc0c7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:22.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.086+0000 7f4a01ef9640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f49fc072730 0x7f49fc0c8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:22.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.086+0000 7f4a01ef9640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f49fc158940 con 0x7f49fc072730 2024-03-19T01:50:22.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.086+0000 7f4a01ef9640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f49fc0aec90 con 0x7f49fc158dd0 2024-03-19T01:50:22.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.086+0000 7f4a01ef9640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f49fc0af1f0 con 0x7f49fc152050 2024-03-19T01:50:22.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.086+0000 7f49fb7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49fc152050 0x7f49fc0746f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:22.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.086+0000 7f49faffd640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f49fc158dd0 0x7f49fc0c7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:22.089 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.086+0000 7f49fb7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49fc152050 0x7f49fc0746f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:55260/0 (socket says 172.21.15.55:55260) 2024-03-19T01:50:22.089 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.086+0000 7f49fb7fe640 1 -- 172.21.15.55:0/1227210213 learned_addr learned my addr 172.21.15.55:0/1227210213 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:50:22.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.094+0000 7f49fb7fe640 1 -- 172.21.15.55:0/1227210213 >> v2:172.21.15.84:3300/0 conn(0x7f49fc158dd0 msgr2=0x7f49fc0c7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:22.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.094+0000 7f49fb7fe640 1 --2- 172.21.15.55:0/1227210213 >> v2:172.21.15.84:3300/0 conn(0x7f49fc158dd0 0x7f49fc0c7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:22.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.094+0000 7f49fb7fe640 1 -- 172.21.15.55:0/1227210213 >> v2:172.21.15.55:3300/0 conn(0x7f49fc072730 msgr2=0x7f49fc0c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:22.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.094+0000 7f49fb7fe640 1 --2- 172.21.15.55:0/1227210213 >> v2:172.21.15.55:3300/0 conn(0x7f49fc072730 0x7f49fc0c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:22.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.094+0000 7f49fb7fe640 1 -- 172.21.15.55:0/1227210213 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49fc0c9640 con 0x7f49fc152050 2024-03-19T01:50:22.099 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.098+0000 7f49fb7fe640 1 --2- 172.21.15.55:0/1227210213 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49fc152050 0x7f49fc0746f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f49ec009870 tx=0x7f49ec01c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68ace01db8a994a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:22.109 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.110+0000 7f49fa7fc640 1 -- 172.21.15.55:0/1227210213 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ec0b0070 con 0x7f49fc152050 2024-03-19T01:50:22.109 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.110+0000 7f49fa7fc640 1 -- 172.21.15.55:0/1227210213 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49ec0ab600 con 0x7f49fc152050 2024-03-19T01:50:22.110 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.110+0000 7f49fa7fc640 1 -- 172.21.15.55:0/1227210213 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ec0ab9d0 con 0x7f49fc152050 2024-03-19T01:50:22.110 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.110+0000 7f4a01ef9640 1 -- 172.21.15.55:0/1227210213 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49fc152050 msgr2=0x7f49fc0746f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:22.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.110+0000 7f4a01ef9640 1 --2- 172.21.15.55:0/1227210213 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49fc152050 0x7f49fc0746f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f49ec009870 tx=0x7f49ec01c520 comp rx=0 tx=0).stop 2024-03-19T01:50:22.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.110+0000 7f4a01ef9640 1 -- 172.21.15.55:0/1227210213 shutdown_connections 2024-03-19T01:50:22.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.110+0000 7f4a01ef9640 1 --2- 172.21.15.55:0/1227210213 >> v2:172.21.15.55:3300/0 conn(0x7f49fc072730 0x7f49fc0c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:22.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.110+0000 7f4a01ef9640 1 --2- 172.21.15.55:0/1227210213 >> v2:172.21.15.84:3300/0 conn(0x7f49fc158dd0 0x7f49fc0c7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:22.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.110+0000 7f4a01ef9640 1 --2- 172.21.15.55:0/1227210213 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49fc152050 0x7f49fc0746f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:22.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.110+0000 7f4a01ef9640 1 -- 172.21.15.55:0/1227210213 >> 172.21.15.55:0/1227210213 conn(0x7f49fc06fa70 msgr2=0x7f49fc06fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:22.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.110+0000 7f4a01ef9640 1 -- 172.21.15.55:0/1227210213 shutdown_connections 2024-03-19T01:50:22.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.110+0000 7f4a01ef9640 1 -- 172.21.15.55:0/1227210213 wait complete. 2024-03-19T01:50:22.112 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.110+0000 7f4a01ef9640 1 Processor -- start 2024-03-19T01:50:22.112 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.110+0000 7f4a01ef9640 1 -- start start 2024-03-19T01:50:22.112 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.110+0000 7f4a01ef9640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f49fc072730 0x7f49fc14cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:22.112 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.110+0000 7f4a01ef9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49fc152050 0x7f49fc0c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:22.113 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.110+0000 7f4a01ef9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49fc158dd0 0x7f49fc1f3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:22.113 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.110+0000 7f4a01ef9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f49fc072ba0 con 0x7f49fc158dd0 2024-03-19T01:50:22.113 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.110+0000 7f4a01ef9640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f49fc0c92a0 con 0x7f49fc072730 2024-03-19T01:50:22.113 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.110+0000 7f4a01ef9640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f49fc0ae950 con 0x7f49fc152050 2024-03-19T01:50:22.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.114+0000 7f49fb7fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f49fc072730 0x7f49fc14cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:22.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.114+0000 7f49fb7fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f49fc072730 0x7f49fc14cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:40472/0 (socket says 172.21.15.55:40472) 2024-03-19T01:50:22.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.114+0000 7f49fb7fe640 1 -- 172.21.15.55:0/3375911231 learned_addr learned my addr 172.21.15.55:0/3375911231 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:50:22.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.114+0000 7f49faffd640 1 --2- 172.21.15.55:0/3375911231 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49fc152050 0x7f49fc0c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:22.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.122+0000 7f49fb7fe640 1 -- 172.21.15.55:0/3375911231 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49fc152050 msgr2=0x7f49fc0c5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:22.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.122+0000 7f49fb7fe640 1 --2- 172.21.15.55:0/3375911231 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49fc152050 0x7f49fc0c5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:22.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.122+0000 7f49fb7fe640 1 -- 172.21.15.55:0/3375911231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49fc158dd0 msgr2=0x7f49fc1f3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:22.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.122+0000 7f49fb7fe640 1 --2- 172.21.15.55:0/3375911231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49fc158dd0 0x7f49fc1f3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:22.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.122+0000 7f49fb7fe640 1 -- 172.21.15.55:0/3375911231 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49fc0cc400 con 0x7f49fc072730 2024-03-19T01:50:22.130 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.130+0000 7f49fb7fe640 1 --2- 172.21.15.55:0/3375911231 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f49fc072730 0x7f49fc14cea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f49ec01cdc0 tx=0x7f49ec0192c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:22.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.138+0000 7f49e3fff640 1 -- 172.21.15.55:0/3375911231 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ec0b0050 con 0x7f49fc072730 2024-03-19T01:50:22.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.138+0000 7f49e3fff640 1 -- 172.21.15.55:0/3375911231 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49ec0bb070 con 0x7f49fc072730 2024-03-19T01:50:22.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.138+0000 7f49e3fff640 1 -- 172.21.15.55:0/3375911231 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ec0b5850 con 0x7f49fc072730 2024-03-19T01:50:22.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.138+0000 7f4a01ef9640 1 -- 172.21.15.55:0/3375911231 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49fc0c3f10 con 0x7f49fc072730 2024-03-19T01:50:22.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.138+0000 7f4a01ef9640 1 -- 172.21.15.55:0/3375911231 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49fc0c44d0 con 0x7f49fc072730 2024-03-19T01:50:22.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.142+0000 7f4a01ef9640 1 -- 172.21.15.55:0/3375911231 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49fc04f590 con 0x7f49fc072730 2024-03-19T01:50:22.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.482+0000 7f49e3fff640 1 -- 172.21.15.55:0/3375911231 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 4) ==== 87027+0+0 (secure 0 0 0) 0x7f49ec0b59f0 con 0x7f49fc072730 2024-03-19T01:50:22.483 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.482+0000 7f49e3fff640 1 --2- 172.21.15.55:0/3375911231 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f49e8067650 0x7f49e8069b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:22.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.482+0000 7f49faffd640 1 --2- 172.21.15.55:0/3375911231 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f49e8067650 0x7f49e8069b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:22.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.482+0000 7f49faffd640 1 --2- 172.21.15.55:0/3375911231 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f49e8067650 0x7f49e8069b10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f49fc150720 tx=0x7f49f0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:22.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.486+0000 7f49e3fff640 1 -- 172.21.15.55:0/3375911231 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 3071+0+0 (secure 0 0 0) 0x7f49ec0ee990 con 0x7f49fc072730 2024-03-19T01:50:22.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.510+0000 7f49e3fff640 1 -- 172.21.15.55:0/3375911231 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f49ec016110 con 0x7f49fc072730 2024-03-19T01:50:22.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.698+0000 7f4a01ef9640 1 -- 172.21.15.55:0/3375911231 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f49fc06c9c0 con 0x7f49fc072730 2024-03-19T01:50:22.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.750+0000 7f49e3fff640 1 -- 172.21.15.55:0/3375911231 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+10894 (secure 0 0 0) 0x7f49ec0f0ff0 con 0x7f49fc072730 2024-03-19T01:50:22.751 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-19T01:50:22.751 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":15,"fsid":"def9d4e5-0bc7-449d-b893-2490cbd10e25","created":"2024-03-19T01:50:02.482954+0000","modified":"2024-03-19T01:50:21.735508+0000","last_up_change":"2024-03-19T01:50:21.735508+0000","last_in_change":"2024-03-19T01:50:12.168944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"35d5e3c5-d089-4f38-8961-98d064188f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6825","nonce":2757675043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6827","nonce":2757675043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6831","nonce":2757675043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6829","nonce":2757675043}]},"public_addr":"172.21.15.55:6825/2757675043","cluster_addr":"172.21.15.55:6827/2757675043","heartbeat_back_addr":"172.21.15.55:6831/2757675043","heartbeat_front_addr":"172.21.15.55:6829/2757675043","state":["exists","up"]},{"osd":1,"uuid":"8e58f791-001a-44d0-a464-ca68a479f70c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6817","nonce":543460007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6819","nonce":543460007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6823","nonce":543460007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6821","nonce":543460007}]},"public_addr":"172.21.15.55:6817/543460007","cluster_addr":"172.21.15.55:6819/543460007","heartbeat_back_addr":"172.21.15.55:6823/543460007","heartbeat_front_addr":"172.21.15.55:6821/543460007","state":["exists","up"]},{"osd":2,"uuid":"d44f38b8-67f4-43de-98a6-f3c520acaf10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6801","nonce":1760253550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6803","nonce":1760253550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6807","nonce":1760253550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6805","nonce":1760253550}]},"public_addr":"172.21.15.55:6801/1760253550","cluster_addr":"172.21.15.55:6803/1760253550","heartbeat_back_addr":"172.21.15.55:6807/1760253550","heartbeat_front_addr":"172.21.15.55:6805/1760253550","state":["exists","up"]},{"osd":3,"uuid":"850a7316-2d74-43f0-8649-b5f8d8f6692e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6809","nonce":2797616448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6811","nonce":2797616448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6815","nonce":2797616448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6813","nonce":2797616448}]},"public_addr":"172.21.15.55:6809/2797616448","cluster_addr":"172.21.15.55:6811/2797616448","heartbeat_back_addr":"172.21.15.55:6815/2797616448","heartbeat_front_addr":"172.21.15.55:6813/2797616448","state":["exists","up"]},{"osd":4,"uuid":"4e6152ae-a77d-4bb6-a855-5782ad5411a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6819","nonce":2938436826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6821","nonce":2938436826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6825","nonce":2938436826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6823","nonce":2938436826}]},"public_addr":"172.21.15.84:6819/2938436826","cluster_addr":"172.21.15.84:6821/2938436826","heartbeat_back_addr":"172.21.15.84:6825/2938436826","heartbeat_front_addr":"172.21.15.84:6823/2938436826","state":["exists","up"]},{"osd":5,"uuid":"1217ceba-dd31-4615-96f1-dfaf57160f3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6803","nonce":460717295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6805","nonce":460717295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6809","nonce":460717295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6807","nonce":460717295}]},"public_addr":"172.21.15.84:6803/460717295","cluster_addr":"172.21.15.84:6805/460717295","heartbeat_back_addr":"172.21.15.84:6809/460717295","heartbeat_front_addr":"172.21.15.84:6807/460717295","state":["exists","up"]},{"osd":6,"uuid":"3d9ed78a-2284-4c28-a24a-70eb812c7836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6827","nonce":416123135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6829","nonce":416123135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6833","nonce":416123135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6831","nonce":416123135}]},"public_addr":"172.21.15.84:6827/416123135","cluster_addr":"172.21.15.84:6829/416123135","heartbeat_back_addr":"172.21.15.84:6833/416123135","heartbeat_front_addr":"172.21.15.84:6831/416123135","state":["exists","up"]},{"osd":7,"uuid":"c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6811","nonce":460083528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6813","nonce":460083528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6817","nonce":460083528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6815","nonce":460083528}]},"public_addr":"172.21.15.84:6811/460083528","cluster_addr":"172.21.15.84:6813/460083528","heartbeat_back_addr":"172.21.15.84:6817/460083528","heartbeat_front_addr":"172.21.15.84:6815/460083528","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.075714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.092461+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.062833+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:50:22.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.750+0000 7f4a01ef9640 1 -- 172.21.15.55:0/3375911231 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f49e8067650 msgr2=0x7f49e8069b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:22.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.750+0000 7f4a01ef9640 1 --2- 172.21.15.55:0/3375911231 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f49e8067650 0x7f49e8069b10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f49fc150720 tx=0x7f49f0009190 comp rx=0 tx=0).stop 2024-03-19T01:50:22.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.750+0000 7f4a01ef9640 1 -- 172.21.15.55:0/3375911231 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f49fc072730 msgr2=0x7f49fc14cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:22.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.750+0000 7f4a01ef9640 1 --2- 172.21.15.55:0/3375911231 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f49fc072730 0x7f49fc14cea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f49ec01cdc0 tx=0x7f49ec0192c0 comp rx=0 tx=0).stop 2024-03-19T01:50:22.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.750+0000 7f4a01ef9640 1 -- 172.21.15.55:0/3375911231 shutdown_connections 2024-03-19T01:50:22.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.750+0000 7f4a01ef9640 1 --2- 172.21.15.55:0/3375911231 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f49e8067650 0x7f49e8069b10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:22.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.750+0000 7f4a01ef9640 1 --2- 172.21.15.55:0/3375911231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49fc158dd0 0x7f49fc1f3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:22.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.750+0000 7f4a01ef9640 1 --2- 172.21.15.55:0/3375911231 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49fc152050 0x7f49fc0c5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:22.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.750+0000 7f4a01ef9640 1 --2- 172.21.15.55:0/3375911231 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f49fc072730 0x7f49fc14cea0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:22.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.750+0000 7f4a01ef9640 1 -- 172.21.15.55:0/3375911231 >> 172.21.15.55:0/3375911231 conn(0x7f49fc06fa70 msgr2=0x7f49fc14ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:22.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.750+0000 7f4a01ef9640 1 -- 172.21.15.55:0/3375911231 shutdown_connections 2024-03-19T01:50:22.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-19T01:50:22.750+0000 7f4a01ef9640 1 -- 172.21.15.55:0/3375911231 wait complete. 2024-03-19T01:50:22.770 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-19T01:50:22.770 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-19T01:50:22.771 DEBUG:tasks.ceph_manager:running mds.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.a.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck'] 2024-03-19T01:50:22.771 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-19T01:50:22.771 DEBUG:teuthology.orchestra.run.smithi055:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.a.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-mds -f --cluster ceph -i a 2024-03-19T01:50:22.773 INFO:tasks.ceph.mds.a:Started 2024-03-19T01:50:22.773 DEBUG:tasks.ceph_manager:running mds.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.c.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck'] 2024-03-19T01:50:22.773 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-19T01:50:22.774 DEBUG:teuthology.orchestra.run.smithi055:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.c.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-mds -f --cluster ceph -i c 2024-03-19T01:50:22.776 INFO:tasks.ceph.mds.c:Started 2024-03-19T01:50:22.776 DEBUG:tasks.ceph_manager:running mds.e under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.e.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck'] 2024-03-19T01:50:22.776 INFO:tasks.ceph.mds.e:Restarting daemon 2024-03-19T01:50:22.776 DEBUG:teuthology.orchestra.run.smithi055:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.e.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-mds -f --cluster ceph -i e 2024-03-19T01:50:22.778 INFO:tasks.ceph.mds.e:Started 2024-03-19T01:50:22.778 DEBUG:tasks.ceph_manager:running mds.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.b.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck'] 2024-03-19T01:50:22.778 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-19T01:50:22.778 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.b.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-mds -f --cluster ceph -i b 2024-03-19T01:50:22.781 INFO:tasks.ceph.mds.b:Started 2024-03-19T01:50:22.781 DEBUG:tasks.ceph_manager:running mds.d under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.d.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck'] 2024-03-19T01:50:22.781 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-19T01:50:22.781 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.d.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-mds -f --cluster ceph -i d 2024-03-19T01:50:22.783 INFO:tasks.ceph.mds.d:Started 2024-03-19T01:50:22.783 DEBUG:tasks.ceph_manager:running mds.f under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.f.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck'] 2024-03-19T01:50:22.784 INFO:tasks.ceph.mds.f:Restarting daemon 2024-03-19T01:50:22.784 DEBUG:teuthology.orchestra.run.smithi084:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.f.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-mds -f --cluster ceph -i f 2024-03-19T01:50:22.786 INFO:tasks.ceph.mds.f:Started 2024-03-19T01:50:22.786 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-19T01:50:22.787 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:50:22.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.906+0000 7fdba23b1640 1 Processor -- start 2024-03-19T01:50:22.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.906+0000 7fdba23b1640 1 -- start start 2024-03-19T01:50:22.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.906+0000 7fdba23b1640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdb9c0c98e0 0x7fdb9c15e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:22.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.906+0000 7fdba23b1640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb9c071280 con 0x7fdb9c154720 2024-03-19T01:50:22.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.906+0000 7fdba23b1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb9c15d900 con 0x7fdb9c154340 2024-03-19T01:50:22.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.906+0000 7fdba23b1640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fdb9c073840 con 0x7fdb9c0c98e0 2024-03-19T01:50:22.910 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.910+0000 7fdb9bfff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fdb9c154340 0x7fdb9c072e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:43670/0 (socket says 172.21.15.55:43670) 2024-03-19T01:50:22.910 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.910+0000 7fdb9bfff640 1 -- 172.21.15.55:0/662307184 learned_addr learned my addr 172.21.15.55:0/662307184 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:50:22.910 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.910+0000 7fdba094e640 1 --2- 172.21.15.55:0/662307184 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdb9c0c98e0 0x7fdb9c15e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:22.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.914+0000 7fdba094e640 1 -- 172.21.15.55:0/662307184 >> v1:172.21.15.84:6789/0 conn(0x7fdb9c154340 legacy=0x7fdb9c072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:22.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.914+0000 7fdba094e640 1 -- 172.21.15.55:0/662307184 >> v1:172.21.15.55:6789/0 conn(0x7fdb9c154720 legacy=0x7fdb9c0c91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:22.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.914+0000 7fdba094e640 1 -- 172.21.15.55:0/662307184 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb9c070cc0 con 0x7fdb9c0c98e0 2024-03-19T01:50:22.920 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.918+0000 7fdba094e640 1 --2- 172.21.15.55:0/662307184 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdb9c0c98e0 0x7fdb9c15e3a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdb94009dc0 tx=0x7fdb9401c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=762ee2a427897286 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:22.927 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.926+0000 7fdb9affd640 1 -- 172.21.15.55:0/662307184 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb940b2070 con 0x7fdb9c0c98e0 2024-03-19T01:50:22.928 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.926+0000 7fdb9affd640 1 -- 172.21.15.55:0/662307184 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb940a5810 con 0x7fdb9c0c98e0 2024-03-19T01:50:22.928 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.926+0000 7fdb9affd640 1 -- 172.21.15.55:0/662307184 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb940a5be0 con 0x7fdb9c0c98e0 2024-03-19T01:50:22.928 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.926+0000 7fdba23b1640 1 -- 172.21.15.55:0/662307184 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdb9c0c98e0 msgr2=0x7fdb9c15e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:22.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.926+0000 7fdba23b1640 1 --2- 172.21.15.55:0/662307184 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdb9c0c98e0 0x7fdb9c15e3a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdb94009dc0 tx=0x7fdb9401c950 comp rx=0 tx=0).stop 2024-03-19T01:50:22.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.926+0000 7fdba23b1640 1 -- 172.21.15.55:0/662307184 shutdown_connections 2024-03-19T01:50:22.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.926+0000 7fdba23b1640 1 --2- 172.21.15.55:0/662307184 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdb9c0c98e0 0x7fdb9c15e3a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:22.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.926+0000 7fdba23b1640 1 -- 172.21.15.55:0/662307184 >> 172.21.15.55:0/662307184 conn(0x7fdb9c0cc9c0 msgr2=0x7fdb9c0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:22.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.926+0000 7fdba23b1640 1 -- 172.21.15.55:0/662307184 shutdown_connections 2024-03-19T01:50:22.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.926+0000 7fdba23b1640 1 -- 172.21.15.55:0/662307184 wait complete. 2024-03-19T01:50:22.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.930+0000 7fdba23b1640 1 Processor -- start 2024-03-19T01:50:22.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.930+0000 7fdba23b1640 1 -- start start 2024-03-19T01:50:22.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.930+0000 7fdba23b1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb9c0c98e0 0x7fdb9c1596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:22.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.930+0000 7fdba23b1640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdb9c154340 0x7fdb9c14d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:22.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.930+0000 7fdba23b1640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdb9c154720 0x7fdb9c14dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:22.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.930+0000 7fdba23b1640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9c168390 con 0x7fdb9c0c98e0 2024-03-19T01:50:22.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.930+0000 7fdba23b1640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9c15b0b0 con 0x7fdb9c154340 2024-03-19T01:50:22.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.930+0000 7fdba23b1640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fdb9c168510 con 0x7fdb9c154720 2024-03-19T01:50:22.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.934+0000 7fdba094e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdb9c154720 0x7fdb9c14dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:22.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.934+0000 7fdb9b7fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdb9c154340 0x7fdb9c14d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:22.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.934+0000 7fdba094e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdb9c154720 0x7fdb9c14dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:55290/0 (socket says 172.21.15.55:55290) 2024-03-19T01:50:22.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.934+0000 7fdba094e640 1 -- 172.21.15.55:0/768707359 learned_addr learned my addr 172.21.15.55:0/768707359 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:50:22.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.938+0000 7fdba094e640 1 -- 172.21.15.55:0/768707359 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdb9c154340 msgr2=0x7fdb9c14d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:22.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.938+0000 7fdba094e640 1 --2- 172.21.15.55:0/768707359 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdb9c154340 0x7fdb9c14d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:22.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.938+0000 7fdba094e640 1 -- 172.21.15.55:0/768707359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb9c0c98e0 msgr2=0x7fdb9c1596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:22.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.938+0000 7fdba094e640 1 --2- 172.21.15.55:0/768707359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb9c0c98e0 0x7fdb9c1596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:22.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.938+0000 7fdba094e640 1 -- 172.21.15.55:0/768707359 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb9c1f4640 con 0x7fdb9c154720 2024-03-19T01:50:22.945 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.942+0000 7fdba094e640 1 --2- 172.21.15.55:0/768707359 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdb9c154720 0x7fdb9c14dc00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdb9401ce30 tx=0x7fdb940196f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:22.956 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.954+0000 7fdb98ff9640 1 -- 172.21.15.55:0/768707359 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb940b2070 con 0x7fdb9c154720 2024-03-19T01:50:22.956 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.954+0000 7fdb98ff9640 1 -- 172.21.15.55:0/768707359 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb940b3040 con 0x7fdb9c154720 2024-03-19T01:50:22.956 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.954+0000 7fdb98ff9640 1 -- 172.21.15.55:0/768707359 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb940b9890 con 0x7fdb9c154720 2024-03-19T01:50:22.956 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.954+0000 7fdba23b1640 1 -- 172.21.15.55:0/768707359 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb9c1f3ac0 con 0x7fdb9c154720 2024-03-19T01:50:22.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.954+0000 7fdba23b1640 1 -- 172.21.15.55:0/768707359 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb9c1f4080 con 0x7fdb9c154720 2024-03-19T01:50:22.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.958+0000 7fdba23b1640 1 -- 172.21.15.55:0/768707359 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb9c068a40 con 0x7fdb9c154720 2024-03-19T01:50:22.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.994+0000 7fdb98ff9640 1 -- 172.21.15.55:0/768707359 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 4) ==== 87027+0+0 (secure 0 0 0) 0x7fdb940c1030 con 0x7fdb9c154720 2024-03-19T01:50:22.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.994+0000 7fdb98ff9640 1 --2- 172.21.15.55:0/768707359 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdb88067650 0x7fdb88069b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:22.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.994+0000 7fdb9bfff640 1 --2- 172.21.15.55:0/768707359 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdb88067650 0x7fdb88069b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:22.997 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:22.998+0000 7fdb9bfff640 1 --2- 172.21.15.55:0/768707359 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdb88067650 0x7fdb88069b10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c001600 tx=0x7fdb8c007a50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:23.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.278+0000 7fdb98ff9640 1 -- 172.21.15.55:0/768707359 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4471+0+0 (secure 0 0 0) 0x7fdb94002020 con 0x7fdb9c154720 2024-03-19T01:50:23.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.290+0000 7fdb98ff9640 1 -- 172.21.15.55:0/768707359 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fdb940c8020 con 0x7fdb9c154720 2024-03-19T01:50:23.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.486+0000 7fdba23b1640 1 -- 172.21.15.55:0/768707359 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdb9c0c7da0 con 0x7fdb9c154720 2024-03-19T01:50:23.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.562+0000 7fdb98ff9640 1 -- 172.21.15.55:0/768707359 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+10894 (secure 0 0 0) 0x7fdb94016540 con 0x7fdb9c154720 2024-03-19T01:50:23.563 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:50:23.563 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":15,"fsid":"def9d4e5-0bc7-449d-b893-2490cbd10e25","created":"2024-03-19T01:50:02.482954+0000","modified":"2024-03-19T01:50:21.735508+0000","last_up_change":"2024-03-19T01:50:21.735508+0000","last_in_change":"2024-03-19T01:50:12.168944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"35d5e3c5-d089-4f38-8961-98d064188f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6825","nonce":2757675043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6827","nonce":2757675043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6831","nonce":2757675043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6829","nonce":2757675043}]},"public_addr":"172.21.15.55:6825/2757675043","cluster_addr":"172.21.15.55:6827/2757675043","heartbeat_back_addr":"172.21.15.55:6831/2757675043","heartbeat_front_addr":"172.21.15.55:6829/2757675043","state":["exists","up"]},{"osd":1,"uuid":"8e58f791-001a-44d0-a464-ca68a479f70c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6817","nonce":543460007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6819","nonce":543460007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6823","nonce":543460007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6821","nonce":543460007}]},"public_addr":"172.21.15.55:6817/543460007","cluster_addr":"172.21.15.55:6819/543460007","heartbeat_back_addr":"172.21.15.55:6823/543460007","heartbeat_front_addr":"172.21.15.55:6821/543460007","state":["exists","up"]},{"osd":2,"uuid":"d44f38b8-67f4-43de-98a6-f3c520acaf10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6801","nonce":1760253550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6803","nonce":1760253550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6807","nonce":1760253550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6805","nonce":1760253550}]},"public_addr":"172.21.15.55:6801/1760253550","cluster_addr":"172.21.15.55:6803/1760253550","heartbeat_back_addr":"172.21.15.55:6807/1760253550","heartbeat_front_addr":"172.21.15.55:6805/1760253550","state":["exists","up"]},{"osd":3,"uuid":"850a7316-2d74-43f0-8649-b5f8d8f6692e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6809","nonce":2797616448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6811","nonce":2797616448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6815","nonce":2797616448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6813","nonce":2797616448}]},"public_addr":"172.21.15.55:6809/2797616448","cluster_addr":"172.21.15.55:6811/2797616448","heartbeat_back_addr":"172.21.15.55:6815/2797616448","heartbeat_front_addr":"172.21.15.55:6813/2797616448","state":["exists","up"]},{"osd":4,"uuid":"4e6152ae-a77d-4bb6-a855-5782ad5411a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6819","nonce":2938436826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6821","nonce":2938436826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6825","nonce":2938436826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6823","nonce":2938436826}]},"public_addr":"172.21.15.84:6819/2938436826","cluster_addr":"172.21.15.84:6821/2938436826","heartbeat_back_addr":"172.21.15.84:6825/2938436826","heartbeat_front_addr":"172.21.15.84:6823/2938436826","state":["exists","up"]},{"osd":5,"uuid":"1217ceba-dd31-4615-96f1-dfaf57160f3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6803","nonce":460717295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6805","nonce":460717295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6809","nonce":460717295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6807","nonce":460717295}]},"public_addr":"172.21.15.84:6803/460717295","cluster_addr":"172.21.15.84:6805/460717295","heartbeat_back_addr":"172.21.15.84:6809/460717295","heartbeat_front_addr":"172.21.15.84:6807/460717295","state":["exists","up"]},{"osd":6,"uuid":"3d9ed78a-2284-4c28-a24a-70eb812c7836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6827","nonce":416123135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6829","nonce":416123135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6833","nonce":416123135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6831","nonce":416123135}]},"public_addr":"172.21.15.84:6827/416123135","cluster_addr":"172.21.15.84:6829/416123135","heartbeat_back_addr":"172.21.15.84:6833/416123135","heartbeat_front_addr":"172.21.15.84:6831/416123135","state":["exists","up"]},{"osd":7,"uuid":"c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6811","nonce":460083528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6813","nonce":460083528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6817","nonce":460083528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6815","nonce":460083528}]},"public_addr":"172.21.15.84:6811/460083528","cluster_addr":"172.21.15.84:6813/460083528","heartbeat_back_addr":"172.21.15.84:6817/460083528","heartbeat_front_addr":"172.21.15.84:6815/460083528","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.075714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.092461+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.062833+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:50:23.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.562+0000 7fdba23b1640 1 -- 172.21.15.55:0/768707359 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdb88067650 msgr2=0x7fdb88069b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:23.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.562+0000 7fdba23b1640 1 --2- 172.21.15.55:0/768707359 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdb88067650 0x7fdb88069b10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c001600 tx=0x7fdb8c007a50 comp rx=0 tx=0).stop 2024-03-19T01:50:23.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.562+0000 7fdba23b1640 1 -- 172.21.15.55:0/768707359 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdb9c154720 msgr2=0x7fdb9c14dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:23.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.562+0000 7fdba23b1640 1 --2- 172.21.15.55:0/768707359 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdb9c154720 0x7fdb9c14dc00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdb9401ce30 tx=0x7fdb940196f0 comp rx=0 tx=0).stop 2024-03-19T01:50:23.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.562+0000 7fdba23b1640 1 -- 172.21.15.55:0/768707359 shutdown_connections 2024-03-19T01:50:23.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.562+0000 7fdba23b1640 1 --2- 172.21.15.55:0/768707359 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdb88067650 0x7fdb88069b10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:23.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.562+0000 7fdba23b1640 1 --2- 172.21.15.55:0/768707359 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdb9c154720 0x7fdb9c14dc00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:23.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.566+0000 7fdba23b1640 1 --2- 172.21.15.55:0/768707359 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdb9c154340 0x7fdb9c14d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:23.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.566+0000 7fdba23b1640 1 --2- 172.21.15.55:0/768707359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb9c0c98e0 0x7fdb9c1596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:23.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.566+0000 7fdba23b1640 1 -- 172.21.15.55:0/768707359 >> 172.21.15.55:0/768707359 conn(0x7fdb9c0cc9c0 msgr2=0x7fdb9c0cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:23.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.566+0000 7fdba23b1640 1 -- 172.21.15.55:0/768707359 shutdown_connections 2024-03-19T01:50:23.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.566+0000 7fdba23b1640 1 -- 172.21.15.55:0/768707359 wait complete. 2024-03-19T01:50:23.581 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-03-19T01:50:23.582 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:50:23.743 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.742+0000 7f678a411640 1 Processor -- start 2024-03-19T01:50:23.743 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.742+0000 7f678a411640 1 -- start start 2024-03-19T01:50:23.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.742+0000 7f678a411640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6784150f50 0x7f678415b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:23.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.742+0000 7f678a411640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f67840ca350 0x7f67840728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:23.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.742+0000 7f678a411640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f67840c3320 0x7f6784157920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:23.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.742+0000 7f678a411640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f67840c3e00 con 0x7f67840ca350 2024-03-19T01:50:23.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.742+0000 7f678a411640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f678415ac10 con 0x7f67840c3320 2024-03-19T01:50:23.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.742+0000 7f678a411640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f6784157e60 con 0x7f6784150f50 2024-03-19T01:50:23.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.742+0000 7f67837fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f67840ca350 0x7f67840728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:23.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.742+0000 7f67837fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f67840ca350 0x7f67840728b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:42524/0 (socket says 172.21.15.55:42524) 2024-03-19T01:50:23.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.742+0000 7f67837fe640 1 -- 172.21.15.55:0/1857512893 learned_addr learned my addr 172.21.15.55:0/1857512893 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:50:23.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.746+0000 7f67837fe640 1 -- 172.21.15.55:0/1857512893 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6784150f50 msgr2=0x7f678415b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:23.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.746+0000 7f67837fe640 1 --2- 172.21.15.55:0/1857512893 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6784150f50 0x7f678415b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:23.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.746+0000 7f67837fe640 1 -- 172.21.15.55:0/1857512893 >> v2:172.21.15.84:3300/0 conn(0x7f67840c3320 msgr2=0x7f6784157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:23.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.746+0000 7f67837fe640 1 --2- 172.21.15.55:0/1857512893 >> v2:172.21.15.84:3300/0 conn(0x7f67840c3320 0x7f6784157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:23.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.746+0000 7f67837fe640 1 -- 172.21.15.55:0/1857512893 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6784151330 con 0x7f67840ca350 2024-03-19T01:50:23.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.750+0000 7f67837fe640 1 --2- 172.21.15.55:0/1857512893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67840ca350 0x7f67840728b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6784151730 tx=0x7f677801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d9a681c91ea55ff server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:23.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.754+0000 7f6782ffd640 1 -- 172.21.15.55:0/1857512893 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67780a6040 con 0x7f67840ca350 2024-03-19T01:50:23.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.754+0000 7f6782ffd640 1 -- 172.21.15.55:0/1857512893 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67780af070 con 0x7f67840ca350 2024-03-19T01:50:23.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.754+0000 7f6782ffd640 1 -- 172.21.15.55:0/1857512893 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67780aa860 con 0x7f67840ca350 2024-03-19T01:50:23.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.754+0000 7f678a411640 1 -- 172.21.15.55:0/1857512893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67840ca350 msgr2=0x7f67840728b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:23.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.754+0000 7f678a411640 1 --2- 172.21.15.55:0/1857512893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67840ca350 0x7f67840728b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6784151730 tx=0x7f677801c520 comp rx=0 tx=0).stop 2024-03-19T01:50:23.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.754+0000 7f678a411640 1 -- 172.21.15.55:0/1857512893 shutdown_connections 2024-03-19T01:50:23.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.754+0000 7f678a411640 1 --2- 172.21.15.55:0/1857512893 >> v2:172.21.15.84:3300/0 conn(0x7f67840c3320 0x7f6784157920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:23.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.754+0000 7f678a411640 1 --2- 172.21.15.55:0/1857512893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67840ca350 0x7f67840728b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:23.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.754+0000 7f678a411640 1 --2- 172.21.15.55:0/1857512893 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6784150f50 0x7f678415b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:23.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.754+0000 7f678a411640 1 -- 172.21.15.55:0/1857512893 >> 172.21.15.55:0/1857512893 conn(0x7f678406e670 msgr2=0x7f678406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:23.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.754+0000 7f678a411640 1 -- 172.21.15.55:0/1857512893 shutdown_connections 2024-03-19T01:50:23.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.754+0000 7f678a411640 1 -- 172.21.15.55:0/1857512893 wait complete. 2024-03-19T01:50:23.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.758+0000 7f678a411640 1 Processor -- start 2024-03-19T01:50:23.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.758+0000 7f678a411640 1 -- start start 2024-03-19T01:50:23.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.758+0000 7f678a411640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f67840c3320 0x7f67840c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:23.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.758+0000 7f678a411640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f67840ca350 0x7f67841f5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:23.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.758+0000 7f678a411640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6784150f50 0x7f67840c46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:23.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.758+0000 7f678a411640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f678406c3b0 con 0x7f6784150f50 2024-03-19T01:50:23.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.758+0000 7f678a411640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f678415a8a0 con 0x7f67840c3320 2024-03-19T01:50:23.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.758+0000 7f678a411640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f678406c530 con 0x7f67840ca350 2024-03-19T01:50:23.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.758+0000 7f67889ae640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6784150f50 0x7f67840c46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:23.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.758+0000 7f67889ae640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6784150f50 0x7f67840c46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:42540/0 (socket says 172.21.15.55:42540) 2024-03-19T01:50:23.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.758+0000 7f67889ae640 1 -- 172.21.15.55:0/898117970 learned_addr learned my addr 172.21.15.55:0/898117970 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:50:23.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.762+0000 7f67889ae640 1 -- 172.21.15.55:0/898117970 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f67840ca350 msgr2=0x7f67841f5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:23.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.762+0000 7f67889ae640 1 --2- 172.21.15.55:0/898117970 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f67840ca350 0x7f67841f5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:23.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.762+0000 7f67889ae640 1 -- 172.21.15.55:0/898117970 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f67840c3320 msgr2=0x7f67840c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:23.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.762+0000 7f67889ae640 1 --2- 172.21.15.55:0/898117970 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f67840c3320 0x7f67840c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:23.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.762+0000 7f67889ae640 1 -- 172.21.15.55:0/898117970 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67840c4140 con 0x7f6784150f50 2024-03-19T01:50:23.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.766+0000 7f67889ae640 1 --2- 172.21.15.55:0/898117970 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6784150f50 0x7f67840c46c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f677c00b9e0 tx=0x7f677c00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:23.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.770+0000 7f6780ff9640 1 -- 172.21.15.55:0/898117970 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c019070 con 0x7f6784150f50 2024-03-19T01:50:23.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.770+0000 7f6780ff9640 1 -- 172.21.15.55:0/898117970 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f677c00cac0 con 0x7f6784150f50 2024-03-19T01:50:23.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.770+0000 7f678a411640 1 -- 172.21.15.55:0/898117970 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67840c4c00 con 0x7f6784150f50 2024-03-19T01:50:23.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.770+0000 7f6780ff9640 1 -- 172.21.15.55:0/898117970 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c014440 con 0x7f6784150f50 2024-03-19T01:50:23.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.770+0000 7f678a411640 1 -- 172.21.15.55:0/898117970 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67840c43e0 con 0x7f6784150f50 2024-03-19T01:50:23.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.770+0000 7f678a411640 1 -- 172.21.15.55:0/898117970 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6784068a40 con 0x7f6784150f50 2024-03-19T01:50:23.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.806+0000 7f6780ff9640 1 -- 172.21.15.55:0/898117970 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 4) ==== 87027+0+0 (secure 0 0 0) 0x7f677c01c070 con 0x7f6784150f50 2024-03-19T01:50:23.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.806+0000 7f6780ff9640 1 --2- 172.21.15.55:0/898117970 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f67700675d0 0x7f6770069a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:23.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.806+0000 7f6783fff640 1 --2- 172.21.15.55:0/898117970 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f67700675d0 0x7f6770069a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:23.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:23.806+0000 7f6783fff640 1 --2- 172.21.15.55:0/898117970 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f67700675d0 0x7f6770069a90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6774009690 tx=0x7f6774006c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:24.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:24.090+0000 7f6780ff9640 1 -- 172.21.15.55:0/898117970 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4471+0+0 (secure 0 0 0) 0x7f677c054000 con 0x7f6784150f50 2024-03-19T01:50:24.092 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:24.090+0000 7f6780ff9640 1 -- 172.21.15.55:0/898117970 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f677c023020 con 0x7f6784150f50 2024-03-19T01:50:24.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:24.370+0000 7f678a411640 1 -- 172.21.15.55:0/898117970 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f678414d650 con 0x7f6784150f50 2024-03-19T01:50:24.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:24.406+0000 7f6780ff9640 1 -- 172.21.15.55:0/898117970 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f677c056660 con 0x7f6784150f50 2024-03-19T01:50:24.409 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:50:24.410 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-19T01:50:24.410 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 1 2024-03-19T01:50:24.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:24.410+0000 7f678a411640 1 -- 172.21.15.55:0/898117970 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f67700675d0 msgr2=0x7f6770069a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:24.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:24.410+0000 7f678a411640 1 --2- 172.21.15.55:0/898117970 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f67700675d0 0x7f6770069a90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6774009690 tx=0x7f6774006c00 comp rx=0 tx=0).stop 2024-03-19T01:50:24.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:24.410+0000 7f678a411640 1 -- 172.21.15.55:0/898117970 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6784150f50 msgr2=0x7f67840c46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:24.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:24.410+0000 7f678a411640 1 --2- 172.21.15.55:0/898117970 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6784150f50 0x7f67840c46c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f677c00b9e0 tx=0x7f677c00bea0 comp rx=0 tx=0).stop 2024-03-19T01:50:24.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:24.410+0000 7f678a411640 1 -- 172.21.15.55:0/898117970 shutdown_connections 2024-03-19T01:50:24.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:24.410+0000 7f678a411640 1 --2- 172.21.15.55:0/898117970 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f67700675d0 0x7f6770069a90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:24.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:24.410+0000 7f678a411640 1 --2- 172.21.15.55:0/898117970 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6784150f50 0x7f67840c46c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:24.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:24.410+0000 7f678a411640 1 --2- 172.21.15.55:0/898117970 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f67840ca350 0x7f67841f5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:24.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:24.410+0000 7f678a411640 1 --2- 172.21.15.55:0/898117970 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f67840c3320 0x7f67840c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:24.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:24.410+0000 7f678a411640 1 -- 172.21.15.55:0/898117970 >> 172.21.15.55:0/898117970 conn(0x7f678406e670 msgr2=0x7f6784074290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:24.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:24.410+0000 7f678a411640 1 -- 172.21.15.55:0/898117970 shutdown_connections 2024-03-19T01:50:24.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:24.410+0000 7f678a411640 1 -- 172.21.15.55:0/898117970 wait complete. 2024-03-19T01:50:26.433 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:50:26.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.538+0000 7f581935d640 1 Processor -- start 2024-03-19T01:50:26.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.538+0000 7f581935d640 1 -- start start 2024-03-19T01:50:26.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.538+0000 7f581935d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f581406c610 0x7f581415ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:26.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.538+0000 7f581935d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f581406ceb0 0x7f58141427f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:26.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.538+0000 7f581935d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5814070e80 con 0x7f581406c9f0 2024-03-19T01:50:26.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.538+0000 7f581935d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f581415b250 con 0x7f581406c610 2024-03-19T01:50:26.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.538+0000 7f581935d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f581406fb00 con 0x7f581406ceb0 2024-03-19T01:50:26.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.538+0000 7f5812ffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f581406c610 0x7f581415ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:26.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.538+0000 7f5812ffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f581406c610 0x7f581415ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:42548/0 (socket says 172.21.15.55:42548) 2024-03-19T01:50:26.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.538+0000 7f5812ffd640 1 -- 172.21.15.55:0/1319388188 learned_addr learned my addr 172.21.15.55:0/1319388188 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:50:26.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.542+0000 7f5812ffd640 1 -- 172.21.15.55:0/1319388188 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f581406ceb0 msgr2=0x7f58141427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:26.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.542+0000 7f5812ffd640 1 --2- 172.21.15.55:0/1319388188 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f581406ceb0 0x7f58141427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:26.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.542+0000 7f5812ffd640 1 -- 172.21.15.55:0/1319388188 >> v1:172.21.15.55:6789/0 conn(0x7f581406c9f0 legacy=0x7f58140c92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:26.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.542+0000 7f5812ffd640 1 -- 172.21.15.55:0/1319388188 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f581406fde0 con 0x7f581406c610 2024-03-19T01:50:26.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.550+0000 7f5812ffd640 1 --2- 172.21.15.55:0/1319388188 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f581406c610 0x7f581415ad10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5804009870 tx=0x7f580401ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618168bf6803b078 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:26.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.554+0000 7f5811ffb640 1 -- 172.21.15.55:0/1319388188 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58040b5070 con 0x7f581406c610 2024-03-19T01:50:26.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.554+0000 7f5811ffb640 1 -- 172.21.15.55:0/1319388188 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58040a8a50 con 0x7f581406c610 2024-03-19T01:50:26.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.554+0000 7f5811ffb640 1 -- 172.21.15.55:0/1319388188 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58040a8e20 con 0x7f581406c610 2024-03-19T01:50:26.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.554+0000 7f581935d640 1 -- 172.21.15.55:0/1319388188 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f581406c610 msgr2=0x7f581415ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:26.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.554+0000 7f581935d640 1 --2- 172.21.15.55:0/1319388188 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f581406c610 0x7f581415ad10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5804009870 tx=0x7f580401ed50 comp rx=0 tx=0).stop 2024-03-19T01:50:26.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.554+0000 7f581935d640 1 -- 172.21.15.55:0/1319388188 shutdown_connections 2024-03-19T01:50:26.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.554+0000 7f581935d640 1 --2- 172.21.15.55:0/1319388188 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f581406ceb0 0x7f58141427f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:26.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.554+0000 7f581935d640 1 --2- 172.21.15.55:0/1319388188 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f581406c610 0x7f581415ad10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:26.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.554+0000 7f581935d640 1 -- 172.21.15.55:0/1319388188 >> 172.21.15.55:0/1319388188 conn(0x7f5814067600 msgr2=0x7f5814071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:26.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.554+0000 7f581935d640 1 -- 172.21.15.55:0/1319388188 shutdown_connections 2024-03-19T01:50:26.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.554+0000 7f581935d640 1 -- 172.21.15.55:0/1319388188 wait complete. 2024-03-19T01:50:26.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.554+0000 7f581935d640 1 Processor -- start 2024-03-19T01:50:26.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.554+0000 7f581935d640 1 -- start start 2024-03-19T01:50:26.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.554+0000 7f581935d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f581406c610 0x7f581413c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:26.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.554+0000 7f581935d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f581406c9f0 0x7f58140cd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:26.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.554+0000 7f581935d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f581406ceb0 0x7f58140cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:26.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.558+0000 7f581935d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5814070680 con 0x7f581406ceb0 2024-03-19T01:50:26.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.558+0000 7f581935d640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f58140703a0 con 0x7f581406c610 2024-03-19T01:50:26.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.558+0000 7f581935d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f5814070960 con 0x7f581406c9f0 2024-03-19T01:50:26.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.558+0000 7f58137fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f581406ceb0 0x7f58140cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:26.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.558+0000 7f58137fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f581406ceb0 0x7f58140cda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:42558/0 (socket says 172.21.15.55:42558) 2024-03-19T01:50:26.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.558+0000 7f58137fe640 1 -- 172.21.15.55:0/1298849262 learned_addr learned my addr 172.21.15.55:0/1298849262 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:50:26.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.562+0000 7f58137fe640 1 -- 172.21.15.55:0/1298849262 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f581406c9f0 msgr2=0x7f58140cd4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:26.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.562+0000 7f58137fe640 1 --2- 172.21.15.55:0/1298849262 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f581406c9f0 0x7f58140cd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:26.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.562+0000 7f58137fe640 1 -- 172.21.15.55:0/1298849262 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f581406c610 msgr2=0x7f581413c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:26.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.562+0000 7f58137fe640 1 --2- 172.21.15.55:0/1298849262 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f581406c610 0x7f581413c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:26.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.562+0000 7f58137fe640 1 -- 172.21.15.55:0/1298849262 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58140ce900 con 0x7f581406ceb0 2024-03-19T01:50:26.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.566+0000 7f58137fe640 1 --2- 172.21.15.55:0/1298849262 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f581406ceb0 0x7f58140cda10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f580c00b970 tx=0x7f580c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:26.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.570+0000 7f57fb7fe640 1 -- 172.21.15.55:0/1298849262 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c019070 con 0x7f581406ceb0 2024-03-19T01:50:26.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.570+0000 7f57fb7fe640 1 -- 172.21.15.55:0/1298849262 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f580c00cb60 con 0x7f581406ceb0 2024-03-19T01:50:26.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.570+0000 7f581935d640 1 -- 172.21.15.55:0/1298849262 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58140ce340 con 0x7f581406ceb0 2024-03-19T01:50:26.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.570+0000 7f57fb7fe640 1 -- 172.21.15.55:0/1298849262 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c014440 con 0x7f581406ceb0 2024-03-19T01:50:26.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.570+0000 7f581935d640 1 -- 172.21.15.55:0/1298849262 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58140c60b0 con 0x7f581406ceb0 2024-03-19T01:50:26.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.570+0000 7f581935d640 1 -- 172.21.15.55:0/1298849262 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5814068a40 con 0x7f581406ceb0 2024-03-19T01:50:26.913 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.914+0000 7f57fb7fe640 1 -- 172.21.15.55:0/1298849262 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 4) ==== 87027+0+0 (secure 0 0 0) 0x7f580c01c070 con 0x7f581406ceb0 2024-03-19T01:50:26.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.914+0000 7f57fb7fe640 1 --2- 172.21.15.55:0/1298849262 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5800067650 0x7f5800069b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:26.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.914+0000 7f5812ffd640 1 --2- 172.21.15.55:0/1298849262 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5800067650 0x7f5800069b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:26.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.914+0000 7f5812ffd640 1 --2- 172.21.15.55:0/1298849262 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5800067650 0x7f5800069b10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f58040b2000 tx=0x7f5804004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:26.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.950+0000 7f57fb7fe640 1 -- 172.21.15.55:0/1298849262 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f580c05c350 con 0x7f581406ceb0 2024-03-19T01:50:26.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:26.950+0000 7f57fb7fe640 1 -- 172.21.15.55:0/1298849262 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f580c023020 con 0x7f581406ceb0 2024-03-19T01:50:27.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:27.190+0000 7f581935d640 1 -- 172.21.15.55:0/1298849262 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f581413cf30 con 0x7f581406ceb0 2024-03-19T01:50:27.277 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:27.274+0000 7f57fb7fe640 1 -- 172.21.15.55:0/1298849262 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f580c056680 con 0x7f581406ceb0 2024-03-19T01:50:27.277 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:50:27.277 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-19T01:50:27.278 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 1 2024-03-19T01:50:27.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:27.278+0000 7f581935d640 1 -- 172.21.15.55:0/1298849262 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5800067650 msgr2=0x7f5800069b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:27.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:27.278+0000 7f581935d640 1 --2- 172.21.15.55:0/1298849262 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5800067650 0x7f5800069b10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f58040b2000 tx=0x7f5804004740 comp rx=0 tx=0).stop 2024-03-19T01:50:27.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:27.278+0000 7f581935d640 1 -- 172.21.15.55:0/1298849262 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f581406ceb0 msgr2=0x7f58140cda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:27.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:27.278+0000 7f581935d640 1 --2- 172.21.15.55:0/1298849262 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f581406ceb0 0x7f58140cda10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f580c00b970 tx=0x7f580c00be30 comp rx=0 tx=0).stop 2024-03-19T01:50:27.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:27.278+0000 7f581935d640 1 -- 172.21.15.55:0/1298849262 shutdown_connections 2024-03-19T01:50:27.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:27.278+0000 7f581935d640 1 --2- 172.21.15.55:0/1298849262 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5800067650 0x7f5800069b10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:27.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:27.278+0000 7f581935d640 1 --2- 172.21.15.55:0/1298849262 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f581406ceb0 0x7f58140cda10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:27.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:27.278+0000 7f581935d640 1 --2- 172.21.15.55:0/1298849262 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f581406c9f0 0x7f58140cd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:27.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:27.278+0000 7f581935d640 1 --2- 172.21.15.55:0/1298849262 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f581406c610 0x7f581413c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:27.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:27.278+0000 7f581935d640 1 -- 172.21.15.55:0/1298849262 >> 172.21.15.55:0/1298849262 conn(0x7f5814067600 msgr2=0x7f58140c2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:27.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:27.278+0000 7f581935d640 1 -- 172.21.15.55:0/1298849262 shutdown_connections 2024-03-19T01:50:27.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:27.278+0000 7f581935d640 1 -- 172.21.15.55:0/1298849262 wait complete. 2024-03-19T01:50:29.298 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:50:29.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.454+0000 7f7eb4877640 1 Processor -- start 2024-03-19T01:50:29.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.454+0000 7f7eb4877640 1 -- start start 2024-03-19T01:50:29.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.454+0000 7f7eb4877640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7eb006bb00 0x7f7eb006c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:29.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.454+0000 7f7eb4877640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7eb006bee0 0x7f7eb0156b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:29.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.454+0000 7f7eb4877640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7eb0159d80 con 0x7f7eb00c6b00 2024-03-19T01:50:29.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.454+0000 7f7eb4877640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f7eb015a060 con 0x7f7eb006bb00 2024-03-19T01:50:29.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.454+0000 7f7eb4877640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f7eb015a340 con 0x7f7eb006bee0 2024-03-19T01:50:29.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.458+0000 7f7eaed9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7eb006bee0 0x7f7eb0156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:29.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.458+0000 7f7eae59c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7eb006bb00 0x7f7eb006c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:29.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.458+0000 7f7eaed9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7eb006bee0 0x7f7eb0156b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:55006/0 (socket says 172.21.15.55:55006) 2024-03-19T01:50:29.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.458+0000 7f7eaed9d640 1 -- 172.21.15.55:0/1871404326 learned_addr learned my addr 172.21.15.55:0/1871404326 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:50:29.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.458+0000 7f7eaed9d640 1 -- 172.21.15.55:0/1871404326 >> v2:172.21.15.55:3300/0 conn(0x7f7eb006bb00 msgr2=0x7f7eb006c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:29.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.458+0000 7f7eaed9d640 1 --2- 172.21.15.55:0/1871404326 >> v2:172.21.15.55:3300/0 conn(0x7f7eb006bb00 0x7f7eb006c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:29.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.458+0000 7f7eaed9d640 1 -- 172.21.15.55:0/1871404326 >> v1:172.21.15.84:6789/0 conn(0x7f7eb00c6b00 legacy=0x7f7eb0150640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:29.461 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.458+0000 7f7eaed9d640 1 -- 172.21.15.55:0/1871404326 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eb015a620 con 0x7f7eb006bee0 2024-03-19T01:50:29.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.466+0000 7f7eaed9d640 1 --2- 172.21.15.55:0/1871404326 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7eb006bee0 0x7f7eb0156b10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7ea800af60 tx=0x7f7ea8020330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ec33898aa66c390 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:29.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.470+0000 7f7ead59a640 1 -- 172.21.15.55:0/1871404326 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ea80b5070 con 0x7f7eb006bee0 2024-03-19T01:50:29.471 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.470+0000 7f7ead59a640 1 -- 172.21.15.55:0/1871404326 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ea800a2d0 con 0x7f7eb006bee0 2024-03-19T01:50:29.471 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.470+0000 7f7ead59a640 1 -- 172.21.15.55:0/1871404326 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ea8008720 con 0x7f7eb006bee0 2024-03-19T01:50:29.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.470+0000 7f7eb4877640 1 -- 172.21.15.55:0/1871404326 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7eb006bee0 msgr2=0x7f7eb0156b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:29.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.470+0000 7f7eb4877640 1 --2- 172.21.15.55:0/1871404326 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7eb006bee0 0x7f7eb0156b10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7ea800af60 tx=0x7f7ea8020330 comp rx=0 tx=0).stop 2024-03-19T01:50:29.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.470+0000 7f7eb4877640 1 -- 172.21.15.55:0/1871404326 shutdown_connections 2024-03-19T01:50:29.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.470+0000 7f7eb4877640 1 --2- 172.21.15.55:0/1871404326 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7eb006bee0 0x7f7eb0156b10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:29.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.470+0000 7f7eb4877640 1 --2- 172.21.15.55:0/1871404326 >> v2:172.21.15.55:3300/0 conn(0x7f7eb006bb00 0x7f7eb006c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:29.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.470+0000 7f7eb4877640 1 -- 172.21.15.55:0/1871404326 >> 172.21.15.55:0/1871404326 conn(0x7f7eb006fb40 msgr2=0x7f7eb006ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:29.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.470+0000 7f7eb4877640 1 -- 172.21.15.55:0/1871404326 shutdown_connections 2024-03-19T01:50:29.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.470+0000 7f7eb4877640 1 -- 172.21.15.55:0/1871404326 wait complete. 2024-03-19T01:50:29.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.474+0000 7f7eb4877640 1 Processor -- start 2024-03-19T01:50:29.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.474+0000 7f7eb4877640 1 -- start start 2024-03-19T01:50:29.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.474+0000 7f7eb4877640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7eb006bb00 0x7f7eb00cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:29.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.474+0000 7f7eb4877640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7eb006bee0 0x7f7eb00cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:29.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.474+0000 7f7eb4877640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7eb00c6b00 0x7f7eb0143f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:29.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.474+0000 7f7eb4877640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7eb0150ed0 con 0x7f7eb006bb00 2024-03-19T01:50:29.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.474+0000 7f7eb4877640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f7eb0150d50 con 0x7f7eb006bee0 2024-03-19T01:50:29.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.474+0000 7f7eb4877640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f7eb00c3600 con 0x7f7eb00c6b00 2024-03-19T01:50:29.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.474+0000 7f7eaed9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7eb00c6b00 0x7f7eb0143f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:29.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.474+0000 7f7eae59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7eb006bb00 0x7f7eb00cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:29.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.474+0000 7f7eaed9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7eb00c6b00 0x7f7eb0143f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:55008/0 (socket says 172.21.15.55:55008) 2024-03-19T01:50:29.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.474+0000 7f7eaed9d640 1 -- 172.21.15.55:0/795275838 learned_addr learned my addr 172.21.15.55:0/795275838 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:50:29.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.474+0000 7f7eadd9b640 1 --2- 172.21.15.55:0/795275838 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7eb006bee0 0x7f7eb00cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:29.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.478+0000 7f7eaed9d640 1 -- 172.21.15.55:0/795275838 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7eb006bee0 msgr2=0x7f7eb00cb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:29.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.478+0000 7f7eaed9d640 1 --2- 172.21.15.55:0/795275838 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7eb006bee0 0x7f7eb00cb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:29.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.478+0000 7f7eaed9d640 1 -- 172.21.15.55:0/795275838 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7eb006bb00 msgr2=0x7f7eb00cb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:29.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.478+0000 7f7eaed9d640 1 --2- 172.21.15.55:0/795275838 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7eb006bb00 0x7f7eb00cb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:29.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.478+0000 7f7eaed9d640 1 -- 172.21.15.55:0/795275838 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eb014e270 con 0x7f7eb00c6b00 2024-03-19T01:50:29.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.482+0000 7f7eaed9d640 1 --2- 172.21.15.55:0/795275838 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7eb00c6b00 0x7f7eb0143f10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7ea801e9e0 tx=0x7f7ea801d0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:29.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.486+0000 7f7e9effd640 1 -- 172.21.15.55:0/795275838 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ea80bf070 con 0x7f7eb00c6b00 2024-03-19T01:50:29.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.486+0000 7f7eb4877640 1 -- 172.21.15.55:0/795275838 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7eb0159300 con 0x7f7eb00c6b00 2024-03-19T01:50:29.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.486+0000 7f7eb4877640 1 -- 172.21.15.55:0/795275838 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7eb01598c0 con 0x7f7eb00c6b00 2024-03-19T01:50:29.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.486+0000 7f7eb4877640 1 -- 172.21.15.55:0/795275838 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7eb0068a40 con 0x7f7eb00c6b00 2024-03-19T01:50:29.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.490+0000 7f7e9effd640 1 -- 172.21.15.55:0/795275838 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ea800a2d0 con 0x7f7eb00c6b00 2024-03-19T01:50:29.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.490+0000 7f7e9effd640 1 -- 172.21.15.55:0/795275838 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ea80b5040 con 0x7f7eb00c6b00 2024-03-19T01:50:29.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.794+0000 7f7e9effd640 1 -- 172.21.15.55:0/795275838 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 4) ==== 87027+0+0 (secure 0 0 0) 0x7f7ea80c2070 con 0x7f7eb00c6b00 2024-03-19T01:50:29.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.794+0000 7f7e9effd640 1 --2- 172.21.15.55:0/795275838 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7e900675d0 0x7f7e90069a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:29.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.794+0000 7f7eae59c640 1 --2- 172.21.15.55:0/795275838 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7e900675d0 0x7f7e90069a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:29.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.794+0000 7f7eae59c640 1 --2- 172.21.15.55:0/795275838 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7e900675d0 0x7f7e90069a90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0009b60 tx=0x7f7ea0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:29.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.798+0000 7f7e9effd640 1 -- 172.21.15.55:0/795275838 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f7ea8121740 con 0x7f7eb00c6b00 2024-03-19T01:50:29.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:29.814+0000 7f7e9effd640 1 -- 172.21.15.55:0/795275838 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f7ea80ca080 con 0x7f7eb00c6b00 2024-03-19T01:50:30.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:30.054+0000 7f7eb4877640 1 -- 172.21.15.55:0/795275838 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7eb006c3c0 con 0x7f7eb00c6b00 2024-03-19T01:50:30.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:30.074+0000 7f7e9effd640 1 -- 172.21.15.55:0/795275838 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f7ea8019e90 con 0x7f7eb00c6b00 2024-03-19T01:50:30.077 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:50:30.078 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-19T01:50:30.078 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 1 2024-03-19T01:50:30.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:30.078+0000 7f7eb4877640 1 -- 172.21.15.55:0/795275838 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7e900675d0 msgr2=0x7f7e90069a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:30.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:30.078+0000 7f7eb4877640 1 --2- 172.21.15.55:0/795275838 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7e900675d0 0x7f7e90069a90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0009b60 tx=0x7f7ea0009190 comp rx=0 tx=0).stop 2024-03-19T01:50:30.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:30.078+0000 7f7eb4877640 1 -- 172.21.15.55:0/795275838 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7eb00c6b00 msgr2=0x7f7eb0143f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:30.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:30.078+0000 7f7eb4877640 1 --2- 172.21.15.55:0/795275838 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7eb00c6b00 0x7f7eb0143f10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7ea801e9e0 tx=0x7f7ea801d0d0 comp rx=0 tx=0).stop 2024-03-19T01:50:30.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:30.078+0000 7f7eb4877640 1 -- 172.21.15.55:0/795275838 shutdown_connections 2024-03-19T01:50:30.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:30.078+0000 7f7eb4877640 1 --2- 172.21.15.55:0/795275838 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7e900675d0 0x7f7e90069a90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:30.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:30.078+0000 7f7eb4877640 1 --2- 172.21.15.55:0/795275838 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7eb00c6b00 0x7f7eb0143f10 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:30.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:30.078+0000 7f7eb4877640 1 --2- 172.21.15.55:0/795275838 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7eb006bee0 0x7f7eb00cb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:30.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:30.078+0000 7f7eb4877640 1 --2- 172.21.15.55:0/795275838 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7eb006bb00 0x7f7eb00cb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:30.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:30.078+0000 7f7eb4877640 1 -- 172.21.15.55:0/795275838 >> 172.21.15.55:0/795275838 conn(0x7f7eb006fb40 msgr2=0x7f7eb006ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:30.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:30.078+0000 7f7eb4877640 1 -- 172.21.15.55:0/795275838 shutdown_connections 2024-03-19T01:50:30.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:30.078+0000 7f7eb4877640 1 -- 172.21.15.55:0/795275838 wait complete. 2024-03-19T01:50:30.704 INFO:tasks.ceph.mds.d.smithi084.stdout:starting mds.d at 2024-03-19T01:50:30.817 INFO:tasks.ceph.mds.b.smithi084.stdout:starting mds.b at 2024-03-19T01:50:31.001 INFO:tasks.ceph.mds.e.smithi055.stdout:starting mds.e at 2024-03-19T01:50:32.102 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:50:32.217 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:32.218+0000 7f97b8bf8640 1 Processor -- start 2024-03-19T01:50:32.217 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:32.218+0000 7f97b8bf8640 1 -- start start 2024-03-19T01:50:32.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:32.218+0000 7f97b8bf8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f97b40c73c0 0x7f97b40c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:32.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:32.218+0000 7f97b8bf8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b40c3f60 con 0x7f97b40c61c0 2024-03-19T01:50:32.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:32.218+0000 7f97b8bf8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b4070fb0 con 0x7f97b41510f0 2024-03-19T01:50:32.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:32.218+0000 7f97b8bf8640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f97b40c6630 con 0x7f97b40c73c0 2024-03-19T01:50:32.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:32.218+0000 7f97b2d9d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f97b40c73c0 0x7f97b40c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:32.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:32.218+0000 7f97b1d9b640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f97b40c61c0 0x7f97b40708a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:42424/0 (socket says 172.21.15.55:42424) 2024-03-19T01:50:32.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:32.218+0000 7f97b1d9b640 1 -- 172.21.15.55:0/2447202556 learned_addr learned my addr 172.21.15.55:0/2447202556 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:50:32.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:32.222+0000 7f97b2d9d640 1 -- 172.21.15.55:0/2447202556 >> v1:172.21.15.84:6789/0 conn(0x7f97b41510f0 legacy=0x7f97b415a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:32.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:32.222+0000 7f97b2d9d640 1 -- 172.21.15.55:0/2447202556 >> v1:172.21.15.55:6789/0 conn(0x7f97b40c61c0 legacy=0x7f97b40708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:32.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:32.222+0000 7f97b2d9d640 1 -- 172.21.15.55:0/2447202556 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b41514d0 con 0x7f97b40c73c0 2024-03-19T01:50:32.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:32.226+0000 7f97b2d9d640 1 --2- 172.21.15.55:0/2447202556 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97b40c73c0 0x7f97b40c77f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f97ac009d90 tx=0x7f97ac01f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df3efd5ac3de42ca server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:33.641 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.638+0000 7f97b159a640 1 -- 172.21.15.55:0/2447202556 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac0b4070 con 0x7f97b40c73c0 2024-03-19T01:50:33.641 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.642+0000 7f97b159a640 1 -- 172.21.15.55:0/2447202556 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97ac0af600 con 0x7f97b40c73c0 2024-03-19T01:50:33.641 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.642+0000 7f97b159a640 1 -- 172.21.15.55:0/2447202556 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac0af9d0 con 0x7f97b40c73c0 2024-03-19T01:50:33.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.642+0000 7f97b8bf8640 1 -- 172.21.15.55:0/2447202556 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97b40c73c0 msgr2=0x7f97b40c77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:33.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.642+0000 7f97b8bf8640 1 --2- 172.21.15.55:0/2447202556 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97b40c73c0 0x7f97b40c77f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f97ac009d90 tx=0x7f97ac01f150 comp rx=0 tx=0).stop 2024-03-19T01:50:33.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.642+0000 7f97b8bf8640 1 -- 172.21.15.55:0/2447202556 shutdown_connections 2024-03-19T01:50:33.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.642+0000 7f97b8bf8640 1 --2- 172.21.15.55:0/2447202556 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97b40c73c0 0x7f97b40c77f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:33.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.642+0000 7f97b8bf8640 1 -- 172.21.15.55:0/2447202556 >> 172.21.15.55:0/2447202556 conn(0x7f97b40cc9c0 msgr2=0x7f97b40ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:33.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.642+0000 7f97b8bf8640 1 -- 172.21.15.55:0/2447202556 shutdown_connections 2024-03-19T01:50:33.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.642+0000 7f97b8bf8640 1 -- 172.21.15.55:0/2447202556 wait complete. 2024-03-19T01:50:33.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.646+0000 7f97b8bf8640 1 Processor -- start 2024-03-19T01:50:33.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.646+0000 7f97b8bf8640 1 -- start start 2024-03-19T01:50:33.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.646+0000 7f97b8bf8640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f97b40c61c0 0x7f97b40744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:33.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.646+0000 7f97b8bf8640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97b40c73c0 0x7f97b41e1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:33.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.646+0000 7f97b8bf8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97b41510f0 0x7f97b41f0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:33.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.646+0000 7f97b8bf8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f97b40c4200 con 0x7f97b41510f0 2024-03-19T01:50:33.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.646+0000 7f97b8bf8640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f97b40c6b50 con 0x7f97b40c73c0 2024-03-19T01:50:33.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.646+0000 7f97b8bf8640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f97b40c44e0 con 0x7f97b40c61c0 2024-03-19T01:50:33.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.646+0000 7f97b1d9b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97b40c73c0 0x7f97b41e1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:33.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.646+0000 7f97b259c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f97b40c61c0 0x7f97b40744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:33.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.650+0000 7f97b1d9b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97b40c73c0 0x7f97b41e1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:47192/0 (socket says 172.21.15.55:47192) 2024-03-19T01:50:33.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.650+0000 7f97b1d9b640 1 -- 172.21.15.55:0/3250542037 learned_addr learned my addr 172.21.15.55:0/3250542037 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:50:33.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.658+0000 7f97b1d9b640 1 -- 172.21.15.55:0/3250542037 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f97b40c61c0 msgr2=0x7f97b40744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:33.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.658+0000 7f97b1d9b640 1 --2- 172.21.15.55:0/3250542037 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f97b40c61c0 0x7f97b40744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:33.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.658+0000 7f97b1d9b640 1 -- 172.21.15.55:0/3250542037 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97b41510f0 msgr2=0x7f97b41f0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:33.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.658+0000 7f97b1d9b640 1 --2- 172.21.15.55:0/3250542037 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97b41510f0 0x7f97b41f0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:33.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.658+0000 7f97b1d9b640 1 -- 172.21.15.55:0/3250542037 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b41f3c10 con 0x7f97b40c73c0 2024-03-19T01:50:33.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.662+0000 7f97b1d9b640 1 --2- 172.21.15.55:0/3250542037 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97b40c73c0 0x7f97b41e1c90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f97a800c450 tx=0x7f97a800c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:33.717 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.714+0000 7f97a2ffd640 1 -- 172.21.15.55:0/3250542037 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a8017070 con 0x7f97b40c73c0 2024-03-19T01:50:33.717 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.714+0000 7f97a2ffd640 1 -- 172.21.15.55:0/3250542037 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97a8005cf0 con 0x7f97b40c73c0 2024-03-19T01:50:33.717 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.714+0000 7f97a2ffd640 1 -- 172.21.15.55:0/3250542037 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a80072f0 con 0x7f97b40c73c0 2024-03-19T01:50:33.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.714+0000 7f97b8bf8640 1 -- 172.21.15.55:0/3250542037 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97b41f3ef0 con 0x7f97b40c73c0 2024-03-19T01:50:33.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.718+0000 7f97b8bf8640 1 -- 172.21.15.55:0/3250542037 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97b41f44b0 con 0x7f97b40c73c0 2024-03-19T01:50:33.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:33.718+0000 7f97b8bf8640 1 -- 172.21.15.55:0/3250542037 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97b4068a40 con 0x7f97b40c73c0 2024-03-19T01:50:33.981 INFO:tasks.ceph.mds.a.smithi055.stdout:starting mds.a at 2024-03-19T01:50:33.986 INFO:tasks.ceph.mds.f.smithi084.stdout:starting mds.f at 2024-03-19T01:50:34.029 INFO:tasks.ceph.mds.c.smithi055.stdout:starting mds.c at 2024-03-19T01:50:34.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:34.058+0000 7f97a2ffd640 1 -- 172.21.15.55:0/3250542037 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 4) ==== 87027+0+0 (secure 0 0 0) 0x7f97a801a070 con 0x7f97b40c73c0 2024-03-19T01:50:34.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:34.058+0000 7f97a2ffd640 1 --2- 172.21.15.55:0/3250542037 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9794067620 0x7f9794069ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:34.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:34.058+0000 7f97b259c640 1 --2- 172.21.15.55:0/3250542037 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9794067620 0x7f9794069ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:34.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:34.062+0000 7f97b259c640 1 --2- 172.21.15.55:0/3250542037 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9794067620 0x7f9794069ae0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f97a4007fd0 tx=0x7f97a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:34.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:34.066+0000 7f97a2ffd640 1 -- 172.21.15.55:0/3250542037 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f97a805b920 con 0x7f97b40c73c0 2024-03-19T01:50:34.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:34.090+0000 7f97a2ffd640 1 -- 172.21.15.55:0/3250542037 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f97a8021020 con 0x7f97b40c73c0 2024-03-19T01:50:34.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:34.334+0000 7f97b8bf8640 1 -- 172.21.15.55:0/3250542037 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97b41e33c0 con 0x7f97b40c73c0 2024-03-19T01:50:38.188 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:50:38.191 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:50:40.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:40.782+0000 7f97a2ffd640 1 -- 172.21.15.55:0/3250542037 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f97a8051c70 con 0x7f97b40c73c0 2024-03-19T01:50:40.882 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:40.882+0000 7f97a2ffd640 1 -- 172.21.15.55:0/3250542037 <== mon.1 v2:172.21.15.84:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f97a80786d0 con 0x7f97b40c73c0 2024-03-19T01:50:40.883 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:50:40.883 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-19T01:50:40.883 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 1 2024-03-19T01:50:40.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:40.886+0000 7f97b8bf8640 1 -- 172.21.15.55:0/3250542037 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9794067620 msgr2=0x7f9794069ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:40.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:40.886+0000 7f97b8bf8640 1 --2- 172.21.15.55:0/3250542037 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9794067620 0x7f9794069ae0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f97a4007fd0 tx=0x7f97a4009040 comp rx=0 tx=0).stop 2024-03-19T01:50:40.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:40.886+0000 7f97b8bf8640 1 -- 172.21.15.55:0/3250542037 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97b40c73c0 msgr2=0x7f97b41e1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:40.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:40.886+0000 7f97b8bf8640 1 --2- 172.21.15.55:0/3250542037 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97b40c73c0 0x7f97b41e1c90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f97a800c450 tx=0x7f97a800c910 comp rx=0 tx=0).stop 2024-03-19T01:50:40.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:40.886+0000 7f97b8bf8640 1 -- 172.21.15.55:0/3250542037 shutdown_connections 2024-03-19T01:50:40.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:40.886+0000 7f97b8bf8640 1 --2- 172.21.15.55:0/3250542037 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9794067620 0x7f9794069ae0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:40.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:40.886+0000 7f97b8bf8640 1 --2- 172.21.15.55:0/3250542037 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97b41510f0 0x7f97b41f0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:40.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:40.886+0000 7f97b8bf8640 1 --2- 172.21.15.55:0/3250542037 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97b40c73c0 0x7f97b41e1c90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:40.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:40.886+0000 7f97b8bf8640 1 --2- 172.21.15.55:0/3250542037 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f97b40c61c0 0x7f97b40744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:40.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:40.886+0000 7f97b8bf8640 1 -- 172.21.15.55:0/3250542037 >> 172.21.15.55:0/3250542037 conn(0x7f97b40cc9c0 msgr2=0x7f97b40ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:40.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:40.886+0000 7f97b8bf8640 1 -- 172.21.15.55:0/3250542037 shutdown_connections 2024-03-19T01:50:40.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:40.886+0000 7f97b8bf8640 1 -- 172.21.15.55:0/3250542037 wait complete. 2024-03-19T01:50:42.910 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:50:43.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.046+0000 7fa0a9e14640 1 Processor -- start 2024-03-19T01:50:43.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.046+0000 7fa0a9e14640 1 -- start start 2024-03-19T01:50:43.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.046+0000 7fa0a9e14640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa0a40ca5b0 0x7fa0a40c2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:43.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.046+0000 7fa0a9e14640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0a40c8a50 con 0x7fa0a41589d0 2024-03-19T01:50:43.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.046+0000 7fa0a9e14640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0a40c9560 con 0x7fa0a415bc50 2024-03-19T01:50:43.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.046+0000 7fa0a9e14640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fa0a40c9b20 con 0x7fa0a40ca5b0 2024-03-19T01:50:43.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.046+0000 7fa0a2ffd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fa0a41589d0 0x7fa0a415b540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:37104/0 (socket says 172.21.15.55:37104) 2024-03-19T01:50:43.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.046+0000 7fa0a2ffd640 1 -- 172.21.15.55:0/822804985 learned_addr learned my addr 172.21.15.55:0/822804985 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:50:43.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.046+0000 7fa0a37fe640 1 --2- 172.21.15.55:0/822804985 >> v2:172.21.15.84:3300/0 conn(0x7fa0a40ca5b0 0x7fa0a40c2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:43.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.050+0000 7fa0a27fc640 1 -- 172.21.15.55:0/822804985 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3301590240 0 0) 0x7fa0a40c8a50 con 0x7fa0a41589d0 2024-03-19T01:50:43.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.050+0000 7fa0a27fc640 1 -- 172.21.15.55:0/822804985 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa08c003530 con 0x7fa0a41589d0 2024-03-19T01:50:43.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.054+0000 7fa0a27fc640 1 -- 172.21.15.55:0/822804985 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1249282864 0 0) 0x7fa08c003530 con 0x7fa0a41589d0 2024-03-19T01:50:43.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.054+0000 7fa0a27fc640 1 -- 172.21.15.55:0/822804985 >> v2:172.21.15.84:3300/0 conn(0x7fa0a40ca5b0 msgr2=0x7fa0a40c2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:43.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.054+0000 7fa0a27fc640 1 --2- 172.21.15.55:0/822804985 >> v2:172.21.15.84:3300/0 conn(0x7fa0a40ca5b0 0x7fa0a40c2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:43.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.054+0000 7fa0a27fc640 1 -- 172.21.15.55:0/822804985 >> v1:172.21.15.84:6789/0 conn(0x7fa0a415bc50 legacy=0x7fa0a41628d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:43.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.054+0000 7fa0a27fc640 1 -- 172.21.15.55:0/822804985 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0a40c9840 con 0x7fa0a41589d0 2024-03-19T01:50:43.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.054+0000 7fa0a27fc640 1 -- 172.21.15.55:0/822804985 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7fa098001680 con 0x7fa0a41589d0 2024-03-19T01:50:43.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.054+0000 7fa0a27fc640 1 -- 172.21.15.55:0/822804985 >> v1:172.21.15.55:6789/0 conn(0x7fa0a41589d0 legacy=0x7fa0a415b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:43.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.054+0000 7fa0a27fc640 1 --2- 172.21.15.55:0/822804985 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa08c003a10 0x7fa08c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:43.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.054+0000 7fa0a27fc640 1 --2- 172.21.15.55:0/822804985 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa08c004810 0x7fa08c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:43.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.054+0000 7fa0a27fc640 1 --2- 172.21.15.55:0/822804985 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa08c005220 0x7fa08c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:43.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.054+0000 7fa0a27fc640 1 -- 172.21.15.55:0/822804985 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa0a40c9b20 con 0x7fa08c003a10 2024-03-19T01:50:43.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.054+0000 7fa0a27fc640 1 -- 172.21.15.55:0/822804985 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa08c00c150 con 0x7fa08c005220 2024-03-19T01:50:43.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.054+0000 7fa0a27fc640 1 -- 172.21.15.55:0/822804985 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa08c00c350 con 0x7fa08c004810 2024-03-19T01:50:43.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.054+0000 7fa0a37fe640 1 --2- 172.21.15.55:0/822804985 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa08c003a10 0x7fa08c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:43.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.054+0000 7fa0a2ffd640 1 --2- 172.21.15.55:0/822804985 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa08c004810 0x7fa08c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:43.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.058+0000 7fa0a2ffd640 1 -- 172.21.15.55:0/822804985 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa08c005220 msgr2=0x7fa08c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:43.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.058+0000 7fa0a2ffd640 1 --2- 172.21.15.55:0/822804985 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa08c005220 0x7fa08c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:43.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.058+0000 7fa0a2ffd640 1 -- 172.21.15.55:0/822804985 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa08c003a10 msgr2=0x7fa08c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:43.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.058+0000 7fa0a2ffd640 1 --2- 172.21.15.55:0/822804985 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa08c003a10 0x7fa08c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:43.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.058+0000 7fa0a2ffd640 1 -- 172.21.15.55:0/822804985 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa08c003530 con 0x7fa08c004810 2024-03-19T01:50:43.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.058+0000 7fa0a3fff640 1 -- 172.21.15.55:0/822804985 reap_dead start 2024-03-19T01:50:43.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.066+0000 7fa0a2ffd640 1 --2- 172.21.15.55:0/822804985 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa08c004810 0x7fa08c004c90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa09800c160 tx=0x7fa09801cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5f8c812284eb8d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:43.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.074+0000 7fa0a27fc640 1 -- 172.21.15.55:0/822804985 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa098007c90 con 0x7fa08c004810 2024-03-19T01:50:43.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.074+0000 7fa0a27fc640 1 -- 172.21.15.55:0/822804985 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0980a9070 con 0x7fa08c004810 2024-03-19T01:50:43.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.074+0000 7fa0a27fc640 1 -- 172.21.15.55:0/822804985 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa098005dc0 con 0x7fa08c004810 2024-03-19T01:50:43.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.074+0000 7fa0a9e14640 1 -- 172.21.15.55:0/822804985 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa08c004810 msgr2=0x7fa08c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:43.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.074+0000 7fa0a9e14640 1 --2- 172.21.15.55:0/822804985 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa08c004810 0x7fa08c004c90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa09800c160 tx=0x7fa09801cb90 comp rx=0 tx=0).stop 2024-03-19T01:50:43.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.074+0000 7fa0a9e14640 1 -- 172.21.15.55:0/822804985 shutdown_connections 2024-03-19T01:50:43.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.074+0000 7fa0a9e14640 1 --2- 172.21.15.55:0/822804985 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa08c004810 0x7fa08c004c90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:43.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.074+0000 7fa0a9e14640 1 -- 172.21.15.55:0/822804985 >> 172.21.15.55:0/822804985 conn(0x7fa0a40715e0 msgr2=0x7fa0a40719f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:43.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.074+0000 7fa0a9e14640 1 -- 172.21.15.55:0/822804985 shutdown_connections 2024-03-19T01:50:43.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.074+0000 7fa0a9e14640 1 -- 172.21.15.55:0/822804985 wait complete. 2024-03-19T01:50:43.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.078+0000 7fa0a9e14640 1 Processor -- start 2024-03-19T01:50:43.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.078+0000 7fa0a9e14640 1 -- start start 2024-03-19T01:50:43.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.078+0000 7fa0a9e14640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa08c004810 0x7fa0a41e6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:43.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.078+0000 7fa0a9e14640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa0a414fb50 0x7fa0a41f2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:43.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.078+0000 7fa0a9e14640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0a41e7150 0x7fa0a41f2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:43.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.078+0000 7fa0a9e14640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa0a40cad20 con 0x7fa0a41e7150 2024-03-19T01:50:43.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.078+0000 7fa0a9e14640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa0a40caba0 con 0x7fa0a414fb50 2024-03-19T01:50:43.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.078+0000 7fa0a9e14640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa0a40caea0 con 0x7fa08c004810 2024-03-19T01:50:43.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.078+0000 7fa0a3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0a41e7150 0x7fa0a41f2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:43.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.078+0000 7fa0a3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0a41e7150 0x7fa0a41f2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57022/0 (socket says 172.21.15.55:57022) 2024-03-19T01:50:43.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.078+0000 7fa0a3fff640 1 -- 172.21.15.55:0/2611619236 learned_addr learned my addr 172.21.15.55:0/2611619236 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:50:43.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.078+0000 7fa0a2ffd640 1 --2- 172.21.15.55:0/2611619236 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa0a414fb50 0x7fa0a41f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:43.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.082+0000 7fa0a37fe640 1 --2- 172.21.15.55:0/2611619236 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa08c004810 0x7fa0a41e6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:43.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.082+0000 7fa0a3fff640 1 -- 172.21.15.55:0/2611619236 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa08c004810 msgr2=0x7fa0a41e6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:43.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.082+0000 7fa0a3fff640 1 --2- 172.21.15.55:0/2611619236 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa08c004810 0x7fa0a41e6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:43.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.082+0000 7fa0a3fff640 1 -- 172.21.15.55:0/2611619236 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa0a414fb50 msgr2=0x7fa0a41f2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:43.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.082+0000 7fa0a3fff640 1 --2- 172.21.15.55:0/2611619236 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa0a414fb50 0x7fa0a41f2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:43.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.082+0000 7fa0a3fff640 1 -- 172.21.15.55:0/2611619236 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0a415a2b0 con 0x7fa0a41e7150 2024-03-19T01:50:43.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.086+0000 7fa0a3fff640 1 --2- 172.21.15.55:0/2611619236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0a41e7150 0x7fa0a41f2e60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4155b60 tx=0x7fa09c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:43.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.090+0000 7fa08bfff640 1 -- 172.21.15.55:0/2611619236 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa09c017070 con 0x7fa0a41e7150 2024-03-19T01:50:43.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.090+0000 7fa08bfff640 1 -- 172.21.15.55:0/2611619236 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa09c0070d0 con 0x7fa0a41e7150 2024-03-19T01:50:43.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.090+0000 7fa08bfff640 1 -- 172.21.15.55:0/2611619236 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa09c0074c0 con 0x7fa0a41e7150 2024-03-19T01:50:43.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.090+0000 7fa0a9e14640 1 -- 172.21.15.55:0/2611619236 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0a4153790 con 0x7fa0a41e7150 2024-03-19T01:50:43.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.090+0000 7fa0a9e14640 1 -- 172.21.15.55:0/2611619236 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0a4153d30 con 0x7fa0a41e7150 2024-03-19T01:50:43.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.090+0000 7fa0a9e14640 1 -- 172.21.15.55:0/2611619236 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0a4068a40 con 0x7fa0a41e7150 2024-03-19T01:50:43.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.390+0000 7fa08bfff640 1 -- 172.21.15.55:0/2611619236 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa09c01a070 con 0x7fa0a41e7150 2024-03-19T01:50:43.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.390+0000 7fa08bfff640 1 --2- 172.21.15.55:0/2611619236 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa0900676e0 0x7fa090069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:43.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.390+0000 7fa0a37fe640 1 --2- 172.21.15.55:0/2611619236 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa0900676e0 0x7fa090069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:43.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.390+0000 7fa0a37fe640 1 --2- 172.21.15.55:0/2611619236 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa0900676e0 0x7fa090069ba0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa094009ba0 tx=0x7fa094008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:43.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.430+0000 7fa08bfff640 1 -- 172.21.15.55:0/2611619236 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7fa09c053670 con 0x7fa0a41e7150 2024-03-19T01:50:43.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.430+0000 7fa08bfff640 1 -- 172.21.15.55:0/2611619236 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa09c021020 con 0x7fa0a41e7150 2024-03-19T01:50:43.641 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.642+0000 7fa0a9e14640 1 -- 172.21.15.55:0/2611619236 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0a40c9210 con 0x7fa0a41e7150 2024-03-19T01:50:43.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.666+0000 7fa08bfff640 1 -- 172.21.15.55:0/2611619236 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1392 (secure 0 0 0) 0x7fa09c0555d0 con 0x7fa0a41e7150 2024-03-19T01:50:43.666 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:50:43.666 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4237,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-19T01:50:43.667 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 2 2024-03-19T01:50:43.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.666+0000 7fa0a9e14640 1 -- 172.21.15.55:0/2611619236 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa0900676e0 msgr2=0x7fa090069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:43.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.666+0000 7fa0a9e14640 1 --2- 172.21.15.55:0/2611619236 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa0900676e0 0x7fa090069ba0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa094009ba0 tx=0x7fa094008040 comp rx=0 tx=0).stop 2024-03-19T01:50:43.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.666+0000 7fa0a9e14640 1 -- 172.21.15.55:0/2611619236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0a41e7150 msgr2=0x7fa0a41f2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:43.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.666+0000 7fa0a9e14640 1 --2- 172.21.15.55:0/2611619236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0a41e7150 0x7fa0a41f2e60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4155b60 tx=0x7fa09c00c960 comp rx=0 tx=0).stop 2024-03-19T01:50:43.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.666+0000 7fa0a9e14640 1 -- 172.21.15.55:0/2611619236 shutdown_connections 2024-03-19T01:50:43.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.666+0000 7fa0a9e14640 1 --2- 172.21.15.55:0/2611619236 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa0900676e0 0x7fa090069ba0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:43.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.666+0000 7fa0a9e14640 1 --2- 172.21.15.55:0/2611619236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0a41e7150 0x7fa0a41f2e60 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:43.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.666+0000 7fa0a9e14640 1 --2- 172.21.15.55:0/2611619236 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa0a414fb50 0x7fa0a41f2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:43.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.666+0000 7fa0a9e14640 1 --2- 172.21.15.55:0/2611619236 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa08c004810 0x7fa0a41e6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:43.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.666+0000 7fa0a9e14640 1 -- 172.21.15.55:0/2611619236 >> 172.21.15.55:0/2611619236 conn(0x7fa0a40715e0 msgr2=0x7fa0a4151730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:43.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.666+0000 7fa0a9e14640 1 -- 172.21.15.55:0/2611619236 shutdown_connections 2024-03-19T01:50:43.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:43.666+0000 7fa0a9e14640 1 -- 172.21.15.55:0/2611619236 wait complete. 2024-03-19T01:50:45.687 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:50:45.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:45.822+0000 7f324e668640 1 Processor -- start 2024-03-19T01:50:45.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:45.822+0000 7f324e668640 1 -- start start 2024-03-19T01:50:45.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:45.822+0000 7f324e668640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f324814d6a0 0x7f32480c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:45.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:45.822+0000 7f324e668640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f32481577f0 0x7f32480c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:45.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:45.826+0000 7f324e668640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32480c6e80 con 0x7f32480c7900 2024-03-19T01:50:45.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:45.826+0000 7f324e668640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f32480c4ca0 con 0x7f324814d6a0 2024-03-19T01:50:45.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:45.826+0000 7f324e668640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f32480c6300 con 0x7f32481577f0 2024-03-19T01:50:45.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:45.826+0000 7f32477fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f32480c7900 0x7f32481570e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:37114/0 (socket says 172.21.15.55:37114) 2024-03-19T01:50:45.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:45.826+0000 7f32477fe640 1 -- 172.21.15.55:0/3064219278 learned_addr learned my addr 172.21.15.55:0/3064219278 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:50:45.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:45.826+0000 7f3247fff640 1 --2- 172.21.15.55:0/3064219278 >> v2:172.21.15.84:3300/0 conn(0x7f324814d6a0 0x7f32480c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:45.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:45.830+0000 7f3246ffd640 1 -- 172.21.15.55:0/3064219278 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1856854996 0 0) 0x7f32480c6e80 con 0x7f32480c7900 2024-03-19T01:50:45.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:45.830+0000 7f3246ffd640 1 -- 172.21.15.55:0/3064219278 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3230003530 con 0x7f32480c7900 2024-03-19T01:50:45.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:45.834+0000 7f3247fff640 1 -- 172.21.15.55:0/3064219278 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f32481577f0 msgr2=0x7f32480c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:45.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:45.834+0000 7f3247fff640 1 --2- 172.21.15.55:0/3064219278 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f32481577f0 0x7f32480c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:45.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:45.834+0000 7f3247fff640 1 -- 172.21.15.55:0/3064219278 >> v1:172.21.15.55:6789/0 conn(0x7f32480c7900 legacy=0x7f32481570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:45.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:45.834+0000 7f3247fff640 1 -- 172.21.15.55:0/3064219278 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32480c65e0 con 0x7f324814d6a0 2024-03-19T01:50:45.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:45.842+0000 7f3247fff640 1 --2- 172.21.15.55:0/3064219278 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f324814d6a0 0x7f32480c73c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f32380097c0 tx=0x7f323801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32acfee22c700a9b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:50.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.122+0000 7f3246ffd640 1 -- 172.21.15.55:0/3064219278 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32380aa600 con 0x7f324814d6a0 2024-03-19T01:50:50.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.122+0000 7f3246ffd640 1 -- 172.21.15.55:0/3064219278 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32380af070 con 0x7f324814d6a0 2024-03-19T01:50:50.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.122+0000 7f3246ffd640 1 -- 172.21.15.55:0/3064219278 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32380aaa40 con 0x7f324814d6a0 2024-03-19T01:50:50.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.122+0000 7f324e668640 1 -- 172.21.15.55:0/3064219278 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f324814d6a0 msgr2=0x7f32480c73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:50.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.122+0000 7f324e668640 1 --2- 172.21.15.55:0/3064219278 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f324814d6a0 0x7f32480c73c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f32380097c0 tx=0x7f323801c4a0 comp rx=0 tx=0).stop 2024-03-19T01:50:50.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.126+0000 7f324e668640 1 -- 172.21.15.55:0/3064219278 shutdown_connections 2024-03-19T01:50:50.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.126+0000 7f324e668640 1 --2- 172.21.15.55:0/3064219278 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f32481577f0 0x7f32480c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:50.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.126+0000 7f324e668640 1 --2- 172.21.15.55:0/3064219278 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f324814d6a0 0x7f32480c73c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:50.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.126+0000 7f324e668640 1 -- 172.21.15.55:0/3064219278 >> 172.21.15.55:0/3064219278 conn(0x7f324806c3b0 msgr2=0x7f324806c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:50.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.126+0000 7f324e668640 1 -- 172.21.15.55:0/3064219278 shutdown_connections 2024-03-19T01:50:50.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.126+0000 7f324e668640 1 -- 172.21.15.55:0/3064219278 wait complete. 2024-03-19T01:50:50.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.126+0000 7f324e668640 1 Processor -- start 2024-03-19T01:50:50.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.126+0000 7f324e668640 1 -- start start 2024-03-19T01:50:50.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.130+0000 7f324e668640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32480c7900 0x7f3248151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:50.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.130+0000 7f324e668640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f324814d6a0 0x7f3248152460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:50.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.130+0000 7f324e668640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32481577f0 0x7f32481431d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:50.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.130+0000 7f324e668640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f324814dc00 con 0x7f32480c7900 2024-03-19T01:50:50.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.130+0000 7f324e668640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f324814da80 con 0x7f32481577f0 2024-03-19T01:50:50.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.130+0000 7f324e668640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f32480c4810 con 0x7f324814d6a0 2024-03-19T01:50:50.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.130+0000 7f32477fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f324814d6a0 0x7f3248152460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:50.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.130+0000 7f324cc05640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32481577f0 0x7f32481431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:50.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.130+0000 7f32477fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f324814d6a0 0x7f3248152460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:43008/0 (socket says 172.21.15.55:43008) 2024-03-19T01:50:50.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.130+0000 7f32477fe640 1 -- 172.21.15.55:0/1684871681 learned_addr learned my addr 172.21.15.55:0/1684871681 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:50:50.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.146+0000 7f32477fe640 1 -- 172.21.15.55:0/1684871681 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32481577f0 msgr2=0x7f32481431d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:50.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.146+0000 7f32477fe640 1 --2- 172.21.15.55:0/1684871681 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32481577f0 0x7f32481431d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:50.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.146+0000 7f32477fe640 1 -- 172.21.15.55:0/1684871681 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32480c7900 msgr2=0x7f3248151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:50.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.146+0000 7f32477fe640 1 --2- 172.21.15.55:0/1684871681 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32480c7900 0x7f3248151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:50.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.146+0000 7f32477fe640 1 -- 172.21.15.55:0/1684871681 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32481582e0 con 0x7f324814d6a0 2024-03-19T01:50:50.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.150+0000 7f32477fe640 1 --2- 172.21.15.55:0/1684871681 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f324814d6a0 0x7f3248152460 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f323c00c400 tx=0x7f323c00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:50.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.158+0000 7f3244ff9640 1 -- 172.21.15.55:0/1684871681 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323c017070 con 0x7f324814d6a0 2024-03-19T01:50:50.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.158+0000 7f3244ff9640 1 -- 172.21.15.55:0/1684871681 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f323c005d10 con 0x7f324814d6a0 2024-03-19T01:50:50.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.158+0000 7f3244ff9640 1 -- 172.21.15.55:0/1684871681 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323c007310 con 0x7f324814d6a0 2024-03-19T01:50:50.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.158+0000 7f324e668640 1 -- 172.21.15.55:0/1684871681 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32480caf30 con 0x7f324814d6a0 2024-03-19T01:50:50.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.158+0000 7f324e668640 1 -- 172.21.15.55:0/1684871681 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f32481544d0 con 0x7f324814d6a0 2024-03-19T01:50:50.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.162+0000 7f324e668640 1 -- 172.21.15.55:0/1684871681 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3248068a40 con 0x7f324814d6a0 2024-03-19T01:50:50.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.466+0000 7f3244ff9640 1 -- 172.21.15.55:0/1684871681 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f323c01a070 con 0x7f324814d6a0 2024-03-19T01:50:50.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.466+0000 7f3244ff9640 1 --2- 172.21.15.55:0/1684871681 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f32340676b0 0x7f3234069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:50.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.466+0000 7f3247fff640 1 --2- 172.21.15.55:0/1684871681 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f32340676b0 0x7f3234069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:50.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.470+0000 7f3247fff640 1 --2- 172.21.15.55:0/1684871681 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f32340676b0 0x7f3234069b70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f32380ae000 tx=0x7f3238004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:50.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.474+0000 7f3244ff9640 1 -- 172.21.15.55:0/1684871681 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f323c053720 con 0x7f324814d6a0 2024-03-19T01:50:50.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.506+0000 7f3244ff9640 1 -- 172.21.15.55:0/1684871681 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f323c021020 con 0x7f324814d6a0 2024-03-19T01:50:50.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:50.810+0000 7f324e668640 1 -- 172.21.15.55:0/1684871681 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32480c5960 con 0x7f324814d6a0 2024-03-19T01:50:55.388 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:55.386+0000 7f3244ff9640 1 -- 172.21.15.55:0/1684871681 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3864 (secure 0 0 0) 0x7f323c055680 con 0x7f324814d6a0 2024-03-19T01:50:55.389 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:50:55.389 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4200,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4237,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-19T01:50:55.389 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 3 2024-03-19T01:50:55.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:55.390+0000 7f324e668640 1 -- 172.21.15.55:0/1684871681 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f32340676b0 msgr2=0x7f3234069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:55.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:55.390+0000 7f324e668640 1 --2- 172.21.15.55:0/1684871681 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f32340676b0 0x7f3234069b70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f32380ae000 tx=0x7f3238004740 comp rx=0 tx=0).stop 2024-03-19T01:50:55.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:55.390+0000 7f324e668640 1 -- 172.21.15.55:0/1684871681 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f324814d6a0 msgr2=0x7f3248152460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:55.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:55.390+0000 7f324e668640 1 --2- 172.21.15.55:0/1684871681 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f324814d6a0 0x7f3248152460 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f323c00c400 tx=0x7f323c00c8c0 comp rx=0 tx=0).stop 2024-03-19T01:50:55.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:55.390+0000 7f324e668640 1 -- 172.21.15.55:0/1684871681 shutdown_connections 2024-03-19T01:50:55.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:55.390+0000 7f324e668640 1 --2- 172.21.15.55:0/1684871681 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f32340676b0 0x7f3234069b70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:55.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:55.390+0000 7f324e668640 1 --2- 172.21.15.55:0/1684871681 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32481577f0 0x7f32481431d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:55.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:55.390+0000 7f324e668640 1 --2- 172.21.15.55:0/1684871681 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f324814d6a0 0x7f3248152460 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:55.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:55.390+0000 7f324e668640 1 --2- 172.21.15.55:0/1684871681 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32480c7900 0x7f3248151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:55.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:55.390+0000 7f324e668640 1 -- 172.21.15.55:0/1684871681 >> 172.21.15.55:0/1684871681 conn(0x7f324806c3b0 msgr2=0x7f3248073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:55.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:55.390+0000 7f324e668640 1 -- 172.21.15.55:0/1684871681 shutdown_connections 2024-03-19T01:50:55.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:55.390+0000 7f324e668640 1 -- 172.21.15.55:0/1684871681 wait complete. 2024-03-19T01:50:57.413 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:50:57.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.546+0000 7f2f5641d640 1 Processor -- start 2024-03-19T01:50:57.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.546+0000 7f2f5641d640 1 -- start start 2024-03-19T01:50:57.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.550+0000 7f2f5641d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2f50154440 0x7f2f5015abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:57.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.550+0000 7f2f5641d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f2f50071c20 0x7f2f500cb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:57.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.550+0000 7f2f5641d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f5015b0f0 con 0x7f2f50071810 2024-03-19T01:50:57.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.550+0000 7f2f5641d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f2f500c7df0 con 0x7f2f50154440 2024-03-19T01:50:57.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.550+0000 7f2f5641d640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f2f500c8090 con 0x7f2f50071c20 2024-03-19T01:50:57.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.550+0000 7f2f549ba640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f2f50071c20 0x7f2f500cb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:57.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.550+0000 7f2f4ffff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2f50154440 0x7f2f5015abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:57.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.550+0000 7f2f4ffff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2f50154440 0x7f2f5015abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60526/0 (socket says 172.21.15.55:60526) 2024-03-19T01:50:57.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.550+0000 7f2f549ba640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f2f50071c20 0x7f2f500cb850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:36144/0 (socket says 172.21.15.55:36144) 2024-03-19T01:50:57.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.550+0000 7f2f4ffff640 1 -- 172.21.15.55:0/3567173002 learned_addr learned my addr 172.21.15.55:0/3567173002 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:50:57.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.554+0000 7f2f4ffff640 1 -- 172.21.15.55:0/3567173002 >> v2:172.21.15.84:3300/0 conn(0x7f2f50071c20 msgr2=0x7f2f500cb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:50:57.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.554+0000 7f2f4ffff640 1 --2- 172.21.15.55:0/3567173002 >> v2:172.21.15.84:3300/0 conn(0x7f2f50071c20 0x7f2f500cb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:57.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.554+0000 7f2f4ffff640 1 -- 172.21.15.55:0/3567173002 >> v1:172.21.15.55:6789/0 conn(0x7f2f50071810 legacy=0x7f2f500c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:57.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.554+0000 7f2f4ffff640 1 -- 172.21.15.55:0/3567173002 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f50154820 con 0x7f2f50154440 2024-03-19T01:50:57.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.562+0000 7f2f4ffff640 1 --2- 172.21.15.55:0/3567173002 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f50154440 0x7f2f5015abb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2f50157d00 tx=0x7f2f4001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9af210e628a1409c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:57.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.566+0000 7f2f4effd640 1 -- 172.21.15.55:0/3567173002 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f400b5070 con 0x7f2f50154440 2024-03-19T01:50:57.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.566+0000 7f2f4effd640 1 -- 172.21.15.55:0/3567173002 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f400a8a10 con 0x7f2f50154440 2024-03-19T01:50:57.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.566+0000 7f2f4effd640 1 -- 172.21.15.55:0/3567173002 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f400a8de0 con 0x7f2f50154440 2024-03-19T01:50:57.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.570+0000 7f2f5641d640 1 -- 172.21.15.55:0/3567173002 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f50154440 msgr2=0x7f2f5015abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:57.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.570+0000 7f2f5641d640 1 --2- 172.21.15.55:0/3567173002 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f50154440 0x7f2f5015abb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2f50157d00 tx=0x7f2f4001ed10 comp rx=0 tx=0).stop 2024-03-19T01:50:57.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.570+0000 7f2f5641d640 1 -- 172.21.15.55:0/3567173002 shutdown_connections 2024-03-19T01:50:57.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.570+0000 7f2f5641d640 1 --2- 172.21.15.55:0/3567173002 >> v2:172.21.15.84:3300/0 conn(0x7f2f50071c20 0x7f2f500cb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:57.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.570+0000 7f2f5641d640 1 --2- 172.21.15.55:0/3567173002 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f50154440 0x7f2f5015abb0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:57.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.570+0000 7f2f5641d640 1 -- 172.21.15.55:0/3567173002 >> 172.21.15.55:0/3567173002 conn(0x7f2f5006f3b0 msgr2=0x7f2f5006f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:57.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.570+0000 7f2f5641d640 1 -- 172.21.15.55:0/3567173002 shutdown_connections 2024-03-19T01:50:57.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.570+0000 7f2f5641d640 1 -- 172.21.15.55:0/3567173002 wait complete. 2024-03-19T01:50:57.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.570+0000 7f2f5641d640 1 Processor -- start 2024-03-19T01:50:57.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.570+0000 7f2f5641d640 1 -- start start 2024-03-19T01:50:57.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.570+0000 7f2f5641d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f50071810 0x7f2f500cb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:57.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.570+0000 7f2f5641d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f50071c20 0x7f2f500c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:57.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.570+0000 7f2f5641d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f50154440 0x7f2f500c50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:57.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.570+0000 7f2f5641d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2f500c88d0 con 0x7f2f50154440 2024-03-19T01:50:57.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.570+0000 7f2f5641d640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2f500c85f0 con 0x7f2f50071c20 2024-03-19T01:50:57.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.570+0000 7f2f5641d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2f500c8bb0 con 0x7f2f50071810 2024-03-19T01:50:57.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.570+0000 7f2f549ba640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f50154440 0x7f2f500c50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:57.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.574+0000 7f2f549ba640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f50154440 0x7f2f500c50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60532/0 (socket says 172.21.15.55:60532) 2024-03-19T01:50:57.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.574+0000 7f2f549ba640 1 -- 172.21.15.55:0/884653209 learned_addr learned my addr 172.21.15.55:0/884653209 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:50:57.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.574+0000 7f2f4f7fe640 1 --2- 172.21.15.55:0/884653209 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f50071c20 0x7f2f500c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:57.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.574+0000 7f2f4ffff640 1 --2- 172.21.15.55:0/884653209 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f50071810 0x7f2f500cb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:57.577 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.578+0000 7f2f549ba640 1 -- 172.21.15.55:0/884653209 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f50071810 msgr2=0x7f2f500cb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:57.577 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.578+0000 7f2f549ba640 1 --2- 172.21.15.55:0/884653209 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f50071810 0x7f2f500cb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:57.577 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.578+0000 7f2f549ba640 1 -- 172.21.15.55:0/884653209 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f50071c20 msgr2=0x7f2f500c4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:57.577 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.578+0000 7f2f549ba640 1 --2- 172.21.15.55:0/884653209 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f50071c20 0x7f2f500c4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:57.577 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.578+0000 7f2f549ba640 1 -- 172.21.15.55:0/884653209 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f50156490 con 0x7f2f50154440 2024-03-19T01:50:57.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.578+0000 7f2f549ba640 1 --2- 172.21.15.55:0/884653209 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f50154440 0x7f2f500c50a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2f48009d20 tx=0x7f2f4800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:57.585 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.582+0000 7f2f4cff9640 1 -- 172.21.15.55:0/884653209 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f48018020 con 0x7f2f50154440 2024-03-19T01:50:57.585 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.582+0000 7f2f4cff9640 1 -- 172.21.15.55:0/884653209 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f48009b10 con 0x7f2f50154440 2024-03-19T01:50:57.585 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.582+0000 7f2f4cff9640 1 -- 172.21.15.55:0/884653209 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f480146c0 con 0x7f2f50154440 2024-03-19T01:50:57.585 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.582+0000 7f2f5641d640 1 -- 172.21.15.55:0/884653209 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f501f0500 con 0x7f2f50154440 2024-03-19T01:50:57.585 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.582+0000 7f2f5641d640 1 -- 172.21.15.55:0/884653209 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f501f0ac0 con 0x7f2f50154440 2024-03-19T01:50:57.585 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.586+0000 7f2f5641d640 1 -- 172.21.15.55:0/884653209 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f50068330 con 0x7f2f50154440 2024-03-19T01:50:57.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.882+0000 7f2f4cff9640 1 -- 172.21.15.55:0/884653209 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f2f4801b020 con 0x7f2f50154440 2024-03-19T01:50:57.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.882+0000 7f2f4cff9640 1 --2- 172.21.15.55:0/884653209 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f300676b0 0x7f2f30069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:50:57.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.886+0000 7f2f4ffff640 1 --2- 172.21.15.55:0/884653209 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f300676b0 0x7f2f30069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:50:57.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.886+0000 7f2f4ffff640 1 --2- 172.21.15.55:0/884653209 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f300676b0 0x7f2f30069b70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2f400b2000 tx=0x7f2f400aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:50:57.919 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.918+0000 7f2f4cff9640 1 -- 172.21.15.55:0/884653209 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f2f480551a0 con 0x7f2f50154440 2024-03-19T01:50:57.919 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:57.918+0000 7f2f4cff9640 1 -- 172.21.15.55:0/884653209 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2f48022030 con 0x7f2f50154440 2024-03-19T01:50:58.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:58.130+0000 7f2f5641d640 1 -- 172.21.15.55:0/884653209 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f5015aed0 con 0x7f2f50154440 2024-03-19T01:50:58.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:58.154+0000 7f2f4cff9640 1 -- 172.21.15.55:0/884653209 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3864 (secure 0 0 0) 0x7f2f48010040 con 0x7f2f50154440 2024-03-19T01:50:58.154 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:50:58.154 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4200,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4237,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-19T01:50:58.155 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 3 2024-03-19T01:50:58.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:58.158+0000 7f2f5641d640 1 -- 172.21.15.55:0/884653209 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f300676b0 msgr2=0x7f2f30069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:58.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:58.158+0000 7f2f5641d640 1 --2- 172.21.15.55:0/884653209 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f300676b0 0x7f2f30069b70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2f400b2000 tx=0x7f2f400aa070 comp rx=0 tx=0).stop 2024-03-19T01:50:58.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:58.158+0000 7f2f5641d640 1 -- 172.21.15.55:0/884653209 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f50154440 msgr2=0x7f2f500c50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:50:58.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:58.158+0000 7f2f5641d640 1 --2- 172.21.15.55:0/884653209 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f50154440 0x7f2f500c50a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2f48009d20 tx=0x7f2f4800c490 comp rx=0 tx=0).stop 2024-03-19T01:50:58.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:58.158+0000 7f2f5641d640 1 -- 172.21.15.55:0/884653209 shutdown_connections 2024-03-19T01:50:58.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:58.158+0000 7f2f5641d640 1 --2- 172.21.15.55:0/884653209 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f300676b0 0x7f2f30069b70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:58.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:58.158+0000 7f2f5641d640 1 --2- 172.21.15.55:0/884653209 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f50154440 0x7f2f500c50a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:58.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:58.158+0000 7f2f5641d640 1 --2- 172.21.15.55:0/884653209 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f50071c20 0x7f2f500c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:58.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:58.158+0000 7f2f5641d640 1 --2- 172.21.15.55:0/884653209 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f50071810 0x7f2f500cb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:50:58.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:58.158+0000 7f2f5641d640 1 -- 172.21.15.55:0/884653209 >> 172.21.15.55:0/884653209 conn(0x7f2f5006f3b0 msgr2=0x7f2f50153770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:50:58.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:58.158+0000 7f2f5641d640 1 -- 172.21.15.55:0/884653209 shutdown_connections 2024-03-19T01:50:58.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:50:58.158+0000 7f2f5641d640 1 -- 172.21.15.55:0/884653209 wait complete. 2024-03-19T01:51:00.177 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:51:00.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.322+0000 7f92f50c0640 1 Processor -- start 2024-03-19T01:51:00.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.322+0000 7f92f50c0640 1 -- start start 2024-03-19T01:51:00.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.326+0000 7f92f50c0640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f92f0154300 0x7f92f015ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:00.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.326+0000 7f92f50c0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f92f006c110 0x7f92f006c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:00.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.326+0000 7f92f50c0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92f00c8840 con 0x7f92f006b2a0 2024-03-19T01:51:00.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.326+0000 7f92f50c0640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f92f015b260 con 0x7f92f0154300 2024-03-19T01:51:00.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.326+0000 7f92f50c0640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f92f00c8ab0 con 0x7f92f006c110 2024-03-19T01:51:00.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.326+0000 7f92ef59e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f92f006c110 0x7f92f006c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:00.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.326+0000 7f92ef59e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f92f006c110 0x7f92f006c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:60714/0 (socket says 172.21.15.55:60714) 2024-03-19T01:51:00.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.326+0000 7f92ef59e640 1 -- 172.21.15.55:0/2076834670 learned_addr learned my addr 172.21.15.55:0/2076834670 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:00.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.334+0000 7f92ef59e640 1 -- 172.21.15.55:0/2076834670 >> v2:172.21.15.55:3300/0 conn(0x7f92f0154300 msgr2=0x7f92f015ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:00.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.334+0000 7f92ef59e640 1 --2- 172.21.15.55:0/2076834670 >> v2:172.21.15.55:3300/0 conn(0x7f92f0154300 0x7f92f015ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:00.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.334+0000 7f92ef59e640 1 -- 172.21.15.55:0/2076834670 >> v1:172.21.15.84:6789/0 conn(0x7f92f006b2a0 legacy=0x7f92f015e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:00.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.334+0000 7f92ef59e640 1 -- 172.21.15.55:0/2076834670 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92f00c8d90 con 0x7f92f006c110 2024-03-19T01:51:00.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.338+0000 7f92ef59e640 1 --2- 172.21.15.55:0/2076834670 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f92f006c110 0x7f92f006c560 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f92e8009e50 tx=0x7f92e801c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12c5e8513316ef52 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:00.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.350+0000 7f92edd9b640 1 -- 172.21.15.55:0/2076834670 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e80a5840 con 0x7f92f006c110 2024-03-19T01:51:00.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.350+0000 7f92edd9b640 1 -- 172.21.15.55:0/2076834670 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92e80a59e0 con 0x7f92f006c110 2024-03-19T01:51:00.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.350+0000 7f92edd9b640 1 -- 172.21.15.55:0/2076834670 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e80a5db0 con 0x7f92f006c110 2024-03-19T01:51:00.351 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.350+0000 7f92f50c0640 1 -- 172.21.15.55:0/2076834670 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f92f006c110 msgr2=0x7f92f006c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:00.351 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.350+0000 7f92f50c0640 1 --2- 172.21.15.55:0/2076834670 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f92f006c110 0x7f92f006c560 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f92e8009e50 tx=0x7f92e801c9c0 comp rx=0 tx=0).stop 2024-03-19T01:51:00.351 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.350+0000 7f92f50c0640 1 -- 172.21.15.55:0/2076834670 shutdown_connections 2024-03-19T01:51:00.351 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.350+0000 7f92f50c0640 1 --2- 172.21.15.55:0/2076834670 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f92f006c110 0x7f92f006c560 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:00.351 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.350+0000 7f92f50c0640 1 --2- 172.21.15.55:0/2076834670 >> v2:172.21.15.55:3300/0 conn(0x7f92f0154300 0x7f92f015ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:00.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.350+0000 7f92f50c0640 1 -- 172.21.15.55:0/2076834670 >> 172.21.15.55:0/2076834670 conn(0x7f92f006fb40 msgr2=0x7f92f006ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:00.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.350+0000 7f92f50c0640 1 -- 172.21.15.55:0/2076834670 shutdown_connections 2024-03-19T01:51:00.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.350+0000 7f92f50c0640 1 -- 172.21.15.55:0/2076834670 wait complete. 2024-03-19T01:51:00.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.350+0000 7f92f50c0640 1 Processor -- start 2024-03-19T01:51:00.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.350+0000 7f92f50c0640 1 -- start start 2024-03-19T01:51:00.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.354+0000 7f92f50c0640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f92f006b2a0 0x7f92f01f6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:00.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.354+0000 7f92f50c0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f92f006c110 0x7f92f01f5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:00.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.354+0000 7f92f50c0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92f0154300 0x7f92f01f5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:00.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.354+0000 7f92f50c0640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f92f015e7d0 con 0x7f92f0154300 2024-03-19T01:51:00.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.354+0000 7f92f50c0640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f92f015dba0 con 0x7f92f006b2a0 2024-03-19T01:51:00.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.354+0000 7f92f50c0640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f92f00c74c0 con 0x7f92f006c110 2024-03-19T01:51:00.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.354+0000 7f92ee59c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f92f006c110 0x7f92f01f5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:00.356 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.354+0000 7f92ee59c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f92f006c110 0x7f92f01f5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:60730/0 (socket says 172.21.15.55:60730) 2024-03-19T01:51:00.356 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.354+0000 7f92ee59c640 1 -- 172.21.15.55:0/1357595449 learned_addr learned my addr 172.21.15.55:0/1357595449 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:00.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.366+0000 7f92ee59c640 1 -- 172.21.15.55:0/1357595449 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f92f006b2a0 msgr2=0x7f92f01f6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:00.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.366+0000 7f92ee59c640 1 --2- 172.21.15.55:0/1357595449 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f92f006b2a0 0x7f92f01f6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:00.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.366+0000 7f92ee59c640 1 -- 172.21.15.55:0/1357595449 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92f0154300 msgr2=0x7f92f01f5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:00.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.366+0000 7f92ee59c640 1 --2- 172.21.15.55:0/1357595449 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92f0154300 0x7f92f01f5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:00.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.366+0000 7f92ee59c640 1 -- 172.21.15.55:0/1357595449 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92f01f6f30 con 0x7f92f006c110 2024-03-19T01:51:00.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.374+0000 7f92ee59c640 1 --2- 172.21.15.55:0/1357595449 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f92f006c110 0x7f92f01f5760 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f92e400c4c0 tx=0x7f92e4001e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:00.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.382+0000 7f92d77fe640 1 -- 172.21.15.55:0/1357595449 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e4017070 con 0x7f92f006c110 2024-03-19T01:51:00.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.382+0000 7f92d77fe640 1 -- 172.21.15.55:0/1357595449 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92e4003600 con 0x7f92f006c110 2024-03-19T01:51:00.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.382+0000 7f92d77fe640 1 -- 172.21.15.55:0/1357595449 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e40039f0 con 0x7f92f006c110 2024-03-19T01:51:00.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.382+0000 7f92f50c0640 1 -- 172.21.15.55:0/1357595449 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92f01f7210 con 0x7f92f006c110 2024-03-19T01:51:00.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.382+0000 7f92f50c0640 1 -- 172.21.15.55:0/1357595449 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f92f01f5240 con 0x7f92f006c110 2024-03-19T01:51:00.384 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.382+0000 7f92f50c0640 1 -- 172.21.15.55:0/1357595449 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92f01f5440 con 0x7f92f006c110 2024-03-19T01:51:00.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.746+0000 7f92d77fe640 1 -- 172.21.15.55:0/1357595449 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f92e401a070 con 0x7f92f006c110 2024-03-19T01:51:00.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.746+0000 7f92d77fe640 1 --2- 172.21.15.55:0/1357595449 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92dc0676e0 0x7f92dc069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:00.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.750+0000 7f92eed9d640 1 --2- 172.21.15.55:0/1357595449 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92dc0676e0 0x7f92dc069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:00.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.750+0000 7f92eed9d640 1 --2- 172.21.15.55:0/1357595449 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92dc0676e0 0x7f92dc069ba0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f92e0009690 tx=0x7f92e0006c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:00.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.758+0000 7f92d77fe640 1 -- 172.21.15.55:0/1357595449 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f92e405ba20 con 0x7f92f006c110 2024-03-19T01:51:00.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:00.786+0000 7f92d77fe640 1 -- 172.21.15.55:0/1357595449 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f92f01f5440 con 0x7f92f006c110 2024-03-19T01:51:01.010 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.010+0000 7f92f50c0640 1 -- 172.21.15.55:0/1357595449 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92f0068a40 con 0x7f92f006c110 2024-03-19T01:51:01.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.214+0000 7f92d77fe640 1 -- 172.21.15.55:0/1357595449 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+5509 (secure 0 0 0) 0x7f92e4021020 con 0x7f92f006c110 2024-03-19T01:51:01.212 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:51:01.212 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4200,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4237,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4249,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6839/354880008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6838","nonce":354880008},{"type":"v1","addr":"172.21.15.84:6839","nonce":354880008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":4255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-03-19T01:51:01.212 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 4 2024-03-19T01:51:01.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.214+0000 7f92f50c0640 1 -- 172.21.15.55:0/1357595449 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92dc0676e0 msgr2=0x7f92dc069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:01.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.214+0000 7f92f50c0640 1 --2- 172.21.15.55:0/1357595449 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92dc0676e0 0x7f92dc069ba0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f92e0009690 tx=0x7f92e0006c00 comp rx=0 tx=0).stop 2024-03-19T01:51:01.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.214+0000 7f92f50c0640 1 -- 172.21.15.55:0/1357595449 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f92f006c110 msgr2=0x7f92f01f5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:01.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.214+0000 7f92f50c0640 1 --2- 172.21.15.55:0/1357595449 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f92f006c110 0x7f92f01f5760 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f92e400c4c0 tx=0x7f92e4001e80 comp rx=0 tx=0).stop 2024-03-19T01:51:01.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.214+0000 7f92f50c0640 1 -- 172.21.15.55:0/1357595449 shutdown_connections 2024-03-19T01:51:01.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.214+0000 7f92f50c0640 1 --2- 172.21.15.55:0/1357595449 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92dc0676e0 0x7f92dc069ba0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:01.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.214+0000 7f92f50c0640 1 --2- 172.21.15.55:0/1357595449 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92f0154300 0x7f92f01f5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:01.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.214+0000 7f92f50c0640 1 --2- 172.21.15.55:0/1357595449 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f92f006c110 0x7f92f01f5760 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:01.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.214+0000 7f92f50c0640 1 --2- 172.21.15.55:0/1357595449 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f92f006b2a0 0x7f92f01f6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:01.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.214+0000 7f92f50c0640 1 -- 172.21.15.55:0/1357595449 >> 172.21.15.55:0/1357595449 conn(0x7f92f006fb40 msgr2=0x7f92f015e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:01.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.214+0000 7f92f50c0640 1 -- 172.21.15.55:0/1357595449 shutdown_connections 2024-03-19T01:51:01.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.214+0000 7f92f50c0640 1 -- 172.21.15.55:0/1357595449 wait complete. 2024-03-19T01:51:01.242 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:51:01.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.370+0000 7f1c2ca0a640 1 Processor -- start 2024-03-19T01:51:01.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.370+0000 7f1c2ca0a640 1 -- start start 2024-03-19T01:51:01.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.374+0000 7f1c2ca0a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1c28152430 0x7f1c28158070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:01.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.374+0000 7f1c2ca0a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c280724c0 con 0x7f1c280c52b0 2024-03-19T01:51:01.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.374+0000 7f1c2ca0a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c2813d290 con 0x7f1c280c89f0 2024-03-19T01:51:01.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.374+0000 7f1c2ca0a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f1c2813d570 con 0x7f1c28152430 2024-03-19T01:51:01.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.374+0000 7f1c25d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1c28152430 0x7f1c28158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:01.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.374+0000 7f1c2659c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f1c280c52b0 0x7f1c28151d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:38616/0 (socket says 172.21.15.55:38616) 2024-03-19T01:51:01.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.374+0000 7f1c2659c640 1 -- 172.21.15.55:0/2615601541 learned_addr learned my addr 172.21.15.55:0/2615601541 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:51:01.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.378+0000 7f1c25d9b640 1 -- 172.21.15.55:0/2615601541 >> v1:172.21.15.84:6789/0 conn(0x7f1c280c89f0 legacy=0x7f1c28071810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:01.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.378+0000 7f1c25d9b640 1 -- 172.21.15.55:0/2615601541 >> v1:172.21.15.55:6789/0 conn(0x7f1c280c52b0 legacy=0x7f1c28151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:01.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.378+0000 7f1c25d9b640 1 -- 172.21.15.55:0/2615601541 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c2813d850 con 0x7f1c28152430 2024-03-19T01:51:01.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.382+0000 7f1c25d9b640 1 --2- 172.21.15.55:0/2615601541 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c28152430 0x7f1c28158070 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c0097f0 tx=0x7f1c1c01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=799efe2004019a3f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:01.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.386+0000 7f1c2559a640 1 -- 172.21.15.55:0/2615601541 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c0b5070 con 0x7f1c28152430 2024-03-19T01:51:01.388 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.386+0000 7f1c2559a640 1 -- 172.21.15.55:0/2615601541 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c1c0a8990 con 0x7f1c28152430 2024-03-19T01:51:01.388 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.386+0000 7f1c2559a640 1 -- 172.21.15.55:0/2615601541 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c0a8d60 con 0x7f1c28152430 2024-03-19T01:51:01.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.386+0000 7f1c2ca0a640 1 -- 172.21.15.55:0/2615601541 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c28152430 msgr2=0x7f1c28158070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:01.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.386+0000 7f1c2ca0a640 1 --2- 172.21.15.55:0/2615601541 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c28152430 0x7f1c28158070 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c0097f0 tx=0x7f1c1c01ec90 comp rx=0 tx=0).stop 2024-03-19T01:51:01.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.386+0000 7f1c2ca0a640 1 -- 172.21.15.55:0/2615601541 shutdown_connections 2024-03-19T01:51:01.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.386+0000 7f1c2ca0a640 1 --2- 172.21.15.55:0/2615601541 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c28152430 0x7f1c28158070 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:01.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.386+0000 7f1c2ca0a640 1 -- 172.21.15.55:0/2615601541 >> 172.21.15.55:0/2615601541 conn(0x7f1c2806f3b0 msgr2=0x7f1c2806f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:01.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.390+0000 7f1c2ca0a640 1 -- 172.21.15.55:0/2615601541 shutdown_connections 2024-03-19T01:51:01.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.390+0000 7f1c2ca0a640 1 -- 172.21.15.55:0/2615601541 wait complete. 2024-03-19T01:51:01.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.390+0000 7f1c2ca0a640 1 Processor -- start 2024-03-19T01:51:01.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.390+0000 7f1c2ca0a640 1 -- start start 2024-03-19T01:51:01.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.390+0000 7f1c2ca0a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c280c52b0 0x7f1c2813f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:01.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.390+0000 7f1c2ca0a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1c280c89f0 0x7f1c2813c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:01.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.390+0000 7f1c2ca0a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1c28152430 0x7f1c280c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:01.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.390+0000 7f1c2ca0a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1c280c5690 con 0x7f1c280c52b0 2024-03-19T01:51:01.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.390+0000 7f1c2ca0a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2813de10 con 0x7f1c28152430 2024-03-19T01:51:01.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.390+0000 7f1c2ca0a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1c280c5810 con 0x7f1c280c89f0 2024-03-19T01:51:01.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.390+0000 7f1c2659c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c280c52b0 0x7f1c2813f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:01.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.390+0000 7f1c2659c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c280c52b0 0x7f1c2813f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47310/0 (socket says 172.21.15.55:47310) 2024-03-19T01:51:01.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.390+0000 7f1c2659c640 1 -- 172.21.15.55:0/1213494034 learned_addr learned my addr 172.21.15.55:0/1213494034 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:01.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.394+0000 7f1c25d9b640 1 --2- 172.21.15.55:0/1213494034 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1c280c89f0 0x7f1c2813c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:01.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.394+0000 7f1c26d9d640 1 --2- 172.21.15.55:0/1213494034 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1c28152430 0x7f1c280c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:01.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.394+0000 7f1c2659c640 1 -- 172.21.15.55:0/1213494034 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1c280c89f0 msgr2=0x7f1c2813c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:01.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.394+0000 7f1c2659c640 1 --2- 172.21.15.55:0/1213494034 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1c280c89f0 0x7f1c2813c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:01.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.394+0000 7f1c2659c640 1 -- 172.21.15.55:0/1213494034 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1c28152430 msgr2=0x7f1c280c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:01.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.394+0000 7f1c2659c640 1 --2- 172.21.15.55:0/1213494034 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1c28152430 0x7f1c280c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:01.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.394+0000 7f1c2659c640 1 -- 172.21.15.55:0/1213494034 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c2815c3f0 con 0x7f1c280c52b0 2024-03-19T01:51:01.398 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.398+0000 7f1c2659c640 1 --2- 172.21.15.55:0/1213494034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c280c52b0 0x7f1c2813f500 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1c1800c450 tx=0x7f1c1800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:01.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.402+0000 7f1c16ffd640 1 -- 172.21.15.55:0/1213494034 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c18017070 con 0x7f1c280c52b0 2024-03-19T01:51:01.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.402+0000 7f1c16ffd640 1 -- 172.21.15.55:0/1213494034 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c18005cf0 con 0x7f1c280c52b0 2024-03-19T01:51:01.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.402+0000 7f1c16ffd640 1 -- 172.21.15.55:0/1213494034 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c18007290 con 0x7f1c280c52b0 2024-03-19T01:51:01.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.402+0000 7f1c2ca0a640 1 -- 172.21.15.55:0/1213494034 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c2815c150 con 0x7f1c280c52b0 2024-03-19T01:51:01.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.402+0000 7f1c2ca0a640 1 -- 172.21.15.55:0/1213494034 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c28140240 con 0x7f1c280c52b0 2024-03-19T01:51:01.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.406+0000 7f1c2ca0a640 1 -- 172.21.15.55:0/1213494034 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c28068330 con 0x7f1c280c52b0 2024-03-19T01:51:01.696 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.694+0000 7f1c16ffd640 1 -- 172.21.15.55:0/1213494034 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1c1801a070 con 0x7f1c280c52b0 2024-03-19T01:51:01.697 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.698+0000 7f1c16ffd640 1 --2- 172.21.15.55:0/1213494034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1c080676b0 0x7f1c08069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:01.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.698+0000 7f1c25d9b640 1 --2- 172.21.15.55:0/1213494034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1c080676b0 0x7f1c08069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:01.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.698+0000 7f1c25d9b640 1 --2- 172.21.15.55:0/1213494034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1c080676b0 0x7f1c08069b70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c0b2000 tx=0x7f1c1c0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:01.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.726+0000 7f1c16ffd640 1 -- 172.21.15.55:0/1213494034 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f1c1805b900 con 0x7f1c280c52b0 2024-03-19T01:51:01.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.726+0000 7f1c16ffd640 1 -- 172.21.15.55:0/1213494034 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1c18021020 con 0x7f1c280c52b0 2024-03-19T01:51:01.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:01.906+0000 7f1c2ca0a640 1 -- 172.21.15.55:0/1213494034 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1c28071d40 con 0x7f1c280c52b0 2024-03-19T01:51:02.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.150+0000 7f1c16ffd640 1 -- 172.21.15.55:0/1213494034 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12800 (secure 0 0 0) 0x7f1c180555e0 con 0x7f1c280c52b0 2024-03-19T01:51:02.152 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:51:02.152 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":18,"fsid":"def9d4e5-0bc7-449d-b893-2490cbd10e25","created":"2024-03-19T01:50:02.482954+0000","modified":"2024-03-19T01:50:26.957021+0000","last_up_change":"2024-03-19T01:50:21.735508+0000","last_in_change":"2024-03-19T01:50:12.168944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:50:23.456904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"35d5e3c5-d089-4f38-8961-98d064188f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6825","nonce":2757675043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6827","nonce":2757675043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6831","nonce":2757675043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6829","nonce":2757675043}]},"public_addr":"172.21.15.55:6825/2757675043","cluster_addr":"172.21.15.55:6827/2757675043","heartbeat_back_addr":"172.21.15.55:6831/2757675043","heartbeat_front_addr":"172.21.15.55:6829/2757675043","state":["exists","up"]},{"osd":1,"uuid":"8e58f791-001a-44d0-a464-ca68a479f70c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6817","nonce":543460007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6819","nonce":543460007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6823","nonce":543460007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6821","nonce":543460007}]},"public_addr":"172.21.15.55:6817/543460007","cluster_addr":"172.21.15.55:6819/543460007","heartbeat_back_addr":"172.21.15.55:6823/543460007","heartbeat_front_addr":"172.21.15.55:6821/543460007","state":["exists","up"]},{"osd":2,"uuid":"d44f38b8-67f4-43de-98a6-f3c520acaf10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6801","nonce":1760253550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6803","nonce":1760253550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6807","nonce":1760253550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6805","nonce":1760253550}]},"public_addr":"172.21.15.55:6801/1760253550","cluster_addr":"172.21.15.55:6803/1760253550","heartbeat_back_addr":"172.21.15.55:6807/1760253550","heartbeat_front_addr":"172.21.15.55:6805/1760253550","state":["exists","up"]},{"osd":3,"uuid":"850a7316-2d74-43f0-8649-b5f8d8f6692e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6809","nonce":2797616448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6811","nonce":2797616448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6815","nonce":2797616448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6813","nonce":2797616448}]},"public_addr":"172.21.15.55:6809/2797616448","cluster_addr":"172.21.15.55:6811/2797616448","heartbeat_back_addr":"172.21.15.55:6815/2797616448","heartbeat_front_addr":"172.21.15.55:6813/2797616448","state":["exists","up"]},{"osd":4,"uuid":"4e6152ae-a77d-4bb6-a855-5782ad5411a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6819","nonce":2938436826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6821","nonce":2938436826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6825","nonce":2938436826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6823","nonce":2938436826}]},"public_addr":"172.21.15.84:6819/2938436826","cluster_addr":"172.21.15.84:6821/2938436826","heartbeat_back_addr":"172.21.15.84:6825/2938436826","heartbeat_front_addr":"172.21.15.84:6823/2938436826","state":["exists","up"]},{"osd":5,"uuid":"1217ceba-dd31-4615-96f1-dfaf57160f3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6803","nonce":460717295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6805","nonce":460717295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6809","nonce":460717295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6807","nonce":460717295}]},"public_addr":"172.21.15.84:6803/460717295","cluster_addr":"172.21.15.84:6805/460717295","heartbeat_back_addr":"172.21.15.84:6809/460717295","heartbeat_front_addr":"172.21.15.84:6807/460717295","state":["exists","up"]},{"osd":6,"uuid":"3d9ed78a-2284-4c28-a24a-70eb812c7836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6827","nonce":416123135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6829","nonce":416123135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6833","nonce":416123135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6831","nonce":416123135}]},"public_addr":"172.21.15.84:6827/416123135","cluster_addr":"172.21.15.84:6829/416123135","heartbeat_back_addr":"172.21.15.84:6833/416123135","heartbeat_front_addr":"172.21.15.84:6831/416123135","state":["exists","up"]},{"osd":7,"uuid":"c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6811","nonce":460083528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6813","nonce":460083528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6817","nonce":460083528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6815","nonce":460083528}]},"public_addr":"172.21.15.84:6811/460083528","cluster_addr":"172.21.15.84:6813/460083528","heartbeat_back_addr":"172.21.15.84:6817/460083528","heartbeat_front_addr":"172.21.15.84:6815/460083528","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.269126+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.121001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.075714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.092461+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.194676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.062833+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.316515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.237111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:51:02.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.154+0000 7f1c2ca0a640 1 -- 172.21.15.55:0/1213494034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1c080676b0 msgr2=0x7f1c08069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:02.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.154+0000 7f1c2ca0a640 1 --2- 172.21.15.55:0/1213494034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1c080676b0 0x7f1c08069b70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c0b2000 tx=0x7f1c1c0aa070 comp rx=0 tx=0).stop 2024-03-19T01:51:02.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.154+0000 7f1c2ca0a640 1 -- 172.21.15.55:0/1213494034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c280c52b0 msgr2=0x7f1c2813f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:02.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.154+0000 7f1c2ca0a640 1 --2- 172.21.15.55:0/1213494034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c280c52b0 0x7f1c2813f500 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1c1800c450 tx=0x7f1c1800c910 comp rx=0 tx=0).stop 2024-03-19T01:51:02.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.154+0000 7f1c2ca0a640 1 -- 172.21.15.55:0/1213494034 shutdown_connections 2024-03-19T01:51:02.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.154+0000 7f1c2ca0a640 1 --2- 172.21.15.55:0/1213494034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1c080676b0 0x7f1c08069b70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:02.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.154+0000 7f1c2ca0a640 1 --2- 172.21.15.55:0/1213494034 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1c28152430 0x7f1c280c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:02.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.154+0000 7f1c2ca0a640 1 --2- 172.21.15.55:0/1213494034 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1c280c89f0 0x7f1c2813c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:02.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.154+0000 7f1c2ca0a640 1 --2- 172.21.15.55:0/1213494034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c280c52b0 0x7f1c2813f500 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:02.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.154+0000 7f1c2ca0a640 1 -- 172.21.15.55:0/1213494034 >> 172.21.15.55:0/1213494034 conn(0x7f1c2806f3b0 msgr2=0x7f1c281e1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:02.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.154+0000 7f1c2ca0a640 1 -- 172.21.15.55:0/1213494034 shutdown_connections 2024-03-19T01:51:02.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.154+0000 7f1c2ca0a640 1 -- 172.21.15.55:0/1213494034 wait complete. 2024-03-19T01:51:02.173 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:50:23.456904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:51:02.174 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:51:02.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.298+0000 7f807a431640 1 Processor -- start 2024-03-19T01:51:02.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.298+0000 7f807a431640 1 -- start start 2024-03-19T01:51:02.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.298+0000 7f807a431640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f80740c2800 0x7f8074156770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:02.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.298+0000 7f807a431640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f8074150c70 0x7f8074158510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:02.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.298+0000 7f807a431640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8074156e70 con 0x7f80740c2420 2024-03-19T01:51:02.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.298+0000 7f807a431640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f80741576f0 con 0x7f80740c2800 2024-03-19T01:51:02.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.298+0000 7f807a431640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f8074157130 con 0x7f8074150c70 2024-03-19T01:51:02.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.298+0000 7f80737fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f80740c2800 0x7f8074156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:02.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.298+0000 7f80737fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f80740c2800 0x7f8074156770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47316/0 (socket says 172.21.15.55:47316) 2024-03-19T01:51:02.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.298+0000 7f80737fe640 1 -- 172.21.15.55:0/3761910635 learned_addr learned my addr 172.21.15.55:0/3761910635 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:02.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.298+0000 7f80789ce640 1 --2- 172.21.15.55:0/3761910635 >> v2:172.21.15.84:3300/0 conn(0x7f8074150c70 0x7f8074158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:02.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.302+0000 7f80737fe640 1 -- 172.21.15.55:0/3761910635 >> v2:172.21.15.84:3300/0 conn(0x7f8074150c70 msgr2=0x7f8074158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:02.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.302+0000 7f80737fe640 1 --2- 172.21.15.55:0/3761910635 >> v2:172.21.15.84:3300/0 conn(0x7f8074150c70 0x7f8074158510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:02.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.302+0000 7f80737fe640 1 -- 172.21.15.55:0/3761910635 >> v1:172.21.15.84:6789/0 conn(0x7f80740c2420 legacy=0x7f8074150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:02.303 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.302+0000 7f80737fe640 1 -- 172.21.15.55:0/3761910635 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8074157410 con 0x7f80740c2800 2024-03-19T01:51:02.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.306+0000 7f80737fe640 1 --2- 172.21.15.55:0/3761910635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80740c2800 0x7f8074156770 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f80740c6000 tx=0x7f806801d5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1462b00de49e649c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:02.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.310+0000 7f8072ffd640 1 -- 172.21.15.55:0/3761910635 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80680ab670 con 0x7f80740c2800 2024-03-19T01:51:02.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.310+0000 7f8072ffd640 1 -- 172.21.15.55:0/3761910635 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80680b0070 con 0x7f80740c2800 2024-03-19T01:51:02.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.310+0000 7f8072ffd640 1 -- 172.21.15.55:0/3761910635 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80680aba30 con 0x7f80740c2800 2024-03-19T01:51:02.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.310+0000 7f807a431640 1 -- 172.21.15.55:0/3761910635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80740c2800 msgr2=0x7f8074156770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:02.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.310+0000 7f807a431640 1 --2- 172.21.15.55:0/3761910635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80740c2800 0x7f8074156770 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f80740c6000 tx=0x7f806801d5f0 comp rx=0 tx=0).stop 2024-03-19T01:51:02.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.310+0000 7f807a431640 1 -- 172.21.15.55:0/3761910635 shutdown_connections 2024-03-19T01:51:02.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.310+0000 7f807a431640 1 --2- 172.21.15.55:0/3761910635 >> v2:172.21.15.84:3300/0 conn(0x7f8074150c70 0x7f8074158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:02.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.310+0000 7f807a431640 1 --2- 172.21.15.55:0/3761910635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80740c2800 0x7f8074156770 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:02.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.310+0000 7f807a431640 1 -- 172.21.15.55:0/3761910635 >> 172.21.15.55:0/3761910635 conn(0x7f807406f3b0 msgr2=0x7f807406f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:02.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.310+0000 7f807a431640 1 -- 172.21.15.55:0/3761910635 shutdown_connections 2024-03-19T01:51:02.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.310+0000 7f807a431640 1 -- 172.21.15.55:0/3761910635 wait complete. 2024-03-19T01:51:02.313 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.314+0000 7f807a431640 1 Processor -- start 2024-03-19T01:51:02.313 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.314+0000 7f807a431640 1 -- start start 2024-03-19T01:51:02.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.314+0000 7f807a431640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80740c2420 0x7f80740c9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:02.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.314+0000 7f807a431640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f80740c2800 0x7f8074071810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:02.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.314+0000 7f807a431640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8074150c70 0x7f8074071d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:02.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.314+0000 7f807a431640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8074158e60 con 0x7f8074150c70 2024-03-19T01:51:02.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.314+0000 7f807a431640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f8074158ce0 con 0x7f80740c2420 2024-03-19T01:51:02.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.314+0000 7f807a431640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f8074163020 con 0x7f80740c2800 2024-03-19T01:51:02.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.314+0000 7f80737fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f80740c2800 0x7f8074071810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:02.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.314+0000 7f8073fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80740c2420 0x7f80740c9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:02.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.314+0000 7f8073fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80740c2420 0x7f80740c9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:44394/0 (socket says 172.21.15.55:44394) 2024-03-19T01:51:02.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.314+0000 7f8073fff640 1 -- 172.21.15.55:0/3018665703 learned_addr learned my addr 172.21.15.55:0/3018665703 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:02.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.326+0000 7f80737fe640 1 -- 172.21.15.55:0/3018665703 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80740c2420 msgr2=0x7f80740c9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:02.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.326+0000 7f80737fe640 1 --2- 172.21.15.55:0/3018665703 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80740c2420 0x7f80740c9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:02.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.326+0000 7f80737fe640 1 -- 172.21.15.55:0/3018665703 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8074150c70 msgr2=0x7f8074071d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:02.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.326+0000 7f80737fe640 1 --2- 172.21.15.55:0/3018665703 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8074150c70 0x7f8074071d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:02.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.326+0000 7f80737fe640 1 -- 172.21.15.55:0/3018665703 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80740cb5e0 con 0x7f80740c2800 2024-03-19T01:51:02.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.334+0000 7f80737fe640 1 --2- 172.21.15.55:0/3018665703 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f80740c2800 0x7f8074071810 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f80680ac000 tx=0x7f806801a390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:02.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.346+0000 7f8070ff9640 1 -- 172.21.15.55:0/3018665703 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80680b4620 con 0x7f80740c2800 2024-03-19T01:51:02.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.346+0000 7f8070ff9640 1 -- 172.21.15.55:0/3018665703 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80680b47c0 con 0x7f80740c2800 2024-03-19T01:51:02.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.346+0000 7f8070ff9640 1 -- 172.21.15.55:0/3018665703 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80680b4b70 con 0x7f80740c2800 2024-03-19T01:51:02.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.346+0000 7f807a431640 1 -- 172.21.15.55:0/3018665703 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f807414b5a0 con 0x7f80740c2800 2024-03-19T01:51:02.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.346+0000 7f807a431640 1 -- 172.21.15.55:0/3018665703 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f80741418d0 con 0x7f80740c2800 2024-03-19T01:51:02.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.350+0000 7f807a431640 1 -- 172.21.15.55:0/3018665703 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8074140af0 con 0x7f80740c2800 2024-03-19T01:51:02.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.662+0000 7f8070ff9640 1 -- 172.21.15.55:0/3018665703 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f80680b4d10 con 0x7f80740c2800 2024-03-19T01:51:02.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.662+0000 7f8070ff9640 1 --2- 172.21.15.55:0/3018665703 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f80600676e0 0x7f8060069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:02.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.662+0000 7f8070ff9640 1 -- 172.21.15.55:0/3018665703 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f80680f5b10 con 0x7f80740c2800 2024-03-19T01:51:02.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.666+0000 7f8073fff640 1 --2- 172.21.15.55:0/3018665703 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f80600676e0 0x7f8060069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:02.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.666+0000 7f8073fff640 1 --2- 172.21.15.55:0/3018665703 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f80600676e0 0x7f8060069ba0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8064001960 tx=0x7f8064009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:02.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.678+0000 7f8070ff9640 1 -- 172.21.15.55:0/3018665703 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f8074140af0 con 0x7f80740c2800 2024-03-19T01:51:02.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:02.870+0000 7f807a431640 1 -- 172.21.15.55:0/3018665703 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f80741416e0 con 0x7f80740c2800 2024-03-19T01:51:03.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.062+0000 7f8070ff9640 1 -- 172.21.15.55:0/3018665703 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) ==== 93+0+10 (secure 0 0 0) 0x7f80680171e0 con 0x7f80740c2800 2024-03-19T01:51:03.062 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 1 2024-03-19T01:51:03.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.066+0000 7f807a431640 1 -- 172.21.15.55:0/3018665703 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f80600676e0 msgr2=0x7f8060069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:03.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.066+0000 7f807a431640 1 --2- 172.21.15.55:0/3018665703 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f80600676e0 0x7f8060069ba0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8064001960 tx=0x7f8064009040 comp rx=0 tx=0).stop 2024-03-19T01:51:03.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.066+0000 7f807a431640 1 -- 172.21.15.55:0/3018665703 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f80740c2800 msgr2=0x7f8074071810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:03.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.066+0000 7f807a431640 1 --2- 172.21.15.55:0/3018665703 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f80740c2800 0x7f8074071810 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f80680ac000 tx=0x7f806801a390 comp rx=0 tx=0).stop 2024-03-19T01:51:03.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.066+0000 7f807a431640 1 -- 172.21.15.55:0/3018665703 shutdown_connections 2024-03-19T01:51:03.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.066+0000 7f807a431640 1 --2- 172.21.15.55:0/3018665703 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f80600676e0 0x7f8060069ba0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:03.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.066+0000 7f807a431640 1 --2- 172.21.15.55:0/3018665703 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8074150c70 0x7f8074071d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:03.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.066+0000 7f807a431640 1 --2- 172.21.15.55:0/3018665703 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f80740c2800 0x7f8074071810 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:03.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.066+0000 7f807a431640 1 --2- 172.21.15.55:0/3018665703 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80740c2420 0x7f80740c9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:03.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.066+0000 7f807a431640 1 -- 172.21.15.55:0/3018665703 >> 172.21.15.55:0/3018665703 conn(0x7f807406f3b0 msgr2=0x7f8074156020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:03.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.066+0000 7f807a431640 1 -- 172.21.15.55:0/3018665703 shutdown_connections 2024-03-19T01:51:03.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.066+0000 7f807a431640 1 -- 172.21.15.55:0/3018665703 wait complete. 2024-03-19T01:51:03.090 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-19T01:51:03.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.210+0000 7f2f934a5640 1 Processor -- start 2024-03-19T01:51:03.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.210+0000 7f2f934a5640 1 -- start start 2024-03-19T01:51:03.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.214+0000 7f2f934a5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2f8c0c2800 0x7f2f8c156770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:03.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.214+0000 7f2f934a5640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f2f8c150c70 0x7f2f8c158510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:03.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.214+0000 7f2f934a5640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f8c156e70 con 0x7f2f8c0c2420 2024-03-19T01:51:03.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.214+0000 7f2f934a5640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f2f8c1576f0 con 0x7f2f8c0c2800 2024-03-19T01:51:03.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.214+0000 7f2f934a5640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f2f8c157130 con 0x7f2f8c150c70 2024-03-19T01:51:03.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.214+0000 7f2f90a40640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2f8c0c2800 0x7f2f8c156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:03.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.214+0000 7f2f90a40640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2f8c0c2800 0x7f2f8c156770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47338/0 (socket says 172.21.15.55:47338) 2024-03-19T01:51:03.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.214+0000 7f2f90a40640 1 -- 172.21.15.55:0/1768157275 learned_addr learned my addr 172.21.15.55:0/1768157275 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:03.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.214+0000 7f2f91a42640 1 --2- 172.21.15.55:0/1768157275 >> v2:172.21.15.84:3300/0 conn(0x7f2f8c150c70 0x7f2f8c158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:03.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.218+0000 7f2f90a40640 1 -- 172.21.15.55:0/1768157275 >> v2:172.21.15.84:3300/0 conn(0x7f2f8c150c70 msgr2=0x7f2f8c158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:03.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.218+0000 7f2f90a40640 1 --2- 172.21.15.55:0/1768157275 >> v2:172.21.15.84:3300/0 conn(0x7f2f8c150c70 0x7f2f8c158510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:03.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.218+0000 7f2f90a40640 1 -- 172.21.15.55:0/1768157275 >> v1:172.21.15.84:6789/0 conn(0x7f2f8c0c2420 legacy=0x7f2f8c150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:03.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.218+0000 7f2f90a40640 1 -- 172.21.15.55:0/1768157275 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f8c157410 con 0x7f2f8c0c2800 2024-03-19T01:51:03.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.222+0000 7f2f90a40640 1 --2- 172.21.15.55:0/1768157275 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f8c0c2800 0x7f2f8c156770 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c0c6000 tx=0x7f2f7c01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d68d42b2cdfa4d4f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:03.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.226+0000 7f2f83fff640 1 -- 172.21.15.55:0/1768157275 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c0ab5f0 con 0x7f2f8c0c2800 2024-03-19T01:51:03.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.226+0000 7f2f83fff640 1 -- 172.21.15.55:0/1768157275 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f7c0b0070 con 0x7f2f8c0c2800 2024-03-19T01:51:03.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.226+0000 7f2f83fff640 1 -- 172.21.15.55:0/1768157275 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c0ab9b0 con 0x7f2f8c0c2800 2024-03-19T01:51:03.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.226+0000 7f2f934a5640 1 -- 172.21.15.55:0/1768157275 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f8c0c2800 msgr2=0x7f2f8c156770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:03.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.226+0000 7f2f934a5640 1 --2- 172.21.15.55:0/1768157275 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f8c0c2800 0x7f2f8c156770 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c0c6000 tx=0x7f2f7c01c4e0 comp rx=0 tx=0).stop 2024-03-19T01:51:03.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.226+0000 7f2f934a5640 1 -- 172.21.15.55:0/1768157275 shutdown_connections 2024-03-19T01:51:03.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.226+0000 7f2f934a5640 1 --2- 172.21.15.55:0/1768157275 >> v2:172.21.15.84:3300/0 conn(0x7f2f8c150c70 0x7f2f8c158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:03.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.226+0000 7f2f934a5640 1 --2- 172.21.15.55:0/1768157275 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f8c0c2800 0x7f2f8c156770 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:03.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.226+0000 7f2f934a5640 1 -- 172.21.15.55:0/1768157275 >> 172.21.15.55:0/1768157275 conn(0x7f2f8c06f3b0 msgr2=0x7f2f8c06f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:03.229 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.226+0000 7f2f934a5640 1 -- 172.21.15.55:0/1768157275 shutdown_connections 2024-03-19T01:51:03.229 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.226+0000 7f2f934a5640 1 -- 172.21.15.55:0/1768157275 wait complete. 2024-03-19T01:51:03.230 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.230+0000 7f2f934a5640 1 Processor -- start 2024-03-19T01:51:03.230 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.230+0000 7f2f934a5640 1 -- start start 2024-03-19T01:51:03.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.230+0000 7f2f934a5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f8c0c2420 0x7f2f8c146700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:03.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.230+0000 7f2f934a5640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f8c0c2800 0x7f2f8c15f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:03.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.230+0000 7f2f934a5640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f8c150c70 0x7f2f8c1ea2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:03.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.230+0000 7f2f934a5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8c158e60 con 0x7f2f8c0c2420 2024-03-19T01:51:03.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.230+0000 7f2f934a5640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8c158ce0 con 0x7f2f8c0c2800 2024-03-19T01:51:03.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.230+0000 7f2f934a5640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2f8c163020 con 0x7f2f8c150c70 2024-03-19T01:51:03.232 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.230+0000 7f2f91241640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f8c0c2420 0x7f2f8c146700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:03.232 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.230+0000 7f2f91241640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f8c0c2420 0x7f2f8c146700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47352/0 (socket says 172.21.15.55:47352) 2024-03-19T01:51:03.232 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.230+0000 7f2f91241640 1 -- 172.21.15.55:0/431753746 learned_addr learned my addr 172.21.15.55:0/431753746 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:03.232 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.230+0000 7f2f91a42640 1 --2- 172.21.15.55:0/431753746 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f8c150c70 0x7f2f8c1ea2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:03.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.234+0000 7f2f91241640 1 -- 172.21.15.55:0/431753746 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f8c150c70 msgr2=0x7f2f8c1ea2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:03.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.234+0000 7f2f91241640 1 --2- 172.21.15.55:0/431753746 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f8c150c70 0x7f2f8c1ea2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:03.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.234+0000 7f2f91241640 1 -- 172.21.15.55:0/431753746 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f8c0c2800 msgr2=0x7f2f8c15f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:03.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.234+0000 7f2f91241640 1 --2- 172.21.15.55:0/431753746 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f8c0c2800 0x7f2f8c15f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:03.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.234+0000 7f2f91241640 1 -- 172.21.15.55:0/431753746 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f8c15fe80 con 0x7f2f8c0c2420 2024-03-19T01:51:03.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.238+0000 7f2f91241640 1 --2- 172.21.15.55:0/431753746 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f8c0c2420 0x7f2f8c146700 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2f8800c490 tx=0x7f2f88001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:03.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.242+0000 7f2f81ffb640 1 -- 172.21.15.55:0/431753746 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f88017070 con 0x7f2f8c0c2420 2024-03-19T01:51:03.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.242+0000 7f2f81ffb640 1 -- 172.21.15.55:0/431753746 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f88003600 con 0x7f2f8c0c2420 2024-03-19T01:51:03.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.242+0000 7f2f81ffb640 1 -- 172.21.15.55:0/431753746 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f88003a30 con 0x7f2f8c0c2420 2024-03-19T01:51:03.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.242+0000 7f2f934a5640 1 -- 172.21.15.55:0/431753746 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f8c1566b0 con 0x7f2f8c0c2420 2024-03-19T01:51:03.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.242+0000 7f2f934a5640 1 -- 172.21.15.55:0/431753746 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f8c160140 con 0x7f2f8c0c2420 2024-03-19T01:51:03.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.242+0000 7f2f934a5640 1 -- 172.21.15.55:0/431753746 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f8c068330 con 0x7f2f8c0c2420 2024-03-19T01:51:03.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.546+0000 7f2f81ffb640 1 -- 172.21.15.55:0/431753746 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f2f8801a070 con 0x7f2f8c0c2420 2024-03-19T01:51:03.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.546+0000 7f2f81ffb640 1 --2- 172.21.15.55:0/431753746 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f70067660 0x7f2f70069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:03.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.546+0000 7f2f90a40640 1 --2- 172.21.15.55:0/431753746 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f70067660 0x7f2f70069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:03.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.550+0000 7f2f90a40640 1 --2- 172.21.15.55:0/431753746 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f70067660 0x7f2f70069b20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c0af000 tx=0x7f2f7c004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:03.585 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.586+0000 7f2f81ffb640 1 -- 172.21.15.55:0/431753746 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f2f8805ba70 con 0x7f2f8c0c2420 2024-03-19T01:51:03.585 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.586+0000 7f2f81ffb640 1 -- 172.21.15.55:0/431753746 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2f88021020 con 0x7f2f8c0c2420 2024-03-19T01:51:03.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.766+0000 7f2f934a5640 1 -- 172.21.15.55:0/431753746 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f2f8c15db70 con 0x7f2f8c0c2420 2024-03-19T01:51:03.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.786+0000 7f2f81ffb640 1 -- 172.21.15.55:0/431753746 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) ==== 84+0+65 (secure 0 0 0) 0x7f2f88055750 con 0x7f2f8c0c2420 2024-03-19T01:51:03.786 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:51:03.786 INFO:teuthology.orchestra.run.smithi055.stdout:[ 2024-03-19T01:51:03.786 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-19T01:51:03.786 INFO:teuthology.orchestra.run.smithi055.stdout: "poolnum": 1, 2024-03-19T01:51:03.786 INFO:teuthology.orchestra.run.smithi055.stdout: "poolname": ".mgr" 2024-03-19T01:51:03.786 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-19T01:51:03.786 INFO:teuthology.orchestra.run.smithi055.stdout:] 2024-03-19T01:51:03.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.786+0000 7f2f934a5640 1 -- 172.21.15.55:0/431753746 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f70067660 msgr2=0x7f2f70069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:03.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.786+0000 7f2f934a5640 1 --2- 172.21.15.55:0/431753746 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f70067660 0x7f2f70069b20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c0af000 tx=0x7f2f7c004740 comp rx=0 tx=0).stop 2024-03-19T01:51:03.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.786+0000 7f2f934a5640 1 -- 172.21.15.55:0/431753746 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f8c0c2420 msgr2=0x7f2f8c146700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:03.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.786+0000 7f2f934a5640 1 --2- 172.21.15.55:0/431753746 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f8c0c2420 0x7f2f8c146700 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2f8800c490 tx=0x7f2f88001e80 comp rx=0 tx=0).stop 2024-03-19T01:51:03.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.786+0000 7f2f934a5640 1 -- 172.21.15.55:0/431753746 shutdown_connections 2024-03-19T01:51:03.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.786+0000 7f2f934a5640 1 --2- 172.21.15.55:0/431753746 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f70067660 0x7f2f70069b20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:03.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.786+0000 7f2f934a5640 1 --2- 172.21.15.55:0/431753746 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f8c150c70 0x7f2f8c1ea2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:03.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.786+0000 7f2f934a5640 1 --2- 172.21.15.55:0/431753746 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f8c0c2800 0x7f2f8c15f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:03.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.786+0000 7f2f934a5640 1 --2- 172.21.15.55:0/431753746 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f8c0c2420 0x7f2f8c146700 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:03.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.786+0000 7f2f934a5640 1 -- 172.21.15.55:0/431753746 >> 172.21.15.55:0/431753746 conn(0x7f2f8c06f3b0 msgr2=0x7f2f8c071810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:03.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.786+0000 7f2f934a5640 1 -- 172.21.15.55:0/431753746 shutdown_connections 2024-03-19T01:51:03.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.786+0000 7f2f934a5640 1 -- 172.21.15.55:0/431753746 wait complete. 2024-03-19T01:51:03.807 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-19T01:51:03.807 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-19T01:51:03.945 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.946+0000 7fb210c04640 1 Processor -- start 2024-03-19T01:51:03.945 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.946+0000 7fb210c04640 1 -- start start 2024-03-19T01:51:03.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.946+0000 7fb210c04640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb20c157a00 0x7fb20c1508b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:03.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.946+0000 7fb210c04640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb20c0c2e40 con 0x7fb20c0c76b0 2024-03-19T01:51:03.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.946+0000 7fb210c04640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb20c156df0 con 0x7fb20c0c7a90 2024-03-19T01:51:03.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.946+0000 7fb210c04640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fb20c0c3100 con 0x7fb20c157a00 2024-03-19T01:51:03.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.946+0000 7fb20ad9d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb20c157a00 0x7fb20c1508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:03.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.946+0000 7fb20ad9d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb20c157a00 0x7fb20c1508b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47366/0 (socket says 172.21.15.55:47366) 2024-03-19T01:51:03.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.946+0000 7fb20ad9d640 1 -- 172.21.15.55:0/2967349098 learned_addr learned my addr 172.21.15.55:0/2967349098 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:03.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.950+0000 7fb20ad9d640 1 -- 172.21.15.55:0/2967349098 >> v1:172.21.15.84:6789/0 conn(0x7fb20c0c7a90 legacy=0x7fb20c1572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:03.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.950+0000 7fb20ad9d640 1 -- 172.21.15.55:0/2967349098 >> v1:172.21.15.55:6789/0 conn(0x7fb20c0c76b0 legacy=0x7fb20c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:03.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.950+0000 7fb20ad9d640 1 -- 172.21.15.55:0/2967349098 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb20c0c33e0 con 0x7fb20c157a00 2024-03-19T01:51:03.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.954+0000 7fb20ad9d640 1 --2- 172.21.15.55:0/2967349098 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb20c157a00 0x7fb20c1508b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb204009e50 tx=0x7fb20401c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85ec6db6b27e6f2b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:03.961 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.962+0000 7fb20959a640 1 -- 172.21.15.55:0/2967349098 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2040a5880 con 0x7fb20c157a00 2024-03-19T01:51:03.961 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.962+0000 7fb20959a640 1 -- 172.21.15.55:0/2967349098 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2040a5a20 con 0x7fb20c157a00 2024-03-19T01:51:03.961 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.962+0000 7fb20959a640 1 -- 172.21.15.55:0/2967349098 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2040a5df0 con 0x7fb20c157a00 2024-03-19T01:51:03.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.962+0000 7fb210c04640 1 -- 172.21.15.55:0/2967349098 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb20c157a00 msgr2=0x7fb20c1508b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:03.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.962+0000 7fb210c04640 1 --2- 172.21.15.55:0/2967349098 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb20c157a00 0x7fb20c1508b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb204009e50 tx=0x7fb20401c9c0 comp rx=0 tx=0).stop 2024-03-19T01:51:03.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.962+0000 7fb210c04640 1 -- 172.21.15.55:0/2967349098 shutdown_connections 2024-03-19T01:51:03.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.962+0000 7fb210c04640 1 --2- 172.21.15.55:0/2967349098 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb20c157a00 0x7fb20c1508b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:03.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.962+0000 7fb210c04640 1 -- 172.21.15.55:0/2967349098 >> 172.21.15.55:0/2967349098 conn(0x7fb20c06f3b0 msgr2=0x7fb20c06f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:03.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.962+0000 7fb210c04640 1 -- 172.21.15.55:0/2967349098 shutdown_connections 2024-03-19T01:51:03.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.962+0000 7fb210c04640 1 -- 172.21.15.55:0/2967349098 wait complete. 2024-03-19T01:51:03.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.962+0000 7fb210c04640 1 Processor -- start 2024-03-19T01:51:03.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.962+0000 7fb210c04640 1 -- start start 2024-03-19T01:51:03.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.962+0000 7fb210c04640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb20c0c76b0 0x7fb20c0c9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:03.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.962+0000 7fb210c04640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb20c0c7a90 0x7fb20c1461f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:03.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.962+0000 7fb210c04640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb20c157a00 0x7fb20c1446c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:03.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.962+0000 7fb210c04640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb20c0c9220 con 0x7fb20c0c76b0 2024-03-19T01:51:03.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.962+0000 7fb210c04640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb20c0c90a0 con 0x7fb20c0c7a90 2024-03-19T01:51:03.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.962+0000 7fb210c04640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fb20c164fa0 con 0x7fb20c157a00 2024-03-19T01:51:03.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.966+0000 7fb20a59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb20c0c76b0 0x7fb20c0c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:03.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.966+0000 7fb20a59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb20c0c76b0 0x7fb20c0c9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47376/0 (socket says 172.21.15.55:47376) 2024-03-19T01:51:03.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.966+0000 7fb20a59c640 1 -- 172.21.15.55:0/4182809988 learned_addr learned my addr 172.21.15.55:0/4182809988 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:03.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.966+0000 7fb20ad9d640 1 --2- 172.21.15.55:0/4182809988 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb20c157a00 0x7fb20c1446c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:03.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.970+0000 7fb20a59c640 1 -- 172.21.15.55:0/4182809988 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb20c157a00 msgr2=0x7fb20c1446c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:03.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.970+0000 7fb20a59c640 1 --2- 172.21.15.55:0/4182809988 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb20c157a00 0x7fb20c1446c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:03.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.970+0000 7fb20a59c640 1 -- 172.21.15.55:0/4182809988 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb20c0c7a90 msgr2=0x7fb20c1461f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:03.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.970+0000 7fb20a59c640 1 --2- 172.21.15.55:0/4182809988 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb20c0c7a90 0x7fb20c1461f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:03.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.970+0000 7fb20a59c640 1 -- 172.21.15.55:0/4182809988 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb20c0ca190 con 0x7fb20c0c76b0 2024-03-19T01:51:03.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.970+0000 7fb20a59c640 1 --2- 172.21.15.55:0/4182809988 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb20c0c76b0 0x7fb20c0c9950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc00c370 tx=0x7fb1fc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:03.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.974+0000 7fb1faffd640 1 -- 172.21.15.55:0/4182809988 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc0052d0 con 0x7fb20c0c76b0 2024-03-19T01:51:03.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.974+0000 7fb1faffd640 1 -- 172.21.15.55:0/4182809988 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1fc005470 con 0x7fb20c0c76b0 2024-03-19T01:51:03.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.974+0000 7fb1faffd640 1 -- 172.21.15.55:0/4182809988 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc007240 con 0x7fb20c0c76b0 2024-03-19T01:51:03.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.974+0000 7fb210c04640 1 -- 172.21.15.55:0/4182809988 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb20c0ca470 con 0x7fb20c0c76b0 2024-03-19T01:51:03.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.974+0000 7fb210c04640 1 -- 172.21.15.55:0/4182809988 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb20c0caa30 con 0x7fb20c0c76b0 2024-03-19T01:51:03.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:03.974+0000 7fb210c04640 1 -- 172.21.15.55:0/4182809988 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb20c068330 con 0x7fb20c0c76b0 2024-03-19T01:51:04.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:04.310+0000 7fb1faffd640 1 -- 172.21.15.55:0/4182809988 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb1fc0073e0 con 0x7fb20c0c76b0 2024-03-19T01:51:04.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:04.310+0000 7fb1faffd640 1 --2- 172.21.15.55:0/4182809988 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb1ec0676b0 0x7fb1ec069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:04.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:04.310+0000 7fb209d9b640 1 --2- 172.21.15.55:0/4182809988 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb1ec0676b0 0x7fb1ec069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:04.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:04.310+0000 7fb209d9b640 1 --2- 172.21.15.55:0/4182809988 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb1ec0676b0 0x7fb1ec069b70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb20c1562c0 tx=0x7fb200006bb0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:04.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:04.338+0000 7fb1faffd640 1 -- 172.21.15.55:0/4182809988 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7fb1fc053870 con 0x7fb20c0c76b0 2024-03-19T01:51:04.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:04.338+0000 7fb1faffd640 1 -- 172.21.15.55:0/4182809988 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fb1fc0557d0 con 0x7fb20c0c76b0 2024-03-19T01:51:04.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:04.554+0000 7fb210c04640 1 -- 172.21.15.55:0/4182809988 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fb20c150820 con 0x7fb20c0c76b0 2024-03-19T01:51:05.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:05.854+0000 7fb1faffd640 1 -- 172.21.15.55:0/4182809988 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) ==== 138+0+0 (secure 0 0 0) 0x7fb1fc059870 con 0x7fb20c0c76b0 2024-03-19T01:51:05.854 INFO:teuthology.orchestra.run.smithi055.stderr:pool 'cephfs_metadata' created 2024-03-19T01:51:05.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:05.858+0000 7fb210c04640 1 -- 172.21.15.55:0/4182809988 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb1ec0676b0 msgr2=0x7fb1ec069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:05.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:05.858+0000 7fb210c04640 1 --2- 172.21.15.55:0/4182809988 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb1ec0676b0 0x7fb1ec069b70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb20c1562c0 tx=0x7fb200006bb0 comp rx=0 tx=0).stop 2024-03-19T01:51:05.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:05.858+0000 7fb210c04640 1 -- 172.21.15.55:0/4182809988 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb20c0c76b0 msgr2=0x7fb20c0c9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:05.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:05.858+0000 7fb210c04640 1 --2- 172.21.15.55:0/4182809988 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb20c0c76b0 0x7fb20c0c9950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc00c370 tx=0x7fb1fc00c830 comp rx=0 tx=0).stop 2024-03-19T01:51:05.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:05.858+0000 7fb210c04640 1 -- 172.21.15.55:0/4182809988 shutdown_connections 2024-03-19T01:51:05.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:05.858+0000 7fb210c04640 1 --2- 172.21.15.55:0/4182809988 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb1ec0676b0 0x7fb1ec069b70 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:05.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:05.858+0000 7fb210c04640 1 --2- 172.21.15.55:0/4182809988 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb20c157a00 0x7fb20c1446c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:05.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:05.858+0000 7fb210c04640 1 --2- 172.21.15.55:0/4182809988 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb20c0c7a90 0x7fb20c1461f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:05.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:05.858+0000 7fb210c04640 1 --2- 172.21.15.55:0/4182809988 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb20c0c76b0 0x7fb20c0c9950 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:05.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:05.858+0000 7fb210c04640 1 -- 172.21.15.55:0/4182809988 >> 172.21.15.55:0/4182809988 conn(0x7fb20c06f3b0 msgr2=0x7fb20c0c6340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:05.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:05.858+0000 7fb210c04640 1 -- 172.21.15.55:0/4182809988 shutdown_connections 2024-03-19T01:51:05.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:05.858+0000 7fb210c04640 1 -- 172.21.15.55:0/4182809988 wait complete. 2024-03-19T01:51:05.884 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-19T01:51:06.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.022+0000 7fee6998d640 1 Processor -- start 2024-03-19T01:51:06.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.022+0000 7fee6998d640 1 -- start start 2024-03-19T01:51:06.025 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.022+0000 7fee6998d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fee640c3520 0x7fee640cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:06.025 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.022+0000 7fee6998d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee6406c230 con 0x7fee6406c630 2024-03-19T01:51:06.025 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.022+0000 7fee6998d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee6406bc70 con 0x7fee64150ae0 2024-03-19T01:51:06.025 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.022+0000 7fee6998d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fee640c3a40 con 0x7fee640c3520 2024-03-19T01:51:06.026 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.026+0000 7fee62ffd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fee64150ae0 0x7fee640c6e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:59280/0 (socket says 172.21.15.55:59280) 2024-03-19T01:51:06.026 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.026+0000 7fee62ffd640 1 -- 172.21.15.55:0/95019389 learned_addr learned my addr 172.21.15.55:0/95019389 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:51:06.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.034+0000 7fee61ffb640 1 -- 172.21.15.55:0/95019389 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 422502609 0 0) 0x7fee6406bc70 con 0x7fee64150ae0 2024-03-19T01:51:06.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.034+0000 7fee61ffb640 1 -- 172.21.15.55:0/95019389 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee4c003450 con 0x7fee64150ae0 2024-03-19T01:51:06.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.046+0000 7fee61ffb640 1 -- 172.21.15.55:0/95019389 <== mon.1 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3617555960 0 0) 0x7fee4c003450 con 0x7fee64150ae0 2024-03-19T01:51:06.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.046+0000 7fee61ffb640 1 -- 172.21.15.55:0/95019389 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fee640c3520 msgr2=0x7fee640cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:06.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.046+0000 7fee61ffb640 1 --2- 172.21.15.55:0/95019389 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fee640c3520 0x7fee640cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:06.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.046+0000 7fee61ffb640 1 -- 172.21.15.55:0/95019389 >> v1:172.21.15.55:6789/0 conn(0x7fee6406c630 legacy=0x7fee64160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:06.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.046+0000 7fee61ffb640 1 -- 172.21.15.55:0/95019389 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee6406bf50 con 0x7fee64150ae0 2024-03-19T01:51:06.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.046+0000 7fee61ffb640 1 -- 172.21.15.55:0/95019389 <== mon.1 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7fee54001680 con 0x7fee64150ae0 2024-03-19T01:51:06.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.046+0000 7fee61ffb640 1 -- 172.21.15.55:0/95019389 >> v1:172.21.15.84:6789/0 conn(0x7fee64150ae0 legacy=0x7fee640c6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:06.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.046+0000 7fee61ffb640 1 --2- 172.21.15.55:0/95019389 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fee640c3520 0x7fee4c003ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:06.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.046+0000 7fee61ffb640 1 --2- 172.21.15.55:0/95019389 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee4c008860 0x7fee4c00ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:06.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.046+0000 7fee61ffb640 1 --2- 172.21.15.55:0/95019389 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fee4c00b190 0x7fee4c00d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:06.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.046+0000 7fee61ffb640 1 -- 172.21.15.55:0/95019389 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fee640c3a40 con 0x7fee4c00b190 2024-03-19T01:51:06.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.046+0000 7fee61ffb640 1 -- 172.21.15.55:0/95019389 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c00deb0 con 0x7fee4c008860 2024-03-19T01:51:06.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.046+0000 7fee61ffb640 1 -- 172.21.15.55:0/95019389 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fee4c00e0b0 con 0x7fee640c3520 2024-03-19T01:51:06.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.050+0000 7fee62ffd640 1 --2- 172.21.15.55:0/95019389 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fee640c3520 0x7fee4c003ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:06.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.054+0000 7fee627fc640 1 --2- 172.21.15.55:0/95019389 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee4c008860 0x7fee4c00ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:06.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.062+0000 7fee62ffd640 1 -- 172.21.15.55:0/95019389 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee4c008860 msgr2=0x7fee4c00ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:06.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.062+0000 7fee62ffd640 1 --2- 172.21.15.55:0/95019389 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee4c008860 0x7fee4c00ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:06.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.062+0000 7fee62ffd640 1 -- 172.21.15.55:0/95019389 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fee4c00b190 msgr2=0x7fee4c00d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:06.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.062+0000 7fee62ffd640 1 --2- 172.21.15.55:0/95019389 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fee4c00b190 0x7fee4c00d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:06.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.062+0000 7fee62ffd640 1 -- 172.21.15.55:0/95019389 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee4c003450 con 0x7fee640c3520 2024-03-19T01:51:06.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.066+0000 7fee62ffd640 1 --2- 172.21.15.55:0/95019389 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fee640c3520 0x7fee4c003ab0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fee5400c190 tx=0x7fee5401cbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f27b7fe34a27a9e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:06.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.074+0000 7fee61ffb640 1 -- 172.21.15.55:0/95019389 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee54007c70 con 0x7fee640c3520 2024-03-19T01:51:06.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.074+0000 7fee61ffb640 1 -- 172.21.15.55:0/95019389 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee540a9070 con 0x7fee640c3520 2024-03-19T01:51:06.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.074+0000 7fee61ffb640 1 -- 172.21.15.55:0/95019389 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee54005da0 con 0x7fee640c3520 2024-03-19T01:51:06.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.074+0000 7fee6998d640 1 -- 172.21.15.55:0/95019389 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fee640c3520 msgr2=0x7fee4c003ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:06.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.074+0000 7fee6998d640 1 --2- 172.21.15.55:0/95019389 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fee640c3520 0x7fee4c003ab0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fee5400c190 tx=0x7fee5401cbe0 comp rx=0 tx=0).stop 2024-03-19T01:51:06.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.074+0000 7fee637fe640 1 -- 172.21.15.55:0/95019389 reap_dead start 2024-03-19T01:51:06.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.074+0000 7fee6998d640 1 -- 172.21.15.55:0/95019389 shutdown_connections 2024-03-19T01:51:06.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.074+0000 7fee6998d640 1 -- 172.21.15.55:0/95019389 >> 172.21.15.55:0/95019389 conn(0x7fee6406e5a0 msgr2=0x7fee64072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:06.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.074+0000 7fee6998d640 1 -- 172.21.15.55:0/95019389 shutdown_connections 2024-03-19T01:51:06.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.074+0000 7fee6998d640 1 -- 172.21.15.55:0/95019389 wait complete. 2024-03-19T01:51:06.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.074+0000 7fee6998d640 1 Processor -- start 2024-03-19T01:51:06.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.074+0000 7fee6998d640 1 -- start start 2024-03-19T01:51:06.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.078+0000 7fee6998d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee640c3520 0x7fee64157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:06.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.078+0000 7fee6998d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fee640b89f0 0x7fee641f0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:06.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.078+0000 7fee6998d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fee640b8dd0 0x7fee641f1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:06.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.078+0000 7fee6998d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fee640c6380 con 0x7fee640b8dd0 2024-03-19T01:51:06.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.078+0000 7fee6998d640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fee640c8200 con 0x7fee640c3520 2024-03-19T01:51:06.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.078+0000 7fee6998d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fee640c6a20 con 0x7fee640b89f0 2024-03-19T01:51:06.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.078+0000 7fee62ffd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee640c3520 0x7fee64157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:06.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.078+0000 7fee62ffd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee640c3520 0x7fee64157100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:44434/0 (socket says 172.21.15.55:44434) 2024-03-19T01:51:06.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.078+0000 7fee62ffd640 1 -- 172.21.15.55:0/3138860210 learned_addr learned my addr 172.21.15.55:0/3138860210 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:06.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.082+0000 7fee627fc640 1 --2- 172.21.15.55:0/3138860210 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fee640b89f0 0x7fee641f0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:06.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.094+0000 7fee627fc640 1 -- 172.21.15.55:0/3138860210 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee640c3520 msgr2=0x7fee64157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:06.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.094+0000 7fee627fc640 1 --2- 172.21.15.55:0/3138860210 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee640c3520 0x7fee64157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:06.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.094+0000 7fee627fc640 1 -- 172.21.15.55:0/3138860210 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fee640b8dd0 msgr2=0x7fee641f1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:06.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.094+0000 7fee627fc640 1 --2- 172.21.15.55:0/3138860210 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fee640b8dd0 0x7fee641f1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:06.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.094+0000 7fee627fc640 1 -- 172.21.15.55:0/3138860210 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee641f0630 con 0x7fee640b89f0 2024-03-19T01:51:06.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.102+0000 7fee627fc640 1 --2- 172.21.15.55:0/3138860210 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fee640b89f0 0x7fee641f0b30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fee5800c5c0 tx=0x7fee5800ca80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:06.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.114+0000 7fee4bfff640 1 -- 172.21.15.55:0/3138860210 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee58015020 con 0x7fee640b89f0 2024-03-19T01:51:06.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.114+0000 7fee6998d640 1 -- 172.21.15.55:0/3138860210 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee64155570 con 0x7fee640b89f0 2024-03-19T01:51:06.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.114+0000 7fee6998d640 1 -- 172.21.15.55:0/3138860210 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fee64155b30 con 0x7fee640b89f0 2024-03-19T01:51:06.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.114+0000 7fee4bfff640 1 -- 172.21.15.55:0/3138860210 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee58004440 con 0x7fee640b89f0 2024-03-19T01:51:06.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.114+0000 7fee4bfff640 1 -- 172.21.15.55:0/3138860210 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee58003800 con 0x7fee640b89f0 2024-03-19T01:51:06.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.114+0000 7fee6998d640 1 -- 172.21.15.55:0/3138860210 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee64068a40 con 0x7fee640b89f0 2024-03-19T01:51:06.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.450+0000 7fee4bfff640 1 -- 172.21.15.55:0/3138860210 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fee58018020 con 0x7fee640b89f0 2024-03-19T01:51:06.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.450+0000 7fee4bfff640 1 --2- 172.21.15.55:0/3138860210 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fee50067660 0x7fee50069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:06.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.450+0000 7fee62ffd640 1 --2- 172.21.15.55:0/3138860210 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fee50067660 0x7fee50069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:06.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.450+0000 7fee62ffd640 1 --2- 172.21.15.55:0/3138860210 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fee50067660 0x7fee50069b20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fee54000c10 tx=0x7fee540b1040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:06.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.462+0000 7fee4bfff640 1 -- 172.21.15.55:0/3138860210 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 5227+0+0 (secure 0 0 0) 0x7fee58054460 con 0x7fee640b89f0 2024-03-19T01:51:06.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.494+0000 7fee4bfff640 1 -- 172.21.15.55:0/3138860210 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fee5801f020 con 0x7fee640b89f0 2024-03-19T01:51:06.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:06.674+0000 7fee6998d640 1 -- 172.21.15.55:0/3138860210 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fee640c4a60 con 0x7fee640b89f0 2024-03-19T01:51:08.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.234+0000 7fee4bfff640 1 -- 172.21.15.55:0/3138860210 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v21) ==== 171+0+0 (secure 0 0 0) 0x7fee58056ac0 con 0x7fee640b89f0 2024-03-19T01:51:08.235 INFO:teuthology.orchestra.run.smithi055.stderr:pool 'cephfs_data' created 2024-03-19T01:51:08.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.238+0000 7fee6998d640 1 -- 172.21.15.55:0/3138860210 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fee50067660 msgr2=0x7fee50069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:08.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.238+0000 7fee6998d640 1 --2- 172.21.15.55:0/3138860210 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fee50067660 0x7fee50069b20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fee54000c10 tx=0x7fee540b1040 comp rx=0 tx=0).stop 2024-03-19T01:51:08.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.238+0000 7fee6998d640 1 -- 172.21.15.55:0/3138860210 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fee640b89f0 msgr2=0x7fee641f0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:08.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.238+0000 7fee6998d640 1 --2- 172.21.15.55:0/3138860210 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fee640b89f0 0x7fee641f0b30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fee5800c5c0 tx=0x7fee5800ca80 comp rx=0 tx=0).stop 2024-03-19T01:51:08.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.238+0000 7fee6998d640 1 -- 172.21.15.55:0/3138860210 shutdown_connections 2024-03-19T01:51:08.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.238+0000 7fee6998d640 1 --2- 172.21.15.55:0/3138860210 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fee50067660 0x7fee50069b20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:08.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.238+0000 7fee6998d640 1 --2- 172.21.15.55:0/3138860210 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fee640b8dd0 0x7fee641f1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:08.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.238+0000 7fee6998d640 1 --2- 172.21.15.55:0/3138860210 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fee640b89f0 0x7fee641f0b30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:08.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.238+0000 7fee6998d640 1 --2- 172.21.15.55:0/3138860210 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee640c3520 0x7fee64157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:08.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.238+0000 7fee6998d640 1 -- 172.21.15.55:0/3138860210 >> 172.21.15.55:0/3138860210 conn(0x7fee6406e5a0 msgr2=0x7fee640cbe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:08.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.238+0000 7fee6998d640 1 -- 172.21.15.55:0/3138860210 shutdown_connections 2024-03-19T01:51:08.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.238+0000 7fee6998d640 1 -- 172.21.15.55:0/3138860210 wait complete. 2024-03-19T01:51:08.259 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-19T01:51:08.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.394+0000 7f5049326640 1 Processor -- start 2024-03-19T01:51:08.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.394+0000 7f5049326640 1 -- start start 2024-03-19T01:51:08.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.394+0000 7f5049326640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f50440c3860 0x7f5044157b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:08.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.394+0000 7f5049326640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f504406b2a0 0x7f50440c3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:08.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.394+0000 7f5049326640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50440c6070 con 0x7f50440c6b00 2024-03-19T01:51:08.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.394+0000 7f5049326640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f50440c6610 con 0x7f50440c3860 2024-03-19T01:51:08.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.394+0000 7f5049326640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f504406b750 con 0x7f504406b2a0 2024-03-19T01:51:08.454 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:51:08.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:51:08.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.470+0000 7f5042ffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f50440c3860 0x7f5044157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:08.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.470+0000 7f5042ffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f50440c3860 0x7f5044157b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47408/0 (socket says 172.21.15.55:47408) 2024-03-19T01:51:08.471 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.470+0000 7f5042ffd640 1 -- 172.21.15.55:0/775548665 learned_addr learned my addr 172.21.15.55:0/775548665 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:08.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.478+0000 7f5042ffd640 1 -- 172.21.15.55:0/775548665 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f504406b2a0 msgr2=0x7f50440c3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:08.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.478+0000 7f5042ffd640 1 --2- 172.21.15.55:0/775548665 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f504406b2a0 0x7f50440c3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:08.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.478+0000 7f5042ffd640 1 -- 172.21.15.55:0/775548665 >> v1:172.21.15.55:6789/0 conn(0x7f50440c6b00 legacy=0x7f50440c58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:08.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.478+0000 7f5042ffd640 1 -- 172.21.15.55:0/775548665 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50440c6330 con 0x7f50440c3860 2024-03-19T01:51:08.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.482+0000 7f5042ffd640 1 --2- 172.21.15.55:0/775548665 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50440c3860 0x7f5044157b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f503400d180 tx=0x7f50340213c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f861b0024951f923 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:08.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.490+0000 7f5041ffb640 1 -- 172.21.15.55:0/775548665 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50340b6070 con 0x7f50440c3860 2024-03-19T01:51:08.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.490+0000 7f5041ffb640 1 -- 172.21.15.55:0/775548665 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f503400b2d0 con 0x7f50440c3860 2024-03-19T01:51:08.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.490+0000 7f5041ffb640 1 -- 172.21.15.55:0/775548665 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503400b480 con 0x7f50440c3860 2024-03-19T01:51:08.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.494+0000 7f5049326640 1 -- 172.21.15.55:0/775548665 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50440c3860 msgr2=0x7f5044157b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:08.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.494+0000 7f5049326640 1 --2- 172.21.15.55:0/775548665 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50440c3860 0x7f5044157b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f503400d180 tx=0x7f50340213c0 comp rx=0 tx=0).stop 2024-03-19T01:51:08.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.494+0000 7f5049326640 1 -- 172.21.15.55:0/775548665 shutdown_connections 2024-03-19T01:51:08.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.494+0000 7f5049326640 1 --2- 172.21.15.55:0/775548665 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f504406b2a0 0x7f50440c3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:08.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.494+0000 7f5049326640 1 --2- 172.21.15.55:0/775548665 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50440c3860 0x7f5044157b20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:08.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.494+0000 7f5049326640 1 -- 172.21.15.55:0/775548665 >> 172.21.15.55:0/775548665 conn(0x7f504406fb40 msgr2=0x7f504406ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:08.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.494+0000 7f5049326640 1 -- 172.21.15.55:0/775548665 shutdown_connections 2024-03-19T01:51:08.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.494+0000 7f5049326640 1 -- 172.21.15.55:0/775548665 wait complete. 2024-03-19T01:51:08.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.494+0000 7f5049326640 1 Processor -- start 2024-03-19T01:51:08.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.494+0000 7f5049326640 1 -- start start 2024-03-19T01:51:08.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.494+0000 7f5049326640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f504406b2a0 0x7f50441517b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:08.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.494+0000 7f5049326640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f50440c3860 0x7f504413db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:08.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.494+0000 7f5049326640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f50440c6b00 0x7f504413a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:08.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.494+0000 7f5049326640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5044157600 con 0x7f504406b2a0 2024-03-19T01:51:08.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.494+0000 7f5049326640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f5044157320 con 0x7f50440c3860 2024-03-19T01:51:08.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.494+0000 7f5049326640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f50440c4180 con 0x7f50440c6b00 2024-03-19T01:51:08.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.498+0000 7f50427fc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f50440c3860 0x7f504413db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:08.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.498+0000 7f50437fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f50440c6b00 0x7f504413a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:08.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.498+0000 7f50437fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f50440c6b00 0x7f504413a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:60802/0 (socket says 172.21.15.55:60802) 2024-03-19T01:51:08.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.498+0000 7f50437fe640 1 -- 172.21.15.55:0/3352593616 learned_addr learned my addr 172.21.15.55:0/3352593616 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:08.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.502+0000 7f50437fe640 1 -- 172.21.15.55:0/3352593616 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f50440c3860 msgr2=0x7f504413db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:08.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.502+0000 7f50437fe640 1 --2- 172.21.15.55:0/3352593616 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f50440c3860 0x7f504413db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:08.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.502+0000 7f50437fe640 1 -- 172.21.15.55:0/3352593616 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f504406b2a0 msgr2=0x7f50441517b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:08.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.502+0000 7f50437fe640 1 --2- 172.21.15.55:0/3352593616 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f504406b2a0 0x7f50441517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:08.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.502+0000 7f50437fe640 1 -- 172.21.15.55:0/3352593616 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5044153ee0 con 0x7f50440c6b00 2024-03-19T01:51:08.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.506+0000 7f50437fe640 1 --2- 172.21.15.55:0/3352593616 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f50440c6b00 0x7f504413a220 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f503c00b3c0 tx=0x7f503c00b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:08.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.514+0000 7f502b7fe640 1 -- 172.21.15.55:0/3352593616 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503c011830 con 0x7f50440c6b00 2024-03-19T01:51:08.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.514+0000 7f502b7fe640 1 -- 172.21.15.55:0/3352593616 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f503c016070 con 0x7f50440c6b00 2024-03-19T01:51:08.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.514+0000 7f502b7fe640 1 -- 172.21.15.55:0/3352593616 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503c011c10 con 0x7f50440c6b00 2024-03-19T01:51:08.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.514+0000 7f5049326640 1 -- 172.21.15.55:0/3352593616 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50441541a0 con 0x7f50440c6b00 2024-03-19T01:51:08.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.514+0000 7f5049326640 1 -- 172.21.15.55:0/3352593616 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f50441e0500 con 0x7f50440c6b00 2024-03-19T01:51:08.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.514+0000 7f5049326640 1 -- 172.21.15.55:0/3352593616 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5044068a40 con 0x7f50440c6b00 2024-03-19T01:51:08.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.542+0000 7f502b7fe640 1 -- 172.21.15.55:0/3352593616 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f503c01b460 con 0x7f50440c6b00 2024-03-19T01:51:08.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.542+0000 7f502b7fe640 1 --2- 172.21.15.55:0/3352593616 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f50300676b0 0x7f5030069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:08.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.542+0000 7f5042ffd640 1 --2- 172.21.15.55:0/3352593616 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f50300676b0 0x7f5030069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:08.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.542+0000 7f5042ffd640 1 --2- 172.21.15.55:0/3352593616 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f50300676b0 0x7f5030069b70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f50340b3000 tx=0x7f5034004ac0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:08.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.546+0000 7f502b7fe640 1 -- 172.21.15.55:0/3352593616 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5609+0+0 (secure 0 0 0) 0x7f503c0545b0 con 0x7f50440c6b00 2024-03-19T01:51:08.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:08.862+0000 7f502b7fe640 1 -- 172.21.15.55:0/3352593616 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f503c056c10 con 0x7f50440c6b00 2024-03-19T01:51:09.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:09.098+0000 7f5049326640 1 -- 172.21.15.55:0/3352593616 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f5044156d20 con 0x7f50440c6b00 2024-03-19T01:51:11.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.422+0000 7f502b7fe640 1 -- 172.21.15.55:0/3352593616 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v5) ==== 172+0+0 (secure 0 0 0) 0x7f503c056510 con 0x7f50440c6b00 2024-03-19T01:51:11.425 INFO:teuthology.orchestra.run.smithi055.stderr:new fs with metadata pool 2 and data pool 3 2024-03-19T01:51:11.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.426+0000 7f5049326640 1 -- 172.21.15.55:0/3352593616 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f50300676b0 msgr2=0x7f5030069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:11.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.426+0000 7f5049326640 1 --2- 172.21.15.55:0/3352593616 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f50300676b0 0x7f5030069b70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f50340b3000 tx=0x7f5034004ac0 comp rx=0 tx=0).stop 2024-03-19T01:51:11.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.426+0000 7f5049326640 1 -- 172.21.15.55:0/3352593616 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f50440c6b00 msgr2=0x7f504413a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:11.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.426+0000 7f5049326640 1 --2- 172.21.15.55:0/3352593616 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f50440c6b00 0x7f504413a220 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f503c00b3c0 tx=0x7f503c00b880 comp rx=0 tx=0).stop 2024-03-19T01:51:11.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.426+0000 7f5049326640 1 -- 172.21.15.55:0/3352593616 shutdown_connections 2024-03-19T01:51:11.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.426+0000 7f5049326640 1 --2- 172.21.15.55:0/3352593616 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f50300676b0 0x7f5030069b70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:11.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.426+0000 7f5049326640 1 --2- 172.21.15.55:0/3352593616 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f50440c6b00 0x7f504413a220 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:11.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.426+0000 7f5049326640 1 --2- 172.21.15.55:0/3352593616 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f50440c3860 0x7f504413db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:11.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.426+0000 7f5049326640 1 --2- 172.21.15.55:0/3352593616 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f504406b2a0 0x7f50441517b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:11.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.426+0000 7f5049326640 1 -- 172.21.15.55:0/3352593616 >> 172.21.15.55:0/3352593616 conn(0x7f504406fb40 msgr2=0x7f50441559d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:11.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.426+0000 7f5049326640 1 -- 172.21.15.55:0/3352593616 shutdown_connections 2024-03-19T01:51:11.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.426+0000 7f5049326640 1 -- 172.21.15.55:0/3352593616 wait complete. 2024-03-19T01:51:11.451 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-19T01:51:11.452 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-19T01:51:11.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.618+0000 7f957afff640 1 Processor -- start 2024-03-19T01:51:11.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.618+0000 7f957afff640 1 -- start start 2024-03-19T01:51:11.621 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.622+0000 7f957afff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f9574154710 0x7f957415ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:11.621 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.622+0000 7f957afff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95740c7a50 0x7f957406b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:11.621 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.622+0000 7f957afff640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95740c9240 con 0x7f9574073720 2024-03-19T01:51:11.621 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.622+0000 7f957afff640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f957415b370 con 0x7f9574154710 2024-03-19T01:51:11.621 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.622+0000 7f957afff640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f9574154af0 con 0x7f95740c7a50 2024-03-19T01:51:11.622 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.622+0000 7f9578d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f9574154710 0x7f957415ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:11.623 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.622+0000 7f9578d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f9574154710 0x7f957415ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33638/0 (socket says 172.21.15.55:33638) 2024-03-19T01:51:11.623 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.622+0000 7f9578d9b640 1 -- 172.21.15.55:0/3478563444 learned_addr learned my addr 172.21.15.55:0/3478563444 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:11.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.642+0000 7f957959c640 1 --2- 172.21.15.55:0/3478563444 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95740c7a50 0x7f957406b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:11.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.706+0000 7f9578d9b640 1 -- 172.21.15.55:0/3478563444 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95740c7a50 msgr2=0x7f957406b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:11.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.706+0000 7f9578d9b640 1 --2- 172.21.15.55:0/3478563444 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95740c7a50 0x7f957406b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:11.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.706+0000 7f9578d9b640 1 -- 172.21.15.55:0/3478563444 >> v1:172.21.15.84:6789/0 conn(0x7f9574073720 legacy=0x7f95740c7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:11.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.706+0000 7f9578d9b640 1 -- 172.21.15.55:0/3478563444 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95740c94c0 con 0x7f9574154710 2024-03-19T01:51:11.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.726+0000 7f9578d9b640 1 --2- 172.21.15.55:0/3478563444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9574154710 0x7f957415ae30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9564009870 tx=0x7f956401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dde8e7b253fd97f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:11.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.730+0000 7f95737fe640 1 -- 172.21.15.55:0/3478563444 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95640b0070 con 0x7f9574154710 2024-03-19T01:51:11.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.730+0000 7f95737fe640 1 -- 172.21.15.55:0/3478563444 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95640ab600 con 0x7f9574154710 2024-03-19T01:51:11.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.730+0000 7f95737fe640 1 -- 172.21.15.55:0/3478563444 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95640ab9d0 con 0x7f9574154710 2024-03-19T01:51:11.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.734+0000 7f957afff640 1 -- 172.21.15.55:0/3478563444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9574154710 msgr2=0x7f957415ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:11.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.734+0000 7f957afff640 1 --2- 172.21.15.55:0/3478563444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9574154710 0x7f957415ae30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9564009870 tx=0x7f956401c560 comp rx=0 tx=0).stop 2024-03-19T01:51:11.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.734+0000 7f957afff640 1 -- 172.21.15.55:0/3478563444 shutdown_connections 2024-03-19T01:51:11.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.734+0000 7f957afff640 1 --2- 172.21.15.55:0/3478563444 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95740c7a50 0x7f957406b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:11.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.734+0000 7f957afff640 1 --2- 172.21.15.55:0/3478563444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9574154710 0x7f957415ae30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:11.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.734+0000 7f957afff640 1 -- 172.21.15.55:0/3478563444 >> 172.21.15.55:0/3478563444 conn(0x7f957406c3b0 msgr2=0x7f957406c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:11.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.734+0000 7f957afff640 1 -- 172.21.15.55:0/3478563444 shutdown_connections 2024-03-19T01:51:11.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.734+0000 7f957afff640 1 -- 172.21.15.55:0/3478563444 wait complete. 2024-03-19T01:51:11.735 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.734+0000 7f957afff640 1 Processor -- start 2024-03-19T01:51:11.735 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.734+0000 7f957afff640 1 -- start start 2024-03-19T01:51:11.736 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.734+0000 7f957afff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9574073720 0x7f95741f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:11.736 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.734+0000 7f957afff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95740c7a50 0x7f9574147be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:11.736 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.734+0000 7f957afff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9574154710 0x7f95740cb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:11.736 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.734+0000 7f957afff640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9574166e00 con 0x7f95740c7a50 2024-03-19T01:51:11.736 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.734+0000 7f957afff640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f9574166c80 con 0x7f9574154710 2024-03-19T01:51:11.736 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.734+0000 7f957afff640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f9574166f80 con 0x7f9574073720 2024-03-19T01:51:11.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.742+0000 7f9573fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95740c7a50 0x7f9574147be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:11.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.746+0000 7f9573fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95740c7a50 0x7f9574147be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33644/0 (socket says 172.21.15.55:33644) 2024-03-19T01:51:11.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.746+0000 7f9573fff640 1 -- 172.21.15.55:0/1219396802 learned_addr learned my addr 172.21.15.55:0/1219396802 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:11.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.750+0000 7f9573fff640 1 -- 172.21.15.55:0/1219396802 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9574073720 msgr2=0x7f95741f3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:11.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.750+0000 7f9573fff640 1 --2- 172.21.15.55:0/1219396802 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9574073720 0x7f95741f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:11.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.750+0000 7f9573fff640 1 -- 172.21.15.55:0/1219396802 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9574154710 msgr2=0x7f95740cb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:11.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.750+0000 7f9573fff640 1 --2- 172.21.15.55:0/1219396802 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9574154710 0x7f95740cb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:11.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.750+0000 7f9573fff640 1 -- 172.21.15.55:0/1219396802 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f957415a630 con 0x7f95740c7a50 2024-03-19T01:51:11.754 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.754+0000 7f9573fff640 1 --2- 172.21.15.55:0/1219396802 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95740c7a50 0x7f9574147be0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f956800c420 tx=0x7f956800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:11.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.758+0000 7f95717fa640 1 -- 172.21.15.55:0/1219396802 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95680052d0 con 0x7f95740c7a50 2024-03-19T01:51:11.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.758+0000 7f95717fa640 1 -- 172.21.15.55:0/1219396802 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9568005470 con 0x7f95740c7a50 2024-03-19T01:51:11.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.758+0000 7f957afff640 1 -- 172.21.15.55:0/1219396802 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f957415b170 con 0x7f95740c7a50 2024-03-19T01:51:11.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.758+0000 7f95717fa640 1 -- 172.21.15.55:0/1219396802 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9568007260 con 0x7f95740c7a50 2024-03-19T01:51:11.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.758+0000 7f957afff640 1 -- 172.21.15.55:0/1219396802 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95741f7290 con 0x7f95740c7a50 2024-03-19T01:51:11.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.758+0000 7f957afff640 1 -- 172.21.15.55:0/1219396802 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9574068330 con 0x7f95740c7a50 2024-03-19T01:51:11.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.790+0000 7f95717fa640 1 -- 172.21.15.55:0/1219396802 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f9568007400 con 0x7f95740c7a50 2024-03-19T01:51:11.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.790+0000 7f95717fa640 1 --2- 172.21.15.55:0/1219396802 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f95600676b0 0x7f9560069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:11.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.794+0000 7f9578d9b640 1 --2- 172.21.15.55:0/1219396802 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f95600676b0 0x7f9560069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:11.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:11.794+0000 7f9578d9b640 1 --2- 172.21.15.55:0/1219396802 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f95600676b0 0x7f9560069b70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f95640ad000 tx=0x7f9564004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:12.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.070+0000 7f95717fa640 1 -- 172.21.15.55:0/1219396802 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5709+0+0 (secure 0 0 0) 0x7f9568053870 con 0x7f95740c7a50 2024-03-19T01:51:12.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.074+0000 7f95717fa640 1 -- 172.21.15.55:0/1219396802 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f95740c9240 con 0x7f95740c7a50 2024-03-19T01:51:12.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.270+0000 7f957afff640 1 -- 172.21.15.55:0/1219396802 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f9574068b10 con 0x7f95740c7a50 2024-03-19T01:51:12.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.742+0000 7f95717fa640 1 -- 172.21.15.55:0/1219396802 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v25) ==== 157+0+0 (secure 0 0 0) 0x7f9568055ed0 con 0x7f95740c7a50 2024-03-19T01:51:12.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.746+0000 7f957afff640 1 -- 172.21.15.55:0/1219396802 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f95600676b0 msgr2=0x7f9560069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:12.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.746+0000 7f957afff640 1 --2- 172.21.15.55:0/1219396802 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f95600676b0 0x7f9560069b70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f95640ad000 tx=0x7f9564004740 comp rx=0 tx=0).stop 2024-03-19T01:51:12.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.746+0000 7f957afff640 1 -- 172.21.15.55:0/1219396802 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95740c7a50 msgr2=0x7f9574147be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:12.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.746+0000 7f957afff640 1 --2- 172.21.15.55:0/1219396802 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95740c7a50 0x7f9574147be0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f956800c420 tx=0x7f956800c8e0 comp rx=0 tx=0).stop 2024-03-19T01:51:12.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.746+0000 7f957afff640 1 -- 172.21.15.55:0/1219396802 shutdown_connections 2024-03-19T01:51:12.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.746+0000 7f957afff640 1 --2- 172.21.15.55:0/1219396802 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f95600676b0 0x7f9560069b70 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:12.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.746+0000 7f957afff640 1 --2- 172.21.15.55:0/1219396802 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9574154710 0x7f95740cb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:12.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.746+0000 7f957afff640 1 --2- 172.21.15.55:0/1219396802 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95740c7a50 0x7f9574147be0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:12.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.746+0000 7f957afff640 1 --2- 172.21.15.55:0/1219396802 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9574073720 0x7f95741f3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:12.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.746+0000 7f957afff640 1 -- 172.21.15.55:0/1219396802 >> 172.21.15.55:0/1219396802 conn(0x7f957406c3b0 msgr2=0x7f957406c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:12.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.746+0000 7f957afff640 1 -- 172.21.15.55:0/1219396802 shutdown_connections 2024-03-19T01:51:12.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.746+0000 7f957afff640 1 -- 172.21.15.55:0/1219396802 wait complete. 2024-03-19T01:51:12.782 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-19T01:51:12.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.934+0000 7f0ce94f9640 1 Processor -- start 2024-03-19T01:51:12.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.934+0000 7f0ce94f9640 1 -- start start 2024-03-19T01:51:12.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.938+0000 7f0ce94f9640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f0ce40c9a90 0x7f0ce4070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:12.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.938+0000 7f0ce94f9640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0ce41540c0 0x7f0ce4154550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:12.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.938+0000 7f0ce94f9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ce415e3a0 con 0x7f0ce40c96b0 2024-03-19T01:51:12.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.938+0000 7f0ce94f9640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f0ce40c7ad0 con 0x7f0ce41540c0 2024-03-19T01:51:12.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.938+0000 7f0ce94f9640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f0ce415e680 con 0x7f0ce40c9a90 2024-03-19T01:51:12.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.938+0000 7f0ce37fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0ce41540c0 0x7f0ce4154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:12.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.938+0000 7f0ce37fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0ce41540c0 0x7f0ce4154550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33646/0 (socket says 172.21.15.55:33646) 2024-03-19T01:51:12.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.938+0000 7f0ce27fc640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f0ce40c9a90 0x7f0ce4070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:12.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.938+0000 7f0ce37fe640 1 -- 172.21.15.55:0/1186102234 learned_addr learned my addr 172.21.15.55:0/1186102234 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:12.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.942+0000 7f0ce37fe640 1 -- 172.21.15.55:0/1186102234 >> v2:172.21.15.84:3300/0 conn(0x7f0ce40c9a90 msgr2=0x7f0ce4070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:12.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.942+0000 7f0ce37fe640 1 --2- 172.21.15.55:0/1186102234 >> v2:172.21.15.84:3300/0 conn(0x7f0ce40c9a90 0x7f0ce4070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:12.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.942+0000 7f0ce37fe640 1 -- 172.21.15.55:0/1186102234 >> v1:172.21.15.84:6789/0 conn(0x7f0ce40c96b0 legacy=0x7f0ce40c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:12.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.942+0000 7f0ce37fe640 1 -- 172.21.15.55:0/1186102234 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ce415a330 con 0x7f0ce41540c0 2024-03-19T01:51:12.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.950+0000 7f0ce37fe640 1 --2- 172.21.15.55:0/1186102234 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ce41540c0 0x7f0ce4154550 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40c9420 tx=0x7f0cdc01c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8cb9fd63daa1450 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:12.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.954+0000 7f0ce1ffb640 1 -- 172.21.15.55:0/1186102234 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc0a56d0 con 0x7f0ce41540c0 2024-03-19T01:51:12.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.954+0000 7f0ce1ffb640 1 -- 172.21.15.55:0/1186102234 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cdc0a5870 con 0x7f0ce41540c0 2024-03-19T01:51:12.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.954+0000 7f0ce1ffb640 1 -- 172.21.15.55:0/1186102234 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc0a5c40 con 0x7f0ce41540c0 2024-03-19T01:51:12.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.954+0000 7f0ce94f9640 1 -- 172.21.15.55:0/1186102234 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ce41540c0 msgr2=0x7f0ce4154550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:12.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.954+0000 7f0ce94f9640 1 --2- 172.21.15.55:0/1186102234 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ce41540c0 0x7f0ce4154550 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40c9420 tx=0x7f0cdc01c920 comp rx=0 tx=0).stop 2024-03-19T01:51:12.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.954+0000 7f0ce94f9640 1 -- 172.21.15.55:0/1186102234 shutdown_connections 2024-03-19T01:51:12.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.954+0000 7f0ce94f9640 1 --2- 172.21.15.55:0/1186102234 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ce41540c0 0x7f0ce4154550 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:12.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.954+0000 7f0ce94f9640 1 --2- 172.21.15.55:0/1186102234 >> v2:172.21.15.84:3300/0 conn(0x7f0ce40c9a90 0x7f0ce4070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:12.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.954+0000 7f0ce94f9640 1 -- 172.21.15.55:0/1186102234 >> 172.21.15.55:0/1186102234 conn(0x7f0ce40cc9c0 msgr2=0x7f0ce40ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:12.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.954+0000 7f0ce94f9640 1 -- 172.21.15.55:0/1186102234 shutdown_connections 2024-03-19T01:51:12.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.954+0000 7f0ce94f9640 1 -- 172.21.15.55:0/1186102234 wait complete. 2024-03-19T01:51:12.956 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.954+0000 7f0ce94f9640 1 Processor -- start 2024-03-19T01:51:12.956 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.954+0000 7f0ce94f9640 1 -- start start 2024-03-19T01:51:12.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.958+0000 7f0ce94f9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ce40c96b0 0x7f0ce414e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:12.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.958+0000 7f0ce94f9640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0ce40c9a90 0x7f0ce41503b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:12.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.958+0000 7f0ce94f9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0ce41540c0 0x7f0ce41460d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:12.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.958+0000 7f0ce94f9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce4166b90 con 0x7f0ce40c96b0 2024-03-19T01:51:12.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.958+0000 7f0ce94f9640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce415b0f0 con 0x7f0ce40c9a90 2024-03-19T01:51:12.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.958+0000 7f0ce94f9640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f0ce4166d10 con 0x7f0ce41540c0 2024-03-19T01:51:12.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.958+0000 7f0ce2ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ce40c96b0 0x7f0ce414e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:12.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.958+0000 7f0ce2ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ce40c96b0 0x7f0ce414e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33650/0 (socket says 172.21.15.55:33650) 2024-03-19T01:51:12.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.958+0000 7f0ce2ffd640 1 -- 172.21.15.55:0/2408973015 learned_addr learned my addr 172.21.15.55:0/2408973015 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:12.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.958+0000 7f0ce27fc640 1 --2- 172.21.15.55:0/2408973015 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0ce40c9a90 0x7f0ce41503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:12.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.958+0000 7f0ce37fe640 1 --2- 172.21.15.55:0/2408973015 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0ce41540c0 0x7f0ce41460d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:12.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.962+0000 7f0ce2ffd640 1 -- 172.21.15.55:0/2408973015 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0ce41540c0 msgr2=0x7f0ce41460d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:12.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.962+0000 7f0ce2ffd640 1 --2- 172.21.15.55:0/2408973015 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0ce41540c0 0x7f0ce41460d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:12.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.962+0000 7f0ce2ffd640 1 -- 172.21.15.55:0/2408973015 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0ce40c9a90 msgr2=0x7f0ce41503b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:12.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.962+0000 7f0ce2ffd640 1 --2- 172.21.15.55:0/2408973015 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0ce40c9a90 0x7f0ce41503b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:12.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.962+0000 7f0ce2ffd640 1 -- 172.21.15.55:0/2408973015 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ce4158070 con 0x7f0ce40c96b0 2024-03-19T01:51:12.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.966+0000 7f0ce2ffd640 1 --2- 172.21.15.55:0/2408973015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ce40c96b0 0x7f0ce414e5d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0cd400b410 tx=0x7f0cd400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:12.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.970+0000 7f0ccb7fe640 1 -- 172.21.15.55:0/2408973015 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4015020 con 0x7f0ce40c96b0 2024-03-19T01:51:12.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.970+0000 7f0ccb7fe640 1 -- 172.21.15.55:0/2408973015 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cd4011830 con 0x7f0ce40c96b0 2024-03-19T01:51:12.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.970+0000 7f0ce94f9640 1 -- 172.21.15.55:0/2408973015 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ce4158610 con 0x7f0ce40c96b0 2024-03-19T01:51:12.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.970+0000 7f0ccb7fe640 1 -- 172.21.15.55:0/2408973015 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4011c20 con 0x7f0ce40c96b0 2024-03-19T01:51:12.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.970+0000 7f0ce94f9640 1 -- 172.21.15.55:0/2408973015 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ce4158a10 con 0x7f0ce40c96b0 2024-03-19T01:51:12.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:12.970+0000 7f0ce94f9640 1 -- 172.21.15.55:0/2408973015 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ce4068a40 con 0x7f0ce40c96b0 2024-03-19T01:51:13.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:13.274+0000 7f0ccb7fe640 1 -- 172.21.15.55:0/2408973015 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f0cd4018020 con 0x7f0ce40c96b0 2024-03-19T01:51:13.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:13.274+0000 7f0ccb7fe640 1 --2- 172.21.15.55:0/2408973015 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0cd00676e0 0x7f0cd0069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:13.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:13.274+0000 7f0ce27fc640 1 --2- 172.21.15.55:0/2408973015 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0cd00676e0 0x7f0cd0069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:13.276 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:13.274+0000 7f0ce27fc640 1 --2- 172.21.15.55:0/2408973015 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0cd00676e0 0x7f0cd0069ba0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40ca100 tx=0x7f0cd8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:13.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:13.310+0000 7f0ccb7fe640 1 -- 172.21.15.55:0/2408973015 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6007+0+0 (secure 0 0 0) 0x7f0cd40543d0 con 0x7f0ce40c96b0 2024-03-19T01:51:13.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:13.310+0000 7f0ccb7fe640 1 -- 172.21.15.55:0/2408973015 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f0cd401f020 con 0x7f0ce40c96b0 2024-03-19T01:51:13.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:13.498+0000 7f0ce94f9640 1 -- 172.21.15.55:0/2408973015 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f0ce4151260 con 0x7f0ce40c96b0 2024-03-19T01:51:15.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.338+0000 7f0ccb7fe640 1 -- 172.21.15.55:0/2408973015 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v27) ==== 228+0+0 (secure 0 0 0) 0x7f0cd4056330 con 0x7f0ce40c96b0 2024-03-19T01:51:15.338 INFO:teuthology.orchestra.run.smithi055.stderr:pool 'cephfs_data_ec' created 2024-03-19T01:51:15.342 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.342+0000 7f0ce94f9640 1 -- 172.21.15.55:0/2408973015 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0cd00676e0 msgr2=0x7f0cd0069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:15.342 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.342+0000 7f0ce94f9640 1 --2- 172.21.15.55:0/2408973015 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0cd00676e0 0x7f0cd0069ba0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40ca100 tx=0x7f0cd8009190 comp rx=0 tx=0).stop 2024-03-19T01:51:15.342 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.342+0000 7f0ce94f9640 1 -- 172.21.15.55:0/2408973015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ce40c96b0 msgr2=0x7f0ce414e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:15.342 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.342+0000 7f0ce94f9640 1 --2- 172.21.15.55:0/2408973015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ce40c96b0 0x7f0ce414e5d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0cd400b410 tx=0x7f0cd400b8d0 comp rx=0 tx=0).stop 2024-03-19T01:51:15.342 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.342+0000 7f0ce94f9640 1 -- 172.21.15.55:0/2408973015 shutdown_connections 2024-03-19T01:51:15.342 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.342+0000 7f0ce94f9640 1 --2- 172.21.15.55:0/2408973015 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0cd00676e0 0x7f0cd0069ba0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:15.342 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.342+0000 7f0ce94f9640 1 --2- 172.21.15.55:0/2408973015 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0ce41540c0 0x7f0ce41460d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:15.342 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.342+0000 7f0ce94f9640 1 --2- 172.21.15.55:0/2408973015 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0ce40c9a90 0x7f0ce41503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:15.342 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.342+0000 7f0ce94f9640 1 --2- 172.21.15.55:0/2408973015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ce40c96b0 0x7f0ce414e5d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:15.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.342+0000 7f0ce94f9640 1 -- 172.21.15.55:0/2408973015 >> 172.21.15.55:0/2408973015 conn(0x7f0ce40cc9c0 msgr2=0x7f0ce40709e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:15.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.342+0000 7f0ce94f9640 1 -- 172.21.15.55:0/2408973015 shutdown_connections 2024-03-19T01:51:15.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.342+0000 7f0ce94f9640 1 -- 172.21.15.55:0/2408973015 wait complete. 2024-03-19T01:51:15.364 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-19T01:51:15.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.506+0000 7fd15c782640 1 Processor -- start 2024-03-19T01:51:15.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.506+0000 7fd15c782640 1 -- start start 2024-03-19T01:51:15.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.506+0000 7fd15c782640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd154152120 0x7fd1540c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:15.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.506+0000 7fd15c782640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd1540c9610 0x7fd154157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:15.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.506+0000 7fd15c782640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fd154070780 0x7fd154158510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:15.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.506+0000 7fd15c782640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fd154152500 con 0x7fd154152120 2024-03-19T01:51:15.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.506+0000 7fd15c782640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fd154070590 con 0x7fd154070780 2024-03-19T01:51:15.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.506+0000 7fd15c782640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd1540c3530 con 0x7fd1540c9610 2024-03-19T01:51:15.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.506+0000 7fd15a51e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd154152120 0x7fd1540c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:15.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.510+0000 7fd159d1d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd1540c9610 0x7fd154157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:15.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.510+0000 7fd15a51e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd154152120 0x7fd1540c90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33666/0 (socket says 172.21.15.55:33666) 2024-03-19T01:51:15.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.510+0000 7fd15a51e640 1 -- 172.21.15.55:0/2080295149 learned_addr learned my addr 172.21.15.55:0/2080295149 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:15.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.510+0000 7fd15a51e640 1 -- 172.21.15.55:0/2080295149 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd1540c9610 msgr2=0x7fd154157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:15.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.510+0000 7fd15a51e640 1 --2- 172.21.15.55:0/2080295149 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd1540c9610 0x7fd154157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:15.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.510+0000 7fd15a51e640 1 -- 172.21.15.55:0/2080295149 >> v2:172.21.15.84:3300/0 conn(0x7fd154070780 msgr2=0x7fd154158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:15.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.510+0000 7fd15a51e640 1 --2- 172.21.15.55:0/2080295149 >> v2:172.21.15.84:3300/0 conn(0x7fd154070780 0x7fd154158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:15.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.510+0000 7fd15a51e640 1 -- 172.21.15.55:0/2080295149 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd15406fff0 con 0x7fd154152120 2024-03-19T01:51:15.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.522+0000 7fd15a51e640 1 --2- 172.21.15.55:0/2080295149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd154152120 0x7fd1540c90d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd154071f20 tx=0x7fd14801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b612340971e6af32 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:15.594 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.594+0000 7fd15951c640 1 -- 172.21.15.55:0/2080295149 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1480b0070 con 0x7fd154152120 2024-03-19T01:51:15.595 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.594+0000 7fd15951c640 1 -- 172.21.15.55:0/2080295149 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1480ab600 con 0x7fd154152120 2024-03-19T01:51:15.595 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.594+0000 7fd15951c640 1 -- 172.21.15.55:0/2080295149 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1480ab9d0 con 0x7fd154152120 2024-03-19T01:51:15.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.594+0000 7fd15c782640 1 -- 172.21.15.55:0/2080295149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd154152120 msgr2=0x7fd1540c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:15.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.594+0000 7fd15c782640 1 --2- 172.21.15.55:0/2080295149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd154152120 0x7fd1540c90d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd154071f20 tx=0x7fd14801c520 comp rx=0 tx=0).stop 2024-03-19T01:51:15.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.594+0000 7fd15c782640 1 -- 172.21.15.55:0/2080295149 shutdown_connections 2024-03-19T01:51:15.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.594+0000 7fd15c782640 1 --2- 172.21.15.55:0/2080295149 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd1540c9610 0x7fd154157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:15.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.594+0000 7fd15c782640 1 --2- 172.21.15.55:0/2080295149 >> v2:172.21.15.84:3300/0 conn(0x7fd154070780 0x7fd154158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:15.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.594+0000 7fd15c782640 1 --2- 172.21.15.55:0/2080295149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd154152120 0x7fd1540c90d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:15.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.594+0000 7fd15c782640 1 -- 172.21.15.55:0/2080295149 >> 172.21.15.55:0/2080295149 conn(0x7fd154067600 msgr2=0x7fd154071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:15.597 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.594+0000 7fd15c782640 1 -- 172.21.15.55:0/2080295149 shutdown_connections 2024-03-19T01:51:15.597 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.594+0000 7fd15c782640 1 -- 172.21.15.55:0/2080295149 wait complete. 2024-03-19T01:51:15.598 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.598+0000 7fd15c782640 1 Processor -- start 2024-03-19T01:51:15.598 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.598+0000 7fd15c782640 1 -- start start 2024-03-19T01:51:15.600 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.598+0000 7fd15c782640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd154070780 0x7fd15414dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:15.600 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.598+0000 7fd15c782640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1540c9610 0x7fd1541462e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:15.600 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.598+0000 7fd15c782640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd154152120 0x7fd154146820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:15.600 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.598+0000 7fd15c782640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd1540c31c0 con 0x7fd1540c9610 2024-03-19T01:51:15.600 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.598+0000 7fd15c782640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd1540c2ee0 con 0x7fd154070780 2024-03-19T01:51:15.600 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.598+0000 7fd15c782640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd1540c86d0 con 0x7fd154152120 2024-03-19T01:51:15.600 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.598+0000 7fd159d1d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1540c9610 0x7fd1541462e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:15.601 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.602+0000 7fd159d1d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1540c9610 0x7fd1541462e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33676/0 (socket says 172.21.15.55:33676) 2024-03-19T01:51:15.601 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.602+0000 7fd159d1d640 1 -- 172.21.15.55:0/1738815393 learned_addr learned my addr 172.21.15.55:0/1738815393 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:15.605 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.606+0000 7fd159d1d640 1 -- 172.21.15.55:0/1738815393 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd154152120 msgr2=0x7fd154146820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:15.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.606+0000 7fd159d1d640 1 --2- 172.21.15.55:0/1738815393 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd154152120 0x7fd154146820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:15.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.606+0000 7fd159d1d640 1 -- 172.21.15.55:0/1738815393 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd154070780 msgr2=0x7fd15414dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:15.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.606+0000 7fd159d1d640 1 --2- 172.21.15.55:0/1738815393 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd154070780 0x7fd15414dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:15.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.606+0000 7fd159d1d640 1 -- 172.21.15.55:0/1738815393 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd154156fe0 con 0x7fd1540c9610 2024-03-19T01:51:15.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.606+0000 7fd159d1d640 1 --2- 172.21.15.55:0/1738815393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1540c9610 0x7fd1541462e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd15000e930 tx=0x7fd15000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:15.612 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.610+0000 7fd146ffd640 1 -- 172.21.15.55:0/1738815393 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd150017070 con 0x7fd1540c9610 2024-03-19T01:51:15.612 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.610+0000 7fd146ffd640 1 -- 172.21.15.55:0/1738815393 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd15000ca00 con 0x7fd1540c9610 2024-03-19T01:51:15.612 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.610+0000 7fd15c782640 1 -- 172.21.15.55:0/1738815393 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1541f3e10 con 0x7fd1540c9610 2024-03-19T01:51:15.612 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.610+0000 7fd146ffd640 1 -- 172.21.15.55:0/1738815393 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15000cdf0 con 0x7fd1540c9610 2024-03-19T01:51:15.612 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.610+0000 7fd15c782640 1 -- 172.21.15.55:0/1738815393 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1540c5a80 con 0x7fd1540c9610 2024-03-19T01:51:15.613 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.614+0000 7fd15c782640 1 -- 172.21.15.55:0/1738815393 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd154068a40 con 0x7fd1540c9610 2024-03-19T01:51:15.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.906+0000 7fd146ffd640 1 -- 172.21.15.55:0/1738815393 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fd15001a070 con 0x7fd1540c9610 2024-03-19T01:51:15.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.906+0000 7fd146ffd640 1 --2- 172.21.15.55:0/1738815393 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd138067660 0x7fd138069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:15.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.906+0000 7fd15a51e640 1 --2- 172.21.15.55:0/1738815393 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd138067660 0x7fd138069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:15.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.906+0000 7fd15a51e640 1 --2- 172.21.15.55:0/1738815393 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd138067660 0x7fd138069b20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd1480ad000 tx=0x7fd148004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:15.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.942+0000 7fd146ffd640 1 -- 172.21.15.55:0/1738815393 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6500+0+0 (secure 0 0 0) 0x7fd150054780 con 0x7fd1540c9610 2024-03-19T01:51:15.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:15.942+0000 7fd146ffd640 1 -- 172.21.15.55:0/1738815393 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fd150056de0 con 0x7fd1540c9610 2024-03-19T01:51:16.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.126+0000 7fd15c782640 1 -- 172.21.15.55:0/1738815393 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fd1540c84b0 con 0x7fd1540c9610 2024-03-19T01:51:16.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.458+0000 7fd146ffd640 1 -- 172.21.15.55:0/1738815393 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v28) ==== 169+0+0 (secure 0 0 0) 0x7fd1500566e0 con 0x7fd1540c9610 2024-03-19T01:51:16.460 INFO:teuthology.orchestra.run.smithi055.stderr:set pool 4 allow_ec_overwrites to true 2024-03-19T01:51:16.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.462+0000 7fd15c782640 1 -- 172.21.15.55:0/1738815393 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd138067660 msgr2=0x7fd138069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:16.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.462+0000 7fd15c782640 1 --2- 172.21.15.55:0/1738815393 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd138067660 0x7fd138069b20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd1480ad000 tx=0x7fd148004740 comp rx=0 tx=0).stop 2024-03-19T01:51:16.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.462+0000 7fd15c782640 1 -- 172.21.15.55:0/1738815393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1540c9610 msgr2=0x7fd1541462e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:16.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.462+0000 7fd15c782640 1 --2- 172.21.15.55:0/1738815393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1540c9610 0x7fd1541462e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd15000e930 tx=0x7fd15000edf0 comp rx=0 tx=0).stop 2024-03-19T01:51:16.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.462+0000 7fd15c782640 1 -- 172.21.15.55:0/1738815393 shutdown_connections 2024-03-19T01:51:16.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.462+0000 7fd15c782640 1 --2- 172.21.15.55:0/1738815393 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd138067660 0x7fd138069b20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:16.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.462+0000 7fd15c782640 1 --2- 172.21.15.55:0/1738815393 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd154152120 0x7fd154146820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:16.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.462+0000 7fd15c782640 1 --2- 172.21.15.55:0/1738815393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1540c9610 0x7fd1541462e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:16.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.462+0000 7fd15c782640 1 --2- 172.21.15.55:0/1738815393 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd154070780 0x7fd15414dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:16.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.462+0000 7fd15c782640 1 -- 172.21.15.55:0/1738815393 >> 172.21.15.55:0/1738815393 conn(0x7fd154067600 msgr2=0x7fd15414c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:16.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.462+0000 7fd15c782640 1 -- 172.21.15.55:0/1738815393 shutdown_connections 2024-03-19T01:51:16.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.462+0000 7fd15c782640 1 -- 172.21.15.55:0/1738815393 wait complete. 2024-03-19T01:51:16.482 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-19T01:51:16.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.618+0000 7fa95001e640 1 Processor -- start 2024-03-19T01:51:16.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.618+0000 7fa95001e640 1 -- start start 2024-03-19T01:51:16.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.618+0000 7fa95001e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa9480cbf30 0x7fa9480cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:16.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.618+0000 7fa95001e640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa94806c6f0 0x7fa9480cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:16.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.618+0000 7fa95001e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa94806cb90 con 0x7fa94815a3b0 2024-03-19T01:51:16.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.618+0000 7fa95001e640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fa9480ce640 con 0x7fa94806c6f0 2024-03-19T01:51:16.621 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.618+0000 7fa95001e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa9480cda30 con 0x7fa9480cbf30 2024-03-19T01:51:16.621 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.622+0000 7fa94d5b9640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa94806c6f0 0x7fa9480cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:16.622 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.622+0000 7fa94d5b9640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa94806c6f0 0x7fa9480cf010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:36660/0 (socket says 172.21.15.55:36660) 2024-03-19T01:51:16.622 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.622+0000 7fa94d5b9640 1 -- 172.21.15.55:0/1484423399 learned_addr learned my addr 172.21.15.55:0/1484423399 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:16.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.626+0000 7fa94d5b9640 1 -- 172.21.15.55:0/1484423399 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa9480cbf30 msgr2=0x7fa9480cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:16.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.626+0000 7fa94d5b9640 1 --2- 172.21.15.55:0/1484423399 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa9480cbf30 0x7fa9480cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:16.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.626+0000 7fa94d5b9640 1 -- 172.21.15.55:0/1484423399 >> v1:172.21.15.55:6789/0 conn(0x7fa94815a3b0 legacy=0x7fa94815a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:16.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.626+0000 7fa94d5b9640 1 -- 172.21.15.55:0/1484423399 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9480cdd10 con 0x7fa94806c6f0 2024-03-19T01:51:16.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.638+0000 7fa94d5b9640 1 --2- 172.21.15.55:0/1484423399 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa94806c6f0 0x7fa9480cf010 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa938009870 tx=0x7fa93801c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad1c164ae9899170 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:16.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.646+0000 7fa94cdb8640 1 -- 172.21.15.55:0/1484423399 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9380b2070 con 0x7fa94806c6f0 2024-03-19T01:51:16.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.646+0000 7fa94cdb8640 1 -- 172.21.15.55:0/1484423399 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9380a9070 con 0x7fa94806c6f0 2024-03-19T01:51:16.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.646+0000 7fa94cdb8640 1 -- 172.21.15.55:0/1484423399 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9380ad880 con 0x7fa94806c6f0 2024-03-19T01:51:16.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.646+0000 7fa95001e640 1 -- 172.21.15.55:0/1484423399 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa94806c6f0 msgr2=0x7fa9480cf010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:16.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.646+0000 7fa95001e640 1 --2- 172.21.15.55:0/1484423399 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa94806c6f0 0x7fa9480cf010 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa938009870 tx=0x7fa93801c490 comp rx=0 tx=0).stop 2024-03-19T01:51:16.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.646+0000 7fa95001e640 1 -- 172.21.15.55:0/1484423399 shutdown_connections 2024-03-19T01:51:16.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.646+0000 7fa95001e640 1 --2- 172.21.15.55:0/1484423399 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa94806c6f0 0x7fa9480cf010 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:16.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.646+0000 7fa95001e640 1 --2- 172.21.15.55:0/1484423399 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa9480cbf30 0x7fa9480cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:16.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.646+0000 7fa95001e640 1 -- 172.21.15.55:0/1484423399 >> 172.21.15.55:0/1484423399 conn(0x7fa9480b16a0 msgr2=0x7fa9480b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:16.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.646+0000 7fa95001e640 1 -- 172.21.15.55:0/1484423399 shutdown_connections 2024-03-19T01:51:16.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.646+0000 7fa95001e640 1 -- 172.21.15.55:0/1484423399 wait complete. 2024-03-19T01:51:16.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.650+0000 7fa95001e640 1 Processor -- start 2024-03-19T01:51:16.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.650+0000 7fa95001e640 1 -- start start 2024-03-19T01:51:16.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.650+0000 7fa95001e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa94806c6f0 0x7fa9480b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:16.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.650+0000 7fa95001e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa9480cbf30 0x7fa94815d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:16.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.650+0000 7fa95001e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa94815a3b0 0x7fa94815ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:16.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.650+0000 7fa95001e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa9480cc310 con 0x7fa94806c6f0 2024-03-19T01:51:16.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.650+0000 7fa95001e640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa9480b23b0 con 0x7fa9480cbf30 2024-03-19T01:51:16.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.650+0000 7fa95001e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa9480cc490 con 0x7fa94815a3b0 2024-03-19T01:51:16.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.650+0000 7fa94d5b9640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa9480cbf30 0x7fa94815d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:16.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.650+0000 7fa94d5b9640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa9480cbf30 0x7fa94815d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:36674/0 (socket says 172.21.15.55:36674) 2024-03-19T01:51:16.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.650+0000 7fa94d5b9640 1 -- 172.21.15.55:0/1926754012 learned_addr learned my addr 172.21.15.55:0/1926754012 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:16.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.658+0000 7fa94d5b9640 1 -- 172.21.15.55:0/1926754012 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa94815a3b0 msgr2=0x7fa94815ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:16.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.658+0000 7fa94d5b9640 1 --2- 172.21.15.55:0/1926754012 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa94815a3b0 0x7fa94815ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:16.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.658+0000 7fa94d5b9640 1 -- 172.21.15.55:0/1926754012 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa94806c6f0 msgr2=0x7fa9480b4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:16.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.658+0000 7fa94d5b9640 1 --2- 172.21.15.55:0/1926754012 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa94806c6f0 0x7fa9480b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:16.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.658+0000 7fa94d5b9640 1 -- 172.21.15.55:0/1926754012 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9480c9250 con 0x7fa9480cbf30 2024-03-19T01:51:16.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.662+0000 7fa94d5b9640 1 --2- 172.21.15.55:0/1926754012 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa9480cbf30 0x7fa94815d890 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa9380b0040 tx=0x7fa93801acd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:16.668 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.666+0000 7fa93e7fc640 1 -- 172.21.15.55:0/1926754012 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9380bc020 con 0x7fa9480cbf30 2024-03-19T01:51:16.668 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.666+0000 7fa93e7fc640 1 -- 172.21.15.55:0/1926754012 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9380b8420 con 0x7fa9480cbf30 2024-03-19T01:51:16.668 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.666+0000 7fa93e7fc640 1 -- 172.21.15.55:0/1926754012 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9380b2040 con 0x7fa9480cbf30 2024-03-19T01:51:16.668 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.670+0000 7fa95001e640 1 -- 172.21.15.55:0/1926754012 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9481f72f0 con 0x7fa9480cbf30 2024-03-19T01:51:16.668 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.670+0000 7fa95001e640 1 -- 172.21.15.55:0/1926754012 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa94815b990 con 0x7fa9480cbf30 2024-03-19T01:51:16.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.670+0000 7fa95001e640 1 -- 172.21.15.55:0/1926754012 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa948068330 con 0x7fa9480cbf30 2024-03-19T01:51:16.956 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.954+0000 7fa93e7fc640 1 -- 172.21.15.55:0/1926754012 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa9380bf020 con 0x7fa9480cbf30 2024-03-19T01:51:16.956 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.954+0000 7fa93e7fc640 1 --2- 172.21.15.55:0/1926754012 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa92c0676e0 0x7fa92c069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:16.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.958+0000 7fa94ddba640 1 --2- 172.21.15.55:0/1926754012 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa92c0676e0 0x7fa92c069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:16.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.958+0000 7fa94ddba640 1 --2- 172.21.15.55:0/1926754012 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa92c0676e0 0x7fa92c069ba0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa944009690 tx=0x7fa944006c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:16.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.958+0000 7fa93e7fc640 1 -- 172.21.15.55:0/1926754012 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6500+0+0 (secure 0 0 0) 0x7fa9380cbe00 con 0x7fa9480cbf30 2024-03-19T01:51:16.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:16.970+0000 7fa93e7fc640 1 -- 172.21.15.55:0/1926754012 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa9380c6030 con 0x7fa9480cbf30 2024-03-19T01:51:17.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:17.182+0000 7fa95001e640 1 -- 172.21.15.55:0/1926754012 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fa9480b2190 con 0x7fa9480cbf30 2024-03-19T01:51:18.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.378+0000 7fa93e7fc640 1 -- 172.21.15.55:0/1926754012 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v8) ==== 137+0+0 (secure 0 0 0) 0x7fa938016110 con 0x7fa9480cbf30 2024-03-19T01:51:18.379 INFO:teuthology.orchestra.run.smithi055.stderr:added data pool 4 to fsmap 2024-03-19T01:51:18.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.382+0000 7fa95001e640 1 -- 172.21.15.55:0/1926754012 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa92c0676e0 msgr2=0x7fa92c069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:18.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.382+0000 7fa95001e640 1 --2- 172.21.15.55:0/1926754012 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa92c0676e0 0x7fa92c069ba0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa944009690 tx=0x7fa944006c00 comp rx=0 tx=0).stop 2024-03-19T01:51:18.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.382+0000 7fa95001e640 1 -- 172.21.15.55:0/1926754012 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa9480cbf30 msgr2=0x7fa94815d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:18.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.382+0000 7fa95001e640 1 --2- 172.21.15.55:0/1926754012 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa9480cbf30 0x7fa94815d890 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa9380b0040 tx=0x7fa93801acd0 comp rx=0 tx=0).stop 2024-03-19T01:51:18.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.382+0000 7fa95001e640 1 -- 172.21.15.55:0/1926754012 shutdown_connections 2024-03-19T01:51:18.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.382+0000 7fa95001e640 1 --2- 172.21.15.55:0/1926754012 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa92c0676e0 0x7fa92c069ba0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:18.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.382+0000 7fa95001e640 1 --2- 172.21.15.55:0/1926754012 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa94815a3b0 0x7fa94815ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:18.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.382+0000 7fa95001e640 1 --2- 172.21.15.55:0/1926754012 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa9480cbf30 0x7fa94815d890 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:18.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.382+0000 7fa95001e640 1 --2- 172.21.15.55:0/1926754012 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa94806c6f0 0x7fa9480b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:18.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.382+0000 7fa95001e640 1 -- 172.21.15.55:0/1926754012 >> 172.21.15.55:0/1926754012 conn(0x7fa9480b16a0 msgr2=0x7fa9480af610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:18.384 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.382+0000 7fa95001e640 1 -- 172.21.15.55:0/1926754012 shutdown_connections 2024-03-19T01:51:18.384 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.382+0000 7fa95001e640 1 -- 172.21.15.55:0/1926754012 wait complete. 2024-03-19T01:51:18.403 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:51:18.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.530+0000 7f6122bb7640 1 Processor -- start 2024-03-19T01:51:18.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.530+0000 7f6122bb7640 1 -- start start 2024-03-19T01:51:18.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.530+0000 7f6122bb7640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f611c070780 0x7f611c15b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:18.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.530+0000 7f6122bb7640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f611c15bb00 0x7f611c15bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:18.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.530+0000 7f6122bb7640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f611c15c4a0 con 0x7f611c152120 2024-03-19T01:51:18.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.530+0000 7f6122bb7640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f611c152500 con 0x7f611c15bb00 2024-03-19T01:51:18.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.530+0000 7f6122bb7640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f611c070590 con 0x7f611c070780 2024-03-19T01:51:18.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.534+0000 7f611bfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f611c070780 0x7f611c15b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:18.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.534+0000 7f6120953640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f611c152120 0x7f611c0c90d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:46882/0 (socket says 172.21.15.55:46882) 2024-03-19T01:51:18.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.534+0000 7f6120953640 1 -- 172.21.15.55:0/3440295865 learned_addr learned my addr 172.21.15.55:0/3440295865 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:51:18.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.542+0000 7f611bfff640 1 -- 172.21.15.55:0/3440295865 >> v2:172.21.15.55:3300/0 conn(0x7f611c15bb00 msgr2=0x7f611c15bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:18.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.542+0000 7f611bfff640 1 --2- 172.21.15.55:0/3440295865 >> v2:172.21.15.55:3300/0 conn(0x7f611c15bb00 0x7f611c15bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:18.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.542+0000 7f611bfff640 1 -- 172.21.15.55:0/3440295865 >> v1:172.21.15.84:6789/0 conn(0x7f611c152120 legacy=0x7f611c0c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:18.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.542+0000 7f611bfff640 1 -- 172.21.15.55:0/3440295865 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f611c06fff0 con 0x7f611c070780 2024-03-19T01:51:18.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.550+0000 7f611bfff640 1 --2- 172.21.15.55:0/3440295865 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f611c070780 0x7f611c15b5c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f61080097c0 tx=0x7f610801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ece0792756fe1c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:18.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.566+0000 7f611b7fe640 1 -- 172.21.15.55:0/3440295865 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61080b0070 con 0x7f611c070780 2024-03-19T01:51:18.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.566+0000 7f611b7fe640 1 -- 172.21.15.55:0/3440295865 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61080ab600 con 0x7f611c070780 2024-03-19T01:51:18.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.566+0000 7f611b7fe640 1 -- 172.21.15.55:0/3440295865 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61080ab9d0 con 0x7f611c070780 2024-03-19T01:51:18.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.566+0000 7f6122bb7640 1 -- 172.21.15.55:0/3440295865 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f611c070780 msgr2=0x7f611c15b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:18.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.566+0000 7f6122bb7640 1 --2- 172.21.15.55:0/3440295865 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f611c070780 0x7f611c15b5c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f61080097c0 tx=0x7f610801c4a0 comp rx=0 tx=0).stop 2024-03-19T01:51:18.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.566+0000 7f6122bb7640 1 -- 172.21.15.55:0/3440295865 shutdown_connections 2024-03-19T01:51:18.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.566+0000 7f6122bb7640 1 --2- 172.21.15.55:0/3440295865 >> v2:172.21.15.55:3300/0 conn(0x7f611c15bb00 0x7f611c15bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:18.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.566+0000 7f6122bb7640 1 --2- 172.21.15.55:0/3440295865 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f611c070780 0x7f611c15b5c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:18.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.566+0000 7f6122bb7640 1 -- 172.21.15.55:0/3440295865 >> 172.21.15.55:0/3440295865 conn(0x7f611c067600 msgr2=0x7f611c071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:18.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.566+0000 7f6122bb7640 1 -- 172.21.15.55:0/3440295865 shutdown_connections 2024-03-19T01:51:18.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.566+0000 7f6122bb7640 1 -- 172.21.15.55:0/3440295865 wait complete. 2024-03-19T01:51:18.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.566+0000 7f6122bb7640 1 Processor -- start 2024-03-19T01:51:18.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.566+0000 7f6122bb7640 1 -- start start 2024-03-19T01:51:18.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.566+0000 7f6122bb7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f611c070780 0x7f611c1f4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:18.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.566+0000 7f6122bb7640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f611c152120 0x7f611c1f0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:18.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.566+0000 7f6122bb7640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f611c15bb00 0x7f611c0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:18.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.566+0000 7f6122bb7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f611c0c31c0 con 0x7f611c070780 2024-03-19T01:51:18.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.566+0000 7f6122bb7640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f611c0c2ee0 con 0x7f611c15bb00 2024-03-19T01:51:18.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.566+0000 7f6122bb7640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f611c0c97e0 con 0x7f611c152120 2024-03-19T01:51:18.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.566+0000 7f6120953640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f611c070780 0x7f611c1f4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:18.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.570+0000 7f6120953640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f611c070780 0x7f611c1f4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33696/0 (socket says 172.21.15.55:33696) 2024-03-19T01:51:18.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.570+0000 7f6120953640 1 -- 172.21.15.55:0/2642109877 learned_addr learned my addr 172.21.15.55:0/2642109877 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:18.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.570+0000 7f6121154640 1 --2- 172.21.15.55:0/2642109877 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f611c15bb00 0x7f611c0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:18.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.570+0000 7f611bfff640 1 --2- 172.21.15.55:0/2642109877 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f611c152120 0x7f611c1f0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:18.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.570+0000 7f6120953640 1 -- 172.21.15.55:0/2642109877 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f611c152120 msgr2=0x7f611c1f0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:18.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.570+0000 7f6120953640 1 --2- 172.21.15.55:0/2642109877 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f611c152120 0x7f611c1f0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:18.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.570+0000 7f6120953640 1 -- 172.21.15.55:0/2642109877 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f611c15bb00 msgr2=0x7f611c0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:18.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.570+0000 7f6120953640 1 --2- 172.21.15.55:0/2642109877 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f611c15bb00 0x7f611c0b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:18.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.570+0000 7f6120953640 1 -- 172.21.15.55:0/2642109877 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f611c15b620 con 0x7f611c070780 2024-03-19T01:51:18.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.574+0000 7f6120953640 1 --2- 172.21.15.55:0/2642109877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f611c070780 0x7f611c1f4fe0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f611000c450 tx=0x7f611000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:18.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.578+0000 7f61197fa640 1 -- 172.21.15.55:0/2642109877 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6110017070 con 0x7f611c070780 2024-03-19T01:51:18.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.578+0000 7f61197fa640 1 -- 172.21.15.55:0/2642109877 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6110005cf0 con 0x7f611c070780 2024-03-19T01:51:18.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.578+0000 7f6122bb7640 1 -- 172.21.15.55:0/2642109877 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f611c15aa60 con 0x7f611c070780 2024-03-19T01:51:18.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.578+0000 7f61197fa640 1 -- 172.21.15.55:0/2642109877 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6110007290 con 0x7f611c070780 2024-03-19T01:51:18.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.578+0000 7f6122bb7640 1 -- 172.21.15.55:0/2642109877 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f611c1f4500 con 0x7f611c070780 2024-03-19T01:51:18.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.582+0000 7f6122bb7640 1 -- 172.21.15.55:0/2642109877 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f611c068a40 con 0x7f611c070780 2024-03-19T01:51:18.599 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.598+0000 7f61197fa640 1 -- 172.21.15.55:0/2642109877 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f611001a070 con 0x7f611c070780 2024-03-19T01:51:18.600 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.598+0000 7f61197fa640 1 --2- 172.21.15.55:0/2642109877 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6100067690 0x7f6100069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:18.600 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.602+0000 7f611bfff640 1 --2- 172.21.15.55:0/2642109877 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6100067690 0x7f6100069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:18.601 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.602+0000 7f611bfff640 1 --2- 172.21.15.55:0/2642109877 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6100067690 0x7f6100069b50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f61080ad000 tx=0x7f61080a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:18.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.890+0000 7f61197fa640 1 -- 172.21.15.55:0/2642109877 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f6110053680 con 0x7f611c070780 2024-03-19T01:51:18.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:18.890+0000 7f61197fa640 1 -- 172.21.15.55:0/2642109877 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f6110021020 con 0x7f611c070780 2024-03-19T01:51:19.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.094+0000 7f6122bb7640 1 -- 172.21.15.55:0/2642109877 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f611c0b0af0 con 0x7f611c070780 2024-03-19T01:51:19.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.126+0000 7f61197fa640 1 -- 172.21.15.55:0/2642109877 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6822 (secure 0 0 0) 0x7f61100555e0 con 0x7f611c070780 2024-03-19T01:51:19.126 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:51:19.126 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4200,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4237,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4249,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6839/354880008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6838","nonce":354880008},{"type":"v1","addr":"172.21.15.84:6839","nonce":354880008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:51:17.241672+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4255,"qdb_cluster":[4255]},"id":1}]} 2024-03-19T01:51:19.127 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 8 2024-03-19T01:51:19.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.126+0000 7f6122bb7640 1 -- 172.21.15.55:0/2642109877 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6100067690 msgr2=0x7f6100069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:19.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.126+0000 7f6122bb7640 1 --2- 172.21.15.55:0/2642109877 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6100067690 0x7f6100069b50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f61080ad000 tx=0x7f61080a6000 comp rx=0 tx=0).stop 2024-03-19T01:51:19.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.126+0000 7f6122bb7640 1 -- 172.21.15.55:0/2642109877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f611c070780 msgr2=0x7f611c1f4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:19.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.126+0000 7f6122bb7640 1 --2- 172.21.15.55:0/2642109877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f611c070780 0x7f611c1f4fe0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f611000c450 tx=0x7f611000c910 comp rx=0 tx=0).stop 2024-03-19T01:51:19.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.126+0000 7f6122bb7640 1 -- 172.21.15.55:0/2642109877 shutdown_connections 2024-03-19T01:51:19.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.130+0000 7f6122bb7640 1 --2- 172.21.15.55:0/2642109877 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6100067690 0x7f6100069b50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:19.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.130+0000 7f6122bb7640 1 --2- 172.21.15.55:0/2642109877 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f611c15bb00 0x7f611c0b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:19.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.130+0000 7f6122bb7640 1 --2- 172.21.15.55:0/2642109877 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f611c152120 0x7f611c1f0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:19.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.130+0000 7f6122bb7640 1 --2- 172.21.15.55:0/2642109877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f611c070780 0x7f611c1f4fe0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:19.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.130+0000 7f6122bb7640 1 -- 172.21.15.55:0/2642109877 >> 172.21.15.55:0/2642109877 conn(0x7f611c067600 msgr2=0x7f611c14d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:19.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.130+0000 7f6122bb7640 1 -- 172.21.15.55:0/2642109877 shutdown_connections 2024-03-19T01:51:19.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.130+0000 7f6122bb7640 1 -- 172.21.15.55:0/2642109877 wait complete. 2024-03-19T01:51:19.145 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:51:19.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.258+0000 7f3a35168640 1 Processor -- start 2024-03-19T01:51:19.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.258+0000 7f3a35168640 1 -- start start 2024-03-19T01:51:19.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.258+0000 7f3a35168640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3a30150c30 0x7f3a300c6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:19.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.258+0000 7f3a35168640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3a3006c6c0 0x7f3a3015ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:19.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.258+0000 7f3a35168640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f3a30067390 0x7f3a3006e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:19.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.258+0000 7f3a35168640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f3a3006cb20 con 0x7f3a30150c30 2024-03-19T01:51:19.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.258+0000 7f3a35168640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f3a3006b340 con 0x7f3a30067390 2024-03-19T01:51:19.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.258+0000 7f3a35168640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3a3006b620 con 0x7f3a3006c6c0 2024-03-19T01:51:19.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.262+0000 7f3a2ed9d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3a30150c30 0x7f3a300c6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:19.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.262+0000 7f3a2ed9d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3a30150c30 0x7f3a300c6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40922/0 (socket says 172.21.15.55:40922) 2024-03-19T01:51:19.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.262+0000 7f3a2ed9d640 1 -- 172.21.15.55:0/3555536864 learned_addr learned my addr 172.21.15.55:0/3555536864 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:19.262 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.262+0000 7f3a2e59c640 1 --2- 172.21.15.55:0/3555536864 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3a3006c6c0 0x7f3a3015ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:19.262 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.262+0000 7f3a2f59e640 1 --2- 172.21.15.55:0/3555536864 >> v2:172.21.15.84:3300/0 conn(0x7f3a30067390 0x7f3a3006e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:19.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.262+0000 7f3a2ed9d640 1 -- 172.21.15.55:0/3555536864 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3a3006c6c0 msgr2=0x7f3a3015ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:19.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.262+0000 7f3a2ed9d640 1 --2- 172.21.15.55:0/3555536864 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3a3006c6c0 0x7f3a3015ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:19.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.262+0000 7f3a2ed9d640 1 -- 172.21.15.55:0/3555536864 >> v2:172.21.15.84:3300/0 conn(0x7f3a30067390 msgr2=0x7f3a3006e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:19.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.262+0000 7f3a2ed9d640 1 --2- 172.21.15.55:0/3555536864 >> v2:172.21.15.84:3300/0 conn(0x7f3a30067390 0x7f3a3006e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:19.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.262+0000 7f3a2ed9d640 1 -- 172.21.15.55:0/3555536864 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3006b900 con 0x7f3a30150c30 2024-03-19T01:51:19.267 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.266+0000 7f3a2ed9d640 1 --2- 172.21.15.55:0/3555536864 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a30150c30 0x7f3a300c6e80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3a300712c0 tx=0x7f3a1801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ea020d62ed32 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:19.269 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.270+0000 7f3a2dd9b640 1 -- 172.21.15.55:0/3555536864 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a180b5070 con 0x7f3a30150c30 2024-03-19T01:51:19.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.270+0000 7f3a2dd9b640 1 -- 172.21.15.55:0/3555536864 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a180a89d0 con 0x7f3a30150c30 2024-03-19T01:51:19.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.270+0000 7f3a2dd9b640 1 -- 172.21.15.55:0/3555536864 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a180a8da0 con 0x7f3a30150c30 2024-03-19T01:51:19.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.270+0000 7f3a35168640 1 -- 172.21.15.55:0/3555536864 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a30150c30 msgr2=0x7f3a300c6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:19.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.270+0000 7f3a35168640 1 --2- 172.21.15.55:0/3555536864 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a30150c30 0x7f3a300c6e80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3a300712c0 tx=0x7f3a1801ed10 comp rx=0 tx=0).stop 2024-03-19T01:51:19.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.270+0000 7f3a35168640 1 -- 172.21.15.55:0/3555536864 shutdown_connections 2024-03-19T01:51:19.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.270+0000 7f3a35168640 1 --2- 172.21.15.55:0/3555536864 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3a3006c6c0 0x7f3a3015ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:19.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.270+0000 7f3a35168640 1 --2- 172.21.15.55:0/3555536864 >> v2:172.21.15.84:3300/0 conn(0x7f3a30067390 0x7f3a3006e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:19.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.270+0000 7f3a35168640 1 --2- 172.21.15.55:0/3555536864 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a30150c30 0x7f3a300c6e80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:19.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.274+0000 7f3a35168640 1 -- 172.21.15.55:0/3555536864 >> 172.21.15.55:0/3555536864 conn(0x7f3a300cc270 msgr2=0x7f3a300cc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:19.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.274+0000 7f3a35168640 1 -- 172.21.15.55:0/3555536864 shutdown_connections 2024-03-19T01:51:19.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.274+0000 7f3a35168640 1 -- 172.21.15.55:0/3555536864 wait complete. 2024-03-19T01:51:19.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.274+0000 7f3a35168640 1 Processor -- start 2024-03-19T01:51:19.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.274+0000 7f3a35168640 1 -- start start 2024-03-19T01:51:19.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.274+0000 7f3a35168640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a30067390 0x7f3a30142f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:19.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.274+0000 7f3a35168640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3a3006c6c0 0x7f3a3014d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:19.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.274+0000 7f3a35168640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3a30150c30 0x7f3a3014bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:19.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.274+0000 7f3a35168640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3006c1a0 con 0x7f3a30067390 2024-03-19T01:51:19.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.274+0000 7f3a35168640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3006bec0 con 0x7f3a30150c30 2024-03-19T01:51:19.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.274+0000 7f3a35168640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3a30067800 con 0x7f3a3006c6c0 2024-03-19T01:51:19.276 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.274+0000 7f3a2ed9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a30067390 0x7f3a30142f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:19.276 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.274+0000 7f3a2ed9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a30067390 0x7f3a30142f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40938/0 (socket says 172.21.15.55:40938) 2024-03-19T01:51:19.276 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.274+0000 7f3a2ed9d640 1 -- 172.21.15.55:0/900522959 learned_addr learned my addr 172.21.15.55:0/900522959 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:19.276 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.274+0000 7f3a2f59e640 1 --2- 172.21.15.55:0/900522959 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3a30150c30 0x7f3a3014bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:19.276 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.274+0000 7f3a2e59c640 1 --2- 172.21.15.55:0/900522959 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3a3006c6c0 0x7f3a3014d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:19.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.278+0000 7f3a2ed9d640 1 -- 172.21.15.55:0/900522959 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3a3006c6c0 msgr2=0x7f3a3014d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:19.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.278+0000 7f3a2ed9d640 1 --2- 172.21.15.55:0/900522959 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3a3006c6c0 0x7f3a3014d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:19.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.278+0000 7f3a2ed9d640 1 -- 172.21.15.55:0/900522959 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3a30150c30 msgr2=0x7f3a3014bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:19.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.278+0000 7f3a2ed9d640 1 --2- 172.21.15.55:0/900522959 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3a30150c30 0x7f3a3014bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:19.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.278+0000 7f3a2ed9d640 1 -- 172.21.15.55:0/900522959 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3014cbd0 con 0x7f3a30067390 2024-03-19T01:51:19.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.282+0000 7f3a2ed9d640 1 --2- 172.21.15.55:0/900522959 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a30067390 0x7f3a30142f20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3a1801d3c0 tx=0x7f3a1801bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:19.285 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.286+0000 7f3a277fe640 1 -- 172.21.15.55:0/900522959 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a180b5070 con 0x7f3a30067390 2024-03-19T01:51:19.285 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.286+0000 7f3a277fe640 1 -- 172.21.15.55:0/900522959 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a180b0a70 con 0x7f3a30067390 2024-03-19T01:51:19.285 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.286+0000 7f3a277fe640 1 -- 172.21.15.55:0/900522959 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a180b0e60 con 0x7f3a30067390 2024-03-19T01:51:19.285 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.286+0000 7f3a35168640 1 -- 172.21.15.55:0/900522959 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a30140370 con 0x7f3a30067390 2024-03-19T01:51:19.285 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.286+0000 7f3a35168640 1 -- 172.21.15.55:0/900522959 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a3014db50 con 0x7f3a30067390 2024-03-19T01:51:19.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.286+0000 7f3a35168640 1 -- 172.21.15.55:0/900522959 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a30068a40 con 0x7f3a30067390 2024-03-19T01:51:19.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.318+0000 7f3a277fe640 1 -- 172.21.15.55:0/900522959 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f3a180c2060 con 0x7f3a30067390 2024-03-19T01:51:19.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.318+0000 7f3a277fe640 1 --2- 172.21.15.55:0/900522959 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3a100676e0 0x7f3a10069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:19.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.318+0000 7f3a2e59c640 1 --2- 172.21.15.55:0/900522959 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3a100676e0 0x7f3a10069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:19.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.318+0000 7f3a2e59c640 1 --2- 172.21.15.55:0/900522959 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3a100676e0 0x7f3a10069ba0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3a30154210 tx=0x7f3a20008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:19.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.522+0000 7f3a277fe640 1 -- 172.21.15.55:0/900522959 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f3a180bce00 con 0x7f3a30067390 2024-03-19T01:51:19.636 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.634+0000 7f3a277fe640 1 -- 172.21.15.55:0/900522959 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f3a3014cbd0 con 0x7f3a30067390 2024-03-19T01:51:19.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:19.818+0000 7f3a35168640 1 -- 172.21.15.55:0/900522959 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3a3014cf40 con 0x7f3a30067390 2024-03-19T01:51:20.010 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.010+0000 7f3a277fe640 1 -- 172.21.15.55:0/900522959 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+18430 (secure 0 0 0) 0x7f3a18018870 con 0x7f3a30067390 2024-03-19T01:51:20.011 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:51:20.011 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":29,"fsid":"def9d4e5-0bc7-449d-b893-2490cbd10e25","created":"2024-03-19T01:50:02.482954+0000","modified":"2024-03-19T01:51:17.228291+0000","last_up_change":"2024-03-19T01:50:21.735508+0000","last_in_change":"2024-03-19T01:50:12.168944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:50:23.456904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:51:04.578583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:51:06.871659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T01:51:14.274442+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"35d5e3c5-d089-4f38-8961-98d064188f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6825","nonce":2757675043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6827","nonce":2757675043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6831","nonce":2757675043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6829","nonce":2757675043}]},"public_addr":"172.21.15.55:6825/2757675043","cluster_addr":"172.21.15.55:6827/2757675043","heartbeat_back_addr":"172.21.15.55:6831/2757675043","heartbeat_front_addr":"172.21.15.55:6829/2757675043","state":["exists","up"]},{"osd":1,"uuid":"8e58f791-001a-44d0-a464-ca68a479f70c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6817","nonce":543460007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6819","nonce":543460007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6823","nonce":543460007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6821","nonce":543460007}]},"public_addr":"172.21.15.55:6817/543460007","cluster_addr":"172.21.15.55:6819/543460007","heartbeat_back_addr":"172.21.15.55:6823/543460007","heartbeat_front_addr":"172.21.15.55:6821/543460007","state":["exists","up"]},{"osd":2,"uuid":"d44f38b8-67f4-43de-98a6-f3c520acaf10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6801","nonce":1760253550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6803","nonce":1760253550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6807","nonce":1760253550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6805","nonce":1760253550}]},"public_addr":"172.21.15.55:6801/1760253550","cluster_addr":"172.21.15.55:6803/1760253550","heartbeat_back_addr":"172.21.15.55:6807/1760253550","heartbeat_front_addr":"172.21.15.55:6805/1760253550","state":["exists","up"]},{"osd":3,"uuid":"850a7316-2d74-43f0-8649-b5f8d8f6692e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6809","nonce":2797616448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6811","nonce":2797616448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6815","nonce":2797616448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6813","nonce":2797616448}]},"public_addr":"172.21.15.55:6809/2797616448","cluster_addr":"172.21.15.55:6811/2797616448","heartbeat_back_addr":"172.21.15.55:6815/2797616448","heartbeat_front_addr":"172.21.15.55:6813/2797616448","state":["exists","up"]},{"osd":4,"uuid":"4e6152ae-a77d-4bb6-a855-5782ad5411a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6819","nonce":2938436826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6821","nonce":2938436826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6825","nonce":2938436826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6823","nonce":2938436826}]},"public_addr":"172.21.15.84:6819/2938436826","cluster_addr":"172.21.15.84:6821/2938436826","heartbeat_back_addr":"172.21.15.84:6825/2938436826","heartbeat_front_addr":"172.21.15.84:6823/2938436826","state":["exists","up"]},{"osd":5,"uuid":"1217ceba-dd31-4615-96f1-dfaf57160f3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6803","nonce":460717295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6805","nonce":460717295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6809","nonce":460717295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6807","nonce":460717295}]},"public_addr":"172.21.15.84:6803/460717295","cluster_addr":"172.21.15.84:6805/460717295","heartbeat_back_addr":"172.21.15.84:6809/460717295","heartbeat_front_addr":"172.21.15.84:6807/460717295","state":["exists","up"]},{"osd":6,"uuid":"3d9ed78a-2284-4c28-a24a-70eb812c7836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6827","nonce":416123135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6829","nonce":416123135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6833","nonce":416123135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6831","nonce":416123135}]},"public_addr":"172.21.15.84:6827/416123135","cluster_addr":"172.21.15.84:6829/416123135","heartbeat_back_addr":"172.21.15.84:6833/416123135","heartbeat_front_addr":"172.21.15.84:6831/416123135","state":["exists","up"]},{"osd":7,"uuid":"c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6811","nonce":460083528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6813","nonce":460083528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6817","nonce":460083528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6815","nonce":460083528}]},"public_addr":"172.21.15.84:6811/460083528","cluster_addr":"172.21.15.84:6813/460083528","heartbeat_back_addr":"172.21.15.84:6817/460083528","heartbeat_front_addr":"172.21.15.84:6815/460083528","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.269126+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.121001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.075714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.092461+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.194676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.062833+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.316515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.237111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:51:20.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.014+0000 7f3a35168640 1 -- 172.21.15.55:0/900522959 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3a100676e0 msgr2=0x7f3a10069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:20.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.014+0000 7f3a35168640 1 --2- 172.21.15.55:0/900522959 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3a100676e0 0x7f3a10069ba0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3a30154210 tx=0x7f3a20008040 comp rx=0 tx=0).stop 2024-03-19T01:51:20.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.014+0000 7f3a35168640 1 -- 172.21.15.55:0/900522959 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a30067390 msgr2=0x7f3a30142f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:20.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.014+0000 7f3a35168640 1 --2- 172.21.15.55:0/900522959 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a30067390 0x7f3a30142f20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3a1801d3c0 tx=0x7f3a1801bab0 comp rx=0 tx=0).stop 2024-03-19T01:51:20.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.014+0000 7f3a35168640 1 -- 172.21.15.55:0/900522959 shutdown_connections 2024-03-19T01:51:20.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.014+0000 7f3a35168640 1 --2- 172.21.15.55:0/900522959 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3a100676e0 0x7f3a10069ba0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:20.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.014+0000 7f3a35168640 1 --2- 172.21.15.55:0/900522959 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3a30150c30 0x7f3a3014bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:20.015 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.014+0000 7f3a35168640 1 --2- 172.21.15.55:0/900522959 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3a3006c6c0 0x7f3a3014d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:20.015 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.014+0000 7f3a35168640 1 --2- 172.21.15.55:0/900522959 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a30067390 0x7f3a30142f20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:20.015 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.014+0000 7f3a35168640 1 -- 172.21.15.55:0/900522959 >> 172.21.15.55:0/900522959 conn(0x7f3a300cc270 msgr2=0x7f3a3014f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:20.015 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.014+0000 7f3a35168640 1 -- 172.21.15.55:0/900522959 shutdown_connections 2024-03-19T01:51:20.015 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.014+0000 7f3a35168640 1 -- 172.21.15.55:0/900522959 wait complete. 2024-03-19T01:51:20.033 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:51:20.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.142+0000 7f97fb4a2640 1 Processor -- start 2024-03-19T01:51:20.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.142+0000 7f97fb4a2640 1 -- start start 2024-03-19T01:51:20.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.142+0000 7f97fb4a2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f97f4072c60 0x7f97f4158a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:20.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.142+0000 7f97fb4a2640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f97f415b540 0x7f97f40c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:20.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.142+0000 7f97fb4a2640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97f4152660 con 0x7f97f40c39c0 2024-03-19T01:51:20.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.142+0000 7f97fb4a2640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f97f415b9b0 con 0x7f97f415b540 2024-03-19T01:51:20.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.142+0000 7f97fb4a2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f97f4073130 con 0x7f97f4072c60 2024-03-19T01:51:20.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.146+0000 7f97f9a3f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f97f415b540 0x7f97f40c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:20.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.146+0000 7f97f8a3d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f97f4072c60 0x7f97f4158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:20.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.146+0000 7f97f8a3d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f97f4072c60 0x7f97f4158a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:37060/0 (socket says 172.21.15.55:37060) 2024-03-19T01:51:20.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.146+0000 7f97f8a3d640 1 -- 172.21.15.55:0/2090522416 learned_addr learned my addr 172.21.15.55:0/2090522416 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:20.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.150+0000 7f97f8a3d640 1 -- 172.21.15.55:0/2090522416 >> v2:172.21.15.84:3300/0 conn(0x7f97f415b540 msgr2=0x7f97f40c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:20.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.150+0000 7f97f8a3d640 1 --2- 172.21.15.55:0/2090522416 >> v2:172.21.15.84:3300/0 conn(0x7f97f415b540 0x7f97f40c60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:20.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.150+0000 7f97f8a3d640 1 -- 172.21.15.55:0/2090522416 >> v1:172.21.15.84:6789/0 conn(0x7f97f40c39c0 legacy=0x7f97f406b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:20.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.150+0000 7f97f8a3d640 1 -- 172.21.15.55:0/2090522416 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f40733f0 con 0x7f97f4072c60 2024-03-19T01:51:20.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.158+0000 7f97f8a3d640 1 --2- 172.21.15.55:0/2090522416 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f97f4072c60 0x7f97f4158a80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009870 tx=0x7f97e401eb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee5cbc7f5b4781fe server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:20.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.170+0000 7f97ebfff640 1 -- 172.21.15.55:0/2090522416 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e40b5070 con 0x7f97f4072c60 2024-03-19T01:51:20.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.170+0000 7f97ebfff640 1 -- 172.21.15.55:0/2090522416 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97e40a87f0 con 0x7f97f4072c60 2024-03-19T01:51:20.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.170+0000 7f97ebfff640 1 -- 172.21.15.55:0/2090522416 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e40a8bc0 con 0x7f97f4072c60 2024-03-19T01:51:20.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.174+0000 7f97fb4a2640 1 -- 172.21.15.55:0/2090522416 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f97f4072c60 msgr2=0x7f97f4158a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:20.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.174+0000 7f97fb4a2640 1 --2- 172.21.15.55:0/2090522416 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f97f4072c60 0x7f97f4158a80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009870 tx=0x7f97e401eb30 comp rx=0 tx=0).stop 2024-03-19T01:51:20.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.174+0000 7f97fb4a2640 1 -- 172.21.15.55:0/2090522416 shutdown_connections 2024-03-19T01:51:20.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.174+0000 7f97fb4a2640 1 --2- 172.21.15.55:0/2090522416 >> v2:172.21.15.84:3300/0 conn(0x7f97f415b540 0x7f97f40c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:20.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.174+0000 7f97fb4a2640 1 --2- 172.21.15.55:0/2090522416 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f97f4072c60 0x7f97f4158a80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:20.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.174+0000 7f97fb4a2640 1 -- 172.21.15.55:0/2090522416 >> 172.21.15.55:0/2090522416 conn(0x7f97f40cc9c0 msgr2=0x7f97f40ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:20.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.174+0000 7f97fb4a2640 1 -- 172.21.15.55:0/2090522416 shutdown_connections 2024-03-19T01:51:20.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.174+0000 7f97fb4a2640 1 -- 172.21.15.55:0/2090522416 wait complete. 2024-03-19T01:51:20.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.174+0000 7f97fb4a2640 1 Processor -- start 2024-03-19T01:51:20.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.174+0000 7f97fb4a2640 1 -- start start 2024-03-19T01:51:20.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.174+0000 7f97fb4a2640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97f4072c60 0x7f97f415a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:20.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.174+0000 7f97fb4a2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f97f40c39c0 0x7f97f41385f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:20.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.174+0000 7f97fb4a2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97f415b540 0x7f97f40c44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:20.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.174+0000 7f97fb4a2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f97f4158560 con 0x7f97f415b540 2024-03-19T01:51:20.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.174+0000 7f97fb4a2640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f97f4158280 con 0x7f97f4072c60 2024-03-19T01:51:20.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.174+0000 7f97fb4a2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f97f4157f50 con 0x7f97f40c39c0 2024-03-19T01:51:20.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.178+0000 7f97f923e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97f4072c60 0x7f97f415a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:20.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.178+0000 7f97f923e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97f4072c60 0x7f97f415a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:39222/0 (socket says 172.21.15.55:39222) 2024-03-19T01:51:20.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.178+0000 7f97f923e640 1 -- 172.21.15.55:0/706575863 learned_addr learned my addr 172.21.15.55:0/706575863 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:20.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.178+0000 7f97f8a3d640 1 --2- 172.21.15.55:0/706575863 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f97f40c39c0 0x7f97f41385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:20.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.182+0000 7f97f9a3f640 1 --2- 172.21.15.55:0/706575863 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97f415b540 0x7f97f40c44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:20.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.186+0000 7f97f923e640 1 -- 172.21.15.55:0/706575863 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f97f40c39c0 msgr2=0x7f97f41385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:20.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.186+0000 7f97f923e640 1 --2- 172.21.15.55:0/706575863 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f97f40c39c0 0x7f97f41385f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:20.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.186+0000 7f97f923e640 1 -- 172.21.15.55:0/706575863 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97f415b540 msgr2=0x7f97f40c44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:20.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.186+0000 7f97f923e640 1 --2- 172.21.15.55:0/706575863 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97f415b540 0x7f97f40c44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:20.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.186+0000 7f97f923e640 1 -- 172.21.15.55:0/706575863 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f40c5f00 con 0x7f97f4072c60 2024-03-19T01:51:20.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.194+0000 7f97f923e640 1 --2- 172.21.15.55:0/706575863 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97f4072c60 0x7f97f415a5c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f97f000c490 tx=0x7f97f0001e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:20.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.206+0000 7f97e9ffb640 1 -- 172.21.15.55:0/706575863 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f0017070 con 0x7f97f4072c60 2024-03-19T01:51:20.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.206+0000 7f97e9ffb640 1 -- 172.21.15.55:0/706575863 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97f0003600 con 0x7f97f4072c60 2024-03-19T01:51:20.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.206+0000 7f97e9ffb640 1 -- 172.21.15.55:0/706575863 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f0003a30 con 0x7f97f4072c60 2024-03-19T01:51:20.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.206+0000 7f97fb4a2640 1 -- 172.21.15.55:0/706575863 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97f41574b0 con 0x7f97f4072c60 2024-03-19T01:51:20.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.206+0000 7f97fb4a2640 1 -- 172.21.15.55:0/706575863 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97f4070e60 con 0x7f97f4072c60 2024-03-19T01:51:20.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.206+0000 7f97fb4a2640 1 -- 172.21.15.55:0/706575863 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97f4068a40 con 0x7f97f4072c60 2024-03-19T01:51:20.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.554+0000 7f97e9ffb640 1 -- 172.21.15.55:0/706575863 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f97f001a070 con 0x7f97f4072c60 2024-03-19T01:51:20.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.558+0000 7f97e9ffb640 1 --2- 172.21.15.55:0/706575863 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f97d80676b0 0x7f97d8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:20.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.558+0000 7f97f8a3d640 1 --2- 172.21.15.55:0/706575863 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f97d80676b0 0x7f97d8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:20.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.558+0000 7f97f8a3d640 1 --2- 172.21.15.55:0/706575863 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f97d80676b0 0x7f97d8069b70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f97e40b2000 tx=0x7f97e40aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:20.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.566+0000 7f97e9ffb640 1 -- 172.21.15.55:0/706575863 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f97f00537c0 con 0x7f97f4072c60 2024-03-19T01:51:20.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.594+0000 7f97e9ffb640 1 -- 172.21.15.55:0/706575863 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f97f0021020 con 0x7f97f4072c60 2024-03-19T01:51:20.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.782+0000 7f97fb4a2640 1 -- 172.21.15.55:0/706575863 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f97f413de50 con 0x7f97f4072c60 2024-03-19T01:51:20.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.958+0000 7f97e9ffb640 1 -- 172.21.15.55:0/706575863 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+18430 (secure 0 0 0) 0x7f97f0055720 con 0x7f97f4072c60 2024-03-19T01:51:20.959 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:51:20.959 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":29,"fsid":"def9d4e5-0bc7-449d-b893-2490cbd10e25","created":"2024-03-19T01:50:02.482954+0000","modified":"2024-03-19T01:51:17.228291+0000","last_up_change":"2024-03-19T01:50:21.735508+0000","last_in_change":"2024-03-19T01:50:12.168944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:50:23.456904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:51:04.578583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:51:06.871659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T01:51:14.274442+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"35d5e3c5-d089-4f38-8961-98d064188f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6825","nonce":2757675043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6827","nonce":2757675043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6831","nonce":2757675043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6829","nonce":2757675043}]},"public_addr":"172.21.15.55:6825/2757675043","cluster_addr":"172.21.15.55:6827/2757675043","heartbeat_back_addr":"172.21.15.55:6831/2757675043","heartbeat_front_addr":"172.21.15.55:6829/2757675043","state":["exists","up"]},{"osd":1,"uuid":"8e58f791-001a-44d0-a464-ca68a479f70c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6817","nonce":543460007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6819","nonce":543460007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6823","nonce":543460007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6821","nonce":543460007}]},"public_addr":"172.21.15.55:6817/543460007","cluster_addr":"172.21.15.55:6819/543460007","heartbeat_back_addr":"172.21.15.55:6823/543460007","heartbeat_front_addr":"172.21.15.55:6821/543460007","state":["exists","up"]},{"osd":2,"uuid":"d44f38b8-67f4-43de-98a6-f3c520acaf10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6801","nonce":1760253550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6803","nonce":1760253550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6807","nonce":1760253550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6805","nonce":1760253550}]},"public_addr":"172.21.15.55:6801/1760253550","cluster_addr":"172.21.15.55:6803/1760253550","heartbeat_back_addr":"172.21.15.55:6807/1760253550","heartbeat_front_addr":"172.21.15.55:6805/1760253550","state":["exists","up"]},{"osd":3,"uuid":"850a7316-2d74-43f0-8649-b5f8d8f6692e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6809","nonce":2797616448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6811","nonce":2797616448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6815","nonce":2797616448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6813","nonce":2797616448}]},"public_addr":"172.21.15.55:6809/2797616448","cluster_addr":"172.21.15.55:6811/2797616448","heartbeat_back_addr":"172.21.15.55:6815/2797616448","heartbeat_front_addr":"172.21.15.55:6813/2797616448","state":["exists","up"]},{"osd":4,"uuid":"4e6152ae-a77d-4bb6-a855-5782ad5411a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6819","nonce":2938436826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6821","nonce":2938436826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6825","nonce":2938436826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6823","nonce":2938436826}]},"public_addr":"172.21.15.84:6819/2938436826","cluster_addr":"172.21.15.84:6821/2938436826","heartbeat_back_addr":"172.21.15.84:6825/2938436826","heartbeat_front_addr":"172.21.15.84:6823/2938436826","state":["exists","up"]},{"osd":5,"uuid":"1217ceba-dd31-4615-96f1-dfaf57160f3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6803","nonce":460717295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6805","nonce":460717295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6809","nonce":460717295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6807","nonce":460717295}]},"public_addr":"172.21.15.84:6803/460717295","cluster_addr":"172.21.15.84:6805/460717295","heartbeat_back_addr":"172.21.15.84:6809/460717295","heartbeat_front_addr":"172.21.15.84:6807/460717295","state":["exists","up"]},{"osd":6,"uuid":"3d9ed78a-2284-4c28-a24a-70eb812c7836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6827","nonce":416123135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6829","nonce":416123135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6833","nonce":416123135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6831","nonce":416123135}]},"public_addr":"172.21.15.84:6827/416123135","cluster_addr":"172.21.15.84:6829/416123135","heartbeat_back_addr":"172.21.15.84:6833/416123135","heartbeat_front_addr":"172.21.15.84:6831/416123135","state":["exists","up"]},{"osd":7,"uuid":"c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6811","nonce":460083528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6813","nonce":460083528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6817","nonce":460083528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6815","nonce":460083528}]},"public_addr":"172.21.15.84:6811/460083528","cluster_addr":"172.21.15.84:6813/460083528","heartbeat_back_addr":"172.21.15.84:6817/460083528","heartbeat_front_addr":"172.21.15.84:6815/460083528","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.269126+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.121001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.075714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.092461+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.194676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.062833+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.316515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.237111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:51:20.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.962+0000 7f97fb4a2640 1 -- 172.21.15.55:0/706575863 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f97d80676b0 msgr2=0x7f97d8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:20.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.962+0000 7f97fb4a2640 1 --2- 172.21.15.55:0/706575863 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f97d80676b0 0x7f97d8069b70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f97e40b2000 tx=0x7f97e40aa070 comp rx=0 tx=0).stop 2024-03-19T01:51:20.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.962+0000 7f97fb4a2640 1 -- 172.21.15.55:0/706575863 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97f4072c60 msgr2=0x7f97f415a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:20.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.962+0000 7f97fb4a2640 1 --2- 172.21.15.55:0/706575863 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97f4072c60 0x7f97f415a5c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f97f000c490 tx=0x7f97f0001e80 comp rx=0 tx=0).stop 2024-03-19T01:51:20.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.962+0000 7f97fb4a2640 1 -- 172.21.15.55:0/706575863 shutdown_connections 2024-03-19T01:51:20.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.962+0000 7f97fb4a2640 1 --2- 172.21.15.55:0/706575863 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f97d80676b0 0x7f97d8069b70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:20.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.962+0000 7f97fb4a2640 1 --2- 172.21.15.55:0/706575863 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f97f415b540 0x7f97f40c44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:20.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.962+0000 7f97fb4a2640 1 --2- 172.21.15.55:0/706575863 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f97f40c39c0 0x7f97f41385f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:20.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.962+0000 7f97fb4a2640 1 --2- 172.21.15.55:0/706575863 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97f4072c60 0x7f97f415a5c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:20.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.962+0000 7f97fb4a2640 1 -- 172.21.15.55:0/706575863 >> 172.21.15.55:0/706575863 conn(0x7f97f40cc9c0 msgr2=0x7f97f4139330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:20.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.962+0000 7f97fb4a2640 1 -- 172.21.15.55:0/706575863 shutdown_connections 2024-03-19T01:51:20.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:20.962+0000 7f97fb4a2640 1 -- 172.21.15.55:0/706575863 wait complete. 2024-03-19T01:51:20.997 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:51:21.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.118+0000 7f9b69ff6640 1 Processor -- start 2024-03-19T01:51:21.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.118+0000 7f9b69ff6640 1 -- start start 2024-03-19T01:51:21.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.122+0000 7f9b69ff6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9b640c61c0 0x7f9b640708a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:21.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.122+0000 7f9b69ff6640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f9b640c73c0 0x7f9b640c7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:21.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.122+0000 7f9b69ff6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b640c6600 con 0x7f9b641510f0 2024-03-19T01:51:21.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.122+0000 7f9b69ff6640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f9b640c3f60 con 0x7f9b640c73c0 2024-03-19T01:51:21.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.122+0000 7f9b69ff6640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f9b641514d0 con 0x7f9b640c61c0 2024-03-19T01:51:21.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.122+0000 7f9b637fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f9b641510f0 0x7f9b6415a410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:53184/0 (socket says 172.21.15.55:53184) 2024-03-19T01:51:21.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.122+0000 7f9b637fe640 1 -- 172.21.15.55:0/2551935629 learned_addr learned my addr 172.21.15.55:0/2551935629 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:51:21.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.126+0000 7f9b627fc640 1 -- 172.21.15.55:0/2551935629 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 393603307 0 0) 0x7f9b640c6600 con 0x7f9b641510f0 2024-03-19T01:51:21.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.126+0000 7f9b627fc640 1 -- 172.21.15.55:0/2551935629 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b4c003530 con 0x7f9b641510f0 2024-03-19T01:51:21.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.130+0000 7f9b627fc640 1 -- 172.21.15.55:0/2551935629 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3434304666 0 0) 0x7f9b4c003530 con 0x7f9b641510f0 2024-03-19T01:51:21.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.130+0000 7f9b627fc640 1 -- 172.21.15.55:0/2551935629 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9b640c61c0 msgr2=0x7f9b640708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:21.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.130+0000 7f9b627fc640 1 --2- 172.21.15.55:0/2551935629 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9b640c61c0 0x7f9b640708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:21.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.130+0000 7f9b627fc640 1 -- 172.21.15.55:0/2551935629 >> v2:172.21.15.84:3300/0 conn(0x7f9b640c73c0 msgr2=0x7f9b640c7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:21.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.130+0000 7f9b627fc640 1 --2- 172.21.15.55:0/2551935629 >> v2:172.21.15.84:3300/0 conn(0x7f9b640c73c0 0x7f9b640c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:21.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.130+0000 7f9b627fc640 1 -- 172.21.15.55:0/2551935629 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b640c6870 con 0x7f9b641510f0 2024-03-19T01:51:21.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.130+0000 7f9b627fc640 1 -- 172.21.15.55:0/2551935629 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f9b5c001680 con 0x7f9b641510f0 2024-03-19T01:51:21.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.130+0000 7f9b627fc640 1 -- 172.21.15.55:0/2551935629 >> v1:172.21.15.55:6789/0 conn(0x7f9b641510f0 legacy=0x7f9b6415a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:21.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.130+0000 7f9b627fc640 1 --2- 172.21.15.55:0/2551935629 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b4c0039c0 0x7f9b4c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:21.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.130+0000 7f9b627fc640 1 --2- 172.21.15.55:0/2551935629 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9b640c61c0 0x7f9b4c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:21.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.130+0000 7f9b627fc640 1 --2- 172.21.15.55:0/2551935629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c004e70 0x7f9b4c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:21.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.130+0000 7f9b627fc640 1 -- 172.21.15.55:0/2551935629 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9b640c3f60 con 0x7f9b4c004e70 2024-03-19T01:51:21.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.130+0000 7f9b627fc640 1 -- 172.21.15.55:0/2551935629 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f9b641514d0 con 0x7f9b4c0039c0 2024-03-19T01:51:21.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.130+0000 7f9b627fc640 1 -- 172.21.15.55:0/2551935629 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f9b4c005630 con 0x7f9b640c61c0 2024-03-19T01:51:21.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.134+0000 7f9b63fff640 1 --2- 172.21.15.55:0/2551935629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c004e70 0x7f9b4c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:21.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.138+0000 7f9b63fff640 1 -- 172.21.15.55:0/2551935629 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9b640c61c0 msgr2=0x7f9b4c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:21.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.138+0000 7f9b63fff640 1 --2- 172.21.15.55:0/2551935629 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9b640c61c0 0x7f9b4c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:21.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.138+0000 7f9b63fff640 1 -- 172.21.15.55:0/2551935629 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b4c0039c0 msgr2=0x7f9b4c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:21.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.138+0000 7f9b63fff640 1 --2- 172.21.15.55:0/2551935629 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b4c0039c0 0x7f9b4c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:21.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.138+0000 7f9b63fff640 1 -- 172.21.15.55:0/2551935629 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4c003530 con 0x7f9b4c004e70 2024-03-19T01:51:21.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.142+0000 7f9b63fff640 1 --2- 172.21.15.55:0/2551935629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c004e70 0x7f9b4c00b9d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9b54009ca0 tx=0x7f9b5401cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555e0155c7dcc306 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:21.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.146+0000 7f9b627fc640 1 -- 172.21.15.55:0/2551935629 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b540b2070 con 0x7f9b4c004e70 2024-03-19T01:51:21.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.146+0000 7f9b627fc640 1 -- 172.21.15.55:0/2551935629 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b540a5ca0 con 0x7f9b4c004e70 2024-03-19T01:51:21.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.146+0000 7f9b627fc640 1 -- 172.21.15.55:0/2551935629 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b54006e60 con 0x7f9b4c004e70 2024-03-19T01:51:21.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.146+0000 7f9b69ff6640 1 -- 172.21.15.55:0/2551935629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c004e70 msgr2=0x7f9b4c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:21.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.146+0000 7f9b69ff6640 1 --2- 172.21.15.55:0/2551935629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c004e70 0x7f9b4c00b9d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9b54009ca0 tx=0x7f9b5401cc70 comp rx=0 tx=0).stop 2024-03-19T01:51:21.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.146+0000 7f9b63fff640 1 -- 172.21.15.55:0/2551935629 reap_dead start 2024-03-19T01:51:21.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.146+0000 7f9b69ff6640 1 -- 172.21.15.55:0/2551935629 shutdown_connections 2024-03-19T01:51:21.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.146+0000 7f9b69ff6640 1 -- 172.21.15.55:0/2551935629 >> 172.21.15.55:0/2551935629 conn(0x7f9b640cc9c0 msgr2=0x7f9b640ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:21.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.146+0000 7f9b69ff6640 1 -- 172.21.15.55:0/2551935629 shutdown_connections 2024-03-19T01:51:21.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.146+0000 7f9b69ff6640 1 -- 172.21.15.55:0/2551935629 wait complete. 2024-03-19T01:51:21.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.146+0000 7f9b69ff6640 1 Processor -- start 2024-03-19T01:51:21.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.146+0000 7f9b69ff6640 1 -- start start 2024-03-19T01:51:21.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.150+0000 7f9b69ff6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c004e70 0x7f9b6414b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:21.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.150+0000 7f9b69ff6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9b6414ad10 0x7f9b6413e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:21.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.150+0000 7f9b69ff6640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b6413c1f0 0x7f9b6413c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:21.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.150+0000 7f9b69ff6640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9b640c44e0 con 0x7f9b4c004e70 2024-03-19T01:51:21.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.150+0000 7f9b69ff6640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f9b640c4200 con 0x7f9b6413c1f0 2024-03-19T01:51:21.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.150+0000 7f9b69ff6640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f9b640c6e30 con 0x7f9b6414ad10 2024-03-19T01:51:21.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.150+0000 7f9b637fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c004e70 0x7f9b6414b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:21.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.150+0000 7f9b637fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c004e70 0x7f9b6414b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40970/0 (socket says 172.21.15.55:40970) 2024-03-19T01:51:21.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.150+0000 7f9b637fe640 1 -- 172.21.15.55:0/2526061442 learned_addr learned my addr 172.21.15.55:0/2526061442 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:21.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.154+0000 7f9b637fe640 1 -- 172.21.15.55:0/2526061442 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9b6414ad10 msgr2=0x7f9b6413e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:21.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.154+0000 7f9b637fe640 1 --2- 172.21.15.55:0/2526061442 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9b6414ad10 0x7f9b6413e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:21.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.154+0000 7f9b637fe640 1 -- 172.21.15.55:0/2526061442 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b6413c1f0 msgr2=0x7f9b6413c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:21.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.154+0000 7f9b637fe640 1 --2- 172.21.15.55:0/2526061442 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b6413c1f0 0x7f9b6413c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:21.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.154+0000 7f9b637fe640 1 -- 172.21.15.55:0/2526061442 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b6414bfb0 con 0x7f9b4c004e70 2024-03-19T01:51:21.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.154+0000 7f9b637fe640 1 --2- 172.21.15.55:0/2526061442 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c004e70 0x7f9b6414b760 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c00c7a0 tx=0x7f9b5c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:21.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.158+0000 7f9b4bfff640 1 -- 172.21.15.55:0/2526061442 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c018070 con 0x7f9b4c004e70 2024-03-19T01:51:21.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.158+0000 7f9b4bfff640 1 -- 172.21.15.55:0/2526061442 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b5c00ce50 con 0x7f9b4c004e70 2024-03-19T01:51:21.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.158+0000 7f9b4bfff640 1 -- 172.21.15.55:0/2526061442 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c0047f0 con 0x7f9b4c004e70 2024-03-19T01:51:21.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.158+0000 7f9b69ff6640 1 -- 172.21.15.55:0/2526061442 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b6414c960 con 0x7f9b4c004e70 2024-03-19T01:51:21.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.158+0000 7f9b69ff6640 1 -- 172.21.15.55:0/2526061442 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b6414cf20 con 0x7f9b4c004e70 2024-03-19T01:51:21.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.158+0000 7f9b69ff6640 1 -- 172.21.15.55:0/2526061442 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b64068a40 con 0x7f9b4c004e70 2024-03-19T01:51:21.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.454+0000 7f9b4bfff640 1 -- 172.21.15.55:0/2526061442 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f9b5c01b070 con 0x7f9b4c004e70 2024-03-19T01:51:21.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.454+0000 7f9b4bfff640 1 --2- 172.21.15.55:0/2526061442 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9b500676e0 0x7f9b50069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:21.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.454+0000 7f9b62ffd640 1 --2- 172.21.15.55:0/2526061442 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9b500676e0 0x7f9b50069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:21.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.454+0000 7f9b62ffd640 1 --2- 172.21.15.55:0/2526061442 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9b500676e0 0x7f9b50069ba0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9b641f0d30 tx=0x7f9b58006cd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:21.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.490+0000 7f9b4bfff640 1 -- 172.21.15.55:0/2526061442 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f9b5c0539e0 con 0x7f9b4c004e70 2024-03-19T01:51:21.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.490+0000 7f9b4bfff640 1 -- 172.21.15.55:0/2526061442 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f9b5c022020 con 0x7f9b4c004e70 2024-03-19T01:51:21.703 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.702+0000 7f9b69ff6640 1 -- 172.21.15.55:0/2526061442 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b64148660 con 0x7f9b4c004e70 2024-03-19T01:51:21.742 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.742+0000 7f9b4bfff640 1 -- 172.21.15.55:0/2526061442 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6822 (secure 0 0 0) 0x7f9b5c056040 con 0x7f9b4c004e70 2024-03-19T01:51:21.742 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:51:21.742 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4200,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4237,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4249,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6839/354880008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6838","nonce":354880008},{"type":"v1","addr":"172.21.15.84:6839","nonce":354880008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:51:17.241672+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4255,"qdb_cluster":[4255]},"id":1}]} 2024-03-19T01:51:21.743 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 8 2024-03-19T01:51:21.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.742+0000 7f9b69ff6640 1 -- 172.21.15.55:0/2526061442 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9b500676e0 msgr2=0x7f9b50069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:21.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.742+0000 7f9b69ff6640 1 --2- 172.21.15.55:0/2526061442 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9b500676e0 0x7f9b50069ba0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9b641f0d30 tx=0x7f9b58006cd0 comp rx=0 tx=0).stop 2024-03-19T01:51:21.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.742+0000 7f9b69ff6640 1 -- 172.21.15.55:0/2526061442 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c004e70 msgr2=0x7f9b6414b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:21.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.742+0000 7f9b69ff6640 1 --2- 172.21.15.55:0/2526061442 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c004e70 0x7f9b6414b760 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c00c7a0 tx=0x7f9b5c00cc60 comp rx=0 tx=0).stop 2024-03-19T01:51:21.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.742+0000 7f9b69ff6640 1 -- 172.21.15.55:0/2526061442 shutdown_connections 2024-03-19T01:51:21.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.742+0000 7f9b69ff6640 1 --2- 172.21.15.55:0/2526061442 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9b500676e0 0x7f9b50069ba0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:21.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.742+0000 7f9b69ff6640 1 --2- 172.21.15.55:0/2526061442 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b6413c1f0 0x7f9b6413c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:21.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.742+0000 7f9b69ff6640 1 --2- 172.21.15.55:0/2526061442 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9b6414ad10 0x7f9b6413e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:21.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.742+0000 7f9b69ff6640 1 --2- 172.21.15.55:0/2526061442 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c004e70 0x7f9b6414b760 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:21.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.742+0000 7f9b69ff6640 1 -- 172.21.15.55:0/2526061442 >> 172.21.15.55:0/2526061442 conn(0x7f9b640cc9c0 msgr2=0x7f9b640c8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:21.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.742+0000 7f9b69ff6640 1 -- 172.21.15.55:0/2526061442 shutdown_connections 2024-03-19T01:51:21.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.742+0000 7f9b69ff6640 1 -- 172.21.15.55:0/2526061442 wait complete. 2024-03-19T01:51:21.762 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:51:21.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.874+0000 7f851577e640 1 Processor -- start 2024-03-19T01:51:21.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.874+0000 7f851577e640 1 -- start start 2024-03-19T01:51:21.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.874+0000 7f851577e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f851013de70 0x7f851006e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:21.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.874+0000 7f851577e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f851006f1b0 0x7f8510158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:21.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.874+0000 7f851577e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f851015b740 con 0x7f851006eb70 2024-03-19T01:51:21.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.874+0000 7f851577e640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f851013e250 con 0x7f851006f1b0 2024-03-19T01:51:21.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.874+0000 7f851577e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f851015ba20 con 0x7f851013de70 2024-03-19T01:51:21.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.874+0000 7f850f7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f851006f1b0 0x7f8510158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:21.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.874+0000 7f850f7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f851006f1b0 0x7f8510158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40974/0 (socket says 172.21.15.55:40974) 2024-03-19T01:51:21.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.874+0000 7f850f7fe640 1 -- 172.21.15.55:0/1959551523 learned_addr learned my addr 172.21.15.55:0/1959551523 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:21.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.878+0000 7f850f7fe640 1 -- 172.21.15.55:0/1959551523 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f851013de70 msgr2=0x7f851006e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:21.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.878+0000 7f850f7fe640 1 --2- 172.21.15.55:0/1959551523 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f851013de70 0x7f851006e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:21.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.878+0000 7f850f7fe640 1 -- 172.21.15.55:0/1959551523 >> v1:172.21.15.84:6789/0 conn(0x7f851006eb70 legacy=0x7f85100c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:21.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.878+0000 7f850f7fe640 1 -- 172.21.15.55:0/1959551523 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f851015bd00 con 0x7f851006f1b0 2024-03-19T01:51:21.882 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.882+0000 7f850f7fe640 1 --2- 172.21.15.55:0/1959551523 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f851006f1b0 0x7f8510158420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8508009e50 tx=0x7f850801f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b378fd04ca7a157 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:21.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.890+0000 7f850dffb640 1 -- 172.21.15.55:0/1959551523 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85080b4070 con 0x7f851006f1b0 2024-03-19T01:51:21.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.890+0000 7f850dffb640 1 -- 172.21.15.55:0/1959551523 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85080af400 con 0x7f851006f1b0 2024-03-19T01:51:21.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.890+0000 7f850dffb640 1 -- 172.21.15.55:0/1959551523 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85080af7d0 con 0x7f851006f1b0 2024-03-19T01:51:21.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.890+0000 7f851577e640 1 -- 172.21.15.55:0/1959551523 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f851006f1b0 msgr2=0x7f8510158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:21.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.890+0000 7f851577e640 1 --2- 172.21.15.55:0/1959551523 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f851006f1b0 0x7f8510158420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8508009e50 tx=0x7f850801f1e0 comp rx=0 tx=0).stop 2024-03-19T01:51:21.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.890+0000 7f851577e640 1 -- 172.21.15.55:0/1959551523 shutdown_connections 2024-03-19T01:51:21.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.890+0000 7f851577e640 1 --2- 172.21.15.55:0/1959551523 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f851006f1b0 0x7f8510158420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:21.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.890+0000 7f851577e640 1 --2- 172.21.15.55:0/1959551523 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f851013de70 0x7f851006e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:21.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.890+0000 7f851577e640 1 -- 172.21.15.55:0/1959551523 >> 172.21.15.55:0/1959551523 conn(0x7f851006fb40 msgr2=0x7f851006ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:21.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.890+0000 7f851577e640 1 -- 172.21.15.55:0/1959551523 shutdown_connections 2024-03-19T01:51:21.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.890+0000 7f851577e640 1 -- 172.21.15.55:0/1959551523 wait complete. 2024-03-19T01:51:21.891 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.890+0000 7f851577e640 1 Processor -- start 2024-03-19T01:51:21.891 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.890+0000 7f851577e640 1 -- start start 2024-03-19T01:51:21.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.890+0000 7f851577e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f851006eb70 0x7f85100ca190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:21.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.890+0000 7f851577e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f851006f1b0 0x7f85101e3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:21.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.890+0000 7f851577e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f851013de70 0x7f85100cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:21.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.890+0000 7f851577e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f85100c5f20 con 0x7f851006f1b0 2024-03-19T01:51:21.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.890+0000 7f851577e640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f851015c2c0 con 0x7f851013de70 2024-03-19T01:51:21.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.890+0000 7f851577e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f85100c60a0 con 0x7f851006eb70 2024-03-19T01:51:21.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.894+0000 7f850e7fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f851006f1b0 0x7f85101e3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:21.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.894+0000 7f850e7fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f851006f1b0 0x7f85101e3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40984/0 (socket says 172.21.15.55:40984) 2024-03-19T01:51:21.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.894+0000 7f850e7fc640 1 -- 172.21.15.55:0/3350296182 learned_addr learned my addr 172.21.15.55:0/3350296182 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:21.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.894+0000 7f850f7fe640 1 --2- 172.21.15.55:0/3350296182 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f851013de70 0x7f85100cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:21.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.894+0000 7f850effd640 1 --2- 172.21.15.55:0/3350296182 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f851006eb70 0x7f85100ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:21.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.898+0000 7f850e7fc640 1 -- 172.21.15.55:0/3350296182 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f851006eb70 msgr2=0x7f85100ca190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:21.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.898+0000 7f850e7fc640 1 --2- 172.21.15.55:0/3350296182 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f851006eb70 0x7f85100ca190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:21.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.898+0000 7f850e7fc640 1 -- 172.21.15.55:0/3350296182 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f851013de70 msgr2=0x7f85100cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:21.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.898+0000 7f850e7fc640 1 --2- 172.21.15.55:0/3350296182 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f851013de70 0x7f85100cb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:21.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.898+0000 7f850e7fc640 1 -- 172.21.15.55:0/3350296182 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f851015aee0 con 0x7f851006f1b0 2024-03-19T01:51:21.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.898+0000 7f850e7fc640 1 --2- 172.21.15.55:0/3350296182 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f851006f1b0 0x7f85101e3d70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f850000c450 tx=0x7f850000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:21.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.902+0000 7f84f77fe640 1 -- 172.21.15.55:0/3350296182 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8500012830 con 0x7f851006f1b0 2024-03-19T01:51:21.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.902+0000 7f84f77fe640 1 -- 172.21.15.55:0/3350296182 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8500017070 con 0x7f851006f1b0 2024-03-19T01:51:21.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.902+0000 7f851577e640 1 -- 172.21.15.55:0/3350296182 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f851015b1c0 con 0x7f851006f1b0 2024-03-19T01:51:21.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.902+0000 7f84f77fe640 1 -- 172.21.15.55:0/3350296182 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8500012c30 con 0x7f851006f1b0 2024-03-19T01:51:21.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.902+0000 7f851577e640 1 -- 172.21.15.55:0/3350296182 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85101f19c0 con 0x7f851006f1b0 2024-03-19T01:51:21.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:21.906+0000 7f851577e640 1 -- 172.21.15.55:0/3350296182 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8510068a40 con 0x7f851006f1b0 2024-03-19T01:51:22.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:22.214+0000 7f84f77fe640 1 -- 172.21.15.55:0/3350296182 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f850001c460 con 0x7f851006f1b0 2024-03-19T01:51:22.217 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:22.218+0000 7f84f77fe640 1 --2- 172.21.15.55:0/3350296182 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f84fc0676b0 0x7f84fc069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:22.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:22.218+0000 7f850effd640 1 --2- 172.21.15.55:0/3350296182 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f84fc0676b0 0x7f84fc069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:22.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:22.218+0000 7f850effd640 1 --2- 172.21.15.55:0/3350296182 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f84fc0676b0 0x7f84fc069b70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f85040098f0 tx=0x7f8504006f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:22.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:22.254+0000 7f84f77fe640 1 -- 172.21.15.55:0/3350296182 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f8500054710 con 0x7f851006f1b0 2024-03-19T01:51:22.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:22.254+0000 7f84f77fe640 1 -- 172.21.15.55:0/3350296182 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f8500056670 con 0x7f851006f1b0 2024-03-19T01:51:22.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:22.442+0000 7f851577e640 1 -- 172.21.15.55:0/3350296182 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f85101526e0 con 0x7f851006f1b0 2024-03-19T01:51:22.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:22.514+0000 7f84f77fe640 1 -- 172.21.15.55:0/3350296182 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+18430 (secure 0 0 0) 0x7f850005a710 con 0x7f851006f1b0 2024-03-19T01:51:22.515 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:51:22.515 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":29,"fsid":"def9d4e5-0bc7-449d-b893-2490cbd10e25","created":"2024-03-19T01:50:02.482954+0000","modified":"2024-03-19T01:51:17.228291+0000","last_up_change":"2024-03-19T01:50:21.735508+0000","last_in_change":"2024-03-19T01:50:12.168944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:50:23.456904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:51:04.578583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:51:06.871659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T01:51:14.274442+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"35d5e3c5-d089-4f38-8961-98d064188f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6825","nonce":2757675043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6827","nonce":2757675043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6831","nonce":2757675043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6829","nonce":2757675043}]},"public_addr":"172.21.15.55:6825/2757675043","cluster_addr":"172.21.15.55:6827/2757675043","heartbeat_back_addr":"172.21.15.55:6831/2757675043","heartbeat_front_addr":"172.21.15.55:6829/2757675043","state":["exists","up"]},{"osd":1,"uuid":"8e58f791-001a-44d0-a464-ca68a479f70c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6817","nonce":543460007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6819","nonce":543460007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6823","nonce":543460007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6821","nonce":543460007}]},"public_addr":"172.21.15.55:6817/543460007","cluster_addr":"172.21.15.55:6819/543460007","heartbeat_back_addr":"172.21.15.55:6823/543460007","heartbeat_front_addr":"172.21.15.55:6821/543460007","state":["exists","up"]},{"osd":2,"uuid":"d44f38b8-67f4-43de-98a6-f3c520acaf10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6801","nonce":1760253550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6803","nonce":1760253550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6807","nonce":1760253550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6805","nonce":1760253550}]},"public_addr":"172.21.15.55:6801/1760253550","cluster_addr":"172.21.15.55:6803/1760253550","heartbeat_back_addr":"172.21.15.55:6807/1760253550","heartbeat_front_addr":"172.21.15.55:6805/1760253550","state":["exists","up"]},{"osd":3,"uuid":"850a7316-2d74-43f0-8649-b5f8d8f6692e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6809","nonce":2797616448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6811","nonce":2797616448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6815","nonce":2797616448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6813","nonce":2797616448}]},"public_addr":"172.21.15.55:6809/2797616448","cluster_addr":"172.21.15.55:6811/2797616448","heartbeat_back_addr":"172.21.15.55:6815/2797616448","heartbeat_front_addr":"172.21.15.55:6813/2797616448","state":["exists","up"]},{"osd":4,"uuid":"4e6152ae-a77d-4bb6-a855-5782ad5411a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6819","nonce":2938436826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6821","nonce":2938436826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6825","nonce":2938436826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6823","nonce":2938436826}]},"public_addr":"172.21.15.84:6819/2938436826","cluster_addr":"172.21.15.84:6821/2938436826","heartbeat_back_addr":"172.21.15.84:6825/2938436826","heartbeat_front_addr":"172.21.15.84:6823/2938436826","state":["exists","up"]},{"osd":5,"uuid":"1217ceba-dd31-4615-96f1-dfaf57160f3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6803","nonce":460717295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6805","nonce":460717295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6809","nonce":460717295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6807","nonce":460717295}]},"public_addr":"172.21.15.84:6803/460717295","cluster_addr":"172.21.15.84:6805/460717295","heartbeat_back_addr":"172.21.15.84:6809/460717295","heartbeat_front_addr":"172.21.15.84:6807/460717295","state":["exists","up"]},{"osd":6,"uuid":"3d9ed78a-2284-4c28-a24a-70eb812c7836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6827","nonce":416123135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6829","nonce":416123135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6833","nonce":416123135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6831","nonce":416123135}]},"public_addr":"172.21.15.84:6827/416123135","cluster_addr":"172.21.15.84:6829/416123135","heartbeat_back_addr":"172.21.15.84:6833/416123135","heartbeat_front_addr":"172.21.15.84:6831/416123135","state":["exists","up"]},{"osd":7,"uuid":"c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6811","nonce":460083528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6813","nonce":460083528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6817","nonce":460083528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6815","nonce":460083528}]},"public_addr":"172.21.15.84:6811/460083528","cluster_addr":"172.21.15.84:6813/460083528","heartbeat_back_addr":"172.21.15.84:6817/460083528","heartbeat_front_addr":"172.21.15.84:6815/460083528","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.269126+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.121001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.075714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.092461+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.194676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.062833+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.316515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.237111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:51:22.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:22.518+0000 7f851577e640 1 -- 172.21.15.55:0/3350296182 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f84fc0676b0 msgr2=0x7f84fc069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:22.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:22.518+0000 7f851577e640 1 --2- 172.21.15.55:0/3350296182 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f84fc0676b0 0x7f84fc069b70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f85040098f0 tx=0x7f8504006f90 comp rx=0 tx=0).stop 2024-03-19T01:51:22.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:22.518+0000 7f851577e640 1 -- 172.21.15.55:0/3350296182 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f851006f1b0 msgr2=0x7f85101e3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:22.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:22.518+0000 7f851577e640 1 --2- 172.21.15.55:0/3350296182 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f851006f1b0 0x7f85101e3d70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f850000c450 tx=0x7f850000c910 comp rx=0 tx=0).stop 2024-03-19T01:51:22.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:22.518+0000 7f851577e640 1 -- 172.21.15.55:0/3350296182 shutdown_connections 2024-03-19T01:51:22.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:22.518+0000 7f851577e640 1 --2- 172.21.15.55:0/3350296182 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f84fc0676b0 0x7f84fc069b70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:22.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:22.518+0000 7f851577e640 1 --2- 172.21.15.55:0/3350296182 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f851013de70 0x7f85100cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:22.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:22.518+0000 7f851577e640 1 --2- 172.21.15.55:0/3350296182 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f851006f1b0 0x7f85101e3d70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:22.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:22.518+0000 7f851577e640 1 --2- 172.21.15.55:0/3350296182 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f851006eb70 0x7f85100ca190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:22.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:22.518+0000 7f851577e640 1 -- 172.21.15.55:0/3350296182 >> 172.21.15.55:0/3350296182 conn(0x7f851006fb40 msgr2=0x7f851013ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:22.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:22.518+0000 7f851577e640 1 -- 172.21.15.55:0/3350296182 shutdown_connections 2024-03-19T01:51:22.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:22.518+0000 7f851577e640 1 -- 172.21.15.55:0/3350296182 wait complete. 2024-03-19T01:51:23.144 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:51:17.241672+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4255}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4255, 'qdb_cluster': [4255]} 2024-03-19T01:51:23.144 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-19T01:51:23.144 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-19T01:51:23.278 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.278+0000 7f6e92c5a640 1 Processor -- start 2024-03-19T01:51:23.278 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.278+0000 7f6e92c5a640 1 -- start start 2024-03-19T01:51:23.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.278+0000 7f6e92c5a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f6e8c154340 0x7f6e8c15e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:23.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.278+0000 7f6e92c5a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6e8c0c73c0 0x7f6e8c15aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:23.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.278+0000 7f6e92c5a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e8c071280 con 0x7f6e8c154720 2024-03-19T01:51:23.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.278+0000 7f6e92c5a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f6e8c073840 con 0x7f6e8c154340 2024-03-19T01:51:23.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.278+0000 7f6e92c5a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f6e8c0c7850 con 0x7f6e8c0c73c0 2024-03-19T01:51:23.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.278+0000 7f6e909f6640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f6e8c154340 0x7f6e8c15e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:23.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.278+0000 7f6e909f6640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f6e8c154340 0x7f6e8c15e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40988/0 (socket says 172.21.15.55:40988) 2024-03-19T01:51:23.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.278+0000 7f6e909f6640 1 -- 172.21.15.55:0/3141555795 learned_addr learned my addr 172.21.15.55:0/3141555795 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:23.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.282+0000 7f6e911f7640 1 --2- 172.21.15.55:0/3141555795 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6e8c0c73c0 0x7f6e8c15aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:23.285 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.286+0000 7f6e909f6640 1 -- 172.21.15.55:0/3141555795 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6e8c0c73c0 msgr2=0x7f6e8c15aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:23.285 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.286+0000 7f6e909f6640 1 --2- 172.21.15.55:0/3141555795 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6e8c0c73c0 0x7f6e8c15aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.285 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.286+0000 7f6e909f6640 1 -- 172.21.15.55:0/3141555795 >> v1:172.21.15.55:6789/0 conn(0x7f6e8c154720 legacy=0x7f6e8c072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:23.285 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.286+0000 7f6e909f6640 1 -- 172.21.15.55:0/3141555795 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e8c15d900 con 0x7f6e8c154340 2024-03-19T01:51:23.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.290+0000 7f6e909f6640 1 --2- 172.21.15.55:0/3141555795 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e8c154340 0x7f6e8c15e3a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6e84009e80 tx=0x7f6e8401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd3f4fadda1685bb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:23.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.298+0000 7f6e8b7fe640 1 -- 172.21.15.55:0/3141555795 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e840a56e0 con 0x7f6e8c154340 2024-03-19T01:51:23.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.298+0000 7f6e8b7fe640 1 -- 172.21.15.55:0/3141555795 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e840a5880 con 0x7f6e8c154340 2024-03-19T01:51:23.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.298+0000 7f6e8b7fe640 1 -- 172.21.15.55:0/3141555795 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e840a5c50 con 0x7f6e8c154340 2024-03-19T01:51:23.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.298+0000 7f6e92c5a640 1 -- 172.21.15.55:0/3141555795 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e8c154340 msgr2=0x7f6e8c15e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:23.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.298+0000 7f6e92c5a640 1 --2- 172.21.15.55:0/3141555795 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e8c154340 0x7f6e8c15e3a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6e84009e80 tx=0x7f6e8401ca10 comp rx=0 tx=0).stop 2024-03-19T01:51:23.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.298+0000 7f6e92c5a640 1 -- 172.21.15.55:0/3141555795 shutdown_connections 2024-03-19T01:51:23.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.298+0000 7f6e92c5a640 1 --2- 172.21.15.55:0/3141555795 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6e8c0c73c0 0x7f6e8c15aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.298+0000 7f6e92c5a640 1 --2- 172.21.15.55:0/3141555795 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e8c154340 0x7f6e8c15e3a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.298+0000 7f6e92c5a640 1 -- 172.21.15.55:0/3141555795 >> 172.21.15.55:0/3141555795 conn(0x7f6e8c0cc9c0 msgr2=0x7f6e8c0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:23.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.298+0000 7f6e92c5a640 1 -- 172.21.15.55:0/3141555795 shutdown_connections 2024-03-19T01:51:23.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.298+0000 7f6e92c5a640 1 -- 172.21.15.55:0/3141555795 wait complete. 2024-03-19T01:51:23.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.298+0000 7f6e92c5a640 1 Processor -- start 2024-03-19T01:51:23.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.298+0000 7f6e92c5a640 1 -- start start 2024-03-19T01:51:23.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.298+0000 7f6e92c5a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e8c0c73c0 0x7f6e8c14e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:23.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.298+0000 7f6e92c5a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e8c154340 0x7f6e8c14a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:23.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.298+0000 7f6e92c5a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6e8c154720 0x7f6e8c1f6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:23.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.298+0000 7f6e92c5a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8c166b70 con 0x7f6e8c154340 2024-03-19T01:51:23.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.298+0000 7f6e92c5a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8c15b1e0 con 0x7f6e8c0c73c0 2024-03-19T01:51:23.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.298+0000 7f6e92c5a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f6e8c166cf0 con 0x7f6e8c154720 2024-03-19T01:51:23.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.298+0000 7f6e8bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e8c154340 0x7f6e8c14a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:23.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.302+0000 7f6e8bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e8c154340 0x7f6e8c14a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40992/0 (socket says 172.21.15.55:40992) 2024-03-19T01:51:23.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.302+0000 7f6e8bfff640 1 -- 172.21.15.55:0/2498464327 learned_addr learned my addr 172.21.15.55:0/2498464327 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:23.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.302+0000 7f6e909f6640 1 --2- 172.21.15.55:0/2498464327 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e8c0c73c0 0x7f6e8c14e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:23.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.302+0000 7f6e911f7640 1 --2- 172.21.15.55:0/2498464327 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6e8c154720 0x7f6e8c1f6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:23.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.302+0000 7f6e8bfff640 1 -- 172.21.15.55:0/2498464327 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6e8c154720 msgr2=0x7f6e8c1f6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:23.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.302+0000 7f6e8bfff640 1 --2- 172.21.15.55:0/2498464327 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6e8c154720 0x7f6e8c1f6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.302+0000 7f6e8bfff640 1 -- 172.21.15.55:0/2498464327 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e8c0c73c0 msgr2=0x7f6e8c14e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:23.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.302+0000 7f6e8bfff640 1 --2- 172.21.15.55:0/2498464327 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e8c0c73c0 0x7f6e8c14e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.302+0000 7f6e8bfff640 1 -- 172.21.15.55:0/2498464327 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e8c152670 con 0x7f6e8c154340 2024-03-19T01:51:23.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.306+0000 7f6e8bfff640 1 --2- 172.21.15.55:0/2498464327 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e8c154340 0x7f6e8c14a280 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6e8000c450 tx=0x7f6e8000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:23.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.310+0000 7f6e897fa640 1 -- 172.21.15.55:0/2498464327 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e80016020 con 0x7f6e8c154340 2024-03-19T01:51:23.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.310+0000 7f6e897fa640 1 -- 172.21.15.55:0/2498464327 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e800052d0 con 0x7f6e8c154340 2024-03-19T01:51:23.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.310+0000 7f6e897fa640 1 -- 172.21.15.55:0/2498464327 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e80007260 con 0x7f6e8c154340 2024-03-19T01:51:23.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.310+0000 7f6e92c5a640 1 -- 172.21.15.55:0/2498464327 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e8c14fda0 con 0x7f6e8c154340 2024-03-19T01:51:23.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.310+0000 7f6e92c5a640 1 -- 172.21.15.55:0/2498464327 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e8c151930 con 0x7f6e8c154340 2024-03-19T01:51:23.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.314+0000 7f6e95ddc000 1 Processor -- start 2024-03-19T01:51:23.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.314+0000 7f6e95ddc000 1 -- start start 2024-03-19T01:51:23.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.314+0000 7f6e95ddc000 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x5583688eb710 0x5583688ebaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:23.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.314+0000 7f6e95ddc000 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x558368926dd0 0x5583689271b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:23.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.314+0000 7f6e95ddc000 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583689276f0 0x558368932060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:23.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.314+0000 7f6e95ddc000 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5583687cd280 con 0x5583689276f0 2024-03-19T01:51:23.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.314+0000 7f6e95ddc000 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x558368735690 con 0x558368926dd0 2024-03-19T01:51:23.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.314+0000 7f6e95ddc000 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x5583686bcc50 con 0x5583688eb710 2024-03-19T01:51:23.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.318+0000 7f6e909f6640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x558368926dd0 0x5583689271b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:23.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.318+0000 7f6e911f7640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x5583688eb710 0x5583688ebaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:23.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.318+0000 7f6e911f7640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x5583688eb710 0x5583688ebaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:37132/0 (socket says 172.21.15.55:37132) 2024-03-19T01:51:23.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.318+0000 7f6e911f7640 1 -- 172.21.15.55:0/324714791 learned_addr learned my addr 172.21.15.55:0/324714791 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:23.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.322+0000 7f6e909f6640 1 -- 172.21.15.55:0/324714791 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x5583688eb710 msgr2=0x5583688ebaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:23.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.322+0000 7f6e909f6640 1 --2- 172.21.15.55:0/324714791 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x5583688eb710 0x5583688ebaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.322+0000 7f6e909f6640 1 -- 172.21.15.55:0/324714791 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583689276f0 msgr2=0x558368932060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:23.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.322+0000 7f6e909f6640 1 --2- 172.21.15.55:0/324714791 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583689276f0 0x558368932060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.322+0000 7f6e909f6640 1 -- 172.21.15.55:0/324714791 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583686bcea0 con 0x558368926dd0 2024-03-19T01:51:23.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.326+0000 7f6e909f6640 1 --2- 172.21.15.55:0/324714791 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x558368926dd0 0x5583689271b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6e840ab000 tx=0x7f6e84004ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26679a479de5d9dc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:23.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.334+0000 7f6e897fa640 1 -- 172.21.15.55:0/2498464327 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f6e80007400 con 0x7f6e8c154340 2024-03-19T01:51:23.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.334+0000 7f6e897fa640 1 --2- 172.21.15.55:0/2498464327 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6e780676e0 0x7f6e78069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:23.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.334+0000 7f6e911f7640 1 --2- 172.21.15.55:0/2498464327 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6e780676e0 0x7f6e78069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:23.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.334+0000 7f6e911f7640 1 --2- 172.21.15.55:0/2498464327 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6e780676e0 0x7f6e78069ba0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c006c10 tx=0x7f6e7c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:23.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.334+0000 7f6e727fc640 1 -- 172.21.15.55:0/324714791 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e840a56e0 con 0x558368926dd0 2024-03-19T01:51:23.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.334+0000 7f6e727fc640 1 -- 172.21.15.55:0/324714791 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e840a5880 con 0x558368926dd0 2024-03-19T01:51:23.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.334+0000 7f6e727fc640 1 -- 172.21.15.55:0/324714791 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e840a5c70 con 0x558368926dd0 2024-03-19T01:51:23.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.334+0000 7f6e95ddc000 1 -- 172.21.15.55:0/324714791 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x558368926dd0 msgr2=0x5583689271b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:23.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.334+0000 7f6e95ddc000 1 --2- 172.21.15.55:0/324714791 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x558368926dd0 0x5583689271b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6e840ab000 tx=0x7f6e84004ac0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.334+0000 7f6e95ddc000 1 -- 172.21.15.55:0/324714791 shutdown_connections 2024-03-19T01:51:23.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.334+0000 7f6e95ddc000 1 --2- 172.21.15.55:0/324714791 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583689276f0 0x558368932060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.334+0000 7f6e95ddc000 1 --2- 172.21.15.55:0/324714791 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x558368926dd0 0x5583689271b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.334+0000 7f6e95ddc000 1 --2- 172.21.15.55:0/324714791 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x5583688eb710 0x5583688ebaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.334+0000 7f6e95ddc000 1 -- 172.21.15.55:0/324714791 >> 172.21.15.55:0/324714791 conn(0x5583688dced0 msgr2=0x558368924b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:23.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.334+0000 7f6e95ddc000 1 -- 172.21.15.55:0/324714791 shutdown_connections 2024-03-19T01:51:23.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.334+0000 7f6e95ddc000 1 -- 172.21.15.55:0/324714791 wait complete. 2024-03-19T01:51:23.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.338+0000 7f6e95ddc000 1 -- start start 2024-03-19T01:51:23.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.338+0000 7f6e95ddc000 1 Processor -- start 2024-03-19T01:51:23.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.338+0000 7f6e897fa640 1 -- 172.21.15.55:0/2498464327 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f6e80053890 con 0x7f6e8c154340 2024-03-19T01:51:23.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.338+0000 7f6e95ddc000 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x5583688eb710 0x558368932f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:23.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.338+0000 7f6e95ddc000 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x558368926dd0 0x5583689334b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:23.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.338+0000 7f6e95ddc000 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583689276f0 0x5583688e5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:23.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.338+0000 7f6e95ddc000 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5583686efb40 con 0x5583689276f0 2024-03-19T01:51:23.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.338+0000 7f6e95ddc000 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x558368731730 con 0x558368926dd0 2024-03-19T01:51:23.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.338+0000 7f6e95ddc000 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x5583687349a0 con 0x5583688eb710 2024-03-19T01:51:23.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.338+0000 7f6e95ddc000 10 client.0 _ll_register_callbacks cb 0x5583688d1740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:51:23.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.338+0000 7f6e909f6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x5583688eb710 0x558368932f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:23.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.338+0000 7f6e911f7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583689276f0 0x5583688e5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:23.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.342+0000 7f6e8bfff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x558368926dd0 0x5583689334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:23.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.342+0000 7f6e911f7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583689276f0 0x5583688e5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41008/0 (socket says 172.21.15.55:41008) 2024-03-19T01:51:23.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.342+0000 7f6e911f7640 1 -- 172.21.15.55:0/3468556916 learned_addr learned my addr 172.21.15.55:0/3468556916 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:23.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.342+0000 7f6e911f7640 1 -- 172.21.15.55:0/3468556916 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x5583688eb710 msgr2=0x558368932f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:23.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.342+0000 7f6e911f7640 1 --2- 172.21.15.55:0/3468556916 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x5583688eb710 0x558368932f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.342+0000 7f6e911f7640 1 -- 172.21.15.55:0/3468556916 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x558368926dd0 msgr2=0x5583689334b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:23.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.342+0000 7f6e911f7640 1 --2- 172.21.15.55:0/3468556916 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x558368926dd0 0x5583689334b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.342+0000 7f6e911f7640 1 -- 172.21.15.55:0/3468556916 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583686bd5b0 con 0x5583689276f0 2024-03-19T01:51:23.347 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.346+0000 7f6e911f7640 1 --2- 172.21.15.55:0/3468556916 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583689276f0 0x5583688e5640 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c014c60 tx=0x7f6e7c0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14394f112c06e44f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:23.347 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.346+0000 7f6e6b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-03-19T01:51:23.351 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.350+0000 7f6e6b7fe640 1 -- 172.21.15.55:0/3468556916 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c01b070 con 0x5583689276f0 2024-03-19T01:51:23.351 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.350+0000 7f6e6b7fe640 1 -- 172.21.15.55:0/3468556916 <== mon.0 v2:172.21.15.55:3300/0 2 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f6e7c020b40 con 0x5583689276f0 2024-03-19T01:51:23.351 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.350+0000 7f6e95ddc000 1 -- 172.21.15.55:0/3468556916 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e840a5880 con 0x5583689276f0 2024-03-19T01:51:23.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.354+0000 7f6e6b7fe640 1 -- 172.21.15.55:0/3468556916 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f6e7c020d40 con 0x5583689276f0 2024-03-19T01:51:23.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.354+0000 7f6e95ddc000 10 client.4311 fetch_fsmap learned FSMap version 8 2024-03-19T01:51:23.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.354+0000 7f6e95ddc000 1 -- 172.21.15.55:0/3468556916 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x5583687b68d0 con 0x5583689276f0 2024-03-19T01:51:23.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.358+0000 7f6e6b7fe640 1 -- 172.21.15.55:0/3468556916 <== mon.0 v2:172.21.15.55:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e840a5880 con 0x5583689276f0 2024-03-19T01:51:23.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.358+0000 7f6e6b7fe640 1 -- 172.21.15.55:0/3468556916 <== mon.0 v2:172.21.15.55:3300/0 5 ==== fsmap(e 8) ==== 4712+0+0 (secure 0 0 0) 0x7f6e7c037080 con 0x5583689276f0 2024-03-19T01:51:23.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.358+0000 7f6e95ddc000 10 client.4311 fetch_fsmap finished waiting for FSMap version 8 2024-03-19T01:51:23.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.358+0000 7f6e95ddc000 10 client.4311 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-19T01:51:23.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.358+0000 7f6e95ddc000 20 client.4311 populate_metadata read hostname 'smithi055' 2024-03-19T01:51:23.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.358+0000 7f6e95ddc000 1 --2- 172.21.15.55:0/3468556916 >> [v2:172.21.15.55:6836/2236779395,v1:172.21.15.55:6837/2236779395] conn(0x5583688e63b0 0x5583688e6790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:23.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.358+0000 7f6e95ddc000 4 client.4311 mds_command: new command op to 4255 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:51:23.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.358+0000 7f6e95ddc000 1 -- 172.21.15.55:0/3468556916 --> [v2:172.21.15.55:6836/2236779395,v1:172.21.15.55:6837/2236779395] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5583686d0bc0 con 0x5583688e63b0 2024-03-19T01:51:23.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.370+0000 7f6e909f6640 1 --2- 172.21.15.55:0/3468556916 >> [v2:172.21.15.55:6836/2236779395,v1:172.21.15.55:6837/2236779395] conn(0x5583688e63b0 0x5583688e6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:23.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.526+0000 7f6e909f6640 1 --2- 172.21.15.55:0/3468556916 >> [v2:172.21.15.55:6836/2236779395,v1:172.21.15.55:6837/2236779395] conn(0x5583688e63b0 0x5583688e6790 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce6dfddcf4b6dd1e server_cookie=80d100e0afdf7212 in_seq=0 out_seq=0 2024-03-19T01:51:23.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.526+0000 7f6e6b7fe640 10 client.4311 ms_handle_connect on v2:172.21.15.55:6836/2236779395 2024-03-19T01:51:23.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.766+0000 7f6e6b7fe640 1 -- 172.21.15.55:0/3468556916 <== mds.0 v2:172.21.15.55:6836/2236779395 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5583686d0bc0 con 0x5583688e63b0 2024-03-19T01:51:23.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.766+0000 7f6e6b7fe640 10 client.4311 handle_command_reply: tid=0 2024-03-19T01:51:23.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.766+0000 7f6e95ddc000 1 client.4311 shutdown 2024-03-19T01:51:23.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.766+0000 7f6e95ddc000 1 -- 172.21.15.55:0/3468556916 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583689276f0 msgr2=0x5583688e5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:23.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.766+0000 7f6e95ddc000 1 --2- 172.21.15.55:0/3468556916 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583689276f0 0x5583688e5640 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c014c60 tx=0x7f6e7c0046d0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.766+0000 7f6e95ddc000 1 -- 172.21.15.55:0/3468556916 shutdown_connections 2024-03-19T01:51:23.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.766+0000 7f6e95ddc000 1 --2- 172.21.15.55:0/3468556916 >> [v2:172.21.15.55:6836/2236779395,v1:172.21.15.55:6837/2236779395] conn(0x5583688e63b0 0x5583688e6790 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.766+0000 7f6e95ddc000 1 --2- 172.21.15.55:0/3468556916 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583689276f0 0x5583688e5640 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.766+0000 7f6e95ddc000 1 --2- 172.21.15.55:0/3468556916 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x558368926dd0 0x5583689334b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.766+0000 7f6e95ddc000 1 --2- 172.21.15.55:0/3468556916 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x5583688eb710 0x558368932f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.766+0000 7f6e95ddc000 1 -- 172.21.15.55:0/3468556916 >> 172.21.15.55:0/3468556916 conn(0x5583688dced0 msgr2=0x558368925dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:23.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.766+0000 7f6e6b7fe640 0 client.4311 ms_handle_reset on v2:172.21.15.55:6836/2236779395 2024-03-19T01:51:23.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.766+0000 7f6e95ddc000 1 -- 172.21.15.55:0/3468556916 shutdown_connections 2024-03-19T01:51:23.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.766+0000 7f6e95ddc000 1 -- 172.21.15.55:0/3468556916 wait complete. 2024-03-19T01:51:23.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.766+0000 7f6e95ddc000 20 client.4311 trim_cache size 0 max 16384 2024-03-19T01:51:23.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.818+0000 7f6e95ddc000 1 Processor -- start 2024-03-19T01:51:23.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.818+0000 7f6e95ddc000 1 -- start start 2024-03-19T01:51:23.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.818+0000 7f6e95ddc000 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583688dced0 0x5583688e3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:23.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.818+0000 7f6e95ddc000 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x5583688eb710 0x558368942810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:23.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.818+0000 7f6e95ddc000 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x558368926dd0 0x558368942d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:23.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.818+0000 7f6e95ddc000 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8c200370 con 0x5583688dced0 2024-03-19T01:51:23.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.818+0000 7f6e95ddc000 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8c1ff6d0 con 0x558368926dd0 2024-03-19T01:51:23.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.818+0000 7f6e95ddc000 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f6e8c1ffd20 con 0x5583688eb710 2024-03-19T01:51:23.820 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.818+0000 7f6e909f6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583688dced0 0x5583688e3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:23.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.822+0000 7f6e911f7640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x558368926dd0 0x558368942d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:23.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.822+0000 7f6e8bfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x5583688eb710 0x558368942810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:23.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.822+0000 7f6e909f6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583688dced0 0x5583688e3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41020/0 (socket says 172.21.15.55:41020) 2024-03-19T01:51:23.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.822+0000 7f6e909f6640 1 -- 172.21.15.55:0/4244084063 learned_addr learned my addr 172.21.15.55:0/4244084063 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:23.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.826+0000 7f6e909f6640 1 -- 172.21.15.55:0/4244084063 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x5583688eb710 msgr2=0x558368942810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:23.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.826+0000 7f6e909f6640 1 --2- 172.21.15.55:0/4244084063 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x5583688eb710 0x558368942810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.826+0000 7f6e909f6640 1 -- 172.21.15.55:0/4244084063 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x558368926dd0 msgr2=0x558368942d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:23.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.826+0000 7f6e909f6640 1 --2- 172.21.15.55:0/4244084063 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x558368926dd0 0x558368942d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.826+0000 7f6e909f6640 1 -- 172.21.15.55:0/4244084063 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55836878d300 con 0x5583688dced0 2024-03-19T01:51:23.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.830+0000 7f6e909f6640 1 --2- 172.21.15.55:0/4244084063 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583688dced0 0x5583688e3010 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6e8401c480 tx=0x7f6e840a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c7aad9fa28494d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:23.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.838+0000 7f6e71ffb640 1 -- 172.21.15.55:0/4244084063 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e840039a0 con 0x5583688dced0 2024-03-19T01:51:23.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.838+0000 7f6e71ffb640 1 -- 172.21.15.55:0/4244084063 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e84003b40 con 0x5583688dced0 2024-03-19T01:51:23.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.838+0000 7f6e71ffb640 1 -- 172.21.15.55:0/4244084063 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e840a5800 con 0x5583688dced0 2024-03-19T01:51:23.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.838+0000 7f6e95ddc000 1 -- 172.21.15.55:0/4244084063 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583688dced0 msgr2=0x5583688e3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:23.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.838+0000 7f6e95ddc000 1 --2- 172.21.15.55:0/4244084063 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583688dced0 0x5583688e3010 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6e8401c480 tx=0x7f6e840a7000 comp rx=0 tx=0).stop 2024-03-19T01:51:23.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.838+0000 7f6e95ddc000 1 -- 172.21.15.55:0/4244084063 shutdown_connections 2024-03-19T01:51:23.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.838+0000 7f6e95ddc000 1 --2- 172.21.15.55:0/4244084063 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x558368926dd0 0x558368942d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.838+0000 7f6e95ddc000 1 --2- 172.21.15.55:0/4244084063 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x5583688eb710 0x558368942810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.838+0000 7f6e95ddc000 1 --2- 172.21.15.55:0/4244084063 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583688dced0 0x5583688e3010 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.838+0000 7f6e95ddc000 1 -- 172.21.15.55:0/4244084063 >> 172.21.15.55:0/4244084063 conn(0x558368926380 msgr2=0x558368942100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:23.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.838+0000 7f6e95ddc000 1 -- 172.21.15.55:0/4244084063 shutdown_connections 2024-03-19T01:51:23.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.838+0000 7f6e95ddc000 1 -- 172.21.15.55:0/4244084063 wait complete. 2024-03-19T01:51:23.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.838+0000 7f6e95ddc000 1 -- start start 2024-03-19T01:51:23.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.842+0000 7f6e95ddc000 1 Processor -- start 2024-03-19T01:51:23.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.842+0000 7f6e95ddc000 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5583688dced0 0x55836899cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:23.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.842+0000 7f6e95ddc000 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583688eb710 0x55836899aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:23.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.842+0000 7f6e95ddc000 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x558368926dd0 0x55836899b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:23.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.842+0000 7f6e95ddc000 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x558368732700 con 0x5583688eb710 2024-03-19T01:51:23.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.842+0000 7f6e95ddc000 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x5583686fd750 con 0x5583688dced0 2024-03-19T01:51:23.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.842+0000 7f6e95ddc000 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x5583686d4540 con 0x558368926dd0 2024-03-19T01:51:23.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.842+0000 7f6e95ddc000 10 client.0 _ll_register_callbacks cb 0x7f6e7c014f60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:51:23.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.842+0000 7f6e8bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583688eb710 0x55836899aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:23.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.842+0000 7f6e8bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583688eb710 0x55836899aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41034/0 (socket says 172.21.15.55:41034) 2024-03-19T01:51:23.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.842+0000 7f6e8bfff640 1 -- 172.21.15.55:0/3342952238 learned_addr learned my addr 172.21.15.55:0/3342952238 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:23.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.842+0000 7f6e909f6640 1 --2- 172.21.15.55:0/3342952238 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5583688dced0 0x55836899cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:23.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.842+0000 7f6e911f7640 1 --2- 172.21.15.55:0/3342952238 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x558368926dd0 0x55836899b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:23.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.846+0000 7f6e8bfff640 1 -- 172.21.15.55:0/3342952238 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x558368926dd0 msgr2=0x55836899b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:23.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.846+0000 7f6e8bfff640 1 --2- 172.21.15.55:0/3342952238 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x558368926dd0 0x55836899b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.846+0000 7f6e8bfff640 1 -- 172.21.15.55:0/3342952238 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5583688dced0 msgr2=0x55836899cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:23.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.846+0000 7f6e8bfff640 1 --2- 172.21.15.55:0/3342952238 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5583688dced0 0x55836899cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.846+0000 7f6e8bfff640 1 -- 172.21.15.55:0/3342952238 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583686e5190 con 0x5583688eb710 2024-03-19T01:51:23.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.850+0000 7f6e8bfff640 1 --2- 172.21.15.55:0/3342952238 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583688eb710 0x55836899aaf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6e80055780 tx=0x7f6e80055860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a685c050941852f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:23.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.850+0000 7f6e6b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-03-19T01:51:23.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.854+0000 7f6e6b7fe640 1 -- 172.21.15.55:0/3342952238 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e80054e50 con 0x5583688eb710 2024-03-19T01:51:23.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.854+0000 7f6e6b7fe640 1 -- 172.21.15.55:0/3342952238 <== mon.0 v2:172.21.15.55:3300/0 2 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f6e80051540 con 0x5583688eb710 2024-03-19T01:51:23.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.854+0000 7f6e95ddc000 1 -- 172.21.15.55:0/3342952238 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e84003b40 con 0x5583688eb710 2024-03-19T01:51:23.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.858+0000 7f6e6b7fe640 1 -- 172.21.15.55:0/3342952238 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f6e80051740 con 0x5583688eb710 2024-03-19T01:51:23.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.858+0000 7f6e95ddc000 10 client.4317 fetch_fsmap learned FSMap version 8 2024-03-19T01:51:23.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.858+0000 7f6e95ddc000 1 -- 172.21.15.55:0/3342952238 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x5583686c94b0 con 0x5583688eb710 2024-03-19T01:51:23.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.862+0000 7f6e6b7fe640 1 -- 172.21.15.55:0/3342952238 <== mon.0 v2:172.21.15.55:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e84003b40 con 0x5583688eb710 2024-03-19T01:51:23.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.862+0000 7f6e6b7fe640 1 -- 172.21.15.55:0/3342952238 <== mon.0 v2:172.21.15.55:3300/0 5 ==== fsmap(e 8) ==== 4712+0+0 (secure 0 0 0) 0x7f6e800546b0 con 0x5583688eb710 2024-03-19T01:51:23.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.862+0000 7f6e95ddc000 10 client.4317 fetch_fsmap finished waiting for FSMap version 8 2024-03-19T01:51:23.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.862+0000 7f6e95ddc000 10 client.4317 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-19T01:51:23.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.862+0000 7f6e95ddc000 20 client.4317 populate_metadata read hostname 'smithi055' 2024-03-19T01:51:23.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.862+0000 7f6e95ddc000 1 --2- 172.21.15.55:0/3342952238 >> [v2:172.21.15.55:6836/2236779395,v1:172.21.15.55:6837/2236779395] conn(0x5583689276f0 0x558368999af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:23.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.862+0000 7f6e95ddc000 4 client.4317 mds_command: new command op to 4255 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T01:51:23.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.862+0000 7f6e95ddc000 1 -- 172.21.15.55:0/3342952238 --> [v2:172.21.15.55:6836/2236779395,v1:172.21.15.55:6837/2236779395] -- command(tid 0: {"prefix": "status"}) -- 0x558368742bd0 con 0x5583689276f0 2024-03-19T01:51:23.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.862+0000 7f6e909f6640 1 --2- 172.21.15.55:0/3342952238 >> [v2:172.21.15.55:6836/2236779395,v1:172.21.15.55:6837/2236779395] conn(0x5583689276f0 0x558368999af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:23.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.870+0000 7f6e909f6640 1 --2- 172.21.15.55:0/3342952238 >> [v2:172.21.15.55:6836/2236779395,v1:172.21.15.55:6837/2236779395] conn(0x5583689276f0 0x558368999af0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa5184d8f28fb79 server_cookie=e2148aab5ce95077 in_seq=0 out_seq=0 2024-03-19T01:51:23.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.870+0000 7f6e6b7fe640 10 client.4317 ms_handle_connect on v2:172.21.15.55:6836/2236779395 2024-03-19T01:51:23.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.890+0000 7f6e6b7fe640 1 -- 172.21.15.55:0/3342952238 <== mds.0 v2:172.21.15.55:6836/2236779395 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x558368742bd0 con 0x5583689276f0 2024-03-19T01:51:23.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.890+0000 7f6e6b7fe640 10 client.4317 handle_command_reply: tid=0 2024-03-19T01:51:23.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.890+0000 7f6e95ddc000 1 client.4317 shutdown 2024-03-19T01:51:23.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e95ddc000 1 -- 172.21.15.55:0/3342952238 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583688eb710 msgr2=0x55836899aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:23.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e95ddc000 1 --2- 172.21.15.55:0/3342952238 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583688eb710 0x55836899aaf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6e80055780 tx=0x7f6e80055860 comp rx=0 tx=0).stop 2024-03-19T01:51:23.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e95ddc000 1 -- 172.21.15.55:0/3342952238 shutdown_connections 2024-03-19T01:51:23.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e95ddc000 1 --2- 172.21.15.55:0/3342952238 >> [v2:172.21.15.55:6836/2236779395,v1:172.21.15.55:6837/2236779395] conn(0x5583689276f0 0x558368999af0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e95ddc000 1 --2- 172.21.15.55:0/3342952238 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x558368926dd0 0x55836899b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e95ddc000 1 --2- 172.21.15.55:0/3342952238 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5583688eb710 0x55836899aaf0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e95ddc000 1 --2- 172.21.15.55:0/3342952238 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5583688dced0 0x55836899cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e6b7fe640 0 client.4317 ms_handle_reset on v2:172.21.15.55:6836/2236779395 2024-03-19T01:51:23.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e95ddc000 1 -- 172.21.15.55:0/3342952238 >> 172.21.15.55:0/3342952238 conn(0x558368926380 msgr2=0x558368942990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:23.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e95ddc000 1 -- 172.21.15.55:0/3342952238 shutdown_connections 2024-03-19T01:51:23.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e95ddc000 1 -- 172.21.15.55:0/3342952238 wait complete. 2024-03-19T01:51:23.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e95ddc000 20 client.4317 trim_cache size 0 max 16384 2024-03-19T01:51:23.894 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-19T01:51:23.894 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_fsid": "def9d4e5-0bc7-449d-b893-2490cbd10e25", 2024-03-19T01:51:23.895 INFO:teuthology.orchestra.run.smithi055.stdout: "whoami": 0, 2024-03-19T01:51:23.895 INFO:teuthology.orchestra.run.smithi055.stdout: "id": 4255, 2024-03-19T01:51:23.895 INFO:teuthology.orchestra.run.smithi055.stdout: "want_state": "up:active", 2024-03-19T01:51:23.895 INFO:teuthology.orchestra.run.smithi055.stdout: "state": "up:active", 2024-03-19T01:51:23.895 INFO:teuthology.orchestra.run.smithi055.stdout: "fs_name": "cephfs", 2024-03-19T01:51:23.895 INFO:teuthology.orchestra.run.smithi055.stdout: "rank_uptime": 12.428834135000001, 2024-03-19T01:51:23.895 INFO:teuthology.orchestra.run.smithi055.stdout: "mdsmap_epoch": 8, 2024-03-19T01:51:23.895 INFO:teuthology.orchestra.run.smithi055.stdout: "osdmap_epoch": 29, 2024-03-19T01:51:23.895 INFO:teuthology.orchestra.run.smithi055.stdout: "osdmap_epoch_barrier": 29, 2024-03-19T01:51:23.895 INFO:teuthology.orchestra.run.smithi055.stdout: "uptime": 49.777137623999998 2024-03-19T01:51:23.895 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-19T01:51:23.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e92c5a640 1 -- 172.21.15.55:0/2498464327 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6e780676e0 msgr2=0x7f6e78069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:23.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e92c5a640 1 --2- 172.21.15.55:0/2498464327 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6e780676e0 0x7f6e78069ba0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c006c10 tx=0x7f6e7c009190 comp rx=0 tx=0).stop 2024-03-19T01:51:23.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e92c5a640 1 -- 172.21.15.55:0/2498464327 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e8c154340 msgr2=0x7f6e8c14a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:23.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e92c5a640 1 --2- 172.21.15.55:0/2498464327 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e8c154340 0x7f6e8c14a280 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6e8000c450 tx=0x7f6e8000c910 comp rx=0 tx=0).stop 2024-03-19T01:51:23.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e92c5a640 1 -- 172.21.15.55:0/2498464327 shutdown_connections 2024-03-19T01:51:23.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e92c5a640 1 --2- 172.21.15.55:0/2498464327 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6e780676e0 0x7f6e78069ba0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e92c5a640 1 --2- 172.21.15.55:0/2498464327 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6e8c154720 0x7f6e8c1f6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e92c5a640 1 --2- 172.21.15.55:0/2498464327 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e8c154340 0x7f6e8c14a280 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e92c5a640 1 --2- 172.21.15.55:0/2498464327 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e8c0c73c0 0x7f6e8c14e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:23.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e92c5a640 1 -- 172.21.15.55:0/2498464327 >> 172.21.15.55:0/2498464327 conn(0x7f6e8c0cc9c0 msgr2=0x7f6e8c14d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:23.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e92c5a640 1 -- 172.21.15.55:0/2498464327 shutdown_connections 2024-03-19T01:51:23.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:23.894+0000 7f6e92c5a640 1 -- 172.21.15.55:0/2498464327 wait complete. 2024-03-19T01:51:23.912 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-19T01:51:23.912 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-19T01:51:23.912 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-19T01:51:24.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.030+0000 7fed8ecd3640 1 Processor -- start 2024-03-19T01:51:24.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.030+0000 7fed8ecd3640 1 -- start start 2024-03-19T01:51:24.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.030+0000 7fed8ecd3640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fed8806bb00 0x7fed8806c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:24.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.030+0000 7fed8ecd3640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fed880c6b00 0x7fed88150640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:24.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.030+0000 7fed8ecd3640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed88159d80 con 0x7fed8806bee0 2024-03-19T01:51:24.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.030+0000 7fed8ecd3640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fed8815a060 con 0x7fed8806bb00 2024-03-19T01:51:24.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.030+0000 7fed8ecd3640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fed8815a340 con 0x7fed880c6b00 2024-03-19T01:51:24.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.034+0000 7fed87fff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fed880c6b00 0x7fed88150640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:24.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.034+0000 7fed87fff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fed880c6b00 0x7fed88150640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:39322/0 (socket says 172.21.15.55:39322) 2024-03-19T01:51:24.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.034+0000 7fed87fff640 1 -- 172.21.15.55:0/599778633 learned_addr learned my addr 172.21.15.55:0/599778633 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:24.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.038+0000 7fed87fff640 1 -- 172.21.15.55:0/599778633 >> v2:172.21.15.55:3300/0 conn(0x7fed8806bb00 msgr2=0x7fed8806c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:24.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.038+0000 7fed87fff640 1 --2- 172.21.15.55:0/599778633 >> v2:172.21.15.55:3300/0 conn(0x7fed8806bb00 0x7fed8806c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:24.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.038+0000 7fed87fff640 1 -- 172.21.15.55:0/599778633 >> v1:172.21.15.55:6789/0 conn(0x7fed8806bee0 legacy=0x7fed88156b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:24.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.038+0000 7fed87fff640 1 -- 172.21.15.55:0/599778633 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8815a620 con 0x7fed880c6b00 2024-03-19T01:51:24.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.046+0000 7fed87fff640 1 --2- 172.21.15.55:0/599778633 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed880c6b00 0x7fed88150640 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fed880c5b90 tx=0x7fed7401ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c39c08ad63a8070 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:24.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.054+0000 7fed877fe640 1 -- 172.21.15.55:0/599778633 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed740b5070 con 0x7fed880c6b00 2024-03-19T01:51:24.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.054+0000 7fed877fe640 1 -- 172.21.15.55:0/599778633 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed740a89d0 con 0x7fed880c6b00 2024-03-19T01:51:24.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.054+0000 7fed877fe640 1 -- 172.21.15.55:0/599778633 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed740a8da0 con 0x7fed880c6b00 2024-03-19T01:51:24.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.058+0000 7fed8ecd3640 1 -- 172.21.15.55:0/599778633 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed880c6b00 msgr2=0x7fed88150640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:24.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.058+0000 7fed8ecd3640 1 --2- 172.21.15.55:0/599778633 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed880c6b00 0x7fed88150640 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fed880c5b90 tx=0x7fed7401ecd0 comp rx=0 tx=0).stop 2024-03-19T01:51:24.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.058+0000 7fed8ecd3640 1 -- 172.21.15.55:0/599778633 shutdown_connections 2024-03-19T01:51:24.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.058+0000 7fed8ecd3640 1 --2- 172.21.15.55:0/599778633 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed880c6b00 0x7fed88150640 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:24.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.058+0000 7fed8ecd3640 1 --2- 172.21.15.55:0/599778633 >> v2:172.21.15.55:3300/0 conn(0x7fed8806bb00 0x7fed8806c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:24.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.058+0000 7fed8ecd3640 1 -- 172.21.15.55:0/599778633 >> 172.21.15.55:0/599778633 conn(0x7fed8806fb40 msgr2=0x7fed8806ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:24.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.058+0000 7fed8ecd3640 1 -- 172.21.15.55:0/599778633 shutdown_connections 2024-03-19T01:51:24.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.058+0000 7fed8ecd3640 1 -- 172.21.15.55:0/599778633 wait complete. 2024-03-19T01:51:24.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.058+0000 7fed8ecd3640 1 Processor -- start 2024-03-19T01:51:24.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.058+0000 7fed8ecd3640 1 -- start start 2024-03-19T01:51:24.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.062+0000 7fed8ecd3640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fed8806bb00 0x7fed88145c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:24.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.062+0000 7fed8ecd3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed8806bee0 0x7fed880c85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:24.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.062+0000 7fed8ecd3640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed880c6b00 0x7fed88146780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:24.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.062+0000 7fed8ecd3640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fed881573a0 con 0x7fed8806bee0 2024-03-19T01:51:24.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.062+0000 7fed8ecd3640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fed88157220 con 0x7fed880c6b00 2024-03-19T01:51:24.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.062+0000 7fed8ecd3640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fed881ef880 con 0x7fed8806bb00 2024-03-19T01:51:24.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.062+0000 7fed87fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed8806bee0 0x7fed880c85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:24.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.062+0000 7fed87fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed8806bee0 0x7fed880c85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41056/0 (socket says 172.21.15.55:41056) 2024-03-19T01:51:24.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.062+0000 7fed87fff640 1 -- 172.21.15.55:0/1931837120 learned_addr learned my addr 172.21.15.55:0/1931837120 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:24.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.062+0000 7fed8ca6f640 1 --2- 172.21.15.55:0/1931837120 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fed8806bb00 0x7fed88145c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:24.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.062+0000 7fed8d270640 1 --2- 172.21.15.55:0/1931837120 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed880c6b00 0x7fed88146780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:24.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.066+0000 7fed87fff640 1 -- 172.21.15.55:0/1931837120 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fed8806bb00 msgr2=0x7fed88145c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:24.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.066+0000 7fed87fff640 1 --2- 172.21.15.55:0/1931837120 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fed8806bb00 0x7fed88145c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:24.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.066+0000 7fed87fff640 1 -- 172.21.15.55:0/1931837120 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed880c6b00 msgr2=0x7fed88146780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:24.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.066+0000 7fed87fff640 1 --2- 172.21.15.55:0/1931837120 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed880c6b00 0x7fed88146780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:24.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.066+0000 7fed87fff640 1 -- 172.21.15.55:0/1931837120 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed881efb60 con 0x7fed8806bee0 2024-03-19T01:51:24.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.066+0000 7fed8ca6f640 1 --2- 172.21.15.55:0/1931837120 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fed8806bb00 0x7fed88145c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:51:24.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.070+0000 7fed87fff640 1 --2- 172.21.15.55:0/1931837120 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed8806bee0 0x7fed880c85f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fed7401c900 tx=0x7fed7401ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:24.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.074+0000 7fed857fa640 1 -- 172.21.15.55:0/1931837120 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed740b5050 con 0x7fed8806bee0 2024-03-19T01:51:24.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.074+0000 7fed857fa640 1 -- 172.21.15.55:0/1931837120 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed740bf070 con 0x7fed8806bee0 2024-03-19T01:51:24.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.074+0000 7fed8ecd3640 1 -- 172.21.15.55:0/1931837120 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed881efe40 con 0x7fed8806bee0 2024-03-19T01:51:24.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.074+0000 7fed857fa640 1 -- 172.21.15.55:0/1931837120 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed740b0cf0 con 0x7fed8806bee0 2024-03-19T01:51:24.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.074+0000 7fed8ecd3640 1 -- 172.21.15.55:0/1931837120 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed881f0400 con 0x7fed8806bee0 2024-03-19T01:51:24.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.074+0000 7fed8ecd3640 1 -- 172.21.15.55:0/1931837120 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed88068a40 con 0x7fed8806bee0 2024-03-19T01:51:24.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.374+0000 7fed857fa640 1 -- 172.21.15.55:0/1931837120 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fed740ba440 con 0x7fed8806bee0 2024-03-19T01:51:24.376 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.374+0000 7fed857fa640 1 --2- 172.21.15.55:0/1931837120 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fed6c0676e0 0x7fed6c069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:24.377 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.378+0000 7fed8ca6f640 1 --2- 172.21.15.55:0/1931837120 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fed6c0676e0 0x7fed6c069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:24.377 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.378+0000 7fed8ca6f640 1 --2- 172.21.15.55:0/1931837120 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fed6c0676e0 0x7fed6c069ba0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fed7c006fd0 tx=0x7fed7c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:24.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.414+0000 7fed857fa640 1 -- 172.21.15.55:0/1931837120 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fed740c5e00 con 0x7fed8806bee0 2024-03-19T01:51:24.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.414+0000 7fed857fa640 1 -- 172.21.15.55:0/1931837120 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fed740c9020 con 0x7fed8806bee0 2024-03-19T01:51:24.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.634+0000 7fed8ecd3640 1 -- 172.21.15.55:0/1931837120 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fed881464e0 con 0x7fed8806bee0 2024-03-19T01:51:24.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.650+0000 7fed857fa640 1 -- 172.21.15.55:0/1931837120 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7fed74018830 con 0x7fed8806bee0 2024-03-19T01:51:24.651 INFO:teuthology.orchestra.run.smithi055.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-19T01:51:24.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.654+0000 7fed8ecd3640 1 -- 172.21.15.55:0/1931837120 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fed6c0676e0 msgr2=0x7fed6c069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:24.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.654+0000 7fed8ecd3640 1 --2- 172.21.15.55:0/1931837120 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fed6c0676e0 0x7fed6c069ba0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fed7c006fd0 tx=0x7fed7c008040 comp rx=0 tx=0).stop 2024-03-19T01:51:24.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.654+0000 7fed8ecd3640 1 -- 172.21.15.55:0/1931837120 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed8806bee0 msgr2=0x7fed880c85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:24.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.654+0000 7fed8ecd3640 1 --2- 172.21.15.55:0/1931837120 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed8806bee0 0x7fed880c85f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fed7401c900 tx=0x7fed7401ba70 comp rx=0 tx=0).stop 2024-03-19T01:51:24.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.654+0000 7fed8ecd3640 1 -- 172.21.15.55:0/1931837120 shutdown_connections 2024-03-19T01:51:24.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.654+0000 7fed8ecd3640 1 --2- 172.21.15.55:0/1931837120 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fed6c0676e0 0x7fed6c069ba0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:24.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.654+0000 7fed8ecd3640 1 --2- 172.21.15.55:0/1931837120 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed880c6b00 0x7fed88146780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:24.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.654+0000 7fed8ecd3640 1 --2- 172.21.15.55:0/1931837120 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed8806bee0 0x7fed880c85f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:24.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.654+0000 7fed8ecd3640 1 --2- 172.21.15.55:0/1931837120 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fed8806bb00 0x7fed88145c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:24.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.654+0000 7fed8ecd3640 1 -- 172.21.15.55:0/1931837120 >> 172.21.15.55:0/1931837120 conn(0x7fed8806fb40 msgr2=0x7fed881558e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:24.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.654+0000 7fed8ecd3640 1 -- 172.21.15.55:0/1931837120 shutdown_connections 2024-03-19T01:51:24.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24.654+0000 7fed8ecd3640 1 -- 172.21.15.55:0/1931837120 wait complete. 2024-03-19T01:51:24.672 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-19T01:51:24.672 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-19T01:51:24.672 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi055.front.sepia.ceph.com 2024-03-19T01:51:24.672 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-19T01:51:24.672 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-19T01:51:24.672 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-19T01:51:24.672 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-19T01:51:24.672 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-19T01:51:24.672 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-19T01:51:24.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:51:24.673 DEBUG:teuthology.orchestra.run.smithi055:> ip addr 2024-03-19T01:51:24.679 INFO:teuthology.orchestra.run.smithi055.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-19T01:51:24.679 INFO:teuthology.orchestra.run.smithi055.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-19T01:51:24.679 INFO:teuthology.orchestra.run.smithi055.stdout: inet 127.0.0.1/8 scope host lo 2024-03-19T01:51:24.679 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-03-19T01:51:24.679 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 ::1/128 scope host 2024-03-19T01:51:24.679 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-03-19T01:51:24.679 INFO:teuthology.orchestra.run.smithi055.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-19T01:51:24.679 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 0c:c4:7a:6b:fd:d6 brd ff:ff:ff:ff:ff:ff 2024-03-19T01:51:24.679 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp7s0f0 2024-03-19T01:51:24.679 INFO:teuthology.orchestra.run.smithi055.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-19T01:51:24.679 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether ac:1f:6b:f8:26:06 brd ff:ff:ff:ff:ff:ff 2024-03-19T01:51:24.679 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp1s0f0 2024-03-19T01:51:24.680 INFO:teuthology.orchestra.run.smithi055.stdout: inet 172.21.15.55/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-19T01:51:24.680 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft 42097sec preferred_lft 42097sec 2024-03-19T01:51:24.680 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 fe80::ae1f:6bff:fef8:2606/64 scope link 2024-03-19T01:51:24.680 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-03-19T01:51:24.680 INFO:teuthology.orchestra.run.smithi055.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-19T01:51:24.680 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 0c:c4:7a:6b:fd:d7 brd ff:ff:ff:ff:ff:ff 2024-03-19T01:51:24.680 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp7s0f1 2024-03-19T01:51:24.680 INFO:teuthology.orchestra.run.smithi055.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-19T01:51:24.680 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether ac:1f:6b:f8:26:07 brd ff:ff:ff:ff:ff:ff 2024-03-19T01:51:24.680 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp1s0f1 2024-03-19T01:51:24.680 INFO:teuthology.orchestra.run.smithi055.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-19T01:51:24.680 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 02:42:1e:d2:95:d9 brd ff:ff:ff:ff:ff:ff 2024-03-19T01:51:24.680 INFO:teuthology.orchestra.run.smithi055.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-19T01:51:24.680 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2024-03-19T01:51:24.681 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-19T01:51:24.681 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T01:51:24.681 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-03-19T01:51:24.681 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link add name ceph-brx type bridge 2024-03-19T01:51:24.681 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip addr flush dev ceph-brx 2024-03-19T01:51:24.681 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link set ceph-brx up 2024-03-19T01:51:24.682 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-19T01:51:24.682 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-03-19T01:51:24.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T01:51:24.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:51:24.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:51:24.796 DEBUG:teuthology.orchestra.run.smithi055:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-19T01:51:24.849 INFO:teuthology.orchestra.run.smithi055.stdout:1 2024-03-19T01:51:24.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:51:24.850 DEBUG:teuthology.orchestra.run.smithi055:> route 2024-03-19T01:51:24.910 INFO:teuthology.orchestra.run.smithi055.stdout:Kernel IP routing table 2024-03-19T01:51:24.910 INFO:teuthology.orchestra.run.smithi055.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-19T01:51:24.910 INFO:teuthology.orchestra.run.smithi055.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-19T01:51:24.910 INFO:teuthology.orchestra.run.smithi055.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-19T01:51:24.911 INFO:teuthology.orchestra.run.smithi055.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-19T01:51:24.911 INFO:teuthology.orchestra.run.smithi055.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-19T01:51:24.911 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T01:51:24.911 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-03-19T01:51:24.911 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-19T01:51:24.911 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-19T01:51:24.911 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-19T01:51:24.911 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-03-19T01:51:24.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T01:51:25.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:51:25.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:51:25.036 DEBUG:teuthology.orchestra.run.smithi055:> ip netns list 2024-03-19T01:51:25.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:51:25.082 DEBUG:teuthology.orchestra.run.smithi055:> ip netns list-id 2024-03-19T01:51:25.129 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T01:51:25.129 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-03-19T01:51:25.129 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T01:51:25.129 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-19T01:51:25.130 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-03-19T01:51:25.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T01:51:25.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:51:25.241 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-19T01:51:25.241 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T01:51:25.241 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-03-19T01:51:25.241 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-19T01:51:25.241 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-19T01:51:25.241 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-19T01:51:25.241 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-19T01:51:25.241 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-19T01:51:25.241 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-03-19T01:51:25.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T01:51:25.376 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:51:25.381 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T01:51:25.382 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2024-03-19T01:51:25.382 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link set brx.0 up 2024-03-19T01:51:25.382 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link set dev brx.0 master ceph-brx 2024-03-19T01:51:25.382 DEBUG:teuthology.orchestra.run.smithi055:> ') 2024-03-19T01:51:25.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T01:51:25.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:51:25.478 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-19T01:51:25.479 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T01:51:25.479 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-19T01:51:25.528 INFO:teuthology.orchestra.run.smithi055.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-19T01:51:25.528 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T01:51:25.528 DEBUG:teuthology.orchestra.run.smithi055:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-19T01:51:25.581 DEBUG:teuthology.orchestra.run.smithi055:> sudo modprobe fuse 2024-03-19T01:51:25.638 DEBUG:teuthology.orchestra.run.smithi055:> cat /proc/self/mounts | awk '{print $2}' 2024-03-19T01:51:25.685 INFO:teuthology.orchestra.run.smithi055.stdout:/sys 2024-03-19T01:51:25.686 INFO:teuthology.orchestra.run.smithi055.stdout:/proc 2024-03-19T01:51:25.686 INFO:teuthology.orchestra.run.smithi055.stdout:/dev 2024-03-19T01:51:25.686 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/pts 2024-03-19T01:51:25.686 INFO:teuthology.orchestra.run.smithi055.stdout:/run 2024-03-19T01:51:25.686 INFO:teuthology.orchestra.run.smithi055.stdout:/ 2024-03-19T01:51:25.686 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/security 2024-03-19T01:51:25.686 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/shm 2024-03-19T01:51:25.687 INFO:teuthology.orchestra.run.smithi055.stdout:/run/lock 2024-03-19T01:51:25.687 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/cgroup 2024-03-19T01:51:25.687 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/pstore 2024-03-19T01:51:25.687 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/bpf 2024-03-19T01:51:25.687 INFO:teuthology.orchestra.run.smithi055.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T01:51:25.687 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/hugepages 2024-03-19T01:51:25.688 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/mqueue 2024-03-19T01:51:25.688 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/debug 2024-03-19T01:51:25.688 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/tracing 2024-03-19T01:51:25.688 INFO:teuthology.orchestra.run.smithi055.stdout:/proc/fs/nfsd 2024-03-19T01:51:25.688 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/fuse/connections 2024-03-19T01:51:25.688 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/config 2024-03-19T01:51:25.689 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-sysusers.service 2024-03-19T01:51:25.689 INFO:teuthology.orchestra.run.smithi055.stdout:/run/rpc_pipefs 2024-03-19T01:51:25.689 INFO:teuthology.orchestra.run.smithi055.stdout:/run/user/1000 2024-03-19T01:51:25.689 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph 2024-03-19T01:51:25.689 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-0 2024-03-19T01:51:25.689 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-1 2024-03-19T01:51:25.690 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-2 2024-03-19T01:51:25.690 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-3 2024-03-19T01:51:25.690 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns 2024-03-19T01:51:25.690 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T01:51:25.690 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T01:51:25.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:51:25.692 DEBUG:teuthology.orchestra.run.smithi055:> ls /sys/fs/fuse/connections 2024-03-19T01:51:25.736 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-19T01:51:25.737 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-19T01:51:25.782 DEBUG:teuthology.orchestra.run.smithi055:> sudo modprobe fuse 2024-03-19T01:51:25.793 DEBUG:teuthology.orchestra.run.smithi055:> cat /proc/self/mounts | awk '{print $2}' 2024-03-19T01:51:25.842 INFO:teuthology.orchestra.run.smithi055.stdout:/sys 2024-03-19T01:51:25.842 INFO:teuthology.orchestra.run.smithi055.stdout:/proc 2024-03-19T01:51:25.842 INFO:teuthology.orchestra.run.smithi055.stdout:/dev 2024-03-19T01:51:25.842 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/pts 2024-03-19T01:51:25.842 INFO:teuthology.orchestra.run.smithi055.stdout:/run 2024-03-19T01:51:25.842 INFO:teuthology.orchestra.run.smithi055.stdout:/ 2024-03-19T01:51:25.842 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/security 2024-03-19T01:51:25.842 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/shm 2024-03-19T01:51:25.842 INFO:teuthology.orchestra.run.smithi055.stdout:/run/lock 2024-03-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/cgroup 2024-03-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/pstore 2024-03-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/bpf 2024-03-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi055.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/hugepages 2024-03-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/mqueue 2024-03-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/debug 2024-03-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/tracing 2024-03-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi055.stdout:/proc/fs/nfsd 2024-03-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/fuse/connections 2024-03-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/config 2024-03-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-sysusers.service 2024-03-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi055.stdout:/run/rpc_pipefs 2024-03-19T01:51:25.843 INFO:teuthology.orchestra.run.smithi055.stdout:/run/user/1000 2024-03-19T01:51:25.844 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph 2024-03-19T01:51:25.844 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-0 2024-03-19T01:51:25.844 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-1 2024-03-19T01:51:25.844 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-2 2024-03-19T01:51:25.844 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-3 2024-03-19T01:51:25.844 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns 2024-03-19T01:51:25.844 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T01:51:25.844 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T01:51:25.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:51:25.844 DEBUG:teuthology.orchestra.run.smithi055:> ls /sys/fs/fuse/connections 2024-03-19T01:51:25.886 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi055.stderr:2024-03-19T01:51:25.886+0000 7f2e28709580 -1 init, newargv = 0x5558340ab0a0 newargc=15 2024-03-19T01:51:25.886 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi055.stderr:ceph-fuse[33232]: starting ceph client 2024-03-19T01:51:26.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi055.stderr:ceph-fuse[33232]: starting fuse 2024-03-19T01:51:26.989 DEBUG:teuthology.orchestra.run.smithi055:> sudo modprobe fuse 2024-03-19T01:51:27.006 DEBUG:teuthology.orchestra.run.smithi055:> cat /proc/self/mounts | awk '{print $2}' 2024-03-19T01:51:27.055 INFO:teuthology.orchestra.run.smithi055.stdout:/sys 2024-03-19T01:51:27.055 INFO:teuthology.orchestra.run.smithi055.stdout:/proc 2024-03-19T01:51:27.055 INFO:teuthology.orchestra.run.smithi055.stdout:/dev 2024-03-19T01:51:27.055 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/pts 2024-03-19T01:51:27.055 INFO:teuthology.orchestra.run.smithi055.stdout:/run 2024-03-19T01:51:27.055 INFO:teuthology.orchestra.run.smithi055.stdout:/ 2024-03-19T01:51:27.055 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/security 2024-03-19T01:51:27.055 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/shm 2024-03-19T01:51:27.055 INFO:teuthology.orchestra.run.smithi055.stdout:/run/lock 2024-03-19T01:51:27.055 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/cgroup 2024-03-19T01:51:27.055 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/pstore 2024-03-19T01:51:27.056 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/bpf 2024-03-19T01:51:27.056 INFO:teuthology.orchestra.run.smithi055.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T01:51:27.056 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/hugepages 2024-03-19T01:51:27.056 INFO:teuthology.orchestra.run.smithi055.stdout:/dev/mqueue 2024-03-19T01:51:27.056 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/debug 2024-03-19T01:51:27.056 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/tracing 2024-03-19T01:51:27.056 INFO:teuthology.orchestra.run.smithi055.stdout:/proc/fs/nfsd 2024-03-19T01:51:27.056 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/fs/fuse/connections 2024-03-19T01:51:27.056 INFO:teuthology.orchestra.run.smithi055.stdout:/sys/kernel/config 2024-03-19T01:51:27.056 INFO:teuthology.orchestra.run.smithi055.stdout:/run/credentials/systemd-sysusers.service 2024-03-19T01:51:27.056 INFO:teuthology.orchestra.run.smithi055.stdout:/run/rpc_pipefs 2024-03-19T01:51:27.056 INFO:teuthology.orchestra.run.smithi055.stdout:/run/user/1000 2024-03-19T01:51:27.056 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph 2024-03-19T01:51:27.057 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-0 2024-03-19T01:51:27.057 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-1 2024-03-19T01:51:27.057 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-2 2024-03-19T01:51:27.057 INFO:teuthology.orchestra.run.smithi055.stdout:/var/lib/ceph/osd/ceph-3 2024-03-19T01:51:27.057 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns 2024-03-19T01:51:27.057 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T01:51:27.057 INFO:teuthology.orchestra.run.smithi055.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T01:51:27.057 INFO:teuthology.orchestra.run.smithi055.stdout:/home/ubuntu/cephtest/mnt.admin 2024-03-19T01:51:27.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:51:27.057 DEBUG:teuthology.orchestra.run.smithi055:> ls /sys/fs/fuse/connections 2024-03-19T01:51:27.101 INFO:teuthology.orchestra.run.smithi055.stdout:45 2024-03-19T01:51:27.101 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-03-19T01:51:27.101 DEBUG:teuthology.orchestra.run.smithi055:> sudo stdin-killer -- python3 -c ' 2024-03-19T01:51:27.102 DEBUG:teuthology.orchestra.run.smithi055:> import glob 2024-03-19T01:51:27.102 DEBUG:teuthology.orchestra.run.smithi055:> import re 2024-03-19T01:51:27.102 DEBUG:teuthology.orchestra.run.smithi055:> import os 2024-03-19T01:51:27.102 DEBUG:teuthology.orchestra.run.smithi055:> import subprocess 2024-03-19T01:51:27.102 DEBUG:teuthology.orchestra.run.smithi055:> 2024-03-19T01:51:27.102 DEBUG:teuthology.orchestra.run.smithi055:> def _find_admin_socket(client_name): 2024-03-19T01:51:27.102 DEBUG:teuthology.orchestra.run.smithi055:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-19T01:51:27.102 DEBUG:teuthology.orchestra.run.smithi055:> files = glob.glob(asok_path) 2024-03-19T01:51:27.102 DEBUG:teuthology.orchestra.run.smithi055:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-19T01:51:27.102 DEBUG:teuthology.orchestra.run.smithi055:> 2024-03-19T01:51:27.102 DEBUG:teuthology.orchestra.run.smithi055:> # Given a non-glob path, it better be there 2024-03-19T01:51:27.102 DEBUG:teuthology.orchestra.run.smithi055:> if "*" not in asok_path: 2024-03-19T01:51:27.102 DEBUG:teuthology.orchestra.run.smithi055:> assert(len(files) == 1) 2024-03-19T01:51:27.102 DEBUG:teuthology.orchestra.run.smithi055:> return files[0] 2024-03-19T01:51:27.103 DEBUG:teuthology.orchestra.run.smithi055:> 2024-03-19T01:51:27.103 DEBUG:teuthology.orchestra.run.smithi055:> for f in files: 2024-03-19T01:51:27.103 DEBUG:teuthology.orchestra.run.smithi055:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-19T01:51:27.103 DEBUG:teuthology.orchestra.run.smithi055:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-19T01:51:27.103 DEBUG:teuthology.orchestra.run.smithi055:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-19T01:51:27.103 DEBUG:teuthology.orchestra.run.smithi055:> contents = proc_f.read() 2024-03-19T01:51:27.103 DEBUG:teuthology.orchestra.run.smithi055:> if mountpoint in contents: 2024-03-19T01:51:27.103 DEBUG:teuthology.orchestra.run.smithi055:> return f 2024-03-19T01:51:27.103 DEBUG:teuthology.orchestra.run.smithi055:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-19T01:51:27.103 DEBUG:teuthology.orchestra.run.smithi055:> 2024-03-19T01:51:27.103 DEBUG:teuthology.orchestra.run.smithi055:> print(_find_admin_socket("client.admin")) 2024-03-19T01:51:27.103 DEBUG:teuthology.orchestra.run.smithi055:> ' 2024-03-19T01:51:27.224 INFO:teuthology.orchestra.run.smithi055.stdout:/var/run/ceph/ceph-client.admin.33232.asok 2024-03-19T01:51:27.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:51:27.236 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.33232.asok 2024-03-19T01:51:27.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:51:27.236 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.33232.asok status 2024-03-19T01:51:27.395 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-19T01:51:27.395 INFO:teuthology.orchestra.run.smithi055.stdout: "metadata": { 2024-03-19T01:51:27.395 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_sha1": "b6ae24918b03b1b8a19c9263a50655bba80bb810", 2024-03-19T01:51:27.396 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-19T01:51:27.396 INFO:teuthology.orchestra.run.smithi055.stdout: "entity_id": "admin", 2024-03-19T01:51:27.396 INFO:teuthology.orchestra.run.smithi055.stdout: "hostname": "smithi055", 2024-03-19T01:51:27.396 INFO:teuthology.orchestra.run.smithi055.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-19T01:51:27.396 INFO:teuthology.orchestra.run.smithi055.stdout: "pid": "33232", 2024-03-19T01:51:27.396 INFO:teuthology.orchestra.run.smithi055.stdout: "root": "/" 2024-03-19T01:51:27.396 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-19T01:51:27.396 INFO:teuthology.orchestra.run.smithi055.stdout: "dentry_count": 0, 2024-03-19T01:51:27.396 INFO:teuthology.orchestra.run.smithi055.stdout: "dentry_pinned_count": 0, 2024-03-19T01:51:27.396 INFO:teuthology.orchestra.run.smithi055.stdout: "id": 4326, 2024-03-19T01:51:27.396 INFO:teuthology.orchestra.run.smithi055.stdout: "inst": { 2024-03-19T01:51:27.396 INFO:teuthology.orchestra.run.smithi055.stdout: "name": { 2024-03-19T01:51:27.396 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "client", 2024-03-19T01:51:27.396 INFO:teuthology.orchestra.run.smithi055.stdout: "num": 4326 2024-03-19T01:51:27.397 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-19T01:51:27.397 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": { 2024-03-19T01:51:27.397 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v1", 2024-03-19T01:51:27.397 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "192.168.0.1:0", 2024-03-19T01:51:27.397 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 2216541051 2024-03-19T01:51:27.397 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-19T01:51:27.397 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-19T01:51:27.397 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": { 2024-03-19T01:51:27.397 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v1", 2024-03-19T01:51:27.397 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "192.168.0.1:0", 2024-03-19T01:51:27.397 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 2216541051 2024-03-19T01:51:27.397 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-19T01:51:27.397 INFO:teuthology.orchestra.run.smithi055.stdout: "inst_str": "client.4326 192.168.0.1:0/2216541051", 2024-03-19T01:51:27.397 INFO:teuthology.orchestra.run.smithi055.stdout: "addr_str": "192.168.0.1:0/2216541051", 2024-03-19T01:51:27.397 INFO:teuthology.orchestra.run.smithi055.stdout: "inode_count": 1, 2024-03-19T01:51:27.398 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_epoch": 8, 2024-03-19T01:51:27.398 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_epoch": 29, 2024-03-19T01:51:27.398 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_epoch_barrier": 0, 2024-03-19T01:51:27.398 INFO:teuthology.orchestra.run.smithi055.stdout: "blocklisted": false, 2024-03-19T01:51:27.398 INFO:teuthology.orchestra.run.smithi055.stdout: "fs_name": "cephfs" 2024-03-19T01:51:27.398 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-19T01:51:27.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:51:27.410 DEBUG:teuthology.orchestra.run.smithi055:> stat --file-system '--printf=%T 2024-03-19T01:51:27.410 DEBUG:teuthology.orchestra.run.smithi055:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-19T01:51:27.435 INFO:teuthology.orchestra.run.smithi055.stdout:fuseblk 2024-03-19T01:51:27.436 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-19T01:51:27.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:51:27.436 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-19T01:51:27.609 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-19T01:51:27.713 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:27 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-19T01:51:28.009 INFO:teuthology.orchestra.run.smithi055.stdout:# file: . 2024-03-19T01:51:28.010 INFO:teuthology.orchestra.run.smithi055.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-19T01:51:28.010 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:51:28.010 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:51:28.019 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:51:28.019 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T01:51:28.025 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:51:28.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.202+0000 7f054c163640 1 Processor -- start 2024-03-19T01:51:28.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.202+0000 7f054c163640 1 -- start start 2024-03-19T01:51:28.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.202+0000 7f054c163640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0544157e60 0x7f0544158310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:28.203 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.202+0000 7f054c163640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0544158850 con 0x7f05440c9680 2024-03-19T01:51:28.203 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.202+0000 7f054c163640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f054406b3d0 con 0x7f0544152300 2024-03-19T01:51:28.203 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.202+0000 7f054c163640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f054406b6b0 con 0x7f0544157e60 2024-03-19T01:51:28.203 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.202+0000 7f054a700640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0544157e60 0x7f0544158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:28.203 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.202+0000 7f05496fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f05440c9680 0x7f05440c5730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:53214/0 (socket says 172.21.15.55:53214) 2024-03-19T01:51:28.203 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.202+0000 7f05496fe640 1 -- 172.21.15.55:0/849790466 learned_addr learned my addr 172.21.15.55:0/849790466 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:51:28.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.206+0000 7f054a700640 1 -- 172.21.15.55:0/849790466 >> v1:172.21.15.84:6789/0 conn(0x7f0544152300 legacy=0x7f05440c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:28.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.206+0000 7f054a700640 1 -- 172.21.15.55:0/849790466 >> v1:172.21.15.55:6789/0 conn(0x7f05440c9680 legacy=0x7f05440c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:28.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.206+0000 7f054a700640 1 -- 172.21.15.55:0/849790466 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054406bf90 con 0x7f0544157e60 2024-03-19T01:51:28.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.214+0000 7f054a700640 1 --2- 172.21.15.55:0/849790466 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0544157e60 0x7f0544158310 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f053c009d90 tx=0x7f053c01c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d46cef5129d4e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:28.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.222+0000 7f0548efd640 1 -- 172.21.15.55:0/849790466 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053c0b2070 con 0x7f0544157e60 2024-03-19T01:51:28.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.222+0000 7f0548efd640 1 -- 172.21.15.55:0/849790466 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f053c0a5760 con 0x7f0544157e60 2024-03-19T01:51:28.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.222+0000 7f0548efd640 1 -- 172.21.15.55:0/849790466 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053c0a5b30 con 0x7f0544157e60 2024-03-19T01:51:28.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.222+0000 7f054c163640 1 -- 172.21.15.55:0/849790466 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0544157e60 msgr2=0x7f0544158310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:28.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.222+0000 7f054c163640 1 --2- 172.21.15.55:0/849790466 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0544157e60 0x7f0544158310 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f053c009d90 tx=0x7f053c01c8a0 comp rx=0 tx=0).stop 2024-03-19T01:51:28.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.222+0000 7f054c163640 1 -- 172.21.15.55:0/849790466 shutdown_connections 2024-03-19T01:51:28.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.222+0000 7f054c163640 1 --2- 172.21.15.55:0/849790466 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0544157e60 0x7f0544158310 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:28.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.222+0000 7f054c163640 1 -- 172.21.15.55:0/849790466 >> 172.21.15.55:0/849790466 conn(0x7f0544071ff0 msgr2=0x7f0544072400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:28.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.222+0000 7f054c163640 1 -- 172.21.15.55:0/849790466 shutdown_connections 2024-03-19T01:51:28.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.222+0000 7f054c163640 1 -- 172.21.15.55:0/849790466 wait complete. 2024-03-19T01:51:28.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.222+0000 7f054c163640 1 Processor -- start 2024-03-19T01:51:28.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.222+0000 7f054c163640 1 -- start start 2024-03-19T01:51:28.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.222+0000 7f054c163640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05440c9680 0x7f0544150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:28.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.222+0000 7f054c163640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0544152300 0x7f054413c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:28.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.222+0000 7f054c163640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0544157e60 0x7f05441f65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:28.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.222+0000 7f054c163640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f05440c5e40 con 0x7f05440c9680 2024-03-19T01:51:28.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.222+0000 7f054c163640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f05440c9190 con 0x7f0544157e60 2024-03-19T01:51:28.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.222+0000 7f054c163640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f05440c5fc0 con 0x7f0544152300 2024-03-19T01:51:28.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.226+0000 7f054a700640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0544157e60 0x7f05441f65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:28.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.226+0000 7f05496fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0544152300 0x7f054413c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:28.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.226+0000 7f054a700640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0544157e60 0x7f05441f65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:39340/0 (socket says 172.21.15.55:39340) 2024-03-19T01:51:28.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.226+0000 7f054a700640 1 -- 172.21.15.55:0/3483964820 learned_addr learned my addr 172.21.15.55:0/3483964820 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:28.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.234+0000 7f05496fe640 1 -- 172.21.15.55:0/3483964820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0544157e60 msgr2=0x7f05441f65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:28.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.234+0000 7f05496fe640 1 --2- 172.21.15.55:0/3483964820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0544157e60 0x7f05441f65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:28.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.234+0000 7f05496fe640 1 -- 172.21.15.55:0/3483964820 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05440c9680 msgr2=0x7f0544150c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:28.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.234+0000 7f05496fe640 1 --2- 172.21.15.55:0/3483964820 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05440c9680 0x7f0544150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:28.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.234+0000 7f05496fe640 1 -- 172.21.15.55:0/3483964820 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05440c5190 con 0x7f0544152300 2024-03-19T01:51:28.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.242+0000 7f05496fe640 1 --2- 172.21.15.55:0/3483964820 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0544152300 0x7f054413c410 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f05440750c0 tx=0x7f053400c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:28.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.254+0000 7f053a7fc640 1 -- 172.21.15.55:0/3483964820 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0534017070 con 0x7f0544152300 2024-03-19T01:51:28.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.254+0000 7f053a7fc640 1 -- 172.21.15.55:0/3483964820 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0534005cf0 con 0x7f0544152300 2024-03-19T01:51:28.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.254+0000 7f053a7fc640 1 -- 172.21.15.55:0/3483964820 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05340072f0 con 0x7f0544152300 2024-03-19T01:51:28.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.254+0000 7f054c163640 1 -- 172.21.15.55:0/3483964820 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0544150450 con 0x7f0544152300 2024-03-19T01:51:28.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.254+0000 7f054c163640 1 -- 172.21.15.55:0/3483964820 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f05440c5430 con 0x7f0544152300 2024-03-19T01:51:28.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.254+0000 7f054c163640 1 -- 172.21.15.55:0/3483964820 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0544068330 con 0x7f0544152300 2024-03-19T01:51:28.605 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.606+0000 7f053a7fc640 1 -- 172.21.15.55:0/3483964820 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f053401a070 con 0x7f0544152300 2024-03-19T01:51:28.605 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.606+0000 7f053a7fc640 1 --2- 172.21.15.55:0/3483964820 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f05280676b0 0x7f0528069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:28.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.606+0000 7f0549eff640 1 --2- 172.21.15.55:0/3483964820 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f05280676b0 0x7f0528069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:28.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.606+0000 7f0549eff640 1 --2- 172.21.15.55:0/3483964820 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f05280676b0 0x7f0528069b70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0540007fd0 tx=0x7f05400079a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:28.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.618+0000 7f053a7fc640 1 -- 172.21.15.55:0/3483964820 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f05340536a0 con 0x7f0544152300 2024-03-19T01:51:28.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.654+0000 7f053a7fc640 1 -- 172.21.15.55:0/3483964820 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f0534021020 con 0x7f0544152300 2024-03-19T01:51:28.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:28.830+0000 7f054c163640 1 -- 172.21.15.55:0/3483964820 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f05440c9460 con 0x7f0544152300 2024-03-19T01:51:29.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.078+0000 7f053a7fc640 1 -- 172.21.15.55:0/3483964820 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+18430 (secure 0 0 0) 0x7f0534055600 con 0x7f0544152300 2024-03-19T01:51:29.080 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:51:29.080 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":29,"fsid":"def9d4e5-0bc7-449d-b893-2490cbd10e25","created":"2024-03-19T01:50:02.482954+0000","modified":"2024-03-19T01:51:17.228291+0000","last_up_change":"2024-03-19T01:50:21.735508+0000","last_in_change":"2024-03-19T01:50:12.168944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:50:23.456904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:51:04.578583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:51:06.871659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T01:51:14.274442+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"35d5e3c5-d089-4f38-8961-98d064188f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6825","nonce":2757675043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6827","nonce":2757675043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6831","nonce":2757675043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6829","nonce":2757675043}]},"public_addr":"172.21.15.55:6825/2757675043","cluster_addr":"172.21.15.55:6827/2757675043","heartbeat_back_addr":"172.21.15.55:6831/2757675043","heartbeat_front_addr":"172.21.15.55:6829/2757675043","state":["exists","up"]},{"osd":1,"uuid":"8e58f791-001a-44d0-a464-ca68a479f70c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6817","nonce":543460007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6819","nonce":543460007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6823","nonce":543460007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6821","nonce":543460007}]},"public_addr":"172.21.15.55:6817/543460007","cluster_addr":"172.21.15.55:6819/543460007","heartbeat_back_addr":"172.21.15.55:6823/543460007","heartbeat_front_addr":"172.21.15.55:6821/543460007","state":["exists","up"]},{"osd":2,"uuid":"d44f38b8-67f4-43de-98a6-f3c520acaf10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6801","nonce":1760253550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6803","nonce":1760253550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6807","nonce":1760253550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6805","nonce":1760253550}]},"public_addr":"172.21.15.55:6801/1760253550","cluster_addr":"172.21.15.55:6803/1760253550","heartbeat_back_addr":"172.21.15.55:6807/1760253550","heartbeat_front_addr":"172.21.15.55:6805/1760253550","state":["exists","up"]},{"osd":3,"uuid":"850a7316-2d74-43f0-8649-b5f8d8f6692e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6809","nonce":2797616448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6811","nonce":2797616448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6815","nonce":2797616448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6813","nonce":2797616448}]},"public_addr":"172.21.15.55:6809/2797616448","cluster_addr":"172.21.15.55:6811/2797616448","heartbeat_back_addr":"172.21.15.55:6815/2797616448","heartbeat_front_addr":"172.21.15.55:6813/2797616448","state":["exists","up"]},{"osd":4,"uuid":"4e6152ae-a77d-4bb6-a855-5782ad5411a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6819","nonce":2938436826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6821","nonce":2938436826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6825","nonce":2938436826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6823","nonce":2938436826}]},"public_addr":"172.21.15.84:6819/2938436826","cluster_addr":"172.21.15.84:6821/2938436826","heartbeat_back_addr":"172.21.15.84:6825/2938436826","heartbeat_front_addr":"172.21.15.84:6823/2938436826","state":["exists","up"]},{"osd":5,"uuid":"1217ceba-dd31-4615-96f1-dfaf57160f3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6803","nonce":460717295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6805","nonce":460717295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6809","nonce":460717295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6807","nonce":460717295}]},"public_addr":"172.21.15.84:6803/460717295","cluster_addr":"172.21.15.84:6805/460717295","heartbeat_back_addr":"172.21.15.84:6809/460717295","heartbeat_front_addr":"172.21.15.84:6807/460717295","state":["exists","up"]},{"osd":6,"uuid":"3d9ed78a-2284-4c28-a24a-70eb812c7836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6827","nonce":416123135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6829","nonce":416123135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6833","nonce":416123135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6831","nonce":416123135}]},"public_addr":"172.21.15.84:6827/416123135","cluster_addr":"172.21.15.84:6829/416123135","heartbeat_back_addr":"172.21.15.84:6833/416123135","heartbeat_front_addr":"172.21.15.84:6831/416123135","state":["exists","up"]},{"osd":7,"uuid":"c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6811","nonce":460083528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6813","nonce":460083528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6817","nonce":460083528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6815","nonce":460083528}]},"public_addr":"172.21.15.84:6811/460083528","cluster_addr":"172.21.15.84:6813/460083528","heartbeat_back_addr":"172.21.15.84:6817/460083528","heartbeat_front_addr":"172.21.15.84:6815/460083528","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.269126+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.121001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.075714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.092461+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.194676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.062833+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.316515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.237111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:51:29.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.082+0000 7f054c163640 1 -- 172.21.15.55:0/3483964820 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f05280676b0 msgr2=0x7f0528069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:29.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.082+0000 7f054c163640 1 --2- 172.21.15.55:0/3483964820 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f05280676b0 0x7f0528069b70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0540007fd0 tx=0x7f05400079a0 comp rx=0 tx=0).stop 2024-03-19T01:51:29.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.082+0000 7f054c163640 1 -- 172.21.15.55:0/3483964820 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0544152300 msgr2=0x7f054413c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:29.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.082+0000 7f054c163640 1 --2- 172.21.15.55:0/3483964820 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0544152300 0x7f054413c410 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f05440750c0 tx=0x7f053400c820 comp rx=0 tx=0).stop 2024-03-19T01:51:29.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.082+0000 7f054c163640 1 -- 172.21.15.55:0/3483964820 shutdown_connections 2024-03-19T01:51:29.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.082+0000 7f054c163640 1 --2- 172.21.15.55:0/3483964820 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f05280676b0 0x7f0528069b70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:29.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.082+0000 7f054c163640 1 --2- 172.21.15.55:0/3483964820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0544157e60 0x7f05441f65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:29.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.082+0000 7f054c163640 1 --2- 172.21.15.55:0/3483964820 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0544152300 0x7f054413c410 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:29.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.082+0000 7f054c163640 1 --2- 172.21.15.55:0/3483964820 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05440c9680 0x7f0544150c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:29.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.082+0000 7f054c163640 1 -- 172.21.15.55:0/3483964820 >> 172.21.15.55:0/3483964820 conn(0x7f0544071ff0 msgr2=0x7f05440723d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:29.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.082+0000 7f054c163640 1 -- 172.21.15.55:0/3483964820 shutdown_connections 2024-03-19T01:51:29.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.082+0000 7f054c163640 1 -- 172.21.15.55:0/3483964820 wait complete. 2024-03-19T01:51:29.098 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:50:23.456904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:51:04.578583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:51:06.871659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T01:51:14.274442+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T01:51:29.098 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:51:29.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.218+0000 7f0461c30640 1 Processor -- start 2024-03-19T01:51:29.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.218+0000 7f0461c30640 1 -- start start 2024-03-19T01:51:29.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.218+0000 7f0461c30640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f045c0c7610 0x7f045c158080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:29.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.218+0000 7f0461c30640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f045c06b2a0 0x7f045c1585c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:29.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.218+0000 7f0461c30640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f045c0748d0 con 0x7f045c151f90 2024-03-19T01:51:29.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.218+0000 7f0461c30640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f045c0cce40 con 0x7f045c0c7610 2024-03-19T01:51:29.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.218+0000 7f0461c30640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f045c06c740 con 0x7f045c06b2a0 2024-03-19T01:51:29.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.222+0000 7f045bfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f045c06b2a0 0x7f045c1585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:29.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.222+0000 7f045bfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f045c06b2a0 0x7f045c1585c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:55142/0 (socket says 172.21.15.55:55142) 2024-03-19T01:51:29.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.222+0000 7f045bfff640 1 -- 172.21.15.55:0/2485701279 learned_addr learned my addr 172.21.15.55:0/2485701279 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:29.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.226+0000 7f045affd640 1 --2- 172.21.15.55:0/2485701279 >> v2:172.21.15.84:3300/0 conn(0x7f045c0c7610 0x7f045c158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:29.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.226+0000 7f045bfff640 1 -- 172.21.15.55:0/2485701279 >> v2:172.21.15.84:3300/0 conn(0x7f045c0c7610 msgr2=0x7f045c158080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:29.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.226+0000 7f045bfff640 1 --2- 172.21.15.55:0/2485701279 >> v2:172.21.15.84:3300/0 conn(0x7f045c0c7610 0x7f045c158080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:29.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.226+0000 7f045bfff640 1 -- 172.21.15.55:0/2485701279 >> v1:172.21.15.84:6789/0 conn(0x7f045c151f90 legacy=0x7f045c0c5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:29.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.226+0000 7f045bfff640 1 -- 172.21.15.55:0/2485701279 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045c0745f0 con 0x7f045c06b2a0 2024-03-19T01:51:29.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.234+0000 7f045bfff640 1 --2- 172.21.15.55:0/2485701279 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f045c06b2a0 0x7f045c1585c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0454009e50 tx=0x7f045401c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99f7fd9ae8780948 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:29.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.242+0000 7f045a7fc640 1 -- 172.21.15.55:0/2485701279 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04540a5760 con 0x7f045c06b2a0 2024-03-19T01:51:29.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.242+0000 7f045a7fc640 1 -- 172.21.15.55:0/2485701279 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04540a5900 con 0x7f045c06b2a0 2024-03-19T01:51:29.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.242+0000 7f045a7fc640 1 -- 172.21.15.55:0/2485701279 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04540a5cd0 con 0x7f045c06b2a0 2024-03-19T01:51:29.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.242+0000 7f0461c30640 1 -- 172.21.15.55:0/2485701279 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f045c06b2a0 msgr2=0x7f045c1585c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:29.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.242+0000 7f0461c30640 1 --2- 172.21.15.55:0/2485701279 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f045c06b2a0 0x7f045c1585c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0454009e50 tx=0x7f045401c9f0 comp rx=0 tx=0).stop 2024-03-19T01:51:29.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.242+0000 7f0461c30640 1 -- 172.21.15.55:0/2485701279 shutdown_connections 2024-03-19T01:51:29.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.242+0000 7f0461c30640 1 --2- 172.21.15.55:0/2485701279 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f045c06b2a0 0x7f045c1585c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:29.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.242+0000 7f0461c30640 1 --2- 172.21.15.55:0/2485701279 >> v2:172.21.15.84:3300/0 conn(0x7f045c0c7610 0x7f045c158080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:29.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.242+0000 7f0461c30640 1 -- 172.21.15.55:0/2485701279 >> 172.21.15.55:0/2485701279 conn(0x7f045c0cc200 msgr2=0x7f045c0cc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:29.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.242+0000 7f0461c30640 1 -- 172.21.15.55:0/2485701279 shutdown_connections 2024-03-19T01:51:29.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.242+0000 7f0461c30640 1 -- 172.21.15.55:0/2485701279 wait complete. 2024-03-19T01:51:29.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.246+0000 7f0461c30640 1 Processor -- start 2024-03-19T01:51:29.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.246+0000 7f0461c30640 1 -- start start 2024-03-19T01:51:29.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.246+0000 7f0461c30640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f045c06b2a0 0x7f045c156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:29.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.246+0000 7f0461c30640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f045c151f90 0x7f045c14a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:29.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.246+0000 7f0461c30640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f045c1625e0 0x7f045c1f64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:29.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.246+0000 7f0461c30640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f045c152370 con 0x7f045c06b2a0 2024-03-19T01:51:29.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.246+0000 7f0461c30640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f045c074290 con 0x7f045c1625e0 2024-03-19T01:51:29.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.246+0000 7f0461c30640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f045c1524f0 con 0x7f045c151f90 2024-03-19T01:51:29.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.246+0000 7f045b7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f045c06b2a0 0x7f045c156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:29.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.246+0000 7f045b7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f045c06b2a0 0x7f045c156bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50568/0 (socket says 172.21.15.55:50568) 2024-03-19T01:51:29.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.246+0000 7f045b7fe640 1 -- 172.21.15.55:0/618884464 learned_addr learned my addr 172.21.15.55:0/618884464 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:29.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.250+0000 7f045affd640 1 --2- 172.21.15.55:0/618884464 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f045c151f90 0x7f045c14a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:29.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.250+0000 7f045b7fe640 1 -- 172.21.15.55:0/618884464 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f045c151f90 msgr2=0x7f045c14a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:29.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.250+0000 7f045b7fe640 1 --2- 172.21.15.55:0/618884464 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f045c151f90 0x7f045c14a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:29.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.250+0000 7f045b7fe640 1 -- 172.21.15.55:0/618884464 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f045c1625e0 msgr2=0x7f045c1f64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:29.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.250+0000 7f045b7fe640 1 --2- 172.21.15.55:0/618884464 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f045c1625e0 0x7f045c1f64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:29.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.250+0000 7f045b7fe640 1 -- 172.21.15.55:0/618884464 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045c0cabc0 con 0x7f045c06b2a0 2024-03-19T01:51:29.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.254+0000 7f045b7fe640 1 --2- 172.21.15.55:0/618884464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f045c06b2a0 0x7f045c156bc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f044c00b3e0 tx=0x7f044c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:29.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.258+0000 7f0443fff640 1 -- 172.21.15.55:0/618884464 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044c015020 con 0x7f045c06b2a0 2024-03-19T01:51:29.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.258+0000 7f0443fff640 1 -- 172.21.15.55:0/618884464 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f044c011850 con 0x7f045c06b2a0 2024-03-19T01:51:29.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.258+0000 7f0461c30640 1 -- 172.21.15.55:0/618884464 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f045c0cae60 con 0x7f045c06b2a0 2024-03-19T01:51:29.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.258+0000 7f0443fff640 1 -- 172.21.15.55:0/618884464 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044c011c40 con 0x7f045c06b2a0 2024-03-19T01:51:29.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.258+0000 7f0461c30640 1 -- 172.21.15.55:0/618884464 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f045c0cb430 con 0x7f045c06b2a0 2024-03-19T01:51:29.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.258+0000 7f0461c30640 1 -- 172.21.15.55:0/618884464 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f045c068330 con 0x7f045c06b2a0 2024-03-19T01:51:29.591 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.590+0000 7f0443fff640 1 -- 172.21.15.55:0/618884464 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f044c018020 con 0x7f045c06b2a0 2024-03-19T01:51:29.591 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.590+0000 7f0443fff640 1 --2- 172.21.15.55:0/618884464 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f04480676e0 0x7f0448069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:29.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.590+0000 7f045affd640 1 --2- 172.21.15.55:0/618884464 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f04480676e0 0x7f0448069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:29.593 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.594+0000 7f045affd640 1 --2- 172.21.15.55:0/618884464 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f04480676e0 0x7f0448069ba0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f045c14d450 tx=0x7f0450009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:29.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.618+0000 7f0443fff640 1 -- 172.21.15.55:0/618884464 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f044c011de0 con 0x7f045c06b2a0 2024-03-19T01:51:29.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.618+0000 7f0443fff640 1 -- 172.21.15.55:0/618884464 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f044c01f020 con 0x7f045c06b2a0 2024-03-19T01:51:29.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.794+0000 7f0461c30640 1 -- 172.21.15.55:0/618884464 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f045c150b20 con 0x7f045c06b2a0 2024-03-19T01:51:29.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.826+0000 7f0443fff640 1 -- 172.21.15.55:0/618884464 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) ==== 93+0+10 (secure 0 0 0) 0x7f044c056a00 con 0x7f045c06b2a0 2024-03-19T01:51:29.825 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 1 2024-03-19T01:51:29.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.826+0000 7f0461c30640 1 -- 172.21.15.55:0/618884464 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f04480676e0 msgr2=0x7f0448069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:29.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.826+0000 7f0461c30640 1 --2- 172.21.15.55:0/618884464 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f04480676e0 0x7f0448069ba0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f045c14d450 tx=0x7f0450009190 comp rx=0 tx=0).stop 2024-03-19T01:51:29.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.826+0000 7f0461c30640 1 -- 172.21.15.55:0/618884464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f045c06b2a0 msgr2=0x7f045c156bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:29.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.826+0000 7f0461c30640 1 --2- 172.21.15.55:0/618884464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f045c06b2a0 0x7f045c156bc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f044c00b3e0 tx=0x7f044c00b8a0 comp rx=0 tx=0).stop 2024-03-19T01:51:29.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.826+0000 7f0461c30640 1 -- 172.21.15.55:0/618884464 shutdown_connections 2024-03-19T01:51:29.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.826+0000 7f0461c30640 1 --2- 172.21.15.55:0/618884464 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f04480676e0 0x7f0448069ba0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:29.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.826+0000 7f0461c30640 1 --2- 172.21.15.55:0/618884464 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f045c1625e0 0x7f045c1f64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:29.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.826+0000 7f0461c30640 1 --2- 172.21.15.55:0/618884464 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f045c151f90 0x7f045c14a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:29.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.826+0000 7f0461c30640 1 --2- 172.21.15.55:0/618884464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f045c06b2a0 0x7f045c156bc0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:29.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.826+0000 7f0461c30640 1 -- 172.21.15.55:0/618884464 >> 172.21.15.55:0/618884464 conn(0x7f045c0cc200 msgr2=0x7f045c0c5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:29.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.826+0000 7f0461c30640 1 -- 172.21.15.55:0/618884464 shutdown_connections 2024-03-19T01:51:29.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.826+0000 7f0461c30640 1 -- 172.21.15.55:0/618884464 wait complete. 2024-03-19T01:51:29.843 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:51:29.961 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.962+0000 7fc2c02e0640 1 Processor -- start 2024-03-19T01:51:29.961 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.962+0000 7fc2c02e0640 1 -- start start 2024-03-19T01:51:29.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.962+0000 7fc2c02e0640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc2b806f1b0 0x7fc2b80c5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:29.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.962+0000 7fc2c02e0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc2b80c5d50 0x7fc2b8158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:29.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.962+0000 7fc2c02e0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2b815b740 con 0x7fc2b813de70 2024-03-19T01:51:29.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.962+0000 7fc2c02e0640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fc2b813e250 con 0x7fc2b806f1b0 2024-03-19T01:51:29.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.962+0000 7fc2c02e0640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc2b815ba20 con 0x7fc2b80c5d50 2024-03-19T01:51:29.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.962+0000 7fc2bd87b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc2b806f1b0 0x7fc2b80c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:29.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.962+0000 7fc2bd87b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc2b806f1b0 0x7fc2b80c5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50576/0 (socket says 172.21.15.55:50576) 2024-03-19T01:51:29.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.962+0000 7fc2bd87b640 1 -- 172.21.15.55:0/2724584057 learned_addr learned my addr 172.21.15.55:0/2724584057 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:29.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.962+0000 7fc2be87d640 1 --2- 172.21.15.55:0/2724584057 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc2b80c5d50 0x7fc2b8158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:29.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.966+0000 7fc2bd87b640 1 -- 172.21.15.55:0/2724584057 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc2b80c5d50 msgr2=0x7fc2b8158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:29.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.966+0000 7fc2bd87b640 1 --2- 172.21.15.55:0/2724584057 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc2b80c5d50 0x7fc2b8158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:29.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.966+0000 7fc2bd87b640 1 -- 172.21.15.55:0/2724584057 >> v1:172.21.15.84:6789/0 conn(0x7fc2b813de70 legacy=0x7fc2b806e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:29.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.966+0000 7fc2bd87b640 1 -- 172.21.15.55:0/2724584057 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b815bd00 con 0x7fc2b806f1b0 2024-03-19T01:51:29.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.970+0000 7fc2bd87b640 1 --2- 172.21.15.55:0/2724584057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2b806f1b0 0x7fc2b80c5810 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac009e50 tx=0x7fc2ac01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f674288cae93f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:29.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.974+0000 7fc2bd07a640 1 -- 172.21.15.55:0/2724584057 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac0a5810 con 0x7fc2b806f1b0 2024-03-19T01:51:29.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.974+0000 7fc2bd07a640 1 -- 172.21.15.55:0/2724584057 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2ac0a59b0 con 0x7fc2b806f1b0 2024-03-19T01:51:29.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.974+0000 7fc2bd07a640 1 -- 172.21.15.55:0/2724584057 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac0a5d80 con 0x7fc2b806f1b0 2024-03-19T01:51:29.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.974+0000 7fc2c02e0640 1 -- 172.21.15.55:0/2724584057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2b806f1b0 msgr2=0x7fc2b80c5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:29.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.974+0000 7fc2c02e0640 1 --2- 172.21.15.55:0/2724584057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2b806f1b0 0x7fc2b80c5810 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac009e50 tx=0x7fc2ac01ca30 comp rx=0 tx=0).stop 2024-03-19T01:51:29.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.974+0000 7fc2c02e0640 1 -- 172.21.15.55:0/2724584057 shutdown_connections 2024-03-19T01:51:29.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.974+0000 7fc2c02e0640 1 --2- 172.21.15.55:0/2724584057 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc2b80c5d50 0x7fc2b8158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:29.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.974+0000 7fc2c02e0640 1 --2- 172.21.15.55:0/2724584057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2b806f1b0 0x7fc2b80c5810 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:29.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.974+0000 7fc2c02e0640 1 -- 172.21.15.55:0/2724584057 >> 172.21.15.55:0/2724584057 conn(0x7fc2b806fb40 msgr2=0x7fc2b806ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:29.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.974+0000 7fc2c02e0640 1 -- 172.21.15.55:0/2724584057 shutdown_connections 2024-03-19T01:51:29.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.974+0000 7fc2c02e0640 1 -- 172.21.15.55:0/2724584057 wait complete. 2024-03-19T01:51:29.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.978+0000 7fc2c02e0640 1 Processor -- start 2024-03-19T01:51:29.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.978+0000 7fc2c02e0640 1 -- start start 2024-03-19T01:51:29.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.978+0000 7fc2c02e0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc2b806f1b0 0x7fc2b813f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:29.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.978+0000 7fc2c02e0640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2b80c5d50 0x7fc2b813f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:29.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.978+0000 7fc2c02e0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2b813de70 0x7fc2b8150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:29.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.978+0000 7fc2c02e0640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b8165170 con 0x7fc2b813de70 2024-03-19T01:51:29.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.978+0000 7fc2c02e0640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b8158d70 con 0x7fc2b80c5d50 2024-03-19T01:51:29.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.978+0000 7fc2c02e0640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc2b81652f0 con 0x7fc2b806f1b0 2024-03-19T01:51:29.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.978+0000 7fc2be87d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2b813de70 0x7fc2b8150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:29.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.978+0000 7fc2be87d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2b813de70 0x7fc2b8150fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50580/0 (socket says 172.21.15.55:50580) 2024-03-19T01:51:29.979 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.978+0000 7fc2be87d640 1 -- 172.21.15.55:0/1987017730 learned_addr learned my addr 172.21.15.55:0/1987017730 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:29.980 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.978+0000 7fc2bd87b640 1 --2- 172.21.15.55:0/1987017730 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2b80c5d50 0x7fc2b813f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:29.980 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.978+0000 7fc2be07c640 1 --2- 172.21.15.55:0/1987017730 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc2b806f1b0 0x7fc2b813f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:29.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.982+0000 7fc2be87d640 1 -- 172.21.15.55:0/1987017730 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc2b806f1b0 msgr2=0x7fc2b813f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:29.983 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.982+0000 7fc2be87d640 1 --2- 172.21.15.55:0/1987017730 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc2b806f1b0 0x7fc2b813f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:29.983 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.982+0000 7fc2be87d640 1 -- 172.21.15.55:0/1987017730 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2b80c5d50 msgr2=0x7fc2b813f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:29.983 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.982+0000 7fc2be87d640 1 --2- 172.21.15.55:0/1987017730 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2b80c5d50 0x7fc2b813f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:29.983 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.982+0000 7fc2be87d640 1 -- 172.21.15.55:0/1987017730 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b815b360 con 0x7fc2b813de70 2024-03-19T01:51:29.984 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.986+0000 7fc2be87d640 1 --2- 172.21.15.55:0/1987017730 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2b813de70 0x7fc2b8150fa0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc2a800d560 tx=0x7fc2a800da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:29.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.990+0000 7fc2b6ffd640 1 -- 172.21.15.55:0/1987017730 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a80125f0 con 0x7fc2b813de70 2024-03-19T01:51:29.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.990+0000 7fc2b6ffd640 1 -- 172.21.15.55:0/1987017730 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2a8019070 con 0x7fc2b813de70 2024-03-19T01:51:29.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.990+0000 7fc2c02e0640 1 -- 172.21.15.55:0/1987017730 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2b8159b20 con 0x7fc2b813de70 2024-03-19T01:51:29.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.990+0000 7fc2b6ffd640 1 -- 172.21.15.55:0/1987017730 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a80129f0 con 0x7fc2b813de70 2024-03-19T01:51:29.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.990+0000 7fc2c02e0640 1 -- 172.21.15.55:0/1987017730 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2b81f1f10 con 0x7fc2b813de70 2024-03-19T01:51:29.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:29.990+0000 7fc2c02e0640 1 -- 172.21.15.55:0/1987017730 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2b8068a40 con 0x7fc2b813de70 2024-03-19T01:51:30.283 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.282+0000 7fc2b6ffd640 1 -- 172.21.15.55:0/1987017730 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc2a8012b90 con 0x7fc2b813de70 2024-03-19T01:51:30.283 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.282+0000 7fc2b6ffd640 1 --2- 172.21.15.55:0/1987017730 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc2a4067660 0x7fc2a4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:30.284 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.282+0000 7fc2be07c640 1 --2- 172.21.15.55:0/1987017730 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc2a4067660 0x7fc2a4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:30.285 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.286+0000 7fc2be07c640 1 --2- 172.21.15.55:0/1987017730 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc2a4067660 0x7fc2a4069b20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0007fd0 tx=0x7fc2b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:30.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.310+0000 7fc2b6ffd640 1 -- 172.21.15.55:0/1987017730 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fc2a8054d50 con 0x7fc2b813de70 2024-03-19T01:51:30.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.310+0000 7fc2b6ffd640 1 -- 172.21.15.55:0/1987017730 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc2a8056cb0 con 0x7fc2b813de70 2024-03-19T01:51:30.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.494+0000 7fc2c02e0640 1 -- 172.21.15.55:0/1987017730 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc2b8152220 con 0x7fc2b813de70 2024-03-19T01:51:30.585 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.586+0000 7fc2b6ffd640 1 -- 172.21.15.55:0/1987017730 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) ==== 104+0+11 (secure 0 0 0) 0x7fc2a805ad50 con 0x7fc2b813de70 2024-03-19T01:51:30.585 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 32 2024-03-19T01:51:30.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.586+0000 7fc2c02e0640 1 -- 172.21.15.55:0/1987017730 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc2a4067660 msgr2=0x7fc2a4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:30.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.586+0000 7fc2c02e0640 1 --2- 172.21.15.55:0/1987017730 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc2a4067660 0x7fc2a4069b20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0007fd0 tx=0x7fc2b0009040 comp rx=0 tx=0).stop 2024-03-19T01:51:30.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.586+0000 7fc2c02e0640 1 -- 172.21.15.55:0/1987017730 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2b813de70 msgr2=0x7fc2b8150fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:30.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.586+0000 7fc2c02e0640 1 --2- 172.21.15.55:0/1987017730 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2b813de70 0x7fc2b8150fa0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc2a800d560 tx=0x7fc2a800da20 comp rx=0 tx=0).stop 2024-03-19T01:51:30.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.586+0000 7fc2c02e0640 1 -- 172.21.15.55:0/1987017730 shutdown_connections 2024-03-19T01:51:30.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.586+0000 7fc2c02e0640 1 --2- 172.21.15.55:0/1987017730 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc2a4067660 0x7fc2a4069b20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:30.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.586+0000 7fc2c02e0640 1 --2- 172.21.15.55:0/1987017730 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2b813de70 0x7fc2b8150fa0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:30.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.586+0000 7fc2c02e0640 1 --2- 172.21.15.55:0/1987017730 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2b80c5d50 0x7fc2b813f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:30.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.586+0000 7fc2c02e0640 1 --2- 172.21.15.55:0/1987017730 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc2b806f1b0 0x7fc2b813f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:30.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.586+0000 7fc2c02e0640 1 -- 172.21.15.55:0/1987017730 >> 172.21.15.55:0/1987017730 conn(0x7fc2b806fb40 msgr2=0x7fc2b806e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:30.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.586+0000 7fc2c02e0640 1 -- 172.21.15.55:0/1987017730 shutdown_connections 2024-03-19T01:51:30.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.586+0000 7fc2c02e0640 1 -- 172.21.15.55:0/1987017730 wait complete. 2024-03-19T01:51:30.604 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:51:30.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.722+0000 7f9416abc640 1 Processor -- start 2024-03-19T01:51:30.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.722+0000 7f9416abc640 1 -- start start 2024-03-19T01:51:30.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.722+0000 7f9416abc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f941014eaa0 0x7f941006b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:30.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.722+0000 7f9416abc640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f941006bec0 0x7f94100c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:30.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.722+0000 7f9416abc640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f941014ee80 con 0x7f94100c6610 2024-03-19T01:51:30.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.722+0000 7f9416abc640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f94100c7540 con 0x7f941006bec0 2024-03-19T01:51:30.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.722+0000 7f9416abc640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f94100c77c0 con 0x7f941014eaa0 2024-03-19T01:51:30.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.722+0000 7f940ffff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f941006bec0 0x7f94100c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:30.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.726+0000 7f940ffff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f941006bec0 0x7f94100c60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50582/0 (socket says 172.21.15.55:50582) 2024-03-19T01:51:30.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.726+0000 7f940ffff640 1 -- 172.21.15.55:0/384149241 learned_addr learned my addr 172.21.15.55:0/384149241 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:30.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.730+0000 7f940ffff640 1 -- 172.21.15.55:0/384149241 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f941014eaa0 msgr2=0x7f941006b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:30.729 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.730+0000 7f940ffff640 1 --2- 172.21.15.55:0/384149241 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f941014eaa0 0x7f941006b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:30.729 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.730+0000 7f940ffff640 1 -- 172.21.15.55:0/384149241 >> v1:172.21.15.55:6789/0 conn(0x7f94100c6610 legacy=0x7f94101585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:30.729 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.730+0000 7f940ffff640 1 -- 172.21.15.55:0/384149241 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94100c7aa0 con 0x7f941006bec0 2024-03-19T01:51:30.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.734+0000 7f940ffff640 1 --2- 172.21.15.55:0/384149241 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f941006bec0 0x7f94100c60d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f93fc009870 tx=0x7f93fc01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27e08031e87f50a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:30.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.738+0000 7f940f7fe640 1 -- 172.21.15.55:0/384149241 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93fc0ab650 con 0x7f941006bec0 2024-03-19T01:51:30.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.738+0000 7f940f7fe640 1 -- 172.21.15.55:0/384149241 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93fc0b0070 con 0x7f941006bec0 2024-03-19T01:51:30.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.738+0000 7f940f7fe640 1 -- 172.21.15.55:0/384149241 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93fc0aba10 con 0x7f941006bec0 2024-03-19T01:51:30.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.738+0000 7f9416abc640 1 -- 172.21.15.55:0/384149241 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f941006bec0 msgr2=0x7f94100c60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:30.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.738+0000 7f9416abc640 1 --2- 172.21.15.55:0/384149241 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f941006bec0 0x7f94100c60d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f93fc009870 tx=0x7f93fc01c490 comp rx=0 tx=0).stop 2024-03-19T01:51:30.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.738+0000 7f9416abc640 1 -- 172.21.15.55:0/384149241 shutdown_connections 2024-03-19T01:51:30.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.738+0000 7f9416abc640 1 --2- 172.21.15.55:0/384149241 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f941006bec0 0x7f94100c60d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:30.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.738+0000 7f9416abc640 1 --2- 172.21.15.55:0/384149241 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f941014eaa0 0x7f941006b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:30.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.738+0000 7f9416abc640 1 -- 172.21.15.55:0/384149241 >> 172.21.15.55:0/384149241 conn(0x7f94100cc9c0 msgr2=0x7f94100ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:30.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.738+0000 7f9416abc640 1 -- 172.21.15.55:0/384149241 shutdown_connections 2024-03-19T01:51:30.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.738+0000 7f9416abc640 1 -- 172.21.15.55:0/384149241 wait complete. 2024-03-19T01:51:30.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.742+0000 7f9416abc640 1 Processor -- start 2024-03-19T01:51:30.742 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.742+0000 7f9416abc640 1 -- start start 2024-03-19T01:51:30.742 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.742+0000 7f9416abc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f941006bec0 0x7f9410155e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:30.742 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.742+0000 7f9416abc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f94100c6610 0x7f9410140d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:30.742 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.742+0000 7f9416abc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f941014eaa0 0x7f9410146550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:30.742 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.742+0000 7f9416abc640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f94100c6d70 con 0x7f941014eaa0 2024-03-19T01:51:30.742 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.742+0000 7f9416abc640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f94100c6bf0 con 0x7f941006bec0 2024-03-19T01:51:30.742 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.742+0000 7f9416abc640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f9410158d00 con 0x7f94100c6610 2024-03-19T01:51:30.743 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.742+0000 7f9415059640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f941014eaa0 0x7f9410146550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:30.743 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.742+0000 7f9415059640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f941014eaa0 0x7f9410146550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50590/0 (socket says 172.21.15.55:50590) 2024-03-19T01:51:30.743 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.742+0000 7f9415059640 1 -- 172.21.15.55:0/2563909246 learned_addr learned my addr 172.21.15.55:0/2563909246 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:30.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.746+0000 7f9415059640 1 -- 172.21.15.55:0/2563909246 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f94100c6610 msgr2=0x7f9410140d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:30.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.746+0000 7f9415059640 1 --2- 172.21.15.55:0/2563909246 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f94100c6610 0x7f9410140d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:30.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.746+0000 7f9415059640 1 -- 172.21.15.55:0/2563909246 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f941006bec0 msgr2=0x7f9410155e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:30.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.746+0000 7f9415059640 1 --2- 172.21.15.55:0/2563909246 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f941006bec0 0x7f9410155e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:30.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.746+0000 7f9415059640 1 -- 172.21.15.55:0/2563909246 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9410157b80 con 0x7f941014eaa0 2024-03-19T01:51:30.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.750+0000 7f9415059640 1 --2- 172.21.15.55:0/2563909246 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f941014eaa0 0x7f9410146550 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f940800b970 tx=0x7f940800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:30.754 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.754+0000 7f940d7fa640 1 -- 172.21.15.55:0/2563909246 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9408019070 con 0x7f941014eaa0 2024-03-19T01:51:30.754 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.754+0000 7f940d7fa640 1 -- 172.21.15.55:0/2563909246 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f940800cb40 con 0x7f941014eaa0 2024-03-19T01:51:30.754 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.754+0000 7f940d7fa640 1 -- 172.21.15.55:0/2563909246 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9408014440 con 0x7f941014eaa0 2024-03-19T01:51:30.754 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.754+0000 7f9416abc640 1 -- 172.21.15.55:0/2563909246 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94100cbe10 con 0x7f941014eaa0 2024-03-19T01:51:30.754 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.754+0000 7f9416abc640 1 -- 172.21.15.55:0/2563909246 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94100cc3d0 con 0x7f941014eaa0 2024-03-19T01:51:30.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.754+0000 7f9416abc640 1 -- 172.21.15.55:0/2563909246 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9410068a40 con 0x7f941014eaa0 2024-03-19T01:51:30.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.786+0000 7f940d7fa640 1 -- 172.21.15.55:0/2563909246 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f940801c070 con 0x7f941014eaa0 2024-03-19T01:51:30.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.786+0000 7f940d7fa640 1 --2- 172.21.15.55:0/2563909246 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f93f4067690 0x7f93f4069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:30.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.786+0000 7f9414858640 1 --2- 172.21.15.55:0/2563909246 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f93f4067690 0x7f93f4069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:30.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.790+0000 7f9414858640 1 --2- 172.21.15.55:0/2563909246 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f93f4067690 0x7f93f4069b50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9404009690 tx=0x7f9404006c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:30.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:30.790+0000 7f940d7fa640 1 -- 172.21.15.55:0/2563909246 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f9408053e00 con 0x7f941014eaa0 2024-03-19T01:51:31.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.074+0000 7f940d7fa640 1 -- 172.21.15.55:0/2563909246 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f9408083e20 con 0x7f941014eaa0 2024-03-19T01:51:31.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.234+0000 7f9416abc640 1 -- 172.21.15.55:0/2563909246 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f94101558e0 con 0x7f941014eaa0 2024-03-19T01:51:31.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.246+0000 7f940d7fa640 1 -- 172.21.15.55:0/2563909246 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) ==== 100+0+11 (secure 0 0 0) 0x7f9408023080 con 0x7f941014eaa0 2024-03-19T01:51:31.245 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 64 2024-03-19T01:51:31.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.246+0000 7f9416abc640 1 -- 172.21.15.55:0/2563909246 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f93f4067690 msgr2=0x7f93f4069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:31.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.246+0000 7f9416abc640 1 --2- 172.21.15.55:0/2563909246 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f93f4067690 0x7f93f4069b50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9404009690 tx=0x7f9404006c00 comp rx=0 tx=0).stop 2024-03-19T01:51:31.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.246+0000 7f9416abc640 1 -- 172.21.15.55:0/2563909246 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f941014eaa0 msgr2=0x7f9410146550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:31.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.246+0000 7f9416abc640 1 --2- 172.21.15.55:0/2563909246 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f941014eaa0 0x7f9410146550 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f940800b970 tx=0x7f940800be30 comp rx=0 tx=0).stop 2024-03-19T01:51:31.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.246+0000 7f9416abc640 1 -- 172.21.15.55:0/2563909246 shutdown_connections 2024-03-19T01:51:31.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.246+0000 7f9416abc640 1 --2- 172.21.15.55:0/2563909246 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f93f4067690 0x7f93f4069b50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:31.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.246+0000 7f9416abc640 1 --2- 172.21.15.55:0/2563909246 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f941014eaa0 0x7f9410146550 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:31.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.246+0000 7f9416abc640 1 --2- 172.21.15.55:0/2563909246 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f94100c6610 0x7f9410140d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:31.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.246+0000 7f9416abc640 1 --2- 172.21.15.55:0/2563909246 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f941006bec0 0x7f9410155e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:31.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.246+0000 7f9416abc640 1 -- 172.21.15.55:0/2563909246 >> 172.21.15.55:0/2563909246 conn(0x7f94100cc9c0 msgr2=0x7f9410070650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:31.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.246+0000 7f9416abc640 1 -- 172.21.15.55:0/2563909246 shutdown_connections 2024-03-19T01:51:31.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.246+0000 7f9416abc640 1 -- 172.21.15.55:0/2563909246 wait complete. 2024-03-19T01:51:31.265 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T01:51:31.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.382+0000 7f1ee5840640 1 Processor -- start 2024-03-19T01:51:31.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.382+0000 7f1ee5840640 1 -- start start 2024-03-19T01:51:31.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.382+0000 7f1ee5840640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f1ee00ca350 0x7f1ee015b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:31.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.382+0000 7f1ee5840640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ee00c3320 0x7f1ee00728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:31.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.382+0000 7f1ee5840640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ee0151330 con 0x7f1ee0150f50 2024-03-19T01:51:31.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.382+0000 7f1ee5840640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f1ee00c3e00 con 0x7f1ee00ca350 2024-03-19T01:51:31.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.382+0000 7f1ee5840640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1ee015ac10 con 0x7f1ee00c3320 2024-03-19T01:51:31.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.382+0000 7f1edeffd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f1ee0150f50 0x7f1ee0157920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:56368/0 (socket says 172.21.15.55:56368) 2024-03-19T01:51:31.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.382+0000 7f1edeffd640 1 -- 172.21.15.55:0/3524936637 learned_addr learned my addr 172.21.15.55:0/3524936637 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:51:31.384 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.386+0000 7f1ed67fc640 1 --2- 172.21.15.55:0/3524936637 >> v2:172.21.15.84:3300/0 conn(0x7f1ee00ca350 0x7f1ee015b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:31.384 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.386+0000 7f1edf7fe640 1 --2- 172.21.15.55:0/3524936637 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ee00c3320 0x7f1ee00728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:31.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.386+0000 7f1ede7fc640 1 -- 172.21.15.55:0/3524936637 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3547764127 0 0) 0x7f1ee0151330 con 0x7f1ee0150f50 2024-03-19T01:51:31.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.386+0000 7f1ede7fc640 1 -- 172.21.15.55:0/3524936637 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ecc003530 con 0x7f1ee0150f50 2024-03-19T01:51:31.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.390+0000 7f1ede7fc640 1 -- 172.21.15.55:0/3524936637 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1726822822 0 0) 0x7f1ecc003530 con 0x7f1ee0150f50 2024-03-19T01:51:31.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.390+0000 7f1ede7fc640 1 -- 172.21.15.55:0/3524936637 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ee00c3320 msgr2=0x7f1ee00728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:31.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.390+0000 7f1ede7fc640 1 --2- 172.21.15.55:0/3524936637 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ee00c3320 0x7f1ee00728b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:31.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.390+0000 7f1ede7fc640 1 -- 172.21.15.55:0/3524936637 >> v2:172.21.15.84:3300/0 conn(0x7f1ee00ca350 msgr2=0x7f1ee015b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:31.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.390+0000 7f1ede7fc640 1 --2- 172.21.15.55:0/3524936637 >> v2:172.21.15.84:3300/0 conn(0x7f1ee00ca350 0x7f1ee015b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:31.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.390+0000 7f1ede7fc640 1 -- 172.21.15.55:0/3524936637 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ee00c3840 con 0x7f1ee0150f50 2024-03-19T01:51:31.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.390+0000 7f1ede7fc640 1 -- 172.21.15.55:0/3524936637 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f1ed0001680 con 0x7f1ee0150f50 2024-03-19T01:51:31.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.390+0000 7f1ede7fc640 1 -- 172.21.15.55:0/3524936637 >> v1:172.21.15.55:6789/0 conn(0x7f1ee0150f50 legacy=0x7f1ee0157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:31.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.390+0000 7f1ede7fc640 1 --2- 172.21.15.55:0/3524936637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ecc0039c0 0x7f1ecc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:31.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.390+0000 7f1ede7fc640 1 --2- 172.21.15.55:0/3524936637 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ee00c3320 0x7f1ecc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:31.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.390+0000 7f1ede7fc640 1 --2- 172.21.15.55:0/3524936637 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1ecc004d50 0x7f1ecc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:31.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.390+0000 7f1ede7fc640 1 -- 172.21.15.55:0/3524936637 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee00c3e00 con 0x7f1ecc0039c0 2024-03-19T01:51:31.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.390+0000 7f1ede7fc640 1 -- 172.21.15.55:0/3524936637 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee015ac10 con 0x7f1ecc004d50 2024-03-19T01:51:31.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.390+0000 7f1ede7fc640 1 -- 172.21.15.55:0/3524936637 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1ecc00b9d0 con 0x7f1ee00c3320 2024-03-19T01:51:31.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.394+0000 7f1edeffd640 1 --2- 172.21.15.55:0/3524936637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ecc0039c0 0x7f1ecc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:31.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.398+0000 7f1edeffd640 1 -- 172.21.15.55:0/3524936637 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ee00c3320 msgr2=0x7f1ecc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:31.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.398+0000 7f1edeffd640 1 --2- 172.21.15.55:0/3524936637 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ee00c3320 0x7f1ecc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:31.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.398+0000 7f1edeffd640 1 -- 172.21.15.55:0/3524936637 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1ecc004d50 msgr2=0x7f1ecc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:31.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.398+0000 7f1edeffd640 1 --2- 172.21.15.55:0/3524936637 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1ecc004d50 0x7f1ecc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:31.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.398+0000 7f1edeffd640 1 -- 172.21.15.55:0/3524936637 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ecc003530 con 0x7f1ecc0039c0 2024-03-19T01:51:31.401 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.402+0000 7f1edeffd640 1 --2- 172.21.15.55:0/3524936637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ecc0039c0 0x7f1ecc003de0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0001060 tx=0x7f1ed001cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=167d208f07d4ef0e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:31.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.402+0000 7f1ede7fc640 1 -- 172.21.15.55:0/3524936637 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0007cb0 con 0x7f1ecc0039c0 2024-03-19T01:51:31.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.402+0000 7f1ede7fc640 1 -- 172.21.15.55:0/3524936637 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ed00a9020 con 0x7f1ecc0039c0 2024-03-19T01:51:31.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.402+0000 7f1ede7fc640 1 -- 172.21.15.55:0/3524936637 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0005b20 con 0x7f1ecc0039c0 2024-03-19T01:51:31.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.406+0000 7f1ee5840640 1 -- 172.21.15.55:0/3524936637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ecc0039c0 msgr2=0x7f1ecc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:31.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.406+0000 7f1ee5840640 1 --2- 172.21.15.55:0/3524936637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ecc0039c0 0x7f1ecc003de0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0001060 tx=0x7f1ed001cbb0 comp rx=0 tx=0).stop 2024-03-19T01:51:31.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.406+0000 7f1edf7fe640 1 -- 172.21.15.55:0/3524936637 reap_dead start 2024-03-19T01:51:31.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.406+0000 7f1ee5840640 1 -- 172.21.15.55:0/3524936637 shutdown_connections 2024-03-19T01:51:31.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.406+0000 7f1ee5840640 1 -- 172.21.15.55:0/3524936637 >> 172.21.15.55:0/3524936637 conn(0x7f1ee006e670 msgr2=0x7f1ee006ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:31.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.406+0000 7f1ee5840640 1 -- 172.21.15.55:0/3524936637 shutdown_connections 2024-03-19T01:51:31.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.406+0000 7f1ee5840640 1 -- 172.21.15.55:0/3524936637 wait complete. 2024-03-19T01:51:31.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.406+0000 7f1ee5840640 1 Processor -- start 2024-03-19T01:51:31.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.406+0000 7f1ee5840640 1 -- start start 2024-03-19T01:51:31.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.406+0000 7f1ee5840640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ecc0039c0 0x7f1ee00cbe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:31.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.406+0000 7f1ee5840640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ee014f3b0 0x7f1ee0147d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:31.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.406+0000 7f1ee5840640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1ee01482d0 0x7f1ee00c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:31.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.406+0000 7f1ee5840640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee006b2a0 con 0x7f1ecc0039c0 2024-03-19T01:51:31.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.406+0000 7f1ee5840640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee0072cc0 con 0x7f1ee01482d0 2024-03-19T01:51:31.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.406+0000 7f1ee5840640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1ee006b420 con 0x7f1ee014f3b0 2024-03-19T01:51:31.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.406+0000 7f1edeffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ecc0039c0 0x7f1ee00cbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:31.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.410+0000 7f1edeffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ecc0039c0 0x7f1ee00cbe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50606/0 (socket says 172.21.15.55:50606) 2024-03-19T01:51:31.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.410+0000 7f1edeffd640 1 -- 172.21.15.55:0/3045264308 learned_addr learned my addr 172.21.15.55:0/3045264308 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:31.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.410+0000 7f1edf7fe640 1 --2- 172.21.15.55:0/3045264308 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1ee01482d0 0x7f1ee00c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:31.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.410+0000 7f1ed67fc640 1 --2- 172.21.15.55:0/3045264308 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ee014f3b0 0x7f1ee0147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:31.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.410+0000 7f1edeffd640 1 -- 172.21.15.55:0/3045264308 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ee014f3b0 msgr2=0x7f1ee0147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:31.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.410+0000 7f1edeffd640 1 --2- 172.21.15.55:0/3045264308 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ee014f3b0 0x7f1ee0147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:31.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.410+0000 7f1edeffd640 1 -- 172.21.15.55:0/3045264308 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1ee01482d0 msgr2=0x7f1ee00c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:31.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.410+0000 7f1edeffd640 1 --2- 172.21.15.55:0/3045264308 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1ee01482d0 0x7f1ee00c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:31.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.410+0000 7f1edeffd640 1 -- 172.21.15.55:0/3045264308 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ee0148760 con 0x7f1ecc0039c0 2024-03-19T01:51:31.415 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.414+0000 7f1edeffd640 1 --2- 172.21.15.55:0/3045264308 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ecc0039c0 0x7f1ee00cbe10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1ed00ab040 tx=0x7f1ecc005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:31.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.418+0000 7f1edcff9640 1 -- 172.21.15.55:0/3045264308 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0005d80 con 0x7f1ecc0039c0 2024-03-19T01:51:31.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.418+0000 7f1edcff9640 1 -- 172.21.15.55:0/3045264308 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ed00a5890 con 0x7f1ecc0039c0 2024-03-19T01:51:31.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.418+0000 7f1ee5840640 1 -- 172.21.15.55:0/3045264308 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ee01504c0 con 0x7f1ecc0039c0 2024-03-19T01:51:31.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.418+0000 7f1edcff9640 1 -- 172.21.15.55:0/3045264308 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed00a5c80 con 0x7f1ecc0039c0 2024-03-19T01:51:31.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.418+0000 7f1ee5840640 1 -- 172.21.15.55:0/3045264308 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ee01472d0 con 0x7f1ecc0039c0 2024-03-19T01:51:31.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.418+0000 7f1ee5840640 1 -- 172.21.15.55:0/3045264308 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ee0068a40 con 0x7f1ecc0039c0 2024-03-19T01:51:31.689 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.690+0000 7f1edcff9640 1 -- 172.21.15.55:0/3045264308 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1ed00047f0 con 0x7f1ecc0039c0 2024-03-19T01:51:31.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.690+0000 7f1edcff9640 1 --2- 172.21.15.55:0/3045264308 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ec00675f0 0x7f1ec0069ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:31.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.690+0000 7f1ed67fc640 1 --2- 172.21.15.55:0/3045264308 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ec00675f0 0x7f1ec0069ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:31.691 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.690+0000 7f1ed67fc640 1 --2- 172.21.15.55:0/3045264308 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ec00675f0 0x7f1ec0069ab0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80093f0 tx=0x7f1ec80094c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:31.717 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.718+0000 7f1edcff9640 1 -- 172.21.15.55:0/3045264308 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f1ed00eda50 con 0x7f1ecc0039c0 2024-03-19T01:51:31.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.718+0000 7f1edcff9640 1 -- 172.21.15.55:0/3045264308 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1ed00168f0 con 0x7f1ecc0039c0 2024-03-19T01:51:31.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.902+0000 7f1ee5840640 1 -- 172.21.15.55:0/3045264308 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1ee00c40a0 con 0x7f1ecc0039c0 2024-03-19T01:51:31.913 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.910+0000 7f1edcff9640 1 -- 172.21.15.55:0/3045264308 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) ==== 103+0+11 (secure 0 0 0) 0x7f1ed00a5e20 con 0x7f1ecc0039c0 2024-03-19T01:51:31.913 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 32 2024-03-19T01:51:31.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.914+0000 7f1ee5840640 1 -- 172.21.15.55:0/3045264308 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ec00675f0 msgr2=0x7f1ec0069ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:31.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.914+0000 7f1ee5840640 1 --2- 172.21.15.55:0/3045264308 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ec00675f0 0x7f1ec0069ab0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80093f0 tx=0x7f1ec80094c0 comp rx=0 tx=0).stop 2024-03-19T01:51:31.916 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.914+0000 7f1ee5840640 1 -- 172.21.15.55:0/3045264308 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ecc0039c0 msgr2=0x7f1ee00cbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:31.916 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.914+0000 7f1ee5840640 1 --2- 172.21.15.55:0/3045264308 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ecc0039c0 0x7f1ee00cbe10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1ed00ab040 tx=0x7f1ecc005840 comp rx=0 tx=0).stop 2024-03-19T01:51:31.916 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.914+0000 7f1ee5840640 1 -- 172.21.15.55:0/3045264308 shutdown_connections 2024-03-19T01:51:31.916 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.914+0000 7f1ee5840640 1 --2- 172.21.15.55:0/3045264308 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ec00675f0 0x7f1ec0069ab0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:31.916 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.914+0000 7f1ee5840640 1 --2- 172.21.15.55:0/3045264308 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1ee01482d0 0x7f1ee00c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:31.916 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.914+0000 7f1ee5840640 1 --2- 172.21.15.55:0/3045264308 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ee014f3b0 0x7f1ee0147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:31.916 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.914+0000 7f1ee5840640 1 --2- 172.21.15.55:0/3045264308 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ecc0039c0 0x7f1ee00cbe10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:31.916 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.914+0000 7f1ee5840640 1 -- 172.21.15.55:0/3045264308 >> 172.21.15.55:0/3045264308 conn(0x7f1ee006e670 msgr2=0x7f1ee006c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:31.916 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.914+0000 7f1ee5840640 1 -- 172.21.15.55:0/3045264308 shutdown_connections 2024-03-19T01:51:31.916 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:31.914+0000 7f1ee5840640 1 -- 172.21.15.55:0/3045264308 wait complete. 2024-03-19T01:51:31.931 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-19T01:51:32.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.054+0000 7f79aeb7d640 1 Processor -- start 2024-03-19T01:51:32.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.054+0000 7f79aeb7d640 1 -- start start 2024-03-19T01:51:32.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.054+0000 7f79aeb7d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f79a80cd470 0x7f79a814e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:32.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.054+0000 7f79aeb7d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f79a80c8010 0x7f79a814ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:32.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.054+0000 7f79aeb7d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79a8158330 con 0x7f79a80cd090 2024-03-19T01:51:32.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.054+0000 7f79aeb7d640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f79a8158e90 con 0x7f79a80c8010 2024-03-19T01:51:32.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.054+0000 7f79aeb7d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f79a81585f0 con 0x7f79a80cd470 2024-03-19T01:51:32.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.054+0000 7f79ad11a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f79a80c8010 0x7f79a814ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:32.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.058+0000 7f79ad11a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f79a80c8010 0x7f79a814ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:44066/0 (socket says 172.21.15.55:44066) 2024-03-19T01:51:32.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.058+0000 7f79ad11a640 1 -- 172.21.15.55:0/133169656 learned_addr learned my addr 172.21.15.55:0/133169656 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:32.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.062+0000 7f79ad11a640 1 -- 172.21.15.55:0/133169656 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f79a80cd470 msgr2=0x7f79a814e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:32.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.062+0000 7f79ad11a640 1 --2- 172.21.15.55:0/133169656 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f79a80cd470 0x7f79a814e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:32.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.062+0000 7f79ad11a640 1 -- 172.21.15.55:0/133169656 >> v1:172.21.15.84:6789/0 conn(0x7f79a80cd090 legacy=0x7f79a80732b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:32.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.062+0000 7f79ad11a640 1 -- 172.21.15.55:0/133169656 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79a81588d0 con 0x7f79a80c8010 2024-03-19T01:51:32.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.070+0000 7f79ad11a640 1 --2- 172.21.15.55:0/133169656 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79a80c8010 0x7f79a814ea90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f79a0009e50 tx=0x7f79a001c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e8ef9863f04b7d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:32.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.082+0000 7f79a77fe640 1 -- 172.21.15.55:0/133169656 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a00a5670 con 0x7f79a80c8010 2024-03-19T01:51:32.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.082+0000 7f79a77fe640 1 -- 172.21.15.55:0/133169656 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79a00a5810 con 0x7f79a80c8010 2024-03-19T01:51:32.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.082+0000 7f79a77fe640 1 -- 172.21.15.55:0/133169656 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a00a5be0 con 0x7f79a80c8010 2024-03-19T01:51:32.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.082+0000 7f79aeb7d640 1 -- 172.21.15.55:0/133169656 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79a80c8010 msgr2=0x7f79a814ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:32.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.082+0000 7f79aeb7d640 1 --2- 172.21.15.55:0/133169656 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79a80c8010 0x7f79a814ea90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f79a0009e50 tx=0x7f79a001c9f0 comp rx=0 tx=0).stop 2024-03-19T01:51:32.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.082+0000 7f79aeb7d640 1 -- 172.21.15.55:0/133169656 shutdown_connections 2024-03-19T01:51:32.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.082+0000 7f79aeb7d640 1 --2- 172.21.15.55:0/133169656 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79a80c8010 0x7f79a814ea90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:32.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.082+0000 7f79aeb7d640 1 --2- 172.21.15.55:0/133169656 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f79a80cd470 0x7f79a814e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:32.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.082+0000 7f79aeb7d640 1 -- 172.21.15.55:0/133169656 >> 172.21.15.55:0/133169656 conn(0x7f79a806e670 msgr2=0x7f79a806ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:32.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.082+0000 7f79aeb7d640 1 -- 172.21.15.55:0/133169656 shutdown_connections 2024-03-19T01:51:32.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.082+0000 7f79aeb7d640 1 -- 172.21.15.55:0/133169656 wait complete. 2024-03-19T01:51:32.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.086+0000 7f79aeb7d640 1 Processor -- start 2024-03-19T01:51:32.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.086+0000 7f79aeb7d640 1 -- start start 2024-03-19T01:51:32.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.086+0000 7f79aeb7d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f79a80c8010 0x7f79a81f7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:32.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.086+0000 7f79aeb7d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79a80cd090 0x7f79a81f3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:32.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.086+0000 7f79aeb7d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f79a80cd470 0x7f79a80c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:32.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.086+0000 7f79aeb7d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f79a8164a90 con 0x7f79a80cd470 2024-03-19T01:51:32.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.086+0000 7f79aeb7d640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f79a8164910 con 0x7f79a80cd090 2024-03-19T01:51:32.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.086+0000 7f79aeb7d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f79a8164c10 con 0x7f79a80c8010 2024-03-19T01:51:32.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.086+0000 7f79ad11a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f79a80cd470 0x7f79a80c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:32.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.086+0000 7f79ad11a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f79a80cd470 0x7f79a80c9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50616/0 (socket says 172.21.15.55:50616) 2024-03-19T01:51:32.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.086+0000 7f79ad11a640 1 -- 172.21.15.55:0/2159536512 learned_addr learned my addr 172.21.15.55:0/2159536512 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:32.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.090+0000 7f79a7fff640 1 --2- 172.21.15.55:0/2159536512 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79a80cd090 0x7f79a81f3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:32.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.090+0000 7f79ad11a640 1 -- 172.21.15.55:0/2159536512 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f79a80c8010 msgr2=0x7f79a81f7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:32.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.090+0000 7f79ad11a640 1 --2- 172.21.15.55:0/2159536512 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f79a80c8010 0x7f79a81f7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:32.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.090+0000 7f79ad11a640 1 -- 172.21.15.55:0/2159536512 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79a80cd090 msgr2=0x7f79a81f3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:32.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.090+0000 7f79ad11a640 1 --2- 172.21.15.55:0/2159536512 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79a80cd090 0x7f79a81f3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:32.092 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.090+0000 7f79ad11a640 1 -- 172.21.15.55:0/2159536512 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79a81f76b0 con 0x7f79a80cd470 2024-03-19T01:51:32.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.094+0000 7f79ad11a640 1 --2- 172.21.15.55:0/2159536512 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f79a80cd470 0x7f79a80c9260 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f79a00af040 tx=0x7f79a0019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:32.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.098+0000 7f79a57fa640 1 -- 172.21.15.55:0/2159536512 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a00bc020 con 0x7f79a80cd470 2024-03-19T01:51:32.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.098+0000 7f79a57fa640 1 -- 172.21.15.55:0/2159536512 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79a00b8670 con 0x7f79a80cd470 2024-03-19T01:51:32.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.098+0000 7f79a57fa640 1 -- 172.21.15.55:0/2159536512 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a00b8a40 con 0x7f79a80cd470 2024-03-19T01:51:32.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.098+0000 7f79aeb7d640 1 -- 172.21.15.55:0/2159536512 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79a80cad10 con 0x7f79a80cd470 2024-03-19T01:51:32.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.098+0000 7f79aeb7d640 1 -- 172.21.15.55:0/2159536512 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79a81f6970 con 0x7f79a80cd470 2024-03-19T01:51:32.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.098+0000 7f79aeb7d640 1 -- 172.21.15.55:0/2159536512 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79a80cb230 con 0x7f79a80cd470 2024-03-19T01:51:32.415 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.414+0000 7f79a57fa640 1 -- 172.21.15.55:0/2159536512 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f79a00bf020 con 0x7f79a80cd470 2024-03-19T01:51:32.415 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.414+0000 7f79a57fa640 1 --2- 172.21.15.55:0/2159536512 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7994067660 0x7f7994069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:32.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.414+0000 7f79ac919640 1 --2- 172.21.15.55:0/2159536512 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7994067660 0x7f7994069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:32.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.418+0000 7f79ac919640 1 --2- 172.21.15.55:0/2159536512 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7994067660 0x7f7994069b20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f799c007fd0 tx=0x7f799c0079a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:32.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.446+0000 7f79a57fa640 1 -- 172.21.15.55:0/2159536512 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f79a00ef560 con 0x7f79a80cd470 2024-03-19T01:51:32.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.450+0000 7f79a57fa640 1 -- 172.21.15.55:0/2159536512 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f79a00c6030 con 0x7f79a80cd470 2024-03-19T01:51:32.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.634+0000 7f79aeb7d640 1 -- 172.21.15.55:0/2159536512 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f79a8068a40 con 0x7f79a80cd470 2024-03-19T01:51:32.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.642+0000 7f79a57fa640 1 -- 172.21.15.55:0/2159536512 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) ==== 80+0+0 (secure 0 0 0) 0x7f79a00165e0 con 0x7f79a80cd470 2024-03-19T01:51:32.644 INFO:teuthology.orchestra.run.smithi055.stderr:listed 0 entries 2024-03-19T01:51:32.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.646+0000 7f79aeb7d640 1 -- 172.21.15.55:0/2159536512 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7994067660 msgr2=0x7f7994069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:32.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.646+0000 7f79aeb7d640 1 --2- 172.21.15.55:0/2159536512 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7994067660 0x7f7994069b20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f799c007fd0 tx=0x7f799c0079a0 comp rx=0 tx=0).stop 2024-03-19T01:51:32.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.646+0000 7f79aeb7d640 1 -- 172.21.15.55:0/2159536512 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f79a80cd470 msgr2=0x7f79a80c9260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:32.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.646+0000 7f79aeb7d640 1 --2- 172.21.15.55:0/2159536512 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f79a80cd470 0x7f79a80c9260 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f79a00af040 tx=0x7f79a0019790 comp rx=0 tx=0).stop 2024-03-19T01:51:32.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.646+0000 7f79aeb7d640 1 -- 172.21.15.55:0/2159536512 shutdown_connections 2024-03-19T01:51:32.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.646+0000 7f79aeb7d640 1 --2- 172.21.15.55:0/2159536512 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7994067660 0x7f7994069b20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:32.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.646+0000 7f79aeb7d640 1 --2- 172.21.15.55:0/2159536512 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f79a80cd470 0x7f79a80c9260 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:32.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.646+0000 7f79aeb7d640 1 --2- 172.21.15.55:0/2159536512 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79a80cd090 0x7f79a81f3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:32.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.646+0000 7f79aeb7d640 1 --2- 172.21.15.55:0/2159536512 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f79a80c8010 0x7f79a81f7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:32.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.646+0000 7f79aeb7d640 1 -- 172.21.15.55:0/2159536512 >> 172.21.15.55:0/2159536512 conn(0x7f79a806e670 msgr2=0x7f79a814e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:32.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.646+0000 7f79aeb7d640 1 -- 172.21.15.55:0/2159536512 shutdown_connections 2024-03-19T01:51:32.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.646+0000 7f79aeb7d640 1 -- 172.21.15.55:0/2159536512 wait complete. 2024-03-19T01:51:32.663 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-19T01:51:32.664 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T01:51:32.668 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:51:32.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.830+0000 7f1bf17fa640 1 Processor -- start 2024-03-19T01:51:32.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.830+0000 7f1bf17fa640 1 -- start start 2024-03-19T01:51:32.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.830+0000 7f1bf17fa640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1bec071050 0x7f1bec0ca210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:32.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.830+0000 7f1bf17fa640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f1bec0c7d60 0x7f1bec15ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:32.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.830+0000 7f1bf17fa640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bec0c8170 con 0x7f1bec1545f0 2024-03-19T01:51:32.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.830+0000 7f1bf17fa640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f1bec15b340 con 0x7f1bec071050 2024-03-19T01:51:32.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.830+0000 7f1bf17fa640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f1bec072c90 con 0x7f1bec0c7d60 2024-03-19T01:51:32.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.834+0000 7f1beb7fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f1bec0c7d60 0x7f1bec15ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:32.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.834+0000 7f1beb7fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f1bec0c7d60 0x7f1bec15ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:44080/0 (socket says 172.21.15.55:44080) 2024-03-19T01:51:32.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.834+0000 7f1beb7fe640 1 -- 172.21.15.55:0/1035047395 learned_addr learned my addr 172.21.15.55:0/1035047395 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:32.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.838+0000 7f1beb7fe640 1 -- 172.21.15.55:0/1035047395 >> v2:172.21.15.55:3300/0 conn(0x7f1bec071050 msgr2=0x7f1bec0ca210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:32.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.838+0000 7f1beb7fe640 1 --2- 172.21.15.55:0/1035047395 >> v2:172.21.15.55:3300/0 conn(0x7f1bec071050 0x7f1bec0ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:32.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.838+0000 7f1beb7fe640 1 -- 172.21.15.55:0/1035047395 >> v1:172.21.15.84:6789/0 conn(0x7f1bec1545f0 legacy=0x7f1bec06b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:32.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.838+0000 7f1beb7fe640 1 -- 172.21.15.55:0/1035047395 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bec073210 con 0x7f1bec0c7d60 2024-03-19T01:51:32.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.846+0000 7f1beb7fe640 1 --2- 172.21.15.55:0/1035047395 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1bec0c7d60 0x7f1bec15ae00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1bec075070 tx=0x7f1be401c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bacfccd9d5e791d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:32.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.966+0000 7f1be9ffb640 1 -- 172.21.15.55:0/1035047395 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be40ac020 con 0x7f1bec0c7d60 2024-03-19T01:51:32.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.966+0000 7f1be9ffb640 1 -- 172.21.15.55:0/1035047395 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1be40a6420 con 0x7f1bec0c7d60 2024-03-19T01:51:32.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.966+0000 7f1be9ffb640 1 -- 172.21.15.55:0/1035047395 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be40a67f0 con 0x7f1bec0c7d60 2024-03-19T01:51:32.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.970+0000 7f1bf17fa640 1 -- 172.21.15.55:0/1035047395 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1bec0c7d60 msgr2=0x7f1bec15ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:32.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.970+0000 7f1bf17fa640 1 --2- 172.21.15.55:0/1035047395 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1bec0c7d60 0x7f1bec15ae00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1bec075070 tx=0x7f1be401c7d0 comp rx=0 tx=0).stop 2024-03-19T01:51:32.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.970+0000 7f1bf17fa640 1 -- 172.21.15.55:0/1035047395 shutdown_connections 2024-03-19T01:51:32.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.970+0000 7f1bf17fa640 1 --2- 172.21.15.55:0/1035047395 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1bec0c7d60 0x7f1bec15ae00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:32.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.970+0000 7f1bf17fa640 1 --2- 172.21.15.55:0/1035047395 >> v2:172.21.15.55:3300/0 conn(0x7f1bec071050 0x7f1bec0ca210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:32.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.970+0000 7f1bf17fa640 1 -- 172.21.15.55:0/1035047395 >> 172.21.15.55:0/1035047395 conn(0x7f1bec070410 msgr2=0x7f1bec070820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:32.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.970+0000 7f1bf17fa640 1 -- 172.21.15.55:0/1035047395 shutdown_connections 2024-03-19T01:51:32.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.970+0000 7f1bf17fa640 1 -- 172.21.15.55:0/1035047395 wait complete. 2024-03-19T01:51:32.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.970+0000 7f1bf17fa640 1 Processor -- start 2024-03-19T01:51:32.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.970+0000 7f1bf17fa640 1 -- start start 2024-03-19T01:51:32.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.970+0000 7f1bf17fa640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1bec071050 0x7f1bec15a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:32.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.970+0000 7f1bf17fa640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1bec0c7d60 0x7f1bec15e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:32.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.970+0000 7f1bf17fa640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1bec1545f0 0x7f1bec0cabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:32.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.970+0000 7f1bf17fa640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec154b50 con 0x7f1bec0c7d60 2024-03-19T01:51:32.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.970+0000 7f1bf17fa640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec1549d0 con 0x7f1bec1545f0 2024-03-19T01:51:32.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.970+0000 7f1bf17fa640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1bec166c70 con 0x7f1bec071050 2024-03-19T01:51:32.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.974+0000 7f1beaffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1bec071050 0x7f1bec15a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:32.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.974+0000 7f1beb7fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1bec1545f0 0x7f1bec0cabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:32.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.974+0000 7f1beaffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1bec071050 0x7f1bec15a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:55252/0 (socket says 172.21.15.55:55252) 2024-03-19T01:51:32.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.974+0000 7f1beaffd640 1 -- 172.21.15.55:0/4026304605 learned_addr learned my addr 172.21.15.55:0/4026304605 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:32.980 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.978+0000 7f1bea7fc640 1 --2- 172.21.15.55:0/4026304605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1bec0c7d60 0x7f1bec15e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:32.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.986+0000 7f1bea7fc640 1 -- 172.21.15.55:0/4026304605 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1bec071050 msgr2=0x7f1bec15a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:32.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.986+0000 7f1bea7fc640 1 --2- 172.21.15.55:0/4026304605 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1bec071050 0x7f1bec15a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:32.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.986+0000 7f1bea7fc640 1 -- 172.21.15.55:0/4026304605 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1bec1545f0 msgr2=0x7f1bec0cabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:32.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.986+0000 7f1bea7fc640 1 --2- 172.21.15.55:0/4026304605 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1bec1545f0 0x7f1bec0cabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:32.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.986+0000 7f1bea7fc640 1 -- 172.21.15.55:0/4026304605 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bec0cc8c0 con 0x7f1bec0c7d60 2024-03-19T01:51:32.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:32.990+0000 7f1bea7fc640 1 --2- 172.21.15.55:0/4026304605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1bec0c7d60 0x7f1bec15e080 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1be000b3f0 tx=0x7f1be000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:33.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.070+0000 7f1bd37fe640 1 -- 172.21.15.55:0/4026304605 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be0015020 con 0x7f1bec0c7d60 2024-03-19T01:51:33.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.070+0000 7f1bd37fe640 1 -- 172.21.15.55:0/4026304605 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1be0011830 con 0x7f1bec0c7d60 2024-03-19T01:51:33.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.070+0000 7f1bf17fa640 1 -- 172.21.15.55:0/4026304605 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bec0ccba0 con 0x7f1bec0c7d60 2024-03-19T01:51:33.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.070+0000 7f1bd37fe640 1 -- 172.21.15.55:0/4026304605 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be0011c20 con 0x7f1bec0c7d60 2024-03-19T01:51:33.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.070+0000 7f1bf17fa640 1 -- 172.21.15.55:0/4026304605 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bec0cd160 con 0x7f1bec0c7d60 2024-03-19T01:51:33.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.070+0000 7f1bf17fa640 1 -- 172.21.15.55:0/4026304605 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bec068a40 con 0x7f1bec0c7d60 2024-03-19T01:51:33.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.362+0000 7f1bd37fe640 1 -- 172.21.15.55:0/4026304605 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1be0018020 con 0x7f1bec0c7d60 2024-03-19T01:51:33.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.362+0000 7f1bd37fe640 1 --2- 172.21.15.55:0/4026304605 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1bd80676e0 0x7f1bd8069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:33.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.362+0000 7f1beaffd640 1 --2- 172.21.15.55:0/4026304605 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1bd80676e0 0x7f1bd8069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:33.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.362+0000 7f1beaffd640 1 --2- 172.21.15.55:0/4026304605 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1bd80676e0 0x7f1bd8069ba0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1bec14e0c0 tx=0x7f1bdc0090e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:33.376 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.374+0000 7f1bd37fe640 1 -- 172.21.15.55:0/4026304605 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f1be00542e0 con 0x7f1bec0c7d60 2024-03-19T01:51:33.376 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.378+0000 7f1bd37fe640 1 -- 172.21.15.55:0/4026304605 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1be001f020 con 0x7f1bec0c7d60 2024-03-19T01:51:33.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.558+0000 7f1bf17fa640 1 -- 172.21.15.55:0/4026304605 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1bec0c7930 con 0x7f1bec0c7d60 2024-03-19T01:51:33.622 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.622+0000 7f1bd37fe640 1 -- 172.21.15.55:0/4026304605 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+18430 (secure 0 0 0) 0x7f1be0056240 con 0x7f1bec0c7d60 2024-03-19T01:51:33.623 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:51:33.623 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":29,"fsid":"def9d4e5-0bc7-449d-b893-2490cbd10e25","created":"2024-03-19T01:50:02.482954+0000","modified":"2024-03-19T01:51:17.228291+0000","last_up_change":"2024-03-19T01:50:21.735508+0000","last_in_change":"2024-03-19T01:50:12.168944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:50:23.456904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:51:04.578583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:51:06.871659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T01:51:14.274442+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"35d5e3c5-d089-4f38-8961-98d064188f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6825","nonce":2757675043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6827","nonce":2757675043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6831","nonce":2757675043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6829","nonce":2757675043}]},"public_addr":"172.21.15.55:6825/2757675043","cluster_addr":"172.21.15.55:6827/2757675043","heartbeat_back_addr":"172.21.15.55:6831/2757675043","heartbeat_front_addr":"172.21.15.55:6829/2757675043","state":["exists","up"]},{"osd":1,"uuid":"8e58f791-001a-44d0-a464-ca68a479f70c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6817","nonce":543460007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6819","nonce":543460007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6823","nonce":543460007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6821","nonce":543460007}]},"public_addr":"172.21.15.55:6817/543460007","cluster_addr":"172.21.15.55:6819/543460007","heartbeat_back_addr":"172.21.15.55:6823/543460007","heartbeat_front_addr":"172.21.15.55:6821/543460007","state":["exists","up"]},{"osd":2,"uuid":"d44f38b8-67f4-43de-98a6-f3c520acaf10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6801","nonce":1760253550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6803","nonce":1760253550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6807","nonce":1760253550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6805","nonce":1760253550}]},"public_addr":"172.21.15.55:6801/1760253550","cluster_addr":"172.21.15.55:6803/1760253550","heartbeat_back_addr":"172.21.15.55:6807/1760253550","heartbeat_front_addr":"172.21.15.55:6805/1760253550","state":["exists","up"]},{"osd":3,"uuid":"850a7316-2d74-43f0-8649-b5f8d8f6692e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6809","nonce":2797616448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6811","nonce":2797616448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6815","nonce":2797616448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6813","nonce":2797616448}]},"public_addr":"172.21.15.55:6809/2797616448","cluster_addr":"172.21.15.55:6811/2797616448","heartbeat_back_addr":"172.21.15.55:6815/2797616448","heartbeat_front_addr":"172.21.15.55:6813/2797616448","state":["exists","up"]},{"osd":4,"uuid":"4e6152ae-a77d-4bb6-a855-5782ad5411a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6819","nonce":2938436826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6821","nonce":2938436826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6825","nonce":2938436826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6823","nonce":2938436826}]},"public_addr":"172.21.15.84:6819/2938436826","cluster_addr":"172.21.15.84:6821/2938436826","heartbeat_back_addr":"172.21.15.84:6825/2938436826","heartbeat_front_addr":"172.21.15.84:6823/2938436826","state":["exists","up"]},{"osd":5,"uuid":"1217ceba-dd31-4615-96f1-dfaf57160f3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6803","nonce":460717295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6805","nonce":460717295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6809","nonce":460717295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6807","nonce":460717295}]},"public_addr":"172.21.15.84:6803/460717295","cluster_addr":"172.21.15.84:6805/460717295","heartbeat_back_addr":"172.21.15.84:6809/460717295","heartbeat_front_addr":"172.21.15.84:6807/460717295","state":["exists","up"]},{"osd":6,"uuid":"3d9ed78a-2284-4c28-a24a-70eb812c7836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6827","nonce":416123135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6829","nonce":416123135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6833","nonce":416123135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6831","nonce":416123135}]},"public_addr":"172.21.15.84:6827/416123135","cluster_addr":"172.21.15.84:6829/416123135","heartbeat_back_addr":"172.21.15.84:6833/416123135","heartbeat_front_addr":"172.21.15.84:6831/416123135","state":["exists","up"]},{"osd":7,"uuid":"c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6811","nonce":460083528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6813","nonce":460083528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6817","nonce":460083528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6815","nonce":460083528}]},"public_addr":"172.21.15.84:6811/460083528","cluster_addr":"172.21.15.84:6813/460083528","heartbeat_back_addr":"172.21.15.84:6817/460083528","heartbeat_front_addr":"172.21.15.84:6815/460083528","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.269126+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.121001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.075714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.092461+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.194676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.062833+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.316515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.237111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:51:33.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.626+0000 7f1bf17fa640 1 -- 172.21.15.55:0/4026304605 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1bd80676e0 msgr2=0x7f1bd8069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:33.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.626+0000 7f1bf17fa640 1 --2- 172.21.15.55:0/4026304605 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1bd80676e0 0x7f1bd8069ba0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1bec14e0c0 tx=0x7f1bdc0090e0 comp rx=0 tx=0).stop 2024-03-19T01:51:33.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.626+0000 7f1bf17fa640 1 -- 172.21.15.55:0/4026304605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1bec0c7d60 msgr2=0x7f1bec15e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:33.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.626+0000 7f1bf17fa640 1 --2- 172.21.15.55:0/4026304605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1bec0c7d60 0x7f1bec15e080 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1be000b3f0 tx=0x7f1be000b8b0 comp rx=0 tx=0).stop 2024-03-19T01:51:33.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.626+0000 7f1bf17fa640 1 -- 172.21.15.55:0/4026304605 shutdown_connections 2024-03-19T01:51:33.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.626+0000 7f1bf17fa640 1 --2- 172.21.15.55:0/4026304605 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1bd80676e0 0x7f1bd8069ba0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:33.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.626+0000 7f1bf17fa640 1 --2- 172.21.15.55:0/4026304605 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1bec1545f0 0x7f1bec0cabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:33.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.626+0000 7f1bf17fa640 1 --2- 172.21.15.55:0/4026304605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1bec0c7d60 0x7f1bec15e080 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:33.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.626+0000 7f1bf17fa640 1 --2- 172.21.15.55:0/4026304605 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1bec071050 0x7f1bec15a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:33.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.626+0000 7f1bf17fa640 1 -- 172.21.15.55:0/4026304605 >> 172.21.15.55:0/4026304605 conn(0x7f1bec070410 msgr2=0x7f1bec14d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:33.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.626+0000 7f1bf17fa640 1 -- 172.21.15.55:0/4026304605 shutdown_connections 2024-03-19T01:51:33.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.626+0000 7f1bf17fa640 1 -- 172.21.15.55:0/4026304605 wait complete. 2024-03-19T01:51:33.642 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:50:23.456904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:51:04.578583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:51:06.871659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T01:51:14.274442+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T01:51:33.642 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:51:33.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.766+0000 7fdac22cb640 1 Processor -- start 2024-03-19T01:51:33.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.766+0000 7fdac22cb640 1 -- start start 2024-03-19T01:51:33.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.766+0000 7fdac22cb640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdabc1544f0 0x7fdabc073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:33.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.766+0000 7fdac22cb640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fdabc15acb0 0x7fdabc0cce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:33.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.766+0000 7fdac22cb640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdabc0c7fa0 con 0x7fdabc06b2a0 2024-03-19T01:51:33.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.766+0000 7fdac22cb640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fdabc15b210 con 0x7fdabc15acb0 2024-03-19T01:51:33.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.766+0000 7fdac22cb640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fdabc0c8240 con 0x7fdabc1544f0 2024-03-19T01:51:33.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.766+0000 7fdabbfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdabc1544f0 0x7fdabc073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:33.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.766+0000 7fdac0868640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fdabc15acb0 0x7fdabc0cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:33.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.766+0000 7fdac0868640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fdabc15acb0 0x7fdabc0cce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:44098/0 (socket says 172.21.15.55:44098) 2024-03-19T01:51:33.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.766+0000 7fdac0868640 1 -- 172.21.15.55:0/3386275440 learned_addr learned my addr 172.21.15.55:0/3386275440 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:33.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.774+0000 7fdabbfff640 1 -- 172.21.15.55:0/3386275440 >> v2:172.21.15.84:3300/0 conn(0x7fdabc15acb0 msgr2=0x7fdabc0cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:33.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.774+0000 7fdabbfff640 1 --2- 172.21.15.55:0/3386275440 >> v2:172.21.15.84:3300/0 conn(0x7fdabc15acb0 0x7fdabc0cce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:33.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.774+0000 7fdabbfff640 1 -- 172.21.15.55:0/3386275440 >> v1:172.21.15.84:6789/0 conn(0x7fdabc06b2a0 legacy=0x7fdabc15e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:33.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.774+0000 7fdabbfff640 1 -- 172.21.15.55:0/3386275440 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdabc15d920 con 0x7fdabc1544f0 2024-03-19T01:51:33.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.778+0000 7fdabbfff640 1 --2- 172.21.15.55:0/3386275440 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdabc1544f0 0x7fdabc073540 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdabc0cdac0 tx=0x7fdaac01ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3431e6f29fd64bf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:33.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.790+0000 7fdabaffd640 1 -- 172.21.15.55:0/3386275440 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac0b5070 con 0x7fdabc1544f0 2024-03-19T01:51:33.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.790+0000 7fdabaffd640 1 -- 172.21.15.55:0/3386275440 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaac0a8910 con 0x7fdabc1544f0 2024-03-19T01:51:33.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.790+0000 7fdabaffd640 1 -- 172.21.15.55:0/3386275440 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac0a8ce0 con 0x7fdabc1544f0 2024-03-19T01:51:33.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.790+0000 7fdac22cb640 1 -- 172.21.15.55:0/3386275440 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdabc1544f0 msgr2=0x7fdabc073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:33.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.790+0000 7fdac22cb640 1 --2- 172.21.15.55:0/3386275440 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdabc1544f0 0x7fdabc073540 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdabc0cdac0 tx=0x7fdaac01ec50 comp rx=0 tx=0).stop 2024-03-19T01:51:33.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.790+0000 7fdac22cb640 1 -- 172.21.15.55:0/3386275440 shutdown_connections 2024-03-19T01:51:33.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.790+0000 7fdac22cb640 1 --2- 172.21.15.55:0/3386275440 >> v2:172.21.15.84:3300/0 conn(0x7fdabc15acb0 0x7fdabc0cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:33.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.790+0000 7fdac22cb640 1 --2- 172.21.15.55:0/3386275440 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdabc1544f0 0x7fdabc073540 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:33.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.790+0000 7fdac22cb640 1 -- 172.21.15.55:0/3386275440 >> 172.21.15.55:0/3386275440 conn(0x7fdabc06e670 msgr2=0x7fdabc06ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:33.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.790+0000 7fdac22cb640 1 -- 172.21.15.55:0/3386275440 shutdown_connections 2024-03-19T01:51:33.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.790+0000 7fdac22cb640 1 -- 172.21.15.55:0/3386275440 wait complete. 2024-03-19T01:51:33.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.794+0000 7fdac22cb640 1 Processor -- start 2024-03-19T01:51:33.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.794+0000 7fdac22cb640 1 -- start start 2024-03-19T01:51:33.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.794+0000 7fdac22cb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdabc06b2a0 0x7fdabc157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:33.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.794+0000 7fdac22cb640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdabc1544f0 0x7fdabc0c4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:33.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.794+0000 7fdac22cb640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdabc15acb0 0x7fdabc0cb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:33.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.794+0000 7fdac22cb640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc0ca120 con 0x7fdabc06b2a0 2024-03-19T01:51:33.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.794+0000 7fdac22cb640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc1548d0 con 0x7fdabc15acb0 2024-03-19T01:51:33.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.794+0000 7fdac22cb640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fdabc0ca2a0 con 0x7fdabc1544f0 2024-03-19T01:51:33.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.794+0000 7fdabbfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdabc06b2a0 0x7fdabc157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:33.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.794+0000 7fdabbfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdabc06b2a0 0x7fdabc157270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50640/0 (socket says 172.21.15.55:50640) 2024-03-19T01:51:33.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.794+0000 7fdabbfff640 1 -- 172.21.15.55:0/3849670998 learned_addr learned my addr 172.21.15.55:0/3849670998 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:33.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.798+0000 7fdac0868640 1 --2- 172.21.15.55:0/3849670998 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdabc15acb0 0x7fdabc0cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:33.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.798+0000 7fdabb7fe640 1 --2- 172.21.15.55:0/3849670998 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdabc1544f0 0x7fdabc0c4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:33.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.798+0000 7fdabbfff640 1 -- 172.21.15.55:0/3849670998 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdabc1544f0 msgr2=0x7fdabc0c4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:33.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.798+0000 7fdabbfff640 1 --2- 172.21.15.55:0/3849670998 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdabc1544f0 0x7fdabc0c4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:33.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.798+0000 7fdabbfff640 1 -- 172.21.15.55:0/3849670998 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdabc15acb0 msgr2=0x7fdabc0cb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:33.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.798+0000 7fdabbfff640 1 --2- 172.21.15.55:0/3849670998 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdabc15acb0 0x7fdabc0cb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:33.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.798+0000 7fdabbfff640 1 -- 172.21.15.55:0/3849670998 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdabc1efdb0 con 0x7fdabc06b2a0 2024-03-19T01:51:33.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.802+0000 7fdabbfff640 1 --2- 172.21.15.55:0/3849670998 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdabc06b2a0 0x7fdabc157270 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdaac01c880 tx=0x7fdaac01b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:33.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.806+0000 7fdab8ff9640 1 -- 172.21.15.55:0/3849670998 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac0bf070 con 0x7fdabc06b2a0 2024-03-19T01:51:33.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.806+0000 7fdab8ff9640 1 -- 172.21.15.55:0/3849670998 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaac0b0700 con 0x7fdabc06b2a0 2024-03-19T01:51:33.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.806+0000 7fdab8ff9640 1 -- 172.21.15.55:0/3849670998 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac0b5040 con 0x7fdabc06b2a0 2024-03-19T01:51:33.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.806+0000 7fdac22cb640 1 -- 172.21.15.55:0/3849670998 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdabc1577b0 con 0x7fdabc06b2a0 2024-03-19T01:51:33.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.806+0000 7fdac22cb640 1 -- 172.21.15.55:0/3849670998 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdabc1f36a0 con 0x7fdabc06b2a0 2024-03-19T01:51:33.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.810+0000 7fdac22cb640 1 -- 172.21.15.55:0/3849670998 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdabc068a40 con 0x7fdabc06b2a0 2024-03-19T01:51:33.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.830+0000 7fdab8ff9640 1 -- 172.21.15.55:0/3849670998 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fdaac0b0ad0 con 0x7fdabc06b2a0 2024-03-19T01:51:33.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.834+0000 7fdab8ff9640 1 --2- 172.21.15.55:0/3849670998 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdaa80676b0 0x7fdaa8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:33.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.834+0000 7fdabb7fe640 1 --2- 172.21.15.55:0/3849670998 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdaa80676b0 0x7fdaa8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:33.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:33.834+0000 7fdabb7fe640 1 --2- 172.21.15.55:0/3849670998 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdaa80676b0 0x7fdaa8069b70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdabc0b1c10 tx=0x7fdab0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:34.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.130+0000 7fdab8ff9640 1 -- 172.21.15.55:0/3849670998 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fdaac0f0d40 con 0x7fdabc06b2a0 2024-03-19T01:51:34.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.130+0000 7fdab8ff9640 1 -- 172.21.15.55:0/3849670998 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fdaac0c8020 con 0x7fdabc06b2a0 2024-03-19T01:51:34.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.306+0000 7fdac22cb640 1 -- 172.21.15.55:0/3849670998 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdabc0c9a70 con 0x7fdabc06b2a0 2024-03-19T01:51:34.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.318+0000 7fdab8ff9640 1 -- 172.21.15.55:0/3849670998 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) ==== 93+0+10 (secure 0 0 0) 0x7fdaac0187b0 con 0x7fdabc06b2a0 2024-03-19T01:51:34.320 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 1 2024-03-19T01:51:34.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.322+0000 7fdac22cb640 1 -- 172.21.15.55:0/3849670998 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdaa80676b0 msgr2=0x7fdaa8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:34.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.322+0000 7fdac22cb640 1 --2- 172.21.15.55:0/3849670998 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdaa80676b0 0x7fdaa8069b70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdabc0b1c10 tx=0x7fdab0008040 comp rx=0 tx=0).stop 2024-03-19T01:51:34.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.322+0000 7fdac22cb640 1 -- 172.21.15.55:0/3849670998 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdabc06b2a0 msgr2=0x7fdabc157270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:34.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.322+0000 7fdac22cb640 1 --2- 172.21.15.55:0/3849670998 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdabc06b2a0 0x7fdabc157270 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdaac01c880 tx=0x7fdaac01b9f0 comp rx=0 tx=0).stop 2024-03-19T01:51:34.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.322+0000 7fdac22cb640 1 -- 172.21.15.55:0/3849670998 shutdown_connections 2024-03-19T01:51:34.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.322+0000 7fdac22cb640 1 --2- 172.21.15.55:0/3849670998 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdaa80676b0 0x7fdaa8069b70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:34.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.322+0000 7fdac22cb640 1 --2- 172.21.15.55:0/3849670998 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdabc15acb0 0x7fdabc0cb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:34.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.322+0000 7fdac22cb640 1 --2- 172.21.15.55:0/3849670998 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdabc1544f0 0x7fdabc0c4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:34.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.322+0000 7fdac22cb640 1 --2- 172.21.15.55:0/3849670998 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdabc06b2a0 0x7fdabc157270 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:34.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.322+0000 7fdac22cb640 1 -- 172.21.15.55:0/3849670998 >> 172.21.15.55:0/3849670998 conn(0x7fdabc06e670 msgr2=0x7fdabc06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:34.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.322+0000 7fdac22cb640 1 -- 172.21.15.55:0/3849670998 shutdown_connections 2024-03-19T01:51:34.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.322+0000 7fdac22cb640 1 -- 172.21.15.55:0/3849670998 wait complete. 2024-03-19T01:51:34.344 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:51:34.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.474+0000 7f55a483a640 1 Processor -- start 2024-03-19T01:51:34.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.474+0000 7f55a483a640 1 -- start start 2024-03-19T01:51:34.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.474+0000 7f55a483a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f55a01510f0 0x7f55a00c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:34.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.474+0000 7f55a483a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f55a015a410 0x7f55a00708a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:34.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.474+0000 7f55a483a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55a00c3f60 con 0x7f55a00c73c0 2024-03-19T01:51:34.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.474+0000 7f55a483a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f55a01514d0 con 0x7f55a015a410 2024-03-19T01:51:34.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.474+0000 7f55a483a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f55a00c6870 con 0x7f55a01510f0 2024-03-19T01:51:34.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.474+0000 7f5597fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f55a015a410 0x7f55a00708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:34.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.474+0000 7f5597fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f55a015a410 0x7f55a00708a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50648/0 (socket says 172.21.15.55:50648) 2024-03-19T01:51:34.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.474+0000 7f5597fff640 1 -- 172.21.15.55:0/2582881249 learned_addr learned my addr 172.21.15.55:0/2582881249 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:34.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.474+0000 7f559e59c640 1 --2- 172.21.15.55:0/2582881249 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f55a01510f0 0x7f55a00c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:34.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.478+0000 7f5597fff640 1 -- 172.21.15.55:0/2582881249 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f55a01510f0 msgr2=0x7f55a00c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:34.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.478+0000 7f5597fff640 1 --2- 172.21.15.55:0/2582881249 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f55a01510f0 0x7f55a00c61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:34.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.478+0000 7f5597fff640 1 -- 172.21.15.55:0/2582881249 >> v1:172.21.15.84:6789/0 conn(0x7f55a00c73c0 legacy=0x7f55a0072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:34.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.478+0000 7f5597fff640 1 -- 172.21.15.55:0/2582881249 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a00c6b50 con 0x7f55a015a410 2024-03-19T01:51:34.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.482+0000 7f5597fff640 1 --2- 172.21.15.55:0/2582881249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55a015a410 0x7f55a00708a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5588009870 tx=0x7f558801ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=426bb79bd218f02d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:34.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.486+0000 7f559dd9b640 1 -- 172.21.15.55:0/2582881249 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55880b5070 con 0x7f55a015a410 2024-03-19T01:51:34.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.486+0000 7f559dd9b640 1 -- 172.21.15.55:0/2582881249 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55880a8a50 con 0x7f55a015a410 2024-03-19T01:51:34.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.486+0000 7f559dd9b640 1 -- 172.21.15.55:0/2582881249 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55880a8e20 con 0x7f55a015a410 2024-03-19T01:51:34.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.486+0000 7f55a483a640 1 -- 172.21.15.55:0/2582881249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55a015a410 msgr2=0x7f55a00708a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:34.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.486+0000 7f55a483a640 1 --2- 172.21.15.55:0/2582881249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55a015a410 0x7f55a00708a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5588009870 tx=0x7f558801ed50 comp rx=0 tx=0).stop 2024-03-19T01:51:34.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.486+0000 7f55a483a640 1 -- 172.21.15.55:0/2582881249 shutdown_connections 2024-03-19T01:51:34.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.490+0000 7f55a483a640 1 --2- 172.21.15.55:0/2582881249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55a015a410 0x7f55a00708a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:34.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.490+0000 7f55a483a640 1 --2- 172.21.15.55:0/2582881249 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f55a01510f0 0x7f55a00c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:34.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.490+0000 7f55a483a640 1 -- 172.21.15.55:0/2582881249 >> 172.21.15.55:0/2582881249 conn(0x7f55a00cc9c0 msgr2=0x7f55a00ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:34.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.490+0000 7f55a483a640 1 -- 172.21.15.55:0/2582881249 shutdown_connections 2024-03-19T01:51:34.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.490+0000 7f55a483a640 1 -- 172.21.15.55:0/2582881249 wait complete. 2024-03-19T01:51:34.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.490+0000 7f55a483a640 1 Processor -- start 2024-03-19T01:51:34.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.490+0000 7f55a483a640 1 -- start start 2024-03-19T01:51:34.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.490+0000 7f55a483a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55a00c73c0 0x7f55a00c37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:34.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.490+0000 7f55a483a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f55a01510f0 0x7f55a0148640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:34.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.490+0000 7f55a483a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f55a015a410 0x7f55a014bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:34.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.490+0000 7f55a483a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f55a00c6e30 con 0x7f55a00c73c0 2024-03-19T01:51:34.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.490+0000 7f55a483a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f55a00c44e0 con 0x7f55a01510f0 2024-03-19T01:51:34.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.490+0000 7f55a483a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f55a00c4a00 con 0x7f55a015a410 2024-03-19T01:51:34.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.490+0000 7f559e59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55a00c73c0 0x7f55a00c37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:34.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.494+0000 7f559e59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55a00c73c0 0x7f55a00c37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50650/0 (socket says 172.21.15.55:50650) 2024-03-19T01:51:34.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.494+0000 7f559e59c640 1 -- 172.21.15.55:0/542174541 learned_addr learned my addr 172.21.15.55:0/542174541 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:34.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.494+0000 7f5597fff640 1 --2- 172.21.15.55:0/542174541 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f55a01510f0 0x7f55a0148640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:34.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.494+0000 7f559ed9d640 1 --2- 172.21.15.55:0/542174541 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f55a015a410 0x7f55a014bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:34.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.494+0000 7f559e59c640 1 -- 172.21.15.55:0/542174541 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f55a015a410 msgr2=0x7f55a014bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:34.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.494+0000 7f559e59c640 1 --2- 172.21.15.55:0/542174541 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f55a015a410 0x7f55a014bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:34.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.494+0000 7f559e59c640 1 -- 172.21.15.55:0/542174541 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f55a01510f0 msgr2=0x7f55a0148640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:34.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.494+0000 7f559e59c640 1 --2- 172.21.15.55:0/542174541 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f55a01510f0 0x7f55a0148640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:34.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.494+0000 7f559e59c640 1 -- 172.21.15.55:0/542174541 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a014deb0 con 0x7f55a00c73c0 2024-03-19T01:51:34.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.498+0000 7f559e59c640 1 --2- 172.21.15.55:0/542174541 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55a00c73c0 0x7f55a00c37e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f559000d800 tx=0x7f559000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:34.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.502+0000 7f5596ffd640 1 -- 172.21.15.55:0/542174541 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559000b6c0 con 0x7f55a00c73c0 2024-03-19T01:51:34.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.502+0000 7f5596ffd640 1 -- 172.21.15.55:0/542174541 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5590019070 con 0x7f55a00c73c0 2024-03-19T01:51:34.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.502+0000 7f5596ffd640 1 -- 172.21.15.55:0/542174541 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559000ba60 con 0x7f55a00c73c0 2024-03-19T01:51:34.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.502+0000 7f55a483a640 1 -- 172.21.15.55:0/542174541 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55a01f3320 con 0x7f55a00c73c0 2024-03-19T01:51:34.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.502+0000 7f55a483a640 1 -- 172.21.15.55:0/542174541 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55a014e150 con 0x7f55a00c73c0 2024-03-19T01:51:34.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.502+0000 7f55a483a640 1 -- 172.21.15.55:0/542174541 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55a0068a40 con 0x7f55a00c73c0 2024-03-19T01:51:34.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.522+0000 7f5596ffd640 1 -- 172.21.15.55:0/542174541 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f559000bc00 con 0x7f55a00c73c0 2024-03-19T01:51:34.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.522+0000 7f5596ffd640 1 --2- 172.21.15.55:0/542174541 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f55800676b0 0x7f5580069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:34.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.526+0000 7f5597fff640 1 --2- 172.21.15.55:0/542174541 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f55800676b0 0x7f5580069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:34.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.526+0000 7f5597fff640 1 --2- 172.21.15.55:0/542174541 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f55800676b0 0x7f5580069b70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f55880b2000 tx=0x7f55880aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:34.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.798+0000 7f5596ffd640 1 -- 172.21.15.55:0/542174541 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f5590054d50 con 0x7f55a00c73c0 2024-03-19T01:51:34.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.798+0000 7f5596ffd640 1 -- 172.21.15.55:0/542174541 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f5590056cb0 con 0x7f55a00c73c0 2024-03-19T01:51:34.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.982+0000 7f55a483a640 1 -- 172.21.15.55:0/542174541 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f55a014bb90 con 0x7f55a00c73c0 2024-03-19T01:51:34.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.986+0000 7f5596ffd640 1 -- 172.21.15.55:0/542174541 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) ==== 104+0+11 (secure 0 0 0) 0x7f559005ad50 con 0x7f55a00c73c0 2024-03-19T01:51:34.987 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 32 2024-03-19T01:51:34.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.990+0000 7f55a483a640 1 -- 172.21.15.55:0/542174541 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f55800676b0 msgr2=0x7f5580069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:34.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.990+0000 7f55a483a640 1 --2- 172.21.15.55:0/542174541 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f55800676b0 0x7f5580069b70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f55880b2000 tx=0x7f55880aa070 comp rx=0 tx=0).stop 2024-03-19T01:51:34.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.990+0000 7f55a483a640 1 -- 172.21.15.55:0/542174541 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55a00c73c0 msgr2=0x7f55a00c37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:34.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.990+0000 7f55a483a640 1 --2- 172.21.15.55:0/542174541 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55a00c73c0 0x7f55a00c37e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f559000d800 tx=0x7f559000dcc0 comp rx=0 tx=0).stop 2024-03-19T01:51:34.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.990+0000 7f55a483a640 1 -- 172.21.15.55:0/542174541 shutdown_connections 2024-03-19T01:51:34.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.990+0000 7f55a483a640 1 --2- 172.21.15.55:0/542174541 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f55800676b0 0x7f5580069b70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:34.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.990+0000 7f55a483a640 1 --2- 172.21.15.55:0/542174541 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f55a015a410 0x7f55a014bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:34.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.990+0000 7f55a483a640 1 --2- 172.21.15.55:0/542174541 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f55a01510f0 0x7f55a0148640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:34.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.990+0000 7f55a483a640 1 --2- 172.21.15.55:0/542174541 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55a00c73c0 0x7f55a00c37e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:34.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.990+0000 7f55a483a640 1 -- 172.21.15.55:0/542174541 >> 172.21.15.55:0/542174541 conn(0x7f55a00cc9c0 msgr2=0x7f55a014a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:34.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.990+0000 7f55a483a640 1 -- 172.21.15.55:0/542174541 shutdown_connections 2024-03-19T01:51:34.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:34.990+0000 7f55a483a640 1 -- 172.21.15.55:0/542174541 wait complete. 2024-03-19T01:51:35.005 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:51:35.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.134+0000 7f4617da3640 1 Processor -- start 2024-03-19T01:51:35.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.134+0000 7f4617da3640 1 -- start start 2024-03-19T01:51:35.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.138+0000 7f4617da3640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f46100c96b0 0x7f4610070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:35.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.138+0000 7f4617da3640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f46101540c0 0x7f4610154550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:35.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.138+0000 7f4617da3640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f461015e3a0 con 0x7f46100c9a90 2024-03-19T01:51:35.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.138+0000 7f4617da3640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f46100c7ad0 con 0x7f46101540c0 2024-03-19T01:51:35.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.138+0000 7f4617da3640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f461015e680 con 0x7f46100c96b0 2024-03-19T01:51:35.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.138+0000 7f4616340640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f46101540c0 0x7f4610154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:35.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.138+0000 7f4616340640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f46101540c0 0x7f4610154550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50654/0 (socket says 172.21.15.55:50654) 2024-03-19T01:51:35.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.138+0000 7f4616340640 1 -- 172.21.15.55:0/2609950868 learned_addr learned my addr 172.21.15.55:0/2609950868 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:35.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.138+0000 7f4615b3f640 1 --2- 172.21.15.55:0/2609950868 >> v2:172.21.15.84:3300/0 conn(0x7f46100c96b0 0x7f4610070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:35.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.142+0000 7f4616340640 1 -- 172.21.15.55:0/2609950868 >> v2:172.21.15.84:3300/0 conn(0x7f46100c96b0 msgr2=0x7f4610070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:35.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.142+0000 7f4616340640 1 --2- 172.21.15.55:0/2609950868 >> v2:172.21.15.84:3300/0 conn(0x7f46100c96b0 0x7f4610070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:35.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.142+0000 7f4616340640 1 -- 172.21.15.55:0/2609950868 >> v1:172.21.15.55:6789/0 conn(0x7f46100c9a90 legacy=0x7f46100c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:35.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.142+0000 7f4616340640 1 -- 172.21.15.55:0/2609950868 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461015a330 con 0x7f46101540c0 2024-03-19T01:51:35.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.146+0000 7f4616340640 1 --2- 172.21.15.55:0/2609950868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46101540c0 0x7f4610154550 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4610072b40 tx=0x7f460001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cdf55a5e1647c51 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:35.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.150+0000 7f4614b3d640 1 -- 172.21.15.55:0/2609950868 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46000a5710 con 0x7f46101540c0 2024-03-19T01:51:35.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.150+0000 7f4614b3d640 1 -- 172.21.15.55:0/2609950868 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46000a58b0 con 0x7f46101540c0 2024-03-19T01:51:35.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.150+0000 7f4614b3d640 1 -- 172.21.15.55:0/2609950868 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46000a5c80 con 0x7f46101540c0 2024-03-19T01:51:35.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.154+0000 7f4617da3640 1 -- 172.21.15.55:0/2609950868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46101540c0 msgr2=0x7f4610154550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:35.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.154+0000 7f4617da3640 1 --2- 172.21.15.55:0/2609950868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46101540c0 0x7f4610154550 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4610072b40 tx=0x7f460001c960 comp rx=0 tx=0).stop 2024-03-19T01:51:35.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.154+0000 7f4617da3640 1 -- 172.21.15.55:0/2609950868 shutdown_connections 2024-03-19T01:51:35.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.154+0000 7f4617da3640 1 --2- 172.21.15.55:0/2609950868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46101540c0 0x7f4610154550 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:35.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.154+0000 7f4617da3640 1 --2- 172.21.15.55:0/2609950868 >> v2:172.21.15.84:3300/0 conn(0x7f46100c96b0 0x7f4610070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:35.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.154+0000 7f4617da3640 1 -- 172.21.15.55:0/2609950868 >> 172.21.15.55:0/2609950868 conn(0x7f46100cc9c0 msgr2=0x7f46100ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:35.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.154+0000 7f4617da3640 1 -- 172.21.15.55:0/2609950868 shutdown_connections 2024-03-19T01:51:35.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.154+0000 7f4617da3640 1 -- 172.21.15.55:0/2609950868 wait complete. 2024-03-19T01:51:35.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.154+0000 7f4617da3640 1 Processor -- start 2024-03-19T01:51:35.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.154+0000 7f4617da3640 1 -- start start 2024-03-19T01:51:35.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.154+0000 7f4617da3640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46100c96b0 0x7f4610154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:35.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.154+0000 7f4617da3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46100c9a90 0x7f4610148af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:35.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.154+0000 7f4617da3640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46101540c0 0x7f46101496a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:35.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.154+0000 7f4617da3640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4610166b90 con 0x7f46100c9a90 2024-03-19T01:51:35.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.154+0000 7f4617da3640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f461015b0f0 con 0x7f46101540c0 2024-03-19T01:51:35.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.154+0000 7f4617da3640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4610166d10 con 0x7f46100c96b0 2024-03-19T01:51:35.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.154+0000 7f461533e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46100c9a90 0x7f4610148af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:35.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.154+0000 7f461533e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46100c9a90 0x7f4610148af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50670/0 (socket says 172.21.15.55:50670) 2024-03-19T01:51:35.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.154+0000 7f461533e640 1 -- 172.21.15.55:0/465334148 learned_addr learned my addr 172.21.15.55:0/465334148 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:35.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.158+0000 7f4616340640 1 --2- 172.21.15.55:0/465334148 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46101540c0 0x7f46101496a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:35.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.158+0000 7f4615b3f640 1 --2- 172.21.15.55:0/465334148 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46100c96b0 0x7f4610154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:35.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.158+0000 7f461533e640 1 -- 172.21.15.55:0/465334148 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46100c96b0 msgr2=0x7f4610154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:35.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.158+0000 7f461533e640 1 --2- 172.21.15.55:0/465334148 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46100c96b0 0x7f4610154530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:35.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.158+0000 7f461533e640 1 -- 172.21.15.55:0/465334148 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46101540c0 msgr2=0x7f46101496a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:35.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.158+0000 7f461533e640 1 --2- 172.21.15.55:0/465334148 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46101540c0 0x7f46101496a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:35.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.158+0000 7f461533e640 1 -- 172.21.15.55:0/465334148 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4610149030 con 0x7f46100c9a90 2024-03-19T01:51:35.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.162+0000 7f461533e640 1 --2- 172.21.15.55:0/465334148 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46100c9a90 0x7f4610148af0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f460400c340 tx=0x7f460400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:35.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.166+0000 7f460e7fc640 1 -- 172.21.15.55:0/465334148 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46040052d0 con 0x7f46100c9a90 2024-03-19T01:51:35.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.166+0000 7f460e7fc640 1 -- 172.21.15.55:0/465334148 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4604005470 con 0x7f46100c9a90 2024-03-19T01:51:35.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.166+0000 7f460e7fc640 1 -- 172.21.15.55:0/465334148 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4604007260 con 0x7f46100c9a90 2024-03-19T01:51:35.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.166+0000 7f4617da3640 1 -- 172.21.15.55:0/465334148 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f461014e5d0 con 0x7f46100c9a90 2024-03-19T01:51:35.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.166+0000 7f4617da3640 1 -- 172.21.15.55:0/465334148 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4610146bf0 con 0x7f46100c9a90 2024-03-19T01:51:35.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.166+0000 7f4617da3640 1 -- 172.21.15.55:0/465334148 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4610068a40 con 0x7f46100c9a90 2024-03-19T01:51:35.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.450+0000 7f460e7fc640 1 -- 172.21.15.55:0/465334148 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f4604007400 con 0x7f46100c9a90 2024-03-19T01:51:35.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.450+0000 7f460e7fc640 1 --2- 172.21.15.55:0/465334148 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f45fc067660 0x7f45fc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:35.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.450+0000 7f4615b3f640 1 --2- 172.21.15.55:0/465334148 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f45fc067660 0x7f45fc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:35.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.450+0000 7f4615b3f640 1 --2- 172.21.15.55:0/465334148 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f45fc067660 0x7f45fc069b20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4608009a50 tx=0x7f4608008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:35.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.474+0000 7f460e7fc640 1 -- 172.21.15.55:0/465334148 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f4604053840 con 0x7f46100c9a90 2024-03-19T01:51:35.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.474+0000 7f460e7fc640 1 -- 172.21.15.55:0/465334148 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4604055ea0 con 0x7f46100c9a90 2024-03-19T01:51:35.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.666+0000 7f4617da3640 1 -- 172.21.15.55:0/465334148 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f46100cd080 con 0x7f46100c9a90 2024-03-19T01:51:35.680 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.678+0000 7f460e7fc640 1 -- 172.21.15.55:0/465334148 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) ==== 100+0+11 (secure 0 0 0) 0x7f46040557a0 con 0x7f46100c9a90 2024-03-19T01:51:35.680 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 64 2024-03-19T01:51:35.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.682+0000 7f4617da3640 1 -- 172.21.15.55:0/465334148 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f45fc067660 msgr2=0x7f45fc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:35.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.682+0000 7f4617da3640 1 --2- 172.21.15.55:0/465334148 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f45fc067660 0x7f45fc069b20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4608009a50 tx=0x7f4608008040 comp rx=0 tx=0).stop 2024-03-19T01:51:35.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.682+0000 7f4617da3640 1 -- 172.21.15.55:0/465334148 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46100c9a90 msgr2=0x7f4610148af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:35.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.682+0000 7f4617da3640 1 --2- 172.21.15.55:0/465334148 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46100c9a90 0x7f4610148af0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f460400c340 tx=0x7f460400c800 comp rx=0 tx=0).stop 2024-03-19T01:51:35.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.682+0000 7f4617da3640 1 -- 172.21.15.55:0/465334148 shutdown_connections 2024-03-19T01:51:35.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.682+0000 7f4617da3640 1 --2- 172.21.15.55:0/465334148 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f45fc067660 0x7f45fc069b20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:35.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.682+0000 7f4617da3640 1 --2- 172.21.15.55:0/465334148 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46101540c0 0x7f46101496a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:35.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.682+0000 7f4617da3640 1 --2- 172.21.15.55:0/465334148 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46100c9a90 0x7f4610148af0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:35.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.682+0000 7f4617da3640 1 --2- 172.21.15.55:0/465334148 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46100c96b0 0x7f4610154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:35.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.682+0000 7f4617da3640 1 -- 172.21.15.55:0/465334148 >> 172.21.15.55:0/465334148 conn(0x7f46100cc9c0 msgr2=0x7f46100c7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:35.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.682+0000 7f4617da3640 1 -- 172.21.15.55:0/465334148 shutdown_connections 2024-03-19T01:51:35.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.682+0000 7f4617da3640 1 -- 172.21.15.55:0/465334148 wait complete. 2024-03-19T01:51:35.699 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T01:51:35.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.814+0000 7fadf9217640 1 Processor -- start 2024-03-19T01:51:35.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.814+0000 7fadf9217640 1 -- start start 2024-03-19T01:51:35.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.818+0000 7fadf9217640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fadf4152120 0x7fadf40c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:35.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.818+0000 7fadf9217640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fadf40c9610 0x7fadf4157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:35.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.818+0000 7fadf9217640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fadf4070780 0x7fadf4158510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:35.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.818+0000 7fadf9217640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fadf4152500 con 0x7fadf4070780 2024-03-19T01:51:35.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.818+0000 7fadf9217640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fadf4070590 con 0x7fadf4152120 2024-03-19T01:51:35.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.818+0000 7fadf9217640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fadf40c3530 con 0x7fadf40c9610 2024-03-19T01:51:35.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.834+0000 7fadf259c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fadf40c9610 0x7fadf4157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:35.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.838+0000 7fadf259c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fadf40c9610 0x7fadf4157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:55274/0 (socket says 172.21.15.55:55274) 2024-03-19T01:51:35.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.838+0000 7fadf259c640 1 -- 172.21.15.55:0/3892576586 learned_addr learned my addr 172.21.15.55:0/3892576586 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:35.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.842+0000 7fadf259c640 1 -- 172.21.15.55:0/3892576586 >> v2:172.21.15.84:3300/0 conn(0x7fadf4152120 msgr2=0x7fadf40c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:35.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.842+0000 7fadf259c640 1 --2- 172.21.15.55:0/3892576586 >> v2:172.21.15.84:3300/0 conn(0x7fadf4152120 0x7fadf40c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:35.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.842+0000 7fadf259c640 1 -- 172.21.15.55:0/3892576586 >> v2:172.21.15.55:3300/0 conn(0x7fadf4070780 msgr2=0x7fadf4158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:35.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.842+0000 7fadf259c640 1 --2- 172.21.15.55:0/3892576586 >> v2:172.21.15.55:3300/0 conn(0x7fadf4070780 0x7fadf4158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:35.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.842+0000 7fadf259c640 1 -- 172.21.15.55:0/3892576586 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadf406fff0 con 0x7fadf40c9610 2024-03-19T01:51:35.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.854+0000 7fadf259c640 1 --2- 172.21.15.55:0/3892576586 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fadf40c9610 0x7fadf4157fd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fadf4072430 tx=0x7fade801c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e16903ad6540505b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:35.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.862+0000 7fadf1d9b640 1 -- 172.21.15.55:0/3892576586 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade80b0070 con 0x7fadf40c9610 2024-03-19T01:51:35.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.862+0000 7fadf1d9b640 1 -- 172.21.15.55:0/3892576586 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fade80ab600 con 0x7fadf40c9610 2024-03-19T01:51:35.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.862+0000 7fadf1d9b640 1 -- 172.21.15.55:0/3892576586 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade80ab9d0 con 0x7fadf40c9610 2024-03-19T01:51:35.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.866+0000 7fadf9217640 1 -- 172.21.15.55:0/3892576586 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fadf40c9610 msgr2=0x7fadf4157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:35.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.866+0000 7fadf9217640 1 --2- 172.21.15.55:0/3892576586 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fadf40c9610 0x7fadf4157fd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fadf4072430 tx=0x7fade801c520 comp rx=0 tx=0).stop 2024-03-19T01:51:35.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.866+0000 7fadf9217640 1 -- 172.21.15.55:0/3892576586 shutdown_connections 2024-03-19T01:51:35.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.866+0000 7fadf9217640 1 --2- 172.21.15.55:0/3892576586 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fadf40c9610 0x7fadf4157fd0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:35.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.866+0000 7fadf9217640 1 --2- 172.21.15.55:0/3892576586 >> v2:172.21.15.55:3300/0 conn(0x7fadf4070780 0x7fadf4158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:35.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.866+0000 7fadf9217640 1 --2- 172.21.15.55:0/3892576586 >> v2:172.21.15.84:3300/0 conn(0x7fadf4152120 0x7fadf40c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:35.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.866+0000 7fadf9217640 1 -- 172.21.15.55:0/3892576586 >> 172.21.15.55:0/3892576586 conn(0x7fadf4067600 msgr2=0x7fadf4071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:35.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.866+0000 7fadf9217640 1 -- 172.21.15.55:0/3892576586 shutdown_connections 2024-03-19T01:51:35.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.866+0000 7fadf9217640 1 -- 172.21.15.55:0/3892576586 wait complete. 2024-03-19T01:51:35.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.866+0000 7fadf9217640 1 Processor -- start 2024-03-19T01:51:35.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.866+0000 7fadf9217640 1 -- start start 2024-03-19T01:51:35.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.866+0000 7fadf9217640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fadf4070780 0x7fadf40b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:35.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.866+0000 7fadf9217640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fadf40c9610 0x7fadf406cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:35.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.866+0000 7fadf9217640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fadf4152120 0x7fadf406d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:35.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.866+0000 7fadf9217640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fadf40c31c0 con 0x7fadf4070780 2024-03-19T01:51:35.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.866+0000 7fadf9217640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fadf40c2ee0 con 0x7fadf4152120 2024-03-19T01:51:35.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.866+0000 7fadf9217640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fadf40c86d0 con 0x7fadf40c9610 2024-03-19T01:51:35.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.866+0000 7fadf2d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fadf4070780 0x7fadf40b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:35.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.870+0000 7fadf359e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fadf4152120 0x7fadf406d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:35.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.870+0000 7fadf359e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fadf4152120 0x7fadf406d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:44160/0 (socket says 172.21.15.55:44160) 2024-03-19T01:51:35.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.870+0000 7fadf359e640 1 -- 172.21.15.55:0/2408862305 learned_addr learned my addr 172.21.15.55:0/2408862305 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:35.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.870+0000 7fadf259c640 1 --2- 172.21.15.55:0/2408862305 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fadf40c9610 0x7fadf406cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:35.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.874+0000 7fadf359e640 1 -- 172.21.15.55:0/2408862305 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fadf40c9610 msgr2=0x7fadf406cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:35.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.874+0000 7fadf359e640 1 --2- 172.21.15.55:0/2408862305 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fadf40c9610 0x7fadf406cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:35.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.874+0000 7fadf359e640 1 -- 172.21.15.55:0/2408862305 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fadf4070780 msgr2=0x7fadf40b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:35.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.874+0000 7fadf359e640 1 --2- 172.21.15.55:0/2408862305 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fadf4070780 0x7fadf40b20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:35.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.874+0000 7fadf359e640 1 -- 172.21.15.55:0/2408862305 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadf41f3990 con 0x7fadf4152120 2024-03-19T01:51:35.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.878+0000 7fadf359e640 1 --2- 172.21.15.55:0/2408862305 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fadf4152120 0x7fadf406d0d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fadec00b9a0 tx=0x7fadec00be60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:35.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.886+0000 7faddb7fe640 1 -- 172.21.15.55:0/2408862305 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadec019070 con 0x7fadf4152120 2024-03-19T01:51:35.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.886+0000 7faddb7fe640 1 -- 172.21.15.55:0/2408862305 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadec00cbc0 con 0x7fadf4152120 2024-03-19T01:51:35.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.886+0000 7faddb7fe640 1 -- 172.21.15.55:0/2408862305 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadec014440 con 0x7fadf4152120 2024-03-19T01:51:35.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.886+0000 7fadf9217640 1 -- 172.21.15.55:0/2408862305 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadf41f3c70 con 0x7fadf4152120 2024-03-19T01:51:35.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.886+0000 7fadf9217640 1 -- 172.21.15.55:0/2408862305 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadf41f4230 con 0x7fadf4152120 2024-03-19T01:51:35.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.886+0000 7fadf9217640 1 -- 172.21.15.55:0/2408862305 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadf4068a40 con 0x7fadf4152120 2024-03-19T01:51:35.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.938+0000 7faddb7fe640 1 -- 172.21.15.55:0/2408862305 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fadec01c070 con 0x7fadf4152120 2024-03-19T01:51:35.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.938+0000 7faddb7fe640 1 --2- 172.21.15.55:0/2408862305 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fade00676b0 0x7fade0069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:35.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.938+0000 7fadf2d9d640 1 --2- 172.21.15.55:0/2408862305 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fade00676b0 0x7fade0069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:35.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:35.938+0000 7fadf2d9d640 1 --2- 172.21.15.55:0/2408862305 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fade00676b0 0x7fade0069b70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fade40098f0 tx=0x7fade4006f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:36.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.234+0000 7faddb7fe640 1 -- 172.21.15.55:0/2408862305 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fadec054130 con 0x7fadf4152120 2024-03-19T01:51:36.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.262+0000 7faddb7fe640 1 -- 172.21.15.55:0/2408862305 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fadec023020 con 0x7fadf4152120 2024-03-19T01:51:36.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.442+0000 7fadf9217640 1 -- 172.21.15.55:0/2408862305 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fadf4070db0 con 0x7fadf4152120 2024-03-19T01:51:36.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.502+0000 7faddb7fe640 1 -- 172.21.15.55:0/2408862305 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) ==== 103+0+11 (secure 0 0 0) 0x7fadec056090 con 0x7fadf4152120 2024-03-19T01:51:36.502 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 32 2024-03-19T01:51:36.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.502+0000 7fadf9217640 1 -- 172.21.15.55:0/2408862305 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fade00676b0 msgr2=0x7fade0069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:36.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.502+0000 7fadf9217640 1 --2- 172.21.15.55:0/2408862305 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fade00676b0 0x7fade0069b70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fade40098f0 tx=0x7fade4006f90 comp rx=0 tx=0).stop 2024-03-19T01:51:36.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.506+0000 7fadf9217640 1 -- 172.21.15.55:0/2408862305 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fadf4152120 msgr2=0x7fadf406d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:36.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.506+0000 7fadf9217640 1 --2- 172.21.15.55:0/2408862305 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fadf4152120 0x7fadf406d0d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fadec00b9a0 tx=0x7fadec00be60 comp rx=0 tx=0).stop 2024-03-19T01:51:36.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.506+0000 7fadf9217640 1 -- 172.21.15.55:0/2408862305 shutdown_connections 2024-03-19T01:51:36.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.506+0000 7fadf9217640 1 --2- 172.21.15.55:0/2408862305 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fade00676b0 0x7fade0069b70 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:36.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.506+0000 7fadf9217640 1 --2- 172.21.15.55:0/2408862305 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fadf4152120 0x7fadf406d0d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:36.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.506+0000 7fadf9217640 1 --2- 172.21.15.55:0/2408862305 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fadf40c9610 0x7fadf406cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:36.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.506+0000 7fadf9217640 1 --2- 172.21.15.55:0/2408862305 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fadf4070780 0x7fadf40b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:36.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.506+0000 7fadf9217640 1 -- 172.21.15.55:0/2408862305 >> 172.21.15.55:0/2408862305 conn(0x7fadf4067600 msgr2=0x7fadf4146580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:36.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.506+0000 7fadf9217640 1 -- 172.21.15.55:0/2408862305 shutdown_connections 2024-03-19T01:51:36.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.506+0000 7fadf9217640 1 -- 172.21.15.55:0/2408862305 wait complete. 2024-03-19T01:51:36.522 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-19T01:51:36.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.634+0000 7fcae6149640 1 Processor -- start 2024-03-19T01:51:36.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.634+0000 7fcae6149640 1 -- start start 2024-03-19T01:51:36.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.634+0000 7fcae6149640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fcae013dd00 0x7fcae00c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:36.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.634+0000 7fcae6149640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fcae015b550 0x7fcae00c5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:36.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.634+0000 7fcae6149640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcae00c8f70 con 0x7fcae00c9780 2024-03-19T01:51:36.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.634+0000 7fcae6149640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fcae00c9250 con 0x7fcae013dd00 2024-03-19T01:51:36.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.634+0000 7fcae6149640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fcae01582b0 con 0x7fcae015b550 2024-03-19T01:51:36.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.634+0000 7fcadf7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fcae013dd00 0x7fcae00c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:36.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.634+0000 7fcadf7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fcae013dd00 0x7fcae00c7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50688/0 (socket says 172.21.15.55:50688) 2024-03-19T01:51:36.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.634+0000 7fcadf7fe640 1 -- 172.21.15.55:0/1129885436 learned_addr learned my addr 172.21.15.55:0/1129885436 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:36.636 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.634+0000 7fcadffff640 1 --2- 172.21.15.55:0/1129885436 >> v2:172.21.15.84:3300/0 conn(0x7fcae015b550 0x7fcae00c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:36.639 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.638+0000 7fcadf7fe640 1 -- 172.21.15.55:0/1129885436 >> v2:172.21.15.84:3300/0 conn(0x7fcae015b550 msgr2=0x7fcae00c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:36.639 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.638+0000 7fcadf7fe640 1 --2- 172.21.15.55:0/1129885436 >> v2:172.21.15.84:3300/0 conn(0x7fcae015b550 0x7fcae00c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:36.639 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.638+0000 7fcadf7fe640 1 -- 172.21.15.55:0/1129885436 >> v1:172.21.15.55:6789/0 conn(0x7fcae00c9780 legacy=0x7fcae00c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:36.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.638+0000 7fcadf7fe640 1 -- 172.21.15.55:0/1129885436 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcae0158590 con 0x7fcae013dd00 2024-03-19T01:51:36.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.650+0000 7fcadf7fe640 1 --2- 172.21.15.55:0/1129885436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcae013dd00 0x7fcae00c7430 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcae0155b30 tx=0x7fcad001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc8d9af8773fca86 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:36.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.654+0000 7fcade7fc640 1 -- 172.21.15.55:0/1129885436 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad00ab430 con 0x7fcae013dd00 2024-03-19T01:51:36.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.654+0000 7fcade7fc640 1 -- 172.21.15.55:0/1129885436 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcad00b0070 con 0x7fcae013dd00 2024-03-19T01:51:36.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.654+0000 7fcade7fc640 1 -- 172.21.15.55:0/1129885436 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad00ab7f0 con 0x7fcae013dd00 2024-03-19T01:51:36.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.654+0000 7fcae6149640 1 -- 172.21.15.55:0/1129885436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcae013dd00 msgr2=0x7fcae00c7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:36.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.654+0000 7fcae6149640 1 --2- 172.21.15.55:0/1129885436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcae013dd00 0x7fcae00c7430 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcae0155b30 tx=0x7fcad001c520 comp rx=0 tx=0).stop 2024-03-19T01:51:36.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.654+0000 7fcae6149640 1 -- 172.21.15.55:0/1129885436 shutdown_connections 2024-03-19T01:51:36.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.654+0000 7fcae6149640 1 --2- 172.21.15.55:0/1129885436 >> v2:172.21.15.84:3300/0 conn(0x7fcae015b550 0x7fcae00c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:36.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.654+0000 7fcae6149640 1 --2- 172.21.15.55:0/1129885436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcae013dd00 0x7fcae00c7430 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:36.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.654+0000 7fcae6149640 1 -- 172.21.15.55:0/1129885436 >> 172.21.15.55:0/1129885436 conn(0x7fcae006e5a0 msgr2=0x7fcae0072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:36.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.654+0000 7fcae6149640 1 -- 172.21.15.55:0/1129885436 shutdown_connections 2024-03-19T01:51:36.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.654+0000 7fcae6149640 1 -- 172.21.15.55:0/1129885436 wait complete. 2024-03-19T01:51:36.657 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.658+0000 7fcae6149640 1 Processor -- start 2024-03-19T01:51:36.657 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.658+0000 7fcae6149640 1 -- start start 2024-03-19T01:51:36.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.658+0000 7fcae6149640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcae00c9780 0x7fcae00c4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:36.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.658+0000 7fcae6149640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcae013dd00 0x7fcae0160de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:36.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.658+0000 7fcae6149640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcae015b550 0x7fcae0161320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:36.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.658+0000 7fcae6149640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcae0165280 con 0x7fcae013dd00 2024-03-19T01:51:36.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.658+0000 7fcae6149640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fcae0165100 con 0x7fcae015b550 2024-03-19T01:51:36.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.658+0000 7fcae6149640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fcae0165400 con 0x7fcae00c9780 2024-03-19T01:51:36.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.658+0000 7fcadeffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcae013dd00 0x7fcae0160de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:36.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.658+0000 7fcadeffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcae013dd00 0x7fcae0160de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50690/0 (socket says 172.21.15.55:50690) 2024-03-19T01:51:36.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.658+0000 7fcadeffd640 1 -- 172.21.15.55:0/2864273070 learned_addr learned my addr 172.21.15.55:0/2864273070 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:36.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.658+0000 7fcadffff640 1 --2- 172.21.15.55:0/2864273070 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcae015b550 0x7fcae0161320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:36.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.658+0000 7fcadf7fe640 1 --2- 172.21.15.55:0/2864273070 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcae00c9780 0x7fcae00c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:36.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.662+0000 7fcadeffd640 1 -- 172.21.15.55:0/2864273070 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcae00c9780 msgr2=0x7fcae00c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:36.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.662+0000 7fcadeffd640 1 --2- 172.21.15.55:0/2864273070 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcae00c9780 0x7fcae00c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:36.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.662+0000 7fcadeffd640 1 -- 172.21.15.55:0/2864273070 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcae015b550 msgr2=0x7fcae0161320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:36.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.662+0000 7fcadeffd640 1 --2- 172.21.15.55:0/2864273070 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcae015b550 0x7fcae0161320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:36.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.662+0000 7fcadeffd640 1 -- 172.21.15.55:0/2864273070 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcae015b0f0 con 0x7fcae013dd00 2024-03-19T01:51:36.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.666+0000 7fcadeffd640 1 --2- 172.21.15.55:0/2864273070 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcae013dd00 0x7fcae0160de0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcac800c370 tx=0x7fcac800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:36.668 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.666+0000 7fcad7fff640 1 -- 172.21.15.55:0/2864273070 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac80052d0 con 0x7fcae013dd00 2024-03-19T01:51:36.668 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.666+0000 7fcad7fff640 1 -- 172.21.15.55:0/2864273070 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcac8005470 con 0x7fcae013dd00 2024-03-19T01:51:36.668 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.666+0000 7fcae6149640 1 -- 172.21.15.55:0/2864273070 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcae015b360 con 0x7fcae013dd00 2024-03-19T01:51:36.668 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.666+0000 7fcad7fff640 1 -- 172.21.15.55:0/2864273070 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac8007260 con 0x7fcae013dd00 2024-03-19T01:51:36.668 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.666+0000 7fcae6149640 1 -- 172.21.15.55:0/2864273070 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcae01538a0 con 0x7fcae013dd00 2024-03-19T01:51:36.668 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.670+0000 7fcae6149640 1 -- 172.21.15.55:0/2864273070 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcae0155330 con 0x7fcae013dd00 2024-03-19T01:51:36.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.954+0000 7fcad7fff640 1 -- 172.21.15.55:0/2864273070 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fcac8007400 con 0x7fcae013dd00 2024-03-19T01:51:36.956 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.954+0000 7fcad7fff640 1 --2- 172.21.15.55:0/2864273070 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcac0067660 0x7fcac0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:36.956 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.954+0000 7fcadf7fe640 1 --2- 172.21.15.55:0/2864273070 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcac0067660 0x7fcac0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:36.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.958+0000 7fcadf7fe640 1 --2- 172.21.15.55:0/2864273070 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcac0067660 0x7fcac0069b20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcad00af000 tx=0x7fcad00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:36.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.990+0000 7fcad7fff640 1 -- 172.21.15.55:0/2864273070 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fcac8053880 con 0x7fcae013dd00 2024-03-19T01:51:36.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:36.994+0000 7fcad7fff640 1 -- 172.21.15.55:0/2864273070 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fcae0155330 con 0x7fcae013dd00 2024-03-19T01:51:37.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:37.178+0000 7fcae6149640 1 -- 172.21.15.55:0/2864273070 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcae0162370 con 0x7fcae013dd00 2024-03-19T01:51:37.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:37.190+0000 7fcad7fff640 1 -- 172.21.15.55:0/2864273070 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) ==== 80+0+0 (secure 0 0 0) 0x7fcac80557e0 con 0x7fcae013dd00 2024-03-19T01:51:37.191 INFO:teuthology.orchestra.run.smithi055.stderr:listed 0 entries 2024-03-19T01:51:37.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:37.194+0000 7fcae6149640 1 -- 172.21.15.55:0/2864273070 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcac0067660 msgr2=0x7fcac0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:37.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:37.194+0000 7fcae6149640 1 --2- 172.21.15.55:0/2864273070 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcac0067660 0x7fcac0069b20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcad00af000 tx=0x7fcad00a7000 comp rx=0 tx=0).stop 2024-03-19T01:51:37.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:37.194+0000 7fcae6149640 1 -- 172.21.15.55:0/2864273070 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcae013dd00 msgr2=0x7fcae0160de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:37.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:37.194+0000 7fcae6149640 1 --2- 172.21.15.55:0/2864273070 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcae013dd00 0x7fcae0160de0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcac800c370 tx=0x7fcac800c830 comp rx=0 tx=0).stop 2024-03-19T01:51:37.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:37.194+0000 7fcae6149640 1 -- 172.21.15.55:0/2864273070 shutdown_connections 2024-03-19T01:51:37.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:37.194+0000 7fcae6149640 1 --2- 172.21.15.55:0/2864273070 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcac0067660 0x7fcac0069b20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:37.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:37.194+0000 7fcae6149640 1 --2- 172.21.15.55:0/2864273070 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcae015b550 0x7fcae0161320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:37.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:37.194+0000 7fcae6149640 1 --2- 172.21.15.55:0/2864273070 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcae013dd00 0x7fcae0160de0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:37.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:37.194+0000 7fcae6149640 1 --2- 172.21.15.55:0/2864273070 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcae00c9780 0x7fcae00c4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:37.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:37.194+0000 7fcae6149640 1 -- 172.21.15.55:0/2864273070 >> 172.21.15.55:0/2864273070 conn(0x7fcae006e5a0 msgr2=0x7fcae0155d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:37.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:37.194+0000 7fcae6149640 1 -- 172.21.15.55:0/2864273070 shutdown_connections 2024-03-19T01:51:37.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:37.194+0000 7fcae6149640 1 -- 172.21.15.55:0/2864273070 wait complete. 2024-03-19T01:51:37.210 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi055.front.sepia.ceph.com... 2024-03-19T01:51:37.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:51:37.211 DEBUG:teuthology.orchestra.run.smithi055:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-19T01:51:37.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi055.stderr:ceph-fuse[33232]: fuse finished with error 0 and tester_r 0 2024-03-19T01:51:37.232 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T01:51:38.490 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:51:38.493 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:51:43.333 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi055.front.sepia.ceph.com 2024-03-19T01:51:43.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:51:43.334 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-19T01:51:43.361 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:51:43.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.525+0000 7fa53b5ff640 1 Processor -- start 2024-03-19T01:51:43.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.525+0000 7fa53b5ff640 1 -- start start 2024-03-19T01:51:43.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.525+0000 7fa53b5ff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa534154300 0x7fa53415ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:43.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.525+0000 7fa53b5ff640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5340c8840 con 0x7fa53406c110 2024-03-19T01:51:43.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.525+0000 7fa53b5ff640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5340c8ab0 con 0x7fa53406b2a0 2024-03-19T01:51:43.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.525+0000 7fa53b5ff640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa53415b260 con 0x7fa534154300 2024-03-19T01:51:43.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.525+0000 7fa539b9c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fa53406c110 0x7fa534164c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:41388/0 (socket says 172.21.15.55:41388) 2024-03-19T01:51:43.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.525+0000 7fa539b9c640 1 -- 172.21.15.55:0/2419489822 learned_addr learned my addr 172.21.15.55:0/2419489822 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:51:43.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.529+0000 7fa52bfff640 1 -- 172.21.15.55:0/2419489822 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 107241541 0 0) 0x7fa5340c8840 con 0x7fa53406c110 2024-03-19T01:51:43.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.529+0000 7fa52bfff640 1 -- 172.21.15.55:0/2419489822 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa520003530 con 0x7fa53406c110 2024-03-19T01:51:43.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.533+0000 7fa52bfff640 1 -- 172.21.15.55:0/2419489822 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3831164293 0 0) 0x7fa520003530 con 0x7fa53406c110 2024-03-19T01:51:43.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.533+0000 7fa52bfff640 1 -- 172.21.15.55:0/2419489822 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa534154300 msgr2=0x7fa53415ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:43.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.533+0000 7fa52bfff640 1 --2- 172.21.15.55:0/2419489822 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa534154300 0x7fa53415ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:43.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.533+0000 7fa52bfff640 1 -- 172.21.15.55:0/2419489822 >> v1:172.21.15.84:6789/0 conn(0x7fa53406b2a0 legacy=0x7fa53415e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:43.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.533+0000 7fa52bfff640 1 -- 172.21.15.55:0/2419489822 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5340c8d90 con 0x7fa53406c110 2024-03-19T01:51:43.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.533+0000 7fa52bfff640 1 -- 172.21.15.55:0/2419489822 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7fa52c001a00 con 0x7fa53406c110 2024-03-19T01:51:43.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.533+0000 7fa52bfff640 1 -- 172.21.15.55:0/2419489822 >> v1:172.21.15.55:6789/0 conn(0x7fa53406c110 legacy=0x7fa534164c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:43.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.533+0000 7fa52bfff640 1 --2- 172.21.15.55:0/2419489822 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa534154300 0x7fa5200039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:43.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.533+0000 7fa52bfff640 1 --2- 172.21.15.55:0/2419489822 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa520004130 0x7fa520004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:43.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.533+0000 7fa52bfff640 1 --2- 172.21.15.55:0/2419489822 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa520004e70 0x7fa52000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:43.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.533+0000 7fa52bfff640 1 -- 172.21.15.55:0/2419489822 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa53415b260 con 0x7fa520004130 2024-03-19T01:51:43.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.533+0000 7fa52bfff640 1 -- 172.21.15.55:0/2419489822 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa520005610 con 0x7fa520004e70 2024-03-19T01:51:43.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.533+0000 7fa52bfff640 1 -- 172.21.15.55:0/2419489822 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa52000bf10 con 0x7fa534154300 2024-03-19T01:51:43.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.537+0000 7fa539b9c640 1 --2- 172.21.15.55:0/2419489822 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa520004e70 0x7fa52000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:43.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.537+0000 7fa538b9a640 1 --2- 172.21.15.55:0/2419489822 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa520004130 0x7fa520004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:43.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.537+0000 7fa53939b640 1 --2- 172.21.15.55:0/2419489822 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa534154300 0x7fa5200039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:43.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.545+0000 7fa539b9c640 1 -- 172.21.15.55:0/2419489822 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa534154300 msgr2=0x7fa5200039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:43.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.545+0000 7fa539b9c640 1 --2- 172.21.15.55:0/2419489822 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa534154300 0x7fa5200039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:43.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.545+0000 7fa539b9c640 1 -- 172.21.15.55:0/2419489822 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa520004130 msgr2=0x7fa520004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:43.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.545+0000 7fa539b9c640 1 --2- 172.21.15.55:0/2419489822 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa520004130 0x7fa520004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:43.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.545+0000 7fa539b9c640 1 -- 172.21.15.55:0/2419489822 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa520003530 con 0x7fa520004e70 2024-03-19T01:51:43.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.549+0000 7fa539b9c640 1 --2- 172.21.15.55:0/2419489822 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa520004e70 0x7fa52000b9d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa52c00a7b0 tx=0x7fa52c01d2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5e05212fcdacdf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:43.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.561+0000 7fa52bfff640 1 -- 172.21.15.55:0/2419489822 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c00a3d0 con 0x7fa520004e70 2024-03-19T01:51:43.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.565+0000 7fa52bfff640 1 -- 172.21.15.55:0/2419489822 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa52c0ad630 con 0x7fa520004e70 2024-03-19T01:51:43.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.565+0000 7fa52bfff640 1 -- 172.21.15.55:0/2419489822 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c0ada60 con 0x7fa520004e70 2024-03-19T01:51:43.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.565+0000 7fa53b5ff640 1 -- 172.21.15.55:0/2419489822 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa520004e70 msgr2=0x7fa52000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:43.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.565+0000 7fa53b5ff640 1 --2- 172.21.15.55:0/2419489822 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa520004e70 0x7fa52000b9d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa52c00a7b0 tx=0x7fa52c01d2c0 comp rx=0 tx=0).stop 2024-03-19T01:51:43.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.565+0000 7fa539b9c640 1 -- 172.21.15.55:0/2419489822 reap_dead start 2024-03-19T01:51:43.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.565+0000 7fa53b5ff640 1 -- 172.21.15.55:0/2419489822 shutdown_connections 2024-03-19T01:51:43.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.565+0000 7fa53b5ff640 1 -- 172.21.15.55:0/2419489822 >> 172.21.15.55:0/2419489822 conn(0x7fa53406fb40 msgr2=0x7fa53406ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:43.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.565+0000 7fa53b5ff640 1 -- 172.21.15.55:0/2419489822 shutdown_connections 2024-03-19T01:51:43.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.565+0000 7fa53b5ff640 1 -- 172.21.15.55:0/2419489822 wait complete. 2024-03-19T01:51:43.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.565+0000 7fa53b5ff640 1 Processor -- start 2024-03-19T01:51:43.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.565+0000 7fa53b5ff640 1 -- start start 2024-03-19T01:51:43.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.565+0000 7fa53b5ff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa520004e70 0x7fa53415c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:43.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.565+0000 7fa53b5ff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa534150ea0 0x7fa5341f3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:43.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.565+0000 7fa53b5ff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5341f3d90 0x7fa5341f0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:43.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.565+0000 7fa53b5ff640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa53415d8c0 con 0x7fa520004e70 2024-03-19T01:51:43.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.565+0000 7fa53b5ff640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa5340c9350 con 0x7fa534150ea0 2024-03-19T01:51:43.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.565+0000 7fa53b5ff640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa53415dba0 con 0x7fa5341f3d90 2024-03-19T01:51:43.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.569+0000 7fa539b9c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5341f3d90 0x7fa5341f0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:43.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.569+0000 7fa539b9c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5341f3d90 0x7fa5341f0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:50278/0 (socket says 172.21.15.55:50278) 2024-03-19T01:51:43.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.569+0000 7fa539b9c640 1 -- 172.21.15.55:0/1469803063 learned_addr learned my addr 172.21.15.55:0/1469803063 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:43.582 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.581+0000 7fa538b9a640 1 --2- 172.21.15.55:0/1469803063 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa534150ea0 0x7fa5341f3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:43.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.641+0000 7fa53939b640 1 --2- 172.21.15.55:0/1469803063 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa520004e70 0x7fa53415c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:43.657 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.657+0000 7fa53939b640 1 -- 172.21.15.55:0/1469803063 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5341f3d90 msgr2=0x7fa5341f0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:43.657 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.657+0000 7fa53939b640 1 --2- 172.21.15.55:0/1469803063 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5341f3d90 0x7fa5341f0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:43.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.657+0000 7fa53939b640 1 -- 172.21.15.55:0/1469803063 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa534150ea0 msgr2=0x7fa5341f3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:43.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.657+0000 7fa53939b640 1 --2- 172.21.15.55:0/1469803063 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa534150ea0 0x7fa5341f3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:43.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.657+0000 7fa53939b640 1 -- 172.21.15.55:0/1469803063 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5341525b0 con 0x7fa520004e70 2024-03-19T01:51:43.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.661+0000 7fa53939b640 1 --2- 172.21.15.55:0/1469803063 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa520004e70 0x7fa53415c950 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa53000e630 tx=0x7fa53000eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:43.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.669+0000 7fa529ffb640 1 -- 172.21.15.55:0/1469803063 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa530004030 con 0x7fa520004e70 2024-03-19T01:51:43.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.673+0000 7fa529ffb640 1 -- 172.21.15.55:0/1469803063 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5300041f0 con 0x7fa520004e70 2024-03-19T01:51:43.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.673+0000 7fa529ffb640 1 -- 172.21.15.55:0/1469803063 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53000c690 con 0x7fa520004e70 2024-03-19T01:51:43.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.673+0000 7fa53b5ff640 1 -- 172.21.15.55:0/1469803063 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa534157950 con 0x7fa520004e70 2024-03-19T01:51:43.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.673+0000 7fa53b5ff640 1 -- 172.21.15.55:0/1469803063 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa53415d5d0 con 0x7fa520004e70 2024-03-19T01:51:43.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:43.673+0000 7fa53b5ff640 1 -- 172.21.15.55:0/1469803063 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa534068a40 con 0x7fa520004e70 2024-03-19T01:51:44.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.001+0000 7fa529ffb640 1 -- 172.21.15.55:0/1469803063 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa53000c830 con 0x7fa520004e70 2024-03-19T01:51:44.003 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.001+0000 7fa529ffb640 1 --2- 172.21.15.55:0/1469803063 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa5180676b0 0x7fa518069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:44.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.001+0000 7fa538b9a640 1 --2- 172.21.15.55:0/1469803063 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa5180676b0 0x7fa518069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:44.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.005+0000 7fa538b9a640 1 --2- 172.21.15.55:0/1469803063 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa5180676b0 0x7fa518069b70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa5340cb5c0 tx=0x7fa524009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:44.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.037+0000 7fa529ffb640 1 -- 172.21.15.55:0/1469803063 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fa5300544f0 con 0x7fa520004e70 2024-03-19T01:51:44.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.037+0000 7fa529ffb640 1 -- 172.21.15.55:0/1469803063 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa530017020 con 0x7fa520004e70 2024-03-19T01:51:44.232 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.229+0000 7fa53b5ff640 1 -- 172.21.15.55:0/1469803063 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa534157f20 con 0x7fa520004e70 2024-03-19T01:51:44.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.341+0000 7fa529ffb640 1 -- 172.21.15.55:0/1469803063 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+18430 (secure 0 0 0) 0x7fa530056450 con 0x7fa520004e70 2024-03-19T01:51:44.341 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:51:44.341 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":29,"fsid":"def9d4e5-0bc7-449d-b893-2490cbd10e25","created":"2024-03-19T01:50:02.482954+0000","modified":"2024-03-19T01:51:17.228291+0000","last_up_change":"2024-03-19T01:50:21.735508+0000","last_in_change":"2024-03-19T01:50:12.168944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:50:23.456904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:51:04.578583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:51:06.871659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T01:51:14.274442+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"35d5e3c5-d089-4f38-8961-98d064188f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6825","nonce":2757675043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6827","nonce":2757675043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6831","nonce":2757675043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6829","nonce":2757675043}]},"public_addr":"172.21.15.55:6825/2757675043","cluster_addr":"172.21.15.55:6827/2757675043","heartbeat_back_addr":"172.21.15.55:6831/2757675043","heartbeat_front_addr":"172.21.15.55:6829/2757675043","state":["exists","up"]},{"osd":1,"uuid":"8e58f791-001a-44d0-a464-ca68a479f70c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6817","nonce":543460007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6819","nonce":543460007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6823","nonce":543460007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6821","nonce":543460007}]},"public_addr":"172.21.15.55:6817/543460007","cluster_addr":"172.21.15.55:6819/543460007","heartbeat_back_addr":"172.21.15.55:6823/543460007","heartbeat_front_addr":"172.21.15.55:6821/543460007","state":["exists","up"]},{"osd":2,"uuid":"d44f38b8-67f4-43de-98a6-f3c520acaf10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6801","nonce":1760253550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6803","nonce":1760253550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6807","nonce":1760253550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6805","nonce":1760253550}]},"public_addr":"172.21.15.55:6801/1760253550","cluster_addr":"172.21.15.55:6803/1760253550","heartbeat_back_addr":"172.21.15.55:6807/1760253550","heartbeat_front_addr":"172.21.15.55:6805/1760253550","state":["exists","up"]},{"osd":3,"uuid":"850a7316-2d74-43f0-8649-b5f8d8f6692e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6809","nonce":2797616448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6811","nonce":2797616448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6815","nonce":2797616448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6813","nonce":2797616448}]},"public_addr":"172.21.15.55:6809/2797616448","cluster_addr":"172.21.15.55:6811/2797616448","heartbeat_back_addr":"172.21.15.55:6815/2797616448","heartbeat_front_addr":"172.21.15.55:6813/2797616448","state":["exists","up"]},{"osd":4,"uuid":"4e6152ae-a77d-4bb6-a855-5782ad5411a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6819","nonce":2938436826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6821","nonce":2938436826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6825","nonce":2938436826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6823","nonce":2938436826}]},"public_addr":"172.21.15.84:6819/2938436826","cluster_addr":"172.21.15.84:6821/2938436826","heartbeat_back_addr":"172.21.15.84:6825/2938436826","heartbeat_front_addr":"172.21.15.84:6823/2938436826","state":["exists","up"]},{"osd":5,"uuid":"1217ceba-dd31-4615-96f1-dfaf57160f3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6803","nonce":460717295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6805","nonce":460717295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6809","nonce":460717295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6807","nonce":460717295}]},"public_addr":"172.21.15.84:6803/460717295","cluster_addr":"172.21.15.84:6805/460717295","heartbeat_back_addr":"172.21.15.84:6809/460717295","heartbeat_front_addr":"172.21.15.84:6807/460717295","state":["exists","up"]},{"osd":6,"uuid":"3d9ed78a-2284-4c28-a24a-70eb812c7836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6827","nonce":416123135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6829","nonce":416123135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6833","nonce":416123135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6831","nonce":416123135}]},"public_addr":"172.21.15.84:6827/416123135","cluster_addr":"172.21.15.84:6829/416123135","heartbeat_back_addr":"172.21.15.84:6833/416123135","heartbeat_front_addr":"172.21.15.84:6831/416123135","state":["exists","up"]},{"osd":7,"uuid":"c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6811","nonce":460083528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6813","nonce":460083528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6817","nonce":460083528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6815","nonce":460083528}]},"public_addr":"172.21.15.84:6811/460083528","cluster_addr":"172.21.15.84:6813/460083528","heartbeat_back_addr":"172.21.15.84:6817/460083528","heartbeat_front_addr":"172.21.15.84:6815/460083528","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.269126+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.121001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.075714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.092461+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.194676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.062833+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.316515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.237111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:51:44.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.341+0000 7fa53b5ff640 1 -- 172.21.15.55:0/1469803063 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa5180676b0 msgr2=0x7fa518069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:44.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.345+0000 7fa53b5ff640 1 --2- 172.21.15.55:0/1469803063 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa5180676b0 0x7fa518069b70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa5340cb5c0 tx=0x7fa524009040 comp rx=0 tx=0).stop 2024-03-19T01:51:44.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.345+0000 7fa53b5ff640 1 -- 172.21.15.55:0/1469803063 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa520004e70 msgr2=0x7fa53415c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:44.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.345+0000 7fa53b5ff640 1 --2- 172.21.15.55:0/1469803063 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa520004e70 0x7fa53415c950 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa53000e630 tx=0x7fa53000eaf0 comp rx=0 tx=0).stop 2024-03-19T01:51:44.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.345+0000 7fa53b5ff640 1 -- 172.21.15.55:0/1469803063 shutdown_connections 2024-03-19T01:51:44.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.345+0000 7fa53b5ff640 1 --2- 172.21.15.55:0/1469803063 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa5180676b0 0x7fa518069b70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:44.346 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.345+0000 7fa53b5ff640 1 --2- 172.21.15.55:0/1469803063 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5341f3d90 0x7fa5341f0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:44.346 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.345+0000 7fa53b5ff640 1 --2- 172.21.15.55:0/1469803063 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa534150ea0 0x7fa5341f3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:44.346 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.345+0000 7fa53b5ff640 1 --2- 172.21.15.55:0/1469803063 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa520004e70 0x7fa53415c950 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:44.346 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.345+0000 7fa53b5ff640 1 -- 172.21.15.55:0/1469803063 >> 172.21.15.55:0/1469803063 conn(0x7fa53406fb40 msgr2=0x7fa5340cc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:44.346 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.345+0000 7fa53b5ff640 1 -- 172.21.15.55:0/1469803063 shutdown_connections 2024-03-19T01:51:44.346 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.345+0000 7fa53b5ff640 1 -- 172.21.15.55:0/1469803063 wait complete. 2024-03-19T01:51:44.366 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:51:44.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.489+0000 7f2b142db640 1 Processor -- start 2024-03-19T01:51:44.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.489+0000 7f2b142db640 1 -- start start 2024-03-19T01:51:44.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.493+0000 7f2b142db640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f2b0c152120 0x7f2b0c0c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:44.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.493+0000 7f2b142db640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2b0c0c9610 0x7f2b0c157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:44.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.493+0000 7f2b142db640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b0c06fff0 con 0x7f2b0c070780 2024-03-19T01:51:44.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.493+0000 7f2b142db640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f2b0c152500 con 0x7f2b0c0c9610 2024-03-19T01:51:44.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.493+0000 7f2b142db640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f2b0c070590 con 0x7f2b0c152120 2024-03-19T01:51:44.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.493+0000 7f2b11876640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2b0c0c9610 0x7f2b0c157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:44.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.493+0000 7f2b11876640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2b0c0c9610 0x7f2b0c157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:42514/0 (socket says 172.21.15.55:42514) 2024-03-19T01:51:44.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.493+0000 7f2b11876640 1 -- 172.21.15.55:0/2840805504 learned_addr learned my addr 172.21.15.55:0/2840805504 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:44.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.493+0000 7f2b12077640 1 --2- 172.21.15.55:0/2840805504 >> v2:172.21.15.84:3300/0 conn(0x7f2b0c152120 0x7f2b0c0c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:44.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.497+0000 7f2b11876640 1 -- 172.21.15.55:0/2840805504 >> v2:172.21.15.84:3300/0 conn(0x7f2b0c152120 msgr2=0x7f2b0c0c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:44.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.497+0000 7f2b11876640 1 --2- 172.21.15.55:0/2840805504 >> v2:172.21.15.84:3300/0 conn(0x7f2b0c152120 0x7f2b0c0c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:44.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.497+0000 7f2b11876640 1 -- 172.21.15.55:0/2840805504 >> v1:172.21.15.84:6789/0 conn(0x7f2b0c070780 legacy=0x7f2b0c158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:44.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.497+0000 7f2b11876640 1 -- 172.21.15.55:0/2840805504 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b0c0c39c0 con 0x7f2b0c0c9610 2024-03-19T01:51:44.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.501+0000 7f2b11876640 1 --2- 172.21.15.55:0/2840805504 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b0c0c9610 0x7f2b0c157fd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c072430 tx=0x7f2afc01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26a230ad29cb43ef server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:44.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.505+0000 7f2b11075640 1 -- 172.21.15.55:0/2840805504 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc0b0070 con 0x7f2b0c0c9610 2024-03-19T01:51:44.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.505+0000 7f2b11075640 1 -- 172.21.15.55:0/2840805504 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2afc0ab600 con 0x7f2b0c0c9610 2024-03-19T01:51:44.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.505+0000 7f2b11075640 1 -- 172.21.15.55:0/2840805504 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc0ab9d0 con 0x7f2b0c0c9610 2024-03-19T01:51:44.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.505+0000 7f2b142db640 1 -- 172.21.15.55:0/2840805504 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b0c0c9610 msgr2=0x7f2b0c157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:44.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.505+0000 7f2b142db640 1 --2- 172.21.15.55:0/2840805504 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b0c0c9610 0x7f2b0c157fd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c072430 tx=0x7f2afc01c4e0 comp rx=0 tx=0).stop 2024-03-19T01:51:44.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.505+0000 7f2b142db640 1 -- 172.21.15.55:0/2840805504 shutdown_connections 2024-03-19T01:51:44.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.505+0000 7f2b142db640 1 --2- 172.21.15.55:0/2840805504 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b0c0c9610 0x7f2b0c157fd0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:44.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.505+0000 7f2b142db640 1 --2- 172.21.15.55:0/2840805504 >> v2:172.21.15.84:3300/0 conn(0x7f2b0c152120 0x7f2b0c0c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:44.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.505+0000 7f2b142db640 1 -- 172.21.15.55:0/2840805504 >> 172.21.15.55:0/2840805504 conn(0x7f2b0c067600 msgr2=0x7f2b0c071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:44.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.505+0000 7f2b142db640 1 -- 172.21.15.55:0/2840805504 shutdown_connections 2024-03-19T01:51:44.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.505+0000 7f2b142db640 1 -- 172.21.15.55:0/2840805504 wait complete. 2024-03-19T01:51:44.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.505+0000 7f2b142db640 1 Processor -- start 2024-03-19T01:51:44.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.505+0000 7f2b142db640 1 -- start start 2024-03-19T01:51:44.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.509+0000 7f2b142db640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b0c070780 0x7f2b0c0b2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:44.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.509+0000 7f2b142db640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b0c0c9610 0x7f2b0c159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:44.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.509+0000 7f2b142db640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2b0c152120 0x7f2b0c15a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:44.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.509+0000 7f2b142db640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c0c86d0 con 0x7f2b0c0c9610 2024-03-19T01:51:44.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.509+0000 7f2b142db640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c0c31c0 con 0x7f2b0c070780 2024-03-19T01:51:44.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.509+0000 7f2b142db640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2b0c165170 con 0x7f2b0c152120 2024-03-19T01:51:44.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.509+0000 7f2b11876640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b0c0c9610 0x7f2b0c159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:44.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.509+0000 7f2b11876640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b0c0c9610 0x7f2b0c159eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:42520/0 (socket says 172.21.15.55:42520) 2024-03-19T01:51:44.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.509+0000 7f2b11876640 1 -- 172.21.15.55:0/2359013317 learned_addr learned my addr 172.21.15.55:0/2359013317 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:44.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.509+0000 7f2b12077640 1 --2- 172.21.15.55:0/2359013317 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b0c070780 0x7f2b0c0b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:44.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.509+0000 7f2b12878640 1 --2- 172.21.15.55:0/2359013317 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2b0c152120 0x7f2b0c15a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:44.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.513+0000 7f2b11876640 1 -- 172.21.15.55:0/2359013317 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2b0c152120 msgr2=0x7f2b0c15a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:44.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.513+0000 7f2b11876640 1 --2- 172.21.15.55:0/2359013317 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2b0c152120 0x7f2b0c15a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:44.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.513+0000 7f2b11876640 1 -- 172.21.15.55:0/2359013317 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b0c070780 msgr2=0x7f2b0c0b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:44.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.513+0000 7f2b11876640 1 --2- 172.21.15.55:0/2359013317 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b0c070780 0x7f2b0c0b2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:44.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.513+0000 7f2b11876640 1 -- 172.21.15.55:0/2359013317 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b0c1f1aa0 con 0x7f2b0c0c9610 2024-03-19T01:51:44.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.513+0000 7f2b11876640 1 --2- 172.21.15.55:0/2359013317 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b0c0c9610 0x7f2b0c159eb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2afc0190f0 tx=0x7f2afc019280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:44.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.517+0000 7f2b02ffd640 1 -- 172.21.15.55:0/2359013317 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc0bb070 con 0x7f2b0c0c9610 2024-03-19T01:51:44.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.517+0000 7f2b02ffd640 1 -- 172.21.15.55:0/2359013317 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2afc0b1040 con 0x7f2b0c0c9610 2024-03-19T01:51:44.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.517+0000 7f2b02ffd640 1 -- 172.21.15.55:0/2359013317 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc0b0070 con 0x7f2b0c0c9610 2024-03-19T01:51:44.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.517+0000 7f2b142db640 1 -- 172.21.15.55:0/2359013317 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b0c1f1d80 con 0x7f2b0c0c9610 2024-03-19T01:51:44.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.517+0000 7f2b142db640 1 -- 172.21.15.55:0/2359013317 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b0c1f2340 con 0x7f2b0c0c9610 2024-03-19T01:51:44.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.517+0000 7f2b142db640 1 -- 172.21.15.55:0/2359013317 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b0c068a40 con 0x7f2b0c0c9610 2024-03-19T01:51:44.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.801+0000 7f2b02ffd640 1 -- 172.21.15.55:0/2359013317 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f2afc0aa7f0 con 0x7f2b0c0c9610 2024-03-19T01:51:44.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.801+0000 7f2b02ffd640 1 --2- 172.21.15.55:0/2359013317 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2af0067660 0x7f2af0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:44.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.801+0000 7f2b12077640 1 --2- 172.21.15.55:0/2359013317 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2af0067660 0x7f2af0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:44.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.801+0000 7f2b12077640 1 --2- 172.21.15.55:0/2359013317 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2af0067660 0x7f2af0069b20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2b080079c0 tx=0x7f2b08008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:44.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.841+0000 7f2b02ffd640 1 -- 172.21.15.55:0/2359013317 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f2afc0ed960 con 0x7f2b0c0c9610 2024-03-19T01:51:44.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:44.845+0000 7f2b02ffd640 1 -- 172.21.15.55:0/2359013317 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2afc0c4020 con 0x7f2b0c0c9610 2024-03-19T01:51:45.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.073+0000 7f2b142db640 1 -- 172.21.15.55:0/2359013317 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2b0c0cba70 con 0x7f2b0c0c9610 2024-03-19T01:51:45.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.153+0000 7f2b02ffd640 1 -- 172.21.15.55:0/2359013317 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+18430 (secure 0 0 0) 0x7f2afc11c810 con 0x7f2b0c0c9610 2024-03-19T01:51:45.154 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:51:45.154 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":29,"fsid":"def9d4e5-0bc7-449d-b893-2490cbd10e25","created":"2024-03-19T01:50:02.482954+0000","modified":"2024-03-19T01:51:17.228291+0000","last_up_change":"2024-03-19T01:50:21.735508+0000","last_in_change":"2024-03-19T01:50:12.168944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:50:23.456904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:51:04.578583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:51:06.871659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T01:51:14.274442+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"35d5e3c5-d089-4f38-8961-98d064188f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6825","nonce":2757675043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6827","nonce":2757675043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6831","nonce":2757675043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6829","nonce":2757675043}]},"public_addr":"172.21.15.55:6825/2757675043","cluster_addr":"172.21.15.55:6827/2757675043","heartbeat_back_addr":"172.21.15.55:6831/2757675043","heartbeat_front_addr":"172.21.15.55:6829/2757675043","state":["exists","up"]},{"osd":1,"uuid":"8e58f791-001a-44d0-a464-ca68a479f70c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6817","nonce":543460007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6819","nonce":543460007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6823","nonce":543460007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6821","nonce":543460007}]},"public_addr":"172.21.15.55:6817/543460007","cluster_addr":"172.21.15.55:6819/543460007","heartbeat_back_addr":"172.21.15.55:6823/543460007","heartbeat_front_addr":"172.21.15.55:6821/543460007","state":["exists","up"]},{"osd":2,"uuid":"d44f38b8-67f4-43de-98a6-f3c520acaf10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6801","nonce":1760253550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6803","nonce":1760253550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6807","nonce":1760253550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6805","nonce":1760253550}]},"public_addr":"172.21.15.55:6801/1760253550","cluster_addr":"172.21.15.55:6803/1760253550","heartbeat_back_addr":"172.21.15.55:6807/1760253550","heartbeat_front_addr":"172.21.15.55:6805/1760253550","state":["exists","up"]},{"osd":3,"uuid":"850a7316-2d74-43f0-8649-b5f8d8f6692e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6809","nonce":2797616448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6811","nonce":2797616448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6815","nonce":2797616448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6813","nonce":2797616448}]},"public_addr":"172.21.15.55:6809/2797616448","cluster_addr":"172.21.15.55:6811/2797616448","heartbeat_back_addr":"172.21.15.55:6815/2797616448","heartbeat_front_addr":"172.21.15.55:6813/2797616448","state":["exists","up"]},{"osd":4,"uuid":"4e6152ae-a77d-4bb6-a855-5782ad5411a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6819","nonce":2938436826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6821","nonce":2938436826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6825","nonce":2938436826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6823","nonce":2938436826}]},"public_addr":"172.21.15.84:6819/2938436826","cluster_addr":"172.21.15.84:6821/2938436826","heartbeat_back_addr":"172.21.15.84:6825/2938436826","heartbeat_front_addr":"172.21.15.84:6823/2938436826","state":["exists","up"]},{"osd":5,"uuid":"1217ceba-dd31-4615-96f1-dfaf57160f3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6803","nonce":460717295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6805","nonce":460717295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6809","nonce":460717295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6807","nonce":460717295}]},"public_addr":"172.21.15.84:6803/460717295","cluster_addr":"172.21.15.84:6805/460717295","heartbeat_back_addr":"172.21.15.84:6809/460717295","heartbeat_front_addr":"172.21.15.84:6807/460717295","state":["exists","up"]},{"osd":6,"uuid":"3d9ed78a-2284-4c28-a24a-70eb812c7836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6827","nonce":416123135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6829","nonce":416123135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6833","nonce":416123135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6831","nonce":416123135}]},"public_addr":"172.21.15.84:6827/416123135","cluster_addr":"172.21.15.84:6829/416123135","heartbeat_back_addr":"172.21.15.84:6833/416123135","heartbeat_front_addr":"172.21.15.84:6831/416123135","state":["exists","up"]},{"osd":7,"uuid":"c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6811","nonce":460083528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6813","nonce":460083528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6817","nonce":460083528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6815","nonce":460083528}]},"public_addr":"172.21.15.84:6811/460083528","cluster_addr":"172.21.15.84:6813/460083528","heartbeat_back_addr":"172.21.15.84:6817/460083528","heartbeat_front_addr":"172.21.15.84:6815/460083528","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.269126+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.121001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.075714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.092461+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.194676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.062833+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.316515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.237111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:51:45.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.153+0000 7f2b142db640 1 -- 172.21.15.55:0/2359013317 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2af0067660 msgr2=0x7f2af0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:45.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.153+0000 7f2b142db640 1 --2- 172.21.15.55:0/2359013317 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2af0067660 0x7f2af0069b20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2b080079c0 tx=0x7f2b08008040 comp rx=0 tx=0).stop 2024-03-19T01:51:45.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.153+0000 7f2b142db640 1 -- 172.21.15.55:0/2359013317 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b0c0c9610 msgr2=0x7f2b0c159eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:45.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.153+0000 7f2b142db640 1 --2- 172.21.15.55:0/2359013317 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b0c0c9610 0x7f2b0c159eb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2afc0190f0 tx=0x7f2afc019280 comp rx=0 tx=0).stop 2024-03-19T01:51:45.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.153+0000 7f2b142db640 1 -- 172.21.15.55:0/2359013317 shutdown_connections 2024-03-19T01:51:45.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.153+0000 7f2b142db640 1 --2- 172.21.15.55:0/2359013317 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2af0067660 0x7f2af0069b20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:45.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.153+0000 7f2b142db640 1 --2- 172.21.15.55:0/2359013317 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2b0c152120 0x7f2b0c15a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:45.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.153+0000 7f2b142db640 1 --2- 172.21.15.55:0/2359013317 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b0c0c9610 0x7f2b0c159eb0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:45.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.153+0000 7f2b142db640 1 --2- 172.21.15.55:0/2359013317 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b0c070780 0x7f2b0c0b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:45.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.153+0000 7f2b142db640 1 -- 172.21.15.55:0/2359013317 >> 172.21.15.55:0/2359013317 conn(0x7f2b0c067600 msgr2=0x7f2b0c06f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:45.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.153+0000 7f2b142db640 1 -- 172.21.15.55:0/2359013317 shutdown_connections 2024-03-19T01:51:45.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.153+0000 7f2b142db640 1 -- 172.21.15.55:0/2359013317 wait complete. 2024-03-19T01:51:45.174 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-19T01:51:45.293 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.293+0000 7f97531c6640 1 Processor -- start 2024-03-19T01:51:45.293 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.293+0000 7f97531c6640 1 -- start start 2024-03-19T01:51:45.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.293+0000 7f97531c6640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f974c154060 0x7f974c06bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:45.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.293+0000 7f97531c6640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f974c0ca0c0 0x7f974c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:45.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.293+0000 7f97531c6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f974c0ccd80 con 0x7f974c070410 2024-03-19T01:51:45.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.293+0000 7f97531c6640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f974c0ccb50 con 0x7f974c0ca0c0 2024-03-19T01:51:45.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.293+0000 7f97531c6640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f974c0cd040 con 0x7f974c154060 2024-03-19T01:51:45.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.293+0000 7f974bfff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f974c0ca0c0 0x7f974c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:45.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.293+0000 7f9751763640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f974c070410 0x7f974c0c7600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:41398/0 (socket says 172.21.15.55:41398) 2024-03-19T01:51:45.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.293+0000 7f9751763640 1 -- 172.21.15.55:0/2833619442 learned_addr learned my addr 172.21.15.55:0/2833619442 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:51:45.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.297+0000 7f974bfff640 1 -- 172.21.15.55:0/2833619442 >> v2:172.21.15.84:3300/0 conn(0x7f974c154060 msgr2=0x7f974c06bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:45.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.297+0000 7f974bfff640 1 --2- 172.21.15.55:0/2833619442 >> v2:172.21.15.84:3300/0 conn(0x7f974c154060 0x7f974c06bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:45.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.297+0000 7f974bfff640 1 -- 172.21.15.55:0/2833619442 >> v1:172.21.15.55:6789/0 conn(0x7f974c070410 legacy=0x7f974c0c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:45.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.297+0000 7f974bfff640 1 -- 172.21.15.55:0/2833619442 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f974c0cd320 con 0x7f974c0ca0c0 2024-03-19T01:51:45.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.301+0000 7f974bfff640 1 --2- 172.21.15.55:0/2833619442 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f974c0ca0c0 0x7f974c15a9b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f974c0ca050 tx=0x7f973801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fca59ff9c8e2164 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:45.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.305+0000 7f974b7fe640 1 -- 172.21.15.55:0/2833619442 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97380b5070 con 0x7f974c0ca0c0 2024-03-19T01:51:45.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.305+0000 7f974b7fe640 1 -- 172.21.15.55:0/2833619442 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97380a8970 con 0x7f974c0ca0c0 2024-03-19T01:51:45.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.305+0000 7f974b7fe640 1 -- 172.21.15.55:0/2833619442 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97380a8d40 con 0x7f974c0ca0c0 2024-03-19T01:51:45.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.309+0000 7f97531c6640 1 -- 172.21.15.55:0/2833619442 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f974c0ca0c0 msgr2=0x7f974c15a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:45.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.309+0000 7f97531c6640 1 --2- 172.21.15.55:0/2833619442 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f974c0ca0c0 0x7f974c15a9b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f974c0ca050 tx=0x7f973801ec70 comp rx=0 tx=0).stop 2024-03-19T01:51:45.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.309+0000 7f97531c6640 1 -- 172.21.15.55:0/2833619442 shutdown_connections 2024-03-19T01:51:45.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.309+0000 7f97531c6640 1 --2- 172.21.15.55:0/2833619442 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f974c0ca0c0 0x7f974c15a9b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:45.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.309+0000 7f97531c6640 1 --2- 172.21.15.55:0/2833619442 >> v2:172.21.15.84:3300/0 conn(0x7f974c154060 0x7f974c06bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:45.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.309+0000 7f97531c6640 1 -- 172.21.15.55:0/2833619442 >> 172.21.15.55:0/2833619442 conn(0x7f974c06c3b0 msgr2=0x7f974c06c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:45.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.309+0000 7f97531c6640 1 -- 172.21.15.55:0/2833619442 shutdown_connections 2024-03-19T01:51:45.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.309+0000 7f97531c6640 1 -- 172.21.15.55:0/2833619442 wait complete. 2024-03-19T01:51:45.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.309+0000 7f97531c6640 1 Processor -- start 2024-03-19T01:51:45.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.309+0000 7f97531c6640 1 -- start start 2024-03-19T01:51:45.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.309+0000 7f97531c6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f974c070410 0x7f974c150d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:45.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.309+0000 7f97531c6640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f974c0ca0c0 0x7f974c1527f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:45.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.309+0000 7f97531c6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f974c154060 0x7f974c14f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:45.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.309+0000 7f97531c6640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f974c15a490 con 0x7f974c070410 2024-03-19T01:51:45.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.309+0000 7f97531c6640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f974c0cd8e0 con 0x7f974c0ca0c0 2024-03-19T01:51:45.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.309+0000 7f97531c6640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f974c15a140 con 0x7f974c154060 2024-03-19T01:51:45.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.309+0000 7f9750f62640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f974c070410 0x7f974c150d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:45.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.309+0000 7f9750f62640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f974c070410 0x7f974c150d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:42540/0 (socket says 172.21.15.55:42540) 2024-03-19T01:51:45.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.309+0000 7f9750f62640 1 -- 172.21.15.55:0/2017152063 learned_addr learned my addr 172.21.15.55:0/2017152063 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:45.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.309+0000 7f974bfff640 1 --2- 172.21.15.55:0/2017152063 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f974c0ca0c0 0x7f974c1527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:45.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.309+0000 7f9751763640 1 --2- 172.21.15.55:0/2017152063 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f974c154060 0x7f974c14f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:45.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.313+0000 7f9750f62640 1 -- 172.21.15.55:0/2017152063 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f974c154060 msgr2=0x7f974c14f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:45.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.313+0000 7f9750f62640 1 --2- 172.21.15.55:0/2017152063 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f974c154060 0x7f974c14f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:45.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.313+0000 7f9750f62640 1 -- 172.21.15.55:0/2017152063 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f974c0ca0c0 msgr2=0x7f974c1527f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:45.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.313+0000 7f9750f62640 1 --2- 172.21.15.55:0/2017152063 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f974c0ca0c0 0x7f974c1527f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:45.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.313+0000 7f9750f62640 1 -- 172.21.15.55:0/2017152063 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f974c0cc650 con 0x7f974c070410 2024-03-19T01:51:45.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.317+0000 7f9750f62640 1 --2- 172.21.15.55:0/2017152063 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f974c070410 0x7f974c150d10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f974000b410 tx=0x7f974000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:45.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.321+0000 7f97497fa640 1 -- 172.21.15.55:0/2017152063 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9740015020 con 0x7f974c070410 2024-03-19T01:51:45.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.321+0000 7f97497fa640 1 -- 172.21.15.55:0/2017152063 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9740011830 con 0x7f974c070410 2024-03-19T01:51:45.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.321+0000 7f97531c6640 1 -- 172.21.15.55:0/2017152063 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f974c151b50 con 0x7f974c070410 2024-03-19T01:51:45.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.321+0000 7f97497fa640 1 -- 172.21.15.55:0/2017152063 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9740011c20 con 0x7f974c070410 2024-03-19T01:51:45.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.321+0000 7f97531c6640 1 -- 172.21.15.55:0/2017152063 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f974c152110 con 0x7f974c070410 2024-03-19T01:51:45.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.321+0000 7f97531c6640 1 -- 172.21.15.55:0/2017152063 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f974c068a40 con 0x7f974c070410 2024-03-19T01:51:45.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.633+0000 7f97497fa640 1 -- 172.21.15.55:0/2017152063 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f9740018020 con 0x7f974c070410 2024-03-19T01:51:45.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.633+0000 7f97497fa640 1 --2- 172.21.15.55:0/2017152063 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f97300676b0 0x7f9730069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:45.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.633+0000 7f974bfff640 1 --2- 172.21.15.55:0/2017152063 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f97300676b0 0x7f9730069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:45.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.633+0000 7f974bfff640 1 --2- 172.21.15.55:0/2017152063 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f97300676b0 0x7f9730069b70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f97380b2000 tx=0x7f97380aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:45.657 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.657+0000 7f97497fa640 1 -- 172.21.15.55:0/2017152063 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f9740054360 con 0x7f974c070410 2024-03-19T01:51:45.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.657+0000 7f97497fa640 1 -- 172.21.15.55:0/2017152063 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f974001f020 con 0x7f974c070410 2024-03-19T01:51:45.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:45.861+0000 7f97531c6640 1 -- 172.21.15.55:0/2017152063 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f974c0c9570 con 0x7f974c070410 2024-03-19T01:51:46.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.677+0000 7f97497fa640 1 -- 172.21.15.55:0/2017152063 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v9) ==== 118+0+0 (secure 0 0 0) 0x7f974c0cc650 con 0x7f974c070410 2024-03-19T01:51:46.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.681+0000 7f97531c6640 1 -- 172.21.15.55:0/2017152063 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f97300676b0 msgr2=0x7f9730069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:46.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.681+0000 7f97531c6640 1 --2- 172.21.15.55:0/2017152063 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f97300676b0 0x7f9730069b70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f97380b2000 tx=0x7f97380aa070 comp rx=0 tx=0).stop 2024-03-19T01:51:46.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.681+0000 7f97531c6640 1 -- 172.21.15.55:0/2017152063 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f974c070410 msgr2=0x7f974c150d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:46.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.681+0000 7f97531c6640 1 --2- 172.21.15.55:0/2017152063 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f974c070410 0x7f974c150d10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f974000b410 tx=0x7f974000b8d0 comp rx=0 tx=0).stop 2024-03-19T01:51:46.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.681+0000 7f97531c6640 1 -- 172.21.15.55:0/2017152063 shutdown_connections 2024-03-19T01:51:46.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.681+0000 7f97531c6640 1 --2- 172.21.15.55:0/2017152063 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f97300676b0 0x7f9730069b70 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:46.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.681+0000 7f97531c6640 1 --2- 172.21.15.55:0/2017152063 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f974c154060 0x7f974c14f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:46.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.681+0000 7f97531c6640 1 --2- 172.21.15.55:0/2017152063 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f974c0ca0c0 0x7f974c1527f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:46.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.681+0000 7f97531c6640 1 --2- 172.21.15.55:0/2017152063 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f974c070410 0x7f974c150d10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:46.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.681+0000 7f97531c6640 1 -- 172.21.15.55:0/2017152063 >> 172.21.15.55:0/2017152063 conn(0x7f974c06c3b0 msgr2=0x7f974c06bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:46.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.681+0000 7f97531c6640 1 -- 172.21.15.55:0/2017152063 shutdown_connections 2024-03-19T01:51:46.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.681+0000 7f97531c6640 1 -- 172.21.15.55:0/2017152063 wait complete. 2024-03-19T01:51:46.699 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300} 2024-03-19T01:51:46.699 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-19T01:51:46.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.825+0000 7f07bf159640 1 Processor -- start 2024-03-19T01:51:46.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.825+0000 7f07bf159640 1 -- start start 2024-03-19T01:51:46.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.825+0000 7f07bf159640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f07b8157b80 0x7f07b8072900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:46.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.825+0000 7f07bf159640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07b80ca2c0 0x7f07b815a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:46.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.825+0000 7f07bf159640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07b8073360 con 0x7f07b8150dc0 2024-03-19T01:51:46.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.825+0000 7f07bf159640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f07b815a9c0 con 0x7f07b8157b80 2024-03-19T01:51:46.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.825+0000 7f07bf159640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f07b8073600 con 0x7f07b80ca2c0 2024-03-19T01:51:46.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.825+0000 7f07b7fff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f07b8157b80 0x7f07b8072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:46.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.825+0000 7f07bd6f6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07b80ca2c0 0x7f07b815a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:46.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.829+0000 7f07b7fff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f07b8157b80 0x7f07b8072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:34506/0 (socket says 172.21.15.55:34506) 2024-03-19T01:51:46.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.829+0000 7f07b7fff640 1 -- 172.21.15.55:0/3413604345 learned_addr learned my addr 172.21.15.55:0/3413604345 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:46.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.833+0000 7f07b7fff640 1 -- 172.21.15.55:0/3413604345 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07b80ca2c0 msgr2=0x7f07b815a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:46.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.833+0000 7f07b7fff640 1 --2- 172.21.15.55:0/3413604345 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07b80ca2c0 0x7f07b815a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:46.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.833+0000 7f07b7fff640 1 -- 172.21.15.55:0/3413604345 >> v1:172.21.15.84:6789/0 conn(0x7f07b8150dc0 legacy=0x7f07b8070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:46.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.833+0000 7f07b7fff640 1 -- 172.21.15.55:0/3413604345 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b81570a0 con 0x7f07b8157b80 2024-03-19T01:51:46.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.841+0000 7f07b7fff640 1 --2- 172.21.15.55:0/3413604345 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07b8157b80 0x7f07b8072900 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f07b8071210 tx=0x7f07ac01ecf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa6fa2d004995325 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:46.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.849+0000 7f07b77fe640 1 -- 172.21.15.55:0/3413604345 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ac0b5070 con 0x7f07b8157b80 2024-03-19T01:51:46.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.849+0000 7f07b77fe640 1 -- 172.21.15.55:0/3413604345 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07ac0a89b0 con 0x7f07b8157b80 2024-03-19T01:51:46.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.849+0000 7f07b77fe640 1 -- 172.21.15.55:0/3413604345 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ac0a8d80 con 0x7f07b8157b80 2024-03-19T01:51:46.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.849+0000 7f07bf159640 1 -- 172.21.15.55:0/3413604345 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07b8157b80 msgr2=0x7f07b8072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:46.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.849+0000 7f07bf159640 1 --2- 172.21.15.55:0/3413604345 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07b8157b80 0x7f07b8072900 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f07b8071210 tx=0x7f07ac01ecf0 comp rx=0 tx=0).stop 2024-03-19T01:51:46.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.849+0000 7f07bf159640 1 -- 172.21.15.55:0/3413604345 shutdown_connections 2024-03-19T01:51:46.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.849+0000 7f07bf159640 1 --2- 172.21.15.55:0/3413604345 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07b80ca2c0 0x7f07b815a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:46.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.849+0000 7f07bf159640 1 --2- 172.21.15.55:0/3413604345 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07b8157b80 0x7f07b8072900 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:46.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.849+0000 7f07bf159640 1 -- 172.21.15.55:0/3413604345 >> 172.21.15.55:0/3413604345 conn(0x7f07b806c580 msgr2=0x7f07b806c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:46.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.849+0000 7f07bf159640 1 -- 172.21.15.55:0/3413604345 shutdown_connections 2024-03-19T01:51:46.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.849+0000 7f07bf159640 1 -- 172.21.15.55:0/3413604345 wait complete. 2024-03-19T01:51:46.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.853+0000 7f07bf159640 1 Processor -- start 2024-03-19T01:51:46.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.853+0000 7f07bf159640 1 -- start start 2024-03-19T01:51:46.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.853+0000 7f07bf159640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07b80ca2c0 0x7f07b814c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:46.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.853+0000 7f07bf159640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07b8150dc0 0x7f07b814b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:46.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.853+0000 7f07bf159640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07b8157b80 0x7f07b81ef9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:46.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.853+0000 7f07bf159640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f07b80c1ce0 con 0x7f07b8150dc0 2024-03-19T01:51:46.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.853+0000 7f07bf159640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f07b80c1b60 con 0x7f07b80ca2c0 2024-03-19T01:51:46.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.853+0000 7f07bf159640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f07b8070e40 con 0x7f07b8157b80 2024-03-19T01:51:46.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.853+0000 7f07b7fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07b8150dc0 0x7f07b814b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:46.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.853+0000 7f07b7fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07b8150dc0 0x7f07b814b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:42550/0 (socket says 172.21.15.55:42550) 2024-03-19T01:51:46.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.853+0000 7f07b7fff640 1 -- 172.21.15.55:0/1322872547 learned_addr learned my addr 172.21.15.55:0/1322872547 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:46.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.853+0000 7f07bd6f6640 1 --2- 172.21.15.55:0/1322872547 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07b8157b80 0x7f07b81ef9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:46.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.857+0000 7f07bcef5640 1 --2- 172.21.15.55:0/1322872547 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07b80ca2c0 0x7f07b814c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:46.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.861+0000 7f07b7fff640 1 -- 172.21.15.55:0/1322872547 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07b8157b80 msgr2=0x7f07b81ef9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:46.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.861+0000 7f07b7fff640 1 --2- 172.21.15.55:0/1322872547 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07b8157b80 0x7f07b81ef9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:46.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.861+0000 7f07b7fff640 1 -- 172.21.15.55:0/1322872547 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07b80ca2c0 msgr2=0x7f07b814c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:46.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.861+0000 7f07b7fff640 1 --2- 172.21.15.55:0/1322872547 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07b80ca2c0 0x7f07b814c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:46.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.861+0000 7f07b7fff640 1 -- 172.21.15.55:0/1322872547 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b814c8d0 con 0x7f07b8150dc0 2024-03-19T01:51:46.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.865+0000 7f07b7fff640 1 --2- 172.21.15.55:0/1322872547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07b8150dc0 0x7f07b814b380 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f07ac01f590 tx=0x7f07ac01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:46.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.869+0000 7f07b57fa640 1 -- 172.21.15.55:0/1322872547 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ac0b5070 con 0x7f07b8150dc0 2024-03-19T01:51:46.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.869+0000 7f07b57fa640 1 -- 172.21.15.55:0/1322872547 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07ac0b09f0 con 0x7f07b8150dc0 2024-03-19T01:51:46.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.869+0000 7f07bf159640 1 -- 172.21.15.55:0/1322872547 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07b81f3e20 con 0x7f07b8150dc0 2024-03-19T01:51:46.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.869+0000 7f07b57fa640 1 -- 172.21.15.55:0/1322872547 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ac0b0de0 con 0x7f07b8150dc0 2024-03-19T01:51:46.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.869+0000 7f07bf159640 1 -- 172.21.15.55:0/1322872547 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07b80cad40 con 0x7f07b8150dc0 2024-03-19T01:51:46.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:46.869+0000 7f07bf159640 1 -- 172.21.15.55:0/1322872547 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07b8068a40 con 0x7f07b8150dc0 2024-03-19T01:51:47.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:47.157+0000 7f07b57fa640 1 -- 172.21.15.55:0/1322872547 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f07ac0b9030 con 0x7f07b8150dc0 2024-03-19T01:51:47.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:47.157+0000 7f07b57fa640 1 --2- 172.21.15.55:0/1322872547 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f07a40676e0 0x7f07a4069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:47.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:47.157+0000 7f07bcef5640 1 --2- 172.21.15.55:0/1322872547 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f07a40676e0 0x7f07a4069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:47.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:47.161+0000 7f07bcef5640 1 --2- 172.21.15.55:0/1322872547 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f07a40676e0 0x7f07a4069ba0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f07a8001f20 tx=0x7f07a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:47.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:47.181+0000 7f07b57fa640 1 -- 172.21.15.55:0/1322872547 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f07ac0a8960 con 0x7f07b8150dc0 2024-03-19T01:51:47.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:47.181+0000 7f07b57fa640 1 -- 172.21.15.55:0/1322872547 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f07ac0c1060 con 0x7f07b8150dc0 2024-03-19T01:51:47.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:47.445+0000 7f07bf159640 1 -- 172.21.15.55:0/1322872547 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f07b8158540 con 0x7f07b8150dc0 2024-03-19T01:51:48.672 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.669+0000 7f07b57fa640 1 -- 172.21.15.55:0/1322872547 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v10) ==== 105+0+0 (secure 0 0 0) 0x7f07ac018850 con 0x7f07b8150dc0 2024-03-19T01:51:48.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.673+0000 7f07bf159640 1 -- 172.21.15.55:0/1322872547 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f07a40676e0 msgr2=0x7f07a4069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:48.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.673+0000 7f07bf159640 1 --2- 172.21.15.55:0/1322872547 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f07a40676e0 0x7f07a4069ba0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f07a8001f20 tx=0x7f07a8009040 comp rx=0 tx=0).stop 2024-03-19T01:51:48.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.673+0000 7f07bf159640 1 -- 172.21.15.55:0/1322872547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07b8150dc0 msgr2=0x7f07b814b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:48.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.673+0000 7f07bf159640 1 --2- 172.21.15.55:0/1322872547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07b8150dc0 0x7f07b814b380 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f07ac01f590 tx=0x7f07ac01ba90 comp rx=0 tx=0).stop 2024-03-19T01:51:48.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.673+0000 7f07bf159640 1 -- 172.21.15.55:0/1322872547 shutdown_connections 2024-03-19T01:51:48.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.673+0000 7f07bf159640 1 --2- 172.21.15.55:0/1322872547 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f07a40676e0 0x7f07a4069ba0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:48.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.673+0000 7f07bf159640 1 --2- 172.21.15.55:0/1322872547 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07b8157b80 0x7f07b81ef9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:48.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.673+0000 7f07bf159640 1 --2- 172.21.15.55:0/1322872547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07b8150dc0 0x7f07b814b380 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:48.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.673+0000 7f07bf159640 1 --2- 172.21.15.55:0/1322872547 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07b80ca2c0 0x7f07b814c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:48.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.673+0000 7f07bf159640 1 -- 172.21.15.55:0/1322872547 >> 172.21.15.55:0/1322872547 conn(0x7f07b806c580 msgr2=0x7f07b806b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:48.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.673+0000 7f07bf159640 1 -- 172.21.15.55:0/1322872547 shutdown_connections 2024-03-19T01:51:48.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.673+0000 7f07bf159640 1 -- 172.21.15.55:0/1322872547 wait complete. 2024-03-19T01:51:48.694 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-19T01:51:48.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.809+0000 7f6fa741e640 1 Processor -- start 2024-03-19T01:51:48.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.809+0000 7f6fa741e640 1 -- start start 2024-03-19T01:51:48.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.813+0000 7f6fa741e640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f6fa0154560 0x7f6fa006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:48.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.813+0000 7f6fa741e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fa015ad80 con 0x7f6fa006b500 2024-03-19T01:51:48.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.813+0000 7f6fa741e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fa006cb20 con 0x7f6fa0074570 2024-03-19T01:51:48.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.813+0000 7f6fa741e640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f6fa015ab50 con 0x7f6fa0154560 2024-03-19T01:51:48.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.813+0000 7f6fa51ba640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f6fa006b500 0x7f6fa006b8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:37518/0 (socket says 172.21.15.55:37518) 2024-03-19T01:51:48.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.813+0000 7f6fa51ba640 1 -- 172.21.15.55:0/3413333019 learned_addr learned my addr 172.21.15.55:0/3413333019 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:51:48.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.813+0000 7f6fa59bb640 1 --2- 172.21.15.55:0/3413333019 >> v2:172.21.15.84:3300/0 conn(0x7f6fa0154560 0x7f6fa006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:48.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.817+0000 7f6f97fff640 1 -- 172.21.15.55:0/3413333019 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3395643296 0 0) 0x7f6fa015ad80 con 0x7f6fa006b500 2024-03-19T01:51:48.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.817+0000 7f6f97fff640 1 -- 172.21.15.55:0/3413333019 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f8c003530 con 0x7f6fa006b500 2024-03-19T01:51:48.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.821+0000 7f6fa59bb640 1 -- 172.21.15.55:0/3413333019 >> v1:172.21.15.84:6789/0 conn(0x7f6fa0074570 legacy=0x7f6fa0153e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:48.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.821+0000 7f6fa59bb640 1 -- 172.21.15.55:0/3413333019 >> v1:172.21.15.55:6789/0 conn(0x7f6fa006b500 legacy=0x7f6fa006b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:48.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.821+0000 7f6fa59bb640 1 -- 172.21.15.55:0/3413333019 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa015b020 con 0x7f6fa0154560 2024-03-19T01:51:48.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.825+0000 7f6fa59bb640 1 --2- 172.21.15.55:0/3413333019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6fa0154560 0x7f6fa006dd00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0071fd0 tx=0x7f6f9801f100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df1b90fd992d58c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:48.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.837+0000 7f6f97fff640 1 -- 172.21.15.55:0/3413333019 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f980b4070 con 0x7f6fa0154560 2024-03-19T01:51:48.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.837+0000 7f6f97fff640 1 -- 172.21.15.55:0/3413333019 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f980af600 con 0x7f6fa0154560 2024-03-19T01:51:48.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.837+0000 7f6f97fff640 1 -- 172.21.15.55:0/3413333019 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f980af9d0 con 0x7f6fa0154560 2024-03-19T01:51:48.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.841+0000 7f6fa741e640 1 -- 172.21.15.55:0/3413333019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6fa0154560 msgr2=0x7f6fa006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:48.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.841+0000 7f6fa741e640 1 --2- 172.21.15.55:0/3413333019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6fa0154560 0x7f6fa006dd00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0071fd0 tx=0x7f6f9801f100 comp rx=0 tx=0).stop 2024-03-19T01:51:48.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.841+0000 7f6fa741e640 1 -- 172.21.15.55:0/3413333019 shutdown_connections 2024-03-19T01:51:48.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.841+0000 7f6fa741e640 1 --2- 172.21.15.55:0/3413333019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6fa0154560 0x7f6fa006dd00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:48.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.841+0000 7f6fa741e640 1 -- 172.21.15.55:0/3413333019 >> 172.21.15.55:0/3413333019 conn(0x7f6fa00cc200 msgr2=0x7f6fa00cc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:48.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.841+0000 7f6fa741e640 1 -- 172.21.15.55:0/3413333019 shutdown_connections 2024-03-19T01:51:48.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.841+0000 7f6fa741e640 1 -- 172.21.15.55:0/3413333019 wait complete. 2024-03-19T01:51:48.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.841+0000 7f6fa741e640 1 Processor -- start 2024-03-19T01:51:48.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.841+0000 7f6fa741e640 1 -- start start 2024-03-19T01:51:48.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.841+0000 7f6fa741e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6fa006b500 0x7f6fa0142090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:48.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.841+0000 7f6fa741e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6fa0074570 0x7f6fa01ef8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:48.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.841+0000 7f6fa741e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6fa0154560 0x7f6fa01efe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:48.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.841+0000 7f6fa741e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa006d7e0 con 0x7f6fa006b500 2024-03-19T01:51:48.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.841+0000 7f6fa741e640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa006c840 con 0x7f6fa0154560 2024-03-19T01:51:48.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.841+0000 7f6fa741e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f6fa006bff0 con 0x7f6fa0074570 2024-03-19T01:51:48.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.845+0000 7f6fa49b9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6fa0074570 0x7f6fa01ef8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:48.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.845+0000 7f6fa59bb640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6fa0154560 0x7f6fa01efe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:48.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.845+0000 7f6fa49b9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6fa0074570 0x7f6fa01ef8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:44498/0 (socket says 172.21.15.55:44498) 2024-03-19T01:51:48.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.845+0000 7f6fa49b9640 1 -- 172.21.15.55:0/367216809 learned_addr learned my addr 172.21.15.55:0/367216809 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:48.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.853+0000 7f6fa49b9640 1 -- 172.21.15.55:0/367216809 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6fa0154560 msgr2=0x7f6fa01efe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:48.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.853+0000 7f6fa49b9640 1 --2- 172.21.15.55:0/367216809 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6fa0154560 0x7f6fa01efe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:48.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.853+0000 7f6fa49b9640 1 -- 172.21.15.55:0/367216809 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6fa006b500 msgr2=0x7f6fa0142090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:48.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.853+0000 7f6fa49b9640 1 --2- 172.21.15.55:0/367216809 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6fa006b500 0x7f6fa0142090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:48.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.853+0000 7f6fa49b9640 1 -- 172.21.15.55:0/367216809 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa00c55a0 con 0x7f6fa0074570 2024-03-19T01:51:48.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.861+0000 7f6fa49b9640 1 --2- 172.21.15.55:0/367216809 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6fa0074570 0x7f6fa01ef8f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6f9000c490 tx=0x7f6f90001e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:48.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.869+0000 7f6f95ffb640 1 -- 172.21.15.55:0/367216809 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f90017070 con 0x7f6fa0074570 2024-03-19T01:51:48.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.869+0000 7f6f95ffb640 1 -- 172.21.15.55:0/367216809 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f90003600 con 0x7f6fa0074570 2024-03-19T01:51:48.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.869+0000 7f6f95ffb640 1 -- 172.21.15.55:0/367216809 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f90003a30 con 0x7f6fa0074570 2024-03-19T01:51:48.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.869+0000 7f6fa741e640 1 -- 172.21.15.55:0/367216809 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fa0142b70 con 0x7f6fa0074570 2024-03-19T01:51:48.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.869+0000 7f6fa741e640 1 -- 172.21.15.55:0/367216809 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fa01f30b0 con 0x7f6fa0074570 2024-03-19T01:51:48.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:48.873+0000 7f6fa741e640 1 -- 172.21.15.55:0/367216809 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fa0068330 con 0x7f6fa0074570 2024-03-19T01:51:49.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:49.173+0000 7f6f95ffb640 1 -- 172.21.15.55:0/367216809 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f6f9001a070 con 0x7f6fa0074570 2024-03-19T01:51:49.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:49.177+0000 7f6f95ffb640 1 --2- 172.21.15.55:0/367216809 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6f84067690 0x7f6f84069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:49.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:49.177+0000 7f6fa51ba640 1 --2- 172.21.15.55:0/367216809 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6f84067690 0x7f6f84069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:49.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:49.177+0000 7f6fa51ba640 1 --2- 172.21.15.55:0/367216809 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6f84067690 0x7f6f84069b50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c007fd0 tx=0x7f6f9c007900 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:49.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:49.181+0000 7f6f95ffb640 1 -- 172.21.15.55:0/367216809 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f6f900537e0 con 0x7f6fa0074570 2024-03-19T01:51:49.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:49.197+0000 7f6f95ffb640 1 -- 172.21.15.55:0/367216809 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f6f90021020 con 0x7f6fa0074570 2024-03-19T01:51:49.472 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:49.469+0000 7f6fa741e640 1 -- 172.21.15.55:0/367216809 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f6fa00cc830 con 0x7f6fa0074570 2024-03-19T01:51:50.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.789+0000 7f6f95ffb640 1 -- 172.21.15.55:0/367216809 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v12) ==== 122+0+0 (secure 0 0 0) 0x7f6f90055740 con 0x7f6fa0074570 2024-03-19T01:51:50.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.793+0000 7f6fa741e640 1 -- 172.21.15.55:0/367216809 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6f84067690 msgr2=0x7f6f84069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:50.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.793+0000 7f6fa741e640 1 --2- 172.21.15.55:0/367216809 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6f84067690 0x7f6f84069b50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c007fd0 tx=0x7f6f9c007900 comp rx=0 tx=0).stop 2024-03-19T01:51:50.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.793+0000 7f6fa741e640 1 -- 172.21.15.55:0/367216809 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6fa0074570 msgr2=0x7f6fa01ef8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:50.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.793+0000 7f6fa741e640 1 --2- 172.21.15.55:0/367216809 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6fa0074570 0x7f6fa01ef8f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6f9000c490 tx=0x7f6f90001e80 comp rx=0 tx=0).stop 2024-03-19T01:51:50.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.793+0000 7f6fa741e640 1 -- 172.21.15.55:0/367216809 shutdown_connections 2024-03-19T01:51:50.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.793+0000 7f6fa741e640 1 --2- 172.21.15.55:0/367216809 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6f84067690 0x7f6f84069b50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:50.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.793+0000 7f6fa741e640 1 --2- 172.21.15.55:0/367216809 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6fa0154560 0x7f6fa01efe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:50.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.793+0000 7f6fa741e640 1 --2- 172.21.15.55:0/367216809 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6fa0074570 0x7f6fa01ef8f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:50.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.793+0000 7f6fa741e640 1 --2- 172.21.15.55:0/367216809 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6fa006b500 0x7f6fa0142090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:50.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.793+0000 7f6fa741e640 1 -- 172.21.15.55:0/367216809 >> 172.21.15.55:0/367216809 conn(0x7f6fa00cc200 msgr2=0x7f6fa00c6ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:50.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.793+0000 7f6fa741e640 1 -- 172.21.15.55:0/367216809 shutdown_connections 2024-03-19T01:51:50.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.793+0000 7f6fa741e640 1 -- 172.21.15.55:0/367216809 wait complete. 2024-03-19T01:51:50.812 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-19T01:51:50.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.941+0000 7fd4a88d9640 1 Processor -- start 2024-03-19T01:51:50.945 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.941+0000 7fd4a88d9640 1 -- start start 2024-03-19T01:51:50.945 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.945+0000 7fd4a88d9640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd4a4070780 0x7fd4a415b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:50.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.945+0000 7fd4a88d9640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4a406fff0 con 0x7fd4a415bb00 2024-03-19T01:51:50.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.945+0000 7fd4a88d9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4a40c39c0 con 0x7fd4a4152120 2024-03-19T01:51:50.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.945+0000 7fd4a88d9640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fd4a4152500 con 0x7fd4a4070780 2024-03-19T01:51:50.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.945+0000 7fd4a1d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd4a4070780 0x7fd4a415b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:50.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.945+0000 7fd4a1d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd4a4070780 0x7fd4a415b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34188/0 (socket says 172.21.15.55:34188) 2024-03-19T01:51:50.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.945+0000 7fd4a1d9b640 1 -- 172.21.15.55:0/467280122 learned_addr learned my addr 172.21.15.55:0/467280122 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:50.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.949+0000 7fd4a1d9b640 1 -- 172.21.15.55:0/467280122 >> v1:172.21.15.84:6789/0 conn(0x7fd4a4152120 legacy=0x7fd4a40c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:50.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.949+0000 7fd4a1d9b640 1 -- 172.21.15.55:0/467280122 >> v1:172.21.15.55:6789/0 conn(0x7fd4a415bb00 legacy=0x7fd4a415bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:50.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.949+0000 7fd4a1d9b640 1 -- 172.21.15.55:0/467280122 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4a40702b0 con 0x7fd4a4070780 2024-03-19T01:51:50.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.957+0000 7fd4a1d9b640 1 --2- 172.21.15.55:0/467280122 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4a4070780 0x7fd4a415b5c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd498009870 tx=0x7fd49801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=163040cf1b1d013f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:50.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.961+0000 7fd4a159a640 1 -- 172.21.15.55:0/467280122 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4980b2070 con 0x7fd4a4070780 2024-03-19T01:51:50.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.961+0000 7fd4a159a640 1 -- 172.21.15.55:0/467280122 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4980a9070 con 0x7fd4a4070780 2024-03-19T01:51:50.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.961+0000 7fd4a159a640 1 -- 172.21.15.55:0/467280122 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4980ad880 con 0x7fd4a4070780 2024-03-19T01:51:50.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.961+0000 7fd4a88d9640 1 -- 172.21.15.55:0/467280122 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4a4070780 msgr2=0x7fd4a415b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:50.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.961+0000 7fd4a88d9640 1 --2- 172.21.15.55:0/467280122 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4a4070780 0x7fd4a415b5c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd498009870 tx=0x7fd49801c560 comp rx=0 tx=0).stop 2024-03-19T01:51:50.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.961+0000 7fd4a88d9640 1 -- 172.21.15.55:0/467280122 shutdown_connections 2024-03-19T01:51:50.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.961+0000 7fd4a88d9640 1 --2- 172.21.15.55:0/467280122 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4a4070780 0x7fd4a415b5c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:50.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.961+0000 7fd4a88d9640 1 -- 172.21.15.55:0/467280122 >> 172.21.15.55:0/467280122 conn(0x7fd4a4067600 msgr2=0x7fd4a4071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:50.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.961+0000 7fd4a88d9640 1 -- 172.21.15.55:0/467280122 shutdown_connections 2024-03-19T01:51:50.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.961+0000 7fd4a88d9640 1 -- 172.21.15.55:0/467280122 wait complete. 2024-03-19T01:51:50.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.965+0000 7fd4a88d9640 1 Processor -- start 2024-03-19T01:51:50.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.965+0000 7fd4a88d9640 1 -- start start 2024-03-19T01:51:50.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.965+0000 7fd4a88d9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd4a4070780 0x7fd4a41481f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:50.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.965+0000 7fd4a88d9640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd4a4152120 0x7fd4a40c5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:50.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.965+0000 7fd4a88d9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4a415bb00 0x7fd4a41f1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:50.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.965+0000 7fd4a88d9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a40c97e0 con 0x7fd4a415bb00 2024-03-19T01:51:50.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.965+0000 7fd4a88d9640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a40c3530 con 0x7fd4a4152120 2024-03-19T01:51:50.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.965+0000 7fd4a88d9640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd4a40c86d0 con 0x7fd4a4070780 2024-03-19T01:51:50.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.965+0000 7fd4a2d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4a415bb00 0x7fd4a41f1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:50.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.965+0000 7fd4a2d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4a415bb00 0x7fd4a41f1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34204/0 (socket says 172.21.15.55:34204) 2024-03-19T01:51:50.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.965+0000 7fd4a2d9d640 1 -- 172.21.15.55:0/2971127629 learned_addr learned my addr 172.21.15.55:0/2971127629 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:50.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.969+0000 7fd4a1d9b640 1 --2- 172.21.15.55:0/2971127629 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd4a4152120 0x7fd4a40c5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:50.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.973+0000 7fd4a2d9d640 1 -- 172.21.15.55:0/2971127629 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd4a4070780 msgr2=0x7fd4a41481f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:50.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.973+0000 7fd4a2d9d640 1 --2- 172.21.15.55:0/2971127629 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd4a4070780 0x7fd4a41481f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:50.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.973+0000 7fd4a2d9d640 1 -- 172.21.15.55:0/2971127629 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd4a4152120 msgr2=0x7fd4a40c5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:50.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.973+0000 7fd4a2d9d640 1 --2- 172.21.15.55:0/2971127629 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd4a4152120 0x7fd4a40c5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:50.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.973+0000 7fd4a2d9d640 1 -- 172.21.15.55:0/2971127629 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4a41f4c60 con 0x7fd4a415bb00 2024-03-19T01:51:50.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.977+0000 7fd4a2d9d640 1 --2- 172.21.15.55:0/2971127629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4a415bb00 0x7fd4a41f1a40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd49c00c7a0 tx=0x7fd49c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:50.984 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.981+0000 7fd492ffd640 1 -- 172.21.15.55:0/2971127629 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49c003b80 con 0x7fd4a415bb00 2024-03-19T01:51:50.984 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.981+0000 7fd492ffd640 1 -- 172.21.15.55:0/2971127629 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd49c003d20 con 0x7fd4a415bb00 2024-03-19T01:51:50.984 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.981+0000 7fd492ffd640 1 -- 172.21.15.55:0/2971127629 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49c00aa90 con 0x7fd4a415bb00 2024-03-19T01:51:50.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.985+0000 7fd4a88d9640 1 -- 172.21.15.55:0/2971127629 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4a41f4f40 con 0x7fd4a415bb00 2024-03-19T01:51:50.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.985+0000 7fd4a88d9640 1 -- 172.21.15.55:0/2971127629 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4a41f5500 con 0x7fd4a415bb00 2024-03-19T01:51:50.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:50.985+0000 7fd4a88d9640 1 -- 172.21.15.55:0/2971127629 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4a4068a40 con 0x7fd4a415bb00 2024-03-19T01:51:51.278 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:51.277+0000 7fd492ffd640 1 -- 172.21.15.55:0/2971127629 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fd49c00ac30 con 0x7fd4a415bb00 2024-03-19T01:51:51.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:51.277+0000 7fd492ffd640 1 --2- 172.21.15.55:0/2971127629 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd484067660 0x7fd484069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:51.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:51.277+0000 7fd4a259c640 1 --2- 172.21.15.55:0/2971127629 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd484067660 0x7fd484069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:51.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:51.281+0000 7fd4a259c640 1 --2- 172.21.15.55:0/2971127629 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd484067660 0x7fd484069b20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd494000f70 tx=0x7fd4940079c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:51.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:51.309+0000 7fd492ffd640 1 -- 172.21.15.55:0/2971127629 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fd49c054820 con 0x7fd4a415bb00 2024-03-19T01:51:51.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:51.309+0000 7fd492ffd640 1 -- 172.21.15.55:0/2971127629 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fd49c056e80 con 0x7fd4a415bb00 2024-03-19T01:51:51.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:51.537+0000 7fd4a88d9640 1 -- 172.21.15.55:0/2971127629 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fd4a40c84b0 con 0x7fd4a415bb00 2024-03-19T01:51:52.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.661+0000 7fd492ffd640 1 -- 172.21.15.55:0/2971127629 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v13) ==== 115+0+0 (secure 0 0 0) 0x7fd49c056780 con 0x7fd4a415bb00 2024-03-19T01:51:52.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.665+0000 7fd4a88d9640 1 -- 172.21.15.55:0/2971127629 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd484067660 msgr2=0x7fd484069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:52.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.665+0000 7fd4a88d9640 1 --2- 172.21.15.55:0/2971127629 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd484067660 0x7fd484069b20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd494000f70 tx=0x7fd4940079c0 comp rx=0 tx=0).stop 2024-03-19T01:51:52.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.665+0000 7fd4a88d9640 1 -- 172.21.15.55:0/2971127629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4a415bb00 msgr2=0x7fd4a41f1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:52.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.665+0000 7fd4a88d9640 1 --2- 172.21.15.55:0/2971127629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4a415bb00 0x7fd4a41f1a40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd49c00c7a0 tx=0x7fd49c00cc60 comp rx=0 tx=0).stop 2024-03-19T01:51:52.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.665+0000 7fd4a88d9640 1 -- 172.21.15.55:0/2971127629 shutdown_connections 2024-03-19T01:51:52.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.665+0000 7fd4a88d9640 1 --2- 172.21.15.55:0/2971127629 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd484067660 0x7fd484069b20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:52.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.665+0000 7fd4a88d9640 1 --2- 172.21.15.55:0/2971127629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4a415bb00 0x7fd4a41f1a40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:52.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.665+0000 7fd4a88d9640 1 --2- 172.21.15.55:0/2971127629 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd4a4152120 0x7fd4a40c5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:52.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.665+0000 7fd4a88d9640 1 --2- 172.21.15.55:0/2971127629 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd4a4070780 0x7fd4a41481f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:52.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.665+0000 7fd4a88d9640 1 -- 172.21.15.55:0/2971127629 >> 172.21.15.55:0/2971127629 conn(0x7fd4a4067600 msgr2=0x7fd4a414d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:52.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.665+0000 7fd4a88d9640 1 -- 172.21.15.55:0/2971127629 shutdown_connections 2024-03-19T01:51:52.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.665+0000 7fd4a88d9640 1 -- 172.21.15.55:0/2971127629 wait complete. 2024-03-19T01:51:52.682 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-19T01:51:52.683 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:51:52.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.793+0000 7fdf63e72640 1 Processor -- start 2024-03-19T01:51:52.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.793+0000 7fdf63e72640 1 -- start start 2024-03-19T01:51:52.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.793+0000 7fdf63e72640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdf5c152120 0x7fdf5c0c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:52.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.793+0000 7fdf63e72640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fdf5c070780 0x7fdf5c1586e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:52.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.793+0000 7fdf63e72640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf5c06fff0 con 0x7fdf5c0c9610 2024-03-19T01:51:52.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.793+0000 7fdf63e72640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fdf5c152500 con 0x7fdf5c070780 2024-03-19T01:51:52.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.793+0000 7fdf63e72640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fdf5c070590 con 0x7fdf5c152120 2024-03-19T01:51:52.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.793+0000 7fdf61c0e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdf5c152120 0x7fdf5c0c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:52.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.793+0000 7fdf6140d640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fdf5c0c9610 0x7fdf5c157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:60114/0 (socket says 172.21.15.55:60114) 2024-03-19T01:51:52.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.797+0000 7fdf6140d640 1 -- 172.21.15.55:0/2561058015 learned_addr learned my addr 172.21.15.55:0/2561058015 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:51:52.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.801+0000 7fdf61c0e640 1 -- 172.21.15.55:0/2561058015 >> v2:172.21.15.55:3300/0 conn(0x7fdf5c070780 msgr2=0x7fdf5c1586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:52.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.801+0000 7fdf61c0e640 1 --2- 172.21.15.55:0/2561058015 >> v2:172.21.15.55:3300/0 conn(0x7fdf5c070780 0x7fdf5c1586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:52.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.801+0000 7fdf61c0e640 1 -- 172.21.15.55:0/2561058015 >> v1:172.21.15.84:6789/0 conn(0x7fdf5c0c9610 legacy=0x7fdf5c157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:52.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.801+0000 7fdf61c0e640 1 -- 172.21.15.55:0/2561058015 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf5c0c39c0 con 0x7fdf5c152120 2024-03-19T01:51:52.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.833+0000 7fdf61c0e640 1 --2- 172.21.15.55:0/2561058015 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdf5c152120 0x7fdf5c0c90d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdf580097c0 tx=0x7fdf5801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ac33bc60407f31d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:52.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.945+0000 7fdf60c0c640 1 -- 172.21.15.55:0/2561058015 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf580b0070 con 0x7fdf5c152120 2024-03-19T01:51:52.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.945+0000 7fdf60c0c640 1 -- 172.21.15.55:0/2561058015 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf580ab600 con 0x7fdf5c152120 2024-03-19T01:51:52.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.945+0000 7fdf60c0c640 1 -- 172.21.15.55:0/2561058015 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf580ab9d0 con 0x7fdf5c152120 2024-03-19T01:51:52.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.945+0000 7fdf63e72640 1 -- 172.21.15.55:0/2561058015 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdf5c152120 msgr2=0x7fdf5c0c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:52.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.945+0000 7fdf63e72640 1 --2- 172.21.15.55:0/2561058015 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdf5c152120 0x7fdf5c0c90d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdf580097c0 tx=0x7fdf5801c4a0 comp rx=0 tx=0).stop 2024-03-19T01:51:52.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.945+0000 7fdf63e72640 1 -- 172.21.15.55:0/2561058015 shutdown_connections 2024-03-19T01:51:52.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.945+0000 7fdf63e72640 1 --2- 172.21.15.55:0/2561058015 >> v2:172.21.15.55:3300/0 conn(0x7fdf5c070780 0x7fdf5c1586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:52.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.945+0000 7fdf63e72640 1 --2- 172.21.15.55:0/2561058015 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdf5c152120 0x7fdf5c0c90d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:52.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.945+0000 7fdf63e72640 1 -- 172.21.15.55:0/2561058015 >> 172.21.15.55:0/2561058015 conn(0x7fdf5c067600 msgr2=0x7fdf5c071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:52.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.945+0000 7fdf63e72640 1 -- 172.21.15.55:0/2561058015 shutdown_connections 2024-03-19T01:51:52.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.945+0000 7fdf63e72640 1 -- 172.21.15.55:0/2561058015 wait complete. 2024-03-19T01:51:52.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.949+0000 7fdf63e72640 1 Processor -- start 2024-03-19T01:51:52.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.949+0000 7fdf63e72640 1 -- start start 2024-03-19T01:51:52.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.953+0000 7fdf63e72640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdf5c070780 0x7fdf5c06c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:52.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.953+0000 7fdf63e72640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf5c0c9610 0x7fdf5c0c72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:52.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.953+0000 7fdf63e72640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdf5c152120 0x7fdf5c161e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:52.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.953+0000 7fdf63e72640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c0c86d0 con 0x7fdf5c0c9610 2024-03-19T01:51:52.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.953+0000 7fdf63e72640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c0c31c0 con 0x7fdf5c152120 2024-03-19T01:51:52.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.953+0000 7fdf63e72640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fdf5c165170 con 0x7fdf5c070780 2024-03-19T01:51:52.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.953+0000 7fdf6140d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf5c0c9610 0x7fdf5c0c72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:52.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.953+0000 7fdf6140d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf5c0c9610 0x7fdf5c0c72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34220/0 (socket says 172.21.15.55:34220) 2024-03-19T01:51:52.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.953+0000 7fdf6140d640 1 -- 172.21.15.55:0/681678477 learned_addr learned my addr 172.21.15.55:0/681678477 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:52.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.957+0000 7fdf6240f640 1 --2- 172.21.15.55:0/681678477 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdf5c152120 0x7fdf5c161e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:52.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.965+0000 7fdf61c0e640 1 --2- 172.21.15.55:0/681678477 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdf5c070780 0x7fdf5c06c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:52.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.969+0000 7fdf6140d640 1 -- 172.21.15.55:0/681678477 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdf5c070780 msgr2=0x7fdf5c06c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:52.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.969+0000 7fdf6140d640 1 --2- 172.21.15.55:0/681678477 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdf5c070780 0x7fdf5c06c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:52.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.969+0000 7fdf6140d640 1 -- 172.21.15.55:0/681678477 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdf5c152120 msgr2=0x7fdf5c161e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:52.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.969+0000 7fdf6140d640 1 --2- 172.21.15.55:0/681678477 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdf5c152120 0x7fdf5c161e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:52.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.969+0000 7fdf6140d640 1 -- 172.21.15.55:0/681678477 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf5c06dde0 con 0x7fdf5c0c9610 2024-03-19T01:51:52.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.973+0000 7fdf6140d640 1 --2- 172.21.15.55:0/681678477 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf5c0c9610 0x7fdf5c0c72e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c00c460 tx=0x7fdf4c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:52.984 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.981+0000 7fdf527fc640 1 -- 172.21.15.55:0/681678477 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c017070 con 0x7fdf5c0c9610 2024-03-19T01:51:52.984 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.981+0000 7fdf527fc640 1 -- 172.21.15.55:0/681678477 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf4c005cf0 con 0x7fdf5c0c9610 2024-03-19T01:51:52.984 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.981+0000 7fdf527fc640 1 -- 172.21.15.55:0/681678477 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c007290 con 0x7fdf5c0c9610 2024-03-19T01:51:52.984 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.981+0000 7fdf63e72640 1 -- 172.21.15.55:0/681678477 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf5c0cd930 con 0x7fdf5c0c9610 2024-03-19T01:51:52.984 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.981+0000 7fdf63e72640 1 -- 172.21.15.55:0/681678477 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf5c0cdbd0 con 0x7fdf5c0c9610 2024-03-19T01:51:52.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:52.985+0000 7fdf63e72640 1 -- 172.21.15.55:0/681678477 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf5c068a40 con 0x7fdf5c0c9610 2024-03-19T01:51:53.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.309+0000 7fdf527fc640 1 -- 172.21.15.55:0/681678477 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fdf4c01a070 con 0x7fdf5c0c9610 2024-03-19T01:51:53.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.309+0000 7fdf527fc640 1 --2- 172.21.15.55:0/681678477 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdf400676b0 0x7fdf40069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:53.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.309+0000 7fdf61c0e640 1 --2- 172.21.15.55:0/681678477 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdf400676b0 0x7fdf40069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:53.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.313+0000 7fdf61c0e640 1 --2- 172.21.15.55:0/681678477 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdf400676b0 0x7fdf40069b70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdf580ad000 tx=0x7fdf580a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:53.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.349+0000 7fdf527fc640 1 -- 172.21.15.55:0/681678477 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fdf4c0534c0 con 0x7fdf5c0c9610 2024-03-19T01:51:53.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.349+0000 7fdf527fc640 1 -- 172.21.15.55:0/681678477 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fdf4c021080 con 0x7fdf5c0c9610 2024-03-19T01:51:53.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.565+0000 7fdf63e72640 1 -- 172.21.15.55:0/681678477 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf5c0c84b0 con 0x7fdf5c0c9610 2024-03-19T01:51:53.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.601+0000 7fdf527fc640 1 -- 172.21.15.55:0/681678477 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6842 (secure 0 0 0) 0x7fdf4c055420 con 0x7fdf5c0c9610 2024-03-19T01:51:53.605 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:51:53.605 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4200,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4237,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:51:51.561381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1],"up":{"mds_0":4255,"mds_1":4249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4249":{"gid":4249,"name":"f","rank":1,"incarnation":11,"state":"up:creating","state_seq":1,"addr":"172.21.15.84:6839/354880008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6838","nonce":354880008},{"type":"v1","addr":"172.21.15.84:6839","nonce":354880008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255]},"id":1}]} 2024-03-19T01:51:53.605 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 13 2024-03-19T01:51:53.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.605+0000 7fdf63e72640 1 -- 172.21.15.55:0/681678477 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdf400676b0 msgr2=0x7fdf40069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:53.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.605+0000 7fdf63e72640 1 --2- 172.21.15.55:0/681678477 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdf400676b0 0x7fdf40069b70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdf580ad000 tx=0x7fdf580a6000 comp rx=0 tx=0).stop 2024-03-19T01:51:53.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.605+0000 7fdf63e72640 1 -- 172.21.15.55:0/681678477 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf5c0c9610 msgr2=0x7fdf5c0c72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:53.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.605+0000 7fdf63e72640 1 --2- 172.21.15.55:0/681678477 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf5c0c9610 0x7fdf5c0c72e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c00c460 tx=0x7fdf4c00c920 comp rx=0 tx=0).stop 2024-03-19T01:51:53.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.605+0000 7fdf63e72640 1 -- 172.21.15.55:0/681678477 shutdown_connections 2024-03-19T01:51:53.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.605+0000 7fdf63e72640 1 --2- 172.21.15.55:0/681678477 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdf400676b0 0x7fdf40069b70 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:53.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.605+0000 7fdf63e72640 1 --2- 172.21.15.55:0/681678477 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdf5c152120 0x7fdf5c161e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:53.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.605+0000 7fdf63e72640 1 --2- 172.21.15.55:0/681678477 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf5c0c9610 0x7fdf5c0c72e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:53.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.605+0000 7fdf63e72640 1 --2- 172.21.15.55:0/681678477 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdf5c070780 0x7fdf5c06c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:53.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.605+0000 7fdf63e72640 1 -- 172.21.15.55:0/681678477 >> 172.21.15.55:0/681678477 conn(0x7fdf5c067600 msgr2=0x7fdf5c0ca460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:53.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.605+0000 7fdf63e72640 1 -- 172.21.15.55:0/681678477 shutdown_connections 2024-03-19T01:51:53.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.605+0000 7fdf63e72640 1 -- 172.21.15.55:0/681678477 wait complete. 2024-03-19T01:51:53.623 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:51:53.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.737+0000 7fd0796df640 1 Processor -- start 2024-03-19T01:51:53.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.737+0000 7fd0796df640 1 -- start start 2024-03-19T01:51:53.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.741+0000 7fd0796df640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fd0741544f0 0x7fd074073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:53.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.741+0000 7fd0796df640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0740c7fa0 con 0x7fd07406b2a0 2024-03-19T01:51:53.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.741+0000 7fd0796df640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0740c8240 con 0x7fd07415acb0 2024-03-19T01:51:53.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.741+0000 7fd0796df640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fd07415b210 con 0x7fd0741544f0 2024-03-19T01:51:53.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.741+0000 7fd06bfff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fd07406b2a0 0x7fd07415e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:37534/0 (socket says 172.21.15.55:37534) 2024-03-19T01:51:53.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.741+0000 7fd06bfff640 1 -- 172.21.15.55:0/4020694002 learned_addr learned my addr 172.21.15.55:0/4020694002 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:51:53.742 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.741+0000 7fd072ffd640 1 --2- 172.21.15.55:0/4020694002 >> v2:172.21.15.84:3300/0 conn(0x7fd0741544f0 0x7fd074073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:53.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.745+0000 7fd0727fc640 1 -- 172.21.15.55:0/4020694002 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2755277037 0 0) 0x7fd0740c7fa0 con 0x7fd07406b2a0 2024-03-19T01:51:53.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.745+0000 7fd0727fc640 1 -- 172.21.15.55:0/4020694002 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd060003530 con 0x7fd07406b2a0 2024-03-19T01:51:53.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.745+0000 7fd0727fc640 1 -- 172.21.15.55:0/4020694002 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 856707517 0 0) 0x7fd060003530 con 0x7fd07406b2a0 2024-03-19T01:51:53.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.745+0000 7fd0727fc640 1 -- 172.21.15.55:0/4020694002 >> v2:172.21.15.84:3300/0 conn(0x7fd0741544f0 msgr2=0x7fd074073540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:53.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.745+0000 7fd0727fc640 1 --2- 172.21.15.55:0/4020694002 >> v2:172.21.15.84:3300/0 conn(0x7fd0741544f0 0x7fd074073540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:53.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.749+0000 7fd0727fc640 1 -- 172.21.15.55:0/4020694002 >> v1:172.21.15.84:6789/0 conn(0x7fd07415acb0 legacy=0x7fd0740cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:53.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.749+0000 7fd0727fc640 1 -- 172.21.15.55:0/4020694002 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd07415d920 con 0x7fd07406b2a0 2024-03-19T01:51:53.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.749+0000 7fd0727fc640 1 -- 172.21.15.55:0/4020694002 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7fd05c0016a0 con 0x7fd07406b2a0 2024-03-19T01:51:53.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.749+0000 7fd0727fc640 1 -- 172.21.15.55:0/4020694002 >> v1:172.21.15.55:6789/0 conn(0x7fd07406b2a0 legacy=0x7fd07415e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:53.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.749+0000 7fd0727fc640 1 --2- 172.21.15.55:0/4020694002 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd060003a10 0x7fd060003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:53.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.749+0000 7fd0727fc640 1 --2- 172.21.15.55:0/4020694002 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd060004810 0x7fd060004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:53.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.749+0000 7fd0727fc640 1 --2- 172.21.15.55:0/4020694002 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0600051d0 0x7fd06000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:53.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.749+0000 7fd0727fc640 1 -- 172.21.15.55:0/4020694002 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd07415b210 con 0x7fd060004810 2024-03-19T01:51:53.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.749+0000 7fd0727fc640 1 -- 172.21.15.55:0/4020694002 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd06000c080 con 0x7fd0600051d0 2024-03-19T01:51:53.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.749+0000 7fd0727fc640 1 -- 172.21.15.55:0/4020694002 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd06000c280 con 0x7fd060003a10 2024-03-19T01:51:53.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.749+0000 7fd072ffd640 1 --2- 172.21.15.55:0/4020694002 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd060003a10 0x7fd060003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:53.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.749+0000 7fd06bfff640 1 --2- 172.21.15.55:0/4020694002 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd060004810 0x7fd060004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:53.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.757+0000 7fd06bfff640 1 -- 172.21.15.55:0/4020694002 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd060003a10 msgr2=0x7fd060003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:53.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.757+0000 7fd06bfff640 1 --2- 172.21.15.55:0/4020694002 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd060003a10 0x7fd060003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:53.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.757+0000 7fd06bfff640 1 -- 172.21.15.55:0/4020694002 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0600051d0 msgr2=0x7fd06000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:53.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.757+0000 7fd06bfff640 1 --2- 172.21.15.55:0/4020694002 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0600051d0 0x7fd06000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:53.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.757+0000 7fd06bfff640 1 -- 172.21.15.55:0/4020694002 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd060003530 con 0x7fd060004810 2024-03-19T01:51:53.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.757+0000 7fd0737fe640 1 -- 172.21.15.55:0/4020694002 reap_dead start 2024-03-19T01:51:53.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.761+0000 7fd06bfff640 1 --2- 172.21.15.55:0/4020694002 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd060004810 0x7fd060004c90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd05c00c160 tx=0x7fd05c01cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ba135cf5b251740 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:53.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.765+0000 7fd0727fc640 1 -- 172.21.15.55:0/4020694002 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c007c70 con 0x7fd060004810 2024-03-19T01:51:53.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.765+0000 7fd0727fc640 1 -- 172.21.15.55:0/4020694002 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd05c0a9020 con 0x7fd060004810 2024-03-19T01:51:53.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.765+0000 7fd0727fc640 1 -- 172.21.15.55:0/4020694002 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c005b20 con 0x7fd060004810 2024-03-19T01:51:53.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.765+0000 7fd0796df640 1 -- 172.21.15.55:0/4020694002 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd060004810 msgr2=0x7fd060004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:53.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.765+0000 7fd0796df640 1 --2- 172.21.15.55:0/4020694002 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd060004810 0x7fd060004c90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd05c00c160 tx=0x7fd05c01cb60 comp rx=0 tx=0).stop 2024-03-19T01:51:53.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.765+0000 7fd0796df640 1 -- 172.21.15.55:0/4020694002 shutdown_connections 2024-03-19T01:51:53.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.765+0000 7fd0796df640 1 --2- 172.21.15.55:0/4020694002 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd060004810 0x7fd060004c90 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:53.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.765+0000 7fd0796df640 1 -- 172.21.15.55:0/4020694002 >> 172.21.15.55:0/4020694002 conn(0x7fd07406e670 msgr2=0x7fd07406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:53.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.765+0000 7fd0796df640 1 -- 172.21.15.55:0/4020694002 shutdown_connections 2024-03-19T01:51:53.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.765+0000 7fd0796df640 1 -- 172.21.15.55:0/4020694002 wait complete. 2024-03-19T01:51:53.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.765+0000 7fd0796df640 1 Processor -- start 2024-03-19T01:51:53.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.765+0000 7fd0796df640 1 -- start start 2024-03-19T01:51:53.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.769+0000 7fd0796df640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd060004810 0x7fd0741e4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:53.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.769+0000 7fd0796df640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0740cb680 0x7fd0741f0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:53.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.769+0000 7fd0796df640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0741f0dd0 0x7fd074159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:53.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.769+0000 7fd0796df640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd074073540 con 0x7fd0740cb680 2024-03-19T01:51:53.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.769+0000 7fd0796df640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd0741548d0 con 0x7fd0741f0dd0 2024-03-19T01:51:53.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.769+0000 7fd0796df640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd0740736c0 con 0x7fd060004810 2024-03-19T01:51:53.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.769+0000 7fd06bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0740cb680 0x7fd0741f0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:53.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.769+0000 7fd06bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0740cb680 0x7fd0741f0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34236/0 (socket says 172.21.15.55:34236) 2024-03-19T01:51:53.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.769+0000 7fd06bfff640 1 -- 172.21.15.55:0/1551525880 learned_addr learned my addr 172.21.15.55:0/1551525880 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:53.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.769+0000 7fd0737fe640 1 --2- 172.21.15.55:0/1551525880 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0741f0dd0 0x7fd074159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:53.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.769+0000 7fd072ffd640 1 --2- 172.21.15.55:0/1551525880 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd060004810 0x7fd0741e4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:53.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.773+0000 7fd06bfff640 1 -- 172.21.15.55:0/1551525880 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd060004810 msgr2=0x7fd0741e4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:53.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.773+0000 7fd06bfff640 1 --2- 172.21.15.55:0/1551525880 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd060004810 0x7fd0741e4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:53.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.773+0000 7fd06bfff640 1 -- 172.21.15.55:0/1551525880 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0741f0dd0 msgr2=0x7fd074159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:53.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.773+0000 7fd06bfff640 1 --2- 172.21.15.55:0/1551525880 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0741f0dd0 0x7fd074159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:53.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.773+0000 7fd06bfff640 1 -- 172.21.15.55:0/1551525880 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0740cc410 con 0x7fd0740cb680 2024-03-19T01:51:53.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.773+0000 7fd0737fe640 1 --2- 172.21.15.55:0/1551525880 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0741f0dd0 0x7fd074159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:51:53.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.773+0000 7fd072ffd640 1 --2- 172.21.15.55:0/1551525880 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd060004810 0x7fd0741e4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:51:53.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.777+0000 7fd06bfff640 1 --2- 172.21.15.55:0/1551525880 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0740cb680 0x7fd0741f0890 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd05c000c10 tx=0x7fd060005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:53.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.777+0000 7fd06b7fe640 1 -- 172.21.15.55:0/1551525880 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c0a5db0 con 0x7fd0740cb680 2024-03-19T01:51:53.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.777+0000 7fd06b7fe640 1 -- 172.21.15.55:0/1551525880 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd05c0b2070 con 0x7fd0740cb680 2024-03-19T01:51:53.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.781+0000 7fd0796df640 1 -- 172.21.15.55:0/1551525880 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0740cc6f0 con 0x7fd0740cb680 2024-03-19T01:51:53.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.781+0000 7fd06b7fe640 1 -- 172.21.15.55:0/1551525880 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c004a30 con 0x7fd0740cb680 2024-03-19T01:51:53.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.781+0000 7fd0796df640 1 -- 172.21.15.55:0/1551525880 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0740cbee0 con 0x7fd0740cb680 2024-03-19T01:51:53.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:53.781+0000 7fd0796df640 1 -- 172.21.15.55:0/1551525880 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd074068a40 con 0x7fd0740cb680 2024-03-19T01:51:54.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.061+0000 7fd06b7fe640 1 -- 172.21.15.55:0/1551525880 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fd05c004bd0 con 0x7fd0740cb680 2024-03-19T01:51:54.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.065+0000 7fd06b7fe640 1 --2- 172.21.15.55:0/1551525880 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd0540676b0 0x7fd054069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:54.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.065+0000 7fd072ffd640 1 --2- 172.21.15.55:0/1551525880 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd0540676b0 0x7fd054069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:54.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.065+0000 7fd072ffd640 1 --2- 172.21.15.55:0/1551525880 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd0540676b0 0x7fd054069b70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd0740689e0 tx=0x7fd064008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:54.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.101+0000 7fd06b7fe640 1 -- 172.21.15.55:0/1551525880 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fd05c0edb40 con 0x7fd0740cb680 2024-03-19T01:51:54.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.101+0000 7fd06b7fe640 1 -- 172.21.15.55:0/1551525880 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fd05c016870 con 0x7fd0740cb680 2024-03-19T01:51:54.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.293+0000 7fd0796df640 1 -- 172.21.15.55:0/1551525880 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd0740cc8c0 con 0x7fd0740cb680 2024-03-19T01:51:54.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.345+0000 7fd06b7fe640 1 -- 172.21.15.55:0/1551525880 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+18430 (secure 0 0 0) 0x7fd05c0f01a0 con 0x7fd0740cb680 2024-03-19T01:51:54.345 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:51:54.346 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":29,"fsid":"def9d4e5-0bc7-449d-b893-2490cbd10e25","created":"2024-03-19T01:50:02.482954+0000","modified":"2024-03-19T01:51:17.228291+0000","last_up_change":"2024-03-19T01:50:21.735508+0000","last_in_change":"2024-03-19T01:50:12.168944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:50:23.456904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:51:04.578583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:51:06.871659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T01:51:14.274442+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"35d5e3c5-d089-4f38-8961-98d064188f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6825","nonce":2757675043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6827","nonce":2757675043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6831","nonce":2757675043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6829","nonce":2757675043}]},"public_addr":"172.21.15.55:6825/2757675043","cluster_addr":"172.21.15.55:6827/2757675043","heartbeat_back_addr":"172.21.15.55:6831/2757675043","heartbeat_front_addr":"172.21.15.55:6829/2757675043","state":["exists","up"]},{"osd":1,"uuid":"8e58f791-001a-44d0-a464-ca68a479f70c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6817","nonce":543460007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6819","nonce":543460007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6823","nonce":543460007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6821","nonce":543460007}]},"public_addr":"172.21.15.55:6817/543460007","cluster_addr":"172.21.15.55:6819/543460007","heartbeat_back_addr":"172.21.15.55:6823/543460007","heartbeat_front_addr":"172.21.15.55:6821/543460007","state":["exists","up"]},{"osd":2,"uuid":"d44f38b8-67f4-43de-98a6-f3c520acaf10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6801","nonce":1760253550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6803","nonce":1760253550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6807","nonce":1760253550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6805","nonce":1760253550}]},"public_addr":"172.21.15.55:6801/1760253550","cluster_addr":"172.21.15.55:6803/1760253550","heartbeat_back_addr":"172.21.15.55:6807/1760253550","heartbeat_front_addr":"172.21.15.55:6805/1760253550","state":["exists","up"]},{"osd":3,"uuid":"850a7316-2d74-43f0-8649-b5f8d8f6692e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6809","nonce":2797616448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6811","nonce":2797616448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6815","nonce":2797616448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6813","nonce":2797616448}]},"public_addr":"172.21.15.55:6809/2797616448","cluster_addr":"172.21.15.55:6811/2797616448","heartbeat_back_addr":"172.21.15.55:6815/2797616448","heartbeat_front_addr":"172.21.15.55:6813/2797616448","state":["exists","up"]},{"osd":4,"uuid":"4e6152ae-a77d-4bb6-a855-5782ad5411a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6819","nonce":2938436826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6821","nonce":2938436826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6825","nonce":2938436826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6823","nonce":2938436826}]},"public_addr":"172.21.15.84:6819/2938436826","cluster_addr":"172.21.15.84:6821/2938436826","heartbeat_back_addr":"172.21.15.84:6825/2938436826","heartbeat_front_addr":"172.21.15.84:6823/2938436826","state":["exists","up"]},{"osd":5,"uuid":"1217ceba-dd31-4615-96f1-dfaf57160f3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6803","nonce":460717295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6805","nonce":460717295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6809","nonce":460717295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6807","nonce":460717295}]},"public_addr":"172.21.15.84:6803/460717295","cluster_addr":"172.21.15.84:6805/460717295","heartbeat_back_addr":"172.21.15.84:6809/460717295","heartbeat_front_addr":"172.21.15.84:6807/460717295","state":["exists","up"]},{"osd":6,"uuid":"3d9ed78a-2284-4c28-a24a-70eb812c7836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6827","nonce":416123135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6829","nonce":416123135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6833","nonce":416123135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6831","nonce":416123135}]},"public_addr":"172.21.15.84:6827/416123135","cluster_addr":"172.21.15.84:6829/416123135","heartbeat_back_addr":"172.21.15.84:6833/416123135","heartbeat_front_addr":"172.21.15.84:6831/416123135","state":["exists","up"]},{"osd":7,"uuid":"c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6811","nonce":460083528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6813","nonce":460083528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6817","nonce":460083528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6815","nonce":460083528}]},"public_addr":"172.21.15.84:6811/460083528","cluster_addr":"172.21.15.84:6813/460083528","heartbeat_back_addr":"172.21.15.84:6817/460083528","heartbeat_front_addr":"172.21.15.84:6815/460083528","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.269126+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.121001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.075714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.092461+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.194676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.062833+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.316515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.237111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:51:54.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.349+0000 7fd0796df640 1 -- 172.21.15.55:0/1551525880 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd0540676b0 msgr2=0x7fd054069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:54.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.349+0000 7fd0796df640 1 --2- 172.21.15.55:0/1551525880 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd0540676b0 0x7fd054069b70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd0740689e0 tx=0x7fd064008040 comp rx=0 tx=0).stop 2024-03-19T01:51:54.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.349+0000 7fd0796df640 1 -- 172.21.15.55:0/1551525880 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0740cb680 msgr2=0x7fd0741f0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:54.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.349+0000 7fd0796df640 1 --2- 172.21.15.55:0/1551525880 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0740cb680 0x7fd0741f0890 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd05c000c10 tx=0x7fd060005840 comp rx=0 tx=0).stop 2024-03-19T01:51:54.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.349+0000 7fd0796df640 1 -- 172.21.15.55:0/1551525880 shutdown_connections 2024-03-19T01:51:54.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.349+0000 7fd0796df640 1 --2- 172.21.15.55:0/1551525880 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd0540676b0 0x7fd054069b70 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:54.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.349+0000 7fd0796df640 1 --2- 172.21.15.55:0/1551525880 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0741f0dd0 0x7fd074159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:54.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.349+0000 7fd0796df640 1 --2- 172.21.15.55:0/1551525880 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0740cb680 0x7fd0741f0890 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:54.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.349+0000 7fd0796df640 1 --2- 172.21.15.55:0/1551525880 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd060004810 0x7fd0741e4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:54.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.349+0000 7fd0796df640 1 -- 172.21.15.55:0/1551525880 >> 172.21.15.55:0/1551525880 conn(0x7fd07406e670 msgr2=0x7fd074070220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:54.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.349+0000 7fd0796df640 1 -- 172.21.15.55:0/1551525880 shutdown_connections 2024-03-19T01:51:54.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.349+0000 7fd0796df640 1 -- 172.21.15.55:0/1551525880 wait complete. 2024-03-19T01:51:54.365 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-19T01:51:54.365 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T01:51:54.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.481+0000 7ffa726d2640 1 Processor -- start 2024-03-19T01:51:54.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.481+0000 7ffa726d2640 1 -- start start 2024-03-19T01:51:54.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.485+0000 7ffa726d2640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ffa6c0c39c0 0x7ffa6c158a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:54.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.485+0000 7ffa726d2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffa6c072c60 0x7ffa6c15b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:54.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.485+0000 7ffa726d2640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa6c152660 con 0x7ffa6c06b9f0 2024-03-19T01:51:54.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.485+0000 7ffa726d2640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7ffa6c073130 con 0x7ffa6c0c39c0 2024-03-19T01:51:54.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.485+0000 7ffa726d2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ffa6c0733f0 con 0x7ffa6c072c60 2024-03-19T01:51:54.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.485+0000 7ffa63fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffa6c072c60 0x7ffa6c15b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:54.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.485+0000 7ffa6bfff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ffa6c0c39c0 0x7ffa6c158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:54.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.485+0000 7ffa63fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffa6c072c60 0x7ffa6c15b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:44554/0 (socket says 172.21.15.55:44554) 2024-03-19T01:51:54.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.485+0000 7ffa6bfff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ffa6c0c39c0 0x7ffa6c158a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:59154/0 (socket says 172.21.15.55:59154) 2024-03-19T01:51:54.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.485+0000 7ffa63fff640 1 -- 172.21.15.55:0/1913842157 learned_addr learned my addr 172.21.15.55:0/1913842157 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:54.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.493+0000 7ffa63fff640 1 -- 172.21.15.55:0/1913842157 >> v2:172.21.15.84:3300/0 conn(0x7ffa6c0c39c0 msgr2=0x7ffa6c158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:54.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.493+0000 7ffa63fff640 1 --2- 172.21.15.55:0/1913842157 >> v2:172.21.15.84:3300/0 conn(0x7ffa6c0c39c0 0x7ffa6c158a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:54.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.493+0000 7ffa63fff640 1 -- 172.21.15.55:0/1913842157 >> v1:172.21.15.84:6789/0 conn(0x7ffa6c06b9f0 legacy=0x7ffa6c0c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:54.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.493+0000 7ffa63fff640 1 -- 172.21.15.55:0/1913842157 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6c0736d0 con 0x7ffa6c072c60 2024-03-19T01:51:54.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.501+0000 7ffa63fff640 1 --2- 172.21.15.55:0/1913842157 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffa6c072c60 0x7ffa6c15b540 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ffa54009870 tx=0x7ffa5401eb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d71f5fc63d17b008 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:54.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.513+0000 7ffa6b7fe640 1 -- 172.21.15.55:0/1913842157 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa540b5070 con 0x7ffa6c072c60 2024-03-19T01:51:54.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.513+0000 7ffa6b7fe640 1 -- 172.21.15.55:0/1913842157 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa540a87f0 con 0x7ffa6c072c60 2024-03-19T01:51:54.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.513+0000 7ffa6b7fe640 1 -- 172.21.15.55:0/1913842157 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa540a8bc0 con 0x7ffa6c072c60 2024-03-19T01:51:54.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.513+0000 7ffa726d2640 1 -- 172.21.15.55:0/1913842157 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffa6c072c60 msgr2=0x7ffa6c15b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:54.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.513+0000 7ffa726d2640 1 --2- 172.21.15.55:0/1913842157 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffa6c072c60 0x7ffa6c15b540 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ffa54009870 tx=0x7ffa5401eb30 comp rx=0 tx=0).stop 2024-03-19T01:51:54.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.513+0000 7ffa726d2640 1 -- 172.21.15.55:0/1913842157 shutdown_connections 2024-03-19T01:51:54.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.513+0000 7ffa726d2640 1 --2- 172.21.15.55:0/1913842157 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffa6c072c60 0x7ffa6c15b540 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:54.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.513+0000 7ffa726d2640 1 --2- 172.21.15.55:0/1913842157 >> v2:172.21.15.84:3300/0 conn(0x7ffa6c0c39c0 0x7ffa6c158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:54.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.513+0000 7ffa726d2640 1 -- 172.21.15.55:0/1913842157 >> 172.21.15.55:0/1913842157 conn(0x7ffa6c0cc9c0 msgr2=0x7ffa6c0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:54.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.513+0000 7ffa726d2640 1 -- 172.21.15.55:0/1913842157 shutdown_connections 2024-03-19T01:51:54.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.513+0000 7ffa726d2640 1 -- 172.21.15.55:0/1913842157 wait complete. 2024-03-19T01:51:54.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.513+0000 7ffa726d2640 1 Processor -- start 2024-03-19T01:51:54.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.513+0000 7ffa726d2640 1 -- start start 2024-03-19T01:51:54.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.517+0000 7ffa726d2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffa6c06b9f0 0x7ffa6c06cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:54.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.517+0000 7ffa726d2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffa6c072c60 0x7ffa6c1f0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:54.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.517+0000 7ffa726d2640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffa6c0c39c0 0x7ffa6c0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:54.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.517+0000 7ffa726d2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6c157f50 con 0x7ffa6c072c60 2024-03-19T01:51:54.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.517+0000 7ffa726d2640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6c158560 con 0x7ffa6c0c39c0 2024-03-19T01:51:54.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.517+0000 7ffa726d2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ffa6c0c67e0 con 0x7ffa6c06b9f0 2024-03-19T01:51:54.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.517+0000 7ffa63fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffa6c072c60 0x7ffa6c1f0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:54.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.517+0000 7ffa63fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffa6c072c60 0x7ffa6c1f0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34240/0 (socket says 172.21.15.55:34240) 2024-03-19T01:51:54.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.517+0000 7ffa63fff640 1 -- 172.21.15.55:0/2489388230 learned_addr learned my addr 172.21.15.55:0/2489388230 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:54.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.517+0000 7ffa70c6f640 1 --2- 172.21.15.55:0/2489388230 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffa6c0c39c0 0x7ffa6c0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:54.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.517+0000 7ffa6bfff640 1 --2- 172.21.15.55:0/2489388230 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffa6c06b9f0 0x7ffa6c06cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:54.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.521+0000 7ffa63fff640 1 -- 172.21.15.55:0/2489388230 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffa6c06b9f0 msgr2=0x7ffa6c06cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:54.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.521+0000 7ffa63fff640 1 --2- 172.21.15.55:0/2489388230 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffa6c06b9f0 0x7ffa6c06cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:54.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.521+0000 7ffa63fff640 1 -- 172.21.15.55:0/2489388230 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffa6c0c39c0 msgr2=0x7ffa6c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:54.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.521+0000 7ffa63fff640 1 --2- 172.21.15.55:0/2489388230 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffa6c0c39c0 0x7ffa6c0c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:54.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.521+0000 7ffa63fff640 1 -- 172.21.15.55:0/2489388230 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6c157bc0 con 0x7ffa6c072c60 2024-03-19T01:51:54.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.521+0000 7ffa63fff640 1 --2- 172.21.15.55:0/2489388230 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffa6c072c60 0x7ffa6c1f0bf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ffa5401c760 tx=0x7ffa5401b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:54.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.525+0000 7ffa697fa640 1 -- 172.21.15.55:0/2489388230 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa540be070 con 0x7ffa6c072c60 2024-03-19T01:51:54.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.525+0000 7ffa697fa640 1 -- 172.21.15.55:0/2489388230 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa540b0640 con 0x7ffa6c072c60 2024-03-19T01:51:54.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.525+0000 7ffa726d2640 1 -- 172.21.15.55:0/2489388230 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa6c0c3ec0 con 0x7ffa6c072c60 2024-03-19T01:51:54.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.525+0000 7ffa697fa640 1 -- 172.21.15.55:0/2489388230 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa540b5040 con 0x7ffa6c072c60 2024-03-19T01:51:54.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.529+0000 7ffa726d2640 1 -- 172.21.15.55:0/2489388230 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa6c0c4480 con 0x7ffa6c072c60 2024-03-19T01:51:54.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.529+0000 7ffa726d2640 1 -- 172.21.15.55:0/2489388230 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa6c068a40 con 0x7ffa6c072c60 2024-03-19T01:51:54.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.565+0000 7ffa697fa640 1 -- 172.21.15.55:0/2489388230 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ffa540b0a10 con 0x7ffa6c072c60 2024-03-19T01:51:54.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.565+0000 7ffa697fa640 1 --2- 172.21.15.55:0/2489388230 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffa4c067660 0x7ffa4c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:54.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.565+0000 7ffa6bfff640 1 --2- 172.21.15.55:0/2489388230 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffa4c067660 0x7ffa4c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:54.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.565+0000 7ffa6bfff640 1 --2- 172.21.15.55:0/2489388230 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffa4c067660 0x7ffa4c069b20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c009a50 tx=0x7ffa5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:54.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.569+0000 7ffa697fa640 1 -- 172.21.15.55:0/2489388230 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7ffa540f0bd0 con 0x7ffa6c072c60 2024-03-19T01:51:54.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:54.857+0000 7ffa697fa640 1 -- 172.21.15.55:0/2489388230 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ffa540c7020 con 0x7ffa6c072c60 2024-03-19T01:51:55.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.033+0000 7ffa726d2640 1 -- 172.21.15.55:0/2489388230 --> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ffa6c0cc410 con 0x7ffa4c067660 2024-03-19T01:51:55.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.061+0000 7ffa697fa640 1 -- 172.21.15.55:0/2489388230 <== mgr.4105 v2:172.21.15.84:6800/998533783 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332684 (secure 0 0 0) 0x7ffa6c0cc410 con 0x7ffa4c067660 2024-03-19T01:51:55.065 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:51:55.067 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-19T01:51:55.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.065+0000 7ffa726d2640 1 -- 172.21.15.55:0/2489388230 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffa4c067660 msgr2=0x7ffa4c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:55.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.065+0000 7ffa726d2640 1 --2- 172.21.15.55:0/2489388230 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffa4c067660 0x7ffa4c069b20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c009a50 tx=0x7ffa5c008040 comp rx=0 tx=0).stop 2024-03-19T01:51:55.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.065+0000 7ffa726d2640 1 -- 172.21.15.55:0/2489388230 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffa6c072c60 msgr2=0x7ffa6c1f0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:55.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.065+0000 7ffa726d2640 1 --2- 172.21.15.55:0/2489388230 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffa6c072c60 0x7ffa6c1f0bf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ffa5401c760 tx=0x7ffa5401b8d0 comp rx=0 tx=0).stop 2024-03-19T01:51:55.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.065+0000 7ffa726d2640 1 -- 172.21.15.55:0/2489388230 shutdown_connections 2024-03-19T01:51:55.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.065+0000 7ffa726d2640 1 --2- 172.21.15.55:0/2489388230 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffa4c067660 0x7ffa4c069b20 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:55.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.065+0000 7ffa726d2640 1 --2- 172.21.15.55:0/2489388230 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffa6c0c39c0 0x7ffa6c0c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:55.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.065+0000 7ffa726d2640 1 --2- 172.21.15.55:0/2489388230 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffa6c072c60 0x7ffa6c1f0bf0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:55.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.065+0000 7ffa726d2640 1 --2- 172.21.15.55:0/2489388230 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffa6c06b9f0 0x7ffa6c06cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:55.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.065+0000 7ffa726d2640 1 -- 172.21.15.55:0/2489388230 >> 172.21.15.55:0/2489388230 conn(0x7ffa6c0cc9c0 msgr2=0x7ffa6c155ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:55.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.065+0000 7ffa726d2640 1 -- 172.21.15.55:0/2489388230 shutdown_connections 2024-03-19T01:51:55.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.065+0000 7ffa726d2640 1 -- 172.21.15.55:0/2489388230 wait complete. 2024-03-19T01:51:55.084 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-03-19T01:51:54.326680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596414,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":247,"num_write_kb":2447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236440,"kb_used_data":3672,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738280,"statfs":{"total":773094113280,"available":772851998720,"internally_reserved":0,"allocated":3760128,"data_stored":2185004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1406,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005119"},"pg_stats":[{"pgid":"2.1f","version":"24'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746887+0000","last_change":"2024-03-19T01:51:06.982533+0000","last_active":"2024-03-19T01:51:17.746887+0000","last_peered":"2024-03-19T01:51:17.746887+0000","last_clean":"2024-03-19T01:51:17.746887+0000","last_became_active":"2024-03-19T01:51:06.981802+0000","last_became_peered":"2024-03-19T01:51:06.981802+0000","last_unstale":"2024-03-19T01:51:17.746887+0000","last_undegraded":"2024-03-19T01:51:17.746887+0000","last_fullsized":"2024-03-19T01:51:17.746887+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:10:01.431003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086436+0000","last_change":"2024-03-19T01:51:09.172057+0000","last_active":"2024-03-19T01:51:18.086436+0000","last_peered":"2024-03-19T01:51:18.086436+0000","last_clean":"2024-03-19T01:51:18.086436+0000","last_became_active":"2024-03-19T01:51:09.171753+0000","last_became_peered":"2024-03-19T01:51:09.171753+0000","last_unstale":"2024-03-19T01:51:18.086436+0000","last_undegraded":"2024-03-19T01:51:18.086436+0000","last_fullsized":"2024-03-19T01:51:18.086436+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:50.575804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087409+0000","last_change":"2024-03-19T01:51:16.537753+0000","last_active":"2024-03-19T01:51:18.087409+0000","last_peered":"2024-03-19T01:51:18.087409+0000","last_clean":"2024-03-19T01:51:18.087409+0000","last_became_active":"2024-03-19T01:51:16.534200+0000","last_became_peered":"2024-03-19T01:51:16.534200+0000","last_unstale":"2024-03-19T01:51:18.087409+0000","last_undegraded":"2024-03-19T01:51:18.087409+0000","last_fullsized":"2024-03-19T01:51:18.087409+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:18:34.208854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"24'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086639+0000","last_change":"2024-03-19T01:51:06.962968+0000","last_active":"2024-03-19T01:51:18.086639+0000","last_peered":"2024-03-19T01:51:18.086639+0000","last_clean":"2024-03-19T01:51:18.086639+0000","last_became_active":"2024-03-19T01:51:06.961856+0000","last_became_peered":"2024-03-19T01:51:06.961856+0000","last_unstale":"2024-03-19T01:51:18.086639+0000","last_undegraded":"2024-03-19T01:51:18.086639+0000","last_fullsized":"2024-03-19T01:51:18.086639+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:49.676651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746193+0000","last_change":"2024-03-19T01:51:09.160338+0000","last_active":"2024-03-19T01:51:17.746193+0000","last_peered":"2024-03-19T01:51:17.746193+0000","last_clean":"2024-03-19T01:51:17.746193+0000","last_became_active":"2024-03-19T01:51:09.158682+0000","last_became_peered":"2024-03-19T01:51:09.158682+0000","last_unstale":"2024-03-19T01:51:17.746193+0000","last_undegraded":"2024-03-19T01:51:17.746193+0000","last_fullsized":"2024-03-19T01:51:17.746193+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521163+0000","last_change":"2024-03-19T01:51:16.592390+0000","last_active":"2024-03-19T01:51:17.521163+0000","last_peered":"2024-03-19T01:51:17.521163+0000","last_clean":"2024-03-19T01:51:17.521163+0000","last_became_active":"2024-03-19T01:51:16.591093+0000","last_became_peered":"2024-03-19T01:51:16.591093+0000","last_unstale":"2024-03-19T01:51:17.521163+0000","last_undegraded":"2024-03-19T01:51:17.521163+0000","last_fullsized":"2024-03-19T01:51:17.521163+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:45.300462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"25'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089684+0000","last_change":"2024-03-19T01:51:07.044049+0000","last_active":"2024-03-19T01:51:18.089684+0000","last_peered":"2024-03-19T01:51:18.089684+0000","last_clean":"2024-03-19T01:51:18.089684+0000","last_became_active":"2024-03-19T01:51:07.043209+0000","last_became_peered":"2024-03-19T01:51:07.043209+0000","last_unstale":"2024-03-19T01:51:18.089684+0000","last_undegraded":"2024-03-19T01:51:18.089684+0000","last_fullsized":"2024-03-19T01:51:18.089684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:42.992419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088027+0000","last_change":"2024-03-19T01:51:09.179615+0000","last_active":"2024-03-19T01:51:18.088027+0000","last_peered":"2024-03-19T01:51:18.088027+0000","last_clean":"2024-03-19T01:51:18.088027+0000","last_became_active":"2024-03-19T01:51:09.173962+0000","last_became_peered":"2024-03-19T01:51:09.173962+0000","last_unstale":"2024-03-19T01:51:18.088027+0000","last_undegraded":"2024-03-19T01:51:18.088027+0000","last_fullsized":"2024-03-19T01:51:18.088027+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:06:12.676868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522548+0000","last_change":"2024-03-19T01:51:16.593210+0000","last_active":"2024-03-19T01:51:17.522548+0000","last_peered":"2024-03-19T01:51:17.522548+0000","last_clean":"2024-03-19T01:51:17.522548+0000","last_became_active":"2024-03-19T01:51:16.591235+0000","last_became_peered":"2024-03-19T01:51:16.591235+0000","last_unstale":"2024-03-19T01:51:17.522548+0000","last_undegraded":"2024-03-19T01:51:17.522548+0000","last_fullsized":"2024-03-19T01:51:17.522548+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:13.248793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"25'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088902+0000","last_change":"2024-03-19T01:51:06.983291+0000","last_active":"2024-03-19T01:51:18.088902+0000","last_peered":"2024-03-19T01:51:18.088902+0000","last_clean":"2024-03-19T01:51:18.088902+0000","last_became_active":"2024-03-19T01:51:06.982309+0000","last_became_peered":"2024-03-19T01:51:06.982309+0000","last_unstale":"2024-03-19T01:51:18.088902+0000","last_undegraded":"2024-03-19T01:51:18.088902+0000","last_fullsized":"2024-03-19T01:51:18.088902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:27.305286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089984+0000","last_change":"2024-03-19T01:51:09.160738+0000","last_active":"2024-03-19T01:51:18.089984+0000","last_peered":"2024-03-19T01:51:18.089984+0000","last_clean":"2024-03-19T01:51:18.089984+0000","last_became_active":"2024-03-19T01:51:09.159951+0000","last_became_peered":"2024-03-19T01:51:09.159951+0000","last_unstale":"2024-03-19T01:51:18.089984+0000","last_undegraded":"2024-03-19T01:51:18.089984+0000","last_fullsized":"2024-03-19T01:51:18.089984+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:30.620041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521585+0000","last_change":"2024-03-19T01:51:16.577378+0000","last_active":"2024-03-19T01:51:17.521585+0000","last_peered":"2024-03-19T01:51:17.521585+0000","last_clean":"2024-03-19T01:51:17.521585+0000","last_became_active":"2024-03-19T01:51:16.576256+0000","last_became_peered":"2024-03-19T01:51:16.576256+0000","last_unstale":"2024-03-19T01:51:17.521585+0000","last_undegraded":"2024-03-19T01:51:17.521585+0000","last_fullsized":"2024-03-19T01:51:17.521585+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:16.096547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426589+0000","last_change":"2024-03-19T01:51:09.185969+0000","last_active":"2024-03-19T01:51:17.426589+0000","last_peered":"2024-03-19T01:51:17.426589+0000","last_clean":"2024-03-19T01:51:17.426589+0000","last_became_active":"2024-03-19T01:51:09.185378+0000","last_became_peered":"2024-03-19T01:51:09.185378+0000","last_unstale":"2024-03-19T01:51:17.426589+0000","last_undegraded":"2024-03-19T01:51:17.426589+0000","last_fullsized":"2024-03-19T01:51:17.426589+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.338974+0000","last_change":"2024-03-19T01:51:06.988067+0000","last_active":"2024-03-19T01:51:50.338974+0000","last_peered":"2024-03-19T01:51:50.338974+0000","last_clean":"2024-03-19T01:51:50.338974+0000","last_became_active":"2024-03-19T01:51:06.987450+0000","last_became_peered":"2024-03-19T01:51:06.987450+0000","last_unstale":"2024-03-19T01:51:50.338974+0000","last_undegraded":"2024-03-19T01:51:50.338974+0000","last_fullsized":"2024-03-19T01:51:50.338974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:56.962922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522867+0000","last_change":"2024-03-19T01:51:16.583190+0000","last_active":"2024-03-19T01:51:17.522867+0000","last_peered":"2024-03-19T01:51:17.522867+0000","last_clean":"2024-03-19T01:51:17.522867+0000","last_became_active":"2024-03-19T01:51:16.582532+0000","last_became_peered":"2024-03-19T01:51:16.582532+0000","last_unstale":"2024-03-19T01:51:17.522867+0000","last_undegraded":"2024-03-19T01:51:17.522867+0000","last_fullsized":"2024-03-19T01:51:17.522867+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:16:49.817940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085997+0000","last_change":"2024-03-19T01:51:09.174687+0000","last_active":"2024-03-19T01:51:18.085997+0000","last_peered":"2024-03-19T01:51:18.085997+0000","last_clean":"2024-03-19T01:51:18.085997+0000","last_became_active":"2024-03-19T01:51:09.174409+0000","last_became_peered":"2024-03-19T01:51:09.174409+0000","last_unstale":"2024-03-19T01:51:18.085997+0000","last_undegraded":"2024-03-19T01:51:18.085997+0000","last_fullsized":"2024-03-19T01:51:18.085997+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:33:07.160350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472300+0000","last_change":"2024-03-19T01:51:06.955434+0000","last_active":"2024-03-19T01:51:17.472300+0000","last_peered":"2024-03-19T01:51:17.472300+0000","last_clean":"2024-03-19T01:51:17.472300+0000","last_became_active":"2024-03-19T01:51:06.954635+0000","last_became_peered":"2024-03-19T01:51:06.954635+0000","last_unstale":"2024-03-19T01:51:17.472300+0000","last_undegraded":"2024-03-19T01:51:17.472300+0000","last_fullsized":"2024-03-19T01:51:17.472300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:41:39.176905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521820+0000","last_change":"2024-03-19T01:51:16.588604+0000","last_active":"2024-03-19T01:51:17.521820+0000","last_peered":"2024-03-19T01:51:17.521820+0000","last_clean":"2024-03-19T01:51:17.521820+0000","last_became_active":"2024-03-19T01:51:16.587483+0000","last_became_peered":"2024-03-19T01:51:16.587483+0000","last_unstale":"2024-03-19T01:51:17.521820+0000","last_undegraded":"2024-03-19T01:51:17.521820+0000","last_fullsized":"2024-03-19T01:51:17.521820+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:51.658547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086030+0000","last_change":"2024-03-19T01:51:09.173987+0000","last_active":"2024-03-19T01:51:18.086030+0000","last_peered":"2024-03-19T01:51:18.086030+0000","last_clean":"2024-03-19T01:51:18.086030+0000","last_became_active":"2024-03-19T01:51:09.173648+0000","last_became_peered":"2024-03-19T01:51:09.173648+0000","last_unstale":"2024-03-19T01:51:18.086030+0000","last_undegraded":"2024-03-19T01:51:18.086030+0000","last_fullsized":"2024-03-19T01:51:18.086030+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:13:17.277014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.374560+0000","last_change":"2024-03-19T01:51:06.981820+0000","last_active":"2024-03-19T01:51:50.374560+0000","last_peered":"2024-03-19T01:51:50.374560+0000","last_clean":"2024-03-19T01:51:50.374560+0000","last_became_active":"2024-03-19T01:51:06.981087+0000","last_became_peered":"2024-03-19T01:51:06.981087+0000","last_unstale":"2024-03-19T01:51:50.374560+0000","last_undegraded":"2024-03-19T01:51:50.374560+0000","last_fullsized":"2024-03-19T01:51:50.374560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:25:59.116138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.473071+0000","last_change":"2024-03-19T01:51:16.579017+0000","last_active":"2024-03-19T01:51:17.473071+0000","last_peered":"2024-03-19T01:51:17.473071+0000","last_clean":"2024-03-19T01:51:17.473071+0000","last_became_active":"2024-03-19T01:51:16.573845+0000","last_became_peered":"2024-03-19T01:51:16.573845+0000","last_unstale":"2024-03-19T01:51:17.473071+0000","last_undegraded":"2024-03-19T01:51:17.473071+0000","last_fullsized":"2024-03-19T01:51:17.473071+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:42:30.142766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522501+0000","last_change":"2024-03-19T01:51:09.192886+0000","last_active":"2024-03-19T01:51:17.522501+0000","last_peered":"2024-03-19T01:51:17.522501+0000","last_clean":"2024-03-19T01:51:17.522501+0000","last_became_active":"2024-03-19T01:51:09.185970+0000","last_became_peered":"2024-03-19T01:51:09.185970+0000","last_unstale":"2024-03-19T01:51:17.522501+0000","last_undegraded":"2024-03-19T01:51:17.522501+0000","last_fullsized":"2024-03-19T01:51:17.522501+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:05.196985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.339562+0000","last_change":"2024-03-19T01:51:06.984875+0000","last_active":"2024-03-19T01:51:50.339562+0000","last_peered":"2024-03-19T01:51:50.339562+0000","last_clean":"2024-03-19T01:51:50.339562+0000","last_became_active":"2024-03-19T01:51:06.983858+0000","last_became_peered":"2024-03-19T01:51:06.983858+0000","last_unstale":"2024-03-19T01:51:50.339562+0000","last_undegraded":"2024-03-19T01:51:50.339562+0000","last_fullsized":"2024-03-19T01:51:50.339562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:14.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522293+0000","last_change":"2024-03-19T01:51:16.591614+0000","last_active":"2024-03-19T01:51:17.522293+0000","last_peered":"2024-03-19T01:51:17.522293+0000","last_clean":"2024-03-19T01:51:17.522293+0000","last_became_active":"2024-03-19T01:51:16.590968+0000","last_became_peered":"2024-03-19T01:51:16.590968+0000","last_unstale":"2024-03-19T01:51:17.522293+0000","last_undegraded":"2024-03-19T01:51:17.522293+0000","last_fullsized":"2024-03-19T01:51:17.522293+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:50.144136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746685+0000","last_change":"2024-03-19T01:51:09.168941+0000","last_active":"2024-03-19T01:51:17.746685+0000","last_peered":"2024-03-19T01:51:17.746685+0000","last_clean":"2024-03-19T01:51:17.746685+0000","last_became_active":"2024-03-19T01:51:09.168243+0000","last_became_peered":"2024-03-19T01:51:09.168243+0000","last_unstale":"2024-03-19T01:51:17.746685+0000","last_undegraded":"2024-03-19T01:51:17.746685+0000","last_fullsized":"2024-03-19T01:51:17.746685+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.369887+0000","last_change":"2024-03-19T01:51:07.042870+0000","last_active":"2024-03-19T01:51:50.369887+0000","last_peered":"2024-03-19T01:51:50.369887+0000","last_clean":"2024-03-19T01:51:50.369887+0000","last_became_active":"2024-03-19T01:51:07.042327+0000","last_became_peered":"2024-03-19T01:51:07.042327+0000","last_unstale":"2024-03-19T01:51:50.369887+0000","last_undegraded":"2024-03-19T01:51:50.369887+0000","last_fullsized":"2024-03-19T01:51:50.369887+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:13.137673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521677+0000","last_change":"2024-03-19T01:51:16.587730+0000","last_active":"2024-03-19T01:51:17.521677+0000","last_peered":"2024-03-19T01:51:17.521677+0000","last_clean":"2024-03-19T01:51:17.521677+0000","last_became_active":"2024-03-19T01:51:16.586614+0000","last_became_peered":"2024-03-19T01:51:16.586614+0000","last_unstale":"2024-03-19T01:51:17.521677+0000","last_undegraded":"2024-03-19T01:51:17.521677+0000","last_fullsized":"2024-03-19T01:51:17.521677+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:44.830637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087909+0000","last_change":"2024-03-19T01:51:09.186397+0000","last_active":"2024-03-19T01:51:18.087909+0000","last_peered":"2024-03-19T01:51:18.087909+0000","last_clean":"2024-03-19T01:51:18.087909+0000","last_became_active":"2024-03-19T01:51:09.186033+0000","last_became_peered":"2024-03-19T01:51:09.186033+0000","last_unstale":"2024-03-19T01:51:18.087909+0000","last_undegraded":"2024-03-19T01:51:18.087909+0000","last_fullsized":"2024-03-19T01:51:18.087909+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:03:04.997711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"26'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.473540+0000","last_change":"2024-03-19T01:51:07.043697+0000","last_active":"2024-03-19T01:51:17.473540+0000","last_peered":"2024-03-19T01:51:17.473540+0000","last_clean":"2024-03-19T01:51:17.473540+0000","last_became_active":"2024-03-19T01:51:07.043185+0000","last_became_peered":"2024-03-19T01:51:07.043185+0000","last_unstale":"2024-03-19T01:51:17.473540+0000","last_undegraded":"2024-03-19T01:51:17.473540+0000","last_fullsized":"2024-03-19T01:51:17.473540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:19.016579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085498+0000","last_change":"2024-03-19T01:51:16.535771+0000","last_active":"2024-03-19T01:51:18.085498+0000","last_peered":"2024-03-19T01:51:18.085498+0000","last_clean":"2024-03-19T01:51:18.085498+0000","last_became_active":"2024-03-19T01:51:16.533661+0000","last_became_peered":"2024-03-19T01:51:16.533661+0000","last_unstale":"2024-03-19T01:51:18.085498+0000","last_undegraded":"2024-03-19T01:51:18.085498+0000","last_fullsized":"2024-03-19T01:51:18.085498+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:55.283782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472472+0000","last_change":"2024-03-19T01:51:09.157526+0000","last_active":"2024-03-19T01:51:17.472472+0000","last_peered":"2024-03-19T01:51:17.472472+0000","last_clean":"2024-03-19T01:51:17.472472+0000","last_became_active":"2024-03-19T01:51:09.156336+0000","last_became_peered":"2024-03-19T01:51:09.156336+0000","last_unstale":"2024-03-19T01:51:17.472472+0000","last_undegraded":"2024-03-19T01:51:17.472472+0000","last_fullsized":"2024-03-19T01:51:17.472472+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090045+0000","last_change":"2024-03-19T01:51:06.982994+0000","last_active":"2024-03-19T01:51:18.090045+0000","last_peered":"2024-03-19T01:51:18.090045+0000","last_clean":"2024-03-19T01:51:18.090045+0000","last_became_active":"2024-03-19T01:51:06.982133+0000","last_became_peered":"2024-03-19T01:51:06.982133+0000","last_unstale":"2024-03-19T01:51:18.090045+0000","last_undegraded":"2024-03-19T01:51:18.090045+0000","last_fullsized":"2024-03-19T01:51:18.090045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:03.574098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746076+0000","last_change":"2024-03-19T01:51:16.578493+0000","last_active":"2024-03-19T01:51:17.746076+0000","last_peered":"2024-03-19T01:51:17.746076+0000","last_clean":"2024-03-19T01:51:17.746076+0000","last_became_active":"2024-03-19T01:51:16.578040+0000","last_became_peered":"2024-03-19T01:51:16.578040+0000","last_unstale":"2024-03-19T01:51:17.746076+0000","last_undegraded":"2024-03-19T01:51:17.746076+0000","last_fullsized":"2024-03-19T01:51:17.746076+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:45.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090208+0000","last_change":"2024-03-19T01:51:09.180758+0000","last_active":"2024-03-19T01:51:18.090208+0000","last_peered":"2024-03-19T01:51:18.090208+0000","last_clean":"2024-03-19T01:51:18.090208+0000","last_became_active":"2024-03-19T01:51:09.180326+0000","last_became_peered":"2024-03-19T01:51:09.180326+0000","last_unstale":"2024-03-19T01:51:18.090208+0000","last_undegraded":"2024-03-19T01:51:18.090208+0000","last_fullsized":"2024-03-19T01:51:18.090208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:56:07.046285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472576+0000","last_change":"2024-03-19T01:51:06.962643+0000","last_active":"2024-03-19T01:51:17.472576+0000","last_peered":"2024-03-19T01:51:17.472576+0000","last_clean":"2024-03-19T01:51:17.472576+0000","last_became_active":"2024-03-19T01:51:06.961862+0000","last_became_peered":"2024-03-19T01:51:06.961862+0000","last_unstale":"2024-03-19T01:51:17.472576+0000","last_undegraded":"2024-03-19T01:51:17.472576+0000","last_fullsized":"2024-03-19T01:51:17.472576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.471619+0000","last_change":"2024-03-19T01:51:16.577730+0000","last_active":"2024-03-19T01:51:17.471619+0000","last_peered":"2024-03-19T01:51:17.471619+0000","last_clean":"2024-03-19T01:51:17.471619+0000","last_became_active":"2024-03-19T01:51:16.576955+0000","last_became_peered":"2024-03-19T01:51:16.576955+0000","last_unstale":"2024-03-19T01:51:17.471619+0000","last_undegraded":"2024-03-19T01:51:17.471619+0000","last_fullsized":"2024-03-19T01:51:17.471619+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:35.409504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522368+0000","last_change":"2024-03-19T01:51:09.183678+0000","last_active":"2024-03-19T01:51:17.522368+0000","last_peered":"2024-03-19T01:51:17.522368+0000","last_clean":"2024-03-19T01:51:17.522368+0000","last_became_active":"2024-03-19T01:51:09.176780+0000","last_became_peered":"2024-03-19T01:51:09.176780+0000","last_unstale":"2024-03-19T01:51:17.522368+0000","last_undegraded":"2024-03-19T01:51:17.522368+0000","last_fullsized":"2024-03-19T01:51:17.522368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:11:02.502333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088948+0000","last_change":"2024-03-19T01:51:06.956307+0000","last_active":"2024-03-19T01:51:18.088948+0000","last_peered":"2024-03-19T01:51:18.088948+0000","last_clean":"2024-03-19T01:51:18.088948+0000","last_became_active":"2024-03-19T01:51:06.955529+0000","last_became_peered":"2024-03-19T01:51:06.955529+0000","last_unstale":"2024-03-19T01:51:18.088948+0000","last_undegraded":"2024-03-19T01:51:18.088948+0000","last_fullsized":"2024-03-19T01:51:18.088948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:10.486012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472717+0000","last_change":"2024-03-19T01:51:16.550718+0000","last_active":"2024-03-19T01:51:17.472717+0000","last_peered":"2024-03-19T01:51:17.472717+0000","last_clean":"2024-03-19T01:51:17.472717+0000","last_became_active":"2024-03-19T01:51:16.550115+0000","last_became_peered":"2024-03-19T01:51:16.550115+0000","last_unstale":"2024-03-19T01:51:17.472717+0000","last_undegraded":"2024-03-19T01:51:17.472717+0000","last_fullsized":"2024-03-19T01:51:17.472717+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:43.172796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.471788+0000","last_change":"2024-03-19T01:51:09.155718+0000","last_active":"2024-03-19T01:51:17.471788+0000","last_peered":"2024-03-19T01:51:17.471788+0000","last_clean":"2024-03-19T01:51:17.471788+0000","last_became_active":"2024-03-19T01:51:09.155014+0000","last_became_peered":"2024-03-19T01:51:09.155014+0000","last_unstale":"2024-03-19T01:51:17.471788+0000","last_undegraded":"2024-03-19T01:51:17.471788+0000","last_fullsized":"2024-03-19T01:51:17.471788+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085568+0000","last_change":"2024-03-19T01:51:06.955372+0000","last_active":"2024-03-19T01:51:18.085568+0000","last_peered":"2024-03-19T01:51:18.085568+0000","last_clean":"2024-03-19T01:51:18.085568+0000","last_became_active":"2024-03-19T01:51:06.954637+0000","last_became_peered":"2024-03-19T01:51:06.954637+0000","last_unstale":"2024-03-19T01:51:18.085568+0000","last_undegraded":"2024-03-19T01:51:18.085568+0000","last_fullsized":"2024-03-19T01:51:18.085568+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:38:20.129564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.473243+0000","last_change":"2024-03-19T01:51:16.575157+0000","last_active":"2024-03-19T01:51:17.473243+0000","last_peered":"2024-03-19T01:51:17.473243+0000","last_clean":"2024-03-19T01:51:17.473243+0000","last_became_active":"2024-03-19T01:51:16.574575+0000","last_became_peered":"2024-03-19T01:51:16.574575+0000","last_unstale":"2024-03-19T01:51:17.473243+0000","last_undegraded":"2024-03-19T01:51:17.473243+0000","last_fullsized":"2024-03-19T01:51:17.473243+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:15:17.718630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.745221+0000","last_change":"2024-03-19T01:51:09.192610+0000","last_active":"2024-03-19T01:51:17.745221+0000","last_peered":"2024-03-19T01:51:17.745221+0000","last_clean":"2024-03-19T01:51:17.745221+0000","last_became_active":"2024-03-19T01:51:09.192280+0000","last_became_peered":"2024-03-19T01:51:09.192280+0000","last_unstale":"2024-03-19T01:51:17.745221+0000","last_undegraded":"2024-03-19T01:51:17.745221+0000","last_fullsized":"2024-03-19T01:51:17.745221+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:15.454555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088226+0000","last_change":"2024-03-19T01:51:06.986895+0000","last_active":"2024-03-19T01:51:18.088226+0000","last_peered":"2024-03-19T01:51:18.088226+0000","last_clean":"2024-03-19T01:51:18.088226+0000","last_became_active":"2024-03-19T01:51:06.985965+0000","last_became_peered":"2024-03-19T01:51:06.985965+0000","last_unstale":"2024-03-19T01:51:18.088226+0000","last_undegraded":"2024-03-19T01:51:18.088226+0000","last_fullsized":"2024-03-19T01:51:18.088226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746886+0000","last_change":"2024-03-19T01:51:16.505278+0000","last_active":"2024-03-19T01:51:17.746886+0000","last_peered":"2024-03-19T01:51:17.746886+0000","last_clean":"2024-03-19T01:51:17.746886+0000","last_became_active":"2024-03-19T01:51:16.504908+0000","last_became_peered":"2024-03-19T01:51:16.504908+0000","last_unstale":"2024-03-19T01:51:17.746886+0000","last_undegraded":"2024-03-19T01:51:17.746886+0000","last_fullsized":"2024-03-19T01:51:17.746886+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:42:30.142766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085144+0000","last_change":"2024-03-19T01:51:09.179725+0000","last_active":"2024-03-19T01:51:18.085144+0000","last_peered":"2024-03-19T01:51:18.085144+0000","last_clean":"2024-03-19T01:51:18.085144+0000","last_became_active":"2024-03-19T01:51:09.179235+0000","last_became_peered":"2024-03-19T01:51:09.179235+0000","last_unstale":"2024-03-19T01:51:18.085144+0000","last_undegraded":"2024-03-19T01:51:18.085144+0000","last_fullsized":"2024-03-19T01:51:18.085144+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:31.395833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.490498+0000","last_change":"2024-03-19T01:51:06.953474+0000","last_active":"2024-03-19T01:51:50.490498+0000","last_peered":"2024-03-19T01:51:50.490498+0000","last_clean":"2024-03-19T01:51:50.490498+0000","last_became_active":"2024-03-19T01:51:06.952898+0000","last_became_peered":"2024-03-19T01:51:06.952898+0000","last_unstale":"2024-03-19T01:51:50.490498+0000","last_undegraded":"2024-03-19T01:51:50.490498+0000","last_fullsized":"2024-03-19T01:51:50.490498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:38.808918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":184,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088303+0000","last_change":"2024-03-19T01:50:26.097484+0000","last_active":"2024-03-19T01:51:18.088303+0000","last_peered":"2024-03-19T01:51:18.088303+0000","last_clean":"2024-03-19T01:51:18.088303+0000","last_became_active":"2024-03-19T01:50:26.095496+0000","last_became_peered":"2024-03-19T01:50:26.095496+0000","last_unstale":"2024-03-19T01:51:18.088303+0000","last_undegraded":"2024-03-19T01:51:18.088303+0000","last_fullsized":"2024-03-19T01:51:18.088303+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:50:24.112070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:50:24.112070+0000","last_clean_scrub_stamp":"2024-03-19T01:50:24.112070+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:05:00.843775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426122+0000","last_change":"2024-03-19T01:51:16.580516+0000","last_active":"2024-03-19T01:51:17.426122+0000","last_peered":"2024-03-19T01:51:17.426122+0000","last_clean":"2024-03-19T01:51:17.426122+0000","last_became_active":"2024-03-19T01:51:16.579906+0000","last_became_peered":"2024-03-19T01:51:16.579906+0000","last_unstale":"2024-03-19T01:51:17.426122+0000","last_undegraded":"2024-03-19T01:51:17.426122+0000","last_fullsized":"2024-03-19T01:51:17.426122+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:53:57.897649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089446+0000","last_change":"2024-03-19T01:51:09.196648+0000","last_active":"2024-03-19T01:51:18.089446+0000","last_peered":"2024-03-19T01:51:18.089446+0000","last_clean":"2024-03-19T01:51:18.089446+0000","last_became_active":"2024-03-19T01:51:09.196093+0000","last_became_peered":"2024-03-19T01:51:09.196093+0000","last_unstale":"2024-03-19T01:51:18.089446+0000","last_undegraded":"2024-03-19T01:51:18.089446+0000","last_fullsized":"2024-03-19T01:51:18.089446+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:54:09.655994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086018+0000","last_change":"2024-03-19T01:51:06.950023+0000","last_active":"2024-03-19T01:51:18.086018+0000","last_peered":"2024-03-19T01:51:18.086018+0000","last_clean":"2024-03-19T01:51:18.086018+0000","last_became_active":"2024-03-19T01:51:06.949234+0000","last_became_peered":"2024-03-19T01:51:06.949234+0000","last_unstale":"2024-03-19T01:51:18.086018+0000","last_undegraded":"2024-03-19T01:51:18.086018+0000","last_fullsized":"2024-03-19T01:51:18.086018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:54:13.131325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425731+0000","last_change":"2024-03-19T01:51:16.554046+0000","last_active":"2024-03-19T01:51:17.425731+0000","last_peered":"2024-03-19T01:51:17.425731+0000","last_clean":"2024-03-19T01:51:17.425731+0000","last_became_active":"2024-03-19T01:51:16.553595+0000","last_became_peered":"2024-03-19T01:51:16.553595+0000","last_unstale":"2024-03-19T01:51:17.425731+0000","last_undegraded":"2024-03-19T01:51:17.425731+0000","last_fullsized":"2024-03-19T01:51:17.425731+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:25.684226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088279+0000","last_change":"2024-03-19T01:51:09.181518+0000","last_active":"2024-03-19T01:51:18.088279+0000","last_peered":"2024-03-19T01:51:18.088279+0000","last_clean":"2024-03-19T01:51:18.088279+0000","last_became_active":"2024-03-19T01:51:09.174122+0000","last_became_peered":"2024-03-19T01:51:09.174122+0000","last_unstale":"2024-03-19T01:51:18.088279+0000","last_undegraded":"2024-03-19T01:51:18.088279+0000","last_fullsized":"2024-03-19T01:51:18.088279+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:18:53.079890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"24'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472911+0000","last_change":"2024-03-19T01:51:06.938396+0000","last_active":"2024-03-19T01:51:17.472911+0000","last_peered":"2024-03-19T01:51:17.472911+0000","last_clean":"2024-03-19T01:51:17.472911+0000","last_became_active":"2024-03-19T01:51:06.937580+0000","last_became_peered":"2024-03-19T01:51:06.937580+0000","last_unstale":"2024-03-19T01:51:17.472911+0000","last_undegraded":"2024-03-19T01:51:17.472911+0000","last_fullsized":"2024-03-19T01:51:17.472911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:08.307376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746471+0000","last_change":"2024-03-19T01:51:16.582111+0000","last_active":"2024-03-19T01:51:17.746471+0000","last_peered":"2024-03-19T01:51:17.746471+0000","last_clean":"2024-03-19T01:51:17.746471+0000","last_became_active":"2024-03-19T01:51:16.581746+0000","last_became_peered":"2024-03-19T01:51:16.581746+0000","last_unstale":"2024-03-19T01:51:17.746471+0000","last_undegraded":"2024-03-19T01:51:17.746471+0000","last_fullsized":"2024-03-19T01:51:17.746471+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:19.442287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090581+0000","last_change":"2024-03-19T01:51:09.195225+0000","last_active":"2024-03-19T01:51:18.090581+0000","last_peered":"2024-03-19T01:51:18.090581+0000","last_clean":"2024-03-19T01:51:18.090581+0000","last_became_active":"2024-03-19T01:51:09.194645+0000","last_became_peered":"2024-03-19T01:51:09.194645+0000","last_unstale":"2024-03-19T01:51:18.090581+0000","last_undegraded":"2024-03-19T01:51:18.090581+0000","last_fullsized":"2024-03-19T01:51:18.090581+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:50:04.024767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"24'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086668+0000","last_change":"2024-03-19T01:51:06.955976+0000","last_active":"2024-03-19T01:51:18.086668+0000","last_peered":"2024-03-19T01:51:18.086668+0000","last_clean":"2024-03-19T01:51:18.086668+0000","last_became_active":"2024-03-19T01:51:06.954951+0000","last_became_peered":"2024-03-19T01:51:06.954951+0000","last_unstale":"2024-03-19T01:51:18.086668+0000","last_undegraded":"2024-03-19T01:51:18.086668+0000","last_fullsized":"2024-03-19T01:51:18.086668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:47.536586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087418+0000","last_change":"2024-03-19T01:51:16.586904+0000","last_active":"2024-03-19T01:51:18.087418+0000","last_peered":"2024-03-19T01:51:18.087418+0000","last_clean":"2024-03-19T01:51:18.087418+0000","last_became_active":"2024-03-19T01:51:16.586298+0000","last_became_peered":"2024-03-19T01:51:16.586298+0000","last_unstale":"2024-03-19T01:51:18.087418+0000","last_undegraded":"2024-03-19T01:51:18.087418+0000","last_fullsized":"2024-03-19T01:51:18.087418+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:42.703415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089106+0000","last_change":"2024-03-19T01:51:09.195400+0000","last_active":"2024-03-19T01:51:18.089106+0000","last_peered":"2024-03-19T01:51:18.089106+0000","last_clean":"2024-03-19T01:51:18.089106+0000","last_became_active":"2024-03-19T01:51:09.195111+0000","last_became_peered":"2024-03-19T01:51:09.195111+0000","last_unstale":"2024-03-19T01:51:18.089106+0000","last_undegraded":"2024-03-19T01:51:18.089106+0000","last_fullsized":"2024-03-19T01:51:18.089106+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:12.084475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"24'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522397+0000","last_change":"2024-03-19T01:51:06.963650+0000","last_active":"2024-03-19T01:51:17.522397+0000","last_peered":"2024-03-19T01:51:17.522397+0000","last_clean":"2024-03-19T01:51:17.522397+0000","last_became_active":"2024-03-19T01:51:06.963028+0000","last_became_peered":"2024-03-19T01:51:06.963028+0000","last_unstale":"2024-03-19T01:51:17.522397+0000","last_undegraded":"2024-03-19T01:51:17.522397+0000","last_fullsized":"2024-03-19T01:51:17.522397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:59:26.369856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521307+0000","last_change":"2024-03-19T01:51:16.588077+0000","last_active":"2024-03-19T01:51:17.521307+0000","last_peered":"2024-03-19T01:51:17.521307+0000","last_clean":"2024-03-19T01:51:17.521307+0000","last_became_active":"2024-03-19T01:51:16.587459+0000","last_became_peered":"2024-03-19T01:51:16.587459+0000","last_unstale":"2024-03-19T01:51:17.521307+0000","last_undegraded":"2024-03-19T01:51:17.521307+0000","last_fullsized":"2024-03-19T01:51:17.521307+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:52.667114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089490+0000","last_change":"2024-03-19T01:51:09.195758+0000","last_active":"2024-03-19T01:51:18.089490+0000","last_peered":"2024-03-19T01:51:18.089490+0000","last_clean":"2024-03-19T01:51:18.089490+0000","last_became_active":"2024-03-19T01:51:09.195139+0000","last_became_peered":"2024-03-19T01:51:09.195139+0000","last_unstale":"2024-03-19T01:51:18.089490+0000","last_undegraded":"2024-03-19T01:51:18.089490+0000","last_fullsized":"2024-03-19T01:51:18.089490+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:18.326414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"29'10","reported_seq":39,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.372190+0000","last_change":"2024-03-19T01:51:06.941200+0000","last_active":"2024-03-19T01:51:50.372190+0000","last_peered":"2024-03-19T01:51:50.372190+0000","last_clean":"2024-03-19T01:51:50.372190+0000","last_became_active":"2024-03-19T01:51:06.940594+0000","last_became_peered":"2024-03-19T01:51:06.940594+0000","last_unstale":"2024-03-19T01:51:50.372190+0000","last_undegraded":"2024-03-19T01:51:50.372190+0000","last_fullsized":"2024-03-19T01:51:50.372190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:25:05.442423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746500+0000","last_change":"2024-03-19T01:51:16.548804+0000","last_active":"2024-03-19T01:51:17.746500+0000","last_peered":"2024-03-19T01:51:17.746500+0000","last_clean":"2024-03-19T01:51:17.746500+0000","last_became_active":"2024-03-19T01:51:16.547800+0000","last_became_peered":"2024-03-19T01:51:16.547800+0000","last_unstale":"2024-03-19T01:51:17.746500+0000","last_undegraded":"2024-03-19T01:51:17.746500+0000","last_fullsized":"2024-03-19T01:51:17.746500+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:43.172796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425380+0000","last_change":"2024-03-19T01:51:09.167987+0000","last_active":"2024-03-19T01:51:17.425380+0000","last_peered":"2024-03-19T01:51:17.425380+0000","last_clean":"2024-03-19T01:51:17.425380+0000","last_became_active":"2024-03-19T01:51:09.167670+0000","last_became_peered":"2024-03-19T01:51:09.167670+0000","last_unstale":"2024-03-19T01:51:17.425380+0000","last_undegraded":"2024-03-19T01:51:17.425380+0000","last_fullsized":"2024-03-19T01:51:17.425380+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425534+0000","last_change":"2024-03-19T01:51:07.048197+0000","last_active":"2024-03-19T01:51:17.425534+0000","last_peered":"2024-03-19T01:51:17.425534+0000","last_clean":"2024-03-19T01:51:17.425534+0000","last_became_active":"2024-03-19T01:51:07.047785+0000","last_became_peered":"2024-03-19T01:51:07.047785+0000","last_unstale":"2024-03-19T01:51:17.425534+0000","last_undegraded":"2024-03-19T01:51:17.425534+0000","last_fullsized":"2024-03-19T01:51:17.425534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:52.323249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087324+0000","last_change":"2024-03-19T01:51:16.583394+0000","last_active":"2024-03-19T01:51:18.087324+0000","last_peered":"2024-03-19T01:51:18.087324+0000","last_clean":"2024-03-19T01:51:18.087324+0000","last_became_active":"2024-03-19T01:51:16.582860+0000","last_became_peered":"2024-03-19T01:51:16.582860+0000","last_unstale":"2024-03-19T01:51:18.087324+0000","last_undegraded":"2024-03-19T01:51:18.087324+0000","last_fullsized":"2024-03-19T01:51:18.087324+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:37.669591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.091680+0000","last_change":"2024-03-19T01:51:09.195008+0000","last_active":"2024-03-19T01:51:18.091680+0000","last_peered":"2024-03-19T01:51:18.091680+0000","last_clean":"2024-03-19T01:51:18.091680+0000","last_became_active":"2024-03-19T01:51:09.194728+0000","last_became_peered":"2024-03-19T01:51:09.194728+0000","last_unstale":"2024-03-19T01:51:18.091680+0000","last_undegraded":"2024-03-19T01:51:18.091680+0000","last_fullsized":"2024-03-19T01:51:18.091680+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:15:10.360817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.490004+0000","last_change":"2024-03-19T01:51:06.956905+0000","last_active":"2024-03-19T01:51:50.490004+0000","last_peered":"2024-03-19T01:51:50.490004+0000","last_clean":"2024-03-19T01:51:50.490004+0000","last_became_active":"2024-03-19T01:51:06.955991+0000","last_became_peered":"2024-03-19T01:51:06.955991+0000","last_unstale":"2024-03-19T01:51:50.490004+0000","last_undegraded":"2024-03-19T01:51:50.490004+0000","last_fullsized":"2024-03-19T01:51:50.490004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:02:45.970853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087664+0000","last_change":"2024-03-19T01:51:16.539701+0000","last_active":"2024-03-19T01:51:18.087664+0000","last_peered":"2024-03-19T01:51:18.087664+0000","last_clean":"2024-03-19T01:51:18.087664+0000","last_became_active":"2024-03-19T01:51:16.539254+0000","last_became_peered":"2024-03-19T01:51:16.539254+0000","last_unstale":"2024-03-19T01:51:18.087664+0000","last_undegraded":"2024-03-19T01:51:18.087664+0000","last_fullsized":"2024-03-19T01:51:18.087664+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:40:24.244131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.745703+0000","last_change":"2024-03-19T01:51:09.188003+0000","last_active":"2024-03-19T01:51:17.745703+0000","last_peered":"2024-03-19T01:51:17.745703+0000","last_clean":"2024-03-19T01:51:17.745703+0000","last_became_active":"2024-03-19T01:51:09.187275+0000","last_became_peered":"2024-03-19T01:51:09.187275+0000","last_unstale":"2024-03-19T01:51:17.745703+0000","last_undegraded":"2024-03-19T01:51:17.745703+0000","last_fullsized":"2024-03-19T01:51:17.745703+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:10.624259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.344999+0000","last_change":"2024-03-19T01:51:06.964413+0000","last_active":"2024-03-19T01:51:50.344999+0000","last_peered":"2024-03-19T01:51:50.344999+0000","last_clean":"2024-03-19T01:51:50.344999+0000","last_became_active":"2024-03-19T01:51:06.963763+0000","last_became_peered":"2024-03-19T01:51:06.963763+0000","last_unstale":"2024-03-19T01:51:50.344999+0000","last_undegraded":"2024-03-19T01:51:50.344999+0000","last_fullsized":"2024-03-19T01:51:50.344999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089726+0000","last_change":"2024-03-19T01:51:16.574401+0000","last_active":"2024-03-19T01:51:18.089726+0000","last_peered":"2024-03-19T01:51:18.089726+0000","last_clean":"2024-03-19T01:51:18.089726+0000","last_became_active":"2024-03-19T01:51:16.573744+0000","last_became_peered":"2024-03-19T01:51:16.573744+0000","last_unstale":"2024-03-19T01:51:18.089726+0000","last_undegraded":"2024-03-19T01:51:18.089726+0000","last_fullsized":"2024-03-19T01:51:18.089726+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:41.414631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089315+0000","last_change":"2024-03-19T01:51:09.187207+0000","last_active":"2024-03-19T01:51:18.089315+0000","last_peered":"2024-03-19T01:51:18.089315+0000","last_clean":"2024-03-19T01:51:18.089315+0000","last_became_active":"2024-03-19T01:51:09.175898+0000","last_became_peered":"2024-03-19T01:51:09.175898+0000","last_unstale":"2024-03-19T01:51:18.089315+0000","last_undegraded":"2024-03-19T01:51:18.089315+0000","last_fullsized":"2024-03-19T01:51:18.089315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:38.387069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'12","reported_seq":41,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.370011+0000","last_change":"2024-03-19T01:51:07.042663+0000","last_active":"2024-03-19T01:51:50.370011+0000","last_peered":"2024-03-19T01:51:50.370011+0000","last_clean":"2024-03-19T01:51:50.370011+0000","last_became_active":"2024-03-19T01:51:07.041262+0000","last_became_peered":"2024-03-19T01:51:07.041262+0000","last_unstale":"2024-03-19T01:51:50.370011+0000","last_undegraded":"2024-03-19T01:51:50.370011+0000","last_fullsized":"2024-03-19T01:51:50.370011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.471923+0000","last_change":"2024-03-19T01:51:16.549744+0000","last_active":"2024-03-19T01:51:17.471923+0000","last_peered":"2024-03-19T01:51:17.471923+0000","last_clean":"2024-03-19T01:51:17.471923+0000","last_became_active":"2024-03-19T01:51:16.549347+0000","last_became_peered":"2024-03-19T01:51:16.549347+0000","last_unstale":"2024-03-19T01:51:17.471923+0000","last_undegraded":"2024-03-19T01:51:17.471923+0000","last_fullsized":"2024-03-19T01:51:17.471923+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:19.442287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522393+0000","last_change":"2024-03-19T01:51:09.194165+0000","last_active":"2024-03-19T01:51:17.522393+0000","last_peered":"2024-03-19T01:51:17.522393+0000","last_clean":"2024-03-19T01:51:17.522393+0000","last_became_active":"2024-03-19T01:51:09.183607+0000","last_became_peered":"2024-03-19T01:51:09.183607+0000","last_unstale":"2024-03-19T01:51:17.522393+0000","last_undegraded":"2024-03-19T01:51:17.522393+0000","last_fullsized":"2024-03-19T01:51:17.522393+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:09.211794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.473060+0000","last_change":"2024-03-19T01:51:06.952515+0000","last_active":"2024-03-19T01:51:17.473060+0000","last_peered":"2024-03-19T01:51:17.473060+0000","last_clean":"2024-03-19T01:51:17.473060+0000","last_became_active":"2024-03-19T01:51:06.951700+0000","last_became_peered":"2024-03-19T01:51:06.951700+0000","last_unstale":"2024-03-19T01:51:17.473060+0000","last_undegraded":"2024-03-19T01:51:17.473060+0000","last_fullsized":"2024-03-19T01:51:17.473060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:13.137673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522152+0000","last_change":"2024-03-19T01:51:16.587027+0000","last_active":"2024-03-19T01:51:17.522152+0000","last_peered":"2024-03-19T01:51:17.522152+0000","last_clean":"2024-03-19T01:51:17.522152+0000","last_became_active":"2024-03-19T01:51:16.586514+0000","last_became_peered":"2024-03-19T01:51:16.586514+0000","last_unstale":"2024-03-19T01:51:17.522152+0000","last_undegraded":"2024-03-19T01:51:17.522152+0000","last_fullsized":"2024-03-19T01:51:17.522152+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:06.637617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746586+0000","last_change":"2024-03-19T01:51:09.172554+0000","last_active":"2024-03-19T01:51:17.746586+0000","last_peered":"2024-03-19T01:51:17.746586+0000","last_clean":"2024-03-19T01:51:17.746586+0000","last_became_active":"2024-03-19T01:51:09.171814+0000","last_became_peered":"2024-03-19T01:51:09.171814+0000","last_unstale":"2024-03-19T01:51:17.746586+0000","last_undegraded":"2024-03-19T01:51:17.746586+0000","last_fullsized":"2024-03-19T01:51:17.746586+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.488757+0000","last_change":"2024-03-19T01:51:07.043404+0000","last_active":"2024-03-19T01:51:50.488757+0000","last_peered":"2024-03-19T01:51:50.488757+0000","last_clean":"2024-03-19T01:51:50.488757+0000","last_became_active":"2024-03-19T01:51:07.043069+0000","last_became_peered":"2024-03-19T01:51:07.043069+0000","last_unstale":"2024-03-19T01:51:50.488757+0000","last_undegraded":"2024-03-19T01:51:50.488757+0000","last_fullsized":"2024-03-19T01:51:50.488757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:13:14.960132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085834+0000","last_change":"2024-03-19T01:51:16.581797+0000","last_active":"2024-03-19T01:51:18.085834+0000","last_peered":"2024-03-19T01:51:18.085834+0000","last_clean":"2024-03-19T01:51:18.085834+0000","last_became_active":"2024-03-19T01:51:16.581174+0000","last_became_peered":"2024-03-19T01:51:16.581174+0000","last_unstale":"2024-03-19T01:51:18.085834+0000","last_undegraded":"2024-03-19T01:51:18.085834+0000","last_fullsized":"2024-03-19T01:51:18.085834+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:29:34.911347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090870+0000","last_change":"2024-03-19T01:51:09.194372+0000","last_active":"2024-03-19T01:51:18.090870+0000","last_peered":"2024-03-19T01:51:18.090870+0000","last_clean":"2024-03-19T01:51:18.090870+0000","last_became_active":"2024-03-19T01:51:09.193656+0000","last_became_peered":"2024-03-19T01:51:09.193656+0000","last_unstale":"2024-03-19T01:51:18.090870+0000","last_undegraded":"2024-03-19T01:51:18.090870+0000","last_fullsized":"2024-03-19T01:51:18.090870+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:23.464803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.371819+0000","last_change":"2024-03-19T01:51:07.046089+0000","last_active":"2024-03-19T01:51:50.371819+0000","last_peered":"2024-03-19T01:51:50.371819+0000","last_clean":"2024-03-19T01:51:50.371819+0000","last_became_active":"2024-03-19T01:51:07.045342+0000","last_became_peered":"2024-03-19T01:51:07.045342+0000","last_unstale":"2024-03-19T01:51:50.371819+0000","last_undegraded":"2024-03-19T01:51:50.371819+0000","last_fullsized":"2024-03-19T01:51:50.371819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:26.105915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087835+0000","last_change":"2024-03-19T01:51:16.572559+0000","last_active":"2024-03-19T01:51:18.087835+0000","last_peered":"2024-03-19T01:51:18.087835+0000","last_clean":"2024-03-19T01:51:18.087835+0000","last_became_active":"2024-03-19T01:51:16.572283+0000","last_became_peered":"2024-03-19T01:51:16.572283+0000","last_unstale":"2024-03-19T01:51:18.087835+0000","last_undegraded":"2024-03-19T01:51:18.087835+0000","last_fullsized":"2024-03-19T01:51:18.087835+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:42.703415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746448+0000","last_change":"2024-03-19T01:51:16.534088+0000","last_active":"2024-03-19T01:51:17.746448+0000","last_peered":"2024-03-19T01:51:17.746448+0000","last_clean":"2024-03-19T01:51:17.746448+0000","last_became_active":"2024-03-19T01:51:16.533665+0000","last_became_peered":"2024-03-19T01:51:16.533665+0000","last_unstale":"2024-03-19T01:51:17.746448+0000","last_undegraded":"2024-03-19T01:51:17.746448+0000","last_fullsized":"2024-03-19T01:51:17.746448+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:35.409504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.471346+0000","last_change":"2024-03-19T01:51:09.168372+0000","last_active":"2024-03-19T01:51:17.471346+0000","last_peered":"2024-03-19T01:51:17.471346+0000","last_clean":"2024-03-19T01:51:17.471346+0000","last_became_active":"2024-03-19T01:51:09.167626+0000","last_became_peered":"2024-03-19T01:51:09.167626+0000","last_unstale":"2024-03-19T01:51:17.471346+0000","last_undegraded":"2024-03-19T01:51:17.471346+0000","last_fullsized":"2024-03-19T01:51:17.471346+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087704+0000","last_change":"2024-03-19T01:51:06.951225+0000","last_active":"2024-03-19T01:51:18.087704+0000","last_peered":"2024-03-19T01:51:18.087704+0000","last_clean":"2024-03-19T01:51:18.087704+0000","last_became_active":"2024-03-19T01:51:06.950826+0000","last_became_peered":"2024-03-19T01:51:06.950826+0000","last_unstale":"2024-03-19T01:51:18.087704+0000","last_undegraded":"2024-03-19T01:51:18.087704+0000","last_fullsized":"2024-03-19T01:51:18.087704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:24.949925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426878+0000","last_change":"2024-03-19T01:51:16.581997+0000","last_active":"2024-03-19T01:51:17.426878+0000","last_peered":"2024-03-19T01:51:17.426878+0000","last_clean":"2024-03-19T01:51:17.426878+0000","last_became_active":"2024-03-19T01:51:16.581412+0000","last_became_peered":"2024-03-19T01:51:16.581412+0000","last_unstale":"2024-03-19T01:51:17.426878+0000","last_undegraded":"2024-03-19T01:51:17.426878+0000","last_fullsized":"2024-03-19T01:51:17.426878+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:45.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085931+0000","last_change":"2024-03-19T01:51:09.161686+0000","last_active":"2024-03-19T01:51:18.085931+0000","last_peered":"2024-03-19T01:51:18.085931+0000","last_clean":"2024-03-19T01:51:18.085931+0000","last_became_active":"2024-03-19T01:51:09.160978+0000","last_became_peered":"2024-03-19T01:51:09.160978+0000","last_unstale":"2024-03-19T01:51:18.085931+0000","last_undegraded":"2024-03-19T01:51:18.085931+0000","last_fullsized":"2024-03-19T01:51:18.085931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:27.266807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088069+0000","last_change":"2024-03-19T01:51:07.047363+0000","last_active":"2024-03-19T01:51:18.088069+0000","last_peered":"2024-03-19T01:51:18.088069+0000","last_clean":"2024-03-19T01:51:18.088069+0000","last_became_active":"2024-03-19T01:51:07.046886+0000","last_became_peered":"2024-03-19T01:51:07.046886+0000","last_unstale":"2024-03-19T01:51:18.088069+0000","last_undegraded":"2024-03-19T01:51:18.088069+0000","last_fullsized":"2024-03-19T01:51:18.088069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:19.418603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"25'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426847+0000","last_change":"2024-03-19T01:51:07.045496+0000","last_active":"2024-03-19T01:51:17.426847+0000","last_peered":"2024-03-19T01:51:17.426847+0000","last_clean":"2024-03-19T01:51:17.426847+0000","last_became_active":"2024-03-19T01:51:07.044554+0000","last_became_peered":"2024-03-19T01:51:07.044554+0000","last_unstale":"2024-03-19T01:51:17.426847+0000","last_undegraded":"2024-03-19T01:51:17.426847+0000","last_fullsized":"2024-03-19T01:51:17.426847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:14.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746492+0000","last_change":"2024-03-19T01:51:09.184934+0000","last_active":"2024-03-19T01:51:17.746492+0000","last_peered":"2024-03-19T01:51:17.746492+0000","last_clean":"2024-03-19T01:51:17.746492+0000","last_became_active":"2024-03-19T01:51:09.184666+0000","last_became_peered":"2024-03-19T01:51:09.184666+0000","last_unstale":"2024-03-19T01:51:17.746492+0000","last_undegraded":"2024-03-19T01:51:17.746492+0000","last_fullsized":"2024-03-19T01:51:17.746492+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:42.786323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085344+0000","last_change":"2024-03-19T01:51:16.575084+0000","last_active":"2024-03-19T01:51:18.085344+0000","last_peered":"2024-03-19T01:51:18.085344+0000","last_clean":"2024-03-19T01:51:18.085344+0000","last_became_active":"2024-03-19T01:51:16.574631+0000","last_became_peered":"2024-03-19T01:51:16.574631+0000","last_unstale":"2024-03-19T01:51:18.085344+0000","last_undegraded":"2024-03-19T01:51:18.085344+0000","last_fullsized":"2024-03-19T01:51:18.085344+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:24:35.634113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.503274+0000","last_change":"2024-03-19T01:51:06.982280+0000","last_active":"2024-03-19T01:51:50.503274+0000","last_peered":"2024-03-19T01:51:50.503274+0000","last_clean":"2024-03-19T01:51:50.503274+0000","last_became_active":"2024-03-19T01:51:06.981704+0000","last_became_peered":"2024-03-19T01:51:06.981704+0000","last_unstale":"2024-03-19T01:51:50.503274+0000","last_undegraded":"2024-03-19T01:51:50.503274+0000","last_fullsized":"2024-03-19T01:51:50.503274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:22:12.657985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521885+0000","last_change":"2024-03-19T01:51:09.184624+0000","last_active":"2024-03-19T01:51:17.521885+0000","last_peered":"2024-03-19T01:51:17.521885+0000","last_clean":"2024-03-19T01:51:17.521885+0000","last_became_active":"2024-03-19T01:51:09.183976+0000","last_became_peered":"2024-03-19T01:51:09.183976+0000","last_unstale":"2024-03-19T01:51:17.521885+0000","last_undegraded":"2024-03-19T01:51:17.521885+0000","last_fullsized":"2024-03-19T01:51:17.521885+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:24:29.313729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086581+0000","last_change":"2024-03-19T01:51:16.576775+0000","last_active":"2024-03-19T01:51:18.086581+0000","last_peered":"2024-03-19T01:51:18.086581+0000","last_clean":"2024-03-19T01:51:18.086581+0000","last_became_active":"2024-03-19T01:51:16.576406+0000","last_became_peered":"2024-03-19T01:51:16.576406+0000","last_unstale":"2024-03-19T01:51:18.086581+0000","last_undegraded":"2024-03-19T01:51:18.086581+0000","last_fullsized":"2024-03-19T01:51:18.086581+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:43.725447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425113+0000","last_change":"2024-03-19T01:51:09.169493+0000","last_active":"2024-03-19T01:51:17.425113+0000","last_peered":"2024-03-19T01:51:17.425113+0000","last_clean":"2024-03-19T01:51:17.425113+0000","last_became_active":"2024-03-19T01:51:09.168231+0000","last_became_peered":"2024-03-19T01:51:09.168231+0000","last_unstale":"2024-03-19T01:51:17.425113+0000","last_undegraded":"2024-03-19T01:51:17.425113+0000","last_fullsized":"2024-03-19T01:51:17.425113+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:10.624259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085344+0000","last_change":"2024-03-19T01:51:09.155483+0000","last_active":"2024-03-19T01:51:18.085344+0000","last_peered":"2024-03-19T01:51:18.085344+0000","last_clean":"2024-03-19T01:51:18.085344+0000","last_became_active":"2024-03-19T01:51:09.154545+0000","last_became_peered":"2024-03-19T01:51:09.154545+0000","last_unstale":"2024-03-19T01:51:18.085344+0000","last_undegraded":"2024-03-19T01:51:18.085344+0000","last_fullsized":"2024-03-19T01:51:18.085344+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:25:09.517323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088404+0000","last_change":"2024-03-19T01:51:09.180498+0000","last_active":"2024-03-19T01:51:18.088404+0000","last_peered":"2024-03-19T01:51:18.088404+0000","last_clean":"2024-03-19T01:51:18.088404+0000","last_became_active":"2024-03-19T01:51:09.179840+0000","last_became_peered":"2024-03-19T01:51:09.179840+0000","last_unstale":"2024-03-19T01:51:18.088404+0000","last_undegraded":"2024-03-19T01:51:18.088404+0000","last_fullsized":"2024-03-19T01:51:18.088404+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:01:21.716571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746226+0000","last_change":"2024-03-19T01:51:09.170389+0000","last_active":"2024-03-19T01:51:17.746226+0000","last_peered":"2024-03-19T01:51:17.746226+0000","last_clean":"2024-03-19T01:51:17.746226+0000","last_became_active":"2024-03-19T01:51:09.163974+0000","last_became_peered":"2024-03-19T01:51:09.163974+0000","last_unstale":"2024-03-19T01:51:17.746226+0000","last_undegraded":"2024-03-19T01:51:17.746226+0000","last_fullsized":"2024-03-19T01:51:17.746226+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087052+0000","last_change":"2024-03-19T01:51:09.177060+0000","last_active":"2024-03-19T01:51:18.087052+0000","last_peered":"2024-03-19T01:51:18.087052+0000","last_clean":"2024-03-19T01:51:18.087052+0000","last_became_active":"2024-03-19T01:51:09.176474+0000","last_became_peered":"2024-03-19T01:51:09.176474+0000","last_unstale":"2024-03-19T01:51:18.087052+0000","last_undegraded":"2024-03-19T01:51:18.087052+0000","last_fullsized":"2024-03-19T01:51:18.087052+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:56.318137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472633+0000","last_change":"2024-03-19T01:51:09.168741+0000","last_active":"2024-03-19T01:51:17.472633+0000","last_peered":"2024-03-19T01:51:17.472633+0000","last_clean":"2024-03-19T01:51:17.472633+0000","last_became_active":"2024-03-19T01:51:09.167982+0000","last_became_peered":"2024-03-19T01:51:09.167982+0000","last_unstale":"2024-03-19T01:51:17.472633+0000","last_undegraded":"2024-03-19T01:51:17.472633+0000","last_fullsized":"2024-03-19T01:51:17.472633+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425915+0000","last_change":"2024-03-19T01:51:09.189814+0000","last_active":"2024-03-19T01:51:17.425915+0000","last_peered":"2024-03-19T01:51:17.425915+0000","last_clean":"2024-03-19T01:51:17.425915+0000","last_became_active":"2024-03-19T01:51:09.189128+0000","last_became_peered":"2024-03-19T01:51:09.189128+0000","last_unstale":"2024-03-19T01:51:17.425915+0000","last_undegraded":"2024-03-19T01:51:17.425915+0000","last_fullsized":"2024-03-19T01:51:17.425915+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087688+0000","last_change":"2024-03-19T01:51:09.183989+0000","last_active":"2024-03-19T01:51:18.087688+0000","last_peered":"2024-03-19T01:51:18.087688+0000","last_clean":"2024-03-19T01:51:18.087688+0000","last_became_active":"2024-03-19T01:51:09.183417+0000","last_became_peered":"2024-03-19T01:51:09.183417+0000","last_unstale":"2024-03-19T01:51:18.087688+0000","last_undegraded":"2024-03-19T01:51:18.087688+0000","last_fullsized":"2024-03-19T01:51:18.087688+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:31.405618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087352+0000","last_change":"2024-03-19T01:51:09.172717+0000","last_active":"2024-03-19T01:51:18.087352+0000","last_peered":"2024-03-19T01:51:18.087352+0000","last_clean":"2024-03-19T01:51:18.087352+0000","last_became_active":"2024-03-19T01:51:09.165872+0000","last_became_peered":"2024-03-19T01:51:09.165872+0000","last_unstale":"2024-03-19T01:51:18.087352+0000","last_undegraded":"2024-03-19T01:51:18.087352+0000","last_fullsized":"2024-03-19T01:51:18.087352+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:10.995492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088202+0000","last_change":"2024-03-19T01:51:09.195116+0000","last_active":"2024-03-19T01:51:18.088202+0000","last_peered":"2024-03-19T01:51:18.088202+0000","last_clean":"2024-03-19T01:51:18.088202+0000","last_became_active":"2024-03-19T01:51:09.194558+0000","last_became_peered":"2024-03-19T01:51:09.194558+0000","last_unstale":"2024-03-19T01:51:18.088202+0000","last_undegraded":"2024-03-19T01:51:18.088202+0000","last_fullsized":"2024-03-19T01:51:18.088202+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:41:53.624234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088160+0000","last_change":"2024-03-19T01:51:09.195141+0000","last_active":"2024-03-19T01:51:18.088160+0000","last_peered":"2024-03-19T01:51:18.088160+0000","last_clean":"2024-03-19T01:51:18.088160+0000","last_became_active":"2024-03-19T01:51:09.194536+0000","last_became_peered":"2024-03-19T01:51:09.194536+0000","last_unstale":"2024-03-19T01:51:18.088160+0000","last_undegraded":"2024-03-19T01:51:18.088160+0000","last_fullsized":"2024-03-19T01:51:18.088160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:31.395833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.745890+0000","last_change":"2024-03-19T01:51:09.189197+0000","last_active":"2024-03-19T01:51:17.745890+0000","last_peered":"2024-03-19T01:51:17.745890+0000","last_clean":"2024-03-19T01:51:17.745890+0000","last_became_active":"2024-03-19T01:51:09.188565+0000","last_became_peered":"2024-03-19T01:51:09.188565+0000","last_unstale":"2024-03-19T01:51:17.745890+0000","last_undegraded":"2024-03-19T01:51:17.745890+0000","last_fullsized":"2024-03-19T01:51:17.745890+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426243+0000","last_change":"2024-03-19T01:51:09.168581+0000","last_active":"2024-03-19T01:51:17.426243+0000","last_peered":"2024-03-19T01:51:17.426243+0000","last_clean":"2024-03-19T01:51:17.426243+0000","last_became_active":"2024-03-19T01:51:09.167812+0000","last_became_peered":"2024-03-19T01:51:09.167812+0000","last_unstale":"2024-03-19T01:51:17.426243+0000","last_undegraded":"2024-03-19T01:51:17.426243+0000","last_fullsized":"2024-03-19T01:51:17.426243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089822+0000","last_change":"2024-03-19T01:51:09.188390+0000","last_active":"2024-03-19T01:51:18.089822+0000","last_peered":"2024-03-19T01:51:18.089822+0000","last_clean":"2024-03-19T01:51:18.089822+0000","last_became_active":"2024-03-19T01:51:09.187104+0000","last_became_peered":"2024-03-19T01:51:09.187104+0000","last_unstale":"2024-03-19T01:51:18.089822+0000","last_undegraded":"2024-03-19T01:51:18.089822+0000","last_fullsized":"2024-03-19T01:51:18.089822+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:37.257686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086435+0000","last_change":"2024-03-19T01:51:09.168451+0000","last_active":"2024-03-19T01:51:18.086435+0000","last_peered":"2024-03-19T01:51:18.086435+0000","last_clean":"2024-03-19T01:51:18.086435+0000","last_became_active":"2024-03-19T01:51:09.167744+0000","last_became_peered":"2024-03-19T01:51:09.167744+0000","last_unstale":"2024-03-19T01:51:18.086435+0000","last_undegraded":"2024-03-19T01:51:18.086435+0000","last_fullsized":"2024-03-19T01:51:18.086435+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:21.735485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089493+0000","last_change":"2024-03-19T01:51:09.196067+0000","last_active":"2024-03-19T01:51:18.089493+0000","last_peered":"2024-03-19T01:51:18.089493+0000","last_clean":"2024-03-19T01:51:18.089493+0000","last_became_active":"2024-03-19T01:51:09.195488+0000","last_became_peered":"2024-03-19T01:51:09.195488+0000","last_unstale":"2024-03-19T01:51:18.089493+0000","last_undegraded":"2024-03-19T01:51:18.089493+0000","last_fullsized":"2024-03-19T01:51:18.089493+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:07:44.120161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087081+0000","last_change":"2024-03-19T01:51:09.193831+0000","last_active":"2024-03-19T01:51:18.087081+0000","last_peered":"2024-03-19T01:51:18.087081+0000","last_clean":"2024-03-19T01:51:18.087081+0000","last_became_active":"2024-03-19T01:51:09.192777+0000","last_became_peered":"2024-03-19T01:51:09.192777+0000","last_unstale":"2024-03-19T01:51:18.087081+0000","last_undegraded":"2024-03-19T01:51:18.087081+0000","last_fullsized":"2024-03-19T01:51:18.087081+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:56.318137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.471571+0000","last_change":"2024-03-19T01:51:09.193581+0000","last_active":"2024-03-19T01:51:17.471571+0000","last_peered":"2024-03-19T01:51:17.471571+0000","last_clean":"2024-03-19T01:51:17.471571+0000","last_became_active":"2024-03-19T01:51:09.192677+0000","last_became_peered":"2024-03-19T01:51:09.192677+0000","last_unstale":"2024-03-19T01:51:17.471571+0000","last_undegraded":"2024-03-19T01:51:17.471571+0000","last_fullsized":"2024-03-19T01:51:17.471571+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.745598+0000","last_change":"2024-03-19T01:51:09.165568+0000","last_active":"2024-03-19T01:51:17.745598+0000","last_peered":"2024-03-19T01:51:17.745598+0000","last_clean":"2024-03-19T01:51:17.745598+0000","last_became_active":"2024-03-19T01:51:09.164753+0000","last_became_peered":"2024-03-19T01:51:09.164753+0000","last_unstale":"2024-03-19T01:51:17.745598+0000","last_undegraded":"2024-03-19T01:51:17.745598+0000","last_fullsized":"2024-03-19T01:51:17.745598+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:28.422798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086531+0000","last_change":"2024-03-19T01:51:09.170898+0000","last_active":"2024-03-19T01:51:18.086531+0000","last_peered":"2024-03-19T01:51:18.086531+0000","last_clean":"2024-03-19T01:51:18.086531+0000","last_became_active":"2024-03-19T01:51:09.170153+0000","last_became_peered":"2024-03-19T01:51:09.170153+0000","last_unstale":"2024-03-19T01:51:18.086531+0000","last_undegraded":"2024-03-19T01:51:18.086531+0000","last_fullsized":"2024-03-19T01:51:18.086531+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:41.125800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522152+0000","last_change":"2024-03-19T01:51:09.185953+0000","last_active":"2024-03-19T01:51:17.522152+0000","last_peered":"2024-03-19T01:51:17.522152+0000","last_clean":"2024-03-19T01:51:17.522152+0000","last_became_active":"2024-03-19T01:51:09.177996+0000","last_became_peered":"2024-03-19T01:51:09.177996+0000","last_unstale":"2024-03-19T01:51:17.522152+0000","last_undegraded":"2024-03-19T01:51:17.522152+0000","last_fullsized":"2024-03-19T01:51:17.522152+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:39:06.045413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426250+0000","last_change":"2024-03-19T01:51:09.183841+0000","last_active":"2024-03-19T01:51:17.426250+0000","last_peered":"2024-03-19T01:51:17.426250+0000","last_clean":"2024-03-19T01:51:17.426250+0000","last_became_active":"2024-03-19T01:51:09.183045+0000","last_became_peered":"2024-03-19T01:51:09.183045+0000","last_unstale":"2024-03-19T01:51:17.426250+0000","last_undegraded":"2024-03-19T01:51:17.426250+0000","last_fullsized":"2024-03-19T01:51:17.426250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425940+0000","last_change":"2024-03-19T01:51:09.184022+0000","last_active":"2024-03-19T01:51:17.425940+0000","last_peered":"2024-03-19T01:51:17.425940+0000","last_clean":"2024-03-19T01:51:17.425940+0000","last_became_active":"2024-03-19T01:51:09.183351+0000","last_became_peered":"2024-03-19T01:51:09.183351+0000","last_unstale":"2024-03-19T01:51:17.425940+0000","last_undegraded":"2024-03-19T01:51:17.425940+0000","last_fullsized":"2024-03-19T01:51:17.425940+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090466+0000","last_change":"2024-03-19T01:51:09.204005+0000","last_active":"2024-03-19T01:51:18.090466+0000","last_peered":"2024-03-19T01:51:18.090466+0000","last_clean":"2024-03-19T01:51:18.090466+0000","last_became_active":"2024-03-19T01:51:09.203545+0000","last_became_peered":"2024-03-19T01:51:09.203545+0000","last_unstale":"2024-03-19T01:51:18.090466+0000","last_undegraded":"2024-03-19T01:51:18.090466+0000","last_fullsized":"2024-03-19T01:51:18.090466+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:42:22.784953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085666+0000","last_change":"2024-03-19T01:51:09.180013+0000","last_active":"2024-03-19T01:51:18.085666+0000","last_peered":"2024-03-19T01:51:18.085666+0000","last_clean":"2024-03-19T01:51:18.085666+0000","last_became_active":"2024-03-19T01:51:09.179541+0000","last_became_peered":"2024-03-19T01:51:09.179541+0000","last_unstale":"2024-03-19T01:51:18.085666+0000","last_undegraded":"2024-03-19T01:51:18.085666+0000","last_fullsized":"2024-03-19T01:51:18.085666+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:19:25.933028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089242+0000","last_change":"2024-03-19T01:51:09.188671+0000","last_active":"2024-03-19T01:51:18.089242+0000","last_peered":"2024-03-19T01:51:18.089242+0000","last_clean":"2024-03-19T01:51:18.089242+0000","last_became_active":"2024-03-19T01:51:09.177627+0000","last_became_peered":"2024-03-19T01:51:09.177627+0000","last_unstale":"2024-03-19T01:51:18.089242+0000","last_undegraded":"2024-03-19T01:51:18.089242+0000","last_fullsized":"2024-03-19T01:51:18.089242+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:53:50.539837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087926+0000","last_change":"2024-03-19T01:51:09.180735+0000","last_active":"2024-03-19T01:51:18.087926+0000","last_peered":"2024-03-19T01:51:18.087926+0000","last_clean":"2024-03-19T01:51:18.087926+0000","last_became_active":"2024-03-19T01:51:09.173235+0000","last_became_peered":"2024-03-19T01:51:09.173235+0000","last_unstale":"2024-03-19T01:51:18.087926+0000","last_undegraded":"2024-03-19T01:51:18.087926+0000","last_fullsized":"2024-03-19T01:51:18.087926+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:31.405618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.091424+0000","last_change":"2024-03-19T01:51:09.188346+0000","last_active":"2024-03-19T01:51:18.091424+0000","last_peered":"2024-03-19T01:51:18.091424+0000","last_clean":"2024-03-19T01:51:18.091424+0000","last_became_active":"2024-03-19T01:51:09.187703+0000","last_became_peered":"2024-03-19T01:51:09.187703+0000","last_unstale":"2024-03-19T01:51:18.091424+0000","last_undegraded":"2024-03-19T01:51:18.091424+0000","last_fullsized":"2024-03-19T01:51:18.091424+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:28.051691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087787+0000","last_change":"2024-03-19T01:51:09.182114+0000","last_active":"2024-03-19T01:51:18.087787+0000","last_peered":"2024-03-19T01:51:18.087787+0000","last_clean":"2024-03-19T01:51:18.087787+0000","last_became_active":"2024-03-19T01:51:09.181056+0000","last_became_peered":"2024-03-19T01:51:09.181056+0000","last_unstale":"2024-03-19T01:51:18.087787+0000","last_undegraded":"2024-03-19T01:51:18.087787+0000","last_fullsized":"2024-03-19T01:51:18.087787+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:03:04.997711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746015+0000","last_change":"2024-03-19T01:51:09.161476+0000","last_active":"2024-03-19T01:51:17.746015+0000","last_peered":"2024-03-19T01:51:17.746015+0000","last_clean":"2024-03-19T01:51:17.746015+0000","last_became_active":"2024-03-19T01:51:09.158609+0000","last_became_peered":"2024-03-19T01:51:09.158609+0000","last_unstale":"2024-03-19T01:51:17.746015+0000","last_undegraded":"2024-03-19T01:51:17.746015+0000","last_fullsized":"2024-03-19T01:51:17.746015+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.473414+0000","last_change":"2024-03-19T01:51:09.165938+0000","last_active":"2024-03-19T01:51:17.473414+0000","last_peered":"2024-03-19T01:51:17.473414+0000","last_clean":"2024-03-19T01:51:17.473414+0000","last_became_active":"2024-03-19T01:51:09.165207+0000","last_became_peered":"2024-03-19T01:51:09.165207+0000","last_unstale":"2024-03-19T01:51:17.473414+0000","last_undegraded":"2024-03-19T01:51:17.473414+0000","last_fullsized":"2024-03-19T01:51:17.473414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090233+0000","last_change":"2024-03-19T01:51:09.176023+0000","last_active":"2024-03-19T01:51:18.090233+0000","last_peered":"2024-03-19T01:51:18.090233+0000","last_clean":"2024-03-19T01:51:18.090233+0000","last_became_active":"2024-03-19T01:51:09.175682+0000","last_became_peered":"2024-03-19T01:51:09.175682+0000","last_unstale":"2024-03-19T01:51:18.090233+0000","last_undegraded":"2024-03-19T01:51:18.090233+0000","last_fullsized":"2024-03-19T01:51:18.090233+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:35.814983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6046,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":12092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509461,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":921600,"data_stored":715706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509463,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":323584,"data_stored":127954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509462,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":124148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509463,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31492,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340348,"statfs":{"total":96636764160,"available":96604516352,"internally_reserved":0,"allocated":323584,"data_stored":124854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509462,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339768,"statfs":{"total":96636764160,"available":96603922432,"internally_reserved":0,"allocated":917504,"data_stored":715050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542165,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":124784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542166,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31496,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340344,"statfs":{"total":96636764160,"available":96604512256,"internally_reserved":0,"allocated":327680,"data_stored":128486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542165,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31476,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340364,"statfs":{"total":96636764160,"available":96604532736,"internally_reserved":0,"allocated":307200,"data_stored":124022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T01:51:55.089 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T01:51:55.283 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.281+0000 7fb6a5ea4640 1 Processor -- start 2024-03-19T01:51:55.283 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.281+0000 7fb6a5ea4640 1 -- start start 2024-03-19T01:51:55.284 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.281+0000 7fb6a5ea4640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb6a00ca350 0x7fb6a015b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:55.284 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.281+0000 7fb6a5ea4640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fb6a00c3320 0x7fb6a00728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:55.284 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.281+0000 7fb6a5ea4640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6a0151330 con 0x7fb6a0150f50 2024-03-19T01:51:55.284 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.281+0000 7fb6a5ea4640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fb6a00c3e00 con 0x7fb6a00c3320 2024-03-19T01:51:55.284 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.281+0000 7fb6a5ea4640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fb6a015ac10 con 0x7fb6a00ca350 2024-03-19T01:51:55.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.293+0000 7fb69effd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb6a00ca350 0x7fb6a015b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:55.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.297+0000 7fb69ffff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fb6a00c3320 0x7fb6a00728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:55.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.297+0000 7fb69effd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb6a00ca350 0x7fb6a015b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:44576/0 (socket says 172.21.15.55:44576) 2024-03-19T01:51:55.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.297+0000 7fb69effd640 1 -- 172.21.15.55:0/2790069050 learned_addr learned my addr 172.21.15.55:0/2790069050 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:55.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.301+0000 7fb69ffff640 1 -- 172.21.15.55:0/2790069050 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb6a00ca350 msgr2=0x7fb6a015b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:55.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.301+0000 7fb69ffff640 1 --2- 172.21.15.55:0/2790069050 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb6a00ca350 0x7fb6a015b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:55.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.301+0000 7fb69ffff640 1 -- 172.21.15.55:0/2790069050 >> v1:172.21.15.84:6789/0 conn(0x7fb6a0150f50 legacy=0x7fb6a0157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:55.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.301+0000 7fb69ffff640 1 -- 172.21.15.55:0/2790069050 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a00c3840 con 0x7fb6a00c3320 2024-03-19T01:51:55.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.313+0000 7fb69ffff640 1 --2- 172.21.15.55:0/2790069050 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6a00c3320 0x7fb6a00728b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb6980097c0 tx=0x7fb69801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3cf82e9e9fcff65 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:55.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.333+0000 7fb69e7fc640 1 -- 172.21.15.55:0/2790069050 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6980aa600 con 0x7fb6a00c3320 2024-03-19T01:51:55.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.333+0000 7fb69e7fc640 1 -- 172.21.15.55:0/2790069050 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6980af070 con 0x7fb6a00c3320 2024-03-19T01:51:55.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.333+0000 7fb69e7fc640 1 -- 172.21.15.55:0/2790069050 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6980aa9c0 con 0x7fb6a00c3320 2024-03-19T01:51:55.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.333+0000 7fb6a5ea4640 1 -- 172.21.15.55:0/2790069050 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6a00c3320 msgr2=0x7fb6a00728b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:55.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.333+0000 7fb6a5ea4640 1 --2- 172.21.15.55:0/2790069050 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6a00c3320 0x7fb6a00728b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb6980097c0 tx=0x7fb69801c4a0 comp rx=0 tx=0).stop 2024-03-19T01:51:55.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.333+0000 7fb6a5ea4640 1 -- 172.21.15.55:0/2790069050 shutdown_connections 2024-03-19T01:51:55.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.333+0000 7fb6a5ea4640 1 --2- 172.21.15.55:0/2790069050 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6a00c3320 0x7fb6a00728b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:55.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.333+0000 7fb6a5ea4640 1 --2- 172.21.15.55:0/2790069050 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb6a00ca350 0x7fb6a015b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:55.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.333+0000 7fb6a5ea4640 1 -- 172.21.15.55:0/2790069050 >> 172.21.15.55:0/2790069050 conn(0x7fb6a006e670 msgr2=0x7fb6a006ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:55.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.333+0000 7fb6a5ea4640 1 -- 172.21.15.55:0/2790069050 shutdown_connections 2024-03-19T01:51:55.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.333+0000 7fb6a5ea4640 1 -- 172.21.15.55:0/2790069050 wait complete. 2024-03-19T01:51:55.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.333+0000 7fb6a5ea4640 1 Processor -- start 2024-03-19T01:51:55.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.337+0000 7fb6a5ea4640 1 -- start start 2024-03-19T01:51:55.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.337+0000 7fb6a5ea4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6a00c3320 0x7fb6a00cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:55.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.337+0000 7fb6a5ea4640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb6a00ca350 0x7fb6a00744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:55.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.337+0000 7fb6a5ea4640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6a0150f50 0x7fb6a01f2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:55.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.337+0000 7fb6a5ea4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a006b420 con 0x7fb6a00c3320 2024-03-19T01:51:55.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.337+0000 7fb6a5ea4640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a006b2a0 con 0x7fb6a0150f50 2024-03-19T01:51:55.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.337+0000 7fb6a5ea4640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fb6a006b5a0 con 0x7fb6a00ca350 2024-03-19T01:51:55.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.337+0000 7fb69effd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb6a00ca350 0x7fb6a00744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:55.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.341+0000 7fb69effd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb6a00ca350 0x7fb6a00744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:44588/0 (socket says 172.21.15.55:44588) 2024-03-19T01:51:55.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.341+0000 7fb69effd640 1 -- 172.21.15.55:0/2919275802 learned_addr learned my addr 172.21.15.55:0/2919275802 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:55.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.341+0000 7fb69ffff640 1 --2- 172.21.15.55:0/2919275802 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6a0150f50 0x7fb6a01f2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:55.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.349+0000 7fb69effd640 1 -- 172.21.15.55:0/2919275802 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6a0150f50 msgr2=0x7fb6a01f2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:55.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.349+0000 7fb69effd640 1 --2- 172.21.15.55:0/2919275802 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6a0150f50 0x7fb6a01f2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:55.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.349+0000 7fb69effd640 1 -- 172.21.15.55:0/2919275802 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6a00c3320 msgr2=0x7fb6a00cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:55.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.349+0000 7fb69effd640 1 --2- 172.21.15.55:0/2919275802 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6a00c3320 0x7fb6a00cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:55.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.349+0000 7fb69effd640 1 -- 172.21.15.55:0/2919275802 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a01432f0 con 0x7fb6a00ca350 2024-03-19T01:51:55.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.353+0000 7fb69effd640 1 --2- 172.21.15.55:0/2919275802 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb6a00ca350 0x7fb6a00744f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb69400e850 tx=0x7fb69400ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:55.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.365+0000 7fb687fff640 1 -- 172.21.15.55:0/2919275802 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb694017070 con 0x7fb6a00ca350 2024-03-19T01:51:55.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.365+0000 7fb687fff640 1 -- 172.21.15.55:0/2919275802 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb69400c940 con 0x7fb6a00ca350 2024-03-19T01:51:55.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.365+0000 7fb687fff640 1 -- 172.21.15.55:0/2919275802 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69400ccf0 con 0x7fb6a00ca350 2024-03-19T01:51:55.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.365+0000 7fb6a5ea4640 1 -- 172.21.15.55:0/2919275802 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6a01426e0 con 0x7fb6a00ca350 2024-03-19T01:51:55.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.365+0000 7fb6a5ea4640 1 -- 172.21.15.55:0/2919275802 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6a0142ca0 con 0x7fb6a00ca350 2024-03-19T01:51:55.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.365+0000 7fb6a5ea4640 1 -- 172.21.15.55:0/2919275802 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6a0068a40 con 0x7fb6a00ca350 2024-03-19T01:51:55.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.685+0000 7fb687fff640 1 -- 172.21.15.55:0/2919275802 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb69401a070 con 0x7fb6a00ca350 2024-03-19T01:51:55.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.685+0000 7fb687fff640 1 --2- 172.21.15.55:0/2919275802 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb68c067690 0x7fb68c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:55.686 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.685+0000 7fb69f7fe640 1 --2- 172.21.15.55:0/2919275802 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb68c067690 0x7fb68c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:55.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.685+0000 7fb69f7fe640 1 --2- 172.21.15.55:0/2919275802 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb68c067690 0x7fb68c069b50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb690007fd0 tx=0x7fb690007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:55.692 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.689+0000 7fb687fff640 1 -- 172.21.15.55:0/2919275802 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fb6940545c0 con 0x7fb6a00ca350 2024-03-19T01:51:55.716 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.713+0000 7fb687fff640 1 -- 172.21.15.55:0/2919275802 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fb694056c20 con 0x7fb6a00ca350 2024-03-19T01:51:55.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.901+0000 7fb6a5ea4640 1 -- 172.21.15.55:0/2919275802 --> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb6a00c7600 con 0x7fb68c067690 2024-03-19T01:51:55.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.921+0000 7fb687fff640 1 -- 172.21.15.55:0/2919275802 <== mgr.4105 v2:172.21.15.84:6800/998533783 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332684 (secure 0 0 0) 0x7fb6a00c7600 con 0x7fb68c067690 2024-03-19T01:51:55.925 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:51:55.929 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-19T01:51:55.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.929+0000 7fb6a5ea4640 1 -- 172.21.15.55:0/2919275802 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb68c067690 msgr2=0x7fb68c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:55.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.929+0000 7fb6a5ea4640 1 --2- 172.21.15.55:0/2919275802 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb68c067690 0x7fb68c069b50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb690007fd0 tx=0x7fb690007d40 comp rx=0 tx=0).stop 2024-03-19T01:51:55.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.929+0000 7fb6a5ea4640 1 -- 172.21.15.55:0/2919275802 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb6a00ca350 msgr2=0x7fb6a00744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:55.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.929+0000 7fb6a5ea4640 1 --2- 172.21.15.55:0/2919275802 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb6a00ca350 0x7fb6a00744f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb69400e850 tx=0x7fb69400ed10 comp rx=0 tx=0).stop 2024-03-19T01:51:55.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.929+0000 7fb6a5ea4640 1 -- 172.21.15.55:0/2919275802 shutdown_connections 2024-03-19T01:51:55.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.929+0000 7fb6a5ea4640 1 --2- 172.21.15.55:0/2919275802 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb68c067690 0x7fb68c069b50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:55.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.929+0000 7fb6a5ea4640 1 --2- 172.21.15.55:0/2919275802 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6a0150f50 0x7fb6a01f2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:55.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.929+0000 7fb6a5ea4640 1 --2- 172.21.15.55:0/2919275802 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb6a00ca350 0x7fb6a00744f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:55.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.929+0000 7fb6a5ea4640 1 --2- 172.21.15.55:0/2919275802 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6a00c3320 0x7fb6a00cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:55.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.929+0000 7fb6a5ea4640 1 -- 172.21.15.55:0/2919275802 >> 172.21.15.55:0/2919275802 conn(0x7fb6a006e670 msgr2=0x7fb6a0143bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:55.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.929+0000 7fb6a5ea4640 1 -- 172.21.15.55:0/2919275802 shutdown_connections 2024-03-19T01:51:55.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:55.929+0000 7fb6a5ea4640 1 -- 172.21.15.55:0/2919275802 wait complete. 2024-03-19T01:51:55.945 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-03-19T01:51:54.326680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596414,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":247,"num_write_kb":2447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236440,"kb_used_data":3672,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738280,"statfs":{"total":773094113280,"available":772851998720,"internally_reserved":0,"allocated":3760128,"data_stored":2185004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1406,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005119"},"pg_stats":[{"pgid":"2.1f","version":"24'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746887+0000","last_change":"2024-03-19T01:51:06.982533+0000","last_active":"2024-03-19T01:51:17.746887+0000","last_peered":"2024-03-19T01:51:17.746887+0000","last_clean":"2024-03-19T01:51:17.746887+0000","last_became_active":"2024-03-19T01:51:06.981802+0000","last_became_peered":"2024-03-19T01:51:06.981802+0000","last_unstale":"2024-03-19T01:51:17.746887+0000","last_undegraded":"2024-03-19T01:51:17.746887+0000","last_fullsized":"2024-03-19T01:51:17.746887+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:10:01.431003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086436+0000","last_change":"2024-03-19T01:51:09.172057+0000","last_active":"2024-03-19T01:51:18.086436+0000","last_peered":"2024-03-19T01:51:18.086436+0000","last_clean":"2024-03-19T01:51:18.086436+0000","last_became_active":"2024-03-19T01:51:09.171753+0000","last_became_peered":"2024-03-19T01:51:09.171753+0000","last_unstale":"2024-03-19T01:51:18.086436+0000","last_undegraded":"2024-03-19T01:51:18.086436+0000","last_fullsized":"2024-03-19T01:51:18.086436+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:50.575804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087409+0000","last_change":"2024-03-19T01:51:16.537753+0000","last_active":"2024-03-19T01:51:18.087409+0000","last_peered":"2024-03-19T01:51:18.087409+0000","last_clean":"2024-03-19T01:51:18.087409+0000","last_became_active":"2024-03-19T01:51:16.534200+0000","last_became_peered":"2024-03-19T01:51:16.534200+0000","last_unstale":"2024-03-19T01:51:18.087409+0000","last_undegraded":"2024-03-19T01:51:18.087409+0000","last_fullsized":"2024-03-19T01:51:18.087409+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:18:34.208854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"24'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086639+0000","last_change":"2024-03-19T01:51:06.962968+0000","last_active":"2024-03-19T01:51:18.086639+0000","last_peered":"2024-03-19T01:51:18.086639+0000","last_clean":"2024-03-19T01:51:18.086639+0000","last_became_active":"2024-03-19T01:51:06.961856+0000","last_became_peered":"2024-03-19T01:51:06.961856+0000","last_unstale":"2024-03-19T01:51:18.086639+0000","last_undegraded":"2024-03-19T01:51:18.086639+0000","last_fullsized":"2024-03-19T01:51:18.086639+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:49.676651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746193+0000","last_change":"2024-03-19T01:51:09.160338+0000","last_active":"2024-03-19T01:51:17.746193+0000","last_peered":"2024-03-19T01:51:17.746193+0000","last_clean":"2024-03-19T01:51:17.746193+0000","last_became_active":"2024-03-19T01:51:09.158682+0000","last_became_peered":"2024-03-19T01:51:09.158682+0000","last_unstale":"2024-03-19T01:51:17.746193+0000","last_undegraded":"2024-03-19T01:51:17.746193+0000","last_fullsized":"2024-03-19T01:51:17.746193+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521163+0000","last_change":"2024-03-19T01:51:16.592390+0000","last_active":"2024-03-19T01:51:17.521163+0000","last_peered":"2024-03-19T01:51:17.521163+0000","last_clean":"2024-03-19T01:51:17.521163+0000","last_became_active":"2024-03-19T01:51:16.591093+0000","last_became_peered":"2024-03-19T01:51:16.591093+0000","last_unstale":"2024-03-19T01:51:17.521163+0000","last_undegraded":"2024-03-19T01:51:17.521163+0000","last_fullsized":"2024-03-19T01:51:17.521163+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:45.300462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"25'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089684+0000","last_change":"2024-03-19T01:51:07.044049+0000","last_active":"2024-03-19T01:51:18.089684+0000","last_peered":"2024-03-19T01:51:18.089684+0000","last_clean":"2024-03-19T01:51:18.089684+0000","last_became_active":"2024-03-19T01:51:07.043209+0000","last_became_peered":"2024-03-19T01:51:07.043209+0000","last_unstale":"2024-03-19T01:51:18.089684+0000","last_undegraded":"2024-03-19T01:51:18.089684+0000","last_fullsized":"2024-03-19T01:51:18.089684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:42.992419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088027+0000","last_change":"2024-03-19T01:51:09.179615+0000","last_active":"2024-03-19T01:51:18.088027+0000","last_peered":"2024-03-19T01:51:18.088027+0000","last_clean":"2024-03-19T01:51:18.088027+0000","last_became_active":"2024-03-19T01:51:09.173962+0000","last_became_peered":"2024-03-19T01:51:09.173962+0000","last_unstale":"2024-03-19T01:51:18.088027+0000","last_undegraded":"2024-03-19T01:51:18.088027+0000","last_fullsized":"2024-03-19T01:51:18.088027+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:06:12.676868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522548+0000","last_change":"2024-03-19T01:51:16.593210+0000","last_active":"2024-03-19T01:51:17.522548+0000","last_peered":"2024-03-19T01:51:17.522548+0000","last_clean":"2024-03-19T01:51:17.522548+0000","last_became_active":"2024-03-19T01:51:16.591235+0000","last_became_peered":"2024-03-19T01:51:16.591235+0000","last_unstale":"2024-03-19T01:51:17.522548+0000","last_undegraded":"2024-03-19T01:51:17.522548+0000","last_fullsized":"2024-03-19T01:51:17.522548+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:13.248793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"25'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088902+0000","last_change":"2024-03-19T01:51:06.983291+0000","last_active":"2024-03-19T01:51:18.088902+0000","last_peered":"2024-03-19T01:51:18.088902+0000","last_clean":"2024-03-19T01:51:18.088902+0000","last_became_active":"2024-03-19T01:51:06.982309+0000","last_became_peered":"2024-03-19T01:51:06.982309+0000","last_unstale":"2024-03-19T01:51:18.088902+0000","last_undegraded":"2024-03-19T01:51:18.088902+0000","last_fullsized":"2024-03-19T01:51:18.088902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:27.305286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089984+0000","last_change":"2024-03-19T01:51:09.160738+0000","last_active":"2024-03-19T01:51:18.089984+0000","last_peered":"2024-03-19T01:51:18.089984+0000","last_clean":"2024-03-19T01:51:18.089984+0000","last_became_active":"2024-03-19T01:51:09.159951+0000","last_became_peered":"2024-03-19T01:51:09.159951+0000","last_unstale":"2024-03-19T01:51:18.089984+0000","last_undegraded":"2024-03-19T01:51:18.089984+0000","last_fullsized":"2024-03-19T01:51:18.089984+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:30.620041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521585+0000","last_change":"2024-03-19T01:51:16.577378+0000","last_active":"2024-03-19T01:51:17.521585+0000","last_peered":"2024-03-19T01:51:17.521585+0000","last_clean":"2024-03-19T01:51:17.521585+0000","last_became_active":"2024-03-19T01:51:16.576256+0000","last_became_peered":"2024-03-19T01:51:16.576256+0000","last_unstale":"2024-03-19T01:51:17.521585+0000","last_undegraded":"2024-03-19T01:51:17.521585+0000","last_fullsized":"2024-03-19T01:51:17.521585+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:16.096547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426589+0000","last_change":"2024-03-19T01:51:09.185969+0000","last_active":"2024-03-19T01:51:17.426589+0000","last_peered":"2024-03-19T01:51:17.426589+0000","last_clean":"2024-03-19T01:51:17.426589+0000","last_became_active":"2024-03-19T01:51:09.185378+0000","last_became_peered":"2024-03-19T01:51:09.185378+0000","last_unstale":"2024-03-19T01:51:17.426589+0000","last_undegraded":"2024-03-19T01:51:17.426589+0000","last_fullsized":"2024-03-19T01:51:17.426589+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.338974+0000","last_change":"2024-03-19T01:51:06.988067+0000","last_active":"2024-03-19T01:51:50.338974+0000","last_peered":"2024-03-19T01:51:50.338974+0000","last_clean":"2024-03-19T01:51:50.338974+0000","last_became_active":"2024-03-19T01:51:06.987450+0000","last_became_peered":"2024-03-19T01:51:06.987450+0000","last_unstale":"2024-03-19T01:51:50.338974+0000","last_undegraded":"2024-03-19T01:51:50.338974+0000","last_fullsized":"2024-03-19T01:51:50.338974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:56.962922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522867+0000","last_change":"2024-03-19T01:51:16.583190+0000","last_active":"2024-03-19T01:51:17.522867+0000","last_peered":"2024-03-19T01:51:17.522867+0000","last_clean":"2024-03-19T01:51:17.522867+0000","last_became_active":"2024-03-19T01:51:16.582532+0000","last_became_peered":"2024-03-19T01:51:16.582532+0000","last_unstale":"2024-03-19T01:51:17.522867+0000","last_undegraded":"2024-03-19T01:51:17.522867+0000","last_fullsized":"2024-03-19T01:51:17.522867+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:16:49.817940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085997+0000","last_change":"2024-03-19T01:51:09.174687+0000","last_active":"2024-03-19T01:51:18.085997+0000","last_peered":"2024-03-19T01:51:18.085997+0000","last_clean":"2024-03-19T01:51:18.085997+0000","last_became_active":"2024-03-19T01:51:09.174409+0000","last_became_peered":"2024-03-19T01:51:09.174409+0000","last_unstale":"2024-03-19T01:51:18.085997+0000","last_undegraded":"2024-03-19T01:51:18.085997+0000","last_fullsized":"2024-03-19T01:51:18.085997+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:33:07.160350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472300+0000","last_change":"2024-03-19T01:51:06.955434+0000","last_active":"2024-03-19T01:51:17.472300+0000","last_peered":"2024-03-19T01:51:17.472300+0000","last_clean":"2024-03-19T01:51:17.472300+0000","last_became_active":"2024-03-19T01:51:06.954635+0000","last_became_peered":"2024-03-19T01:51:06.954635+0000","last_unstale":"2024-03-19T01:51:17.472300+0000","last_undegraded":"2024-03-19T01:51:17.472300+0000","last_fullsized":"2024-03-19T01:51:17.472300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:41:39.176905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521820+0000","last_change":"2024-03-19T01:51:16.588604+0000","last_active":"2024-03-19T01:51:17.521820+0000","last_peered":"2024-03-19T01:51:17.521820+0000","last_clean":"2024-03-19T01:51:17.521820+0000","last_became_active":"2024-03-19T01:51:16.587483+0000","last_became_peered":"2024-03-19T01:51:16.587483+0000","last_unstale":"2024-03-19T01:51:17.521820+0000","last_undegraded":"2024-03-19T01:51:17.521820+0000","last_fullsized":"2024-03-19T01:51:17.521820+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:51.658547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086030+0000","last_change":"2024-03-19T01:51:09.173987+0000","last_active":"2024-03-19T01:51:18.086030+0000","last_peered":"2024-03-19T01:51:18.086030+0000","last_clean":"2024-03-19T01:51:18.086030+0000","last_became_active":"2024-03-19T01:51:09.173648+0000","last_became_peered":"2024-03-19T01:51:09.173648+0000","last_unstale":"2024-03-19T01:51:18.086030+0000","last_undegraded":"2024-03-19T01:51:18.086030+0000","last_fullsized":"2024-03-19T01:51:18.086030+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:13:17.277014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.374560+0000","last_change":"2024-03-19T01:51:06.981820+0000","last_active":"2024-03-19T01:51:50.374560+0000","last_peered":"2024-03-19T01:51:50.374560+0000","last_clean":"2024-03-19T01:51:50.374560+0000","last_became_active":"2024-03-19T01:51:06.981087+0000","last_became_peered":"2024-03-19T01:51:06.981087+0000","last_unstale":"2024-03-19T01:51:50.374560+0000","last_undegraded":"2024-03-19T01:51:50.374560+0000","last_fullsized":"2024-03-19T01:51:50.374560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:25:59.116138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.473071+0000","last_change":"2024-03-19T01:51:16.579017+0000","last_active":"2024-03-19T01:51:17.473071+0000","last_peered":"2024-03-19T01:51:17.473071+0000","last_clean":"2024-03-19T01:51:17.473071+0000","last_became_active":"2024-03-19T01:51:16.573845+0000","last_became_peered":"2024-03-19T01:51:16.573845+0000","last_unstale":"2024-03-19T01:51:17.473071+0000","last_undegraded":"2024-03-19T01:51:17.473071+0000","last_fullsized":"2024-03-19T01:51:17.473071+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:42:30.142766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522501+0000","last_change":"2024-03-19T01:51:09.192886+0000","last_active":"2024-03-19T01:51:17.522501+0000","last_peered":"2024-03-19T01:51:17.522501+0000","last_clean":"2024-03-19T01:51:17.522501+0000","last_became_active":"2024-03-19T01:51:09.185970+0000","last_became_peered":"2024-03-19T01:51:09.185970+0000","last_unstale":"2024-03-19T01:51:17.522501+0000","last_undegraded":"2024-03-19T01:51:17.522501+0000","last_fullsized":"2024-03-19T01:51:17.522501+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:05.196985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.339562+0000","last_change":"2024-03-19T01:51:06.984875+0000","last_active":"2024-03-19T01:51:50.339562+0000","last_peered":"2024-03-19T01:51:50.339562+0000","last_clean":"2024-03-19T01:51:50.339562+0000","last_became_active":"2024-03-19T01:51:06.983858+0000","last_became_peered":"2024-03-19T01:51:06.983858+0000","last_unstale":"2024-03-19T01:51:50.339562+0000","last_undegraded":"2024-03-19T01:51:50.339562+0000","last_fullsized":"2024-03-19T01:51:50.339562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:14.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522293+0000","last_change":"2024-03-19T01:51:16.591614+0000","last_active":"2024-03-19T01:51:17.522293+0000","last_peered":"2024-03-19T01:51:17.522293+0000","last_clean":"2024-03-19T01:51:17.522293+0000","last_became_active":"2024-03-19T01:51:16.590968+0000","last_became_peered":"2024-03-19T01:51:16.590968+0000","last_unstale":"2024-03-19T01:51:17.522293+0000","last_undegraded":"2024-03-19T01:51:17.522293+0000","last_fullsized":"2024-03-19T01:51:17.522293+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:50.144136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746685+0000","last_change":"2024-03-19T01:51:09.168941+0000","last_active":"2024-03-19T01:51:17.746685+0000","last_peered":"2024-03-19T01:51:17.746685+0000","last_clean":"2024-03-19T01:51:17.746685+0000","last_became_active":"2024-03-19T01:51:09.168243+0000","last_became_peered":"2024-03-19T01:51:09.168243+0000","last_unstale":"2024-03-19T01:51:17.746685+0000","last_undegraded":"2024-03-19T01:51:17.746685+0000","last_fullsized":"2024-03-19T01:51:17.746685+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.369887+0000","last_change":"2024-03-19T01:51:07.042870+0000","last_active":"2024-03-19T01:51:50.369887+0000","last_peered":"2024-03-19T01:51:50.369887+0000","last_clean":"2024-03-19T01:51:50.369887+0000","last_became_active":"2024-03-19T01:51:07.042327+0000","last_became_peered":"2024-03-19T01:51:07.042327+0000","last_unstale":"2024-03-19T01:51:50.369887+0000","last_undegraded":"2024-03-19T01:51:50.369887+0000","last_fullsized":"2024-03-19T01:51:50.369887+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:13.137673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521677+0000","last_change":"2024-03-19T01:51:16.587730+0000","last_active":"2024-03-19T01:51:17.521677+0000","last_peered":"2024-03-19T01:51:17.521677+0000","last_clean":"2024-03-19T01:51:17.521677+0000","last_became_active":"2024-03-19T01:51:16.586614+0000","last_became_peered":"2024-03-19T01:51:16.586614+0000","last_unstale":"2024-03-19T01:51:17.521677+0000","last_undegraded":"2024-03-19T01:51:17.521677+0000","last_fullsized":"2024-03-19T01:51:17.521677+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:44.830637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087909+0000","last_change":"2024-03-19T01:51:09.186397+0000","last_active":"2024-03-19T01:51:18.087909+0000","last_peered":"2024-03-19T01:51:18.087909+0000","last_clean":"2024-03-19T01:51:18.087909+0000","last_became_active":"2024-03-19T01:51:09.186033+0000","last_became_peered":"2024-03-19T01:51:09.186033+0000","last_unstale":"2024-03-19T01:51:18.087909+0000","last_undegraded":"2024-03-19T01:51:18.087909+0000","last_fullsized":"2024-03-19T01:51:18.087909+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:03:04.997711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"26'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.473540+0000","last_change":"2024-03-19T01:51:07.043697+0000","last_active":"2024-03-19T01:51:17.473540+0000","last_peered":"2024-03-19T01:51:17.473540+0000","last_clean":"2024-03-19T01:51:17.473540+0000","last_became_active":"2024-03-19T01:51:07.043185+0000","last_became_peered":"2024-03-19T01:51:07.043185+0000","last_unstale":"2024-03-19T01:51:17.473540+0000","last_undegraded":"2024-03-19T01:51:17.473540+0000","last_fullsized":"2024-03-19T01:51:17.473540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:19.016579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085498+0000","last_change":"2024-03-19T01:51:16.535771+0000","last_active":"2024-03-19T01:51:18.085498+0000","last_peered":"2024-03-19T01:51:18.085498+0000","last_clean":"2024-03-19T01:51:18.085498+0000","last_became_active":"2024-03-19T01:51:16.533661+0000","last_became_peered":"2024-03-19T01:51:16.533661+0000","last_unstale":"2024-03-19T01:51:18.085498+0000","last_undegraded":"2024-03-19T01:51:18.085498+0000","last_fullsized":"2024-03-19T01:51:18.085498+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:55.283782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472472+0000","last_change":"2024-03-19T01:51:09.157526+0000","last_active":"2024-03-19T01:51:17.472472+0000","last_peered":"2024-03-19T01:51:17.472472+0000","last_clean":"2024-03-19T01:51:17.472472+0000","last_became_active":"2024-03-19T01:51:09.156336+0000","last_became_peered":"2024-03-19T01:51:09.156336+0000","last_unstale":"2024-03-19T01:51:17.472472+0000","last_undegraded":"2024-03-19T01:51:17.472472+0000","last_fullsized":"2024-03-19T01:51:17.472472+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090045+0000","last_change":"2024-03-19T01:51:06.982994+0000","last_active":"2024-03-19T01:51:18.090045+0000","last_peered":"2024-03-19T01:51:18.090045+0000","last_clean":"2024-03-19T01:51:18.090045+0000","last_became_active":"2024-03-19T01:51:06.982133+0000","last_became_peered":"2024-03-19T01:51:06.982133+0000","last_unstale":"2024-03-19T01:51:18.090045+0000","last_undegraded":"2024-03-19T01:51:18.090045+0000","last_fullsized":"2024-03-19T01:51:18.090045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:03.574098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746076+0000","last_change":"2024-03-19T01:51:16.578493+0000","last_active":"2024-03-19T01:51:17.746076+0000","last_peered":"2024-03-19T01:51:17.746076+0000","last_clean":"2024-03-19T01:51:17.746076+0000","last_became_active":"2024-03-19T01:51:16.578040+0000","last_became_peered":"2024-03-19T01:51:16.578040+0000","last_unstale":"2024-03-19T01:51:17.746076+0000","last_undegraded":"2024-03-19T01:51:17.746076+0000","last_fullsized":"2024-03-19T01:51:17.746076+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:45.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090208+0000","last_change":"2024-03-19T01:51:09.180758+0000","last_active":"2024-03-19T01:51:18.090208+0000","last_peered":"2024-03-19T01:51:18.090208+0000","last_clean":"2024-03-19T01:51:18.090208+0000","last_became_active":"2024-03-19T01:51:09.180326+0000","last_became_peered":"2024-03-19T01:51:09.180326+0000","last_unstale":"2024-03-19T01:51:18.090208+0000","last_undegraded":"2024-03-19T01:51:18.090208+0000","last_fullsized":"2024-03-19T01:51:18.090208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:56:07.046285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472576+0000","last_change":"2024-03-19T01:51:06.962643+0000","last_active":"2024-03-19T01:51:17.472576+0000","last_peered":"2024-03-19T01:51:17.472576+0000","last_clean":"2024-03-19T01:51:17.472576+0000","last_became_active":"2024-03-19T01:51:06.961862+0000","last_became_peered":"2024-03-19T01:51:06.961862+0000","last_unstale":"2024-03-19T01:51:17.472576+0000","last_undegraded":"2024-03-19T01:51:17.472576+0000","last_fullsized":"2024-03-19T01:51:17.472576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.471619+0000","last_change":"2024-03-19T01:51:16.577730+0000","last_active":"2024-03-19T01:51:17.471619+0000","last_peered":"2024-03-19T01:51:17.471619+0000","last_clean":"2024-03-19T01:51:17.471619+0000","last_became_active":"2024-03-19T01:51:16.576955+0000","last_became_peered":"2024-03-19T01:51:16.576955+0000","last_unstale":"2024-03-19T01:51:17.471619+0000","last_undegraded":"2024-03-19T01:51:17.471619+0000","last_fullsized":"2024-03-19T01:51:17.471619+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:35.409504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522368+0000","last_change":"2024-03-19T01:51:09.183678+0000","last_active":"2024-03-19T01:51:17.522368+0000","last_peered":"2024-03-19T01:51:17.522368+0000","last_clean":"2024-03-19T01:51:17.522368+0000","last_became_active":"2024-03-19T01:51:09.176780+0000","last_became_peered":"2024-03-19T01:51:09.176780+0000","last_unstale":"2024-03-19T01:51:17.522368+0000","last_undegraded":"2024-03-19T01:51:17.522368+0000","last_fullsized":"2024-03-19T01:51:17.522368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:11:02.502333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088948+0000","last_change":"2024-03-19T01:51:06.956307+0000","last_active":"2024-03-19T01:51:18.088948+0000","last_peered":"2024-03-19T01:51:18.088948+0000","last_clean":"2024-03-19T01:51:18.088948+0000","last_became_active":"2024-03-19T01:51:06.955529+0000","last_became_peered":"2024-03-19T01:51:06.955529+0000","last_unstale":"2024-03-19T01:51:18.088948+0000","last_undegraded":"2024-03-19T01:51:18.088948+0000","last_fullsized":"2024-03-19T01:51:18.088948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:10.486012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472717+0000","last_change":"2024-03-19T01:51:16.550718+0000","last_active":"2024-03-19T01:51:17.472717+0000","last_peered":"2024-03-19T01:51:17.472717+0000","last_clean":"2024-03-19T01:51:17.472717+0000","last_became_active":"2024-03-19T01:51:16.550115+0000","last_became_peered":"2024-03-19T01:51:16.550115+0000","last_unstale":"2024-03-19T01:51:17.472717+0000","last_undegraded":"2024-03-19T01:51:17.472717+0000","last_fullsized":"2024-03-19T01:51:17.472717+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:43.172796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.471788+0000","last_change":"2024-03-19T01:51:09.155718+0000","last_active":"2024-03-19T01:51:17.471788+0000","last_peered":"2024-03-19T01:51:17.471788+0000","last_clean":"2024-03-19T01:51:17.471788+0000","last_became_active":"2024-03-19T01:51:09.155014+0000","last_became_peered":"2024-03-19T01:51:09.155014+0000","last_unstale":"2024-03-19T01:51:17.471788+0000","last_undegraded":"2024-03-19T01:51:17.471788+0000","last_fullsized":"2024-03-19T01:51:17.471788+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085568+0000","last_change":"2024-03-19T01:51:06.955372+0000","last_active":"2024-03-19T01:51:18.085568+0000","last_peered":"2024-03-19T01:51:18.085568+0000","last_clean":"2024-03-19T01:51:18.085568+0000","last_became_active":"2024-03-19T01:51:06.954637+0000","last_became_peered":"2024-03-19T01:51:06.954637+0000","last_unstale":"2024-03-19T01:51:18.085568+0000","last_undegraded":"2024-03-19T01:51:18.085568+0000","last_fullsized":"2024-03-19T01:51:18.085568+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:38:20.129564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.473243+0000","last_change":"2024-03-19T01:51:16.575157+0000","last_active":"2024-03-19T01:51:17.473243+0000","last_peered":"2024-03-19T01:51:17.473243+0000","last_clean":"2024-03-19T01:51:17.473243+0000","last_became_active":"2024-03-19T01:51:16.574575+0000","last_became_peered":"2024-03-19T01:51:16.574575+0000","last_unstale":"2024-03-19T01:51:17.473243+0000","last_undegraded":"2024-03-19T01:51:17.473243+0000","last_fullsized":"2024-03-19T01:51:17.473243+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:15:17.718630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.745221+0000","last_change":"2024-03-19T01:51:09.192610+0000","last_active":"2024-03-19T01:51:17.745221+0000","last_peered":"2024-03-19T01:51:17.745221+0000","last_clean":"2024-03-19T01:51:17.745221+0000","last_became_active":"2024-03-19T01:51:09.192280+0000","last_became_peered":"2024-03-19T01:51:09.192280+0000","last_unstale":"2024-03-19T01:51:17.745221+0000","last_undegraded":"2024-03-19T01:51:17.745221+0000","last_fullsized":"2024-03-19T01:51:17.745221+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:15.454555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088226+0000","last_change":"2024-03-19T01:51:06.986895+0000","last_active":"2024-03-19T01:51:18.088226+0000","last_peered":"2024-03-19T01:51:18.088226+0000","last_clean":"2024-03-19T01:51:18.088226+0000","last_became_active":"2024-03-19T01:51:06.985965+0000","last_became_peered":"2024-03-19T01:51:06.985965+0000","last_unstale":"2024-03-19T01:51:18.088226+0000","last_undegraded":"2024-03-19T01:51:18.088226+0000","last_fullsized":"2024-03-19T01:51:18.088226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746886+0000","last_change":"2024-03-19T01:51:16.505278+0000","last_active":"2024-03-19T01:51:17.746886+0000","last_peered":"2024-03-19T01:51:17.746886+0000","last_clean":"2024-03-19T01:51:17.746886+0000","last_became_active":"2024-03-19T01:51:16.504908+0000","last_became_peered":"2024-03-19T01:51:16.504908+0000","last_unstale":"2024-03-19T01:51:17.746886+0000","last_undegraded":"2024-03-19T01:51:17.746886+0000","last_fullsized":"2024-03-19T01:51:17.746886+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:42:30.142766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085144+0000","last_change":"2024-03-19T01:51:09.179725+0000","last_active":"2024-03-19T01:51:18.085144+0000","last_peered":"2024-03-19T01:51:18.085144+0000","last_clean":"2024-03-19T01:51:18.085144+0000","last_became_active":"2024-03-19T01:51:09.179235+0000","last_became_peered":"2024-03-19T01:51:09.179235+0000","last_unstale":"2024-03-19T01:51:18.085144+0000","last_undegraded":"2024-03-19T01:51:18.085144+0000","last_fullsized":"2024-03-19T01:51:18.085144+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:31.395833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.490498+0000","last_change":"2024-03-19T01:51:06.953474+0000","last_active":"2024-03-19T01:51:50.490498+0000","last_peered":"2024-03-19T01:51:50.490498+0000","last_clean":"2024-03-19T01:51:50.490498+0000","last_became_active":"2024-03-19T01:51:06.952898+0000","last_became_peered":"2024-03-19T01:51:06.952898+0000","last_unstale":"2024-03-19T01:51:50.490498+0000","last_undegraded":"2024-03-19T01:51:50.490498+0000","last_fullsized":"2024-03-19T01:51:50.490498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:38.808918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":184,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088303+0000","last_change":"2024-03-19T01:50:26.097484+0000","last_active":"2024-03-19T01:51:18.088303+0000","last_peered":"2024-03-19T01:51:18.088303+0000","last_clean":"2024-03-19T01:51:18.088303+0000","last_became_active":"2024-03-19T01:50:26.095496+0000","last_became_peered":"2024-03-19T01:50:26.095496+0000","last_unstale":"2024-03-19T01:51:18.088303+0000","last_undegraded":"2024-03-19T01:51:18.088303+0000","last_fullsized":"2024-03-19T01:51:18.088303+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:50:24.112070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:50:24.112070+0000","last_clean_scrub_stamp":"2024-03-19T01:50:24.112070+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:05:00.843775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426122+0000","last_change":"2024-03-19T01:51:16.580516+0000","last_active":"2024-03-19T01:51:17.426122+0000","last_peered":"2024-03-19T01:51:17.426122+0000","last_clean":"2024-03-19T01:51:17.426122+0000","last_became_active":"2024-03-19T01:51:16.579906+0000","last_became_peered":"2024-03-19T01:51:16.579906+0000","last_unstale":"2024-03-19T01:51:17.426122+0000","last_undegraded":"2024-03-19T01:51:17.426122+0000","last_fullsized":"2024-03-19T01:51:17.426122+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:53:57.897649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089446+0000","last_change":"2024-03-19T01:51:09.196648+0000","last_active":"2024-03-19T01:51:18.089446+0000","last_peered":"2024-03-19T01:51:18.089446+0000","last_clean":"2024-03-19T01:51:18.089446+0000","last_became_active":"2024-03-19T01:51:09.196093+0000","last_became_peered":"2024-03-19T01:51:09.196093+0000","last_unstale":"2024-03-19T01:51:18.089446+0000","last_undegraded":"2024-03-19T01:51:18.089446+0000","last_fullsized":"2024-03-19T01:51:18.089446+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:54:09.655994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086018+0000","last_change":"2024-03-19T01:51:06.950023+0000","last_active":"2024-03-19T01:51:18.086018+0000","last_peered":"2024-03-19T01:51:18.086018+0000","last_clean":"2024-03-19T01:51:18.086018+0000","last_became_active":"2024-03-19T01:51:06.949234+0000","last_became_peered":"2024-03-19T01:51:06.949234+0000","last_unstale":"2024-03-19T01:51:18.086018+0000","last_undegraded":"2024-03-19T01:51:18.086018+0000","last_fullsized":"2024-03-19T01:51:18.086018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:54:13.131325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425731+0000","last_change":"2024-03-19T01:51:16.554046+0000","last_active":"2024-03-19T01:51:17.425731+0000","last_peered":"2024-03-19T01:51:17.425731+0000","last_clean":"2024-03-19T01:51:17.425731+0000","last_became_active":"2024-03-19T01:51:16.553595+0000","last_became_peered":"2024-03-19T01:51:16.553595+0000","last_unstale":"2024-03-19T01:51:17.425731+0000","last_undegraded":"2024-03-19T01:51:17.425731+0000","last_fullsized":"2024-03-19T01:51:17.425731+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:25.684226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088279+0000","last_change":"2024-03-19T01:51:09.181518+0000","last_active":"2024-03-19T01:51:18.088279+0000","last_peered":"2024-03-19T01:51:18.088279+0000","last_clean":"2024-03-19T01:51:18.088279+0000","last_became_active":"2024-03-19T01:51:09.174122+0000","last_became_peered":"2024-03-19T01:51:09.174122+0000","last_unstale":"2024-03-19T01:51:18.088279+0000","last_undegraded":"2024-03-19T01:51:18.088279+0000","last_fullsized":"2024-03-19T01:51:18.088279+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:18:53.079890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"24'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472911+0000","last_change":"2024-03-19T01:51:06.938396+0000","last_active":"2024-03-19T01:51:17.472911+0000","last_peered":"2024-03-19T01:51:17.472911+0000","last_clean":"2024-03-19T01:51:17.472911+0000","last_became_active":"2024-03-19T01:51:06.937580+0000","last_became_peered":"2024-03-19T01:51:06.937580+0000","last_unstale":"2024-03-19T01:51:17.472911+0000","last_undegraded":"2024-03-19T01:51:17.472911+0000","last_fullsized":"2024-03-19T01:51:17.472911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:08.307376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746471+0000","last_change":"2024-03-19T01:51:16.582111+0000","last_active":"2024-03-19T01:51:17.746471+0000","last_peered":"2024-03-19T01:51:17.746471+0000","last_clean":"2024-03-19T01:51:17.746471+0000","last_became_active":"2024-03-19T01:51:16.581746+0000","last_became_peered":"2024-03-19T01:51:16.581746+0000","last_unstale":"2024-03-19T01:51:17.746471+0000","last_undegraded":"2024-03-19T01:51:17.746471+0000","last_fullsized":"2024-03-19T01:51:17.746471+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:19.442287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090581+0000","last_change":"2024-03-19T01:51:09.195225+0000","last_active":"2024-03-19T01:51:18.090581+0000","last_peered":"2024-03-19T01:51:18.090581+0000","last_clean":"2024-03-19T01:51:18.090581+0000","last_became_active":"2024-03-19T01:51:09.194645+0000","last_became_peered":"2024-03-19T01:51:09.194645+0000","last_unstale":"2024-03-19T01:51:18.090581+0000","last_undegraded":"2024-03-19T01:51:18.090581+0000","last_fullsized":"2024-03-19T01:51:18.090581+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:50:04.024767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"24'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086668+0000","last_change":"2024-03-19T01:51:06.955976+0000","last_active":"2024-03-19T01:51:18.086668+0000","last_peered":"2024-03-19T01:51:18.086668+0000","last_clean":"2024-03-19T01:51:18.086668+0000","last_became_active":"2024-03-19T01:51:06.954951+0000","last_became_peered":"2024-03-19T01:51:06.954951+0000","last_unstale":"2024-03-19T01:51:18.086668+0000","last_undegraded":"2024-03-19T01:51:18.086668+0000","last_fullsized":"2024-03-19T01:51:18.086668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:47.536586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087418+0000","last_change":"2024-03-19T01:51:16.586904+0000","last_active":"2024-03-19T01:51:18.087418+0000","last_peered":"2024-03-19T01:51:18.087418+0000","last_clean":"2024-03-19T01:51:18.087418+0000","last_became_active":"2024-03-19T01:51:16.586298+0000","last_became_peered":"2024-03-19T01:51:16.586298+0000","last_unstale":"2024-03-19T01:51:18.087418+0000","last_undegraded":"2024-03-19T01:51:18.087418+0000","last_fullsized":"2024-03-19T01:51:18.087418+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:42.703415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089106+0000","last_change":"2024-03-19T01:51:09.195400+0000","last_active":"2024-03-19T01:51:18.089106+0000","last_peered":"2024-03-19T01:51:18.089106+0000","last_clean":"2024-03-19T01:51:18.089106+0000","last_became_active":"2024-03-19T01:51:09.195111+0000","last_became_peered":"2024-03-19T01:51:09.195111+0000","last_unstale":"2024-03-19T01:51:18.089106+0000","last_undegraded":"2024-03-19T01:51:18.089106+0000","last_fullsized":"2024-03-19T01:51:18.089106+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:12.084475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"24'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522397+0000","last_change":"2024-03-19T01:51:06.963650+0000","last_active":"2024-03-19T01:51:17.522397+0000","last_peered":"2024-03-19T01:51:17.522397+0000","last_clean":"2024-03-19T01:51:17.522397+0000","last_became_active":"2024-03-19T01:51:06.963028+0000","last_became_peered":"2024-03-19T01:51:06.963028+0000","last_unstale":"2024-03-19T01:51:17.522397+0000","last_undegraded":"2024-03-19T01:51:17.522397+0000","last_fullsized":"2024-03-19T01:51:17.522397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:59:26.369856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521307+0000","last_change":"2024-03-19T01:51:16.588077+0000","last_active":"2024-03-19T01:51:17.521307+0000","last_peered":"2024-03-19T01:51:17.521307+0000","last_clean":"2024-03-19T01:51:17.521307+0000","last_became_active":"2024-03-19T01:51:16.587459+0000","last_became_peered":"2024-03-19T01:51:16.587459+0000","last_unstale":"2024-03-19T01:51:17.521307+0000","last_undegraded":"2024-03-19T01:51:17.521307+0000","last_fullsized":"2024-03-19T01:51:17.521307+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:52.667114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089490+0000","last_change":"2024-03-19T01:51:09.195758+0000","last_active":"2024-03-19T01:51:18.089490+0000","last_peered":"2024-03-19T01:51:18.089490+0000","last_clean":"2024-03-19T01:51:18.089490+0000","last_became_active":"2024-03-19T01:51:09.195139+0000","last_became_peered":"2024-03-19T01:51:09.195139+0000","last_unstale":"2024-03-19T01:51:18.089490+0000","last_undegraded":"2024-03-19T01:51:18.089490+0000","last_fullsized":"2024-03-19T01:51:18.089490+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:18.326414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"29'10","reported_seq":39,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.372190+0000","last_change":"2024-03-19T01:51:06.941200+0000","last_active":"2024-03-19T01:51:50.372190+0000","last_peered":"2024-03-19T01:51:50.372190+0000","last_clean":"2024-03-19T01:51:50.372190+0000","last_became_active":"2024-03-19T01:51:06.940594+0000","last_became_peered":"2024-03-19T01:51:06.940594+0000","last_unstale":"2024-03-19T01:51:50.372190+0000","last_undegraded":"2024-03-19T01:51:50.372190+0000","last_fullsized":"2024-03-19T01:51:50.372190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:25:05.442423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746500+0000","last_change":"2024-03-19T01:51:16.548804+0000","last_active":"2024-03-19T01:51:17.746500+0000","last_peered":"2024-03-19T01:51:17.746500+0000","last_clean":"2024-03-19T01:51:17.746500+0000","last_became_active":"2024-03-19T01:51:16.547800+0000","last_became_peered":"2024-03-19T01:51:16.547800+0000","last_unstale":"2024-03-19T01:51:17.746500+0000","last_undegraded":"2024-03-19T01:51:17.746500+0000","last_fullsized":"2024-03-19T01:51:17.746500+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:43.172796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425380+0000","last_change":"2024-03-19T01:51:09.167987+0000","last_active":"2024-03-19T01:51:17.425380+0000","last_peered":"2024-03-19T01:51:17.425380+0000","last_clean":"2024-03-19T01:51:17.425380+0000","last_became_active":"2024-03-19T01:51:09.167670+0000","last_became_peered":"2024-03-19T01:51:09.167670+0000","last_unstale":"2024-03-19T01:51:17.425380+0000","last_undegraded":"2024-03-19T01:51:17.425380+0000","last_fullsized":"2024-03-19T01:51:17.425380+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425534+0000","last_change":"2024-03-19T01:51:07.048197+0000","last_active":"2024-03-19T01:51:17.425534+0000","last_peered":"2024-03-19T01:51:17.425534+0000","last_clean":"2024-03-19T01:51:17.425534+0000","last_became_active":"2024-03-19T01:51:07.047785+0000","last_became_peered":"2024-03-19T01:51:07.047785+0000","last_unstale":"2024-03-19T01:51:17.425534+0000","last_undegraded":"2024-03-19T01:51:17.425534+0000","last_fullsized":"2024-03-19T01:51:17.425534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:52.323249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087324+0000","last_change":"2024-03-19T01:51:16.583394+0000","last_active":"2024-03-19T01:51:18.087324+0000","last_peered":"2024-03-19T01:51:18.087324+0000","last_clean":"2024-03-19T01:51:18.087324+0000","last_became_active":"2024-03-19T01:51:16.582860+0000","last_became_peered":"2024-03-19T01:51:16.582860+0000","last_unstale":"2024-03-19T01:51:18.087324+0000","last_undegraded":"2024-03-19T01:51:18.087324+0000","last_fullsized":"2024-03-19T01:51:18.087324+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:37.669591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.091680+0000","last_change":"2024-03-19T01:51:09.195008+0000","last_active":"2024-03-19T01:51:18.091680+0000","last_peered":"2024-03-19T01:51:18.091680+0000","last_clean":"2024-03-19T01:51:18.091680+0000","last_became_active":"2024-03-19T01:51:09.194728+0000","last_became_peered":"2024-03-19T01:51:09.194728+0000","last_unstale":"2024-03-19T01:51:18.091680+0000","last_undegraded":"2024-03-19T01:51:18.091680+0000","last_fullsized":"2024-03-19T01:51:18.091680+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:15:10.360817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.490004+0000","last_change":"2024-03-19T01:51:06.956905+0000","last_active":"2024-03-19T01:51:50.490004+0000","last_peered":"2024-03-19T01:51:50.490004+0000","last_clean":"2024-03-19T01:51:50.490004+0000","last_became_active":"2024-03-19T01:51:06.955991+0000","last_became_peered":"2024-03-19T01:51:06.955991+0000","last_unstale":"2024-03-19T01:51:50.490004+0000","last_undegraded":"2024-03-19T01:51:50.490004+0000","last_fullsized":"2024-03-19T01:51:50.490004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:02:45.970853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087664+0000","last_change":"2024-03-19T01:51:16.539701+0000","last_active":"2024-03-19T01:51:18.087664+0000","last_peered":"2024-03-19T01:51:18.087664+0000","last_clean":"2024-03-19T01:51:18.087664+0000","last_became_active":"2024-03-19T01:51:16.539254+0000","last_became_peered":"2024-03-19T01:51:16.539254+0000","last_unstale":"2024-03-19T01:51:18.087664+0000","last_undegraded":"2024-03-19T01:51:18.087664+0000","last_fullsized":"2024-03-19T01:51:18.087664+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:40:24.244131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.745703+0000","last_change":"2024-03-19T01:51:09.188003+0000","last_active":"2024-03-19T01:51:17.745703+0000","last_peered":"2024-03-19T01:51:17.745703+0000","last_clean":"2024-03-19T01:51:17.745703+0000","last_became_active":"2024-03-19T01:51:09.187275+0000","last_became_peered":"2024-03-19T01:51:09.187275+0000","last_unstale":"2024-03-19T01:51:17.745703+0000","last_undegraded":"2024-03-19T01:51:17.745703+0000","last_fullsized":"2024-03-19T01:51:17.745703+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:10.624259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.344999+0000","last_change":"2024-03-19T01:51:06.964413+0000","last_active":"2024-03-19T01:51:50.344999+0000","last_peered":"2024-03-19T01:51:50.344999+0000","last_clean":"2024-03-19T01:51:50.344999+0000","last_became_active":"2024-03-19T01:51:06.963763+0000","last_became_peered":"2024-03-19T01:51:06.963763+0000","last_unstale":"2024-03-19T01:51:50.344999+0000","last_undegraded":"2024-03-19T01:51:50.344999+0000","last_fullsized":"2024-03-19T01:51:50.344999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089726+0000","last_change":"2024-03-19T01:51:16.574401+0000","last_active":"2024-03-19T01:51:18.089726+0000","last_peered":"2024-03-19T01:51:18.089726+0000","last_clean":"2024-03-19T01:51:18.089726+0000","last_became_active":"2024-03-19T01:51:16.573744+0000","last_became_peered":"2024-03-19T01:51:16.573744+0000","last_unstale":"2024-03-19T01:51:18.089726+0000","last_undegraded":"2024-03-19T01:51:18.089726+0000","last_fullsized":"2024-03-19T01:51:18.089726+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:41.414631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089315+0000","last_change":"2024-03-19T01:51:09.187207+0000","last_active":"2024-03-19T01:51:18.089315+0000","last_peered":"2024-03-19T01:51:18.089315+0000","last_clean":"2024-03-19T01:51:18.089315+0000","last_became_active":"2024-03-19T01:51:09.175898+0000","last_became_peered":"2024-03-19T01:51:09.175898+0000","last_unstale":"2024-03-19T01:51:18.089315+0000","last_undegraded":"2024-03-19T01:51:18.089315+0000","last_fullsized":"2024-03-19T01:51:18.089315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:38.387069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'12","reported_seq":41,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.370011+0000","last_change":"2024-03-19T01:51:07.042663+0000","last_active":"2024-03-19T01:51:50.370011+0000","last_peered":"2024-03-19T01:51:50.370011+0000","last_clean":"2024-03-19T01:51:50.370011+0000","last_became_active":"2024-03-19T01:51:07.041262+0000","last_became_peered":"2024-03-19T01:51:07.041262+0000","last_unstale":"2024-03-19T01:51:50.370011+0000","last_undegraded":"2024-03-19T01:51:50.370011+0000","last_fullsized":"2024-03-19T01:51:50.370011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.471923+0000","last_change":"2024-03-19T01:51:16.549744+0000","last_active":"2024-03-19T01:51:17.471923+0000","last_peered":"2024-03-19T01:51:17.471923+0000","last_clean":"2024-03-19T01:51:17.471923+0000","last_became_active":"2024-03-19T01:51:16.549347+0000","last_became_peered":"2024-03-19T01:51:16.549347+0000","last_unstale":"2024-03-19T01:51:17.471923+0000","last_undegraded":"2024-03-19T01:51:17.471923+0000","last_fullsized":"2024-03-19T01:51:17.471923+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:19.442287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522393+0000","last_change":"2024-03-19T01:51:09.194165+0000","last_active":"2024-03-19T01:51:17.522393+0000","last_peered":"2024-03-19T01:51:17.522393+0000","last_clean":"2024-03-19T01:51:17.522393+0000","last_became_active":"2024-03-19T01:51:09.183607+0000","last_became_peered":"2024-03-19T01:51:09.183607+0000","last_unstale":"2024-03-19T01:51:17.522393+0000","last_undegraded":"2024-03-19T01:51:17.522393+0000","last_fullsized":"2024-03-19T01:51:17.522393+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:09.211794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.473060+0000","last_change":"2024-03-19T01:51:06.952515+0000","last_active":"2024-03-19T01:51:17.473060+0000","last_peered":"2024-03-19T01:51:17.473060+0000","last_clean":"2024-03-19T01:51:17.473060+0000","last_became_active":"2024-03-19T01:51:06.951700+0000","last_became_peered":"2024-03-19T01:51:06.951700+0000","last_unstale":"2024-03-19T01:51:17.473060+0000","last_undegraded":"2024-03-19T01:51:17.473060+0000","last_fullsized":"2024-03-19T01:51:17.473060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:13.137673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522152+0000","last_change":"2024-03-19T01:51:16.587027+0000","last_active":"2024-03-19T01:51:17.522152+0000","last_peered":"2024-03-19T01:51:17.522152+0000","last_clean":"2024-03-19T01:51:17.522152+0000","last_became_active":"2024-03-19T01:51:16.586514+0000","last_became_peered":"2024-03-19T01:51:16.586514+0000","last_unstale":"2024-03-19T01:51:17.522152+0000","last_undegraded":"2024-03-19T01:51:17.522152+0000","last_fullsized":"2024-03-19T01:51:17.522152+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:06.637617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746586+0000","last_change":"2024-03-19T01:51:09.172554+0000","last_active":"2024-03-19T01:51:17.746586+0000","last_peered":"2024-03-19T01:51:17.746586+0000","last_clean":"2024-03-19T01:51:17.746586+0000","last_became_active":"2024-03-19T01:51:09.171814+0000","last_became_peered":"2024-03-19T01:51:09.171814+0000","last_unstale":"2024-03-19T01:51:17.746586+0000","last_undegraded":"2024-03-19T01:51:17.746586+0000","last_fullsized":"2024-03-19T01:51:17.746586+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.488757+0000","last_change":"2024-03-19T01:51:07.043404+0000","last_active":"2024-03-19T01:51:50.488757+0000","last_peered":"2024-03-19T01:51:50.488757+0000","last_clean":"2024-03-19T01:51:50.488757+0000","last_became_active":"2024-03-19T01:51:07.043069+0000","last_became_peered":"2024-03-19T01:51:07.043069+0000","last_unstale":"2024-03-19T01:51:50.488757+0000","last_undegraded":"2024-03-19T01:51:50.488757+0000","last_fullsized":"2024-03-19T01:51:50.488757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:13:14.960132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085834+0000","last_change":"2024-03-19T01:51:16.581797+0000","last_active":"2024-03-19T01:51:18.085834+0000","last_peered":"2024-03-19T01:51:18.085834+0000","last_clean":"2024-03-19T01:51:18.085834+0000","last_became_active":"2024-03-19T01:51:16.581174+0000","last_became_peered":"2024-03-19T01:51:16.581174+0000","last_unstale":"2024-03-19T01:51:18.085834+0000","last_undegraded":"2024-03-19T01:51:18.085834+0000","last_fullsized":"2024-03-19T01:51:18.085834+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:29:34.911347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090870+0000","last_change":"2024-03-19T01:51:09.194372+0000","last_active":"2024-03-19T01:51:18.090870+0000","last_peered":"2024-03-19T01:51:18.090870+0000","last_clean":"2024-03-19T01:51:18.090870+0000","last_became_active":"2024-03-19T01:51:09.193656+0000","last_became_peered":"2024-03-19T01:51:09.193656+0000","last_unstale":"2024-03-19T01:51:18.090870+0000","last_undegraded":"2024-03-19T01:51:18.090870+0000","last_fullsized":"2024-03-19T01:51:18.090870+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:23.464803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.371819+0000","last_change":"2024-03-19T01:51:07.046089+0000","last_active":"2024-03-19T01:51:50.371819+0000","last_peered":"2024-03-19T01:51:50.371819+0000","last_clean":"2024-03-19T01:51:50.371819+0000","last_became_active":"2024-03-19T01:51:07.045342+0000","last_became_peered":"2024-03-19T01:51:07.045342+0000","last_unstale":"2024-03-19T01:51:50.371819+0000","last_undegraded":"2024-03-19T01:51:50.371819+0000","last_fullsized":"2024-03-19T01:51:50.371819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:26.105915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087835+0000","last_change":"2024-03-19T01:51:16.572559+0000","last_active":"2024-03-19T01:51:18.087835+0000","last_peered":"2024-03-19T01:51:18.087835+0000","last_clean":"2024-03-19T01:51:18.087835+0000","last_became_active":"2024-03-19T01:51:16.572283+0000","last_became_peered":"2024-03-19T01:51:16.572283+0000","last_unstale":"2024-03-19T01:51:18.087835+0000","last_undegraded":"2024-03-19T01:51:18.087835+0000","last_fullsized":"2024-03-19T01:51:18.087835+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:42.703415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746448+0000","last_change":"2024-03-19T01:51:16.534088+0000","last_active":"2024-03-19T01:51:17.746448+0000","last_peered":"2024-03-19T01:51:17.746448+0000","last_clean":"2024-03-19T01:51:17.746448+0000","last_became_active":"2024-03-19T01:51:16.533665+0000","last_became_peered":"2024-03-19T01:51:16.533665+0000","last_unstale":"2024-03-19T01:51:17.746448+0000","last_undegraded":"2024-03-19T01:51:17.746448+0000","last_fullsized":"2024-03-19T01:51:17.746448+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:35.409504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.471346+0000","last_change":"2024-03-19T01:51:09.168372+0000","last_active":"2024-03-19T01:51:17.471346+0000","last_peered":"2024-03-19T01:51:17.471346+0000","last_clean":"2024-03-19T01:51:17.471346+0000","last_became_active":"2024-03-19T01:51:09.167626+0000","last_became_peered":"2024-03-19T01:51:09.167626+0000","last_unstale":"2024-03-19T01:51:17.471346+0000","last_undegraded":"2024-03-19T01:51:17.471346+0000","last_fullsized":"2024-03-19T01:51:17.471346+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087704+0000","last_change":"2024-03-19T01:51:06.951225+0000","last_active":"2024-03-19T01:51:18.087704+0000","last_peered":"2024-03-19T01:51:18.087704+0000","last_clean":"2024-03-19T01:51:18.087704+0000","last_became_active":"2024-03-19T01:51:06.950826+0000","last_became_peered":"2024-03-19T01:51:06.950826+0000","last_unstale":"2024-03-19T01:51:18.087704+0000","last_undegraded":"2024-03-19T01:51:18.087704+0000","last_fullsized":"2024-03-19T01:51:18.087704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:24.949925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426878+0000","last_change":"2024-03-19T01:51:16.581997+0000","last_active":"2024-03-19T01:51:17.426878+0000","last_peered":"2024-03-19T01:51:17.426878+0000","last_clean":"2024-03-19T01:51:17.426878+0000","last_became_active":"2024-03-19T01:51:16.581412+0000","last_became_peered":"2024-03-19T01:51:16.581412+0000","last_unstale":"2024-03-19T01:51:17.426878+0000","last_undegraded":"2024-03-19T01:51:17.426878+0000","last_fullsized":"2024-03-19T01:51:17.426878+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:45.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085931+0000","last_change":"2024-03-19T01:51:09.161686+0000","last_active":"2024-03-19T01:51:18.085931+0000","last_peered":"2024-03-19T01:51:18.085931+0000","last_clean":"2024-03-19T01:51:18.085931+0000","last_became_active":"2024-03-19T01:51:09.160978+0000","last_became_peered":"2024-03-19T01:51:09.160978+0000","last_unstale":"2024-03-19T01:51:18.085931+0000","last_undegraded":"2024-03-19T01:51:18.085931+0000","last_fullsized":"2024-03-19T01:51:18.085931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:27.266807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088069+0000","last_change":"2024-03-19T01:51:07.047363+0000","last_active":"2024-03-19T01:51:18.088069+0000","last_peered":"2024-03-19T01:51:18.088069+0000","last_clean":"2024-03-19T01:51:18.088069+0000","last_became_active":"2024-03-19T01:51:07.046886+0000","last_became_peered":"2024-03-19T01:51:07.046886+0000","last_unstale":"2024-03-19T01:51:18.088069+0000","last_undegraded":"2024-03-19T01:51:18.088069+0000","last_fullsized":"2024-03-19T01:51:18.088069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:19.418603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"25'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426847+0000","last_change":"2024-03-19T01:51:07.045496+0000","last_active":"2024-03-19T01:51:17.426847+0000","last_peered":"2024-03-19T01:51:17.426847+0000","last_clean":"2024-03-19T01:51:17.426847+0000","last_became_active":"2024-03-19T01:51:07.044554+0000","last_became_peered":"2024-03-19T01:51:07.044554+0000","last_unstale":"2024-03-19T01:51:17.426847+0000","last_undegraded":"2024-03-19T01:51:17.426847+0000","last_fullsized":"2024-03-19T01:51:17.426847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:14.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746492+0000","last_change":"2024-03-19T01:51:09.184934+0000","last_active":"2024-03-19T01:51:17.746492+0000","last_peered":"2024-03-19T01:51:17.746492+0000","last_clean":"2024-03-19T01:51:17.746492+0000","last_became_active":"2024-03-19T01:51:09.184666+0000","last_became_peered":"2024-03-19T01:51:09.184666+0000","last_unstale":"2024-03-19T01:51:17.746492+0000","last_undegraded":"2024-03-19T01:51:17.746492+0000","last_fullsized":"2024-03-19T01:51:17.746492+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:42.786323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085344+0000","last_change":"2024-03-19T01:51:16.575084+0000","last_active":"2024-03-19T01:51:18.085344+0000","last_peered":"2024-03-19T01:51:18.085344+0000","last_clean":"2024-03-19T01:51:18.085344+0000","last_became_active":"2024-03-19T01:51:16.574631+0000","last_became_peered":"2024-03-19T01:51:16.574631+0000","last_unstale":"2024-03-19T01:51:18.085344+0000","last_undegraded":"2024-03-19T01:51:18.085344+0000","last_fullsized":"2024-03-19T01:51:18.085344+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:24:35.634113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.503274+0000","last_change":"2024-03-19T01:51:06.982280+0000","last_active":"2024-03-19T01:51:50.503274+0000","last_peered":"2024-03-19T01:51:50.503274+0000","last_clean":"2024-03-19T01:51:50.503274+0000","last_became_active":"2024-03-19T01:51:06.981704+0000","last_became_peered":"2024-03-19T01:51:06.981704+0000","last_unstale":"2024-03-19T01:51:50.503274+0000","last_undegraded":"2024-03-19T01:51:50.503274+0000","last_fullsized":"2024-03-19T01:51:50.503274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:22:12.657985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521885+0000","last_change":"2024-03-19T01:51:09.184624+0000","last_active":"2024-03-19T01:51:17.521885+0000","last_peered":"2024-03-19T01:51:17.521885+0000","last_clean":"2024-03-19T01:51:17.521885+0000","last_became_active":"2024-03-19T01:51:09.183976+0000","last_became_peered":"2024-03-19T01:51:09.183976+0000","last_unstale":"2024-03-19T01:51:17.521885+0000","last_undegraded":"2024-03-19T01:51:17.521885+0000","last_fullsized":"2024-03-19T01:51:17.521885+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:24:29.313729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086581+0000","last_change":"2024-03-19T01:51:16.576775+0000","last_active":"2024-03-19T01:51:18.086581+0000","last_peered":"2024-03-19T01:51:18.086581+0000","last_clean":"2024-03-19T01:51:18.086581+0000","last_became_active":"2024-03-19T01:51:16.576406+0000","last_became_peered":"2024-03-19T01:51:16.576406+0000","last_unstale":"2024-03-19T01:51:18.086581+0000","last_undegraded":"2024-03-19T01:51:18.086581+0000","last_fullsized":"2024-03-19T01:51:18.086581+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:43.725447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425113+0000","last_change":"2024-03-19T01:51:09.169493+0000","last_active":"2024-03-19T01:51:17.425113+0000","last_peered":"2024-03-19T01:51:17.425113+0000","last_clean":"2024-03-19T01:51:17.425113+0000","last_became_active":"2024-03-19T01:51:09.168231+0000","last_became_peered":"2024-03-19T01:51:09.168231+0000","last_unstale":"2024-03-19T01:51:17.425113+0000","last_undegraded":"2024-03-19T01:51:17.425113+0000","last_fullsized":"2024-03-19T01:51:17.425113+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:10.624259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085344+0000","last_change":"2024-03-19T01:51:09.155483+0000","last_active":"2024-03-19T01:51:18.085344+0000","last_peered":"2024-03-19T01:51:18.085344+0000","last_clean":"2024-03-19T01:51:18.085344+0000","last_became_active":"2024-03-19T01:51:09.154545+0000","last_became_peered":"2024-03-19T01:51:09.154545+0000","last_unstale":"2024-03-19T01:51:18.085344+0000","last_undegraded":"2024-03-19T01:51:18.085344+0000","last_fullsized":"2024-03-19T01:51:18.085344+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:25:09.517323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088404+0000","last_change":"2024-03-19T01:51:09.180498+0000","last_active":"2024-03-19T01:51:18.088404+0000","last_peered":"2024-03-19T01:51:18.088404+0000","last_clean":"2024-03-19T01:51:18.088404+0000","last_became_active":"2024-03-19T01:51:09.179840+0000","last_became_peered":"2024-03-19T01:51:09.179840+0000","last_unstale":"2024-03-19T01:51:18.088404+0000","last_undegraded":"2024-03-19T01:51:18.088404+0000","last_fullsized":"2024-03-19T01:51:18.088404+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:01:21.716571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746226+0000","last_change":"2024-03-19T01:51:09.170389+0000","last_active":"2024-03-19T01:51:17.746226+0000","last_peered":"2024-03-19T01:51:17.746226+0000","last_clean":"2024-03-19T01:51:17.746226+0000","last_became_active":"2024-03-19T01:51:09.163974+0000","last_became_peered":"2024-03-19T01:51:09.163974+0000","last_unstale":"2024-03-19T01:51:17.746226+0000","last_undegraded":"2024-03-19T01:51:17.746226+0000","last_fullsized":"2024-03-19T01:51:17.746226+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087052+0000","last_change":"2024-03-19T01:51:09.177060+0000","last_active":"2024-03-19T01:51:18.087052+0000","last_peered":"2024-03-19T01:51:18.087052+0000","last_clean":"2024-03-19T01:51:18.087052+0000","last_became_active":"2024-03-19T01:51:09.176474+0000","last_became_peered":"2024-03-19T01:51:09.176474+0000","last_unstale":"2024-03-19T01:51:18.087052+0000","last_undegraded":"2024-03-19T01:51:18.087052+0000","last_fullsized":"2024-03-19T01:51:18.087052+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:56.318137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472633+0000","last_change":"2024-03-19T01:51:09.168741+0000","last_active":"2024-03-19T01:51:17.472633+0000","last_peered":"2024-03-19T01:51:17.472633+0000","last_clean":"2024-03-19T01:51:17.472633+0000","last_became_active":"2024-03-19T01:51:09.167982+0000","last_became_peered":"2024-03-19T01:51:09.167982+0000","last_unstale":"2024-03-19T01:51:17.472633+0000","last_undegraded":"2024-03-19T01:51:17.472633+0000","last_fullsized":"2024-03-19T01:51:17.472633+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425915+0000","last_change":"2024-03-19T01:51:09.189814+0000","last_active":"2024-03-19T01:51:17.425915+0000","last_peered":"2024-03-19T01:51:17.425915+0000","last_clean":"2024-03-19T01:51:17.425915+0000","last_became_active":"2024-03-19T01:51:09.189128+0000","last_became_peered":"2024-03-19T01:51:09.189128+0000","last_unstale":"2024-03-19T01:51:17.425915+0000","last_undegraded":"2024-03-19T01:51:17.425915+0000","last_fullsized":"2024-03-19T01:51:17.425915+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087688+0000","last_change":"2024-03-19T01:51:09.183989+0000","last_active":"2024-03-19T01:51:18.087688+0000","last_peered":"2024-03-19T01:51:18.087688+0000","last_clean":"2024-03-19T01:51:18.087688+0000","last_became_active":"2024-03-19T01:51:09.183417+0000","last_became_peered":"2024-03-19T01:51:09.183417+0000","last_unstale":"2024-03-19T01:51:18.087688+0000","last_undegraded":"2024-03-19T01:51:18.087688+0000","last_fullsized":"2024-03-19T01:51:18.087688+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:31.405618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087352+0000","last_change":"2024-03-19T01:51:09.172717+0000","last_active":"2024-03-19T01:51:18.087352+0000","last_peered":"2024-03-19T01:51:18.087352+0000","last_clean":"2024-03-19T01:51:18.087352+0000","last_became_active":"2024-03-19T01:51:09.165872+0000","last_became_peered":"2024-03-19T01:51:09.165872+0000","last_unstale":"2024-03-19T01:51:18.087352+0000","last_undegraded":"2024-03-19T01:51:18.087352+0000","last_fullsized":"2024-03-19T01:51:18.087352+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:10.995492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088202+0000","last_change":"2024-03-19T01:51:09.195116+0000","last_active":"2024-03-19T01:51:18.088202+0000","last_peered":"2024-03-19T01:51:18.088202+0000","last_clean":"2024-03-19T01:51:18.088202+0000","last_became_active":"2024-03-19T01:51:09.194558+0000","last_became_peered":"2024-03-19T01:51:09.194558+0000","last_unstale":"2024-03-19T01:51:18.088202+0000","last_undegraded":"2024-03-19T01:51:18.088202+0000","last_fullsized":"2024-03-19T01:51:18.088202+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:41:53.624234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088160+0000","last_change":"2024-03-19T01:51:09.195141+0000","last_active":"2024-03-19T01:51:18.088160+0000","last_peered":"2024-03-19T01:51:18.088160+0000","last_clean":"2024-03-19T01:51:18.088160+0000","last_became_active":"2024-03-19T01:51:09.194536+0000","last_became_peered":"2024-03-19T01:51:09.194536+0000","last_unstale":"2024-03-19T01:51:18.088160+0000","last_undegraded":"2024-03-19T01:51:18.088160+0000","last_fullsized":"2024-03-19T01:51:18.088160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:31.395833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.745890+0000","last_change":"2024-03-19T01:51:09.189197+0000","last_active":"2024-03-19T01:51:17.745890+0000","last_peered":"2024-03-19T01:51:17.745890+0000","last_clean":"2024-03-19T01:51:17.745890+0000","last_became_active":"2024-03-19T01:51:09.188565+0000","last_became_peered":"2024-03-19T01:51:09.188565+0000","last_unstale":"2024-03-19T01:51:17.745890+0000","last_undegraded":"2024-03-19T01:51:17.745890+0000","last_fullsized":"2024-03-19T01:51:17.745890+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426243+0000","last_change":"2024-03-19T01:51:09.168581+0000","last_active":"2024-03-19T01:51:17.426243+0000","last_peered":"2024-03-19T01:51:17.426243+0000","last_clean":"2024-03-19T01:51:17.426243+0000","last_became_active":"2024-03-19T01:51:09.167812+0000","last_became_peered":"2024-03-19T01:51:09.167812+0000","last_unstale":"2024-03-19T01:51:17.426243+0000","last_undegraded":"2024-03-19T01:51:17.426243+0000","last_fullsized":"2024-03-19T01:51:17.426243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089822+0000","last_change":"2024-03-19T01:51:09.188390+0000","last_active":"2024-03-19T01:51:18.089822+0000","last_peered":"2024-03-19T01:51:18.089822+0000","last_clean":"2024-03-19T01:51:18.089822+0000","last_became_active":"2024-03-19T01:51:09.187104+0000","last_became_peered":"2024-03-19T01:51:09.187104+0000","last_unstale":"2024-03-19T01:51:18.089822+0000","last_undegraded":"2024-03-19T01:51:18.089822+0000","last_fullsized":"2024-03-19T01:51:18.089822+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:37.257686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086435+0000","last_change":"2024-03-19T01:51:09.168451+0000","last_active":"2024-03-19T01:51:18.086435+0000","last_peered":"2024-03-19T01:51:18.086435+0000","last_clean":"2024-03-19T01:51:18.086435+0000","last_became_active":"2024-03-19T01:51:09.167744+0000","last_became_peered":"2024-03-19T01:51:09.167744+0000","last_unstale":"2024-03-19T01:51:18.086435+0000","last_undegraded":"2024-03-19T01:51:18.086435+0000","last_fullsized":"2024-03-19T01:51:18.086435+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:21.735485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089493+0000","last_change":"2024-03-19T01:51:09.196067+0000","last_active":"2024-03-19T01:51:18.089493+0000","last_peered":"2024-03-19T01:51:18.089493+0000","last_clean":"2024-03-19T01:51:18.089493+0000","last_became_active":"2024-03-19T01:51:09.195488+0000","last_became_peered":"2024-03-19T01:51:09.195488+0000","last_unstale":"2024-03-19T01:51:18.089493+0000","last_undegraded":"2024-03-19T01:51:18.089493+0000","last_fullsized":"2024-03-19T01:51:18.089493+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:07:44.120161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087081+0000","last_change":"2024-03-19T01:51:09.193831+0000","last_active":"2024-03-19T01:51:18.087081+0000","last_peered":"2024-03-19T01:51:18.087081+0000","last_clean":"2024-03-19T01:51:18.087081+0000","last_became_active":"2024-03-19T01:51:09.192777+0000","last_became_peered":"2024-03-19T01:51:09.192777+0000","last_unstale":"2024-03-19T01:51:18.087081+0000","last_undegraded":"2024-03-19T01:51:18.087081+0000","last_fullsized":"2024-03-19T01:51:18.087081+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:56.318137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.471571+0000","last_change":"2024-03-19T01:51:09.193581+0000","last_active":"2024-03-19T01:51:17.471571+0000","last_peered":"2024-03-19T01:51:17.471571+0000","last_clean":"2024-03-19T01:51:17.471571+0000","last_became_active":"2024-03-19T01:51:09.192677+0000","last_became_peered":"2024-03-19T01:51:09.192677+0000","last_unstale":"2024-03-19T01:51:17.471571+0000","last_undegraded":"2024-03-19T01:51:17.471571+0000","last_fullsized":"2024-03-19T01:51:17.471571+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.745598+0000","last_change":"2024-03-19T01:51:09.165568+0000","last_active":"2024-03-19T01:51:17.745598+0000","last_peered":"2024-03-19T01:51:17.745598+0000","last_clean":"2024-03-19T01:51:17.745598+0000","last_became_active":"2024-03-19T01:51:09.164753+0000","last_became_peered":"2024-03-19T01:51:09.164753+0000","last_unstale":"2024-03-19T01:51:17.745598+0000","last_undegraded":"2024-03-19T01:51:17.745598+0000","last_fullsized":"2024-03-19T01:51:17.745598+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:28.422798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086531+0000","last_change":"2024-03-19T01:51:09.170898+0000","last_active":"2024-03-19T01:51:18.086531+0000","last_peered":"2024-03-19T01:51:18.086531+0000","last_clean":"2024-03-19T01:51:18.086531+0000","last_became_active":"2024-03-19T01:51:09.170153+0000","last_became_peered":"2024-03-19T01:51:09.170153+0000","last_unstale":"2024-03-19T01:51:18.086531+0000","last_undegraded":"2024-03-19T01:51:18.086531+0000","last_fullsized":"2024-03-19T01:51:18.086531+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:41.125800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522152+0000","last_change":"2024-03-19T01:51:09.185953+0000","last_active":"2024-03-19T01:51:17.522152+0000","last_peered":"2024-03-19T01:51:17.522152+0000","last_clean":"2024-03-19T01:51:17.522152+0000","last_became_active":"2024-03-19T01:51:09.177996+0000","last_became_peered":"2024-03-19T01:51:09.177996+0000","last_unstale":"2024-03-19T01:51:17.522152+0000","last_undegraded":"2024-03-19T01:51:17.522152+0000","last_fullsized":"2024-03-19T01:51:17.522152+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:39:06.045413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426250+0000","last_change":"2024-03-19T01:51:09.183841+0000","last_active":"2024-03-19T01:51:17.426250+0000","last_peered":"2024-03-19T01:51:17.426250+0000","last_clean":"2024-03-19T01:51:17.426250+0000","last_became_active":"2024-03-19T01:51:09.183045+0000","last_became_peered":"2024-03-19T01:51:09.183045+0000","last_unstale":"2024-03-19T01:51:17.426250+0000","last_undegraded":"2024-03-19T01:51:17.426250+0000","last_fullsized":"2024-03-19T01:51:17.426250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425940+0000","last_change":"2024-03-19T01:51:09.184022+0000","last_active":"2024-03-19T01:51:17.425940+0000","last_peered":"2024-03-19T01:51:17.425940+0000","last_clean":"2024-03-19T01:51:17.425940+0000","last_became_active":"2024-03-19T01:51:09.183351+0000","last_became_peered":"2024-03-19T01:51:09.183351+0000","last_unstale":"2024-03-19T01:51:17.425940+0000","last_undegraded":"2024-03-19T01:51:17.425940+0000","last_fullsized":"2024-03-19T01:51:17.425940+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090466+0000","last_change":"2024-03-19T01:51:09.204005+0000","last_active":"2024-03-19T01:51:18.090466+0000","last_peered":"2024-03-19T01:51:18.090466+0000","last_clean":"2024-03-19T01:51:18.090466+0000","last_became_active":"2024-03-19T01:51:09.203545+0000","last_became_peered":"2024-03-19T01:51:09.203545+0000","last_unstale":"2024-03-19T01:51:18.090466+0000","last_undegraded":"2024-03-19T01:51:18.090466+0000","last_fullsized":"2024-03-19T01:51:18.090466+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:42:22.784953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085666+0000","last_change":"2024-03-19T01:51:09.180013+0000","last_active":"2024-03-19T01:51:18.085666+0000","last_peered":"2024-03-19T01:51:18.085666+0000","last_clean":"2024-03-19T01:51:18.085666+0000","last_became_active":"2024-03-19T01:51:09.179541+0000","last_became_peered":"2024-03-19T01:51:09.179541+0000","last_unstale":"2024-03-19T01:51:18.085666+0000","last_undegraded":"2024-03-19T01:51:18.085666+0000","last_fullsized":"2024-03-19T01:51:18.085666+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:19:25.933028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089242+0000","last_change":"2024-03-19T01:51:09.188671+0000","last_active":"2024-03-19T01:51:18.089242+0000","last_peered":"2024-03-19T01:51:18.089242+0000","last_clean":"2024-03-19T01:51:18.089242+0000","last_became_active":"2024-03-19T01:51:09.177627+0000","last_became_peered":"2024-03-19T01:51:09.177627+0000","last_unstale":"2024-03-19T01:51:18.089242+0000","last_undegraded":"2024-03-19T01:51:18.089242+0000","last_fullsized":"2024-03-19T01:51:18.089242+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:53:50.539837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087926+0000","last_change":"2024-03-19T01:51:09.180735+0000","last_active":"2024-03-19T01:51:18.087926+0000","last_peered":"2024-03-19T01:51:18.087926+0000","last_clean":"2024-03-19T01:51:18.087926+0000","last_became_active":"2024-03-19T01:51:09.173235+0000","last_became_peered":"2024-03-19T01:51:09.173235+0000","last_unstale":"2024-03-19T01:51:18.087926+0000","last_undegraded":"2024-03-19T01:51:18.087926+0000","last_fullsized":"2024-03-19T01:51:18.087926+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:31.405618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.091424+0000","last_change":"2024-03-19T01:51:09.188346+0000","last_active":"2024-03-19T01:51:18.091424+0000","last_peered":"2024-03-19T01:51:18.091424+0000","last_clean":"2024-03-19T01:51:18.091424+0000","last_became_active":"2024-03-19T01:51:09.187703+0000","last_became_peered":"2024-03-19T01:51:09.187703+0000","last_unstale":"2024-03-19T01:51:18.091424+0000","last_undegraded":"2024-03-19T01:51:18.091424+0000","last_fullsized":"2024-03-19T01:51:18.091424+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:28.051691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087787+0000","last_change":"2024-03-19T01:51:09.182114+0000","last_active":"2024-03-19T01:51:18.087787+0000","last_peered":"2024-03-19T01:51:18.087787+0000","last_clean":"2024-03-19T01:51:18.087787+0000","last_became_active":"2024-03-19T01:51:09.181056+0000","last_became_peered":"2024-03-19T01:51:09.181056+0000","last_unstale":"2024-03-19T01:51:18.087787+0000","last_undegraded":"2024-03-19T01:51:18.087787+0000","last_fullsized":"2024-03-19T01:51:18.087787+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:03:04.997711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746015+0000","last_change":"2024-03-19T01:51:09.161476+0000","last_active":"2024-03-19T01:51:17.746015+0000","last_peered":"2024-03-19T01:51:17.746015+0000","last_clean":"2024-03-19T01:51:17.746015+0000","last_became_active":"2024-03-19T01:51:09.158609+0000","last_became_peered":"2024-03-19T01:51:09.158609+0000","last_unstale":"2024-03-19T01:51:17.746015+0000","last_undegraded":"2024-03-19T01:51:17.746015+0000","last_fullsized":"2024-03-19T01:51:17.746015+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.473414+0000","last_change":"2024-03-19T01:51:09.165938+0000","last_active":"2024-03-19T01:51:17.473414+0000","last_peered":"2024-03-19T01:51:17.473414+0000","last_clean":"2024-03-19T01:51:17.473414+0000","last_became_active":"2024-03-19T01:51:09.165207+0000","last_became_peered":"2024-03-19T01:51:09.165207+0000","last_unstale":"2024-03-19T01:51:17.473414+0000","last_undegraded":"2024-03-19T01:51:17.473414+0000","last_fullsized":"2024-03-19T01:51:17.473414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090233+0000","last_change":"2024-03-19T01:51:09.176023+0000","last_active":"2024-03-19T01:51:18.090233+0000","last_peered":"2024-03-19T01:51:18.090233+0000","last_clean":"2024-03-19T01:51:18.090233+0000","last_became_active":"2024-03-19T01:51:09.175682+0000","last_became_peered":"2024-03-19T01:51:09.175682+0000","last_unstale":"2024-03-19T01:51:18.090233+0000","last_undegraded":"2024-03-19T01:51:18.090233+0000","last_fullsized":"2024-03-19T01:51:18.090233+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:35.814983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6046,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":12092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509461,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":921600,"data_stored":715706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509463,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":323584,"data_stored":127954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509462,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":124148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509463,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31492,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340348,"statfs":{"total":96636764160,"available":96604516352,"internally_reserved":0,"allocated":323584,"data_stored":124854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509462,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339768,"statfs":{"total":96636764160,"available":96603922432,"internally_reserved":0,"allocated":917504,"data_stored":715050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542165,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":124784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542166,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31496,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340344,"statfs":{"total":96636764160,"available":96604512256,"internally_reserved":0,"allocated":327680,"data_stored":128486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542165,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31476,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340364,"statfs":{"total":96636764160,"available":96604532736,"internally_reserved":0,"allocated":307200,"data_stored":124022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T01:51:55.950 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-19T01:51:55.951 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-19T01:51:55.951 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-19T01:51:55.951 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-19T01:51:55.951 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-19T01:51:56.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.105+0000 7ff6f9fb1640 1 Processor -- start 2024-03-19T01:51:56.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.105+0000 7ff6f9fb1640 1 -- start start 2024-03-19T01:51:56.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.109+0000 7ff6f9fb1640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff6f4150f50 0x7ff6f415b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:56.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.109+0000 7ff6f9fb1640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6f4151330 con 0x7ff6f40c3320 2024-03-19T01:51:56.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.109+0000 7ff6f9fb1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6f40c3840 con 0x7ff6f40ca350 2024-03-19T01:51:56.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.109+0000 7ff6f9fb1640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7ff6f40c3e00 con 0x7ff6f4150f50 2024-03-19T01:51:56.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.109+0000 7ff6f3fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7ff6f40c3320 0x7ff6f40ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:37544/0 (socket says 172.21.15.55:37544) 2024-03-19T01:51:56.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.109+0000 7ff6f3fff640 1 -- 172.21.15.55:0/3080945379 learned_addr learned my addr 172.21.15.55:0/3080945379 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:51:56.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.109+0000 7ff6f37fe640 1 --2- 172.21.15.55:0/3080945379 >> v2:172.21.15.84:3300/0 conn(0x7ff6f4150f50 0x7ff6f415b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:56.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.113+0000 7ff6f2ffd640 1 -- 172.21.15.55:0/3080945379 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2122522791 0 0) 0x7ff6f4151330 con 0x7ff6f40c3320 2024-03-19T01:51:56.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.113+0000 7ff6f2ffd640 1 -- 172.21.15.55:0/3080945379 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6e0005740 con 0x7ff6f40c3320 2024-03-19T01:51:56.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.117+0000 7ff6f2ffd640 1 -- 172.21.15.55:0/3080945379 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 351598164 0 0) 0x7ff6e0005740 con 0x7ff6f40c3320 2024-03-19T01:51:56.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.117+0000 7ff6f2ffd640 1 -- 172.21.15.55:0/3080945379 >> v2:172.21.15.84:3300/0 conn(0x7ff6f4150f50 msgr2=0x7ff6f415b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:56.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.117+0000 7ff6f2ffd640 1 --2- 172.21.15.55:0/3080945379 >> v2:172.21.15.84:3300/0 conn(0x7ff6f4150f50 0x7ff6f415b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:56.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.117+0000 7ff6f2ffd640 1 -- 172.21.15.55:0/3080945379 >> v1:172.21.15.84:6789/0 conn(0x7ff6f40ca350 legacy=0x7ff6f4157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:56.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.117+0000 7ff6f2ffd640 1 -- 172.21.15.55:0/3080945379 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6f40c3b20 con 0x7ff6f40c3320 2024-03-19T01:51:56.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.117+0000 7ff6f2ffd640 1 -- 172.21.15.55:0/3080945379 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7ff6ec001a00 con 0x7ff6f40c3320 2024-03-19T01:51:56.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.117+0000 7ff6f2ffd640 1 -- 172.21.15.55:0/3080945379 >> v1:172.21.15.55:6789/0 conn(0x7ff6f40c3320 legacy=0x7ff6f40ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:56.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.117+0000 7ff6f2ffd640 1 --2- 172.21.15.55:0/3080945379 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff6e0005bd0 0x7ff6e0005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:56.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.117+0000 7ff6f2ffd640 1 --2- 172.21.15.55:0/3080945379 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff6e0006a20 0x7ff6e0006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:56.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.117+0000 7ff6f2ffd640 1 --2- 172.21.15.55:0/3080945379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e00073e0 0x7ff6e000dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:56.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.117+0000 7ff6f2ffd640 1 -- 172.21.15.55:0/3080945379 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f40c3e00 con 0x7ff6e00073e0 2024-03-19T01:51:56.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.117+0000 7ff6f2ffd640 1 -- 172.21.15.55:0/3080945379 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e000e280 con 0x7ff6e0005bd0 2024-03-19T01:51:56.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.117+0000 7ff6f2ffd640 1 -- 172.21.15.55:0/3080945379 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff6e000e480 con 0x7ff6e0006a20 2024-03-19T01:51:56.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.117+0000 7ff6ebfff640 1 --2- 172.21.15.55:0/3080945379 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff6e0006a20 0x7ff6e0006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:56.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.117+0000 7ff6f3fff640 1 --2- 172.21.15.55:0/3080945379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e00073e0 0x7ff6e000dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:56.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.125+0000 7ff6f3fff640 1 -- 172.21.15.55:0/3080945379 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff6e0006a20 msgr2=0x7ff6e0006ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:56.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.125+0000 7ff6f3fff640 1 --2- 172.21.15.55:0/3080945379 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff6e0006a20 0x7ff6e0006ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:56.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.125+0000 7ff6f3fff640 1 -- 172.21.15.55:0/3080945379 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff6e0005bd0 msgr2=0x7ff6e0005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:56.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.125+0000 7ff6f3fff640 1 --2- 172.21.15.55:0/3080945379 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff6e0005bd0 0x7ff6e0005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:56.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.125+0000 7ff6f3fff640 1 -- 172.21.15.55:0/3080945379 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6e0005740 con 0x7ff6e00073e0 2024-03-19T01:51:56.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.125+0000 7ff6f3fff640 1 -- 172.21.15.55:0/3080945379 reap_dead start 2024-03-19T01:51:56.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.129+0000 7ff6f3fff640 1 --2- 172.21.15.55:0/3080945379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e00073e0 0x7ff6e000dbe0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec00a830 tx=0x7ff6ec01d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b96b119f62fcd0e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:56.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.133+0000 7ff6f2ffd640 1 -- 172.21.15.55:0/3080945379 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6ec00a480 con 0x7ff6e00073e0 2024-03-19T01:51:56.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.133+0000 7ff6f2ffd640 1 -- 172.21.15.55:0/3080945379 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6ec0ac620 con 0x7ff6e00073e0 2024-03-19T01:51:56.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.133+0000 7ff6f2ffd640 1 -- 172.21.15.55:0/3080945379 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6ec0aca10 con 0x7ff6e00073e0 2024-03-19T01:51:56.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.133+0000 7ff6f9fb1640 1 -- 172.21.15.55:0/3080945379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e00073e0 msgr2=0x7ff6e000dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:56.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.133+0000 7ff6f9fb1640 1 --2- 172.21.15.55:0/3080945379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e00073e0 0x7ff6e000dbe0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec00a830 tx=0x7ff6ec01d360 comp rx=0 tx=0).stop 2024-03-19T01:51:56.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.133+0000 7ff6f9fb1640 1 -- 172.21.15.55:0/3080945379 shutdown_connections 2024-03-19T01:51:56.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.133+0000 7ff6f9fb1640 1 --2- 172.21.15.55:0/3080945379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e00073e0 0x7ff6e000dbe0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:56.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.133+0000 7ff6f9fb1640 1 -- 172.21.15.55:0/3080945379 >> 172.21.15.55:0/3080945379 conn(0x7ff6f406e670 msgr2=0x7ff6f406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:56.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.133+0000 7ff6f9fb1640 1 -- 172.21.15.55:0/3080945379 shutdown_connections 2024-03-19T01:51:56.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.133+0000 7ff6f9fb1640 1 -- 172.21.15.55:0/3080945379 wait complete. 2024-03-19T01:51:56.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.133+0000 7ff6f9fb1640 1 Processor -- start 2024-03-19T01:51:56.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.133+0000 7ff6f9fb1640 1 -- start start 2024-03-19T01:51:56.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.137+0000 7ff6f9fb1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e00073e0 0x7ff6f414e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:56.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.137+0000 7ff6f9fb1640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff6f406ba30 0x7ff6f41558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:56.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.137+0000 7ff6f9fb1640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff6f40cbc90 0x7ff6f40cc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:56.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.137+0000 7ff6f9fb1640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f415a8a0 con 0x7ff6e00073e0 2024-03-19T01:51:56.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.137+0000 7ff6f9fb1640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f415ac10 con 0x7ff6f40cbc90 2024-03-19T01:51:56.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.137+0000 7ff6f9fb1640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff6f415b0a0 con 0x7ff6f406ba30 2024-03-19T01:51:56.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.137+0000 7ff6f37fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e00073e0 0x7ff6f414e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:56.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.137+0000 7ff6f37fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e00073e0 0x7ff6f414e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34262/0 (socket says 172.21.15.55:34262) 2024-03-19T01:51:56.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.137+0000 7ff6f37fe640 1 -- 172.21.15.55:0/399688053 learned_addr learned my addr 172.21.15.55:0/399688053 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:56.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.137+0000 7ff6ebfff640 1 --2- 172.21.15.55:0/399688053 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff6f406ba30 0x7ff6f41558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:56.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.137+0000 7ff6f3fff640 1 --2- 172.21.15.55:0/399688053 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff6f40cbc90 0x7ff6f40cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:56.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.141+0000 7ff6f37fe640 1 -- 172.21.15.55:0/399688053 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff6f406ba30 msgr2=0x7ff6f41558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:56.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.141+0000 7ff6f37fe640 1 --2- 172.21.15.55:0/399688053 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff6f406ba30 0x7ff6f41558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:56.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.141+0000 7ff6f37fe640 1 -- 172.21.15.55:0/399688053 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff6f40cbc90 msgr2=0x7ff6f40cc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:56.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.141+0000 7ff6f37fe640 1 --2- 172.21.15.55:0/399688053 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff6f40cbc90 0x7ff6f40cc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:56.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.141+0000 7ff6f37fe640 1 -- 172.21.15.55:0/399688053 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6f406c260 con 0x7ff6e00073e0 2024-03-19T01:51:56.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.145+0000 7ff6f37fe640 1 --2- 172.21.15.55:0/399688053 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e00073e0 0x7ff6f414e2b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400e820 tx=0x7ff6e400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:56.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.149+0000 7ff6f0ff9640 1 -- 172.21.15.55:0/399688053 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4017070 con 0x7ff6e00073e0 2024-03-19T01:51:56.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.149+0000 7ff6f0ff9640 1 -- 172.21.15.55:0/399688053 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6e400c8e0 con 0x7ff6e00073e0 2024-03-19T01:51:56.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.149+0000 7ff6f9fb1640 1 -- 172.21.15.55:0/399688053 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6f406c500 con 0x7ff6e00073e0 2024-03-19T01:51:56.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.149+0000 7ff6f0ff9640 1 -- 172.21.15.55:0/399688053 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e400ccb0 con 0x7ff6e00073e0 2024-03-19T01:51:56.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.149+0000 7ff6f9fb1640 1 -- 172.21.15.55:0/399688053 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6f406ce10 con 0x7ff6e00073e0 2024-03-19T01:51:56.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.149+0000 7ff6f9fb1640 1 -- 172.21.15.55:0/399688053 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6f4068a40 con 0x7ff6e00073e0 2024-03-19T01:51:56.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.181+0000 7ff6f0ff9640 1 -- 172.21.15.55:0/399688053 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ff6e401a070 con 0x7ff6e00073e0 2024-03-19T01:51:56.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.181+0000 7ff6f0ff9640 1 --2- 172.21.15.55:0/399688053 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff6d4067660 0x7ff6d4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:56.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.181+0000 7ff6ebfff640 1 --2- 172.21.15.55:0/399688053 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff6d4067660 0x7ff6d4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:56.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.181+0000 7ff6ebfff640 1 --2- 172.21.15.55:0/399688053 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff6d4067660 0x7ff6d4069b20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4155790 tx=0x7ff6dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:56.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.449+0000 7ff6f0ff9640 1 -- 172.21.15.55:0/399688053 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7ff6e4054500 con 0x7ff6e00073e0 2024-03-19T01:51:56.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.449+0000 7ff6f0ff9640 1 -- 172.21.15.55:0/399688053 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ff6e4056460 con 0x7ff6e00073e0 2024-03-19T01:51:56.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.657+0000 7ff6f9fb1640 1 -- 172.21.15.55:0/399688053 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7ff6f4145f00 con 0x7ff6e00073e0 2024-03-19T01:51:56.926 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.925+0000 7ff6f0ff9640 1 -- 172.21.15.55:0/399688053 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+166522 (secure 0 0 0) 0x7ff6e405a500 con 0x7ff6e00073e0 2024-03-19T01:51:56.928 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:51:56.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.929+0000 7ff6f9fb1640 1 -- 172.21.15.55:0/399688053 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff6d4067660 msgr2=0x7ff6d4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:56.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.929+0000 7ff6f9fb1640 1 --2- 172.21.15.55:0/399688053 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff6d4067660 0x7ff6d4069b20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4155790 tx=0x7ff6dc009040 comp rx=0 tx=0).stop 2024-03-19T01:51:56.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.929+0000 7ff6f9fb1640 1 -- 172.21.15.55:0/399688053 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e00073e0 msgr2=0x7ff6f414e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:56.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.929+0000 7ff6f9fb1640 1 --2- 172.21.15.55:0/399688053 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e00073e0 0x7ff6f414e2b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400e820 tx=0x7ff6e400ece0 comp rx=0 tx=0).stop 2024-03-19T01:51:56.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.929+0000 7ff6f9fb1640 1 -- 172.21.15.55:0/399688053 shutdown_connections 2024-03-19T01:51:56.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.929+0000 7ff6f9fb1640 1 --2- 172.21.15.55:0/399688053 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff6d4067660 0x7ff6d4069b20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:56.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.929+0000 7ff6f9fb1640 1 --2- 172.21.15.55:0/399688053 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff6f40cbc90 0x7ff6f40cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:56.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.929+0000 7ff6f9fb1640 1 --2- 172.21.15.55:0/399688053 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff6f406ba30 0x7ff6f41558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:56.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.929+0000 7ff6f9fb1640 1 --2- 172.21.15.55:0/399688053 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e00073e0 0x7ff6f414e2b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:56.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.929+0000 7ff6f9fb1640 1 -- 172.21.15.55:0/399688053 >> 172.21.15.55:0/399688053 conn(0x7ff6f406e670 msgr2=0x7ff6f414a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:56.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.929+0000 7ff6f9fb1640 1 -- 172.21.15.55:0/399688053 shutdown_connections 2024-03-19T01:51:56.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:56.929+0000 7ff6f9fb1640 1 -- 172.21.15.55:0/399688053 wait complete. 2024-03-19T01:51:56.950 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6800","nonce":998533783},{"type":"v1","addr":"172.21.15.84:6801","nonce":998533783}]},"active_addr":"172.21.15.84:6801/998533783","active_change":"2024-03-19T01:50:09.601585+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4108,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":1108900289}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":244975565}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":148767853}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":3920656118}]}]} 2024-03-19T01:51:56.954 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-19T01:51:56.955 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-19T01:51:56.955 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:51:57.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.085+0000 7f68d2dcd640 1 Processor -- start 2024-03-19T01:51:57.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.085+0000 7f68d2dcd640 1 -- start start 2024-03-19T01:51:57.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.085+0000 7f68d2dcd640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f68cc152120 0x7f68cc0c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:57.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.085+0000 7f68d2dcd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f68cc0c9610 0x7f68cc157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:57.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.085+0000 7f68d2dcd640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68cc06fff0 con 0x7f68cc070780 2024-03-19T01:51:57.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.085+0000 7f68d2dcd640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f68cc152500 con 0x7f68cc0c9610 2024-03-19T01:51:57.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.085+0000 7f68d2dcd640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f68cc070590 con 0x7f68cc152120 2024-03-19T01:51:57.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.085+0000 7f68cbfff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f68cc0c9610 0x7f68cc157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:57.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.085+0000 7f68cbfff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f68cc0c9610 0x7f68cc157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34276/0 (socket says 172.21.15.55:34276) 2024-03-19T01:51:57.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.085+0000 7f68cbfff640 1 -- 172.21.15.55:0/553373323 learned_addr learned my addr 172.21.15.55:0/553373323 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:57.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.089+0000 7f68cbfff640 1 -- 172.21.15.55:0/553373323 >> v2:172.21.15.84:3300/0 conn(0x7f68cc152120 msgr2=0x7f68cc0c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:57.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.089+0000 7f68cbfff640 1 --2- 172.21.15.55:0/553373323 >> v2:172.21.15.84:3300/0 conn(0x7f68cc152120 0x7f68cc0c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:57.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.089+0000 7f68cbfff640 1 -- 172.21.15.55:0/553373323 >> v1:172.21.15.84:6789/0 conn(0x7f68cc070780 legacy=0x7f68cc158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:57.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.089+0000 7f68cbfff640 1 -- 172.21.15.55:0/553373323 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68cc0c39c0 con 0x7f68cc0c9610 2024-03-19T01:51:57.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.093+0000 7f68cbfff640 1 --2- 172.21.15.55:0/553373323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68cc0c9610 0x7f68cc157fd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f68cc072430 tx=0x7f68b801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a802220da827607 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:57.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.093+0000 7f68cb7fe640 1 -- 172.21.15.55:0/553373323 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b80b0070 con 0x7f68cc0c9610 2024-03-19T01:51:57.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.097+0000 7f68cb7fe640 1 -- 172.21.15.55:0/553373323 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68b80ab600 con 0x7f68cc0c9610 2024-03-19T01:51:57.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.097+0000 7f68cb7fe640 1 -- 172.21.15.55:0/553373323 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b80ab9d0 con 0x7f68cc0c9610 2024-03-19T01:51:57.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.097+0000 7f68d2dcd640 1 -- 172.21.15.55:0/553373323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68cc0c9610 msgr2=0x7f68cc157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:57.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.097+0000 7f68d2dcd640 1 --2- 172.21.15.55:0/553373323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68cc0c9610 0x7f68cc157fd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f68cc072430 tx=0x7f68b801c4e0 comp rx=0 tx=0).stop 2024-03-19T01:51:57.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.097+0000 7f68d2dcd640 1 -- 172.21.15.55:0/553373323 shutdown_connections 2024-03-19T01:51:57.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.097+0000 7f68d2dcd640 1 --2- 172.21.15.55:0/553373323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68cc0c9610 0x7f68cc157fd0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:57.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.097+0000 7f68d2dcd640 1 --2- 172.21.15.55:0/553373323 >> v2:172.21.15.84:3300/0 conn(0x7f68cc152120 0x7f68cc0c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:57.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.097+0000 7f68d2dcd640 1 -- 172.21.15.55:0/553373323 >> 172.21.15.55:0/553373323 conn(0x7f68cc067600 msgr2=0x7f68cc071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:57.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.097+0000 7f68d2dcd640 1 -- 172.21.15.55:0/553373323 shutdown_connections 2024-03-19T01:51:57.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.097+0000 7f68d2dcd640 1 -- 172.21.15.55:0/553373323 wait complete. 2024-03-19T01:51:57.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.101+0000 7f68d2dcd640 1 Processor -- start 2024-03-19T01:51:57.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.101+0000 7f68d2dcd640 1 -- start start 2024-03-19T01:51:57.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.101+0000 7f68d2dcd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68cc070780 0x7f68cc14b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:57.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.101+0000 7f68d2dcd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68cc0c9610 0x7f68cc159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:57.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.101+0000 7f68d2dcd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68cc152120 0x7f68cc0b1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:57.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.101+0000 7f68d2dcd640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f68cc0c86d0 con 0x7f68cc0c9610 2024-03-19T01:51:57.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.101+0000 7f68d2dcd640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f68cc0c31c0 con 0x7f68cc070780 2024-03-19T01:51:57.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.101+0000 7f68d2dcd640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f68cc165170 con 0x7f68cc152120 2024-03-19T01:51:57.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.101+0000 7f68cbfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68cc0c9610 0x7f68cc159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:57.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.101+0000 7f68cbfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68cc0c9610 0x7f68cc159b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34278/0 (socket says 172.21.15.55:34278) 2024-03-19T01:51:57.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.101+0000 7f68cbfff640 1 -- 172.21.15.55:0/2666153748 learned_addr learned my addr 172.21.15.55:0/2666153748 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:57.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.101+0000 7f68d136a640 1 --2- 172.21.15.55:0/2666153748 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68cc152120 0x7f68cc0b1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:57.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.105+0000 7f68cbfff640 1 -- 172.21.15.55:0/2666153748 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68cc152120 msgr2=0x7f68cc0b1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:57.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.105+0000 7f68cbfff640 1 --2- 172.21.15.55:0/2666153748 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68cc152120 0x7f68cc0b1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:57.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.105+0000 7f68cbfff640 1 -- 172.21.15.55:0/2666153748 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68cc070780 msgr2=0x7f68cc14b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:57.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.105+0000 7f68cbfff640 1 --2- 172.21.15.55:0/2666153748 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68cc070780 0x7f68cc14b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:57.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.105+0000 7f68cbfff640 1 -- 172.21.15.55:0/2666153748 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68cc1f1750 con 0x7f68cc0c9610 2024-03-19T01:51:57.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.109+0000 7f68cbfff640 1 --2- 172.21.15.55:0/2666153748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68cc0c9610 0x7f68cc159b90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f68b80ae040 tx=0x7f68b801ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:57.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.113+0000 7f68c97fa640 1 -- 172.21.15.55:0/2666153748 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b80ba020 con 0x7f68cc0c9610 2024-03-19T01:51:57.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.113+0000 7f68c97fa640 1 -- 172.21.15.55:0/2666153748 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68b80b6640 con 0x7f68cc0c9610 2024-03-19T01:51:57.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.113+0000 7f68c97fa640 1 -- 172.21.15.55:0/2666153748 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b80b0040 con 0x7f68cc0c9610 2024-03-19T01:51:57.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.113+0000 7f68d2dcd640 1 -- 172.21.15.55:0/2666153748 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68cc0c6660 con 0x7f68cc0c9610 2024-03-19T01:51:57.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.113+0000 7f68d2dcd640 1 -- 172.21.15.55:0/2666153748 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68cc0c6c20 con 0x7f68cc0c9610 2024-03-19T01:51:57.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.113+0000 7f68d2dcd640 1 -- 172.21.15.55:0/2666153748 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68cc068a40 con 0x7f68cc0c9610 2024-03-19T01:51:57.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.133+0000 7f68c97fa640 1 -- 172.21.15.55:0/2666153748 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f68b80bd020 con 0x7f68cc0c9610 2024-03-19T01:51:57.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.137+0000 7f68c97fa640 1 --2- 172.21.15.55:0/2666153748 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68b00676e0 0x7f68b0069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:57.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.137+0000 7f68d0b69640 1 --2- 172.21.15.55:0/2666153748 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68b00676e0 0x7f68b0069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:57.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.137+0000 7f68d0b69640 1 --2- 172.21.15.55:0/2666153748 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68b00676e0 0x7f68b0069ba0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f68c0009690 tx=0x7f68c0006c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:57.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.421+0000 7f68c97fa640 1 -- 172.21.15.55:0/2666153748 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f68b80ee560 con 0x7f68cc0c9610 2024-03-19T01:51:57.422 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.421+0000 7f68c97fa640 1 -- 172.21.15.55:0/2666153748 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f68b80c4030 con 0x7f68cc0c9610 2024-03-19T01:51:57.614 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.613+0000 7f68d2dcd640 1 -- 172.21.15.55:0/2666153748 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f68cc0c84b0 con 0x7f68cc0c9610 2024-03-19T01:51:57.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.769+0000 7f68c97fa640 1 -- 172.21.15.55:0/2666153748 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+18430 (secure 0 0 0) 0x7f68b80160d0 con 0x7f68cc0c9610 2024-03-19T01:51:57.772 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:51:57.772 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":29,"fsid":"def9d4e5-0bc7-449d-b893-2490cbd10e25","created":"2024-03-19T01:50:02.482954+0000","modified":"2024-03-19T01:51:17.228291+0000","last_up_change":"2024-03-19T01:50:21.735508+0000","last_in_change":"2024-03-19T01:50:12.168944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:50:23.456904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:51:04.578583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:51:06.871659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T01:51:14.274442+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"35d5e3c5-d089-4f38-8961-98d064188f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6825","nonce":2757675043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6827","nonce":2757675043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6831","nonce":2757675043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6829","nonce":2757675043}]},"public_addr":"172.21.15.55:6825/2757675043","cluster_addr":"172.21.15.55:6827/2757675043","heartbeat_back_addr":"172.21.15.55:6831/2757675043","heartbeat_front_addr":"172.21.15.55:6829/2757675043","state":["exists","up"]},{"osd":1,"uuid":"8e58f791-001a-44d0-a464-ca68a479f70c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6817","nonce":543460007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6819","nonce":543460007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6823","nonce":543460007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6821","nonce":543460007}]},"public_addr":"172.21.15.55:6817/543460007","cluster_addr":"172.21.15.55:6819/543460007","heartbeat_back_addr":"172.21.15.55:6823/543460007","heartbeat_front_addr":"172.21.15.55:6821/543460007","state":["exists","up"]},{"osd":2,"uuid":"d44f38b8-67f4-43de-98a6-f3c520acaf10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6801","nonce":1760253550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6803","nonce":1760253550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6807","nonce":1760253550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6805","nonce":1760253550}]},"public_addr":"172.21.15.55:6801/1760253550","cluster_addr":"172.21.15.55:6803/1760253550","heartbeat_back_addr":"172.21.15.55:6807/1760253550","heartbeat_front_addr":"172.21.15.55:6805/1760253550","state":["exists","up"]},{"osd":3,"uuid":"850a7316-2d74-43f0-8649-b5f8d8f6692e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6809","nonce":2797616448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6811","nonce":2797616448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6815","nonce":2797616448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6813","nonce":2797616448}]},"public_addr":"172.21.15.55:6809/2797616448","cluster_addr":"172.21.15.55:6811/2797616448","heartbeat_back_addr":"172.21.15.55:6815/2797616448","heartbeat_front_addr":"172.21.15.55:6813/2797616448","state":["exists","up"]},{"osd":4,"uuid":"4e6152ae-a77d-4bb6-a855-5782ad5411a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6819","nonce":2938436826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6821","nonce":2938436826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6825","nonce":2938436826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6823","nonce":2938436826}]},"public_addr":"172.21.15.84:6819/2938436826","cluster_addr":"172.21.15.84:6821/2938436826","heartbeat_back_addr":"172.21.15.84:6825/2938436826","heartbeat_front_addr":"172.21.15.84:6823/2938436826","state":["exists","up"]},{"osd":5,"uuid":"1217ceba-dd31-4615-96f1-dfaf57160f3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6803","nonce":460717295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6805","nonce":460717295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6809","nonce":460717295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6807","nonce":460717295}]},"public_addr":"172.21.15.84:6803/460717295","cluster_addr":"172.21.15.84:6805/460717295","heartbeat_back_addr":"172.21.15.84:6809/460717295","heartbeat_front_addr":"172.21.15.84:6807/460717295","state":["exists","up"]},{"osd":6,"uuid":"3d9ed78a-2284-4c28-a24a-70eb812c7836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6827","nonce":416123135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6829","nonce":416123135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6833","nonce":416123135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6831","nonce":416123135}]},"public_addr":"172.21.15.84:6827/416123135","cluster_addr":"172.21.15.84:6829/416123135","heartbeat_back_addr":"172.21.15.84:6833/416123135","heartbeat_front_addr":"172.21.15.84:6831/416123135","state":["exists","up"]},{"osd":7,"uuid":"c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6811","nonce":460083528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6813","nonce":460083528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6817","nonce":460083528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6815","nonce":460083528}]},"public_addr":"172.21.15.84:6811/460083528","cluster_addr":"172.21.15.84:6813/460083528","heartbeat_back_addr":"172.21.15.84:6817/460083528","heartbeat_front_addr":"172.21.15.84:6815/460083528","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.269126+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.121001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.075714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.092461+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.194676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.062833+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.316515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.237111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:51:57.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.773+0000 7f68d2dcd640 1 -- 172.21.15.55:0/2666153748 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68b00676e0 msgr2=0x7f68b0069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:57.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.773+0000 7f68d2dcd640 1 --2- 172.21.15.55:0/2666153748 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68b00676e0 0x7f68b0069ba0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f68c0009690 tx=0x7f68c0006c00 comp rx=0 tx=0).stop 2024-03-19T01:51:57.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.773+0000 7f68d2dcd640 1 -- 172.21.15.55:0/2666153748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68cc0c9610 msgr2=0x7f68cc159b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:57.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.773+0000 7f68d2dcd640 1 --2- 172.21.15.55:0/2666153748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68cc0c9610 0x7f68cc159b90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f68b80ae040 tx=0x7f68b801ad20 comp rx=0 tx=0).stop 2024-03-19T01:51:57.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.773+0000 7f68d2dcd640 1 -- 172.21.15.55:0/2666153748 shutdown_connections 2024-03-19T01:51:57.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.773+0000 7f68d2dcd640 1 --2- 172.21.15.55:0/2666153748 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68b00676e0 0x7f68b0069ba0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:57.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.773+0000 7f68d2dcd640 1 --2- 172.21.15.55:0/2666153748 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68cc152120 0x7f68cc0b1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:57.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.773+0000 7f68d2dcd640 1 --2- 172.21.15.55:0/2666153748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68cc0c9610 0x7f68cc159b90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:57.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.773+0000 7f68d2dcd640 1 --2- 172.21.15.55:0/2666153748 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68cc070780 0x7f68cc14b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:57.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.773+0000 7f68d2dcd640 1 -- 172.21.15.55:0/2666153748 >> 172.21.15.55:0/2666153748 conn(0x7f68cc067600 msgr2=0x7f68cc06f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:57.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.773+0000 7f68d2dcd640 1 -- 172.21.15.55:0/2666153748 shutdown_connections 2024-03-19T01:51:57.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.773+0000 7f68d2dcd640 1 -- 172.21.15.55:0/2666153748 wait complete. 2024-03-19T01:51:57.797 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-19T01:51:57.797 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:51:57.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.909+0000 7f575df38640 1 Processor -- start 2024-03-19T01:51:57.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.909+0000 7f575df38640 1 -- start start 2024-03-19T01:51:57.913 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.913+0000 7f575df38640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f575815a490 0x7f5758072900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:57.913 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.913+0000 7f575df38640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57580ca350 0x7f5758070500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:57.913 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.913+0000 7f575df38640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57580738b0 con 0x7f5758150e50 2024-03-19T01:51:57.913 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.913+0000 7f575df38640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f5758072f40 con 0x7f575815a490 2024-03-19T01:51:57.913 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.913+0000 7f575df38640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f57580732f0 con 0x7f57580ca350 2024-03-19T01:51:57.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.913+0000 7f57577fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f5758150e50 0x7f57581578b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:37560/0 (socket says 172.21.15.55:37560) 2024-03-19T01:51:57.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.913+0000 7f57577fe640 1 -- 172.21.15.55:0/4012482331 learned_addr learned my addr 172.21.15.55:0/4012482331 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:51:57.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.913+0000 7f5757fff640 1 --2- 172.21.15.55:0/4012482331 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57580ca350 0x7f5758070500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:57.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.913+0000 7f5756ffd640 1 --2- 172.21.15.55:0/4012482331 >> v2:172.21.15.84:3300/0 conn(0x7f575815a490 0x7f5758072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:57.917 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.917+0000 7f57567fc640 1 -- 172.21.15.55:0/4012482331 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 32466156 0 0) 0x7f57580738b0 con 0x7f5758150e50 2024-03-19T01:51:57.917 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.917+0000 7f57567fc640 1 -- 172.21.15.55:0/4012482331 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5740003530 con 0x7f5758150e50 2024-03-19T01:51:57.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.921+0000 7f57567fc640 1 -- 172.21.15.55:0/4012482331 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2763350314 0 0) 0x7f5740003530 con 0x7f5758150e50 2024-03-19T01:51:57.922 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.921+0000 7f57567fc640 1 -- 172.21.15.55:0/4012482331 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57580ca350 msgr2=0x7f5758070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:57.922 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.921+0000 7f57567fc640 1 --2- 172.21.15.55:0/4012482331 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57580ca350 0x7f5758070500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:57.922 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.921+0000 7f57567fc640 1 -- 172.21.15.55:0/4012482331 >> v2:172.21.15.84:3300/0 conn(0x7f575815a490 msgr2=0x7f5758072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:57.922 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.921+0000 7f57567fc640 1 --2- 172.21.15.55:0/4012482331 >> v2:172.21.15.84:3300/0 conn(0x7f575815a490 0x7f5758072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:57.922 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.921+0000 7f57567fc640 1 -- 172.21.15.55:0/4012482331 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57580735d0 con 0x7f5758150e50 2024-03-19T01:51:57.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.921+0000 7f57567fc640 1 -- 172.21.15.55:0/4012482331 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f574c001680 con 0x7f5758150e50 2024-03-19T01:51:57.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.921+0000 7f57567fc640 1 -- 172.21.15.55:0/4012482331 >> v1:172.21.15.55:6789/0 conn(0x7f5758150e50 legacy=0x7f57581578b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:57.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.921+0000 7f57567fc640 1 --2- 172.21.15.55:0/4012482331 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57400039c0 0x7f5740003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:57.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.921+0000 7f57567fc640 1 --2- 172.21.15.55:0/4012482331 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57580ca350 0x7f5740004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:57.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.921+0000 7f57567fc640 1 --2- 172.21.15.55:0/4012482331 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5740004d50 0x7f57400051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:57.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.921+0000 7f57567fc640 1 -- 172.21.15.55:0/4012482331 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5758072f40 con 0x7f57400039c0 2024-03-19T01:51:57.924 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.921+0000 7f57567fc640 1 -- 172.21.15.55:0/4012482331 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f57580732f0 con 0x7f5740004d50 2024-03-19T01:51:57.924 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.921+0000 7f57567fc640 1 -- 172.21.15.55:0/4012482331 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f574000b9d0 con 0x7f57580ca350 2024-03-19T01:51:57.924 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.921+0000 7f57577fe640 1 --2- 172.21.15.55:0/4012482331 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57400039c0 0x7f5740003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:57.925 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.925+0000 7f57577fe640 1 -- 172.21.15.55:0/4012482331 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57580ca350 msgr2=0x7f5740004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:57.925 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.925+0000 7f57577fe640 1 --2- 172.21.15.55:0/4012482331 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57580ca350 0x7f5740004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:57.925 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.925+0000 7f57577fe640 1 -- 172.21.15.55:0/4012482331 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5740004d50 msgr2=0x7f57400051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:57.925 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.925+0000 7f57577fe640 1 --2- 172.21.15.55:0/4012482331 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5740004d50 0x7f57400051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:57.926 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.925+0000 7f57577fe640 1 -- 172.21.15.55:0/4012482331 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5740003530 con 0x7f57400039c0 2024-03-19T01:51:57.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.929+0000 7f57577fe640 1 --2- 172.21.15.55:0/4012482331 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57400039c0 0x7f5740003de0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f574c00c190 tx=0x7f574c01cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d387feebbf3504 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:57.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.933+0000 7f57567fc640 1 -- 172.21.15.55:0/4012482331 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574c007c70 con 0x7f57400039c0 2024-03-19T01:51:57.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.933+0000 7f57567fc640 1 -- 172.21.15.55:0/4012482331 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f574c0a9020 con 0x7f57400039c0 2024-03-19T01:51:57.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.933+0000 7f57567fc640 1 -- 172.21.15.55:0/4012482331 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574c005ae0 con 0x7f57400039c0 2024-03-19T01:51:57.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.933+0000 7f575df38640 1 -- 172.21.15.55:0/4012482331 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57400039c0 msgr2=0x7f5740003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:57.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.933+0000 7f575df38640 1 --2- 172.21.15.55:0/4012482331 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57400039c0 0x7f5740003de0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f574c00c190 tx=0x7f574c01cbe0 comp rx=0 tx=0).stop 2024-03-19T01:51:57.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.933+0000 7f5757fff640 1 -- 172.21.15.55:0/4012482331 reap_dead start 2024-03-19T01:51:57.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.933+0000 7f575df38640 1 -- 172.21.15.55:0/4012482331 shutdown_connections 2024-03-19T01:51:57.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.933+0000 7f575df38640 1 -- 172.21.15.55:0/4012482331 >> 172.21.15.55:0/4012482331 conn(0x7f575806c3b0 msgr2=0x7f575806c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:57.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.933+0000 7f575df38640 1 -- 172.21.15.55:0/4012482331 shutdown_connections 2024-03-19T01:51:57.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.933+0000 7f575df38640 1 -- 172.21.15.55:0/4012482331 wait complete. 2024-03-19T01:51:57.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.933+0000 7f575df38640 1 Processor -- start 2024-03-19T01:51:57.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.933+0000 7f575df38640 1 -- start start 2024-03-19T01:51:57.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.933+0000 7f575df38640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57400039c0 0x7f57581451d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:57.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.933+0000 7f575df38640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57581472c0 0x7f57581f6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:57.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.933+0000 7f575df38640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5758149610 0x7f57581f6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:57.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.933+0000 7f575df38640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f57581513b0 con 0x7f57400039c0 2024-03-19T01:51:57.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.933+0000 7f575df38640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f5758151230 con 0x7f5758149610 2024-03-19T01:51:57.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.933+0000 7f575df38640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f57580c1b60 con 0x7f57581472c0 2024-03-19T01:51:57.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.937+0000 7f57577fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57400039c0 0x7f57581451d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:57.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.937+0000 7f57577fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57400039c0 0x7f57581451d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34296/0 (socket says 172.21.15.55:34296) 2024-03-19T01:51:57.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.937+0000 7f57577fe640 1 -- 172.21.15.55:0/2588736605 learned_addr learned my addr 172.21.15.55:0/2588736605 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:57.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.937+0000 7f5756ffd640 1 --2- 172.21.15.55:0/2588736605 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57581472c0 0x7f57581f6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:57.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.937+0000 7f5757fff640 1 --2- 172.21.15.55:0/2588736605 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5758149610 0x7f57581f6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:57.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.941+0000 7f57577fe640 1 -- 172.21.15.55:0/2588736605 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57581472c0 msgr2=0x7f57581f6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:57.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.941+0000 7f57577fe640 1 --2- 172.21.15.55:0/2588736605 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57581472c0 0x7f57581f6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:57.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.941+0000 7f57577fe640 1 -- 172.21.15.55:0/2588736605 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5758149610 msgr2=0x7f57581f6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:57.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.941+0000 7f57577fe640 1 --2- 172.21.15.55:0/2588736605 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5758149610 0x7f57581f6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:57.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.941+0000 7f57577fe640 1 -- 172.21.15.55:0/2588736605 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5758147820 con 0x7f57400039c0 2024-03-19T01:51:57.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.941+0000 7f5757fff640 1 --2- 172.21.15.55:0/2588736605 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5758149610 0x7f57581f6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:51:57.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.941+0000 7f57577fe640 1 --2- 172.21.15.55:0/2588736605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57400039c0 0x7f57581451d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f574c007040 tx=0x7f5740005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:57.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.945+0000 7f573ffff640 1 -- 172.21.15.55:0/2588736605 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574c004bb0 con 0x7f57400039c0 2024-03-19T01:51:57.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.945+0000 7f573ffff640 1 -- 172.21.15.55:0/2588736605 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f574c004d50 con 0x7f57400039c0 2024-03-19T01:51:57.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.945+0000 7f573ffff640 1 -- 172.21.15.55:0/2588736605 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574c0072f0 con 0x7f57400039c0 2024-03-19T01:51:57.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.945+0000 7f575df38640 1 -- 172.21.15.55:0/2588736605 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57581f3890 con 0x7f57400039c0 2024-03-19T01:51:57.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.945+0000 7f575df38640 1 -- 172.21.15.55:0/2588736605 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f575814b480 con 0x7f57400039c0 2024-03-19T01:51:57.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:57.945+0000 7f575df38640 1 -- 172.21.15.55:0/2588736605 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5758068a40 con 0x7f57400039c0 2024-03-19T01:51:58.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.237+0000 7f573ffff640 1 -- 172.21.15.55:0/2588736605 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f574c007490 con 0x7f57400039c0 2024-03-19T01:51:58.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.237+0000 7f573ffff640 1 --2- 172.21.15.55:0/2588736605 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f57440676b0 0x7f5744069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.237+0000 7f5756ffd640 1 --2- 172.21.15.55:0/2588736605 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f57440676b0 0x7f5744069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.237+0000 7f5756ffd640 1 --2- 172.21.15.55:0/2588736605 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f57440676b0 0x7f5744069b70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f57581476c0 tx=0x7f57480094c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:58.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.273+0000 7f573ffff640 1 -- 172.21.15.55:0/2588736605 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f574c0ed5c0 con 0x7f57400039c0 2024-03-19T01:51:58.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.273+0000 7f573ffff640 1 -- 172.21.15.55:0/2588736605 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f5758147820 con 0x7f57400039c0 2024-03-19T01:51:58.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.449+0000 7f575df38640 1 -- 172.21.15.55:0/2588736605 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f575814bbb0 con 0x7f57400039c0 2024-03-19T01:51:58.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.569+0000 7f573ffff640 1 -- 172.21.15.55:0/2588736605 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+18430 (secure 0 0 0) 0x7f574c0b4070 con 0x7f57400039c0 2024-03-19T01:51:58.572 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:51:58.572 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":29,"fsid":"def9d4e5-0bc7-449d-b893-2490cbd10e25","created":"2024-03-19T01:50:02.482954+0000","modified":"2024-03-19T01:51:17.228291+0000","last_up_change":"2024-03-19T01:50:21.735508+0000","last_in_change":"2024-03-19T01:50:12.168944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:50:23.456904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:51:04.578583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:51:06.871659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T01:51:14.274442+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"35d5e3c5-d089-4f38-8961-98d064188f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6825","nonce":2757675043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6827","nonce":2757675043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6831","nonce":2757675043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6829","nonce":2757675043}]},"public_addr":"172.21.15.55:6825/2757675043","cluster_addr":"172.21.15.55:6827/2757675043","heartbeat_back_addr":"172.21.15.55:6831/2757675043","heartbeat_front_addr":"172.21.15.55:6829/2757675043","state":["exists","up"]},{"osd":1,"uuid":"8e58f791-001a-44d0-a464-ca68a479f70c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6817","nonce":543460007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6819","nonce":543460007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6823","nonce":543460007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6821","nonce":543460007}]},"public_addr":"172.21.15.55:6817/543460007","cluster_addr":"172.21.15.55:6819/543460007","heartbeat_back_addr":"172.21.15.55:6823/543460007","heartbeat_front_addr":"172.21.15.55:6821/543460007","state":["exists","up"]},{"osd":2,"uuid":"d44f38b8-67f4-43de-98a6-f3c520acaf10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6801","nonce":1760253550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6803","nonce":1760253550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6807","nonce":1760253550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6805","nonce":1760253550}]},"public_addr":"172.21.15.55:6801/1760253550","cluster_addr":"172.21.15.55:6803/1760253550","heartbeat_back_addr":"172.21.15.55:6807/1760253550","heartbeat_front_addr":"172.21.15.55:6805/1760253550","state":["exists","up"]},{"osd":3,"uuid":"850a7316-2d74-43f0-8649-b5f8d8f6692e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6809","nonce":2797616448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6811","nonce":2797616448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6815","nonce":2797616448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6813","nonce":2797616448}]},"public_addr":"172.21.15.55:6809/2797616448","cluster_addr":"172.21.15.55:6811/2797616448","heartbeat_back_addr":"172.21.15.55:6815/2797616448","heartbeat_front_addr":"172.21.15.55:6813/2797616448","state":["exists","up"]},{"osd":4,"uuid":"4e6152ae-a77d-4bb6-a855-5782ad5411a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6819","nonce":2938436826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6821","nonce":2938436826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6825","nonce":2938436826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6823","nonce":2938436826}]},"public_addr":"172.21.15.84:6819/2938436826","cluster_addr":"172.21.15.84:6821/2938436826","heartbeat_back_addr":"172.21.15.84:6825/2938436826","heartbeat_front_addr":"172.21.15.84:6823/2938436826","state":["exists","up"]},{"osd":5,"uuid":"1217ceba-dd31-4615-96f1-dfaf57160f3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6803","nonce":460717295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6805","nonce":460717295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6809","nonce":460717295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6807","nonce":460717295}]},"public_addr":"172.21.15.84:6803/460717295","cluster_addr":"172.21.15.84:6805/460717295","heartbeat_back_addr":"172.21.15.84:6809/460717295","heartbeat_front_addr":"172.21.15.84:6807/460717295","state":["exists","up"]},{"osd":6,"uuid":"3d9ed78a-2284-4c28-a24a-70eb812c7836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6827","nonce":416123135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6829","nonce":416123135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6833","nonce":416123135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6831","nonce":416123135}]},"public_addr":"172.21.15.84:6827/416123135","cluster_addr":"172.21.15.84:6829/416123135","heartbeat_back_addr":"172.21.15.84:6833/416123135","heartbeat_front_addr":"172.21.15.84:6831/416123135","state":["exists","up"]},{"osd":7,"uuid":"c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6811","nonce":460083528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6813","nonce":460083528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6817","nonce":460083528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6815","nonce":460083528}]},"public_addr":"172.21.15.84:6811/460083528","cluster_addr":"172.21.15.84:6813/460083528","heartbeat_back_addr":"172.21.15.84:6817/460083528","heartbeat_front_addr":"172.21.15.84:6815/460083528","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.269126+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.121001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.075714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.092461+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.194676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.062833+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.316515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.237111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:51:58.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.573+0000 7f575df38640 1 -- 172.21.15.55:0/2588736605 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f57440676b0 msgr2=0x7f5744069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.573+0000 7f575df38640 1 --2- 172.21.15.55:0/2588736605 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f57440676b0 0x7f5744069b70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f57581476c0 tx=0x7f57480094c0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.573+0000 7f575df38640 1 -- 172.21.15.55:0/2588736605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57400039c0 msgr2=0x7f57581451d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.573+0000 7f575df38640 1 --2- 172.21.15.55:0/2588736605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57400039c0 0x7f57581451d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f574c007040 tx=0x7f5740005840 comp rx=0 tx=0).stop 2024-03-19T01:51:58.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.573+0000 7f575df38640 1 -- 172.21.15.55:0/2588736605 shutdown_connections 2024-03-19T01:51:58.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.573+0000 7f575df38640 1 --2- 172.21.15.55:0/2588736605 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f57440676b0 0x7f5744069b70 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.573+0000 7f575df38640 1 --2- 172.21.15.55:0/2588736605 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5758149610 0x7f57581f6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.573+0000 7f575df38640 1 --2- 172.21.15.55:0/2588736605 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57581472c0 0x7f57581f6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.573+0000 7f575df38640 1 --2- 172.21.15.55:0/2588736605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57400039c0 0x7f57581451d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.573+0000 7f575df38640 1 -- 172.21.15.55:0/2588736605 >> 172.21.15.55:0/2588736605 conn(0x7f575806c3b0 msgr2=0x7f575806c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:58.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.573+0000 7f575df38640 1 -- 172.21.15.55:0/2588736605 shutdown_connections 2024-03-19T01:51:58.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.573+0000 7f575df38640 1 -- 172.21.15.55:0/2588736605 wait complete. 2024-03-19T01:51:58.598 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-19T01:51:58.598 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-19T01:51:58.599 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-19T01:51:58.599 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-19T01:51:58.599 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-19T01:51:58.600 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-19T01:51:58.601 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-19T01:51:58.601 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-19T01:51:58.693 INFO:tasks.ceph.mds.c.smithi055.stderr:2024-03-19T01:51:58.685+0000 f9f5640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-19T01:51:58.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.793+0000 7fa57cfa2640 1 Processor -- start 2024-03-19T01:51:58.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.793+0000 7fa57cfa2640 1 -- start start 2024-03-19T01:51:58.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.793+0000 7fa57cfa2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5781562e0 0x7fa5780c7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.793+0000 7fa57cfa2640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa5780c7e70 0x7fa5780cb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.793+0000 7fa57cfa2640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa578156770 con 0x7fa57814ccf0 2024-03-19T01:51:58.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.793+0000 7fa57cfa2640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fa5780c43d0 con 0x7fa5780c7e70 2024-03-19T01:51:58.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.793+0000 7fa57cfa2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa578156a50 con 0x7fa5781562e0 2024-03-19T01:51:58.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.797+0000 7f880f1a0640 1 Processor -- start 2024-03-19T01:51:58.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.797+0000 7f880f1a0640 1 -- start start 2024-03-19T01:51:58.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.797+0000 7fa57659c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fa57814ccf0 0x7fa578159f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:33158/0 (socket says 172.21.15.55:33158) 2024-03-19T01:51:58.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.797+0000 7fa57659c640 1 -- 172.21.15.55:0/795312671 learned_addr learned my addr 172.21.15.55:0/795312671 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:51:58.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.797+0000 7f880f1a0640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f880806fed0 0x7f8808159f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.797+0000 7f880f1a0640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f880815a4d0 0x7f88080c7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.797+0000 7f880f1a0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8808156770 con 0x7f88081562e0 2024-03-19T01:51:58.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.797+0000 7f880f1a0640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f88080c43d0 con 0x7f880815a4d0 2024-03-19T01:51:58.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.797+0000 7f880f1a0640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f8808156a50 con 0x7f880806fed0 2024-03-19T01:51:58.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.797+0000 7f1b3b163640 1 Processor -- start 2024-03-19T01:51:58.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.797+0000 7f1b3b163640 1 -- start start 2024-03-19T01:51:58.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.797+0000 7f1b3b163640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1b340caff0 0x7f1b341619d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.801+0000 7f880cf3c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f88081562e0 0x7f880814cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:33170/0 (socket says 172.21.15.55:33170) 2024-03-19T01:51:58.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.801+0000 7f880cf3c640 1 -- 172.21.15.55:0/3637556370 learned_addr learned my addr 172.21.15.55:0/3637556370 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:51:58.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.797+0000 7f1b3b163640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f1b340c8100 0x7f1b340c8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.797+0000 7f1b3b163640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b3415e530 con 0x7f1b34158870 2024-03-19T01:51:58.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.797+0000 7f1b3b163640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f1b340c9d50 con 0x7f1b340caff0 2024-03-19T01:51:58.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.797+0000 7f1b3b163640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f1b3415e7d0 con 0x7f1b340c8100 2024-03-19T01:51:58.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.801+0000 7fa575d9b640 1 --2- 172.21.15.55:0/795312671 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5781562e0 0x7fa5780c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.801+0000 7f1b3a161640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f1b34158870 0x7f1b340c22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:38534/0 (socket says 172.21.15.55:38534) 2024-03-19T01:51:58.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.801+0000 7f1b3a161640 1 -- 172.21.15.55:0/966516624 learned_addr learned my addr 172.21.15.55:0/966516624 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:51:58.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.801+0000 7fa576d9d640 1 --2- 172.21.15.55:0/795312671 >> v2:172.21.15.55:3300/0 conn(0x7fa5780c7e70 0x7fa5780cb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.805+0000 7f8807fff640 1 --2- 172.21.15.55:0/3637556370 >> v2:172.21.15.84:3300/0 conn(0x7f880806fed0 0x7f8808159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.809+0000 7fa575d9b640 1 -- 172.21.15.55:0/795312671 >> v2:172.21.15.55:3300/0 conn(0x7fa5780c7e70 msgr2=0x7fa5780cb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:58.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.809+0000 7fa575d9b640 1 --2- 172.21.15.55:0/795312671 >> v2:172.21.15.55:3300/0 conn(0x7fa5780c7e70 0x7fa5780cb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.809+0000 7fa575d9b640 1 -- 172.21.15.55:0/795312671 >> v1:172.21.15.84:6789/0 conn(0x7fa57814ccf0 legacy=0x7fa578159f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.809+0000 7fa575d9b640 1 -- 172.21.15.55:0/795312671 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa578156d30 con 0x7fa5781562e0 2024-03-19T01:51:58.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.813+0000 7fb24d963640 1 Processor -- start 2024-03-19T01:51:58.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.813+0000 7fb24d963640 1 -- start start 2024-03-19T01:51:58.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.813+0000 7f1b39960640 1 --2- 172.21.15.55:0/966516624 >> v2:172.21.15.55:3300/0 conn(0x7f1b340caff0 0x7f1b341619d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.813+0000 7fb24d963640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb2480c3520 0x7fb2480cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.813+0000 7fb24d963640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fb24806c630 0x7fb2480cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.813+0000 7fb24d963640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb24806c230 con 0x7fb248150ae0 2024-03-19T01:51:58.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.813+0000 7fb24d963640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fb2480c3a40 con 0x7fb2480c3520 2024-03-19T01:51:58.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.813+0000 7fb24d963640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fb2480c8200 con 0x7fb24806c630 2024-03-19T01:51:58.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.813+0000 7fa575d9b640 1 --2- 172.21.15.55:0/795312671 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5781562e0 0x7fa5780c7930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa5680097c0 tx=0x7fa56801ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37093d2f22fb7856 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:58.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.813+0000 7f135d0ce640 1 Processor -- start 2024-03-19T01:51:58.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.813+0000 7f135d0ce640 1 -- start start 2024-03-19T01:51:58.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.813+0000 7f963a2bc640 1 Processor -- start 2024-03-19T01:51:58.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.817+0000 7f963a2bc640 1 -- start start 2024-03-19T01:51:58.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.817+0000 7f135d0ce640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f13500d5220 0x7f135004be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.817+0000 7f135d0ce640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f13500d5600 0x7f135004c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.817+0000 7f135d0ce640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f135004b0e0 con 0x7f1350007960 2024-03-19T01:51:58.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.817+0000 7f135d0ce640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f135004b9f0 con 0x7f13500d5600 2024-03-19T01:51:58.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.817+0000 7f135d0ce640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f1350004bf0 con 0x7f13500d5220 2024-03-19T01:51:58.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.817+0000 7f880d73d640 1 --2- 172.21.15.55:0/3637556370 >> v2:172.21.15.55:3300/0 conn(0x7f880815a4d0 0x7f88080c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.817+0000 7f963a2bc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f963413ca40 0x7f9634073090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.817+0000 7f963a2bc640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f96340c6860 0x7f9634156f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.817+0000 7f963a2bc640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96340735d0 con 0x7f96340c6400 2024-03-19T01:51:58.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.817+0000 7f963a2bc640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f96340c6d20 con 0x7f96340c6860 2024-03-19T01:51:58.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.817+0000 7f963a2bc640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f96340c7000 con 0x7f963413ca40 2024-03-19T01:51:58.820 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.821+0000 7f1b39960640 1 -- 172.21.15.55:0/966516624 >> v2:172.21.15.84:3300/0 conn(0x7f1b340c8100 msgr2=0x7f1b340c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:58.820 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.821+0000 7f1b39960640 1 --2- 172.21.15.55:0/966516624 >> v2:172.21.15.84:3300/0 conn(0x7f1b340c8100 0x7f1b340c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.821+0000 7f1b39960640 1 -- 172.21.15.55:0/966516624 >> v1:172.21.15.55:6789/0 conn(0x7f1b34158870 legacy=0x7f1b340c22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.821+0000 7f1b39960640 1 -- 172.21.15.55:0/966516624 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3415eab0 con 0x7f1b340caff0 2024-03-19T01:51:58.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.821+0000 7f9633fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f963413ca40 0x7f9634073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.825+0000 7fa57559a640 1 -- 172.21.15.55:0/795312671 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5680b5070 con 0x7fa5781562e0 2024-03-19T01:51:58.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.825+0000 7f9633fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f963413ca40 0x7f9634073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:42586/0 (socket says 172.21.15.55:42586) 2024-03-19T01:51:58.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.825+0000 7f9633fff640 1 -- 172.21.15.55:0/2251594363 learned_addr learned my addr 172.21.15.55:0/2251594363 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:58.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.825+0000 7fa57559a640 1 -- 172.21.15.55:0/795312671 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5680a8990 con 0x7fa5781562e0 2024-03-19T01:51:58.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.825+0000 7fa57559a640 1 -- 172.21.15.55:0/795312671 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5680a8d60 con 0x7fa5781562e0 2024-03-19T01:51:58.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.825+0000 7fa57cfa2640 1 -- 172.21.15.55:0/795312671 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5781562e0 msgr2=0x7fa5780c7930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.825+0000 7fa57cfa2640 1 --2- 172.21.15.55:0/795312671 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5781562e0 0x7fa5780c7930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa5680097c0 tx=0x7fa56801ec90 comp rx=0 tx=0).stop 2024-03-19T01:51:58.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.825+0000 7fa57cfa2640 1 -- 172.21.15.55:0/795312671 shutdown_connections 2024-03-19T01:51:58.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.825+0000 7fa57cfa2640 1 --2- 172.21.15.55:0/795312671 >> v2:172.21.15.55:3300/0 conn(0x7fa5780c7e70 0x7fa5780cb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.825+0000 7fa57cfa2640 1 --2- 172.21.15.55:0/795312671 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5781562e0 0x7fa5780c7930 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.825+0000 7fa57cfa2640 1 -- 172.21.15.55:0/795312671 >> 172.21.15.55:0/795312671 conn(0x7fa57806f3b0 msgr2=0x7fa57806f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:58.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.825+0000 7fa57cfa2640 1 -- 172.21.15.55:0/795312671 shutdown_connections 2024-03-19T01:51:58.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.825+0000 7fa57cfa2640 1 -- 172.21.15.55:0/795312671 wait complete. 2024-03-19T01:51:58.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.825+0000 7f8807fff640 1 -- 172.21.15.55:0/3637556370 >> v2:172.21.15.55:3300/0 conn(0x7f880815a4d0 msgr2=0x7f88080c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:58.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.825+0000 7f8807fff640 1 --2- 172.21.15.55:0/3637556370 >> v2:172.21.15.55:3300/0 conn(0x7f880815a4d0 0x7f88080c7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.825+0000 7f8807fff640 1 -- 172.21.15.55:0/3637556370 >> v1:172.21.15.84:6789/0 conn(0x7f88081562e0 legacy=0x7f880814cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.825+0000 7f8807fff640 1 -- 172.21.15.55:0/3637556370 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8808156d30 con 0x7f880806fed0 2024-03-19T01:51:58.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.825+0000 7fa57cfa2640 1 Processor -- start 2024-03-19T01:51:58.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.829+0000 7fa57cfa2640 1 -- start start 2024-03-19T01:51:58.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.829+0000 7fa57cfa2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5780c7e70 0x7fa578072ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.829+0000 7fa57cfa2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa57814ccf0 0x7fa57814afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.829+0000 7fa57cfa2640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa578072560 0x7fa5781f4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.829+0000 7fa57cfa2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa57814d0d0 con 0x7fa57814ccf0 2024-03-19T01:51:58.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.829+0000 7fa57cfa2640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa578159a70 con 0x7fa578072560 2024-03-19T01:51:58.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.829+0000 7fa57cfa2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa5781508f0 con 0x7fa5780c7e70 2024-03-19T01:51:58.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.833+0000 7f9633fff640 1 -- 172.21.15.55:0/2251594363 >> v2:172.21.15.84:3300/0 conn(0x7f96340c6860 msgr2=0x7f9634156f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:58.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.833+0000 7f9633fff640 1 --2- 172.21.15.55:0/2251594363 >> v2:172.21.15.84:3300/0 conn(0x7f96340c6860 0x7f9634156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.833+0000 7f9633fff640 1 -- 172.21.15.55:0/2251594363 >> v1:172.21.15.55:6789/0 conn(0x7f96340c6400 legacy=0x7f96340ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:58.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.833+0000 7f9633fff640 1 -- 172.21.15.55:0/2251594363 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96340c41a0 con 0x7f963413ca40 2024-03-19T01:51:58.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.841+0000 7f8807fff640 1 --2- 172.21.15.55:0/3637556370 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f880806fed0 0x7f8808159f90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f88080c2cf0 tx=0x7f87f801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3364d0c0f1f7d2ce server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:58.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.841+0000 7fa57659c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5780c7e70 0x7fa578072ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.841+0000 7f9633fff640 1 --2- 172.21.15.55:0/2251594363 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f963413ca40 0x7f9634073090 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f962800bb40 tx=0x7f962801ead0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2fc5bede0002195 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:58.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.841+0000 7f1b39960640 1 --2- 172.21.15.55:0/966516624 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b340caff0 0x7f1b341619d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1b340700e0 tx=0x7f1b3001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a319b3a14f0081 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:58.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.845+0000 7fa57659c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5780c7e70 0x7fa578072ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:42590/0 (socket says 172.21.15.55:42590) 2024-03-19T01:51:58.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.845+0000 7fa57659c640 1 -- 172.21.15.55:0/3088667650 learned_addr learned my addr 172.21.15.55:0/3088667650 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:58.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.845+0000 7f13577fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f13500d5600 0x7f135004c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.845+0000 7fab2295b640 1 Processor -- start 2024-03-19T01:51:58.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.845+0000 7fab2295b640 1 -- start start 2024-03-19T01:51:58.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.845+0000 7eff66f70640 1 Processor -- start 2024-03-19T01:51:58.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.845+0000 7eff66f70640 1 -- start start 2024-03-19T01:51:58.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.845+0000 7fab2295b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fab1c1544f0 0x7fab1c073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.845+0000 7f13577fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f13500d5600 0x7f135004c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52876/0 (socket says 172.21.15.55:52876) 2024-03-19T01:51:58.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.845+0000 7f13577fe640 1 -- 172.21.15.55:0/109029442 learned_addr learned my addr 172.21.15.55:0/109029442 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:58.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.845+0000 7fab2295b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab1c0c7fa0 con 0x7fab1c15acb0 2024-03-19T01:51:58.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.845+0000 7fab2295b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab1c0c8240 con 0x7fab1c06b2a0 2024-03-19T01:51:58.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.845+0000 7fab2295b640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fab1c15b210 con 0x7fab1c1544f0 2024-03-19T01:51:58.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.845+0000 7fb2477fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fb24806c630 0x7fb2480cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.845+0000 7eff66f70640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7eff600c9a90 0x7eff60070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.845+0000 7eff66f70640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7eff601540c0 0x7eff60154550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.845+0000 7eff66f70640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff6015e3a0 con 0x7eff600c96b0 2024-03-19T01:51:58.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.845+0000 7eff66f70640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7eff600c7ad0 con 0x7eff600c9a90 2024-03-19T01:51:58.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.845+0000 7eff66f70640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7eff6015e680 con 0x7eff601540c0 2024-03-19T01:51:58.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.849+0000 7fa576d9d640 1 --2- 172.21.15.55:0/3088667650 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa578072560 0x7fa5781f4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.853+0000 7fb2477fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fb24806c630 0x7fb2480cc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:55678/0 (socket says 172.21.15.55:55678) 2024-03-19T01:51:58.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.853+0000 7fb2477fe640 1 -- 172.21.15.55:0/592315306 learned_addr learned my addr 172.21.15.55:0/592315306 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:58.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.857+0000 7fa57659c640 1 -- 172.21.15.55:0/3088667650 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa578072560 msgr2=0x7fa5781f4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.857+0000 7fa57659c640 1 --2- 172.21.15.55:0/3088667650 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa578072560 0x7fa5781f4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.857+0000 7fa57659c640 1 -- 172.21.15.55:0/3088667650 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa57814ccf0 msgr2=0x7fa57814afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.857+0000 7fa57659c640 1 --2- 172.21.15.55:0/3088667650 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa57814ccf0 0x7fa57814afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.857+0000 7fa57659c640 1 -- 172.21.15.55:0/3088667650 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa578073790 con 0x7fa5780c7e70 2024-03-19T01:51:58.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.861+0000 7fb2477fe640 1 -- 172.21.15.55:0/592315306 >> v2:172.21.15.55:3300/0 conn(0x7fb2480c3520 msgr2=0x7fb2480cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:58.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.861+0000 7fb2477fe640 1 --2- 172.21.15.55:0/592315306 >> v2:172.21.15.55:3300/0 conn(0x7fb2480c3520 0x7fb2480cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.861+0000 7fb2477fe640 1 -- 172.21.15.55:0/592315306 >> v1:172.21.15.55:6789/0 conn(0x7fb248150ae0 legacy=0x7fb2480c6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:58.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.861+0000 7fb2477fe640 1 -- 172.21.15.55:0/592315306 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb24806bc70 con 0x7fb24806c630 2024-03-19T01:51:58.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.869+0000 7eff6550d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7eff601540c0 0x7eff60154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.869+0000 7f9632ffd640 1 -- 172.21.15.55:0/2251594363 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96280b4070 con 0x7f963413ca40 2024-03-19T01:51:58.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.869+0000 7f9632ffd640 1 -- 172.21.15.55:0/2251594363 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96280a7860 con 0x7f963413ca40 2024-03-19T01:51:58.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.869+0000 7f9632ffd640 1 -- 172.21.15.55:0/2251594363 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96280a7c30 con 0x7f963413ca40 2024-03-19T01:51:58.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.869+0000 7f963a2bc640 1 -- 172.21.15.55:0/2251594363 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f963413ca40 msgr2=0x7f9634073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.869+0000 7f963a2bc640 1 --2- 172.21.15.55:0/2251594363 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f963413ca40 0x7f9634073090 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f962800bb40 tx=0x7f962801ead0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.869+0000 7f963a2bc640 1 -- 172.21.15.55:0/2251594363 shutdown_connections 2024-03-19T01:51:58.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.869+0000 7f963a2bc640 1 --2- 172.21.15.55:0/2251594363 >> v2:172.21.15.84:3300/0 conn(0x7f96340c6860 0x7f9634156f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.869+0000 7f963a2bc640 1 --2- 172.21.15.55:0/2251594363 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f963413ca40 0x7f9634073090 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.869+0000 7f963a2bc640 1 -- 172.21.15.55:0/2251594363 >> 172.21.15.55:0/2251594363 conn(0x7f96340703c0 msgr2=0x7f96340707d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:58.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.869+0000 7f963a2bc640 1 -- 172.21.15.55:0/2251594363 shutdown_connections 2024-03-19T01:51:58.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.869+0000 7f963a2bc640 1 -- 172.21.15.55:0/2251594363 wait complete. 2024-03-19T01:51:58.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.873+0000 7f963a2bc640 1 Processor -- start 2024-03-19T01:51:58.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.873+0000 7f963a2bc640 1 -- start start 2024-03-19T01:51:58.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.873+0000 7fab13fff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fab1c06b2a0 0x7fab1c15e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:33186/0 (socket says 172.21.15.55:33186) 2024-03-19T01:51:58.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.873+0000 7fab13fff640 1 -- 172.21.15.55:0/3811181705 learned_addr learned my addr 172.21.15.55:0/3811181705 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:51:58.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.873+0000 7f963a2bc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96340c6400 0x7f96341ef3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.873+0000 7f963a2bc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96340c6860 0x7f9634134110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.873+0000 7fb2477fe640 1 --2- 172.21.15.55:0/592315306 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb24806c630 0x7fb2480cc4d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb24806f6d0 tx=0x7fb24001d8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae195d8e4e37b677 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:58.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.873+0000 7f963a2bc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f963413ca40 0x7f9634134650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.873+0000 7f963a2bc640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f96340cd2a0 con 0x7f96340c6400 2024-03-19T01:51:58.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.873+0000 7f963a2bc640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f96340c4a40 con 0x7f96340c6860 2024-03-19T01:51:58.876 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.873+0000 7f963a2bc640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f96340cd420 con 0x7f963413ca40 2024-03-19T01:51:58.876 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.873+0000 7fa57659c640 1 --2- 172.21.15.55:0/3088667650 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5780c7e70 0x7fa578072ab0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa570001a60 tx=0x7fa570001e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:58.876 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.873+0000 7eff6550d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7eff601540c0 0x7eff60154550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:55712/0 (socket says 172.21.15.55:55712) 2024-03-19T01:51:58.876 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.873+0000 7eff6550d640 1 -- 172.21.15.55:0/1954751820 learned_addr learned my addr 172.21.15.55:0/1954751820 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:58.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.877+0000 7f1357fff640 1 --2- 172.21.15.55:0/109029442 >> v2:172.21.15.84:3300/0 conn(0x7f13500d5220 0x7f135004be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.893+0000 7eff6550d640 1 -- 172.21.15.55:0/1954751820 >> v2:172.21.15.55:3300/0 conn(0x7eff600c9a90 msgr2=0x7eff60070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:58.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.893+0000 7eff6550d640 1 --2- 172.21.15.55:0/1954751820 >> v2:172.21.15.55:3300/0 conn(0x7eff600c9a90 0x7eff60070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.893+0000 7eff6550d640 1 -- 172.21.15.55:0/1954751820 >> v1:172.21.15.55:6789/0 conn(0x7eff600c96b0 legacy=0x7eff600c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:58.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.893+0000 7eff6550d640 1 -- 172.21.15.55:0/1954751820 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff6015a330 con 0x7eff601540c0 2024-03-19T01:51:58.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.901+0000 7f9638859640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f963413ca40 0x7f9634134650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.901+0000 7f1357fff640 1 -- 172.21.15.55:0/109029442 >> v2:172.21.15.55:3300/0 conn(0x7f13500d5600 msgr2=0x7f135004c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:58.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.901+0000 7f1357fff640 1 --2- 172.21.15.55:0/109029442 >> v2:172.21.15.55:3300/0 conn(0x7f13500d5600 0x7f135004c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.901+0000 7f1357fff640 1 -- 172.21.15.55:0/109029442 >> v1:172.21.15.84:6789/0 conn(0x7f1350007960 legacy=0x7f1350006bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:58.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.901+0000 7f1357fff640 1 -- 172.21.15.55:0/109029442 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13500dbe80 con 0x7f13500d5220 2024-03-19T01:51:58.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.901+0000 7f9638859640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f963413ca40 0x7f9634134650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:42604/0 (socket says 172.21.15.55:42604) 2024-03-19T01:51:58.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.901+0000 7f9638859640 1 -- 172.21.15.55:0/3952056332 learned_addr learned my addr 172.21.15.55:0/3952056332 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:58.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.909+0000 7f1357fff640 1 --2- 172.21.15.55:0/109029442 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f13500d5220 0x7f135004be80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1350007e70 tx=0x7f134c01fd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd111902e6c092fe server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:58.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.909+0000 7eff6550d640 1 --2- 172.21.15.55:0/1954751820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff601540c0 0x7eff60154550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7eff60072890 tx=0x7eff5801c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d181085d9697aaea server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:58.913 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.913+0000 7f9633fff640 1 --2- 172.21.15.55:0/3952056332 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96340c6400 0x7f96341ef3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.925 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.925+0000 7f96337fe640 1 --2- 172.21.15.55:0/3952056332 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96340c6860 0x7f9634134110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.925 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.925+0000 7fab1bfff640 1 --2- 172.21.15.55:0/3811181705 >> v2:172.21.15.55:3300/0 conn(0x7fab1c1544f0 0x7fab1c073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.929+0000 7f88077fe640 1 -- 172.21.15.55:0/3637556370 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f80b1070 con 0x7f880806fed0 2024-03-19T01:51:58.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.929+0000 7f88077fe640 1 -- 172.21.15.55:0/3637556370 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87f80ac5f0 con 0x7f880806fed0 2024-03-19T01:51:58.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.929+0000 7f88077fe640 1 -- 172.21.15.55:0/3637556370 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f80ac9c0 con 0x7f880806fed0 2024-03-19T01:51:58.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.929+0000 7f880f1a0640 1 -- 172.21.15.55:0/3637556370 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f880806fed0 msgr2=0x7f8808159f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.929+0000 7f880f1a0640 1 --2- 172.21.15.55:0/3637556370 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f880806fed0 0x7f8808159f90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f88080c2cf0 tx=0x7f87f801c480 comp rx=0 tx=0).stop 2024-03-19T01:51:58.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.929+0000 7f880f1a0640 1 -- 172.21.15.55:0/3637556370 shutdown_connections 2024-03-19T01:51:58.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.929+0000 7f880f1a0640 1 --2- 172.21.15.55:0/3637556370 >> v2:172.21.15.55:3300/0 conn(0x7f880815a4d0 0x7f88080c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.929+0000 7f880f1a0640 1 --2- 172.21.15.55:0/3637556370 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f880806fed0 0x7f8808159f90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.929+0000 7f880f1a0640 1 -- 172.21.15.55:0/3637556370 >> 172.21.15.55:0/3637556370 conn(0x7f880806f3b0 msgr2=0x7f880806f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:58.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.929+0000 7f880f1a0640 1 -- 172.21.15.55:0/3637556370 shutdown_connections 2024-03-19T01:51:58.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.929+0000 7f880f1a0640 1 -- 172.21.15.55:0/3637556370 wait complete. 2024-03-19T01:51:58.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.929+0000 7fab1bfff640 1 -- 172.21.15.55:0/3811181705 >> v1:172.21.15.84:6789/0 conn(0x7fab1c06b2a0 legacy=0x7fab1c15e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.929+0000 7fab1bfff640 1 -- 172.21.15.55:0/3811181705 >> v1:172.21.15.55:6789/0 conn(0x7fab1c15acb0 legacy=0x7fab1c0cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:58.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.929+0000 7fab1bfff640 1 -- 172.21.15.55:0/3811181705 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab1c15d920 con 0x7fab1c1544f0 2024-03-19T01:51:58.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.929+0000 7f880f1a0640 1 Processor -- start 2024-03-19T01:51:58.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.929+0000 7f880f1a0640 1 -- start start 2024-03-19T01:51:58.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.933+0000 7f880f1a0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f880806fed0 0x7f8808071c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.933+0000 7f880f1a0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f880815a4d0 0x7f88081f35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.933+0000 7f880f1a0640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f880814c1e0 0x7f88081f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.933+0000 7f880f1a0640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f880815adb0 con 0x7f880806fed0 2024-03-19T01:51:58.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.933+0000 7f880f1a0640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f880815ac30 con 0x7f880814c1e0 2024-03-19T01:51:58.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.933+0000 7f880f1a0640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f880815af30 con 0x7f880815a4d0 2024-03-19T01:51:58.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.933+0000 7f9633fff640 1 -- 172.21.15.55:0/3952056332 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f963413ca40 msgr2=0x7f9634134650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.933+0000 7f9633fff640 1 --2- 172.21.15.55:0/3952056332 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f963413ca40 0x7f9634134650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.933+0000 7f9633fff640 1 -- 172.21.15.55:0/3952056332 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96340c6860 msgr2=0x7f9634134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.933+0000 7f9633fff640 1 --2- 172.21.15.55:0/3952056332 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96340c6860 0x7f9634134110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.933+0000 7f9633fff640 1 -- 172.21.15.55:0/3952056332 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96341326b0 con 0x7f96340c6400 2024-03-19T01:51:58.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.937+0000 7fab1bfff640 1 --2- 172.21.15.55:0/3811181705 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab1c1544f0 0x7fab1c073540 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fab1c0ca790 tx=0x7fab0c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b31addf68826c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:58.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.937+0000 7f9633fff640 1 --2- 172.21.15.55:0/3952056332 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96340c6400 0x7f96341ef3e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f96280ae000 tx=0x7f962801b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:58.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.941+0000 7f880cf3c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f880806fed0 0x7f8808071c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.945 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.945+0000 7f880cf3c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f880806fed0 0x7f8808071c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52926/0 (socket says 172.21.15.55:52926) 2024-03-19T01:51:58.945 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.945+0000 7f880cf3c640 1 -- 172.21.15.55:0/1480968147 learned_addr learned my addr 172.21.15.55:0/1480968147 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:58.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.945+0000 7fab1b7fe640 1 -- 172.21.15.55:0/3811181705 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0c0b5070 con 0x7fab1c1544f0 2024-03-19T01:51:58.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.945+0000 7fab1b7fe640 1 -- 172.21.15.55:0/3811181705 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab0c0a8950 con 0x7fab1c1544f0 2024-03-19T01:51:58.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.945+0000 7fab1b7fe640 1 -- 172.21.15.55:0/3811181705 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0c0a8d20 con 0x7fab1c1544f0 2024-03-19T01:51:58.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.945+0000 7fab2295b640 1 -- 172.21.15.55:0/3811181705 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab1c1544f0 msgr2=0x7fab1c073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.945+0000 7fab2295b640 1 --2- 172.21.15.55:0/3811181705 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab1c1544f0 0x7fab1c073540 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fab1c0ca790 tx=0x7fab0c01ec50 comp rx=0 tx=0).stop 2024-03-19T01:51:58.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.945+0000 7fab2295b640 1 -- 172.21.15.55:0/3811181705 shutdown_connections 2024-03-19T01:51:58.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.945+0000 7fab2295b640 1 --2- 172.21.15.55:0/3811181705 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab1c1544f0 0x7fab1c073540 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.945+0000 7fab2295b640 1 -- 172.21.15.55:0/3811181705 >> 172.21.15.55:0/3811181705 conn(0x7fab1c06e670 msgr2=0x7fab1c06ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:58.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.945+0000 7fab2295b640 1 -- 172.21.15.55:0/3811181705 shutdown_connections 2024-03-19T01:51:58.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.945+0000 7fab2295b640 1 -- 172.21.15.55:0/3811181705 wait complete. 2024-03-19T01:51:58.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.945+0000 7fab2295b640 1 Processor -- start 2024-03-19T01:51:58.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.945+0000 7fab2295b640 1 -- start start 2024-03-19T01:51:58.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.945+0000 7f1b3915f640 1 -- 172.21.15.55:0/966516624 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b300b5070 con 0x7f1b340caff0 2024-03-19T01:51:58.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.945+0000 7f1b3915f640 1 -- 172.21.15.55:0/966516624 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b300a8990 con 0x7f1b340caff0 2024-03-19T01:51:58.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f1b3915f640 1 -- 172.21.15.55:0/966516624 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b300a8da0 con 0x7f1b340caff0 2024-03-19T01:51:58.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7fab2295b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab1c06b2a0 0x7fab1c153940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7fab2295b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fab1c1544f0 0x7fab1c1f3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7fab2295b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fab1c15acb0 0x7fab1c150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f1b3b163640 1 -- 172.21.15.55:0/966516624 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b340caff0 msgr2=0x7f1b341619d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f1b3b163640 1 --2- 172.21.15.55:0/966516624 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b340caff0 0x7f1b341619d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1b340700e0 tx=0x7f1b3001ec90 comp rx=0 tx=0).stop 2024-03-19T01:51:58.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7fab2295b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fab1c0cd540 con 0x7fab1c06b2a0 2024-03-19T01:51:58.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7fab2295b640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fab1c1548d0 con 0x7fab1c1544f0 2024-03-19T01:51:58.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7fab2295b640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fab1c0cd6c0 con 0x7fab1c15acb0 2024-03-19T01:51:58.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f1b3b163640 1 -- 172.21.15.55:0/966516624 shutdown_connections 2024-03-19T01:51:58.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f1b3b163640 1 --2- 172.21.15.55:0/966516624 >> v2:172.21.15.84:3300/0 conn(0x7f1b340c8100 0x7f1b340c8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f1b3b163640 1 --2- 172.21.15.55:0/966516624 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b340caff0 0x7f1b341619d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f1b3b163640 1 -- 172.21.15.55:0/966516624 >> 172.21.15.55:0/966516624 conn(0x7f1b34067400 msgr2=0x7f1b3406b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:58.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f1b3b163640 1 -- 172.21.15.55:0/966516624 shutdown_connections 2024-03-19T01:51:58.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f1b3b163640 1 -- 172.21.15.55:0/966516624 wait complete. 2024-03-19T01:51:58.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f1b3b163640 1 Processor -- start 2024-03-19T01:51:58.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f1b3b163640 1 -- start start 2024-03-19T01:51:58.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f1b3b163640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b340c8100 0x7f1b3415da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f1b3b163640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b340caff0 0x7f1b340c3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f1b3b163640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b34158870 0x7f1b340c4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f1b3b163640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3415f2b0 con 0x7f1b34158870 2024-03-19T01:51:58.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f1b3b163640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f1b340ca010 con 0x7f1b340c8100 2024-03-19T01:51:58.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f1b3b163640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1b340c99a0 con 0x7f1b340caff0 2024-03-19T01:51:58.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f880cf3c640 1 -- 172.21.15.55:0/1480968147 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f880815a4d0 msgr2=0x7f88081f35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f880cf3c640 1 --2- 172.21.15.55:0/1480968147 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f880815a4d0 0x7f88081f35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f880cf3c640 1 -- 172.21.15.55:0/1480968147 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f880814c1e0 msgr2=0x7f88081f3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f880cf3c640 1 --2- 172.21.15.55:0/1480968147 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f880814c1e0 0x7f88081f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.949+0000 7f880cf3c640 1 -- 172.21.15.55:0/1480968147 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88081401d0 con 0x7f880806fed0 2024-03-19T01:51:58.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.953+0000 7f9630ff9640 1 -- 172.21.15.55:0/3952056332 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96280be070 con 0x7f96340c6400 2024-03-19T01:51:58.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.953+0000 7f9630ff9640 1 -- 172.21.15.55:0/3952056332 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96280af850 con 0x7f96340c6400 2024-03-19T01:51:58.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.953+0000 7f963a2bc640 1 -- 172.21.15.55:0/3952056332 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9634132990 con 0x7f96340c6400 2024-03-19T01:51:58.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.953+0000 7f9630ff9640 1 -- 172.21.15.55:0/3952056332 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96280b4040 con 0x7f96340c6400 2024-03-19T01:51:58.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.953+0000 7f963a2bc640 1 -- 172.21.15.55:0/3952056332 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9634132f50 con 0x7f96340c6400 2024-03-19T01:51:58.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.953+0000 7f963a2bc640 1 -- 172.21.15.55:0/3952056332 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9634068a40 con 0x7f96340c6400 2024-03-19T01:51:58.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.957+0000 7f880cf3c640 1 --2- 172.21.15.55:0/1480968147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f880806fed0 0x7f8808071c80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f87fc00c450 tx=0x7f87fc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:58.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.957+0000 7f1b3a962640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b34158870 0x7f1b340c4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.961+0000 7fb245ffb640 1 -- 172.21.15.55:0/592315306 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2400a6600 con 0x7fb24806c630 2024-03-19T01:51:58.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.961+0000 7fb245ffb640 1 -- 172.21.15.55:0/592315306 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2400a67a0 con 0x7fb24806c630 2024-03-19T01:51:58.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.961+0000 7fb245ffb640 1 -- 172.21.15.55:0/592315306 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2400a6b70 con 0x7fb24806c630 2024-03-19T01:51:58.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.961+0000 7fb24d963640 1 -- 172.21.15.55:0/592315306 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb24806c630 msgr2=0x7fb2480cc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.961+0000 7fb24d963640 1 --2- 172.21.15.55:0/592315306 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb24806c630 0x7fb2480cc4d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb24806f6d0 tx=0x7fb24001d8e0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.961+0000 7fb24d963640 1 -- 172.21.15.55:0/592315306 shutdown_connections 2024-03-19T01:51:58.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.961+0000 7fb24d963640 1 --2- 172.21.15.55:0/592315306 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb24806c630 0x7fb2480cc4d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.961+0000 7fb24d963640 1 --2- 172.21.15.55:0/592315306 >> v2:172.21.15.55:3300/0 conn(0x7fb2480c3520 0x7fb2480cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.961+0000 7fb24d963640 1 -- 172.21.15.55:0/592315306 >> 172.21.15.55:0/592315306 conn(0x7fb24806e5a0 msgr2=0x7fb248072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:58.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.961+0000 7fb24d963640 1 -- 172.21.15.55:0/592315306 shutdown_connections 2024-03-19T01:51:58.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.961+0000 7fb24d963640 1 -- 172.21.15.55:0/592315306 wait complete. 2024-03-19T01:51:58.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.961+0000 7fb24d963640 1 Processor -- start 2024-03-19T01:51:58.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.961+0000 7fb24d963640 1 -- start start 2024-03-19T01:51:58.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.965+0000 7fb24d963640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb24806c630 0x7fb248141450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.965+0000 7fb24d963640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb2480c3520 0x7fb2481f1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.965+0000 7fb24d963640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb248150ae0 0x7fb248153950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.965+0000 7fb24d963640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb2480c6ba0 con 0x7fb248150ae0 2024-03-19T01:51:58.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.965+0000 7fb24d963640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb2480c6a20 con 0x7fb2480c3520 2024-03-19T01:51:58.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.965+0000 7fb24d963640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fb2480c8690 con 0x7fb24806c630 2024-03-19T01:51:58.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.965+0000 7f1b3a962640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b34158870 0x7f1b340c4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52958/0 (socket says 172.21.15.55:52958) 2024-03-19T01:51:58.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.965+0000 7f1b3a962640 1 -- 172.21.15.55:0/1030693805 learned_addr learned my addr 172.21.15.55:0/1030693805 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:58.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.969+0000 7f88057fa640 1 -- 172.21.15.55:0/1480968147 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc017070 con 0x7f880806fed0 2024-03-19T01:51:58.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.969+0000 7f88057fa640 1 -- 172.21.15.55:0/1480968147 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87fc005cf0 con 0x7f880806fed0 2024-03-19T01:51:58.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.969+0000 7f88057fa640 1 -- 172.21.15.55:0/1480968147 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc0072f0 con 0x7f880806fed0 2024-03-19T01:51:58.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.969+0000 7f880f1a0640 1 -- 172.21.15.55:0/1480968147 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f880814a650 con 0x7f880806fed0 2024-03-19T01:51:58.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.969+0000 7f880f1a0640 1 -- 172.21.15.55:0/1480968147 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88080726a0 con 0x7f880806fed0 2024-03-19T01:51:58.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.969+0000 7fab1bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab1c06b2a0 0x7fab1c153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.969+0000 7f880f1a0640 1 -- 172.21.15.55:0/1480968147 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8808068330 con 0x7f880806fed0 2024-03-19T01:51:58.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.973+0000 7f1356ffd640 1 -- 172.21.15.55:0/109029442 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134c0b6070 con 0x7f13500d5220 2024-03-19T01:51:58.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.973+0000 7f1b3a962640 1 -- 172.21.15.55:0/1030693805 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b340caff0 msgr2=0x7f1b340c3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.973+0000 7f1b3a962640 1 --2- 172.21.15.55:0/1030693805 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b340caff0 0x7f1b340c3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.973+0000 7f1356ffd640 1 -- 172.21.15.55:0/109029442 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f134c0a9a40 con 0x7f13500d5220 2024-03-19T01:51:58.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.973+0000 7f1356ffd640 1 -- 172.21.15.55:0/109029442 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134c0a9e10 con 0x7f13500d5220 2024-03-19T01:51:58.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.973+0000 7f1b3a962640 1 -- 172.21.15.55:0/1030693805 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b340c8100 msgr2=0x7f1b3415da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.973+0000 7f1b3a962640 1 --2- 172.21.15.55:0/1030693805 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b340c8100 0x7f1b3415da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.973+0000 7f1b3a962640 1 -- 172.21.15.55:0/1030693805 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3415c9b0 con 0x7f1b34158870 2024-03-19T01:51:58.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.973+0000 7f135d0ce640 1 -- 172.21.15.55:0/109029442 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f13500d5220 msgr2=0x7f135004be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.973+0000 7f135d0ce640 1 --2- 172.21.15.55:0/109029442 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f13500d5220 0x7f135004be80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1350007e70 tx=0x7f134c01fd40 comp rx=0 tx=0).stop 2024-03-19T01:51:58.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.973+0000 7f135d0ce640 1 -- 172.21.15.55:0/109029442 shutdown_connections 2024-03-19T01:51:58.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.973+0000 7f135d0ce640 1 --2- 172.21.15.55:0/109029442 >> v2:172.21.15.55:3300/0 conn(0x7f13500d5600 0x7f135004c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.973+0000 7f135d0ce640 1 --2- 172.21.15.55:0/109029442 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f13500d5220 0x7f135004be80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.973+0000 7f135d0ce640 1 -- 172.21.15.55:0/109029442 >> 172.21.15.55:0/109029442 conn(0x7f13500505e0 msgr2=0x7f13500509f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:58.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.973+0000 7f135d0ce640 1 -- 172.21.15.55:0/109029442 shutdown_connections 2024-03-19T01:51:58.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.973+0000 7f135d0ce640 1 -- 172.21.15.55:0/109029442 wait complete. 2024-03-19T01:51:58.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.973+0000 7fab1bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab1c06b2a0 0x7fab1c153940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52942/0 (socket says 172.21.15.55:52942) 2024-03-19T01:51:58.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.973+0000 7fab1bfff640 1 -- 172.21.15.55:0/3996690030 learned_addr learned my addr 172.21.15.55:0/3996690030 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:58.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.973+0000 7f135d0ce640 1 Processor -- start 2024-03-19T01:51:58.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.973+0000 7f135d0ce640 1 -- start start 2024-03-19T01:51:58.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.977+0000 7f135d0ce640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1350007960 0x7f13500317f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.977+0000 7f135d0ce640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f13500d5220 0x7f1350175550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.977+0000 7f135d0ce640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13500304c0 0x7f1350034f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.977+0000 7f135d0ce640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f13500dbb40 con 0x7f13500304c0 2024-03-19T01:51:58.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.977+0000 7f135d0ce640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f135004b680 con 0x7f13500d5220 2024-03-19T01:51:58.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.977+0000 7f135d0ce640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f13500072e0 con 0x7f1350007960 2024-03-19T01:51:58.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.981+0000 7fab13fff640 1 --2- 172.21.15.55:0/3996690030 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fab1c1544f0 0x7fab1c1f3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.989+0000 7fab13fff640 1 -- 172.21.15.55:0/3996690030 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fab1c15acb0 msgr2=0x7fab1c150230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.989+0000 7fab13fff640 1 --2- 172.21.15.55:0/3996690030 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fab1c15acb0 0x7fab1c150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.989+0000 7fab13fff640 1 -- 172.21.15.55:0/3996690030 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab1c06b2a0 msgr2=0x7fab1c153940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:58.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.989+0000 7fab13fff640 1 --2- 172.21.15.55:0/3996690030 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab1c06b2a0 0x7fab1c153940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:58.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.989+0000 7fab13fff640 1 -- 172.21.15.55:0/3996690030 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab1c14eee0 con 0x7fab1c1544f0 2024-03-19T01:51:58.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.993+0000 7fab13fff640 1 --2- 172.21.15.55:0/3996690030 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fab1c1544f0 0x7fab1c1f3720 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fab0400c430 tx=0x7fab0400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:58.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.997+0000 7f13577fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f13500d5220 0x7f1350175550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.997 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.997+0000 7f88057fa640 1 -- 172.21.15.55:0/1480968147 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f87fc01a070 con 0x7f880806fed0 2024-03-19T01:51:58.997 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.997+0000 7fb2467fc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb2480c3520 0x7fb2481f1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.997+0000 7f88057fa640 1 --2- 172.21.15.55:0/1480968147 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f87f40676b0 0x7f87f4069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:58.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.997+0000 7f8807fff640 1 --2- 172.21.15.55:0/1480968147 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f87f40676b0 0x7f87f4069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:58.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.997+0000 7fb2467fc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb2480c3520 0x7fb2481f1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:55772/0 (socket says 172.21.15.55:55772) 2024-03-19T01:51:58.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.997+0000 7fb2467fc640 1 -- 172.21.15.55:0/3883317258 learned_addr learned my addr 172.21.15.55:0/3883317258 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:58.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:58.997+0000 7f8807fff640 1 --2- 172.21.15.55:0/1480968147 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f87f40676b0 0x7f87f4069b70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f87f80ae000 tx=0x7f87f8004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.001+0000 7f88057fa640 1 -- 172.21.15.55:0/1480968147 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f87fc053a30 con 0x7f880806fed0 2024-03-19T01:51:59.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.001+0000 7f88057fa640 1 --2- 172.21.15.55:0/1480968147 >> [v2:172.21.15.55:6816/543460007,v1:172.21.15.55:6817/543460007] conn(0x7f87f4070d20 0x7f87f4073160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.001+0000 7f88057fa640 1 -- 172.21.15.55:0/1480968147 --> [v2:172.21.15.55:6816/543460007,v1:172.21.15.55:6817/543460007] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f87f4073830 con 0x7f87f4070d20 2024-03-19T01:51:59.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.001+0000 7f88057fa640 1 -- 172.21.15.55:0/1480968147 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f87fc051c30 con 0x7f880806fed0 2024-03-19T01:51:59.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.001+0000 7f880d73d640 1 --2- 172.21.15.55:0/1480968147 >> [v2:172.21.15.55:6816/543460007,v1:172.21.15.55:6817/543460007] conn(0x7f87f4070d20 0x7f87f4073160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.001+0000 7f880d73d640 1 --2- 172.21.15.55:0/1480968147 >> [v2:172.21.15.55:6816/543460007,v1:172.21.15.55:6817/543460007] conn(0x7f87f4070d20 0x7f87f4073160 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.001+0000 7f1b3a962640 1 --2- 172.21.15.55:0/1030693805 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b34158870 0x7f1b340c4210 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00b970 tx=0x7f1b2c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.003 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.001+0000 7f88057fa640 1 -- 172.21.15.55:0/1480968147 <== osd.1 v2:172.21.15.55:6816/543460007 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f87f4073830 con 0x7f87f4070d20 2024-03-19T01:51:59.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.001+0000 7fa566ffd640 1 -- 172.21.15.55:0/3088667650 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa570018070 con 0x7fa5780c7e70 2024-03-19T01:51:59.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.001+0000 7fa566ffd640 1 -- 172.21.15.55:0/3088667650 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5700020b0 con 0x7fa5780c7e70 2024-03-19T01:51:59.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.001+0000 7fa566ffd640 1 -- 172.21.15.55:0/3088667650 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa57000d8e0 con 0x7fa5780c7e70 2024-03-19T01:51:59.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.001+0000 7fa57cfa2640 1 -- 172.21.15.55:0/3088667650 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5781560e0 con 0x7fa5780c7e70 2024-03-19T01:51:59.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.001+0000 7fa57cfa2640 1 -- 172.21.15.55:0/3088667650 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5780731d0 con 0x7fa5780c7e70 2024-03-19T01:51:59.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.005+0000 7fa57cfa2640 1 -- 172.21.15.55:0/3088667650 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa5780cb670 con 0x7fa5780c7e70 2024-03-19T01:51:59.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.005+0000 7fb246ffd640 1 --2- 172.21.15.55:0/3883317258 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb24806c630 0x7fb248141450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.005+0000 7f1357fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1350007960 0x7f13500317f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.005+0000 7f1357fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1350007960 0x7f13500317f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:42634/0 (socket says 172.21.15.55:42634) 2024-03-19T01:51:59.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.005+0000 7f1357fff640 1 -- 172.21.15.55:0/300527546 learned_addr learned my addr 172.21.15.55:0/300527546 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:59.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.013+0000 7f1357fff640 1 -- 172.21.15.55:0/300527546 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f13500d5220 msgr2=0x7f1350175550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.013+0000 7f1357fff640 1 --2- 172.21.15.55:0/300527546 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f13500d5220 0x7f1350175550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.013+0000 7f1357fff640 1 -- 172.21.15.55:0/300527546 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13500304c0 msgr2=0x7f1350034f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.013+0000 7f1357fff640 1 --2- 172.21.15.55:0/300527546 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13500304c0 0x7f1350034f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.013+0000 7f1357fff640 1 -- 172.21.15.55:0/300527546 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1350031e80 con 0x7f1350007960 2024-03-19T01:51:59.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.013+0000 7fb2477fe640 1 --2- 172.21.15.55:0/3883317258 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb248150ae0 0x7fb248153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.017+0000 7f880f1a0640 1 -- 172.21.15.55:0/1480968147 --> [v2:172.21.15.55:6816/543460007,v1:172.21.15.55:6817/543460007] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f88080c7710 con 0x7f87f4070d20 2024-03-19T01:51:59.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.017+0000 7f88057fa640 1 -- 172.21.15.55:0/1480968147 <== osd.1 v2:172.21.15.55:6816/543460007 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f88080c7710 con 0x7f87f4070d20 2024-03-19T01:51:59.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.017+0000 7f880f1a0640 1 -- 172.21.15.55:0/1480968147 >> [v2:172.21.15.55:6816/543460007,v1:172.21.15.55:6817/543460007] conn(0x7f87f4070d20 msgr2=0x7f87f4073160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.017+0000 7f880f1a0640 1 --2- 172.21.15.55:0/1480968147 >> [v2:172.21.15.55:6816/543460007,v1:172.21.15.55:6817/543460007] conn(0x7f87f4070d20 0x7f87f4073160 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.017+0000 7f880f1a0640 1 -- 172.21.15.55:0/1480968147 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f87f40676b0 msgr2=0x7f87f4069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.017+0000 7f880f1a0640 1 --2- 172.21.15.55:0/1480968147 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f87f40676b0 0x7f87f4069b70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f87f80ae000 tx=0x7f87f8004740 comp rx=0 tx=0).stop 2024-03-19T01:51:59.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.017+0000 7f880f1a0640 1 -- 172.21.15.55:0/1480968147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f880806fed0 msgr2=0x7f8808071c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.017+0000 7f880f1a0640 1 --2- 172.21.15.55:0/1480968147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f880806fed0 0x7f8808071c80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f87fc00c450 tx=0x7f87fc00c910 comp rx=0 tx=0).stop 2024-03-19T01:51:59.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.017+0000 7f880d73d640 1 -- 172.21.15.55:0/1480968147 reap_dead start 2024-03-19T01:51:59.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.017+0000 7f880f1a0640 1 -- 172.21.15.55:0/1480968147 shutdown_connections 2024-03-19T01:51:59.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.017+0000 7f880f1a0640 1 -- 172.21.15.55:0/1480968147 >> 172.21.15.55:0/1480968147 conn(0x7f880806f3b0 msgr2=0x7f88080c4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:59.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.017+0000 7f880f1a0640 1 -- 172.21.15.55:0/1480968147 shutdown_connections 2024-03-19T01:51:59.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.017+0000 7f880f1a0640 1 -- 172.21.15.55:0/1480968147 wait complete. 2024-03-19T01:51:59.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.021+0000 7fb246ffd640 1 -- 172.21.15.55:0/3883317258 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb2480c3520 msgr2=0x7fb2481f1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.021+0000 7fb246ffd640 1 --2- 172.21.15.55:0/3883317258 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb2480c3520 0x7fb2481f1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.021+0000 7fb246ffd640 1 -- 172.21.15.55:0/3883317258 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb248150ae0 msgr2=0x7fb248153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.021+0000 7fb246ffd640 1 --2- 172.21.15.55:0/3883317258 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb248150ae0 0x7fb248153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.021+0000 7fb246ffd640 1 -- 172.21.15.55:0/3883317258 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb248141990 con 0x7fb24806c630 2024-03-19T01:51:59.033 INFO:teuthology.orchestra.run.smithi055.stdout:64424509465 2024-03-19T01:51:59.033 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-19T01:51:59.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.033+0000 7f9630ff9640 1 -- 172.21.15.55:0/3952056332 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f96280c1070 con 0x7f96340c6400 2024-03-19T01:51:59.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.033+0000 7f9630ff9640 1 --2- 172.21.15.55:0/3952056332 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f96200676e0 0x7f9620069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.033+0000 7f96337fe640 1 --2- 172.21.15.55:0/3952056332 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f96200676e0 0x7f9620069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.033+0000 7f96337fe640 1 --2- 172.21.15.55:0/3952056332 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f96200676e0 0x7f9620069ba0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f96340c8a40 tx=0x7f9624009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.037+0000 7f9630ff9640 1 -- 172.21.15.55:0/3952056332 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f96280f1320 con 0x7f96340c6400 2024-03-19T01:51:59.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.037+0000 7f9630ff9640 1 --2- 172.21.15.55:0/3952056332 >> [v2:172.21.15.84:6802/460717295,v1:172.21.15.84:6803/460717295] conn(0x7f9620070de0 0x7f9620073220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.037+0000 7f9630ff9640 1 -- 172.21.15.55:0/3952056332 --> [v2:172.21.15.84:6802/460717295,v1:172.21.15.84:6803/460717295] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f96280b0e50 con 0x7f9620070de0 2024-03-19T01:51:59.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.037+0000 7f9630ff9640 1 -- 172.21.15.55:0/3952056332 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f96280ef520 con 0x7f96340c6400 2024-03-19T01:51:59.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.037+0000 7f9638859640 1 --2- 172.21.15.55:0/3952056332 >> [v2:172.21.15.84:6802/460717295,v1:172.21.15.84:6803/460717295] conn(0x7f9620070de0 0x7f9620073220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.037+0000 7f9638859640 1 --2- 172.21.15.55:0/3952056332 >> [v2:172.21.15.84:6802/460717295,v1:172.21.15.84:6803/460717295] conn(0x7f9620070de0 0x7f9620073220 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.037+0000 7fb246ffd640 1 --2- 172.21.15.55:0/3883317258 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb24806c630 0x7fb248141450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb23c00b440 tx=0x7fb23c00b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.037+0000 7f1b2affd640 1 -- 172.21.15.55:0/1030693805 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c019070 con 0x7f1b34158870 2024-03-19T01:51:59.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.037+0000 7f1b2affd640 1 -- 172.21.15.55:0/1030693805 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b2c00cb60 con 0x7f1b34158870 2024-03-19T01:51:59.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.037+0000 7f1b3b163640 1 -- 172.21.15.55:0/1030693805 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b340c4750 con 0x7f1b34158870 2024-03-19T01:51:59.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.037+0000 7f1b2affd640 1 -- 172.21.15.55:0/1030693805 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c014440 con 0x7f1b34158870 2024-03-19T01:51:59.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.037+0000 7f1b3b163640 1 -- 172.21.15.55:0/1030693805 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b3415cc50 con 0x7f1b34158870 2024-03-19T01:51:59.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.037+0000 7f9630ff9640 1 -- 172.21.15.55:0/3952056332 <== osd.5 v2:172.21.15.84:6802/460717295 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96280b0e50 con 0x7f9620070de0 2024-03-19T01:51:59.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.037+0000 7f1b3b163640 1 -- 172.21.15.55:0/1030693805 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1b341f4580 con 0x7f1b34158870 2024-03-19T01:51:59.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.041+0000 7f1357fff640 1 --2- 172.21.15.55:0/300527546 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1350007960 0x7f13500317f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f134c0b4040 tx=0x7f134c01cae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.045+0000 7fb22ffff640 1 -- 172.21.15.55:0/3883317258 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23c015020 con 0x7fb24806c630 2024-03-19T01:51:59.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.045+0000 7fb22ffff640 1 -- 172.21.15.55:0/3883317258 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb23c011850 con 0x7fb24806c630 2024-03-19T01:51:59.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.045+0000 7fb24d963640 1 -- 172.21.15.55:0/3883317258 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2481f1cc0 con 0x7fb24806c630 2024-03-19T01:51:59.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.045+0000 7fb24d963640 1 -- 172.21.15.55:0/3883317258 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb248143ab0 con 0x7fb24806c630 2024-03-19T01:51:59.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.045+0000 7fb22ffff640 1 -- 172.21.15.55:0/3883317258 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23c011c40 con 0x7fb24806c630 2024-03-19T01:51:59.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.045+0000 7fb24d963640 1 -- 172.21.15.55:0/3883317258 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb228001040 con 0x7fb24806c630 2024-03-19T01:51:59.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.053+0000 7f963a2bc640 1 -- 172.21.15.55:0/3952056332 --> [v2:172.21.15.84:6802/460717295,v1:172.21.15.84:6803/460717295] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9634134b90 con 0x7f9620070de0 2024-03-19T01:51:59.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.057+0000 7f9630ff9640 1 -- 172.21.15.55:0/3952056332 <== osd.5 v2:172.21.15.84:6802/460717295 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9634134b90 con 0x7f9620070de0 2024-03-19T01:51:59.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.057+0000 7f963a2bc640 1 -- 172.21.15.55:0/3952056332 >> [v2:172.21.15.84:6802/460717295,v1:172.21.15.84:6803/460717295] conn(0x7f9620070de0 msgr2=0x7f9620073220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.057+0000 7f963a2bc640 1 --2- 172.21.15.55:0/3952056332 >> [v2:172.21.15.84:6802/460717295,v1:172.21.15.84:6803/460717295] conn(0x7f9620070de0 0x7f9620073220 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.057+0000 7f963a2bc640 1 -- 172.21.15.55:0/3952056332 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f96200676e0 msgr2=0x7f9620069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.057+0000 7f963a2bc640 1 --2- 172.21.15.55:0/3952056332 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f96200676e0 0x7f9620069ba0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f96340c8a40 tx=0x7f9624009040 comp rx=0 tx=0).stop 2024-03-19T01:51:59.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.057+0000 7f963a2bc640 1 -- 172.21.15.55:0/3952056332 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96340c6400 msgr2=0x7f96341ef3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.057+0000 7f963a2bc640 1 --2- 172.21.15.55:0/3952056332 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96340c6400 0x7f96341ef3e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f96280ae000 tx=0x7f962801b870 comp rx=0 tx=0).stop 2024-03-19T01:51:59.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.057+0000 7f9638859640 1 -- 172.21.15.55:0/3952056332 reap_dead start 2024-03-19T01:51:59.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.057+0000 7f963a2bc640 1 -- 172.21.15.55:0/3952056332 shutdown_connections 2024-03-19T01:51:59.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.057+0000 7f963a2bc640 1 -- 172.21.15.55:0/3952056332 >> 172.21.15.55:0/3952056332 conn(0x7f96340703c0 msgr2=0x7f963413aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:59.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.057+0000 7f963a2bc640 1 -- 172.21.15.55:0/3952056332 shutdown_connections 2024-03-19T01:51:59.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.057+0000 7f963a2bc640 1 -- 172.21.15.55:0/3952056332 wait complete. 2024-03-19T01:51:59.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.057+0000 7f1354ff9640 1 -- 172.21.15.55:0/300527546 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134c0c1070 con 0x7f1350007960 2024-03-19T01:51:59.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.057+0000 7f1354ff9640 1 -- 172.21.15.55:0/300527546 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f134c0b2df0 con 0x7f1350007960 2024-03-19T01:51:59.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.057+0000 7f135d0ce640 1 -- 172.21.15.55:0/300527546 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13500358d0 con 0x7f1350007960 2024-03-19T01:51:59.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.057+0000 7f1354ff9640 1 -- 172.21.15.55:0/300527546 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134c0b6070 con 0x7f1350007960 2024-03-19T01:51:59.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.057+0000 7f135d0ce640 1 -- 172.21.15.55:0/300527546 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1350031070 con 0x7f1350007960 2024-03-19T01:51:59.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.061+0000 7f135d0ce640 1 -- 172.21.15.55:0/300527546 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1350005720 con 0x7f1350007960 2024-03-19T01:51:59.074 INFO:teuthology.orchestra.run.smithi055.stdout:60129542167 2024-03-19T01:51:59.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.073+0000 7f1b2affd640 1 -- 172.21.15.55:0/1030693805 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1b2c01c070 con 0x7f1b34158870 2024-03-19T01:51:59.075 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-19T01:51:59.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.073+0000 7f1b2affd640 1 --2- 172.21.15.55:0/1030693805 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1b180676b0 0x7f1b18069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.073+0000 7f1b3a161640 1 --2- 172.21.15.55:0/1030693805 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1b180676b0 0x7f1b18069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.073+0000 7f1b3a161640 1 --2- 172.21.15.55:0/1030693805 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1b180676b0 0x7f1b18069b70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1b24001600 tx=0x7f1b24007a50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.085+0000 7f1b2affd640 1 -- 172.21.15.55:0/1030693805 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f1b2c054390 con 0x7f1b34158870 2024-03-19T01:51:59.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.085+0000 7f1b2affd640 1 --2- 172.21.15.55:0/1030693805 >> [v2:172.21.15.84:6810/460083528,v1:172.21.15.84:6811/460083528] conn(0x7f1b18070d30 0x7f1b18073170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.085+0000 7f1b2affd640 1 -- 172.21.15.55:0/1030693805 --> [v2:172.21.15.84:6810/460083528,v1:172.21.15.84:6811/460083528] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1b2c00eea0 con 0x7f1b18070d30 2024-03-19T01:51:59.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.085+0000 7f1b2affd640 1 -- 172.21.15.55:0/1030693805 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1b2c052590 con 0x7f1b34158870 2024-03-19T01:51:59.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.085+0000 7f1b39960640 1 --2- 172.21.15.55:0/1030693805 >> [v2:172.21.15.84:6810/460083528,v1:172.21.15.84:6811/460083528] conn(0x7f1b18070d30 0x7f1b18073170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.085+0000 7f1b39960640 1 --2- 172.21.15.55:0/1030693805 >> [v2:172.21.15.84:6810/460083528,v1:172.21.15.84:6811/460083528] conn(0x7f1b18070d30 0x7f1b18073170 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.085+0000 7f1b2affd640 1 -- 172.21.15.55:0/1030693805 <== osd.7 v2:172.21.15.84:6810/460083528 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1b2c00eea0 con 0x7f1b18070d30 2024-03-19T01:51:59.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.089+0000 7fa566ffd640 1 -- 172.21.15.55:0/3088667650 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa57001b070 con 0x7fa5780c7e70 2024-03-19T01:51:59.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.089+0000 7fa566ffd640 1 --2- 172.21.15.55:0/3088667650 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa5580676b0 0x7fa558069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.089+0000 7fa575d9b640 1 --2- 172.21.15.55:0/3088667650 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa5580676b0 0x7fa558069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.089+0000 7fa575d9b640 1 --2- 172.21.15.55:0/3088667650 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa5580676b0 0x7fa558069b70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa5680b2000 tx=0x7fa568004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.101+0000 7f1b3b163640 1 -- 172.21.15.55:0/1030693805 --> [v2:172.21.15.84:6810/460083528,v1:172.21.15.84:6811/460083528] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1b3404f590 con 0x7f1b18070d30 2024-03-19T01:51:59.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.101+0000 7f1b2affd640 1 -- 172.21.15.55:0/1030693805 <== osd.7 v2:172.21.15.84:6810/460083528 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1b3404f590 con 0x7f1b18070d30 2024-03-19T01:51:59.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.101+0000 7f1b3b163640 1 -- 172.21.15.55:0/1030693805 >> [v2:172.21.15.84:6810/460083528,v1:172.21.15.84:6811/460083528] conn(0x7f1b18070d30 msgr2=0x7f1b18073170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.101+0000 7f1b3b163640 1 --2- 172.21.15.55:0/1030693805 >> [v2:172.21.15.84:6810/460083528,v1:172.21.15.84:6811/460083528] conn(0x7f1b18070d30 0x7f1b18073170 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.101+0000 7f1b3b163640 1 -- 172.21.15.55:0/1030693805 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1b180676b0 msgr2=0x7f1b18069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.101+0000 7f1b3b163640 1 --2- 172.21.15.55:0/1030693805 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1b180676b0 0x7f1b18069b70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1b24001600 tx=0x7f1b24007a50 comp rx=0 tx=0).stop 2024-03-19T01:51:59.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.101+0000 7f1b3b163640 1 -- 172.21.15.55:0/1030693805 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b34158870 msgr2=0x7f1b340c4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.101+0000 7f1b3b163640 1 --2- 172.21.15.55:0/1030693805 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b34158870 0x7f1b340c4210 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00b970 tx=0x7f1b2c00be30 comp rx=0 tx=0).stop 2024-03-19T01:51:59.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.101+0000 7f1b3a962640 1 -- 172.21.15.55:0/1030693805 reap_dead start 2024-03-19T01:51:59.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.101+0000 7f1b3b163640 1 -- 172.21.15.55:0/1030693805 shutdown_connections 2024-03-19T01:51:59.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.101+0000 7f1b3b163640 1 -- 172.21.15.55:0/1030693805 >> 172.21.15.55:0/1030693805 conn(0x7f1b34067400 msgr2=0x7f1b341621c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:59.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.101+0000 7f1b3b163640 1 -- 172.21.15.55:0/1030693805 shutdown_connections 2024-03-19T01:51:59.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.101+0000 7f1b3b163640 1 -- 172.21.15.55:0/1030693805 wait complete. 2024-03-19T01:51:59.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.109+0000 7eff5f7fe640 1 -- 172.21.15.55:0/1954751820 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff580a56d0 con 0x7eff601540c0 2024-03-19T01:51:59.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.109+0000 7eff5f7fe640 1 -- 172.21.15.55:0/1954751820 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff580a5870 con 0x7eff601540c0 2024-03-19T01:51:59.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.109+0000 7eff5f7fe640 1 -- 172.21.15.55:0/1954751820 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff580a5c40 con 0x7eff601540c0 2024-03-19T01:51:59.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.109+0000 7fa566ffd640 1 -- 172.21.15.55:0/3088667650 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fa570053de0 con 0x7fa5780c7e70 2024-03-19T01:51:59.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.109+0000 7fa566ffd640 1 --2- 172.21.15.55:0/3088667650 >> [v2:172.21.15.55:6824/2757675043,v1:172.21.15.55:6825/2757675043] conn(0x7fa558070d30 0x7fa558073170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.109+0000 7fa566ffd640 1 -- 172.21.15.55:0/3088667650 --> [v2:172.21.15.55:6824/2757675043,v1:172.21.15.55:6825/2757675043] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa558073840 con 0x7fa558070d30 2024-03-19T01:51:59.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.109+0000 7eff66f70640 1 -- 172.21.15.55:0/1954751820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff601540c0 msgr2=0x7eff60154550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.109+0000 7eff66f70640 1 --2- 172.21.15.55:0/1954751820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff601540c0 0x7eff60154550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7eff60072890 tx=0x7eff5801c920 comp rx=0 tx=0).stop 2024-03-19T01:51:59.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.109+0000 7eff66f70640 1 -- 172.21.15.55:0/1954751820 shutdown_connections 2024-03-19T01:51:59.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.109+0000 7eff66f70640 1 --2- 172.21.15.55:0/1954751820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff601540c0 0x7eff60154550 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.109+0000 7eff66f70640 1 --2- 172.21.15.55:0/1954751820 >> v2:172.21.15.55:3300/0 conn(0x7eff600c9a90 0x7eff60070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.109+0000 7eff66f70640 1 -- 172.21.15.55:0/1954751820 >> 172.21.15.55:0/1954751820 conn(0x7eff600cc9c0 msgr2=0x7eff600ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:59.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.113+0000 7fab197fa640 1 -- 172.21.15.55:0/3996690030 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab04017070 con 0x7fab1c1544f0 2024-03-19T01:51:59.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.113+0000 7eff66f70640 1 -- 172.21.15.55:0/1954751820 shutdown_connections 2024-03-19T01:51:59.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.113+0000 7fa576d9d640 1 --2- 172.21.15.55:0/3088667650 >> [v2:172.21.15.55:6824/2757675043,v1:172.21.15.55:6825/2757675043] conn(0x7fa558070d30 0x7fa558073170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.113+0000 7fab197fa640 1 -- 172.21.15.55:0/3996690030 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab04005cf0 con 0x7fab1c1544f0 2024-03-19T01:51:59.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.113+0000 7fab2295b640 1 -- 172.21.15.55:0/3996690030 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab1c1529f0 con 0x7fab1c1544f0 2024-03-19T01:51:59.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.113+0000 7fab2295b640 1 -- 172.21.15.55:0/3996690030 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab1c152fb0 con 0x7fab1c1544f0 2024-03-19T01:51:59.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.113+0000 7fab197fa640 1 -- 172.21.15.55:0/3996690030 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab040072d0 con 0x7fab1c1544f0 2024-03-19T01:51:59.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.113+0000 7fab2295b640 1 -- 172.21.15.55:0/3996690030 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fab00001040 con 0x7fab1c1544f0 2024-03-19T01:51:59.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.113+0000 7eff66f70640 1 -- 172.21.15.55:0/1954751820 wait complete. 2024-03-19T01:51:59.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.113+0000 7eff66f70640 1 Processor -- start 2024-03-19T01:51:59.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.113+0000 7eff66f70640 1 -- start start 2024-03-19T01:51:59.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.113+0000 7fa576d9d640 1 --2- 172.21.15.55:0/3088667650 >> [v2:172.21.15.55:6824/2757675043,v1:172.21.15.55:6825/2757675043] conn(0x7fa558070d30 0x7fa558073170 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.113+0000 7eff66f70640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff600c96b0 0x7eff601522f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.113+0000 7eff66f70640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff600c9a90 0x7eff60144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.113+0000 7eff66f70640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7eff601540c0 0x7eff601f3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.113+0000 7eff66f70640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7eff60166b90 con 0x7eff600c9a90 2024-03-19T01:51:59.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.113+0000 7eff66f70640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7eff6015b0f0 con 0x7eff600c96b0 2024-03-19T01:51:59.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.113+0000 7eff66f70640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7eff60166d10 con 0x7eff601540c0 2024-03-19T01:51:59.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.113+0000 7fa566ffd640 1 -- 172.21.15.55:0/3088667650 <== osd.0 v2:172.21.15.55:6824/2757675043 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa558073840 con 0x7fa558070d30 2024-03-19T01:51:59.129 INFO:teuthology.orchestra.run.smithi055.stdout:64424509463 2024-03-19T01:51:59.129 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-19T01:51:59.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.129+0000 7fa57cfa2640 1 -- 172.21.15.55:0/3088667650 --> [v2:172.21.15.55:6824/2757675043,v1:172.21.15.55:6825/2757675043] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa578068330 con 0x7fa558070d30 2024-03-19T01:51:59.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.129+0000 7eff64d0c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff600c96b0 0x7eff601522f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.129+0000 7fa566ffd640 1 -- 172.21.15.55:0/3088667650 <== osd.0 v2:172.21.15.55:6824/2757675043 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa578068330 con 0x7fa558070d30 2024-03-19T01:51:59.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.129+0000 7fa57cfa2640 1 -- 172.21.15.55:0/3088667650 >> [v2:172.21.15.55:6824/2757675043,v1:172.21.15.55:6825/2757675043] conn(0x7fa558070d30 msgr2=0x7fa558073170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.129+0000 7fa57cfa2640 1 --2- 172.21.15.55:0/3088667650 >> [v2:172.21.15.55:6824/2757675043,v1:172.21.15.55:6825/2757675043] conn(0x7fa558070d30 0x7fa558073170 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.129+0000 7fa57cfa2640 1 -- 172.21.15.55:0/3088667650 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa5580676b0 msgr2=0x7fa558069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.129+0000 7fa57cfa2640 1 --2- 172.21.15.55:0/3088667650 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa5580676b0 0x7fa558069b70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa5680b2000 tx=0x7fa568004740 comp rx=0 tx=0).stop 2024-03-19T01:51:59.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.129+0000 7fa57cfa2640 1 -- 172.21.15.55:0/3088667650 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5780c7e70 msgr2=0x7fa578072ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.129+0000 7fa57cfa2640 1 --2- 172.21.15.55:0/3088667650 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5780c7e70 0x7fa578072ab0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa570001a60 tx=0x7fa570001e60 comp rx=0 tx=0).stop 2024-03-19T01:51:59.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.129+0000 7fa576d9d640 1 -- 172.21.15.55:0/3088667650 reap_dead start 2024-03-19T01:51:59.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.129+0000 7fa57cfa2640 1 -- 172.21.15.55:0/3088667650 shutdown_connections 2024-03-19T01:51:59.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.129+0000 7fa57cfa2640 1 -- 172.21.15.55:0/3088667650 >> 172.21.15.55:0/3088667650 conn(0x7fa57806f3b0 msgr2=0x7fa578146550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:59.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.129+0000 7fa57cfa2640 1 -- 172.21.15.55:0/3088667650 shutdown_connections 2024-03-19T01:51:59.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.129+0000 7fa57cfa2640 1 -- 172.21.15.55:0/3088667650 wait complete. 2024-03-19T01:51:59.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.129+0000 7eff64d0c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff600c96b0 0x7eff601522f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:55782/0 (socket says 172.21.15.55:55782) 2024-03-19T01:51:59.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.129+0000 7eff64d0c640 1 -- 172.21.15.55:0/1064317876 learned_addr learned my addr 172.21.15.55:0/1064317876 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:59.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.133+0000 7fb22ffff640 1 -- 172.21.15.55:0/3883317258 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb23c018020 con 0x7fb24806c630 2024-03-19T01:51:59.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.133+0000 7fb22ffff640 1 --2- 172.21.15.55:0/3883317258 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb2340676e0 0x7fb234069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.133+0000 7fb2467fc640 1 --2- 172.21.15.55:0/3883317258 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb2340676e0 0x7fb234069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.133+0000 7fb2467fc640 1 --2- 172.21.15.55:0/3883317258 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb2340676e0 0x7fb234069ba0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb2480cbf00 tx=0x7fb238008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.149+0000 7fab197fa640 1 -- 172.21.15.55:0/3996690030 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fab0401a070 con 0x7fab1c1544f0 2024-03-19T01:51:59.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.149+0000 7fab197fa640 1 --2- 172.21.15.55:0/3996690030 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7faafc067660 0x7faafc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.149+0000 7fab1bfff640 1 --2- 172.21.15.55:0/3996690030 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7faafc067660 0x7faafc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.153 INFO:teuthology.orchestra.run.smithi055.stdout:64424509464 2024-03-19T01:51:59.153 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-19T01:51:59.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.153+0000 7fab1bfff640 1 --2- 172.21.15.55:0/3996690030 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7faafc067660 0x7faafc069b20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fab0c01ec90 tx=0x7fab0c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.153+0000 7fab197fa640 1 -- 172.21.15.55:0/3996690030 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fab04053a20 con 0x7fab1c1544f0 2024-03-19T01:51:59.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.153+0000 7fab197fa640 1 --2- 172.21.15.55:0/3996690030 >> [v2:172.21.15.84:6818/2938436826,v1:172.21.15.84:6819/2938436826] conn(0x7faafc070c50 0x7faafc073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.153+0000 7fab197fa640 1 -- 172.21.15.55:0/3996690030 --> [v2:172.21.15.84:6818/2938436826,v1:172.21.15.84:6819/2938436826] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fab0400ee50 con 0x7faafc070c50 2024-03-19T01:51:59.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.153+0000 7fab20ef8640 1 --2- 172.21.15.55:0/3996690030 >> [v2:172.21.15.84:6818/2938436826,v1:172.21.15.84:6819/2938436826] conn(0x7faafc070c50 0x7faafc073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.153+0000 7fab20ef8640 1 --2- 172.21.15.55:0/3996690030 >> [v2:172.21.15.84:6818/2938436826,v1:172.21.15.84:6819/2938436826] conn(0x7faafc070c50 0x7faafc073090 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.157+0000 7fab197fa640 1 -- 172.21.15.55:0/3996690030 <== osd.4 v2:172.21.15.84:6818/2938436826 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fab0400ee50 con 0x7faafc070c50 2024-03-19T01:51:59.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.157+0000 7f1354ff9640 1 -- 172.21.15.55:0/300527546 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f134c0c4070 con 0x7f1350007960 2024-03-19T01:51:59.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.157+0000 7f1354ff9640 1 --2- 172.21.15.55:0/300527546 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f133c067660 0x7f133c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.157+0000 7f13577fe640 1 --2- 172.21.15.55:0/300527546 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f133c067660 0x7f133c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.157+0000 7f13577fe640 1 --2- 172.21.15.55:0/300527546 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f133c067660 0x7f133c069b20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1350046ca0 tx=0x7f1344008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.157+0000 7eff64d0c640 1 -- 172.21.15.55:0/1064317876 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7eff601540c0 msgr2=0x7eff601f3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.157+0000 7eff64d0c640 1 --2- 172.21.15.55:0/1064317876 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7eff601540c0 0x7eff601f3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.157+0000 7eff64d0c640 1 -- 172.21.15.55:0/1064317876 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff600c9a90 msgr2=0x7eff60144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.157+0000 7eff64d0c640 1 --2- 172.21.15.55:0/1064317876 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff600c9a90 0x7eff60144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.157+0000 7eff64d0c640 1 -- 172.21.15.55:0/1064317876 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff60144c40 con 0x7eff600c96b0 2024-03-19T01:51:59.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.161+0000 7eff64d0c640 1 --2- 172.21.15.55:0/1064317876 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff600c96b0 0x7eff601522f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7eff5400b3e0 tx=0x7eff5400b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.165+0000 7eff5d7fa640 1 -- 172.21.15.55:0/1064317876 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff54015020 con 0x7eff600c96b0 2024-03-19T01:51:59.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.165+0000 7eff66f70640 1 -- 172.21.15.55:0/1064317876 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff601f2e20 con 0x7eff600c96b0 2024-03-19T01:51:59.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.165+0000 7eff66f70640 1 -- 172.21.15.55:0/1064317876 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff6014ee00 con 0x7eff600c96b0 2024-03-19T01:51:59.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.165+0000 7f1354ff9640 1 -- 172.21.15.55:0/300527546 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f134c0f1880 con 0x7f1350007960 2024-03-19T01:51:59.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.165+0000 7f1354ff9640 1 --2- 172.21.15.55:0/300527546 >> [v2:172.21.15.55:6808/2797616448,v1:172.21.15.55:6809/2797616448] conn(0x7f133c070c50 0x7f133c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.165+0000 7f1354ff9640 1 -- 172.21.15.55:0/300527546 --> [v2:172.21.15.55:6808/2797616448,v1:172.21.15.55:6809/2797616448] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f134c0caec0 con 0x7f133c070c50 2024-03-19T01:51:59.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.165+0000 7f135c8cd640 1 --2- 172.21.15.55:0/300527546 >> [v2:172.21.15.55:6808/2797616448,v1:172.21.15.55:6809/2797616448] conn(0x7f133c070c50 0x7f133c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.165+0000 7eff66f70640 1 -- 172.21.15.55:0/1064317876 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7eff60068a40 con 0x7eff600c96b0 2024-03-19T01:51:59.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.169+0000 7f135c8cd640 1 --2- 172.21.15.55:0/300527546 >> [v2:172.21.15.55:6808/2797616448,v1:172.21.15.55:6809/2797616448] conn(0x7f133c070c50 0x7f133c073090 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.169+0000 7eff5d7fa640 1 -- 172.21.15.55:0/1064317876 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff60068a40 con 0x7eff600c96b0 2024-03-19T01:51:59.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.169+0000 7eff5d7fa640 1 -- 172.21.15.55:0/1064317876 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff54011a30 con 0x7eff600c96b0 2024-03-19T01:51:59.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.169+0000 7f1354ff9640 1 -- 172.21.15.55:0/300527546 <== osd.3 v2:172.21.15.55:6808/2797616448 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f134c0caec0 con 0x7f133c070c50 2024-03-19T01:51:59.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.169+0000 7f2308963640 1 Processor -- start 2024-03-19T01:51:59.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.169+0000 7f2308963640 1 -- start start 2024-03-19T01:51:59.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.173+0000 7f2308963640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f23040c6550 0x7f23040c9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.173+0000 7f2308963640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f23040c42a0 0x7f23040c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.173+0000 7f2308963640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2304156260 con 0x7f23040c9b30 2024-03-19T01:51:59.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.173+0000 7f2308963640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f23040c4c30 con 0x7f23040c6550 2024-03-19T01:51:59.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.173+0000 7f2308963640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2304156500 con 0x7f23040c42a0 2024-03-19T01:51:59.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.177+0000 7f230359e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f23040c42a0 0x7f23040c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.181+0000 7fb22ffff640 1 -- 172.21.15.55:0/3883317258 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fb23c01be00 con 0x7fb24806c630 2024-03-19T01:51:59.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.181+0000 7fb22ffff640 1 --2- 172.21.15.55:0/3883317258 >> [v2:172.21.15.55:6800/1760253550,v1:172.21.15.55:6801/1760253550] conn(0x7fb234070d50 0x7fb234073190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.181+0000 7fb22ffff640 1 -- 172.21.15.55:0/3883317258 --> [v2:172.21.15.55:6800/1760253550,v1:172.21.15.55:6801/1760253550] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb23c00af30 con 0x7fb234070d50 2024-03-19T01:51:59.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.181+0000 7fb2477fe640 1 --2- 172.21.15.55:0/3883317258 >> [v2:172.21.15.55:6800/1760253550,v1:172.21.15.55:6801/1760253550] conn(0x7fb234070d50 0x7fb234073190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.181+0000 7fab2295b640 1 -- 172.21.15.55:0/3996690030 --> [v2:172.21.15.84:6818/2938436826,v1:172.21.15.84:6819/2938436826] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fab00002de0 con 0x7faafc070c50 2024-03-19T01:51:59.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.181+0000 7fb2477fe640 1 --2- 172.21.15.55:0/3883317258 >> [v2:172.21.15.55:6800/1760253550,v1:172.21.15.55:6801/1760253550] conn(0x7fb234070d50 0x7fb234073190 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.181+0000 7fab197fa640 1 -- 172.21.15.55:0/3996690030 <== osd.4 v2:172.21.15.84:6818/2938436826 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fab00002de0 con 0x7faafc070c50 2024-03-19T01:51:59.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.181+0000 7fab2295b640 1 -- 172.21.15.55:0/3996690030 >> [v2:172.21.15.84:6818/2938436826,v1:172.21.15.84:6819/2938436826] conn(0x7faafc070c50 msgr2=0x7faafc073090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.181+0000 7fab2295b640 1 --2- 172.21.15.55:0/3996690030 >> [v2:172.21.15.84:6818/2938436826,v1:172.21.15.84:6819/2938436826] conn(0x7faafc070c50 0x7faafc073090 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.181+0000 7fab2295b640 1 -- 172.21.15.55:0/3996690030 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7faafc067660 msgr2=0x7faafc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.181+0000 7fab2295b640 1 --2- 172.21.15.55:0/3996690030 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7faafc067660 0x7faafc069b20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fab0c01ec90 tx=0x7fab0c008040 comp rx=0 tx=0).stop 2024-03-19T01:51:59.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.181+0000 7fab2295b640 1 -- 172.21.15.55:0/3996690030 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fab1c1544f0 msgr2=0x7fab1c1f3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.181+0000 7fab2295b640 1 --2- 172.21.15.55:0/3996690030 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fab1c1544f0 0x7fab1c1f3720 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fab0400c430 tx=0x7fab0400c8f0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.181+0000 7fab20ef8640 1 -- 172.21.15.55:0/3996690030 reap_dead start 2024-03-19T01:51:59.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.181+0000 7fab2295b640 1 -- 172.21.15.55:0/3996690030 shutdown_connections 2024-03-19T01:51:59.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.181+0000 7fab2295b640 1 -- 172.21.15.55:0/3996690030 >> 172.21.15.55:0/3996690030 conn(0x7fab1c06e670 msgr2=0x7fab1c0cbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:59.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.181+0000 7fab2295b640 1 -- 172.21.15.55:0/3996690030 shutdown_connections 2024-03-19T01:51:59.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.181+0000 7fab2295b640 1 -- 172.21.15.55:0/3996690030 wait complete. 2024-03-19T01:51:59.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.185+0000 7f230359e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f23040c42a0 0x7f23040c46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:42658/0 (socket says 172.21.15.55:42658) 2024-03-19T01:51:59.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.185+0000 7f230359e640 1 -- 172.21.15.55:0/635894051 learned_addr learned my addr 172.21.15.55:0/635894051 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:59.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.181+0000 7fb22ffff640 1 -- 172.21.15.55:0/3883317258 <== osd.2 v2:172.21.15.55:6800/1760253550 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb23c00af30 con 0x7fb234070d50 2024-03-19T01:51:59.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.193+0000 7eff5d7fa640 1 -- 172.21.15.55:0/1064317876 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7eff54018020 con 0x7eff600c96b0 2024-03-19T01:51:59.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.193+0000 7eff5d7fa640 1 --2- 172.21.15.55:0/1064317876 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7eff44067690 0x7eff44069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.197+0000 7eff5ffff640 1 --2- 172.21.15.55:0/1064317876 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7eff44067690 0x7eff44069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.197+0000 7eff5ffff640 1 --2- 172.21.15.55:0/1064317876 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7eff44067690 0x7eff44069b50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7eff600c8230 tx=0x7eff4c006c20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.197+0000 7eff5d7fa640 1 -- 172.21.15.55:0/1064317876 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7eff54054360 con 0x7eff600c96b0 2024-03-19T01:51:59.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.197+0000 7eff5d7fa640 1 --2- 172.21.15.55:0/1064317876 >> [v2:172.21.15.84:6826/416123135,v1:172.21.15.84:6827/416123135] conn(0x7eff44070c80 0x7eff440730c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.197+0000 7eff5d7fa640 1 -- 172.21.15.55:0/1064317876 --> [v2:172.21.15.84:6826/416123135,v1:172.21.15.84:6827/416123135] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7eff5400af30 con 0x7eff44070c80 2024-03-19T01:51:59.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.197+0000 7eff6550d640 1 --2- 172.21.15.55:0/1064317876 >> [v2:172.21.15.84:6826/416123135,v1:172.21.15.84:6827/416123135] conn(0x7eff44070c80 0x7eff440730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.197+0000 7eff6550d640 1 --2- 172.21.15.55:0/1064317876 >> [v2:172.21.15.84:6826/416123135,v1:172.21.15.84:6827/416123135] conn(0x7eff44070c80 0x7eff440730c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.201+0000 7f230259c640 1 --2- 172.21.15.55:0/635894051 >> v2:172.21.15.84:3300/0 conn(0x7f23040c6550 0x7f23040c9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.201+0000 7eff5d7fa640 1 -- 172.21.15.55:0/1064317876 <== osd.6 v2:172.21.15.84:6826/416123135 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7eff5400af30 con 0x7eff44070c80 2024-03-19T01:51:59.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.209+0000 7f135d0ce640 1 -- 172.21.15.55:0/300527546 --> [v2:172.21.15.55:6808/2797616448,v1:172.21.15.55:6809/2797616448] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1350035df0 con 0x7f133c070c50 2024-03-19T01:51:59.209 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.209+0000 7f1354ff9640 1 -- 172.21.15.55:0/300527546 <== osd.3 v2:172.21.15.55:6808/2797616448 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1350035df0 con 0x7f133c070c50 2024-03-19T01:51:59.210 INFO:teuthology.orchestra.run.smithi055.stdout:64424509464 2024-03-19T01:51:59.210 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-19T01:51:59.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.209+0000 7f135d0ce640 1 -- 172.21.15.55:0/300527546 >> [v2:172.21.15.55:6808/2797616448,v1:172.21.15.55:6809/2797616448] conn(0x7f133c070c50 msgr2=0x7f133c073090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.209+0000 7f135d0ce640 1 --2- 172.21.15.55:0/300527546 >> [v2:172.21.15.55:6808/2797616448,v1:172.21.15.55:6809/2797616448] conn(0x7f133c070c50 0x7f133c073090 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.209+0000 7f135d0ce640 1 -- 172.21.15.55:0/300527546 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f133c067660 msgr2=0x7f133c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.209+0000 7f135d0ce640 1 --2- 172.21.15.55:0/300527546 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f133c067660 0x7f133c069b20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1350046ca0 tx=0x7f1344008040 comp rx=0 tx=0).stop 2024-03-19T01:51:59.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.209+0000 7f135d0ce640 1 -- 172.21.15.55:0/300527546 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1350007960 msgr2=0x7f13500317f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.209+0000 7f135d0ce640 1 --2- 172.21.15.55:0/300527546 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1350007960 0x7f13500317f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f134c0b4040 tx=0x7f134c01cae0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.209+0000 7f135c8cd640 1 -- 172.21.15.55:0/300527546 reap_dead start 2024-03-19T01:51:59.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.209+0000 7f135d0ce640 1 -- 172.21.15.55:0/300527546 shutdown_connections 2024-03-19T01:51:59.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.209+0000 7f135d0ce640 1 -- 172.21.15.55:0/300527546 >> 172.21.15.55:0/300527546 conn(0x7f13500505e0 msgr2=0x7f13500d4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:59.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.209+0000 7f135d0ce640 1 -- 172.21.15.55:0/300527546 shutdown_connections 2024-03-19T01:51:59.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.209+0000 7f135d0ce640 1 -- 172.21.15.55:0/300527546 wait complete. 2024-03-19T01:51:59.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.233+0000 7fb24d963640 1 -- 172.21.15.55:0/3883317258 --> [v2:172.21.15.55:6800/1760253550,v1:172.21.15.55:6801/1760253550] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb228002de0 con 0x7fb234070d50 2024-03-19T01:51:59.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.233+0000 7eff66f70640 1 -- 172.21.15.55:0/1064317876 --> [v2:172.21.15.84:6826/416123135,v1:172.21.15.84:6827/416123135] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7eff6015d5f0 con 0x7eff44070c80 2024-03-19T01:51:59.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.233+0000 7fb22ffff640 1 -- 172.21.15.55:0/3883317258 <== osd.2 v2:172.21.15.55:6800/1760253550 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb228002de0 con 0x7fb234070d50 2024-03-19T01:51:59.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.233+0000 7eff5d7fa640 1 -- 172.21.15.55:0/1064317876 <== osd.6 v2:172.21.15.84:6826/416123135 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7eff6015d5f0 con 0x7eff44070c80 2024-03-19T01:51:59.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.233+0000 7fb22dffb640 1 -- 172.21.15.55:0/3883317258 >> [v2:172.21.15.55:6800/1760253550,v1:172.21.15.55:6801/1760253550] conn(0x7fb234070d50 msgr2=0x7fb234073190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.233+0000 7fb22dffb640 1 --2- 172.21.15.55:0/3883317258 >> [v2:172.21.15.55:6800/1760253550,v1:172.21.15.55:6801/1760253550] conn(0x7fb234070d50 0x7fb234073190 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.233+0000 7eff66f70640 1 -- 172.21.15.55:0/1064317876 >> [v2:172.21.15.84:6826/416123135,v1:172.21.15.84:6827/416123135] conn(0x7eff44070c80 msgr2=0x7eff440730c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.233+0000 7eff66f70640 1 --2- 172.21.15.55:0/1064317876 >> [v2:172.21.15.84:6826/416123135,v1:172.21.15.84:6827/416123135] conn(0x7eff44070c80 0x7eff440730c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.233+0000 7eff66f70640 1 -- 172.21.15.55:0/1064317876 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7eff44067690 msgr2=0x7eff44069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.233+0000 7eff66f70640 1 --2- 172.21.15.55:0/1064317876 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7eff44067690 0x7eff44069b50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7eff600c8230 tx=0x7eff4c006c20 comp rx=0 tx=0).stop 2024-03-19T01:51:59.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.233+0000 7eff66f70640 1 -- 172.21.15.55:0/1064317876 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff600c96b0 msgr2=0x7eff601522f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.233+0000 7eff66f70640 1 --2- 172.21.15.55:0/1064317876 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff600c96b0 0x7eff601522f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7eff5400b3e0 tx=0x7eff5400b8a0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.233+0000 7eff6550d640 1 -- 172.21.15.55:0/1064317876 reap_dead start 2024-03-19T01:51:59.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.233+0000 7eff66f70640 1 -- 172.21.15.55:0/1064317876 shutdown_connections 2024-03-19T01:51:59.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.233+0000 7eff66f70640 1 -- 172.21.15.55:0/1064317876 >> 172.21.15.55:0/1064317876 conn(0x7eff600cc9c0 msgr2=0x7eff600c6b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:59.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.233+0000 7eff66f70640 1 -- 172.21.15.55:0/1064317876 shutdown_connections 2024-03-19T01:51:59.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.233+0000 7eff66f70640 1 -- 172.21.15.55:0/1064317876 wait complete. 2024-03-19T01:51:59.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.237+0000 7fb22dffb640 1 -- 172.21.15.55:0/3883317258 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb2340676e0 msgr2=0x7fb234069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.237+0000 7fb22dffb640 1 --2- 172.21.15.55:0/3883317258 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb2340676e0 0x7fb234069ba0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb2480cbf00 tx=0x7fb238008040 comp rx=0 tx=0).stop 2024-03-19T01:51:59.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.237+0000 7fb22dffb640 1 -- 172.21.15.55:0/3883317258 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb24806c630 msgr2=0x7fb248141450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.237+0000 7fb22dffb640 1 --2- 172.21.15.55:0/3883317258 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb24806c630 0x7fb248141450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb23c00b440 tx=0x7fb23c00b900 comp rx=0 tx=0).stop 2024-03-19T01:51:59.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.237+0000 7fb2477fe640 1 -- 172.21.15.55:0/3883317258 reap_dead start 2024-03-19T01:51:59.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.237+0000 7fb22dffb640 1 -- 172.21.15.55:0/3883317258 shutdown_connections 2024-03-19T01:51:59.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.237+0000 7fb22dffb640 1 -- 172.21.15.55:0/3883317258 >> 172.21.15.55:0/3883317258 conn(0x7fb24806e5a0 msgr2=0x7fb24814dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:59.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.237+0000 7fb22dffb640 1 -- 172.21.15.55:0/3883317258 shutdown_connections 2024-03-19T01:51:59.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.237+0000 7fb22dffb640 1 -- 172.21.15.55:0/3883317258 wait complete. 2024-03-19T01:51:59.240 INFO:teuthology.orchestra.run.smithi055.stdout:60129542168 2024-03-19T01:51:59.240 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-19T01:51:59.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.241+0000 7f5ad06bf640 1 Processor -- start 2024-03-19T01:51:59.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.241+0000 7f5ad06bf640 1 -- start start 2024-03-19T01:51:59.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.241+0000 7f5ad06bf640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f5ac00f6bc0 0x7f5ac00fd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.241+0000 7f5ad06bf640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f5ac0045450 0x7f5ac0005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.241+0000 7f5ad06bf640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5ac0046c50 0x7f5ac004b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.241+0000 7f5ace45b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f5ac00f6bc0 0x7f5ac00fd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.241+0000 7f5ace45b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f5ac00f6bc0 0x7f5ac00fd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52996/0 (socket says 172.21.15.55:52996) 2024-03-19T01:51:59.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.241+0000 7f5ace45b640 1 -- 172.21.15.55:0/3782478839 learned_addr learned my addr 172.21.15.55:0/3782478839 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:59.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.241+0000 7f5ad06bf640 1 -- 172.21.15.55:0/3782478839 --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f5ac0050e70 con 0x7f5ac00f6bc0 2024-03-19T01:51:59.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.241+0000 7f5ad06bf640 1 -- 172.21.15.55:0/3782478839 --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f5ac00467c0 con 0x7f5ac0045450 2024-03-19T01:51:59.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.241+0000 7f5ad06bf640 1 -- 172.21.15.55:0/3782478839 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f5ac0045bb0 con 0x7f5ac0046c50 2024-03-19T01:51:59.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.245+0000 7f5ace45b640 1 -- 172.21.15.55:0/3782478839 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5ac0046c50 msgr2=0x7f5ac004b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:59.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.245+0000 7f5ace45b640 1 --2- 172.21.15.55:0/3782478839 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5ac0046c50 0x7f5ac004b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.245+0000 7f5ace45b640 1 -- 172.21.15.55:0/3782478839 >> v2:172.21.15.84:3300/0 conn(0x7f5ac0045450 msgr2=0x7f5ac0005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:59.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.245+0000 7f5ace45b640 1 --2- 172.21.15.55:0/3782478839 >> v2:172.21.15.84:3300/0 conn(0x7f5ac0045450 0x7f5ac0005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.245+0000 7f5ace45b640 1 -- 172.21.15.55:0/3782478839 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac0045e90 con 0x7f5ac00f6bc0 2024-03-19T01:51:59.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.253+0000 7f5ace45b640 1 --2- 172.21.15.55:0/3782478839 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ac00f6bc0 0x7f5ac00fd380 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000ddd0 tx=0x7f5ab8027870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fcc7f8b7661f527 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.257+0000 7f2301d9b640 1 -- 172.21.15.55:0/635894051 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 962083650 0 0) 0x7f2304156260 con 0x7f23040c9b30 2024-03-19T01:51:59.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.257+0000 7f2301d9b640 1 -- 172.21.15.55:0/635894051 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22ec003530 con 0x7f23040c9b30 2024-03-19T01:51:59.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.257+0000 7f5acd459640 1 -- 172.21.15.55:0/3782478839 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab80ba020 con 0x7f5ac00f6bc0 2024-03-19T01:51:59.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.257+0000 7f5acd459640 1 -- 172.21.15.55:0/3782478839 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab80b2620 con 0x7f5ac00f6bc0 2024-03-19T01:51:59.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.257+0000 7f5acd459640 1 -- 172.21.15.55:0/3782478839 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab80b29f0 con 0x7f5ac00f6bc0 2024-03-19T01:51:59.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.257+0000 7f5ad06bf640 1 -- 172.21.15.55:0/3782478839 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ac00f6bc0 msgr2=0x7f5ac00fd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.257+0000 7f5ad06bf640 1 --2- 172.21.15.55:0/3782478839 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ac00f6bc0 0x7f5ac00fd380 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000ddd0 tx=0x7f5ab8027870 comp rx=0 tx=0).stop 2024-03-19T01:51:59.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.257+0000 7f5ad06bf640 1 -- 172.21.15.55:0/3782478839 shutdown_connections 2024-03-19T01:51:59.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.257+0000 7f5ad06bf640 1 --2- 172.21.15.55:0/3782478839 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5ac0046c50 0x7f5ac004b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.257+0000 7f5ad06bf640 1 --2- 172.21.15.55:0/3782478839 >> v2:172.21.15.84:3300/0 conn(0x7f5ac0045450 0x7f5ac0005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.257+0000 7f5ad06bf640 1 --2- 172.21.15.55:0/3782478839 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ac00f6bc0 0x7f5ac00fd380 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.257+0000 7f5ad06bf640 1 -- 172.21.15.55:0/3782478839 >> 172.21.15.55:0/3782478839 conn(0x7f5ac0050190 msgr2=0x7f5ac00505a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:59.262 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.261+0000 7f5ad06bf640 1 -- 172.21.15.55:0/3782478839 shutdown_connections 2024-03-19T01:51:59.262 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.261+0000 7f5ad06bf640 1 -- 172.21.15.55:0/3782478839 wait complete. 2024-03-19T01:51:59.264 INFO:teuthology.orchestra.run.smithi055.stdout:64424509465 2024-03-19T01:51:59.264 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-19T01:51:59.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.261+0000 7f5ad06bf640 1 Processor -- start 2024-03-19T01:51:59.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.265+0000 7f5ad06bf640 1 -- start start 2024-03-19T01:51:59.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.265+0000 7f5ad06bf640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5ac0045450 0x7f5ac0195cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.265+0000 7f5ad06bf640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ac0046c50 0x7f5ac00ff8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.265+0000 7f5ad06bf640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5ac00f6bc0 0x7f5ac00ffdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.265+0000 7f5ad06bf640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac00f6fa0 con 0x7f5ac0046c50 2024-03-19T01:51:59.267 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.265+0000 7f5ad06bf640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac0046450 con 0x7f5ac0045450 2024-03-19T01:51:59.267 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.265+0000 7f5ad06bf640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f5ac004c610 con 0x7f5ac00f6bc0 2024-03-19T01:51:59.267 INFO:teuthology.orchestra.run.smithi055.stdout:60129542167 2024-03-19T01:51:59.268 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-19T01:51:59.269 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.265+0000 7f5acdc5a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ac0046c50 0x7f5ac00ff8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.269 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.265+0000 7f5acdc5a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ac0046c50 0x7f5ac00ff8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52998/0 (socket says 172.21.15.55:52998) 2024-03-19T01:51:59.269 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.265+0000 7f5acdc5a640 1 -- 172.21.15.55:0/3899978323 learned_addr learned my addr 172.21.15.55:0/3899978323 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:59.269 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.265+0000 7f230359e640 1 -- 172.21.15.55:0/635894051 >> v2:172.21.15.84:3300/0 conn(0x7f23040c6550 msgr2=0x7f23040c9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:59.269 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.265+0000 7f230359e640 1 --2- 172.21.15.55:0/635894051 >> v2:172.21.15.84:3300/0 conn(0x7f23040c6550 0x7f23040c9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.269 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.265+0000 7f230359e640 1 -- 172.21.15.55:0/635894051 >> v1:172.21.15.84:6789/0 conn(0x7f23040c9b30 legacy=0x7f230415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.270 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.265+0000 7f230359e640 1 -- 172.21.15.55:0/635894051 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23041567c0 con 0x7f23040c42a0 2024-03-19T01:51:59.270 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.269+0000 7f5acec5c640 1 --2- 172.21.15.55:0/3899978323 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5ac00f6bc0 0x7f5ac00ffdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.269+0000 7f5acdc5a640 1 -- 172.21.15.55:0/3899978323 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5ac00f6bc0 msgr2=0x7f5ac00ffdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.269+0000 7f5acdc5a640 1 --2- 172.21.15.55:0/3899978323 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5ac00f6bc0 0x7f5ac00ffdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.269+0000 7f5acdc5a640 1 -- 172.21.15.55:0/3899978323 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5ac0045450 msgr2=0x7f5ac0195cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.269+0000 7f5acdc5a640 1 --2- 172.21.15.55:0/3899978323 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5ac0045450 0x7f5ac0195cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.269+0000 7f5acdc5a640 1 -- 172.21.15.55:0/3899978323 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac0188b20 con 0x7f5ac0046c50 2024-03-19T01:51:59.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.273+0000 7f230359e640 1 --2- 172.21.15.55:0/635894051 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f23040c42a0 0x7f23040c46f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009870 tx=0x7f22f801ff80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3c0c37799c27335 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.273+0000 7f5acdc5a640 1 --2- 172.21.15.55:0/3899978323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ac0046c50 0x7f5ac00ff8b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400b440 tx=0x7f5ac400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.277+0000 7f5abeffd640 1 -- 172.21.15.55:0/3899978323 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4015020 con 0x7f5ac0046c50 2024-03-19T01:51:59.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.277+0000 7f5abeffd640 1 -- 172.21.15.55:0/3899978323 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ac40118b0 con 0x7f5ac0046c50 2024-03-19T01:51:59.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.277+0000 7f5abeffd640 1 -- 172.21.15.55:0/3899978323 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4011ca0 con 0x7f5ac0046c50 2024-03-19T01:51:59.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.277+0000 7f5ad06bf640 1 -- 172.21.15.55:0/3899978323 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ac0196800 con 0x7f5ac0046c50 2024-03-19T01:51:59.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.277+0000 7f5ad06bf640 1 -- 172.21.15.55:0/3899978323 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ac00fdac0 con 0x7f5ac0046c50 2024-03-19T01:51:59.284 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.281+0000 7f2301d9b640 1 -- 172.21.15.55:0/635894051 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f80b6070 con 0x7f23040c42a0 2024-03-19T01:51:59.284 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.281+0000 7f2301d9b640 1 -- 172.21.15.55:0/635894051 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22f80a9d90 con 0x7f23040c42a0 2024-03-19T01:51:59.284 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.285+0000 7f2301d9b640 1 -- 172.21.15.55:0/635894051 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f80b1660 con 0x7f23040c42a0 2024-03-19T01:51:59.285 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.285+0000 7f2308963640 1 -- 172.21.15.55:0/635894051 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f23040c42a0 msgr2=0x7f23040c46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.285 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.285+0000 7f2308963640 1 --2- 172.21.15.55:0/635894051 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f23040c42a0 0x7f23040c46f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009870 tx=0x7f22f801ff80 comp rx=0 tx=0).stop 2024-03-19T01:51:59.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.285+0000 7f2308963640 1 -- 172.21.15.55:0/635894051 shutdown_connections 2024-03-19T01:51:59.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.285+0000 7f2308963640 1 --2- 172.21.15.55:0/635894051 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f23040c42a0 0x7f23040c46f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.285+0000 7f2308963640 1 --2- 172.21.15.55:0/635894051 >> v2:172.21.15.84:3300/0 conn(0x7f23040c6550 0x7f23040c9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.285+0000 7f2308963640 1 -- 172.21.15.55:0/635894051 >> 172.21.15.55:0/635894051 conn(0x7f2304067400 msgr2=0x7f230406b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:59.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.285+0000 7f2308963640 1 -- 172.21.15.55:0/635894051 shutdown_connections 2024-03-19T01:51:59.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.285+0000 7f2308963640 1 -- 172.21.15.55:0/635894051 wait complete. 2024-03-19T01:51:59.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.285+0000 7f2308963640 1 Processor -- start 2024-03-19T01:51:59.288 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.285+0000 7f5ad06bf640 1 -- 172.21.15.55:0/3899978323 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ab00052a0 con 0x7f5ac0046c50 2024-03-19T01:51:59.288 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.285+0000 7f2308963640 1 -- start start 2024-03-19T01:51:59.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.289+0000 7f2308963640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f23040c42a0 0x7f2304144ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.289+0000 7f2308963640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23040c6550 0x7f2304145910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.289+0000 7f2308963640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23040c9b30 0x7f2304149a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.289+0000 7f2308963640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2304157060 con 0x7f23040c6550 2024-03-19T01:51:59.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.289+0000 7f2308963640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2304156d80 con 0x7f23040c9b30 2024-03-19T01:51:59.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.289+0000 7f2308963640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f230414d1c0 con 0x7f23040c42a0 2024-03-19T01:51:59.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.289+0000 7f2302d9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f23040c42a0 0x7f2304144ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.289+0000 7f2302d9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f23040c42a0 0x7f2304144ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:42684/0 (socket says 172.21.15.55:42684) 2024-03-19T01:51:59.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.289+0000 7f2302d9d640 1 -- 172.21.15.55:0/3815110467 learned_addr learned my addr 172.21.15.55:0/3815110467 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:59.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.289+0000 7f230359e640 1 --2- 172.21.15.55:0/3815110467 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23040c9b30 0x7f2304149a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.293+0000 7f2302d9d640 1 -- 172.21.15.55:0/3815110467 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23040c9b30 msgr2=0x7f2304149a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.293+0000 7f2302d9d640 1 --2- 172.21.15.55:0/3815110467 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23040c9b30 0x7f2304149a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.293+0000 7f2302d9d640 1 -- 172.21.15.55:0/3815110467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23040c6550 msgr2=0x7f2304145910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.293+0000 7f2302d9d640 1 --2- 172.21.15.55:0/3815110467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23040c6550 0x7f2304145910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.293+0000 7f2302d9d640 1 -- 172.21.15.55:0/3815110467 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2304147690 con 0x7f23040c42a0 2024-03-19T01:51:59.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.297+0000 7f2302d9d640 1 --2- 172.21.15.55:0/3815110467 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f23040c42a0 0x7f2304144ff0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00c7a0 tx=0x7f22fc00cc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.305+0000 7f22eb7fe640 1 -- 172.21.15.55:0/3815110467 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc003b80 con 0x7f23040c42a0 2024-03-19T01:51:59.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.305+0000 7f22eb7fe640 1 -- 172.21.15.55:0/3815110467 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22fc003d20 con 0x7f23040c42a0 2024-03-19T01:51:59.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.305+0000 7f2308963640 1 -- 172.21.15.55:0/3815110467 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23041557f0 con 0x7f23040c42a0 2024-03-19T01:51:59.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.305+0000 7f2308963640 1 -- 172.21.15.55:0/3815110467 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2304158bf0 con 0x7f23040c42a0 2024-03-19T01:51:59.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.305+0000 7f22eb7fe640 1 -- 172.21.15.55:0/3815110467 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc00aa70 con 0x7f23040c42a0 2024-03-19T01:51:59.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.309+0000 7f2308963640 1 -- 172.21.15.55:0/3815110467 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f230404f590 con 0x7f23040c42a0 2024-03-19T01:51:59.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.313+0000 7f40438cd640 1 Processor -- start 2024-03-19T01:51:59.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.317+0000 7f40438cd640 1 -- start start 2024-03-19T01:51:59.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.317+0000 7f40438cd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f403c1644e0 0x7f403c16a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.317+0000 7f40438cd640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f403c0c7f10 0x7f403c0c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.317+0000 7f40438cd640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f403c16a9c0 con 0x7f403c071a80 2024-03-19T01:51:59.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.317+0000 7f40438cd640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f403c16aca0 con 0x7f403c0c7f10 2024-03-19T01:51:59.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.317+0000 7f40438cd640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f403c16af80 con 0x7f403c1644e0 2024-03-19T01:51:59.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.317+0000 7f4041e6a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f403c0c7f10 0x7f403c0c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.321+0000 7f4041e6a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f403c0c7f10 0x7f403c0c66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:55810/0 (socket says 172.21.15.55:55810) 2024-03-19T01:51:59.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.321+0000 7f4041e6a640 1 -- 172.21.15.55:0/1639862465 learned_addr learned my addr 172.21.15.55:0/1639862465 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:59.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.321+0000 7f4041e6a640 1 -- 172.21.15.55:0/1639862465 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f403c1644e0 msgr2=0x7f403c16a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:59.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.321+0000 7f4041e6a640 1 --2- 172.21.15.55:0/1639862465 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f403c1644e0 0x7f403c16a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.321+0000 7f4041e6a640 1 -- 172.21.15.55:0/1639862465 >> v1:172.21.15.55:6789/0 conn(0x7f403c071a80 legacy=0x7f403c1677b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:59.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.321+0000 7f4041e6a640 1 -- 172.21.15.55:0/1639862465 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f403c166fb0 con 0x7f403c0c7f10 2024-03-19T01:51:59.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.325+0000 7f4041e6a640 1 --2- 172.21.15.55:0/1639862465 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f403c0c7f10 0x7f403c0c66a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4038008870 tx=0x7f4038020f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=259d371552a254f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.329+0000 7ff37bd86640 1 Processor -- start 2024-03-19T01:51:59.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.329+0000 7ff37bd86640 1 -- start start 2024-03-19T01:51:59.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.329+0000 7ff37bd86640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff374152120 0x7ff3740c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.329+0000 7ff37bd86640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff374070780 0x7ff3741586e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.329+0000 7ff37bd86640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff37406fff0 con 0x7ff3740c9610 2024-03-19T01:51:59.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.329+0000 7ff37bd86640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7ff374152500 con 0x7ff374152120 2024-03-19T01:51:59.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.329+0000 7ff37bd86640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7ff374070590 con 0x7ff374070780 2024-03-19T01:51:59.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.333+0000 7f4033fff640 1 -- 172.21.15.55:0/1639862465 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40380b0070 con 0x7f403c0c7f10 2024-03-19T01:51:59.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.333+0000 7f4033fff640 1 -- 172.21.15.55:0/1639862465 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40380aac00 con 0x7f403c0c7f10 2024-03-19T01:51:59.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.333+0000 7f4033fff640 1 -- 172.21.15.55:0/1639862465 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40380b4400 con 0x7f403c0c7f10 2024-03-19T01:51:59.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.333+0000 7ff37a323640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff374070780 0x7ff3741586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.333+0000 7f40438cd640 1 -- 172.21.15.55:0/1639862465 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f403c0c7f10 msgr2=0x7f403c0c66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.333+0000 7f40438cd640 1 --2- 172.21.15.55:0/1639862465 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f403c0c7f10 0x7f403c0c66a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4038008870 tx=0x7f4038020f30 comp rx=0 tx=0).stop 2024-03-19T01:51:59.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.333+0000 7ff37a323640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff374070780 0x7ff3741586e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:55826/0 (socket says 172.21.15.55:55826) 2024-03-19T01:51:59.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.333+0000 7ff37a323640 1 -- 172.21.15.55:0/341312070 learned_addr learned my addr 172.21.15.55:0/341312070 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:59.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.333+0000 7f40438cd640 1 -- 172.21.15.55:0/1639862465 shutdown_connections 2024-03-19T01:51:59.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.333+0000 7f40438cd640 1 --2- 172.21.15.55:0/1639862465 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f403c0c7f10 0x7f403c0c66a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.333+0000 7f40438cd640 1 --2- 172.21.15.55:0/1639862465 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f403c1644e0 0x7f403c16a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.333+0000 7f40438cd640 1 -- 172.21.15.55:0/1639862465 >> 172.21.15.55:0/1639862465 conn(0x7f403c06d250 msgr2=0x7f403c06d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:59.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.333+0000 7f40438cd640 1 -- 172.21.15.55:0/1639862465 shutdown_connections 2024-03-19T01:51:59.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.333+0000 7f40438cd640 1 -- 172.21.15.55:0/1639862465 wait complete. 2024-03-19T01:51:59.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.333+0000 7f40438cd640 1 Processor -- start 2024-03-19T01:51:59.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.337+0000 7ff37a323640 1 -- 172.21.15.55:0/341312070 >> v2:172.21.15.55:3300/0 conn(0x7ff374152120 msgr2=0x7ff3740c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:59.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.337+0000 7ff37a323640 1 --2- 172.21.15.55:0/341312070 >> v2:172.21.15.55:3300/0 conn(0x7ff374152120 0x7ff3740c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.337+0000 7ff37a323640 1 -- 172.21.15.55:0/341312070 >> v1:172.21.15.84:6789/0 conn(0x7ff3740c9610 legacy=0x7ff374157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:59.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.333+0000 7f40438cd640 1 -- start start 2024-03-19T01:51:59.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.337+0000 7f40438cd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f403c071a80 0x7f403c168b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.337+0000 7ff37a323640 1 -- 172.21.15.55:0/341312070 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3740c39c0 con 0x7ff374070780 2024-03-19T01:51:59.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.337+0000 7f40438cd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f403c0c7f10 0x7f403c0c5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.337+0000 7f40438cd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f403c1644e0 0x7f403c20dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.337+0000 7f40438cd640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f403c164a40 con 0x7f403c0c7f10 2024-03-19T01:51:59.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.337+0000 7f40438cd640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f403c1648c0 con 0x7f403c1644e0 2024-03-19T01:51:59.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.337+0000 7f40438cd640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f403c0c7470 con 0x7f403c071a80 2024-03-19T01:51:59.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.337+0000 7f4041e6a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f403c1644e0 0x7f403c20dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.337+0000 7f22eb7fe640 1 -- 172.21.15.55:0/3815110467 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f22fc00ac10 con 0x7f23040c42a0 2024-03-19T01:51:59.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.337+0000 7f22eb7fe640 1 --2- 172.21.15.55:0/3815110467 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f22f00676b0 0x7f22f0069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.337+0000 7f230259c640 1 --2- 172.21.15.55:0/3815110467 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f22f00676b0 0x7f22f0069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.341+0000 7f5abeffd640 1 -- 172.21.15.55:0/3899978323 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f5ac4018020 con 0x7f5ac0046c50 2024-03-19T01:51:59.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.341+0000 7f230259c640 1 --2- 172.21.15.55:0/3815110467 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f22f00676b0 0x7f22f0069b70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f23041f0260 tx=0x7f22f4009e50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.341+0000 7f4041e6a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f403c1644e0 0x7f403c20dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:55834/0 (socket says 172.21.15.55:55834) 2024-03-19T01:51:59.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.341+0000 7f4041e6a640 1 -- 172.21.15.55:0/1954656492 learned_addr learned my addr 172.21.15.55:0/1954656492 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:59.342 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.341+0000 7f5abeffd640 1 --2- 172.21.15.55:0/3899978323 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5aac067660 0x7f5aac069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.342 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.341+0000 7f5ace45b640 1 --2- 172.21.15.55:0/3899978323 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5aac067660 0x7f5aac069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.342 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.341+0000 7f5ace45b640 1 --2- 172.21.15.55:0/3899978323 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5aac067660 0x7f5aac069b20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80bb000 tx=0x7f5ab8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.341+0000 7ff37a323640 1 --2- 172.21.15.55:0/341312070 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff374070780 0x7ff3741586e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff3740cb940 tx=0x7ff36c01c9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2af0647e6cea7769 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.345+0000 7f22eb7fe640 1 -- 172.21.15.55:0/3815110467 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f22fc054880 con 0x7f23040c42a0 2024-03-19T01:51:59.347 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.345+0000 7f5abeffd640 1 -- 172.21.15.55:0/3899978323 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f5ac40545e0 con 0x7f5ac0046c50 2024-03-19T01:51:59.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.349+0000 7f4041e6a640 1 -- 172.21.15.55:0/1954656492 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f403c071a80 msgr2=0x7f403c168b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.349+0000 7f4041e6a640 1 --2- 172.21.15.55:0/1954656492 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f403c071a80 0x7f403c168b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.349+0000 7f4041e6a640 1 -- 172.21.15.55:0/1954656492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f403c0c7f10 msgr2=0x7f403c0c5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.349+0000 7f4041e6a640 1 --2- 172.21.15.55:0/1954656492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f403c0c7f10 0x7f403c0c5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.349+0000 7f4041e6a640 1 -- 172.21.15.55:0/1954656492 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f403c1f2220 con 0x7f403c1644e0 2024-03-19T01:51:59.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.353+0000 7ff378b20640 1 -- 172.21.15.55:0/341312070 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36c0a5670 con 0x7ff374070780 2024-03-19T01:51:59.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.353+0000 7ff378b20640 1 -- 172.21.15.55:0/341312070 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff36c0a5810 con 0x7ff374070780 2024-03-19T01:51:59.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.353+0000 7ff378b20640 1 -- 172.21.15.55:0/341312070 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36c0a5be0 con 0x7ff374070780 2024-03-19T01:51:59.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.357+0000 7ff37bd86640 1 -- 172.21.15.55:0/341312070 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff374070780 msgr2=0x7ff3741586e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.357+0000 7ff37bd86640 1 --2- 172.21.15.55:0/341312070 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff374070780 0x7ff3741586e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff3740cb940 tx=0x7ff36c01c9b0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.357+0000 7ff37bd86640 1 -- 172.21.15.55:0/341312070 shutdown_connections 2024-03-19T01:51:59.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.357+0000 7ff37bd86640 1 --2- 172.21.15.55:0/341312070 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff374070780 0x7ff3741586e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.357+0000 7ff37bd86640 1 --2- 172.21.15.55:0/341312070 >> v2:172.21.15.55:3300/0 conn(0x7ff374152120 0x7ff3740c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.357+0000 7ff37bd86640 1 -- 172.21.15.55:0/341312070 >> 172.21.15.55:0/341312070 conn(0x7ff374067600 msgr2=0x7ff374071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:59.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.357+0000 7ff37bd86640 1 -- 172.21.15.55:0/341312070 shutdown_connections 2024-03-19T01:51:59.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.357+0000 7ff37bd86640 1 -- 172.21.15.55:0/341312070 wait complete. 2024-03-19T01:51:59.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.357+0000 7f4041e6a640 1 --2- 172.21.15.55:0/1954656492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f403c1644e0 0x7f403c20dd60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f403801f5e0 tx=0x7f403801dcd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.357+0000 7ff37bd86640 1 Processor -- start 2024-03-19T01:51:59.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.357+0000 7ff37bd86640 1 -- start start 2024-03-19T01:51:59.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.357+0000 7ff37bd86640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff374070780 0x7ff3740cb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.361+0000 7ff37bd86640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3740c9610 0x7ff3740b2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.361+0000 7ff37bd86640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff374152120 0x7ff374143c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.361+0000 7ff37bd86640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff3740c86d0 con 0x7ff374152120 2024-03-19T01:51:59.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.361+0000 7ff37bd86640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff3740c31c0 con 0x7ff3740c9610 2024-03-19T01:51:59.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.361+0000 7ff37bd86640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff37406c3b0 con 0x7ff374070780 2024-03-19T01:51:59.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.365+0000 7ff379321640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3740c9610 0x7ff3740b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.365+0000 7f4031ffb640 1 -- 172.21.15.55:0/1954656492 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40380c3070 con 0x7f403c1644e0 2024-03-19T01:51:59.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.365+0000 7f4031ffb640 1 -- 172.21.15.55:0/1954656492 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40380b4e20 con 0x7f403c1644e0 2024-03-19T01:51:59.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.365+0000 7f4031ffb640 1 -- 172.21.15.55:0/1954656492 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40380b0040 con 0x7f403c1644e0 2024-03-19T01:51:59.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.365+0000 7f40438cd640 1 -- 172.21.15.55:0/1954656492 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f403c1f24e0 con 0x7f403c1644e0 2024-03-19T01:51:59.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.365+0000 7f40438cd640 1 -- 172.21.15.55:0/1954656492 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f403c1f2aa0 con 0x7f403c1644e0 2024-03-19T01:51:59.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.365+0000 7ff379321640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3740c9610 0x7ff3740b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:55848/0 (socket says 172.21.15.55:55848) 2024-03-19T01:51:59.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.365+0000 7ff379321640 1 -- 172.21.15.55:0/2354848291 learned_addr learned my addr 172.21.15.55:0/2354848291 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:59.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.365+0000 7f40438cd640 1 -- 172.21.15.55:0/1954656492 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f403c04f590 con 0x7f403c1644e0 2024-03-19T01:51:59.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.397+0000 7f4eebccf640 1 Processor -- start 2024-03-19T01:51:59.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.397+0000 7f4eebccf640 1 -- start start 2024-03-19T01:51:59.398 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.397+0000 7f4eebccf640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4ee4150ae0 0x7f4ee40c6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.398 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.397+0000 7f4eebccf640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ee40c3520 0x7f4ee40cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.398 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.397+0000 7f4eebccf640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f4ee406c630 0x7f4ee40cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.398 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.397+0000 7f4eebccf640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f4ee40c3a40 con 0x7f4ee4150ae0 2024-03-19T01:51:59.398 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.397+0000 7f4eebccf640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f4ee40c8200 con 0x7f4ee406c630 2024-03-19T01:51:59.398 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.397+0000 7f4eebccf640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4ee406c230 con 0x7f4ee40c3520 2024-03-19T01:51:59.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.417+0000 7f4ee9a6b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4ee4150ae0 0x7f4ee40c6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.421+0000 7f4ee9a6b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4ee4150ae0 0x7f4ee40c6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53036/0 (socket says 172.21.15.55:53036) 2024-03-19T01:51:59.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.421+0000 7f4ee9a6b640 1 -- 172.21.15.55:0/3317045318 learned_addr learned my addr 172.21.15.55:0/3317045318 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:59.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.421+0000 7ff37a323640 1 --2- 172.21.15.55:0/2354848291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff374152120 0x7ff374143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.421+0000 7f193c67e640 1 Processor -- start 2024-03-19T01:51:59.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.421+0000 7f193c67e640 1 -- start start 2024-03-19T01:51:59.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.421+0000 7fef29bba640 1 Processor -- start 2024-03-19T01:51:59.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.421+0000 7fef29bba640 1 -- start start 2024-03-19T01:51:59.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.421+0000 7f193c67e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f193406c3b0 0x7f193415e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.421+0000 7f193c67e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f193406f840 con 0x7f193406c790 2024-03-19T01:51:59.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.421+0000 7f193c67e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f193406fb20 con 0x7f1934072190 2024-03-19T01:51:59.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.421+0000 7f193c67e640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f193406fe00 con 0x7f193406c3b0 2024-03-19T01:51:59.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.425+0000 7fef29bba640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fef1c0d5000 0x7fef1c0db7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.425+0000 7fef29bba640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fef1c0458a0 0x7fef1c006340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.425+0000 7fef29bba640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef1c046000 con 0x7fef1c0470a0 2024-03-19T01:51:59.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.425+0000 7fef29bba640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fef1c0512c0 con 0x7fef1c0458a0 2024-03-19T01:51:59.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.425+0000 7fef29bba640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fef1c046c10 con 0x7fef1c0d5000 2024-03-19T01:51:59.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.433+0000 7f11ab925640 1 Processor -- start 2024-03-19T01:51:59.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.433+0000 7f11ab925640 1 -- start start 2024-03-19T01:51:59.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.433+0000 7f11ab925640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f11a41510f0 0x7f11a40c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.433+0000 7f11ab925640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f11a40c73c0 0x7f11a40c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.433+0000 7f11ab925640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11a40c3f60 con 0x7f11a415a410 2024-03-19T01:51:59.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.433+0000 7f11ab925640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f11a4070fb0 con 0x7f11a40c73c0 2024-03-19T01:51:59.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.433+0000 7f11ab925640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f11a41514d0 con 0x7f11a41510f0 2024-03-19T01:51:59.605 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.605+0000 7f22eb7fe640 1 -- 172.21.15.55:0/3815110467 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f22fc056ee0 con 0x7f23040c42a0 2024-03-19T01:51:59.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.605+0000 7ff379b22640 1 --2- 172.21.15.55:0/2354848291 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff374070780 0x7ff3740cb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.605+0000 7f4ee926a640 1 --2- 172.21.15.55:0/3317045318 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ee40c3520 0x7f4ee40cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.605+0000 7f4ee9a6b640 1 -- 172.21.15.55:0/3317045318 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ee40c3520 msgr2=0x7f4ee40cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:59.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.605+0000 7f4ee9a6b640 1 --2- 172.21.15.55:0/3317045318 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ee40c3520 0x7f4ee40cbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.605+0000 7f4ee9a6b640 1 -- 172.21.15.55:0/3317045318 >> v2:172.21.15.84:3300/0 conn(0x7f4ee406c630 msgr2=0x7f4ee40cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:59.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.605+0000 7f4ee9a6b640 1 --2- 172.21.15.55:0/3317045318 >> v2:172.21.15.84:3300/0 conn(0x7f4ee406c630 0x7f4ee40cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.605+0000 7f4ee9a6b640 1 -- 172.21.15.55:0/3317045318 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ee406bc70 con 0x7f4ee4150ae0 2024-03-19T01:51:59.614 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.613+0000 7ff379b22640 1 -- 172.21.15.55:0/2354848291 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3740c9610 msgr2=0x7ff3740b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.614 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.613+0000 7fef28bb8640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fef1c0d5000 0x7fef1c0db7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.615 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.613+0000 7ff379b22640 1 --2- 172.21.15.55:0/2354848291 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3740c9610 0x7ff3740b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.613+0000 7ff379b22640 1 -- 172.21.15.55:0/2354848291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff374152120 msgr2=0x7ff374143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.613+0000 7ff379b22640 1 --2- 172.21.15.55:0/2354848291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff374152120 0x7ff374143c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.613+0000 7ff379b22640 1 -- 172.21.15.55:0/2354848291 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3740b0e80 con 0x7ff374070780 2024-03-19T01:51:59.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.617+0000 7fef28bb8640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fef1c0d5000 0x7fef1c0db7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:42720/0 (socket says 172.21.15.55:42720) 2024-03-19T01:51:59.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.617+0000 7fef28bb8640 1 -- 172.21.15.55:0/3162143887 learned_addr learned my addr 172.21.15.55:0/3162143887 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:59.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.617+0000 7f5abeffd640 1 -- 172.21.15.55:0/3899978323 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f5ac401f020 con 0x7f5ac0046c50 2024-03-19T01:51:59.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.617+0000 7ff379b22640 1 --2- 172.21.15.55:0/2354848291 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff374070780 0x7ff3740cb8b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff36400b410 tx=0x7ff36400b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.622 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.621+0000 7fef28bb8640 1 -- 172.21.15.55:0/3162143887 >> v2:172.21.15.84:3300/0 conn(0x7fef1c0458a0 msgr2=0x7fef1c006340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:59.622 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.621+0000 7fef28bb8640 1 --2- 172.21.15.55:0/3162143887 >> v2:172.21.15.84:3300/0 conn(0x7fef1c0458a0 0x7fef1c006340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.623 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.621+0000 7fef28bb8640 1 -- 172.21.15.55:0/3162143887 >> v1:172.21.15.84:6789/0 conn(0x7fef1c0470a0 legacy=0x7fef1c04b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:59.623 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.621+0000 7fef28bb8640 1 -- 172.21.15.55:0/3162143887 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef1c0462e0 con 0x7fef1c0d5000 2024-03-19T01:51:59.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.625+0000 7ff36a7fc640 1 -- 172.21.15.55:0/2354848291 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff364015020 con 0x7ff374070780 2024-03-19T01:51:59.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.625+0000 7ff36a7fc640 1 -- 172.21.15.55:0/2354848291 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff364011830 con 0x7ff374070780 2024-03-19T01:51:59.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.625+0000 7ff37bd86640 1 -- 172.21.15.55:0/2354848291 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3740b1720 con 0x7ff374070780 2024-03-19T01:51:59.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.625+0000 7ff36a7fc640 1 -- 172.21.15.55:0/2354848291 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff364011c20 con 0x7ff374070780 2024-03-19T01:51:59.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.625+0000 7ff37bd86640 1 -- 172.21.15.55:0/2354848291 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3741406b0 con 0x7ff374070780 2024-03-19T01:51:59.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.629+0000 7ff37bd86640 1 -- 172.21.15.55:0/2354848291 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff374068a40 con 0x7ff374070780 2024-03-19T01:51:59.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.629+0000 7f4ee9a6b640 1 --2- 172.21.15.55:0/3317045318 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ee4150ae0 0x7f4ee40c6e80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40c6120 tx=0x7f4ee001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e20f971a52d7b588 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.631 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.629+0000 7fef28bb8640 1 --2- 172.21.15.55:0/3162143887 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fef1c0d5000 0x7fef1c0db7c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fef18007360 tx=0x7fef1801fd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc4eb075330df0d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.633+0000 7f193a41a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f193406c3b0 0x7f193415e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.633+0000 7f193a41a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f193406c3b0 0x7f193415e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53040/0 (socket says 172.21.15.55:53040) 2024-03-19T01:51:59.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.633+0000 7f193a41a640 1 -- 172.21.15.55:0/168841755 learned_addr learned my addr 172.21.15.55:0/168841755 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:59.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.637+0000 7f193a41a640 1 -- 172.21.15.55:0/168841755 >> v1:172.21.15.84:6789/0 conn(0x7f1934072190 legacy=0x7f193406f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:59.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.637+0000 7f193a41a640 1 -- 172.21.15.55:0/168841755 >> v1:172.21.15.55:6789/0 conn(0x7f193406c790 legacy=0x7f19340c30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:59.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.637+0000 7f193a41a640 1 -- 172.21.15.55:0/168841755 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19340700e0 con 0x7f193406c3b0 2024-03-19T01:51:59.639 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.637+0000 7f11a9ec2640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f11a40c73c0 0x7f11a40c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.637+0000 7f11a9ec2640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f11a40c73c0 0x7f11a40c77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53056/0 (socket says 172.21.15.55:53056) 2024-03-19T01:51:59.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.637+0000 7f11a9ec2640 1 -- 172.21.15.55:0/414744305 learned_addr learned my addr 172.21.15.55:0/414744305 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:59.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.645+0000 7f11a9ec2640 1 -- 172.21.15.55:0/414744305 >> v2:172.21.15.84:3300/0 conn(0x7f11a41510f0 msgr2=0x7f11a40c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:59.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.645+0000 7f11a9ec2640 1 --2- 172.21.15.55:0/414744305 >> v2:172.21.15.84:3300/0 conn(0x7f11a41510f0 0x7f11a40c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.645+0000 7f11a9ec2640 1 -- 172.21.15.55:0/414744305 >> v1:172.21.15.55:6789/0 conn(0x7f11a415a410 legacy=0x7f11a40708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:51:59.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.645+0000 7f11a9ec2640 1 -- 172.21.15.55:0/414744305 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a40c6870 con 0x7f11a40c73c0 2024-03-19T01:51:59.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.645+0000 7f4ee8a69640 1 -- 172.21.15.55:0/3317045318 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee00b0070 con 0x7f4ee4150ae0 2024-03-19T01:51:59.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.645+0000 7f4ee8a69640 1 -- 172.21.15.55:0/3317045318 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ee00ab600 con 0x7f4ee4150ae0 2024-03-19T01:51:59.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.645+0000 7f4ee8a69640 1 -- 172.21.15.55:0/3317045318 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee00ab9d0 con 0x7f4ee4150ae0 2024-03-19T01:51:59.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.645+0000 7f4eebccf640 1 -- 172.21.15.55:0/3317045318 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ee4150ae0 msgr2=0x7f4ee40c6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.645+0000 7f4eebccf640 1 --2- 172.21.15.55:0/3317045318 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ee4150ae0 0x7f4ee40c6e80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40c6120 tx=0x7f4ee001c490 comp rx=0 tx=0).stop 2024-03-19T01:51:59.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.649+0000 7f4eebccf640 1 -- 172.21.15.55:0/3317045318 shutdown_connections 2024-03-19T01:51:59.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.649+0000 7f4eebccf640 1 --2- 172.21.15.55:0/3317045318 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ee40c3520 0x7f4ee40cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.649+0000 7f4eebccf640 1 --2- 172.21.15.55:0/3317045318 >> v2:172.21.15.84:3300/0 conn(0x7f4ee406c630 0x7f4ee40cc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.649+0000 7f4eebccf640 1 --2- 172.21.15.55:0/3317045318 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ee4150ae0 0x7f4ee40c6e80 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.649+0000 7f4eebccf640 1 -- 172.21.15.55:0/3317045318 >> 172.21.15.55:0/3317045318 conn(0x7f4ee406e5a0 msgr2=0x7f4ee4072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:59.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.649+0000 7f4eebccf640 1 -- 172.21.15.55:0/3317045318 shutdown_connections 2024-03-19T01:51:59.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.649+0000 7f4eebccf640 1 -- 172.21.15.55:0/3317045318 wait complete. 2024-03-19T01:51:59.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.649+0000 7f4eebccf640 1 Processor -- start 2024-03-19T01:51:59.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.649+0000 7f4eebccf640 1 -- start start 2024-03-19T01:51:59.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.649+0000 7f193a41a640 1 --2- 172.21.15.55:0/168841755 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f193406c3b0 0x7f193415e490 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1930009870 tx=0x7f193001c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9875a30e4473f59f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.649+0000 7f4eebccf640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ee406c630 0x7f4ee41544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.649+0000 7f4eebccf640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ee40c3520 0x7f4ee4141030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.649+0000 7f4eebccf640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ee4150ae0 0x7f4ee4154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.649+0000 7f4eebccf640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee40c6a20 con 0x7f4ee40c3520 2024-03-19T01:51:59.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.649+0000 7f4eebccf640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee40c6380 con 0x7f4ee4150ae0 2024-03-19T01:51:59.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.649+0000 7f4eebccf640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4ee40c6ba0 con 0x7f4ee406c630 2024-03-19T01:51:59.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.653+0000 7f1939418640 1 -- 172.21.15.55:0/168841755 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19300ab5f0 con 0x7f193406c3b0 2024-03-19T01:51:59.657 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.657+0000 7f4ee926a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ee40c3520 0x7f4ee4141030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.657+0000 7f4ee926a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ee40c3520 0x7f4ee4141030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53060/0 (socket says 172.21.15.55:53060) 2024-03-19T01:51:59.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.657+0000 7f4ee926a640 1 -- 172.21.15.55:0/1475027555 learned_addr learned my addr 172.21.15.55:0/1475027555 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:59.661 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.661+0000 7f1939418640 1 -- 172.21.15.55:0/168841755 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19300b0070 con 0x7f193406c3b0 2024-03-19T01:51:59.661 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.661+0000 7f1939418640 1 -- 172.21.15.55:0/168841755 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19300ab9b0 con 0x7f193406c3b0 2024-03-19T01:51:59.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.661+0000 7f193c67e640 1 -- 172.21.15.55:0/168841755 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f193406c3b0 msgr2=0x7f193415e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.661+0000 7f193c67e640 1 --2- 172.21.15.55:0/168841755 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f193406c3b0 0x7f193415e490 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1930009870 tx=0x7f193001c4e0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.661+0000 7f193c67e640 1 -- 172.21.15.55:0/168841755 shutdown_connections 2024-03-19T01:51:59.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.661+0000 7f193c67e640 1 --2- 172.21.15.55:0/168841755 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f193406c3b0 0x7f193415e490 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.661+0000 7f193c67e640 1 -- 172.21.15.55:0/168841755 >> 172.21.15.55:0/168841755 conn(0x7f1934071590 msgr2=0x7f19340719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:59.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.661+0000 7f193c67e640 1 -- 172.21.15.55:0/168841755 shutdown_connections 2024-03-19T01:51:59.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.661+0000 7f193c67e640 1 -- 172.21.15.55:0/168841755 wait complete. 2024-03-19T01:51:59.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.661+0000 7f4ee926a640 1 -- 172.21.15.55:0/1475027555 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ee406c630 msgr2=0x7f4ee41544b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.661+0000 7f4ee926a640 1 --2- 172.21.15.55:0/1475027555 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ee406c630 0x7f4ee41544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.661+0000 7f4ee926a640 1 -- 172.21.15.55:0/1475027555 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ee4150ae0 msgr2=0x7f4ee4154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.661+0000 7f4ee926a640 1 --2- 172.21.15.55:0/1475027555 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ee4150ae0 0x7f4ee4154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.661+0000 7f4ee926a640 1 -- 172.21.15.55:0/1475027555 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ee4163b70 con 0x7f4ee40c3520 2024-03-19T01:51:59.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.665+0000 7f193c67e640 1 Processor -- start 2024-03-19T01:51:59.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.665+0000 7f193c67e640 1 -- start start 2024-03-19T01:51:59.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.665+0000 7f193c67e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f193406c3b0 0x7f19340c5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.665+0000 7f193c67e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f193406c790 0x7f19341534d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.665+0000 7f193c67e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1934072190 0x7f1934160da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.665+0000 7f193c67e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f193416c7c0 con 0x7f193406c790 2024-03-19T01:51:59.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.665+0000 7f193c67e640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f193416c640 con 0x7f193406c3b0 2024-03-19T01:51:59.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.665+0000 7f193c67e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f193416c940 con 0x7f1934072190 2024-03-19T01:51:59.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.665+0000 7f11a9ec2640 1 --2- 172.21.15.55:0/414744305 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11a40c73c0 0x7f11a40c77f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f11a415b510 tx=0x7f119c01f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e7da9409df19de8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.673+0000 7f4ee926a640 1 --2- 172.21.15.55:0/1475027555 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ee40c3520 0x7f4ee4141030 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4ed400d800 tx=0x7f4ed400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.673+0000 7f119bfff640 1 -- 172.21.15.55:0/414744305 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119c0b4070 con 0x7f11a40c73c0 2024-03-19T01:51:59.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.673+0000 7f119bfff640 1 -- 172.21.15.55:0/414744305 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f119c0af400 con 0x7f11a40c73c0 2024-03-19T01:51:59.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.673+0000 7f119bfff640 1 -- 172.21.15.55:0/414744305 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119c0af7d0 con 0x7f11a40c73c0 2024-03-19T01:51:59.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.673+0000 7f11ab925640 1 -- 172.21.15.55:0/414744305 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11a40c73c0 msgr2=0x7f11a40c77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.673+0000 7f11ab925640 1 --2- 172.21.15.55:0/414744305 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11a40c73c0 0x7f11a40c77f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f11a415b510 tx=0x7f119c01f1a0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.673+0000 7f11ab925640 1 -- 172.21.15.55:0/414744305 shutdown_connections 2024-03-19T01:51:59.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.673+0000 7f11ab925640 1 --2- 172.21.15.55:0/414744305 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11a40c73c0 0x7f11a40c77f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.673+0000 7f11ab925640 1 --2- 172.21.15.55:0/414744305 >> v2:172.21.15.84:3300/0 conn(0x7f11a41510f0 0x7f11a40c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.673+0000 7f11ab925640 1 -- 172.21.15.55:0/414744305 >> 172.21.15.55:0/414744305 conn(0x7f11a40cc9c0 msgr2=0x7f11a40ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:59.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.673+0000 7f11ab925640 1 -- 172.21.15.55:0/414744305 shutdown_connections 2024-03-19T01:51:59.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.673+0000 7f11ab925640 1 -- 172.21.15.55:0/414744305 wait complete. 2024-03-19T01:51:59.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.677+0000 7f11ab925640 1 Processor -- start 2024-03-19T01:51:59.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.677+0000 7f11ab925640 1 -- start start 2024-03-19T01:51:59.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.677+0000 7f11ab925640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f11a40c73c0 0x7f11a413afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.677+0000 7f11ab925640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11a41510f0 0x7f11a406cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.677+0000 7f11ab925640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f11a415a410 0x7f11a40cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.677+0000 7f11ab925640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f11a40c44e0 con 0x7f11a41510f0 2024-03-19T01:51:59.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.677+0000 7f11ab925640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f11a40c4200 con 0x7f11a415a410 2024-03-19T01:51:59.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.677+0000 7f11ab925640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f11a40c6e30 con 0x7f11a40c73c0 2024-03-19T01:51:59.681 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.681+0000 7f1939c19640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f193406c790 0x7f19341534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.681+0000 7f1939c19640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f193406c790 0x7f19341534d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53066/0 (socket says 172.21.15.55:53066) 2024-03-19T01:51:59.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.681+0000 7f1939c19640 1 -- 172.21.15.55:0/4098951980 learned_addr learned my addr 172.21.15.55:0/4098951980 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:59.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.681+0000 7f11a8ec0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11a41510f0 0x7f11a406cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.685+0000 7f4eda7fc640 1 -- 172.21.15.55:0/1475027555 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed400b6c0 con 0x7f4ee40c3520 2024-03-19T01:51:59.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.685+0000 7f4eda7fc640 1 -- 172.21.15.55:0/1475027555 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ed4019070 con 0x7f4ee40c3520 2024-03-19T01:51:59.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.685+0000 7f4eda7fc640 1 -- 172.21.15.55:0/1475027555 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed400ba60 con 0x7f4ee40c3520 2024-03-19T01:51:59.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.685+0000 7f11a8ec0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11a41510f0 0x7f11a406cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53070/0 (socket says 172.21.15.55:53070) 2024-03-19T01:51:59.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.685+0000 7f11a8ec0640 1 -- 172.21.15.55:0/3643999475 learned_addr learned my addr 172.21.15.55:0/3643999475 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:51:59.686 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.685+0000 7f4eebccf640 1 -- 172.21.15.55:0/1475027555 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ee4145f20 con 0x7f4ee40c3520 2024-03-19T01:51:59.686 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.685+0000 7f4eebccf640 1 -- 172.21.15.55:0/1475027555 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ee4162dd0 con 0x7f4ee40c3520 2024-03-19T01:51:59.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.685+0000 7f4eebccf640 1 -- 172.21.15.55:0/1475027555 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ee4068a40 con 0x7f4ee40c3520 2024-03-19T01:51:59.688 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.689+0000 7f1939c19640 1 -- 172.21.15.55:0/4098951980 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1934072190 msgr2=0x7f1934160da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.688 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.689+0000 7f1939c19640 1 --2- 172.21.15.55:0/4098951980 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1934072190 0x7f1934160da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.688 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.689+0000 7f1939c19640 1 -- 172.21.15.55:0/4098951980 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f193406c3b0 msgr2=0x7f19340c5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.688 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.689+0000 7f1939c19640 1 --2- 172.21.15.55:0/4098951980 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f193406c3b0 0x7f19340c5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.688 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.689+0000 7f1939c19640 1 -- 172.21.15.55:0/4098951980 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19340c50a0 con 0x7f193406c790 2024-03-19T01:51:59.692 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.693+0000 7f4031ffb640 1 -- 172.21.15.55:0/1954656492 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f40380c6070 con 0x7f403c1644e0 2024-03-19T01:51:59.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.693+0000 7f4031ffb640 1 --2- 172.21.15.55:0/1954656492 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f40200676b0 0x7f4020069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.693+0000 7f4041669640 1 --2- 172.21.15.55:0/1954656492 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f40200676b0 0x7f4020069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.693+0000 7f4041669640 1 --2- 172.21.15.55:0/1954656492 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f40200676b0 0x7f4020069b70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f402c009690 tx=0x7f402c006c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.696 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.693+0000 7f11a8ec0640 1 -- 172.21.15.55:0/3643999475 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f11a40c73c0 msgr2=0x7f11a413afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.696 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.693+0000 7f11a8ec0640 1 --2- 172.21.15.55:0/3643999475 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f11a40c73c0 0x7f11a413afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.696 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.693+0000 7f11a8ec0640 1 -- 172.21.15.55:0/3643999475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f11a415a410 msgr2=0x7f11a40cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.696 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.693+0000 7f11a8ec0640 1 --2- 172.21.15.55:0/3643999475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f11a415a410 0x7f11a40cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.696 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.693+0000 7f11a8ec0640 1 -- 172.21.15.55:0/3643999475 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a406c010 con 0x7f11a41510f0 2024-03-19T01:51:59.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.697+0000 7f4031ffb640 1 -- 172.21.15.55:0/1954656492 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f40380f19e0 con 0x7f403c1644e0 2024-03-19T01:51:59.701 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.701+0000 7f1939c19640 1 --2- 172.21.15.55:0/4098951980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f193406c790 0x7f19341534d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f192400b3e0 tx=0x7f192400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.717+0000 7f4031ffb640 1 -- 172.21.15.55:0/1954656492 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f40380cd020 con 0x7f403c1644e0 2024-03-19T01:51:59.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.733+0000 7f4eda7fc640 1 -- 172.21.15.55:0/1475027555 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f4ed400bc00 con 0x7f4ee40c3520 2024-03-19T01:51:59.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.733+0000 7f4eda7fc640 1 --2- 172.21.15.55:0/1475027555 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4ec8067690 0x7f4ec8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.733+0000 7f11a8ec0640 1 --2- 172.21.15.55:0/3643999475 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11a41510f0 0x7f11a406cdb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f119400c490 tx=0x7f1194001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.733+0000 7f4ee9a6b640 1 --2- 172.21.15.55:0/1475027555 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4ec8067690 0x7f4ec8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.735 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.733+0000 7f4ee9a6b640 1 --2- 172.21.15.55:0/1475027555 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4ec8067690 0x7f4ec8069b50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4ee00ad000 tx=0x7f4ee0004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.738 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.737+0000 7f4eda7fc640 1 -- 172.21.15.55:0/1475027555 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f4ed4054e80 con 0x7f4ee40c3520 2024-03-19T01:51:59.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.793+0000 7f2308963640 1 -- 172.21.15.55:0/3815110467 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f23040c4a10 con 0x7f23040c42a0 2024-03-19T01:51:59.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.817+0000 7f5ad06bf640 1 -- 172.21.15.55:0/3899978323 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f5ab0005860 con 0x7f5ac0046c50 2024-03-19T01:51:59.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.961+0000 7f40438cd640 1 -- 172.21.15.55:0/1954656492 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f403c08bb40 con 0x7f403c1644e0 2024-03-19T01:51:59.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.973+0000 7f4031ffb640 1 -- 172.21.15.55:0/1954656492 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f403801aa90 con 0x7f403c1644e0 2024-03-19T01:51:59.973 INFO:teuthology.orchestra.run.smithi055.stdout:64424509461 2024-03-19T01:51:59.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.973+0000 7f40438cd640 1 -- 172.21.15.55:0/1954656492 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f40200676b0 msgr2=0x7f4020069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.973+0000 7f40438cd640 1 --2- 172.21.15.55:0/1954656492 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f40200676b0 0x7f4020069b70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f402c009690 tx=0x7f402c006c00 comp rx=0 tx=0).stop 2024-03-19T01:51:59.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.973+0000 7f40438cd640 1 -- 172.21.15.55:0/1954656492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f403c1644e0 msgr2=0x7f403c20dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.973+0000 7f40438cd640 1 --2- 172.21.15.55:0/1954656492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f403c1644e0 0x7f403c20dd60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f403801f5e0 tx=0x7f403801dcd0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.973+0000 7f40438cd640 1 -- 172.21.15.55:0/1954656492 shutdown_connections 2024-03-19T01:51:59.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.973+0000 7f40438cd640 1 --2- 172.21.15.55:0/1954656492 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f40200676b0 0x7f4020069b70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.973+0000 7f40438cd640 1 --2- 172.21.15.55:0/1954656492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f403c1644e0 0x7f403c20dd60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.973+0000 7f40438cd640 1 --2- 172.21.15.55:0/1954656492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f403c0c7f10 0x7f403c0c5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.973+0000 7f40438cd640 1 --2- 172.21.15.55:0/1954656492 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f403c071a80 0x7f403c168b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.973+0000 7f40438cd640 1 -- 172.21.15.55:0/1954656492 >> 172.21.15.55:0/1954656492 conn(0x7f403c06d250 msgr2=0x7f403c06d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:59.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.973+0000 7f40438cd640 1 -- 172.21.15.55:0/1954656492 shutdown_connections 2024-03-19T01:51:59.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.973+0000 7f40438cd640 1 -- 172.21.15.55:0/1954656492 wait complete. 2024-03-19T01:51:59.981 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.981+0000 7f192affd640 1 -- 172.21.15.55:0/4098951980 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1924015020 con 0x7f193406c790 2024-03-19T01:51:59.981 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.981+0000 7f192affd640 1 -- 172.21.15.55:0/4098951980 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1924011850 con 0x7f193406c790 2024-03-19T01:51:59.981 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.981+0000 7f193c67e640 1 -- 172.21.15.55:0/4098951980 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1934154620 con 0x7f193406c790 2024-03-19T01:51:59.981 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.981+0000 7f192affd640 1 -- 172.21.15.55:0/4098951980 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1924011c40 con 0x7f193406c790 2024-03-19T01:51:59.981 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.981+0000 7f193c67e640 1 -- 172.21.15.55:0/4098951980 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19341fbc00 con 0x7f193406c790 2024-03-19T01:51:59.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.981+0000 7f193c67e640 1 -- 172.21.15.55:0/4098951980 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1934068a40 con 0x7f193406c790 2024-03-19T01:51:59.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.985+0000 7ff36a7fc640 1 -- 172.21.15.55:0/2354848291 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ff364018020 con 0x7ff374070780 2024-03-19T01:51:59.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.985+0000 7ff36a7fc640 1 --2- 172.21.15.55:0/2354848291 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff3580676b0 0x7ff358069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:51:59.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.985+0000 7ff379321640 1 --2- 172.21.15.55:0/2354848291 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff3580676b0 0x7ff358069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:51:59.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.985+0000 7ff379321640 1 --2- 172.21.15.55:0/2354848291 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff3580676b0 0x7ff358069b70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff370007fd0 tx=0x7ff370009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:51:59.991 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509463 got 64424509461 for osd.7 2024-03-19T01:51:59.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.993+0000 7f4eda7fc640 1 -- 172.21.15.55:0/1475027555 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4ed40574e0 con 0x7f4ee40c3520 2024-03-19T01:51:59.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.993+0000 7f1199ffb640 1 -- 172.21.15.55:0/3643999475 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1194017070 con 0x7f11a41510f0 2024-03-19T01:51:59.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.993+0000 7f1199ffb640 1 -- 172.21.15.55:0/3643999475 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1194003600 con 0x7f11a41510f0 2024-03-19T01:51:59.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.993+0000 7f1199ffb640 1 -- 172.21.15.55:0/3643999475 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1194003a30 con 0x7f11a41510f0 2024-03-19T01:51:59.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.993+0000 7f11ab925640 1 -- 172.21.15.55:0/3643999475 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11a406c2d0 con 0x7f11a41510f0 2024-03-19T01:51:59.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.993+0000 7f11ab925640 1 -- 172.21.15.55:0/3643999475 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11a406c890 con 0x7f11a41510f0 2024-03-19T01:51:59.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.993+0000 7ff36a7fc640 1 -- 172.21.15.55:0/2354848291 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7ff364054270 con 0x7ff374070780 2024-03-19T01:51:59.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.997+0000 7f11ab925640 1 -- 172.21.15.55:0/3643999475 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11a4068a40 con 0x7f11a41510f0 2024-03-19T01:51:59.997 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.997+0000 7fef237fe640 1 -- 172.21.15.55:0/3162143887 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef18008be0 con 0x7fef1c0d5000 2024-03-19T01:51:59.997 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.997+0000 7fef237fe640 1 -- 172.21.15.55:0/3162143887 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef18008d80 con 0x7fef1c0d5000 2024-03-19T01:51:59.997 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.997+0000 7fef237fe640 1 -- 172.21.15.55:0/3162143887 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef180a9b10 con 0x7fef1c0d5000 2024-03-19T01:51:59.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.997+0000 7fef29bba640 1 -- 172.21.15.55:0/3162143887 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fef1c0d5000 msgr2=0x7fef1c0db7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:51:59.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.997+0000 7fef29bba640 1 --2- 172.21.15.55:0/3162143887 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fef1c0d5000 0x7fef1c0db7c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fef18007360 tx=0x7fef1801fd20 comp rx=0 tx=0).stop 2024-03-19T01:51:59.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.997+0000 7fef29bba640 1 -- 172.21.15.55:0/3162143887 shutdown_connections 2024-03-19T01:51:59.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.997+0000 7fef29bba640 1 --2- 172.21.15.55:0/3162143887 >> v2:172.21.15.84:3300/0 conn(0x7fef1c0458a0 0x7fef1c006340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.997+0000 7fef29bba640 1 --2- 172.21.15.55:0/3162143887 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fef1c0d5000 0x7fef1c0db7c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:51:59.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.997+0000 7fef29bba640 1 -- 172.21.15.55:0/3162143887 >> 172.21.15.55:0/3162143887 conn(0x7fef1c0505e0 msgr2=0x7fef1c0509f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:51:59.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.997+0000 7fef29bba640 1 -- 172.21.15.55:0/3162143887 shutdown_connections 2024-03-19T01:51:59.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.997+0000 7fef29bba640 1 -- 172.21.15.55:0/3162143887 wait complete. 2024-03-19T01:52:00.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.997+0000 7fef29bba640 1 Processor -- start 2024-03-19T01:52:00.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:51:59.997+0000 7fef29bba640 1 -- start start 2024-03-19T01:52:00.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.001+0000 7fef29bba640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fef1c0458a0 0x7fef1c171760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:00.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.001+0000 7fef29bba640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef1c0470a0 0x7fef1c006be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:00.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.001+0000 7fef29bba640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef1c0d5000 0x7fef1c048210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:00.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.001+0000 7fef29bba640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c0d53e0 con 0x7fef1c0470a0 2024-03-19T01:52:00.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.001+0000 7fef29bba640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c0468a0 con 0x7fef1c0d5000 2024-03-19T01:52:00.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.001+0000 7fef29bba640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fef1c04bc00 con 0x7fef1c0458a0 2024-03-19T01:52:00.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.001+0000 7fef293b9640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef1c0d5000 0x7fef1c048210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:00.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.001+0000 7fef293b9640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef1c0d5000 0x7fef1c048210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:55896/0 (socket says 172.21.15.55:55896) 2024-03-19T01:52:00.003 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.001+0000 7fef293b9640 1 -- 172.21.15.55:0/16399306 learned_addr learned my addr 172.21.15.55:0/16399306 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:00.008 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.009+0000 7fef293b9640 1 -- 172.21.15.55:0/16399306 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fef1c0458a0 msgr2=0x7fef1c171760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:00.008 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.009+0000 7fef293b9640 1 --2- 172.21.15.55:0/16399306 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fef1c0458a0 0x7fef1c171760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.008 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.009+0000 7fef293b9640 1 -- 172.21.15.55:0/16399306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef1c0470a0 msgr2=0x7fef1c006be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:00.008 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.009+0000 7fef293b9640 1 --2- 172.21.15.55:0/16399306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef1c0470a0 0x7fef1c006be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.009+0000 7fef293b9640 1 -- 172.21.15.55:0/16399306 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef1c04a130 con 0x7fef1c0d5000 2024-03-19T01:52:00.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.009+0000 7ff36a7fc640 1 -- 172.21.15.55:0/2354848291 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ff36401f020 con 0x7ff374070780 2024-03-19T01:52:00.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.013+0000 7fef293b9640 1 --2- 172.21.15.55:0/16399306 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef1c0d5000 0x7fef1c048210 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fef2406a310 tx=0x7fef24000b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:00.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.021+0000 7fef217fa640 1 -- 172.21.15.55:0/16399306 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef24073040 con 0x7fef1c0d5000 2024-03-19T01:52:00.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.021+0000 7fef217fa640 1 -- 172.21.15.55:0/16399306 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef240731e0 con 0x7fef1c0d5000 2024-03-19T01:52:00.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.021+0000 7fef29bba640 1 -- 172.21.15.55:0/16399306 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef1c04a9b0 con 0x7fef1c0d5000 2024-03-19T01:52:00.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.021+0000 7fef217fa640 1 -- 172.21.15.55:0/16399306 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef240735d0 con 0x7fef1c0d5000 2024-03-19T01:52:00.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.021+0000 7fef29bba640 1 -- 172.21.15.55:0/16399306 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef1c04a6d0 con 0x7fef1c0d5000 2024-03-19T01:52:00.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.021+0000 7fef29bba640 1 -- 172.21.15.55:0/16399306 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef1c171ca0 con 0x7fef1c0d5000 2024-03-19T01:52:00.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.029+0000 7f22eb7fe640 1 -- 172.21.15.55:0/3815110467 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f22fc0567e0 con 0x7f23040c42a0 2024-03-19T01:52:00.031 INFO:teuthology.orchestra.run.smithi055.stdout:64424509463 2024-03-19T01:52:00.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.029+0000 7f2308963640 1 -- 172.21.15.55:0/3815110467 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f22f00676b0 msgr2=0x7f22f0069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:00.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.029+0000 7f2308963640 1 --2- 172.21.15.55:0/3815110467 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f22f00676b0 0x7f22f0069b70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f23041f0260 tx=0x7f22f4009e50 comp rx=0 tx=0).stop 2024-03-19T01:52:00.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.033+0000 7f2308963640 1 -- 172.21.15.55:0/3815110467 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f23040c42a0 msgr2=0x7f2304144ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:00.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.033+0000 7f2308963640 1 --2- 172.21.15.55:0/3815110467 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f23040c42a0 0x7f2304144ff0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00c7a0 tx=0x7f22fc00cc60 comp rx=0 tx=0).stop 2024-03-19T01:52:00.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.033+0000 7f2308963640 1 -- 172.21.15.55:0/3815110467 shutdown_connections 2024-03-19T01:52:00.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.033+0000 7f2308963640 1 --2- 172.21.15.55:0/3815110467 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f22f00676b0 0x7f22f0069b70 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.033+0000 7f2308963640 1 --2- 172.21.15.55:0/3815110467 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23040c9b30 0x7f2304149a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.033+0000 7f2308963640 1 --2- 172.21.15.55:0/3815110467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23040c6550 0x7f2304145910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.033+0000 7f2308963640 1 --2- 172.21.15.55:0/3815110467 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f23040c42a0 0x7f2304144ff0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.033+0000 7f2308963640 1 -- 172.21.15.55:0/3815110467 >> 172.21.15.55:0/3815110467 conn(0x7f2304067400 msgr2=0x7f230406b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:00.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.033+0000 7f2308963640 1 -- 172.21.15.55:0/3815110467 shutdown_connections 2024-03-19T01:52:00.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.033+0000 7f2308963640 1 -- 172.21.15.55:0/3815110467 wait complete. 2024-03-19T01:52:00.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.049+0000 7fef217fa640 1 -- 172.21.15.55:0/16399306 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fef24073770 con 0x7fef1c0d5000 2024-03-19T01:52:00.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.049+0000 7fef217fa640 1 --2- 172.21.15.55:0/16399306 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fef08067660 0x7fef08069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:00.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.049+0000 7fef28bb8640 1 --2- 172.21.15.55:0/16399306 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fef08067660 0x7fef08069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:00.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.049+0000 7fef28bb8640 1 --2- 172.21.15.55:0/16399306 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fef08067660 0x7fef08069b20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fef180b2000 tx=0x7fef18004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:00.056 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509465 got 64424509463 for osd.1 2024-03-19T01:52:00.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.317+0000 7f4eebccf640 1 -- 172.21.15.55:0/1475027555 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f4ee414d370 con 0x7f4ee40c3520 2024-03-19T01:52:00.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.329+0000 7ff37bd86640 1 -- 172.21.15.55:0/2354848291 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7ff3740c84b0 con 0x7ff374070780 2024-03-19T01:52:00.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.365+0000 7ff36a7fc640 1 -- 172.21.15.55:0/2354848291 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff3640568d0 con 0x7ff374070780 2024-03-19T01:52:00.368 INFO:teuthology.orchestra.run.smithi055.stdout:64424509462 2024-03-19T01:52:00.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.369+0000 7ff37bd86640 1 -- 172.21.15.55:0/2354848291 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff3580676b0 msgr2=0x7ff358069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:00.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.369+0000 7ff37bd86640 1 --2- 172.21.15.55:0/2354848291 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff3580676b0 0x7ff358069b70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff370007fd0 tx=0x7ff370009040 comp rx=0 tx=0).stop 2024-03-19T01:52:00.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.369+0000 7ff37bd86640 1 -- 172.21.15.55:0/2354848291 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff374070780 msgr2=0x7ff3740cb8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:00.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.369+0000 7ff37bd86640 1 --2- 172.21.15.55:0/2354848291 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff374070780 0x7ff3740cb8b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff36400b410 tx=0x7ff36400b810 comp rx=0 tx=0).stop 2024-03-19T01:52:00.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.369+0000 7ff37bd86640 1 -- 172.21.15.55:0/2354848291 shutdown_connections 2024-03-19T01:52:00.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.369+0000 7ff37bd86640 1 --2- 172.21.15.55:0/2354848291 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff3580676b0 0x7ff358069b70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.369+0000 7ff37bd86640 1 --2- 172.21.15.55:0/2354848291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff374152120 0x7ff374143c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.369+0000 7ff37bd86640 1 --2- 172.21.15.55:0/2354848291 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3740c9610 0x7ff3740b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.369+0000 7ff37bd86640 1 --2- 172.21.15.55:0/2354848291 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff374070780 0x7ff3740cb8b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.369+0000 7ff37bd86640 1 -- 172.21.15.55:0/2354848291 >> 172.21.15.55:0/2354848291 conn(0x7ff374067600 msgr2=0x7ff37414f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:00.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.369+0000 7ff37bd86640 1 -- 172.21.15.55:0/2354848291 shutdown_connections 2024-03-19T01:52:00.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.369+0000 7ff37bd86640 1 -- 172.21.15.55:0/2354848291 wait complete. 2024-03-19T01:52:00.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.369+0000 7fef217fa640 1 -- 172.21.15.55:0/16399306 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fef240d01f0 con 0x7fef1c0d5000 2024-03-19T01:52:00.384 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.385+0000 7fef217fa640 1 -- 172.21.15.55:0/16399306 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fef1c171ca0 con 0x7fef1c0d5000 2024-03-19T01:52:00.386 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509464 got 64424509462 for osd.0 2024-03-19T01:52:00.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.393+0000 7f192affd640 1 -- 172.21.15.55:0/4098951980 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1924018020 con 0x7f193406c790 2024-03-19T01:52:00.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.393+0000 7f192affd640 1 --2- 172.21.15.55:0/4098951980 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f19180676b0 0x7f1918069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:00.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.393+0000 7f193a41a640 1 --2- 172.21.15.55:0/4098951980 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f19180676b0 0x7f1918069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:00.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.393+0000 7f193a41a640 1 --2- 172.21.15.55:0/4098951980 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f19180676b0 0x7f1918069b70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f19300af000 tx=0x7f1930004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:00.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.417+0000 7f192affd640 1 -- 172.21.15.55:0/4098951980 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f1924011de0 con 0x7f193406c790 2024-03-19T01:52:00.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.417+0000 7f192affd640 1 -- 172.21.15.55:0/4098951980 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f192401f020 con 0x7f193406c790 2024-03-19T01:52:00.422 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.421+0000 7f5abeffd640 1 -- 172.21.15.55:0/3899978323 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5ac400d040 con 0x7f5ac0046c50 2024-03-19T01:52:00.422 INFO:teuthology.orchestra.run.smithi055.stdout:60129542166 2024-03-19T01:52:00.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.425+0000 7f5ad06bf640 1 -- 172.21.15.55:0/3899978323 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5aac067660 msgr2=0x7f5aac069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:00.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.425+0000 7f5ad06bf640 1 --2- 172.21.15.55:0/3899978323 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5aac067660 0x7f5aac069b20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80bb000 tx=0x7f5ab8008040 comp rx=0 tx=0).stop 2024-03-19T01:52:00.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.425+0000 7f5ad06bf640 1 -- 172.21.15.55:0/3899978323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ac0046c50 msgr2=0x7f5ac00ff8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:00.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.425+0000 7f5ad06bf640 1 --2- 172.21.15.55:0/3899978323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ac0046c50 0x7f5ac00ff8b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400b440 tx=0x7f5ac400b900 comp rx=0 tx=0).stop 2024-03-19T01:52:00.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.425+0000 7f5ad06bf640 1 -- 172.21.15.55:0/3899978323 shutdown_connections 2024-03-19T01:52:00.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.425+0000 7f5ad06bf640 1 --2- 172.21.15.55:0/3899978323 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5aac067660 0x7f5aac069b20 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.425+0000 7f5ad06bf640 1 --2- 172.21.15.55:0/3899978323 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5ac00f6bc0 0x7f5ac00ffdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.425+0000 7f5ad06bf640 1 --2- 172.21.15.55:0/3899978323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5ac0046c50 0x7f5ac00ff8b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.425+0000 7f5ad06bf640 1 --2- 172.21.15.55:0/3899978323 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5ac0045450 0x7f5ac0195cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.425+0000 7f5ad06bf640 1 -- 172.21.15.55:0/3899978323 >> 172.21.15.55:0/3899978323 conn(0x7f5ac0050190 msgr2=0x7f5ac00f44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:00.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.425+0000 7f5ad06bf640 1 -- 172.21.15.55:0/3899978323 shutdown_connections 2024-03-19T01:52:00.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.425+0000 7f5ad06bf640 1 -- 172.21.15.55:0/3899978323 wait complete. 2024-03-19T01:52:00.447 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542167 got 60129542166 for osd.5 2024-03-19T01:52:00.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.545+0000 7fef29bba640 1 -- 172.21.15.55:0/16399306 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fef1c005720 con 0x7fef1c0d5000 2024-03-19T01:52:00.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.553+0000 7fef217fa640 1 -- 172.21.15.55:0/16399306 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fef240a2080 con 0x7fef1c0d5000 2024-03-19T01:52:00.553 INFO:teuthology.orchestra.run.smithi055.stdout:64424509463 2024-03-19T01:52:00.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.553+0000 7fef29bba640 1 -- 172.21.15.55:0/16399306 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fef08067660 msgr2=0x7fef08069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:00.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.553+0000 7fef29bba640 1 --2- 172.21.15.55:0/16399306 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fef08067660 0x7fef08069b20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fef180b2000 tx=0x7fef18004740 comp rx=0 tx=0).stop 2024-03-19T01:52:00.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.553+0000 7fef29bba640 1 -- 172.21.15.55:0/16399306 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef1c0d5000 msgr2=0x7fef1c048210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:00.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.553+0000 7fef29bba640 1 --2- 172.21.15.55:0/16399306 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef1c0d5000 0x7fef1c048210 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fef2406a310 tx=0x7fef24000b10 comp rx=0 tx=0).stop 2024-03-19T01:52:00.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.553+0000 7fef29bba640 1 -- 172.21.15.55:0/16399306 shutdown_connections 2024-03-19T01:52:00.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.553+0000 7fef29bba640 1 --2- 172.21.15.55:0/16399306 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fef08067660 0x7fef08069b20 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.553+0000 7fef29bba640 1 --2- 172.21.15.55:0/16399306 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef1c0d5000 0x7fef1c048210 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.553+0000 7fef29bba640 1 --2- 172.21.15.55:0/16399306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef1c0470a0 0x7fef1c006be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.553+0000 7fef29bba640 1 --2- 172.21.15.55:0/16399306 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fef1c0458a0 0x7fef1c171760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.553+0000 7fef29bba640 1 -- 172.21.15.55:0/16399306 >> 172.21.15.55:0/16399306 conn(0x7fef1c0505e0 msgr2=0x7fef1c0d9dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:00.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.553+0000 7fef29bba640 1 -- 172.21.15.55:0/16399306 shutdown_connections 2024-03-19T01:52:00.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.553+0000 7fef29bba640 1 -- 172.21.15.55:0/16399306 wait complete. 2024-03-19T01:52:00.570 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509465 got 64424509463 for osd.6 2024-03-19T01:52:00.576 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.577+0000 7f193c67e640 1 -- 172.21.15.55:0/4098951980 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f19340c8700 con 0x7f193406c790 2024-03-19T01:52:00.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.705+0000 7f1199ffb640 1 -- 172.21.15.55:0/3643999475 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f119401a070 con 0x7f11a41510f0 2024-03-19T01:52:00.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.705+0000 7f1199ffb640 1 --2- 172.21.15.55:0/3643999475 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f11880676b0 0x7f1188069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:00.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.709+0000 7f11a96c1640 1 --2- 172.21.15.55:0/3643999475 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f11880676b0 0x7f1188069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:00.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.709+0000 7f11a96c1640 1 --2- 172.21.15.55:0/3643999475 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f11880676b0 0x7f1188069b70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f11a0009730 tx=0x7f11a0006ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:00.729 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.729+0000 7f1199ffb640 1 -- 172.21.15.55:0/3643999475 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f1194053730 con 0x7f11a41510f0 2024-03-19T01:52:00.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.729+0000 7f1199ffb640 1 -- 172.21.15.55:0/3643999475 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1194021020 con 0x7f11a41510f0 2024-03-19T01:52:00.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.841+0000 7f192affd640 1 -- 172.21.15.55:0/4098951980 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1924056ac0 con 0x7f193406c790 2024-03-19T01:52:00.844 INFO:teuthology.orchestra.run.smithi055.stdout:60129542167 2024-03-19T01:52:00.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.845+0000 7f193c67e640 1 -- 172.21.15.55:0/4098951980 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f19180676b0 msgr2=0x7f1918069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:00.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.845+0000 7f193c67e640 1 --2- 172.21.15.55:0/4098951980 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f19180676b0 0x7f1918069b70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f19300af000 tx=0x7f1930004740 comp rx=0 tx=0).stop 2024-03-19T01:52:00.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.845+0000 7f193c67e640 1 -- 172.21.15.55:0/4098951980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f193406c790 msgr2=0x7f19341534d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:00.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.845+0000 7f193c67e640 1 --2- 172.21.15.55:0/4098951980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f193406c790 0x7f19341534d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f192400b3e0 tx=0x7f192400b8a0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.845+0000 7f193c67e640 1 -- 172.21.15.55:0/4098951980 shutdown_connections 2024-03-19T01:52:00.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.845+0000 7f193c67e640 1 --2- 172.21.15.55:0/4098951980 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f19180676b0 0x7f1918069b70 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.845+0000 7f193c67e640 1 --2- 172.21.15.55:0/4098951980 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1934072190 0x7f1934160da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.845+0000 7f193c67e640 1 --2- 172.21.15.55:0/4098951980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f193406c790 0x7f19341534d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.845+0000 7f193c67e640 1 --2- 172.21.15.55:0/4098951980 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f193406c3b0 0x7f19340c5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.845+0000 7f193c67e640 1 -- 172.21.15.55:0/4098951980 >> 172.21.15.55:0/4098951980 conn(0x7f1934071590 msgr2=0x7f19340cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:00.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.845+0000 7f193c67e640 1 -- 172.21.15.55:0/4098951980 shutdown_connections 2024-03-19T01:52:00.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.845+0000 7f193c67e640 1 -- 172.21.15.55:0/4098951980 wait complete. 2024-03-19T01:52:00.866 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542168 got 60129542167 for osd.3 2024-03-19T01:52:00.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.865+0000 7f4eda7fc640 1 -- 172.21.15.55:0/1475027555 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4ed4056de0 con 0x7f4ee40c3520 2024-03-19T01:52:00.867 INFO:teuthology.orchestra.run.smithi055.stdout:64424509462 2024-03-19T01:52:00.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.869+0000 7f4eebccf640 1 -- 172.21.15.55:0/1475027555 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4ec8067690 msgr2=0x7f4ec8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:00.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.869+0000 7f4eebccf640 1 --2- 172.21.15.55:0/1475027555 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4ec8067690 0x7f4ec8069b50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4ee00ad000 tx=0x7f4ee0004740 comp rx=0 tx=0).stop 2024-03-19T01:52:00.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.869+0000 7f4eebccf640 1 -- 172.21.15.55:0/1475027555 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ee40c3520 msgr2=0x7f4ee4141030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:00.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.869+0000 7f4eebccf640 1 --2- 172.21.15.55:0/1475027555 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ee40c3520 0x7f4ee4141030 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4ed400d800 tx=0x7f4ed400dcc0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.869+0000 7f4eebccf640 1 -- 172.21.15.55:0/1475027555 shutdown_connections 2024-03-19T01:52:00.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.869+0000 7f4eebccf640 1 --2- 172.21.15.55:0/1475027555 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4ec8067690 0x7f4ec8069b50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.869+0000 7f4eebccf640 1 --2- 172.21.15.55:0/1475027555 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ee4150ae0 0x7f4ee4154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.869+0000 7f4eebccf640 1 --2- 172.21.15.55:0/1475027555 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ee40c3520 0x7f4ee4141030 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.869+0000 7f4eebccf640 1 --2- 172.21.15.55:0/1475027555 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ee406c630 0x7f4ee41544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.869+0000 7f4eebccf640 1 -- 172.21.15.55:0/1475027555 >> 172.21.15.55:0/1475027555 conn(0x7f4ee406e5a0 msgr2=0x7f4ee4142700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:00.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.869+0000 7f4eebccf640 1 -- 172.21.15.55:0/1475027555 shutdown_connections 2024-03-19T01:52:00.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.869+0000 7f4eebccf640 1 -- 172.21.15.55:0/1475027555 wait complete. 2024-03-19T01:52:00.885 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509464 got 64424509462 for osd.4 2024-03-19T01:52:00.926 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.925+0000 7f11ab925640 1 -- 172.21.15.55:0/3643999475 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f11a40c64e0 con 0x7f11a41510f0 2024-03-19T01:52:00.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.961+0000 7f1199ffb640 1 -- 172.21.15.55:0/3643999475 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1194055d90 con 0x7f11a41510f0 2024-03-19T01:52:00.962 INFO:teuthology.orchestra.run.smithi055.stdout:60129542166 2024-03-19T01:52:00.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.965+0000 7f11ab925640 1 -- 172.21.15.55:0/3643999475 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f11880676b0 msgr2=0x7f1188069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:00.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.965+0000 7f11ab925640 1 --2- 172.21.15.55:0/3643999475 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f11880676b0 0x7f1188069b70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f11a0009730 tx=0x7f11a0006ca0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.965+0000 7f11ab925640 1 -- 172.21.15.55:0/3643999475 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11a41510f0 msgr2=0x7f11a406cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:00.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.965+0000 7f11ab925640 1 --2- 172.21.15.55:0/3643999475 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11a41510f0 0x7f11a406cdb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f119400c490 tx=0x7f1194001e80 comp rx=0 tx=0).stop 2024-03-19T01:52:00.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.965+0000 7f11ab925640 1 -- 172.21.15.55:0/3643999475 shutdown_connections 2024-03-19T01:52:00.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.965+0000 7f11ab925640 1 --2- 172.21.15.55:0/3643999475 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f11880676b0 0x7f1188069b70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.965+0000 7f11ab925640 1 --2- 172.21.15.55:0/3643999475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f11a415a410 0x7f11a40cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.965+0000 7f11ab925640 1 --2- 172.21.15.55:0/3643999475 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11a41510f0 0x7f11a406cdb0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.965+0000 7f11ab925640 1 --2- 172.21.15.55:0/3643999475 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f11a40c73c0 0x7f11a413afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:00.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.965+0000 7f11ab925640 1 -- 172.21.15.55:0/3643999475 >> 172.21.15.55:0/3643999475 conn(0x7f11a40cc9c0 msgr2=0x7f11a414fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:00.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.965+0000 7f11ab925640 1 -- 172.21.15.55:0/3643999475 shutdown_connections 2024-03-19T01:52:00.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:00.965+0000 7f11ab925640 1 -- 172.21.15.55:0/3643999475 wait complete. 2024-03-19T01:52:00.980 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542167 got 60129542166 for osd.2 2024-03-19T01:52:00.992 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-19T01:52:01.057 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-19T01:52:01.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.133+0000 7f63cbae3640 1 Processor -- start 2024-03-19T01:52:01.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.137+0000 7f63cbae3640 1 -- start start 2024-03-19T01:52:01.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.137+0000 7f63cbae3640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f63c40ca350 0x7f63c415b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.137+0000 7f63cbae3640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f63c40c3320 0x7f63c40728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.137+0000 7f63cbae3640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63c4151330 con 0x7f63c4150f50 2024-03-19T01:52:01.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.137+0000 7f63cbae3640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f63c40c3e00 con 0x7f63c40ca350 2024-03-19T01:52:01.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.137+0000 7f63cbae3640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f63c415ac10 con 0x7f63c40c3320 2024-03-19T01:52:01.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.137+0000 7f63c907e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f63c40ca350 0x7f63c415b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:01.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.137+0000 7f63c907e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f63c40ca350 0x7f63c415b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53092/0 (socket says 172.21.15.55:53092) 2024-03-19T01:52:01.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.137+0000 7f63c907e640 1 -- 172.21.15.55:0/1955027628 learned_addr learned my addr 172.21.15.55:0/1955027628 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:01.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.141+0000 7f63c907e640 1 -- 172.21.15.55:0/1955027628 >> v2:172.21.15.84:3300/0 conn(0x7f63c40c3320 msgr2=0x7f63c40728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:01.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.141+0000 7f63c907e640 1 --2- 172.21.15.55:0/1955027628 >> v2:172.21.15.84:3300/0 conn(0x7f63c40c3320 0x7f63c40728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.141+0000 7f63c907e640 1 -- 172.21.15.55:0/1955027628 >> v1:172.21.15.84:6789/0 conn(0x7f63c4150f50 legacy=0x7f63c4157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:01.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.141+0000 7f63c907e640 1 -- 172.21.15.55:0/1955027628 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c40c3840 con 0x7f63c40ca350 2024-03-19T01:52:01.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.145+0000 7f63c907e640 1 --2- 172.21.15.55:0/1955027628 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63c40ca350 0x7f63c415b0a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f63c415ab80 tx=0x7f63b401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a2d6e54d099086 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:01.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.149+0000 7f63c887d640 1 -- 172.21.15.55:0/1955027628 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b40a6040 con 0x7f63c40ca350 2024-03-19T01:52:01.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.149+0000 7f63c887d640 1 -- 172.21.15.55:0/1955027628 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63b40af070 con 0x7f63c40ca350 2024-03-19T01:52:01.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.149+0000 7f63c887d640 1 -- 172.21.15.55:0/1955027628 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b40aa860 con 0x7f63c40ca350 2024-03-19T01:52:01.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.149+0000 7f63cbae3640 1 -- 172.21.15.55:0/1955027628 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63c40ca350 msgr2=0x7f63c415b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:01.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.149+0000 7f63cbae3640 1 --2- 172.21.15.55:0/1955027628 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63c40ca350 0x7f63c415b0a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f63c415ab80 tx=0x7f63b401c4e0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.149+0000 7f63cbae3640 1 -- 172.21.15.55:0/1955027628 shutdown_connections 2024-03-19T01:52:01.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.149+0000 7f63cbae3640 1 --2- 172.21.15.55:0/1955027628 >> v2:172.21.15.84:3300/0 conn(0x7f63c40c3320 0x7f63c40728b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.149+0000 7f63cbae3640 1 --2- 172.21.15.55:0/1955027628 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63c40ca350 0x7f63c415b0a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.149+0000 7f63cbae3640 1 -- 172.21.15.55:0/1955027628 >> 172.21.15.55:0/1955027628 conn(0x7f63c406e670 msgr2=0x7f63c406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:01.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.149+0000 7f63cbae3640 1 -- 172.21.15.55:0/1955027628 shutdown_connections 2024-03-19T01:52:01.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.149+0000 7f63cbae3640 1 -- 172.21.15.55:0/1955027628 wait complete. 2024-03-19T01:52:01.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.149+0000 7f63cbae3640 1 Processor -- start 2024-03-19T01:52:01.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.149+0000 7f63cbae3640 1 -- start start 2024-03-19T01:52:01.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.153+0000 7f63cbae3640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f63c40c3320 0x7f63c41438a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.153+0000 7f63cbae3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63c40ca350 0x7f63c4146880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.153+0000 7f63cbae3640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f63c4150f50 0x7f63c40cba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.153+0000 7f63cbae3640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f63c41639b0 con 0x7f63c40ca350 2024-03-19T01:52:01.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.153+0000 7f63cbae3640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f63c4163830 con 0x7f63c40c3320 2024-03-19T01:52:01.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.153+0000 7f63cbae3640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f63c4163b30 con 0x7f63c4150f50 2024-03-19T01:52:01.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.153+0000 7f63c907e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63c40ca350 0x7f63c4146880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:01.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.153+0000 7f63c907e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63c40ca350 0x7f63c4146880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53104/0 (socket says 172.21.15.55:53104) 2024-03-19T01:52:01.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.153+0000 7f63c907e640 1 -- 172.21.15.55:0/2526851781 learned_addr learned my addr 172.21.15.55:0/2526851781 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:01.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.157+0000 7f63c907e640 1 -- 172.21.15.55:0/2526851781 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f63c4150f50 msgr2=0x7f63c40cba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:01.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.157+0000 7f63c907e640 1 --2- 172.21.15.55:0/2526851781 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f63c4150f50 0x7f63c40cba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.157+0000 7f63c907e640 1 -- 172.21.15.55:0/2526851781 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f63c40c3320 msgr2=0x7f63c41438a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:01.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.157+0000 7f63c907e640 1 --2- 172.21.15.55:0/2526851781 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f63c40c3320 0x7f63c41438a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.157+0000 7f63c907e640 1 -- 172.21.15.55:0/2526851781 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c40c6440 con 0x7f63c40ca350 2024-03-19T01:52:01.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.157+0000 7f63c907e640 1 --2- 172.21.15.55:0/2526851781 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63c40ca350 0x7f63c4146880 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f63b401ab90 tx=0x7f63b401ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:01.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.165+0000 7f63ba7fc640 1 -- 172.21.15.55:0/2526851781 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b40bd3f0 con 0x7f63c40ca350 2024-03-19T01:52:01.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.165+0000 7f63ba7fc640 1 -- 172.21.15.55:0/2526851781 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63b40bd5b0 con 0x7f63c40ca350 2024-03-19T01:52:01.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.165+0000 7f63cbae3640 1 -- 172.21.15.55:0/2526851781 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63c4147360 con 0x7f63c40ca350 2024-03-19T01:52:01.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.165+0000 7f63ba7fc640 1 -- 172.21.15.55:0/2526851781 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b40bd9a0 con 0x7f63c40ca350 2024-03-19T01:52:01.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.165+0000 7f63cbae3640 1 -- 172.21.15.55:0/2526851781 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63c40c6160 con 0x7f63c40ca350 2024-03-19T01:52:01.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.165+0000 7f63cbae3640 1 -- 172.21.15.55:0/2526851781 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63c4068a40 con 0x7f63c40ca350 2024-03-19T01:52:01.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.181+0000 7fe67bddc640 1 Processor -- start 2024-03-19T01:52:01.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.181+0000 7fe67bddc640 1 -- start start 2024-03-19T01:52:01.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.181+0000 7fe67bddc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe674154300 0x7fe67415ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.181+0000 7fe67bddc640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe67406b2a0 0x7fe67415e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.181+0000 7fe67bddc640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6740c8840 con 0x7fe67415e600 2024-03-19T01:52:01.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.181+0000 7fe67bddc640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fe67415b260 con 0x7fe67406b2a0 2024-03-19T01:52:01.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.181+0000 7fe67bddc640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fe6740c8ab0 con 0x7fe674154300 2024-03-19T01:52:01.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.225+0000 7fe67a379640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fe67415e600 0x7fe67406c110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:33234/0 (socket says 172.21.15.55:33234) 2024-03-19T01:52:01.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.225+0000 7fe67a379640 1 -- 172.21.15.55:0/3366220940 learned_addr learned my addr 172.21.15.55:0/3366220940 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:52:01.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.237+0000 7fe678b76640 1 -- 172.21.15.55:0/3366220940 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2070366085 0 0) 0x7fe6740c8840 con 0x7fe67415e600 2024-03-19T01:52:01.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.237+0000 7fe678b76640 1 -- 172.21.15.55:0/3366220940 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe660003530 con 0x7fe67415e600 2024-03-19T01:52:01.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.241+0000 7fe678b76640 1 -- 172.21.15.55:0/3366220940 <== mon.1 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3749691538 0 0) 0x7fe660003530 con 0x7fe67415e600 2024-03-19T01:52:01.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.241+0000 7fe678b76640 1 -- 172.21.15.55:0/3366220940 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe674154300 msgr2=0x7fe67415ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:01.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.241+0000 7fe678b76640 1 --2- 172.21.15.55:0/3366220940 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe674154300 0x7fe67415ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.241+0000 7fe678b76640 1 -- 172.21.15.55:0/3366220940 >> v2:172.21.15.55:3300/0 conn(0x7fe67406b2a0 msgr2=0x7fe67415e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:01.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.241+0000 7fe678b76640 1 --2- 172.21.15.55:0/3366220940 >> v2:172.21.15.55:3300/0 conn(0x7fe67406b2a0 0x7fe67415e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.241+0000 7fe678b76640 1 -- 172.21.15.55:0/3366220940 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6740c8d90 con 0x7fe67415e600 2024-03-19T01:52:01.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.241+0000 7fe678b76640 1 -- 172.21.15.55:0/3366220940 <== mon.1 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7fe66c001a00 con 0x7fe67415e600 2024-03-19T01:52:01.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.241+0000 7fe678b76640 1 -- 172.21.15.55:0/3366220940 >> v1:172.21.15.84:6789/0 conn(0x7fe67415e600 legacy=0x7fe67406c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:01.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.241+0000 7fe678b76640 1 --2- 172.21.15.55:0/3366220940 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe6600039c0 0x7fe660003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.241+0000 7fe678b76640 1 --2- 172.21.15.55:0/3366220940 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe674154300 0x7fe660004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.241+0000 7fe678b76640 1 --2- 172.21.15.55:0/3366220940 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe660004e70 0x7fe66000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.241+0000 7fe678b76640 1 -- 172.21.15.55:0/3366220940 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe67415b260 con 0x7fe660004e70 2024-03-19T01:52:01.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.241+0000 7fe678b76640 1 -- 172.21.15.55:0/3366220940 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fe6740c8ab0 con 0x7fe6600039c0 2024-03-19T01:52:01.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.241+0000 7fe678b76640 1 -- 172.21.15.55:0/3366220940 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fe660005630 con 0x7fe674154300 2024-03-19T01:52:01.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.245+0000 7fe679b78640 1 --2- 172.21.15.55:0/3366220940 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe6600039c0 0x7fe660003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:01.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.245+0000 7fe679377640 1 --2- 172.21.15.55:0/3366220940 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe674154300 0x7fe660004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:01.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.249+0000 7fe679377640 1 -- 172.21.15.55:0/3366220940 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe6600039c0 msgr2=0x7fe660003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:01.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.249+0000 7fe679377640 1 --2- 172.21.15.55:0/3366220940 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe6600039c0 0x7fe660003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.249+0000 7fe679377640 1 -- 172.21.15.55:0/3366220940 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe660004e70 msgr2=0x7fe66000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:01.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.249+0000 7fe679377640 1 --2- 172.21.15.55:0/3366220940 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe660004e70 0x7fe66000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.249+0000 7fe679377640 1 -- 172.21.15.55:0/3366220940 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe660003530 con 0x7fe674154300 2024-03-19T01:52:01.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.253+0000 7fe679377640 1 --2- 172.21.15.55:0/3366220940 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe674154300 0x7fe660004810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe664009890 tx=0x7fe66401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29fed6560b58a53c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:01.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.257+0000 7fe678b76640 1 -- 172.21.15.55:0/3366220940 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6640b0070 con 0x7fe674154300 2024-03-19T01:52:01.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.261+0000 7fe678b76640 1 -- 172.21.15.55:0/3366220940 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6640ab620 con 0x7fe674154300 2024-03-19T01:52:01.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.261+0000 7fe678b76640 1 -- 172.21.15.55:0/3366220940 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe664009460 con 0x7fe674154300 2024-03-19T01:52:01.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.261+0000 7fe67bddc640 1 -- 172.21.15.55:0/3366220940 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe674154300 msgr2=0x7fe660004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:01.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.261+0000 7fe67bddc640 1 --2- 172.21.15.55:0/3366220940 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe674154300 0x7fe660004810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe664009890 tx=0x7fe66401c3c0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.262 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.261+0000 7fe67a379640 1 -- 172.21.15.55:0/3366220940 reap_dead start 2024-03-19T01:52:01.262 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.261+0000 7fe67bddc640 1 -- 172.21.15.55:0/3366220940 shutdown_connections 2024-03-19T01:52:01.262 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.261+0000 7fe67bddc640 1 -- 172.21.15.55:0/3366220940 >> 172.21.15.55:0/3366220940 conn(0x7fe67406fb40 msgr2=0x7fe67406ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:01.262 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.261+0000 7fe67bddc640 1 -- 172.21.15.55:0/3366220940 shutdown_connections 2024-03-19T01:52:01.262 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.261+0000 7fe67bddc640 1 -- 172.21.15.55:0/3366220940 wait complete. 2024-03-19T01:52:01.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.261+0000 7fe67bddc640 1 Processor -- start 2024-03-19T01:52:01.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.261+0000 7fe67bddc640 1 -- start start 2024-03-19T01:52:01.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.265+0000 7fe67bddc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe674154300 0x7fe6741f0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.265+0000 7fe67bddc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe6741513f0 0x7fe674143cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.265+0000 7fe67bddc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe674146880 0x7fe674150300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.265+0000 7fe67bddc640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe67415d8c0 con 0x7fe674146880 2024-03-19T01:52:01.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.265+0000 7fe67bddc640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fe6740c9350 con 0x7fe674154300 2024-03-19T01:52:01.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.265+0000 7fe67bddc640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fe67415dba0 con 0x7fe6741513f0 2024-03-19T01:52:01.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.265+0000 7fe679b78640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe674154300 0x7fe6741f0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:01.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.265+0000 7fe679377640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe6741513f0 0x7fe674143cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:01.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.265+0000 7fe679b78640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe674154300 0x7fe6741f0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:55932/0 (socket says 172.21.15.55:55932) 2024-03-19T01:52:01.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.265+0000 7fe679b78640 1 -- 172.21.15.55:0/3136738196 learned_addr learned my addr 172.21.15.55:0/3136738196 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:01.269 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.269+0000 7fe679b78640 1 -- 172.21.15.55:0/3136738196 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe6741513f0 msgr2=0x7fe674143cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:01.270 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.269+0000 7fe679b78640 1 --2- 172.21.15.55:0/3136738196 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe6741513f0 0x7fe674143cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.270 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.269+0000 7fe679b78640 1 -- 172.21.15.55:0/3136738196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe674146880 msgr2=0x7fe674150300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:01.270 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.269+0000 7fe679b78640 1 --2- 172.21.15.55:0/3136738196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe674146880 0x7fe674150300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.270 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.269+0000 7fe679b78640 1 -- 172.21.15.55:0/3136738196 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6740cb5d0 con 0x7fe674154300 2024-03-19T01:52:01.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.273+0000 7fe679b78640 1 --2- 172.21.15.55:0/3136738196 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe674154300 0x7fe6741f0910 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe67000e870 tx=0x7fe67000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:01.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.277+0000 7fe66a7fc640 1 -- 172.21.15.55:0/3136738196 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe67000ee60 con 0x7fe674154300 2024-03-19T01:52:01.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.277+0000 7fe66a7fc640 1 -- 172.21.15.55:0/3136738196 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe670004030 con 0x7fe674154300 2024-03-19T01:52:01.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.277+0000 7fe66a7fc640 1 -- 172.21.15.55:0/3136738196 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe670004400 con 0x7fe674154300 2024-03-19T01:52:01.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.277+0000 7fe67bddc640 1 -- 172.21.15.55:0/3136738196 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe674147070 con 0x7fe674154300 2024-03-19T01:52:01.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.277+0000 7fe67bddc640 1 -- 172.21.15.55:0/3136738196 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe67414e750 con 0x7fe674154300 2024-03-19T01:52:01.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.277+0000 7fe67bddc640 1 -- 172.21.15.55:0/3136738196 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe674147590 con 0x7fe674154300 2024-03-19T01:52:01.386 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-19T01:52:01.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.445+0000 7f63ba7fc640 1 -- 172.21.15.55:0/2526851781 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f63b40bdb40 con 0x7f63c40ca350 2024-03-19T01:52:01.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.445+0000 7f63ba7fc640 1 --2- 172.21.15.55:0/2526851781 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f63a80676b0 0x7f63a8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.445+0000 7f63c987f640 1 --2- 172.21.15.55:0/2526851781 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f63a80676b0 0x7f63a8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:01.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.445+0000 7f63c987f640 1 --2- 172.21.15.55:0/2526851781 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f63a80676b0 0x7f63a8069b70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009650 tx=0x7f63c0006bb0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:01.447 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-19T01:52:01.461 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.461+0000 7f63ba7fc640 1 -- 172.21.15.55:0/2526851781 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f63b40b4de0 con 0x7f63c40ca350 2024-03-19T01:52:01.461 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.461+0000 7f63ba7fc640 1 -- 172.21.15.55:0/2526851781 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f63b40c3060 con 0x7f63c40ca350 2024-03-19T01:52:01.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.513+0000 7fa7dab1a640 1 Processor -- start 2024-03-19T01:52:01.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.513+0000 7fa7dab1a640 1 -- start start 2024-03-19T01:52:01.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.513+0000 7fa7dab1a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa7d415bc50 0x7fa7d40c2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.513+0000 7fa7dab1a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa7d41589d0 0x7fa7d40ca990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.513+0000 7fa7dab1a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7d40c8a50 con 0x7fa7d40ca5b0 2024-03-19T01:52:01.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.513+0000 7fa7dab1a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fa7d40c9b20 con 0x7fa7d41589d0 2024-03-19T01:52:01.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.513+0000 7fa7dab1a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa7d4158f10 con 0x7fa7d415bc50 2024-03-19T01:52:01.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.517+0000 7fa7d90b7640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa7d41589d0 0x7fa7d40ca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:01.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.517+0000 7fa7d3fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa7d415bc50 0x7fa7d40c2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:01.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.517+0000 7fa7d90b7640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa7d41589d0 0x7fa7d40ca990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53138/0 (socket says 172.21.15.55:53138) 2024-03-19T01:52:01.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.517+0000 7fa7d90b7640 1 -- 172.21.15.55:0/1671836147 learned_addr learned my addr 172.21.15.55:0/1671836147 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:01.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.521+0000 7fa7d90b7640 1 -- 172.21.15.55:0/1671836147 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa7d415bc50 msgr2=0x7fa7d40c2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:01.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.521+0000 7fa7d90b7640 1 --2- 172.21.15.55:0/1671836147 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa7d415bc50 0x7fa7d40c2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.521+0000 7fa7d90b7640 1 -- 172.21.15.55:0/1671836147 >> v1:172.21.15.84:6789/0 conn(0x7fa7d40ca5b0 legacy=0x7fa7d415b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:01.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.521+0000 7fa7d90b7640 1 -- 172.21.15.55:0/1671836147 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d40c9560 con 0x7fa7d41589d0 2024-03-19T01:52:01.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.525+0000 7fa7d90b7640 1 --2- 172.21.15.55:0/1671836147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7d41589d0 0x7fa7d40ca990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc009e50 tx=0x7fa7cc01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f2947adc1b9ab5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:01.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.529+0000 7fa7d37fe640 1 -- 172.21.15.55:0/1671836147 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc0a55e0 con 0x7fa7d41589d0 2024-03-19T01:52:01.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.529+0000 7fa7d37fe640 1 -- 172.21.15.55:0/1671836147 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7cc0a5780 con 0x7fa7d41589d0 2024-03-19T01:52:01.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.529+0000 7fa7d37fe640 1 -- 172.21.15.55:0/1671836147 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc0a5b50 con 0x7fa7d41589d0 2024-03-19T01:52:01.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.529+0000 7fa7dab1a640 1 -- 172.21.15.55:0/1671836147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7d41589d0 msgr2=0x7fa7d40ca990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:01.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.529+0000 7fa7dab1a640 1 --2- 172.21.15.55:0/1671836147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7d41589d0 0x7fa7d40ca990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc009e50 tx=0x7fa7cc01c960 comp rx=0 tx=0).stop 2024-03-19T01:52:01.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.529+0000 7fa7dab1a640 1 -- 172.21.15.55:0/1671836147 shutdown_connections 2024-03-19T01:52:01.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.529+0000 7fa7dab1a640 1 --2- 172.21.15.55:0/1671836147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7d41589d0 0x7fa7d40ca990 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.529+0000 7fa7dab1a640 1 --2- 172.21.15.55:0/1671836147 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa7d415bc50 0x7fa7d40c2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.529+0000 7fa7dab1a640 1 -- 172.21.15.55:0/1671836147 >> 172.21.15.55:0/1671836147 conn(0x7fa7d40715e0 msgr2=0x7fa7d40719f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:01.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.529+0000 7fa7dab1a640 1 -- 172.21.15.55:0/1671836147 shutdown_connections 2024-03-19T01:52:01.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.529+0000 7fa7dab1a640 1 -- 172.21.15.55:0/1671836147 wait complete. 2024-03-19T01:52:01.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.529+0000 7fa7dab1a640 1 Processor -- start 2024-03-19T01:52:01.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.529+0000 7fa7dab1a640 1 -- start start 2024-03-19T01:52:01.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.529+0000 7fa7dab1a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7d40ca5b0 0x7fa7d41f4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.529+0000 7fa7dab1a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7d41589d0 0x7fa7d4152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.529+0000 7fa7dab1a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa7d415bc50 0x7fa7d4152f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.529+0000 7fa7dab1a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d4165270 con 0x7fa7d41589d0 2024-03-19T01:52:01.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.533+0000 7fa7dab1a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d41650f0 con 0x7fa7d40ca5b0 2024-03-19T01:52:01.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.533+0000 7fa7dab1a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa7d41653f0 con 0x7fa7d415bc50 2024-03-19T01:52:01.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.533+0000 7fa7d3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7d41589d0 0x7fa7d4152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:01.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.533+0000 7fa7d3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7d41589d0 0x7fa7d4152a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53142/0 (socket says 172.21.15.55:53142) 2024-03-19T01:52:01.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.533+0000 7fa7d3fff640 1 -- 172.21.15.55:0/969560266 learned_addr learned my addr 172.21.15.55:0/969560266 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:01.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.533+0000 7fa7d90b7640 1 --2- 172.21.15.55:0/969560266 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa7d415bc50 0x7fa7d4152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:01.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.537+0000 7fa7d3fff640 1 -- 172.21.15.55:0/969560266 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa7d415bc50 msgr2=0x7fa7d4152f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:01.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.537+0000 7fa7d3fff640 1 --2- 172.21.15.55:0/969560266 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa7d415bc50 0x7fa7d4152f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.537+0000 7fa7d3fff640 1 -- 172.21.15.55:0/969560266 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7d40ca5b0 msgr2=0x7fa7d41f4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:01.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.537+0000 7fa7d3fff640 1 --2- 172.21.15.55:0/969560266 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7d40ca5b0 0x7fa7d41f4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.537+0000 7fa7d3fff640 1 -- 172.21.15.55:0/969560266 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d40c7ce0 con 0x7fa7d41589d0 2024-03-19T01:52:01.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.541+0000 7fa7d3fff640 1 --2- 172.21.15.55:0/969560266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7d41589d0 0x7fa7d4152a30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000e930 tx=0x7fa7c000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:01.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.545+0000 7fa7d17fa640 1 -- 172.21.15.55:0/969560266 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0017070 con 0x7fa7d41589d0 2024-03-19T01:52:01.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.545+0000 7fa7d17fa640 1 -- 172.21.15.55:0/969560266 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7c000ca00 con 0x7fa7d41589d0 2024-03-19T01:52:01.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.545+0000 7fa7dab1a640 1 -- 172.21.15.55:0/969560266 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7d40c7f80 con 0x7fa7d41589d0 2024-03-19T01:52:01.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.545+0000 7fa7dab1a640 1 -- 172.21.15.55:0/969560266 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7d414d4b0 con 0x7fa7d41589d0 2024-03-19T01:52:01.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.545+0000 7fa7d17fa640 1 -- 172.21.15.55:0/969560266 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c000cdb0 con 0x7fa7d41589d0 2024-03-19T01:52:01.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.545+0000 7fa7dab1a640 1 -- 172.21.15.55:0/969560266 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7d4068a40 con 0x7fa7d41589d0 2024-03-19T01:52:01.571 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-19T01:52:01.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.577+0000 7fa7d17fa640 1 -- 172.21.15.55:0/969560266 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa7c001a070 con 0x7fa7d41589d0 2024-03-19T01:52:01.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.577+0000 7fa7d17fa640 1 --2- 172.21.15.55:0/969560266 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa7b8067660 0x7fa7b8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.577+0000 7fa7d88b6640 1 --2- 172.21.15.55:0/969560266 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa7b8067660 0x7fa7b8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:01.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.577+0000 7fa7d88b6640 1 --2- 172.21.15.55:0/969560266 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa7b8067660 0x7fa7b8069b20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa7c8009650 tx=0x7fa7c8006bb0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:01.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.581+0000 7fa7d17fa640 1 -- 172.21.15.55:0/969560266 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fa7c0054770 con 0x7fa7d41589d0 2024-03-19T01:52:01.615 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.613+0000 7fc400fb8640 1 Processor -- start 2024-03-19T01:52:01.615 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.613+0000 7fc400fb8640 1 -- start start 2024-03-19T01:52:01.616 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.617+0000 7fc400fb8640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3fc06c610 0x7fc3fc15ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.617+0000 7fc400fb8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc3fc06c9f0 0x7fc3fc0c92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.617+0000 7fc400fb8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3fc070e80 con 0x7fc3fc0c9800 2024-03-19T01:52:01.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.617+0000 7fc400fb8640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fc3fc15b250 con 0x7fc3fc06c9f0 2024-03-19T01:52:01.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.617+0000 7fc400fb8640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc3fc06fb00 con 0x7fc3fc06c610 2024-03-19T01:52:01.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.617+0000 7fc3fa59c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3fc06c610 0x7fc3fc15ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:01.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.617+0000 7fc3fa59c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3fc06c610 0x7fc3fc15ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:42830/0 (socket says 172.21.15.55:42830) 2024-03-19T01:52:01.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.617+0000 7fc3fa59c640 1 -- 172.21.15.55:0/2391970748 learned_addr learned my addr 172.21.15.55:0/2391970748 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:01.622 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.621+0000 7fc3fa59c640 1 -- 172.21.15.55:0/2391970748 >> v2:172.21.15.55:3300/0 conn(0x7fc3fc06c9f0 msgr2=0x7fc3fc0c92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:01.622 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.621+0000 7fc3fa59c640 1 --2- 172.21.15.55:0/2391970748 >> v2:172.21.15.55:3300/0 conn(0x7fc3fc06c9f0 0x7fc3fc0c92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.622 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.621+0000 7fc3fa59c640 1 -- 172.21.15.55:0/2391970748 >> v1:172.21.15.84:6789/0 conn(0x7fc3fc0c9800 legacy=0x7fc3fc1427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:01.622 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.621+0000 7fc3fa59c640 1 -- 172.21.15.55:0/2391970748 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3fc06fde0 con 0x7fc3fc06c610 2024-03-19T01:52:01.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.625+0000 7fc3fa59c640 1 --2- 172.21.15.55:0/2391970748 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3fc06c610 0x7fc3fc15ad10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec009870 tx=0x7fc3ec01ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75be14b24d8c9e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:01.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.629+0000 7fc3f9d9b640 1 -- 172.21.15.55:0/2391970748 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec0b5070 con 0x7fc3fc06c610 2024-03-19T01:52:01.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.633+0000 7fc3f9d9b640 1 -- 172.21.15.55:0/2391970748 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3ec0a89d0 con 0x7fc3fc06c610 2024-03-19T01:52:01.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.633+0000 7fc3f9d9b640 1 -- 172.21.15.55:0/2391970748 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec0a8da0 con 0x7fc3fc06c610 2024-03-19T01:52:01.637 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.633+0000 7fc400fb8640 1 -- 172.21.15.55:0/2391970748 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3fc06c610 msgr2=0x7fc3fc15ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:01.637 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.633+0000 7fc400fb8640 1 --2- 172.21.15.55:0/2391970748 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3fc06c610 0x7fc3fc15ad10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec009870 tx=0x7fc3ec01ed10 comp rx=0 tx=0).stop 2024-03-19T01:52:01.637 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.633+0000 7fc400fb8640 1 -- 172.21.15.55:0/2391970748 shutdown_connections 2024-03-19T01:52:01.637 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.633+0000 7fc400fb8640 1 --2- 172.21.15.55:0/2391970748 >> v2:172.21.15.55:3300/0 conn(0x7fc3fc06c9f0 0x7fc3fc0c92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.633+0000 7fc400fb8640 1 --2- 172.21.15.55:0/2391970748 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3fc06c610 0x7fc3fc15ad10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.633+0000 7fc400fb8640 1 -- 172.21.15.55:0/2391970748 >> 172.21.15.55:0/2391970748 conn(0x7fc3fc067600 msgr2=0x7fc3fc071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:01.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.633+0000 7fc400fb8640 1 -- 172.21.15.55:0/2391970748 shutdown_connections 2024-03-19T01:52:01.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.633+0000 7fc400fb8640 1 -- 172.21.15.55:0/2391970748 wait complete. 2024-03-19T01:52:01.639 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.637+0000 7fc400fb8640 1 Processor -- start 2024-03-19T01:52:01.639 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.637+0000 7fc400fb8640 1 -- start start 2024-03-19T01:52:01.639 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.637+0000 7fc400fb8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3fc06c610 0x7fc3fc141d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.639 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.637+0000 7fc400fb8640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3fc06c9f0 0x7fc3fc1387b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.637+0000 7fc400fb8640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3fc0c9800 0x7fc3fc0c4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.637+0000 7fc400fb8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc3fc070680 con 0x7fc3fc06c610 2024-03-19T01:52:01.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.637+0000 7fc400fb8640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc3fc0703a0 con 0x7fc3fc0c9800 2024-03-19T01:52:01.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.637+0000 7fc400fb8640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc3fc070960 con 0x7fc3fc06c9f0 2024-03-19T01:52:01.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.637+0000 7fc3f3fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3fc06c9f0 0x7fc3fc1387b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:01.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.637+0000 7fc3f3fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3fc06c9f0 0x7fc3fc1387b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:42846/0 (socket says 172.21.15.55:42846) 2024-03-19T01:52:01.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.637+0000 7fc3f3fff640 1 -- 172.21.15.55:0/3080344912 learned_addr learned my addr 172.21.15.55:0/3080344912 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:01.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.645+0000 7fc3f3fff640 1 -- 172.21.15.55:0/3080344912 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3fc0c9800 msgr2=0x7fc3fc0c4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:01.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.645+0000 7fc3f3fff640 1 --2- 172.21.15.55:0/3080344912 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3fc0c9800 0x7fc3fc0c4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.645+0000 7fc3f3fff640 1 -- 172.21.15.55:0/3080344912 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3fc06c610 msgr2=0x7fc3fc141d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:01.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.645+0000 7fc3f3fff640 1 --2- 172.21.15.55:0/3080344912 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3fc06c610 0x7fc3fc141d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.645+0000 7fc3f3fff640 1 -- 172.21.15.55:0/3080344912 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3fc13a0f0 con 0x7fc3fc06c9f0 2024-03-19T01:52:01.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.649+0000 7fc3f3fff640 1 --2- 172.21.15.55:0/3080344912 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3fc06c9f0 0x7fc3fc1387b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400b4b0 tx=0x7fc3e400b970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:01.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.653+0000 7fc3f2ffd640 1 -- 172.21.15.55:0/3080344912 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4014070 con 0x7fc3fc06c9f0 2024-03-19T01:52:01.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.653+0000 7fc400fb8640 1 -- 172.21.15.55:0/3080344912 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3fc13a3d0 con 0x7fc3fc06c9f0 2024-03-19T01:52:01.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.653+0000 7fc400fb8640 1 -- 172.21.15.55:0/3080344912 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3fc13a990 con 0x7fc3fc06c9f0 2024-03-19T01:52:01.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.653+0000 7fc400fb8640 1 -- 172.21.15.55:0/3080344912 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3fc068a40 con 0x7fc3fc06c9f0 2024-03-19T01:52:01.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.657+0000 7fc3f2ffd640 1 -- 172.21.15.55:0/3080344912 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3e4010610 con 0x7fc3fc06c9f0 2024-03-19T01:52:01.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.657+0000 7fc3f2ffd640 1 -- 172.21.15.55:0/3080344912 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e401e060 con 0x7fc3fc06c9f0 2024-03-19T01:52:01.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.681+0000 7f5a5214f640 1 Processor -- start 2024-03-19T01:52:01.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.681+0000 7f5a5214f640 1 -- start start 2024-03-19T01:52:01.684 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.681+0000 7f5a5214f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f5a4c0ca210 0x7f5a4c0c7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.684 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.681+0000 7f5a5214f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a4c072c90 con 0x7f5a4c071050 2024-03-19T01:52:01.684 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.685+0000 7f5a5214f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a4c073210 con 0x7f5a4c1545f0 2024-03-19T01:52:01.684 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.685+0000 7f5a5214f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f5a4c06c0c0 con 0x7f5a4c0ca210 2024-03-19T01:52:01.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.689+0000 7f63cbae3640 1 -- 172.21.15.55:0/2526851781 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f63c40c70d0 con 0x7f63c40ca350 2024-03-19T01:52:01.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.721+0000 7fe66a7fc640 1 -- 172.21.15.55:0/3136738196 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fe67000c940 con 0x7fe674154300 2024-03-19T01:52:01.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.725+0000 7fe66a7fc640 1 --2- 172.21.15.55:0/3136738196 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe658067690 0x7fe658069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.725+0000 7fe679377640 1 --2- 172.21.15.55:0/3136738196 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe658067690 0x7fe658069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:01.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.725+0000 7fe679377640 1 --2- 172.21.15.55:0/3136738196 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe658067690 0x7fe658069b50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe6640ad000 tx=0x7fe664004850 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:01.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.729+0000 7fe66a7fc640 1 -- 172.21.15.55:0/3136738196 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fe6700545c0 con 0x7fe674154300 2024-03-19T01:52:01.753 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.753+0000 7fe66a7fc640 1 -- 172.21.15.55:0/3136738196 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fe670017020 con 0x7fe674154300 2024-03-19T01:52:01.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.757+0000 7f5a4b7fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f5a4c1545f0 0x7f5a4c06b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:33272/0 (socket says 172.21.15.55:33272) 2024-03-19T01:52:01.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.757+0000 7f5a4b7fe640 1 -- 172.21.15.55:0/2353757198 learned_addr learned my addr 172.21.15.55:0/2353757198 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:52:01.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.765+0000 7f5a4a7fc640 1 -- 172.21.15.55:0/2353757198 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3168764910 0 0) 0x7f5a4c073210 con 0x7f5a4c1545f0 2024-03-19T01:52:01.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.765+0000 7f5a4a7fc640 1 -- 172.21.15.55:0/2353757198 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a34003530 con 0x7f5a4c1545f0 2024-03-19T01:52:01.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.769+0000 7f5a4a7fc640 1 -- 172.21.15.55:0/2353757198 <== mon.1 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3421221855 0 0) 0x7f5a34003530 con 0x7f5a4c1545f0 2024-03-19T01:52:01.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.769+0000 7f5a4a7fc640 1 -- 172.21.15.55:0/2353757198 >> v2:172.21.15.55:3300/0 conn(0x7f5a4c0ca210 msgr2=0x7f5a4c0c7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:01.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.769+0000 7f5a4a7fc640 1 --2- 172.21.15.55:0/2353757198 >> v2:172.21.15.55:3300/0 conn(0x7f5a4c0ca210 0x7f5a4c0c7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.769+0000 7f5a4a7fc640 1 -- 172.21.15.55:0/2353757198 >> v1:172.21.15.55:6789/0 conn(0x7f5a4c071050 legacy=0x7f5a4c06b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:01.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.769+0000 7f5a4a7fc640 1 -- 172.21.15.55:0/2353757198 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a4c072f30 con 0x7f5a4c1545f0 2024-03-19T01:52:01.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.769+0000 7f5a4a7fc640 1 -- 172.21.15.55:0/2353757198 <== mon.1 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f5a3c001680 con 0x7f5a4c1545f0 2024-03-19T01:52:01.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.769+0000 7f5a4a7fc640 1 -- 172.21.15.55:0/2353757198 >> v1:172.21.15.84:6789/0 conn(0x7f5a4c1545f0 legacy=0x7f5a4c06b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:01.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.769+0000 7f5a4a7fc640 1 --2- 172.21.15.55:0/2353757198 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a34003a10 0x7f5a34003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.769+0000 7f5a4a7fc640 1 --2- 172.21.15.55:0/2353757198 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a34004810 0x7f5a34004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.769+0000 7f5a4a7fc640 1 --2- 172.21.15.55:0/2353757198 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a340053d0 0x7f5a3400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.769+0000 7f5a4a7fc640 1 -- 172.21.15.55:0/2353757198 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4c06c0c0 con 0x7f5a340053d0 2024-03-19T01:52:01.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.769+0000 7f5a4a7fc640 1 -- 172.21.15.55:0/2353757198 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3400c240 con 0x7f5a34003a10 2024-03-19T01:52:01.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.769+0000 7f5a4a7fc640 1 -- 172.21.15.55:0/2353757198 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f5a3400c440 con 0x7f5a34004810 2024-03-19T01:52:01.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.773+0000 7f5a4b7fe640 1 --2- 172.21.15.55:0/2353757198 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a34003a10 0x7f5a34003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:01.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.777+0000 7f5a4b7fe640 1 -- 172.21.15.55:0/2353757198 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a34004810 msgr2=0x7f5a34004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:01.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.777+0000 7f5a4b7fe640 1 --2- 172.21.15.55:0/2353757198 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a34004810 0x7f5a34004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.777+0000 7f5a4b7fe640 1 -- 172.21.15.55:0/2353757198 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a340053d0 msgr2=0x7f5a3400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:01.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.777+0000 7f5a4b7fe640 1 --2- 172.21.15.55:0/2353757198 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a340053d0 0x7f5a3400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.777+0000 7f5a4b7fe640 1 -- 172.21.15.55:0/2353757198 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a34003530 con 0x7f5a34003a10 2024-03-19T01:52:01.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.777+0000 7f5a4bfff640 1 -- 172.21.15.55:0/2353757198 reap_dead start 2024-03-19T01:52:01.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.785+0000 7f5a4b7fe640 1 --2- 172.21.15.55:0/2353757198 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a34003a10 0x7f5a34003e30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c00c190 tx=0x7f5a3c01cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d734b3342cf317ea server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:01.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.789+0000 7f5a4a7fc640 1 -- 172.21.15.55:0/2353757198 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c007c70 con 0x7f5a34003a10 2024-03-19T01:52:01.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.789+0000 7f5a4a7fc640 1 -- 172.21.15.55:0/2353757198 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a3c0a9070 con 0x7f5a34003a10 2024-03-19T01:52:01.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.789+0000 7f5a4a7fc640 1 -- 172.21.15.55:0/2353757198 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c005da0 con 0x7f5a34003a10 2024-03-19T01:52:01.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.789+0000 7f5a5214f640 1 -- 172.21.15.55:0/2353757198 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a34003a10 msgr2=0x7f5a34003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:01.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.789+0000 7f5a5214f640 1 --2- 172.21.15.55:0/2353757198 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a34003a10 0x7f5a34003e30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c00c190 tx=0x7f5a3c01cbc0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.789+0000 7f5a5214f640 1 -- 172.21.15.55:0/2353757198 shutdown_connections 2024-03-19T01:52:01.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.789+0000 7f5a5214f640 1 --2- 172.21.15.55:0/2353757198 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a34003a10 0x7f5a34003e30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.789+0000 7f5a5214f640 1 -- 172.21.15.55:0/2353757198 >> 172.21.15.55:0/2353757198 conn(0x7f5a4c070410 msgr2=0x7f5a4c070820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:01.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.789+0000 7f5a5214f640 1 -- 172.21.15.55:0/2353757198 shutdown_connections 2024-03-19T01:52:01.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.789+0000 7f5a5214f640 1 -- 172.21.15.55:0/2353757198 wait complete. 2024-03-19T01:52:01.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.793+0000 7f5a5214f640 1 Processor -- start 2024-03-19T01:52:01.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.793+0000 7f5a5214f640 1 -- start start 2024-03-19T01:52:01.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.793+0000 7f5a5214f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a34003a10 0x7f5a4c14f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.793+0000 7f5a5214f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a4c0ccb10 0x7f5a4c1f3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.793+0000 7f5a5214f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a4c152790 0x7f5a4c152c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.793+0000 7f5a5214f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4c0c9a10 con 0x7f5a4c152790 2024-03-19T01:52:01.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.793+0000 7f5a5214f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4c0ca620 con 0x7f5a4c0ccb10 2024-03-19T01:52:01.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.793+0000 7f5a5214f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f5a4c1549d0 con 0x7f5a34003a10 2024-03-19T01:52:01.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.797+0000 7f5a4affd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a4c0ccb10 0x7f5a4c1f3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:01.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.797+0000 7f5a4affd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a4c0ccb10 0x7f5a4c1f3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:55974/0 (socket says 172.21.15.55:55974) 2024-03-19T01:52:01.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.797+0000 7f5a4affd640 1 -- 172.21.15.55:0/2974681700 learned_addr learned my addr 172.21.15.55:0/2974681700 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:01.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.801+0000 7f5a4affd640 1 -- 172.21.15.55:0/2974681700 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a34003a10 msgr2=0x7f5a4c14f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:01.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.801+0000 7f5a4affd640 1 --2- 172.21.15.55:0/2974681700 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a34003a10 0x7f5a4c14f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.801+0000 7f5a4affd640 1 -- 172.21.15.55:0/2974681700 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a4c152790 msgr2=0x7f5a4c152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:01.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.801+0000 7f5a4affd640 1 --2- 172.21.15.55:0/2974681700 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a4c152790 0x7f5a4c152c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.801+0000 7f5a4affd640 1 -- 172.21.15.55:0/2974681700 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a4c15dd70 con 0x7f5a4c0ccb10 2024-03-19T01:52:01.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.805+0000 7f5a4affd640 1 --2- 172.21.15.55:0/2974681700 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a4c0ccb10 0x7f5a4c1f3f90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5a4000c3c0 tx=0x7f5a4000c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:01.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.809+0000 7f5a33fff640 1 -- 172.21.15.55:0/2974681700 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a40017070 con 0x7f5a4c0ccb10 2024-03-19T01:52:01.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.809+0000 7f5a33fff640 1 -- 172.21.15.55:0/2974681700 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a40003600 con 0x7f5a4c0ccb10 2024-03-19T01:52:01.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.809+0000 7f5a5214f640 1 -- 172.21.15.55:0/2974681700 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a4c14fae0 con 0x7f5a4c0ccb10 2024-03-19T01:52:01.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.809+0000 7f5a33fff640 1 -- 172.21.15.55:0/2974681700 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a400039f0 con 0x7f5a4c0ccb10 2024-03-19T01:52:01.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.809+0000 7f5a5214f640 1 -- 172.21.15.55:0/2974681700 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a4c1f44d0 con 0x7f5a4c0ccb10 2024-03-19T01:52:01.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.813+0000 7f5a5214f640 1 -- 172.21.15.55:0/2974681700 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a4c068a40 con 0x7f5a4c0ccb10 2024-03-19T01:52:01.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.845+0000 7f5a33fff640 1 -- 172.21.15.55:0/2974681700 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f5a4001a070 con 0x7f5a4c0ccb10 2024-03-19T01:52:01.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.845+0000 7f5a33fff640 1 --2- 172.21.15.55:0/2974681700 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5a380676b0 0x7f5a38069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:01.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.845+0000 7f5a4b7fe640 1 --2- 172.21.15.55:0/2974681700 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5a380676b0 0x7f5a38069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:01.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.845+0000 7f5a4b7fe640 1 --2- 172.21.15.55:0/2974681700 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5a380676b0 0x7f5a38069b70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c0ab000 tx=0x7f5a4c15a380 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:01.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.849+0000 7fa7d17fa640 1 -- 172.21.15.55:0/969560266 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa7c00566d0 con 0x7fa7d41589d0 2024-03-19T01:52:01.867 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-19T01:52:01.876 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.877+0000 7f63ba7fc640 1 -- 172.21.15.55:0/2526851781 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f63b40160d0 con 0x7f63c40ca350 2024-03-19T01:52:01.877 INFO:teuthology.orchestra.run.smithi055.stdout:64424509463 2024-03-19T01:52:01.880 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.877+0000 7f63cbae3640 1 -- 172.21.15.55:0/2526851781 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f63a80676b0 msgr2=0x7f63a8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:01.880 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.877+0000 7f63cbae3640 1 --2- 172.21.15.55:0/2526851781 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f63a80676b0 0x7f63a8069b70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009650 tx=0x7f63c0006bb0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.880 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.877+0000 7f63cbae3640 1 -- 172.21.15.55:0/2526851781 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63c40ca350 msgr2=0x7f63c4146880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:01.880 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.877+0000 7f63cbae3640 1 --2- 172.21.15.55:0/2526851781 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63c40ca350 0x7f63c4146880 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f63b401ab90 tx=0x7f63b401ad20 comp rx=0 tx=0).stop 2024-03-19T01:52:01.880 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.877+0000 7f63cbae3640 1 -- 172.21.15.55:0/2526851781 shutdown_connections 2024-03-19T01:52:01.881 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.877+0000 7f63cbae3640 1 --2- 172.21.15.55:0/2526851781 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f63a80676b0 0x7f63a8069b70 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.881 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.877+0000 7f63cbae3640 1 --2- 172.21.15.55:0/2526851781 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f63c4150f50 0x7f63c40cba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.881 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.877+0000 7f63cbae3640 1 --2- 172.21.15.55:0/2526851781 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63c40ca350 0x7f63c4146880 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.881 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.877+0000 7f63cbae3640 1 --2- 172.21.15.55:0/2526851781 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f63c40c3320 0x7f63c41438a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:01.881 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.877+0000 7f63cbae3640 1 -- 172.21.15.55:0/2526851781 >> 172.21.15.55:0/2526851781 conn(0x7f63c406e670 msgr2=0x7f63c414f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:01.881 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.881+0000 7f63cbae3640 1 -- 172.21.15.55:0/2526851781 shutdown_connections 2024-03-19T01:52:01.881 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.881+0000 7f63cbae3640 1 -- 172.21.15.55:0/2526851781 wait complete. 2024-03-19T01:52:01.885 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-19T01:52:01.907 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509463 got 64424509463 for osd.7 2024-03-19T01:52:01.908 DEBUG:teuthology.parallel:result is None 2024-03-19T01:52:01.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:01.945+0000 7fe67bddc640 1 -- 172.21.15.55:0/3136738196 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fe674068a40 con 0x7fe674154300 2024-03-19T01:52:01.981 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-19T01:52:02.003 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.001+0000 7fc3f2ffd640 1 -- 172.21.15.55:0/3080344912 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc3e4017070 con 0x7fc3fc06c9f0 2024-03-19T01:52:02.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.001+0000 7fc3f2ffd640 1 --2- 172.21.15.55:0/3080344912 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3dc067690 0x7fc3dc069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:02.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.005+0000 7fc3fa59c640 1 --2- 172.21.15.55:0/3080344912 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3dc067690 0x7fc3dc069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:02.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.005+0000 7fc3fa59c640 1 --2- 172.21.15.55:0/3080344912 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3dc067690 0x7fc3dc069b50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec0b2000 tx=0x7fc3ec004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:02.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.005+0000 7fc3f2ffd640 1 -- 172.21.15.55:0/3080344912 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fc3e4053da0 con 0x7fc3fc06c9f0 2024-03-19T01:52:02.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.021+0000 7fc3f2ffd640 1 -- 172.21.15.55:0/3080344912 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc3e4056400 con 0x7fc3fc06c9f0 2024-03-19T01:52:02.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.025+0000 7fa7dab1a640 1 -- 172.21.15.55:0/969560266 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa7d40b0510 con 0x7fa7d41589d0 2024-03-19T01:52:02.031 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.029+0000 7fe7b0d4f640 1 Processor -- start 2024-03-19T01:52:02.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.029+0000 7fe7b0d4f640 1 -- start start 2024-03-19T01:52:02.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.029+0000 7fa7d17fa640 1 -- 172.21.15.55:0/969560266 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa7c005a770 con 0x7fa7d41589d0 2024-03-19T01:52:02.032 INFO:teuthology.orchestra.run.smithi055.stdout:64424509464 2024-03-19T01:52:02.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.033+0000 7fe7b0d4f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fe7ac1510f0 0x7fe7ac0c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:02.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.033+0000 7fe7b0d4f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe7ac15a410 0x7fe7ac0708a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:02.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.033+0000 7fe7b0d4f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7ac0c3f60 con 0x7fe7ac0c73c0 2024-03-19T01:52:02.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.033+0000 7fe7b0d4f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fe7ac1514d0 con 0x7fe7ac15a410 2024-03-19T01:52:02.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.033+0000 7fe7b0d4f640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fe7ac0c6870 con 0x7fe7ac1510f0 2024-03-19T01:52:02.035 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.033+0000 7fa7dab1a640 1 -- 172.21.15.55:0/969560266 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa7b8067660 msgr2=0x7fa7b8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.033+0000 7fa7dab1a640 1 --2- 172.21.15.55:0/969560266 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa7b8067660 0x7fa7b8069b20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa7c8009650 tx=0x7fa7c8006bb0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.033+0000 7fa7dab1a640 1 -- 172.21.15.55:0/969560266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7d41589d0 msgr2=0x7fa7d4152a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.033+0000 7fa7dab1a640 1 --2- 172.21.15.55:0/969560266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7d41589d0 0x7fa7d4152a30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000e930 tx=0x7fa7c000edf0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.033+0000 7fa7dab1a640 1 -- 172.21.15.55:0/969560266 shutdown_connections 2024-03-19T01:52:02.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.033+0000 7fa7dab1a640 1 --2- 172.21.15.55:0/969560266 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa7b8067660 0x7fa7b8069b20 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.033+0000 7fa7dab1a640 1 --2- 172.21.15.55:0/969560266 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa7d415bc50 0x7fa7d4152f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.033+0000 7fa7dab1a640 1 --2- 172.21.15.55:0/969560266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa7d41589d0 0x7fa7d4152a30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.033+0000 7fa7dab1a640 1 --2- 172.21.15.55:0/969560266 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7d40ca5b0 0x7fa7d41f4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.033+0000 7fa7dab1a640 1 -- 172.21.15.55:0/969560266 >> 172.21.15.55:0/969560266 conn(0x7fa7d40715e0 msgr2=0x7fa7d40719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:02.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.033+0000 7fa7dab1a640 1 -- 172.21.15.55:0/969560266 shutdown_connections 2024-03-19T01:52:02.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.033+0000 7fa7dab1a640 1 -- 172.21.15.55:0/969560266 wait complete. 2024-03-19T01:52:02.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.041+0000 7fe7a9d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe7ac15a410 0x7fe7ac0708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:02.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.041+0000 7fe7a9d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe7ac15a410 0x7fe7ac0708a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53180/0 (socket says 172.21.15.55:53180) 2024-03-19T01:52:02.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.041+0000 7fe7a9d9b640 1 -- 172.21.15.55:0/3499546345 learned_addr learned my addr 172.21.15.55:0/3499546345 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:02.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.045+0000 7fe7a9d9b640 1 -- 172.21.15.55:0/3499546345 >> v2:172.21.15.84:3300/0 conn(0x7fe7ac1510f0 msgr2=0x7fe7ac0c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:02.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.045+0000 7fe7a9d9b640 1 --2- 172.21.15.55:0/3499546345 >> v2:172.21.15.84:3300/0 conn(0x7fe7ac1510f0 0x7fe7ac0c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.045+0000 7fe7a9d9b640 1 -- 172.21.15.55:0/3499546345 >> v1:172.21.15.84:6789/0 conn(0x7fe7ac0c73c0 legacy=0x7fe7ac072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:02.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.045+0000 7fe7a9d9b640 1 -- 172.21.15.55:0/3499546345 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7ac0c6b50 con 0x7fe7ac15a410 2024-03-19T01:52:02.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.049+0000 7fe7a9d9b640 1 --2- 172.21.15.55:0/3499546345 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7ac15a410 0x7fe7ac0708a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac0cc900 tx=0x7fe7a001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bb823865d69cf1e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:02.057 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509464 got 64424509464 for osd.0 2024-03-19T01:52:02.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.057+0000 7fe7a959a640 1 -- 172.21.15.55:0/3499546345 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a00b5070 con 0x7fe7ac15a410 2024-03-19T01:52:02.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.057+0000 7fe7a959a640 1 -- 172.21.15.55:0/3499546345 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7a00a89d0 con 0x7fe7ac15a410 2024-03-19T01:52:02.058 DEBUG:teuthology.parallel:result is None 2024-03-19T01:52:02.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.057+0000 7f4f65dc1640 1 Processor -- start 2024-03-19T01:52:02.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.057+0000 7f4f65dc1640 1 -- start start 2024-03-19T01:52:02.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.057+0000 7f4f65dc1640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4f60158c00 0x7f4f60072e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:02.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.057+0000 7f4f65dc1640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f6015e6c0 con 0x7f4f60158820 2024-03-19T01:52:02.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.057+0000 7f4f65dc1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f60161dd0 con 0x7f4f600c79b0 2024-03-19T01:52:02.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.057+0000 7f4f65dc1640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f4f601620b0 con 0x7f4f60158c00 2024-03-19T01:52:02.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.057+0000 7fe7a959a640 1 -- 172.21.15.55:0/3499546345 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a00a8da0 con 0x7fe7ac15a410 2024-03-19T01:52:02.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.057+0000 7fe7b0d4f640 1 -- 172.21.15.55:0/3499546345 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7ac15a410 msgr2=0x7fe7ac0708a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.057+0000 7fe7b0d4f640 1 --2- 172.21.15.55:0/3499546345 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7ac15a410 0x7fe7ac0708a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac0cc900 tx=0x7fe7a001ecf0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.057+0000 7fe7b0d4f640 1 -- 172.21.15.55:0/3499546345 shutdown_connections 2024-03-19T01:52:02.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.057+0000 7fe7b0d4f640 1 --2- 172.21.15.55:0/3499546345 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7ac15a410 0x7fe7ac0708a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.057+0000 7fe7b0d4f640 1 --2- 172.21.15.55:0/3499546345 >> v2:172.21.15.84:3300/0 conn(0x7fe7ac1510f0 0x7fe7ac0c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.057+0000 7fe7b0d4f640 1 -- 172.21.15.55:0/3499546345 >> 172.21.15.55:0/3499546345 conn(0x7fe7ac0cc9c0 msgr2=0x7fe7ac0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:02.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.057+0000 7fe7b0d4f640 1 -- 172.21.15.55:0/3499546345 shutdown_connections 2024-03-19T01:52:02.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.057+0000 7fe7b0d4f640 1 -- 172.21.15.55:0/3499546345 wait complete. 2024-03-19T01:52:02.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.057+0000 7f4f57fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4f60158c00 0x7f4f60072e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:02.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.061+0000 7fe7b0d4f640 1 Processor -- start 2024-03-19T01:52:02.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.061+0000 7fe7b0d4f640 1 -- start start 2024-03-19T01:52:02.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.061+0000 7f4f57fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4f60158c00 0x7f4f60072e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53186/0 (socket says 172.21.15.55:53186) 2024-03-19T01:52:02.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.061+0000 7f4f57fff640 1 -- 172.21.15.55:0/1812662092 learned_addr learned my addr 172.21.15.55:0/1812662092 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:02.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.061+0000 7fe7b0d4f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7ac0c73c0 0x7fe7ac1306f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:02.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.061+0000 7fe7b0d4f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7ac1510f0 0x7fe7ac131e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:02.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.061+0000 7fe7b0d4f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7ac15a410 0x7fe7ac1f4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:02.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.061+0000 7fe7b0d4f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ac0c6e30 con 0x7fe7ac1510f0 2024-03-19T01:52:02.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.061+0000 7fe7b0d4f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ac0c44e0 con 0x7fe7ac15a410 2024-03-19T01:52:02.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.061+0000 7fe7b0d4f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fe7ac0c4a00 con 0x7fe7ac0c73c0 2024-03-19T01:52:02.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.061+0000 7fe7aa59c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7ac0c73c0 0x7fe7ac1306f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:02.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.061+0000 7fe7aa59c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7ac0c73c0 0x7fe7ac1306f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:42876/0 (socket says 172.21.15.55:42876) 2024-03-19T01:52:02.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.061+0000 7fe7aa59c640 1 -- 172.21.15.55:0/2611546435 learned_addr learned my addr 172.21.15.55:0/2611546435 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:02.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.065+0000 7fe7a9d9b640 1 --2- 172.21.15.55:0/2611546435 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7ac1510f0 0x7fe7ac131e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:02.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.065+0000 7f4f57fff640 1 -- 172.21.15.55:0/1812662092 >> v1:172.21.15.84:6789/0 conn(0x7f4f600c79b0 legacy=0x7f4f6015ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:02.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.065+0000 7f4f57fff640 1 -- 172.21.15.55:0/1812662092 >> v1:172.21.15.55:6789/0 conn(0x7f4f60158820 legacy=0x7f4f60070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.065+0000 7f4f57fff640 1 -- 172.21.15.55:0/1812662092 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f60162390 con 0x7f4f60158c00 2024-03-19T01:52:02.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.065+0000 7fe7aa59c640 1 -- 172.21.15.55:0/2611546435 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7ac15a410 msgr2=0x7fe7ac1f4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.065+0000 7fe7aa59c640 1 --2- 172.21.15.55:0/2611546435 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7ac15a410 0x7fe7ac1f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.069+0000 7fe7aa59c640 1 -- 172.21.15.55:0/2611546435 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7ac1510f0 msgr2=0x7fe7ac131e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.069+0000 7fe7aa59c640 1 --2- 172.21.15.55:0/2611546435 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7ac1510f0 0x7fe7ac131e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.069+0000 7fe7aa59c640 1 -- 172.21.15.55:0/2611546435 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7ac130c30 con 0x7fe7ac0c73c0 2024-03-19T01:52:02.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.069+0000 7fe7aa59c640 1 --2- 172.21.15.55:0/2611546435 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7ac0c73c0 0x7fe7ac1306f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe79c00b550 tx=0x7fe79c00ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:02.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.073+0000 7f4f57fff640 1 --2- 172.21.15.55:0/1812662092 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f60158c00 0x7f4f60072e20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4f48009870 tx=0x7f4f4801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b12fd93ab4007d59 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:02.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.077+0000 7fe79affd640 1 -- 172.21.15.55:0/2611546435 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe79c014070 con 0x7fe7ac0c73c0 2024-03-19T01:52:02.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.077+0000 7fe7b0d4f640 1 -- 172.21.15.55:0/2611546435 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7ac13ec30 con 0x7fe7ac0c73c0 2024-03-19T01:52:02.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.077+0000 7fe7b0d4f640 1 -- 172.21.15.55:0/2611546435 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7ac06b2a0 con 0x7fe7ac0c73c0 2024-03-19T01:52:02.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.077+0000 7fe79affd640 1 -- 172.21.15.55:0/2611546435 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe79c010890 con 0x7fe7ac0c73c0 2024-03-19T01:52:02.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.077+0000 7fe79affd640 1 -- 172.21.15.55:0/2611546435 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe79c010c80 con 0x7fe7ac0c73c0 2024-03-19T01:52:02.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.077+0000 7fe7b0d4f640 1 -- 172.21.15.55:0/2611546435 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7900052a0 con 0x7fe7ac0c73c0 2024-03-19T01:52:02.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.077+0000 7f4f5effd640 1 -- 172.21.15.55:0/1812662092 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f480aa650 con 0x7f4f60158c00 2024-03-19T01:52:02.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.077+0000 7f4f5effd640 1 -- 172.21.15.55:0/1812662092 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f480af070 con 0x7f4f60158c00 2024-03-19T01:52:02.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.077+0000 7f4f5effd640 1 -- 172.21.15.55:0/1812662092 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f480aaa10 con 0x7f4f60158c00 2024-03-19T01:52:02.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.081+0000 7f4f65dc1640 1 -- 172.21.15.55:0/1812662092 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f60158c00 msgr2=0x7f4f60072e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.081+0000 7f4f65dc1640 1 --2- 172.21.15.55:0/1812662092 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f60158c00 0x7f4f60072e20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4f48009870 tx=0x7f4f4801c4f0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.081+0000 7f4f65dc1640 1 -- 172.21.15.55:0/1812662092 shutdown_connections 2024-03-19T01:52:02.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.081+0000 7f4f65dc1640 1 --2- 172.21.15.55:0/1812662092 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f60158c00 0x7f4f60072e20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.081+0000 7f4f65dc1640 1 -- 172.21.15.55:0/1812662092 >> 172.21.15.55:0/1812662092 conn(0x7f4f600cc9c0 msgr2=0x7f4f600ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:02.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.081+0000 7f4f65dc1640 1 -- 172.21.15.55:0/1812662092 shutdown_connections 2024-03-19T01:52:02.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.081+0000 7f4f65dc1640 1 -- 172.21.15.55:0/1812662092 wait complete. 2024-03-19T01:52:02.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.081+0000 7f4f65dc1640 1 Processor -- start 2024-03-19T01:52:02.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.081+0000 7f4f65dc1640 1 -- start start 2024-03-19T01:52:02.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.081+0000 7f4f65dc1640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4f600c79b0 0x7f4f60157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:02.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.081+0000 7f4f65dc1640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f60158820 0x7f4f602035d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:02.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.081+0000 7f4f65dc1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f60158c00 0x7f4f60154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:02.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.081+0000 7f4f65dc1640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6016cb20 con 0x7f4f60158c00 2024-03-19T01:52:02.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.081+0000 7f4f65dc1640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6016c9a0 con 0x7f4f60158820 2024-03-19T01:52:02.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.081+0000 7f4f65dc1640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4f6016cca0 con 0x7f4f600c79b0 2024-03-19T01:52:02.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.081+0000 7f4f5ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f60158c00 0x7f4f60154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:02.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.085+0000 7f4f5ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f60158c00 0x7f4f60154b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53216/0 (socket says 172.21.15.55:53216) 2024-03-19T01:52:02.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.085+0000 7f4f5ffff640 1 -- 172.21.15.55:0/1170748642 learned_addr learned my addr 172.21.15.55:0/1170748642 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:02.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.085+0000 7f4f5f7fe640 1 --2- 172.21.15.55:0/1170748642 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4f600c79b0 0x7f4f60157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:02.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.089+0000 7f4f5ffff640 1 -- 172.21.15.55:0/1170748642 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4f600c79b0 msgr2=0x7f4f60157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.089+0000 7f4f5ffff640 1 --2- 172.21.15.55:0/1170748642 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4f600c79b0 0x7f4f60157720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.089+0000 7f4f5ffff640 1 -- 172.21.15.55:0/1170748642 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f60158820 msgr2=0x7f4f602035d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.089+0000 7f4f5ffff640 1 --2- 172.21.15.55:0/1170748642 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f60158820 0x7f4f602035d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.089+0000 7f4f5ffff640 1 -- 172.21.15.55:0/1170748642 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f60156540 con 0x7f4f60158c00 2024-03-19T01:52:02.092 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.089+0000 7f4f5ffff640 1 --2- 172.21.15.55:0/1170748642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f60158c00 0x7f4f60154b90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4f5800b410 tx=0x7f4f5800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:02.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.097+0000 7f4f5cff9640 1 -- 172.21.15.55:0/1170748642 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f58015020 con 0x7f4f60158c00 2024-03-19T01:52:02.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.097+0000 7f4f5cff9640 1 -- 172.21.15.55:0/1170748642 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f58011830 con 0x7f4f60158c00 2024-03-19T01:52:02.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.097+0000 7f4f5cff9640 1 -- 172.21.15.55:0/1170748642 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f58011c20 con 0x7f4f60158c00 2024-03-19T01:52:02.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.097+0000 7f4f65dc1640 1 -- 172.21.15.55:0/1170748642 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f6015d5a0 con 0x7f4f60158c00 2024-03-19T01:52:02.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.097+0000 7f4f65dc1640 1 -- 172.21.15.55:0/1170748642 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f60204010 con 0x7f4f60158c00 2024-03-19T01:52:02.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.097+0000 7f4f65dc1640 1 -- 172.21.15.55:0/1170748642 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f60204250 con 0x7f4f60158c00 2024-03-19T01:52:02.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.105+0000 7fc3fba14640 1 Processor -- start 2024-03-19T01:52:02.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.105+0000 7fc3fba14640 1 -- start start 2024-03-19T01:52:02.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.105+0000 7fc3fba14640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc3f40c61c0 0x7fc3f40708a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:02.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.105+0000 7fc3fba14640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3f40c3f60 con 0x7fc3f40c73c0 2024-03-19T01:52:02.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.105+0000 7fc3fba14640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3f41514d0 con 0x7fc3f41510f0 2024-03-19T01:52:02.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.105+0000 7fc3fba14640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fc3f40c6630 con 0x7fc3f40c61c0 2024-03-19T01:52:02.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.161+0000 7f5a33fff640 1 -- 172.21.15.55:0/2974681700 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f5a40053650 con 0x7f5a4c0ccb10 2024-03-19T01:52:02.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.181+0000 7f5a33fff640 1 -- 172.21.15.55:0/2974681700 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f5a40021020 con 0x7f5a4c0ccb10 2024-03-19T01:52:02.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.201+0000 7fe66a7fc640 1 -- 172.21.15.55:0/3136738196 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe670056520 con 0x7fe674154300 2024-03-19T01:52:02.204 INFO:teuthology.orchestra.run.smithi055.stdout:64424509465 2024-03-19T01:52:02.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.201+0000 7fc400fb8640 1 -- 172.21.15.55:0/3080344912 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fc3fc13e190 con 0x7fc3fc06c9f0 2024-03-19T01:52:02.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.205+0000 7fe67bddc640 1 -- 172.21.15.55:0/3136738196 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe658067690 msgr2=0x7fe658069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.205+0000 7fe67bddc640 1 --2- 172.21.15.55:0/3136738196 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe658067690 0x7fe658069b50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe6640ad000 tx=0x7fe664004850 comp rx=0 tx=0).stop 2024-03-19T01:52:02.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.205+0000 7fe67bddc640 1 -- 172.21.15.55:0/3136738196 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe674154300 msgr2=0x7fe6741f0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.205+0000 7fe67bddc640 1 --2- 172.21.15.55:0/3136738196 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe674154300 0x7fe6741f0910 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe67000e870 tx=0x7fe67000ed30 comp rx=0 tx=0).stop 2024-03-19T01:52:02.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.205+0000 7fe67bddc640 1 -- 172.21.15.55:0/3136738196 shutdown_connections 2024-03-19T01:52:02.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.205+0000 7fe67bddc640 1 --2- 172.21.15.55:0/3136738196 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe658067690 0x7fe658069b50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.205+0000 7fe67bddc640 1 --2- 172.21.15.55:0/3136738196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe674146880 0x7fe674150300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.205+0000 7fe67bddc640 1 --2- 172.21.15.55:0/3136738196 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe6741513f0 0x7fe674143cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.205+0000 7fe67bddc640 1 --2- 172.21.15.55:0/3136738196 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe674154300 0x7fe6741f0910 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.205+0000 7fe67bddc640 1 -- 172.21.15.55:0/3136738196 >> 172.21.15.55:0/3136738196 conn(0x7fe67406fb40 msgr2=0x7fe6740c4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:02.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.205+0000 7fe67bddc640 1 -- 172.21.15.55:0/3136738196 shutdown_connections 2024-03-19T01:52:02.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.205+0000 7fe67bddc640 1 -- 172.21.15.55:0/3136738196 wait complete. 2024-03-19T01:52:02.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.213+0000 7fc3f97b0640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fc3f41510f0 0x7fc3f415a410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:33294/0 (socket says 172.21.15.55:33294) 2024-03-19T01:52:02.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.213+0000 7fc3f97b0640 1 -- 172.21.15.55:0/1670762027 learned_addr learned my addr 172.21.15.55:0/1670762027 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:52:02.227 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509465 got 64424509465 for osd.1 2024-03-19T01:52:02.228 DEBUG:teuthology.parallel:result is None 2024-03-19T01:52:02.356 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.353+0000 7f5a5214f640 1 -- 172.21.15.55:0/2974681700 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f5a4c0c42a0 con 0x7f5a4c0ccb10 2024-03-19T01:52:02.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.361+0000 7f5a33fff640 1 -- 172.21.15.55:0/2974681700 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5a40055cb0 con 0x7f5a4c0ccb10 2024-03-19T01:52:02.404 INFO:teuthology.orchestra.run.smithi055.stdout:64424509465 2024-03-19T01:52:02.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.361+0000 7f5a5214f640 1 -- 172.21.15.55:0/2974681700 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5a380676b0 msgr2=0x7f5a38069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.361+0000 7f5a5214f640 1 --2- 172.21.15.55:0/2974681700 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5a380676b0 0x7f5a38069b70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c0ab000 tx=0x7f5a4c15a380 comp rx=0 tx=0).stop 2024-03-19T01:52:02.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.361+0000 7f5a5214f640 1 -- 172.21.15.55:0/2974681700 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a4c0ccb10 msgr2=0x7f5a4c1f3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.361+0000 7f5a5214f640 1 --2- 172.21.15.55:0/2974681700 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a4c0ccb10 0x7f5a4c1f3f90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5a4000c3c0 tx=0x7f5a4000c880 comp rx=0 tx=0).stop 2024-03-19T01:52:02.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.361+0000 7f5a5214f640 1 -- 172.21.15.55:0/2974681700 shutdown_connections 2024-03-19T01:52:02.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.361+0000 7f5a5214f640 1 --2- 172.21.15.55:0/2974681700 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5a380676b0 0x7f5a38069b70 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.361+0000 7f5a5214f640 1 --2- 172.21.15.55:0/2974681700 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a4c152790 0x7f5a4c152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.361+0000 7f5a5214f640 1 --2- 172.21.15.55:0/2974681700 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a4c0ccb10 0x7f5a4c1f3f90 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.361+0000 7f5a5214f640 1 --2- 172.21.15.55:0/2974681700 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a34003a10 0x7f5a4c14f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.361+0000 7f5a5214f640 1 -- 172.21.15.55:0/2974681700 >> 172.21.15.55:0/2974681700 conn(0x7f5a4c070410 msgr2=0x7f5a4c0c6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:02.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.361+0000 7f5a5214f640 1 -- 172.21.15.55:0/2974681700 shutdown_connections 2024-03-19T01:52:02.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.361+0000 7f5a5214f640 1 -- 172.21.15.55:0/2974681700 wait complete. 2024-03-19T01:52:02.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.365+0000 7f4f5cff9640 1 -- 172.21.15.55:0/1170748642 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f4f58018020 con 0x7f4f60158c00 2024-03-19T01:52:02.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.365+0000 7f4f5cff9640 1 --2- 172.21.15.55:0/1170748642 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4f40067660 0x7f4f40069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:02.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.365+0000 7f4f5f7fe640 1 --2- 172.21.15.55:0/1170748642 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4f40067660 0x7f4f40069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:02.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.365+0000 7f4f5f7fe640 1 --2- 172.21.15.55:0/1170748642 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4f40067660 0x7f4f40069b20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4f50001cc0 tx=0x7f4f50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:02.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.377+0000 7f4f5cff9640 1 -- 172.21.15.55:0/1170748642 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f4f58054440 con 0x7f4f60158c00 2024-03-19T01:52:02.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.377+0000 7f4f5cff9640 1 -- 172.21.15.55:0/1170748642 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4f60204250 con 0x7f4f60158c00 2024-03-19T01:52:02.408 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509465 got 64424509465 for osd.6 2024-03-19T01:52:02.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.381+0000 7fc3f8faf640 1 --2- 172.21.15.55:0/1670762027 >> v2:172.21.15.55:3300/0 conn(0x7fc3f40c61c0 0x7fc3f40708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:02.408 DEBUG:teuthology.parallel:result is None 2024-03-19T01:52:02.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.393+0000 7fe79affd640 1 -- 172.21.15.55:0/2611546435 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fe79c017070 con 0x7fe7ac0c73c0 2024-03-19T01:52:02.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.393+0000 7fe79affd640 1 --2- 172.21.15.55:0/2611546435 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe78c067660 0x7fe78c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:02.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.393+0000 7fe7a9d9b640 1 --2- 172.21.15.55:0/2611546435 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe78c067660 0x7fe78c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:02.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.393+0000 7fe7a9d9b640 1 --2- 172.21.15.55:0/2611546435 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe78c067660 0x7fe78c069b20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe7a00b2000 tx=0x7fe7a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:02.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.397+0000 7fc3f2ffd640 1 -- 172.21.15.55:0/3080344912 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc3e4055d00 con 0x7fc3fc06c9f0 2024-03-19T01:52:02.410 INFO:teuthology.orchestra.run.smithi055.stdout:60129542167 2024-03-19T01:52:02.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.397+0000 7fc3f8faf640 1 -- 172.21.15.55:0/1670762027 >> v1:172.21.15.84:6789/0 conn(0x7fc3f41510f0 legacy=0x7fc3f415a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.397+0000 7fc3f8faf640 1 -- 172.21.15.55:0/1670762027 >> v1:172.21.15.55:6789/0 conn(0x7fc3f40c73c0 legacy=0x7fc3f41617a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.397+0000 7fc3f8faf640 1 -- 172.21.15.55:0/1670762027 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f40c6870 con 0x7fc3f40c61c0 2024-03-19T01:52:02.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.397+0000 7fc400fb8640 1 -- 172.21.15.55:0/3080344912 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3dc067690 msgr2=0x7fc3dc069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.397+0000 7fc400fb8640 1 --2- 172.21.15.55:0/3080344912 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3dc067690 0x7fc3dc069b50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec0b2000 tx=0x7fc3ec004740 comp rx=0 tx=0).stop 2024-03-19T01:52:02.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.397+0000 7fc400fb8640 1 -- 172.21.15.55:0/3080344912 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3fc06c9f0 msgr2=0x7fc3fc1387b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.397+0000 7fc400fb8640 1 --2- 172.21.15.55:0/3080344912 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3fc06c9f0 0x7fc3fc1387b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400b4b0 tx=0x7fc3e400b970 comp rx=0 tx=0).stop 2024-03-19T01:52:02.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.397+0000 7fc400fb8640 1 -- 172.21.15.55:0/3080344912 shutdown_connections 2024-03-19T01:52:02.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.397+0000 7fc400fb8640 1 --2- 172.21.15.55:0/3080344912 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3dc067690 0x7fc3dc069b50 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.397+0000 7fc400fb8640 1 --2- 172.21.15.55:0/3080344912 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3fc0c9800 0x7fc3fc0c4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.397+0000 7fc400fb8640 1 --2- 172.21.15.55:0/3080344912 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3fc06c9f0 0x7fc3fc1387b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.397+0000 7fc400fb8640 1 --2- 172.21.15.55:0/3080344912 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3fc06c610 0x7fc3fc141d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.397+0000 7fc400fb8640 1 -- 172.21.15.55:0/3080344912 >> 172.21.15.55:0/3080344912 conn(0x7fc3fc067600 msgr2=0x7fc3fc158020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:02.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.397+0000 7fc400fb8640 1 -- 172.21.15.55:0/3080344912 shutdown_connections 2024-03-19T01:52:02.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.397+0000 7fc400fb8640 1 -- 172.21.15.55:0/3080344912 wait complete. 2024-03-19T01:52:02.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.405+0000 7fc3f8faf640 1 --2- 172.21.15.55:0/1670762027 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3f40c61c0 0x7fc3f40708a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc3e40097c0 tx=0x7fc3e401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee79dc5ecb021737 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:02.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.405+0000 7fe79affd640 1 -- 172.21.15.55:0/2611546435 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fe79c054160 con 0x7fe7ac0c73c0 2024-03-19T01:52:02.418 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542167 got 60129542167 for osd.5 2024-03-19T01:52:02.418 DEBUG:teuthology.parallel:result is None 2024-03-19T01:52:02.440 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.441+0000 7fe79affd640 1 -- 172.21.15.55:0/2611546435 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fe79c0567c0 con 0x7fe7ac0c73c0 2024-03-19T01:52:02.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.473+0000 7fc3ebfff640 1 -- 172.21.15.55:0/1670762027 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e40b5070 con 0x7fc3f40c61c0 2024-03-19T01:52:02.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.473+0000 7fc3ebfff640 1 -- 172.21.15.55:0/1670762027 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3e40a8990 con 0x7fc3f40c61c0 2024-03-19T01:52:02.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.473+0000 7fc3ebfff640 1 -- 172.21.15.55:0/1670762027 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e40a8d60 con 0x7fc3f40c61c0 2024-03-19T01:52:02.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.477+0000 7fc3fba14640 1 -- 172.21.15.55:0/1670762027 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3f40c61c0 msgr2=0x7fc3f40708a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.477+0000 7fc3fba14640 1 --2- 172.21.15.55:0/1670762027 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3f40c61c0 0x7fc3f40708a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc3e40097c0 tx=0x7fc3e401ec90 comp rx=0 tx=0).stop 2024-03-19T01:52:02.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.477+0000 7fc3fba14640 1 -- 172.21.15.55:0/1670762027 shutdown_connections 2024-03-19T01:52:02.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.477+0000 7fc3fba14640 1 --2- 172.21.15.55:0/1670762027 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3f40c61c0 0x7fc3f40708a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.477+0000 7fc3fba14640 1 -- 172.21.15.55:0/1670762027 >> 172.21.15.55:0/1670762027 conn(0x7fc3f40cc9c0 msgr2=0x7fc3f40ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:02.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.477+0000 7fc3fba14640 1 -- 172.21.15.55:0/1670762027 shutdown_connections 2024-03-19T01:52:02.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.477+0000 7fc3fba14640 1 -- 172.21.15.55:0/1670762027 wait complete. 2024-03-19T01:52:02.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.477+0000 7fc3fba14640 1 Processor -- start 2024-03-19T01:52:02.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.477+0000 7fc3fba14640 1 -- start start 2024-03-19T01:52:02.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.477+0000 7fc3fba14640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3f40c61c0 0x7fc3f406c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:02.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.477+0000 7fc3fba14640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3f40c73c0 0x7fc3f40c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:02.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.477+0000 7fc3fba14640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3f41510f0 0x7fc3f41ef4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:02.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.477+0000 7fc3fba14640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f40c44e0 con 0x7fc3f40c61c0 2024-03-19T01:52:02.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.477+0000 7fc3fba14640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f40c4200 con 0x7fc3f40c73c0 2024-03-19T01:52:02.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.477+0000 7fc3fba14640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc3f40c6e30 con 0x7fc3f41510f0 2024-03-19T01:52:02.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.481+0000 7fc3f97b0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3f40c61c0 0x7fc3f406c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:02.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.481+0000 7fc3f97b0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3f40c61c0 0x7fc3f406c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53238/0 (socket says 172.21.15.55:53238) 2024-03-19T01:52:02.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.481+0000 7fc3f97b0640 1 -- 172.21.15.55:0/3078041043 learned_addr learned my addr 172.21.15.55:0/3078041043 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:02.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.485+0000 7fc3f97b0640 1 -- 172.21.15.55:0/3078041043 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3f41510f0 msgr2=0x7fc3f41ef4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.485+0000 7fc3f97b0640 1 --2- 172.21.15.55:0/3078041043 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3f41510f0 0x7fc3f41ef4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.485+0000 7fc3f97b0640 1 -- 172.21.15.55:0/3078041043 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3f40c73c0 msgr2=0x7fc3f40c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.485+0000 7fc3f97b0640 1 --2- 172.21.15.55:0/3078041043 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3f40c73c0 0x7fc3f40c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.485+0000 7fc3f97b0640 1 -- 172.21.15.55:0/3078041043 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f414c540 con 0x7fc3f40c61c0 2024-03-19T01:52:02.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.489+0000 7fc3f97b0640 1 --2- 172.21.15.55:0/3078041043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3f40c61c0 0x7fc3f406c4c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc3f000c530 tx=0x7fc3f000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:02.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.493+0000 7fc3e9ffb640 1 -- 172.21.15.55:0/3078041043 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0017070 con 0x7fc3f40c61c0 2024-03-19T01:52:02.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.493+0000 7fc3e9ffb640 1 -- 172.21.15.55:0/3078041043 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3f0005cf0 con 0x7fc3f40c61c0 2024-03-19T01:52:02.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.493+0000 7fc3fba14640 1 -- 172.21.15.55:0/3078041043 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3f414c820 con 0x7fc3f40c61c0 2024-03-19T01:52:02.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.493+0000 7fc3e9ffb640 1 -- 172.21.15.55:0/3078041043 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0007270 con 0x7fc3f40c61c0 2024-03-19T01:52:02.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.493+0000 7fc3fba14640 1 -- 172.21.15.55:0/3078041043 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3f414cde0 con 0x7fc3f40c61c0 2024-03-19T01:52:02.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.497+0000 7fc3fba14640 1 -- 172.21.15.55:0/3078041043 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3f4068a40 con 0x7fc3f40c61c0 2024-03-19T01:52:02.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.525+0000 7fc3e9ffb640 1 -- 172.21.15.55:0/3078041043 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc3f001a070 con 0x7fc3f40c61c0 2024-03-19T01:52:02.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.525+0000 7fc3e9ffb640 1 --2- 172.21.15.55:0/3078041043 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3d8067690 0x7fc3d8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:02.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.525+0000 7fc3f8faf640 1 --2- 172.21.15.55:0/3078041043 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3d8067690 0x7fc3d8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:02.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.529+0000 7fc3f8faf640 1 --2- 172.21.15.55:0/3078041043 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3d8067690 0x7fc3d8069b50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc3e40b2000 tx=0x7fc3e4004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:02.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.533+0000 7fc3e9ffb640 1 -- 172.21.15.55:0/3078041043 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fc3f00536a0 con 0x7fc3f40c61c0 2024-03-19T01:52:02.576 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.573+0000 7f4f65dc1640 1 -- 172.21.15.55:0/1170748642 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f4f60068a40 con 0x7f4f60158c00 2024-03-19T01:52:02.611 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.609+0000 7fe7b0d4f640 1 -- 172.21.15.55:0/2611546435 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fe790005860 con 0x7fe7ac0c73c0 2024-03-19T01:52:02.616 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.613+0000 7fe79affd640 1 -- 172.21.15.55:0/2611546435 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe79c010e20 con 0x7fe7ac0c73c0 2024-03-19T01:52:02.616 INFO:teuthology.orchestra.run.smithi055.stdout:60129542168 2024-03-19T01:52:02.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.617+0000 7fe7b0d4f640 1 -- 172.21.15.55:0/2611546435 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe78c067660 msgr2=0x7fe78c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.617+0000 7fe7b0d4f640 1 --2- 172.21.15.55:0/2611546435 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe78c067660 0x7fe78c069b20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe7a00b2000 tx=0x7fe7a0008040 comp rx=0 tx=0).stop 2024-03-19T01:52:02.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.617+0000 7fe7b0d4f640 1 -- 172.21.15.55:0/2611546435 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7ac0c73c0 msgr2=0x7fe7ac1306f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.617+0000 7fe7b0d4f640 1 --2- 172.21.15.55:0/2611546435 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7ac0c73c0 0x7fe7ac1306f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe79c00b550 tx=0x7fe79c00ba10 comp rx=0 tx=0).stop 2024-03-19T01:52:02.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.617+0000 7fe7b0d4f640 1 -- 172.21.15.55:0/2611546435 shutdown_connections 2024-03-19T01:52:02.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.617+0000 7fe7b0d4f640 1 --2- 172.21.15.55:0/2611546435 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe78c067660 0x7fe78c069b20 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.617+0000 7fe7b0d4f640 1 --2- 172.21.15.55:0/2611546435 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7ac15a410 0x7fe7ac1f4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.617+0000 7fe7b0d4f640 1 --2- 172.21.15.55:0/2611546435 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7ac1510f0 0x7fe7ac131e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.617+0000 7fe7b0d4f640 1 --2- 172.21.15.55:0/2611546435 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7ac0c73c0 0x7fe7ac1306f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.617+0000 7fe7b0d4f640 1 -- 172.21.15.55:0/2611546435 >> 172.21.15.55:0/2611546435 conn(0x7fe7ac0cc9c0 msgr2=0x7fe7ac14fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:02.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.617+0000 7fe7b0d4f640 1 -- 172.21.15.55:0/2611546435 shutdown_connections 2024-03-19T01:52:02.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.617+0000 7fe7b0d4f640 1 -- 172.21.15.55:0/2611546435 wait complete. 2024-03-19T01:52:02.634 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542168 got 60129542168 for osd.3 2024-03-19T01:52:02.634 DEBUG:teuthology.parallel:result is None 2024-03-19T01:52:02.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.817+0000 7fc3e9ffb640 1 -- 172.21.15.55:0/3078041043 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc3f0055600 con 0x7fc3f40c61c0 2024-03-19T01:52:02.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.949+0000 7f4f5cff9640 1 -- 172.21.15.55:0/1170748642 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4f5801f020 con 0x7f4f60158c00 2024-03-19T01:52:02.950 INFO:teuthology.orchestra.run.smithi055.stdout:64424509464 2024-03-19T01:52:02.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.953+0000 7f4f65dc1640 1 -- 172.21.15.55:0/1170748642 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4f40067660 msgr2=0x7f4f40069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.953+0000 7f4f65dc1640 1 --2- 172.21.15.55:0/1170748642 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4f40067660 0x7f4f40069b20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4f50001cc0 tx=0x7f4f50009040 comp rx=0 tx=0).stop 2024-03-19T01:52:02.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.953+0000 7f4f65dc1640 1 -- 172.21.15.55:0/1170748642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f60158c00 msgr2=0x7f4f60154b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:02.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.953+0000 7f4f65dc1640 1 --2- 172.21.15.55:0/1170748642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f60158c00 0x7f4f60154b90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4f5800b410 tx=0x7f4f5800b8d0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.953+0000 7f4f65dc1640 1 -- 172.21.15.55:0/1170748642 shutdown_connections 2024-03-19T01:52:02.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.953+0000 7f4f65dc1640 1 --2- 172.21.15.55:0/1170748642 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4f40067660 0x7f4f40069b20 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.953+0000 7f4f65dc1640 1 --2- 172.21.15.55:0/1170748642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f60158c00 0x7f4f60154b90 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.953+0000 7f4f65dc1640 1 --2- 172.21.15.55:0/1170748642 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f60158820 0x7f4f602035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.953+0000 7f4f65dc1640 1 --2- 172.21.15.55:0/1170748642 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4f600c79b0 0x7f4f60157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:02.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.953+0000 7f4f65dc1640 1 -- 172.21.15.55:0/1170748642 >> 172.21.15.55:0/1170748642 conn(0x7f4f600cc9c0 msgr2=0x7f4f600c9410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:02.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.953+0000 7f4f65dc1640 1 -- 172.21.15.55:0/1170748642 shutdown_connections 2024-03-19T01:52:02.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:02.953+0000 7f4f65dc1640 1 -- 172.21.15.55:0/1170748642 wait complete. 2024-03-19T01:52:02.974 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509464 got 64424509464 for osd.4 2024-03-19T01:52:02.974 DEBUG:teuthology.parallel:result is None 2024-03-19T01:52:03.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.001+0000 7fc3fba14640 1 -- 172.21.15.55:0/3078041043 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fc3f414a980 con 0x7fc3f40c61c0 2024-03-19T01:52:03.010 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.009+0000 7fc3e9ffb640 1 -- 172.21.15.55:0/3078041043 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc3f00596a0 con 0x7fc3f40c61c0 2024-03-19T01:52:03.010 INFO:teuthology.orchestra.run.smithi055.stdout:60129542167 2024-03-19T01:52:03.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.013+0000 7fc3fba14640 1 -- 172.21.15.55:0/3078041043 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3d8067690 msgr2=0x7fc3d8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:03.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.013+0000 7fc3fba14640 1 --2- 172.21.15.55:0/3078041043 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3d8067690 0x7fc3d8069b50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc3e40b2000 tx=0x7fc3e4004740 comp rx=0 tx=0).stop 2024-03-19T01:52:03.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.013+0000 7fc3fba14640 1 -- 172.21.15.55:0/3078041043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3f40c61c0 msgr2=0x7fc3f406c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:03.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.013+0000 7fc3fba14640 1 --2- 172.21.15.55:0/3078041043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3f40c61c0 0x7fc3f406c4c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc3f000c530 tx=0x7fc3f000c9f0 comp rx=0 tx=0).stop 2024-03-19T01:52:03.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.013+0000 7fc3fba14640 1 -- 172.21.15.55:0/3078041043 shutdown_connections 2024-03-19T01:52:03.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.013+0000 7fc3fba14640 1 --2- 172.21.15.55:0/3078041043 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3d8067690 0x7fc3d8069b50 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:03.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.013+0000 7fc3fba14640 1 --2- 172.21.15.55:0/3078041043 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3f41510f0 0x7fc3f41ef4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:03.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.013+0000 7fc3fba14640 1 --2- 172.21.15.55:0/3078041043 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3f40c73c0 0x7fc3f40c8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:03.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.013+0000 7fc3fba14640 1 --2- 172.21.15.55:0/3078041043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3f40c61c0 0x7fc3f406c4c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:03.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.013+0000 7fc3fba14640 1 -- 172.21.15.55:0/3078041043 >> 172.21.15.55:0/3078041043 conn(0x7fc3f40cc9c0 msgr2=0x7fc3f41501d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:03.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.013+0000 7fc3fba14640 1 -- 172.21.15.55:0/3078041043 shutdown_connections 2024-03-19T01:52:03.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.013+0000 7fc3fba14640 1 -- 172.21.15.55:0/3078041043 wait complete. 2024-03-19T01:52:03.028 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542167 got 60129542167 for osd.2 2024-03-19T01:52:03.029 DEBUG:teuthology.parallel:result is None 2024-03-19T01:52:03.029 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-19T01:52:03.029 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T01:52:03.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.177+0000 7fc003156640 1 Processor -- start 2024-03-19T01:52:03.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.177+0000 7fc003156640 1 -- start start 2024-03-19T01:52:03.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.177+0000 7fc003156640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fbffc1510f0 0x7fbffc0c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:03.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.177+0000 7fc003156640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbffc15a410 0x7fbffc0708a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:03.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.177+0000 7fc003156640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbffc0c3f60 con 0x7fbffc0c73c0 2024-03-19T01:52:03.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.177+0000 7fc003156640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fbffc1514d0 con 0x7fbffc1510f0 2024-03-19T01:52:03.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.177+0000 7fc003156640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbffc0c6870 con 0x7fbffc15a410 2024-03-19T01:52:03.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.181+0000 7fc0016f3640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fbffc0c73c0 0x7fbffc072890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:33296/0 (socket says 172.21.15.55:33296) 2024-03-19T01:52:03.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.181+0000 7fc0016f3640 1 -- 172.21.15.55:0/2311757418 learned_addr learned my addr 172.21.15.55:0/2311757418 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:52:03.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.181+0000 7fbffbfff640 1 --2- 172.21.15.55:0/2311757418 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbffc15a410 0x7fbffc0708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:03.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.185+0000 7fbffbfff640 1 -- 172.21.15.55:0/2311757418 >> v2:172.21.15.55:3300/0 conn(0x7fbffc1510f0 msgr2=0x7fbffc0c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:03.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.185+0000 7fbffbfff640 1 --2- 172.21.15.55:0/2311757418 >> v2:172.21.15.55:3300/0 conn(0x7fbffc1510f0 0x7fbffc0c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:03.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.185+0000 7fbffbfff640 1 -- 172.21.15.55:0/2311757418 >> v1:172.21.15.84:6789/0 conn(0x7fbffc0c73c0 legacy=0x7fbffc072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:03.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.185+0000 7fbffbfff640 1 -- 172.21.15.55:0/2311757418 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbffc0c6b50 con 0x7fbffc15a410 2024-03-19T01:52:03.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.189+0000 7fbffbfff640 1 --2- 172.21.15.55:0/2311757418 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbffc15a410 0x7fbffc0708a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbff00097c0 tx=0x7fbff001ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=285d421be3ca1ccc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:03.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.193+0000 7fbffb7fe640 1 -- 172.21.15.55:0/2311757418 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff00b5070 con 0x7fbffc15a410 2024-03-19T01:52:03.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.197+0000 7fbffb7fe640 1 -- 172.21.15.55:0/2311757418 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbff00a8990 con 0x7fbffc15a410 2024-03-19T01:52:03.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.197+0000 7fbffb7fe640 1 -- 172.21.15.55:0/2311757418 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff00a8d60 con 0x7fbffc15a410 2024-03-19T01:52:03.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.197+0000 7fc003156640 1 -- 172.21.15.55:0/2311757418 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbffc15a410 msgr2=0x7fbffc0708a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:03.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.197+0000 7fc003156640 1 --2- 172.21.15.55:0/2311757418 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbffc15a410 0x7fbffc0708a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbff00097c0 tx=0x7fbff001ec90 comp rx=0 tx=0).stop 2024-03-19T01:52:03.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.197+0000 7fc003156640 1 -- 172.21.15.55:0/2311757418 shutdown_connections 2024-03-19T01:52:03.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.197+0000 7fc003156640 1 --2- 172.21.15.55:0/2311757418 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbffc15a410 0x7fbffc0708a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:03.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.197+0000 7fc003156640 1 --2- 172.21.15.55:0/2311757418 >> v2:172.21.15.55:3300/0 conn(0x7fbffc1510f0 0x7fbffc0c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:03.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.197+0000 7fc003156640 1 -- 172.21.15.55:0/2311757418 >> 172.21.15.55:0/2311757418 conn(0x7fbffc0cc9c0 msgr2=0x7fbffc0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:03.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.197+0000 7fc003156640 1 -- 172.21.15.55:0/2311757418 shutdown_connections 2024-03-19T01:52:03.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.197+0000 7fc003156640 1 -- 172.21.15.55:0/2311757418 wait complete. 2024-03-19T01:52:03.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.201+0000 7fc003156640 1 Processor -- start 2024-03-19T01:52:03.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.201+0000 7fc003156640 1 -- start start 2024-03-19T01:52:03.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.201+0000 7fc003156640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbffc0c73c0 0x7fbffc1563e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:03.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.201+0000 7fc003156640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbffc1510f0 0x7fbffc1f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:03.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.201+0000 7fc003156640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbffc15a410 0x7fbffc1f4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:03.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.201+0000 7fc003156640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbffc0c6e30 con 0x7fbffc0c73c0 2024-03-19T01:52:03.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.201+0000 7fc003156640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fbffc0c44e0 con 0x7fbffc1510f0 2024-03-19T01:52:03.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.201+0000 7fc003156640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbffc0c4a00 con 0x7fbffc15a410 2024-03-19T01:52:03.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.201+0000 7fbffbfff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbffc1510f0 0x7fbffc1f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:03.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.205+0000 7fc0016f3640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbffc15a410 0x7fbffc1f4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:03.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.205+0000 7fc0016f3640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbffc15a410 0x7fbffc1f4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:42906/0 (socket says 172.21.15.55:42906) 2024-03-19T01:52:03.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.205+0000 7fc0016f3640 1 -- 172.21.15.55:0/281355093 learned_addr learned my addr 172.21.15.55:0/281355093 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:03.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.209+0000 7fc000ef2640 1 --2- 172.21.15.55:0/281355093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbffc0c73c0 0x7fbffc1563e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:03.217 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.217+0000 7fc000ef2640 1 -- 172.21.15.55:0/281355093 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbffc15a410 msgr2=0x7fbffc1f4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:03.217 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.217+0000 7fc000ef2640 1 --2- 172.21.15.55:0/281355093 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbffc15a410 0x7fbffc1f4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:03.217 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.217+0000 7fc000ef2640 1 -- 172.21.15.55:0/281355093 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbffc1510f0 msgr2=0x7fbffc1f3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:03.217 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.217+0000 7fc000ef2640 1 --2- 172.21.15.55:0/281355093 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbffc1510f0 0x7fbffc1f3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:03.217 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.217+0000 7fc000ef2640 1 -- 172.21.15.55:0/281355093 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbffc151870 con 0x7fbffc0c73c0 2024-03-19T01:52:03.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.221+0000 7fc000ef2640 1 --2- 172.21.15.55:0/281355093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbffc0c73c0 0x7fbffc1563e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbfec00bd70 tx=0x7fbfec00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:03.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.225+0000 7fbff97fa640 1 -- 172.21.15.55:0/281355093 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec019070 con 0x7fbffc0c73c0 2024-03-19T01:52:03.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.225+0000 7fbff97fa640 1 -- 172.21.15.55:0/281355093 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfec014440 con 0x7fbffc0c73c0 2024-03-19T01:52:03.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.225+0000 7fc003156640 1 -- 172.21.15.55:0/281355093 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbffc154170 con 0x7fbffc0c73c0 2024-03-19T01:52:03.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.225+0000 7fbff97fa640 1 -- 172.21.15.55:0/281355093 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec014830 con 0x7fbffc0c73c0 2024-03-19T01:52:03.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.225+0000 7fc003156640 1 -- 172.21.15.55:0/281355093 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbffc153a40 con 0x7fbffc0c73c0 2024-03-19T01:52:03.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.225+0000 7fc003156640 1 -- 172.21.15.55:0/281355093 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbffc068a40 con 0x7fbffc0c73c0 2024-03-19T01:52:03.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.261+0000 7fbff97fa640 1 -- 172.21.15.55:0/281355093 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fbfec01c070 con 0x7fbffc0c73c0 2024-03-19T01:52:03.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.261+0000 7fbff97fa640 1 --2- 172.21.15.55:0/281355093 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbfe80676b0 0x7fbfe8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:03.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.261+0000 7fbffbfff640 1 --2- 172.21.15.55:0/281355093 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbfe80676b0 0x7fbfe8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:03.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.265+0000 7fbffbfff640 1 --2- 172.21.15.55:0/281355093 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbfe80676b0 0x7fbfe8069b70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbff00b2000 tx=0x7fbff0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:03.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.265+0000 7fbff97fa640 1 -- 172.21.15.55:0/281355093 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fbfec0543e0 con 0x7fbffc0c73c0 2024-03-19T01:52:03.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.529+0000 7fbff97fa640 1 -- 172.21.15.55:0/281355093 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fbfec023020 con 0x7fbffc0c73c0 2024-03-19T01:52:03.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.745+0000 7fc003156640 1 -- 172.21.15.55:0/281355093 --> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbffc157110 con 0x7fbfe80676b0 2024-03-19T01:52:03.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.761+0000 7fbff97fa640 1 -- 172.21.15.55:0/281355093 <== mgr.4105 v2:172.21.15.84:6800/998533783 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332643 (secure 0 0 0) 0x7fbffc157110 con 0x7fbfe80676b0 2024-03-19T01:52:03.766 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:52:03.768 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-19T01:52:03.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.765+0000 7fc003156640 1 -- 172.21.15.55:0/281355093 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbfe80676b0 msgr2=0x7fbfe8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:03.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.765+0000 7fc003156640 1 --2- 172.21.15.55:0/281355093 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbfe80676b0 0x7fbfe8069b70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbff00b2000 tx=0x7fbff0008040 comp rx=0 tx=0).stop 2024-03-19T01:52:03.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.765+0000 7fc003156640 1 -- 172.21.15.55:0/281355093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbffc0c73c0 msgr2=0x7fbffc1563e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:03.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.765+0000 7fc003156640 1 --2- 172.21.15.55:0/281355093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbffc0c73c0 0x7fbffc1563e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbfec00bd70 tx=0x7fbfec00be60 comp rx=0 tx=0).stop 2024-03-19T01:52:03.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.765+0000 7fc003156640 1 -- 172.21.15.55:0/281355093 shutdown_connections 2024-03-19T01:52:03.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.765+0000 7fc003156640 1 --2- 172.21.15.55:0/281355093 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbfe80676b0 0x7fbfe8069b70 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:03.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.765+0000 7fc003156640 1 --2- 172.21.15.55:0/281355093 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbffc15a410 0x7fbffc1f4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:03.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.765+0000 7fc003156640 1 --2- 172.21.15.55:0/281355093 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbffc1510f0 0x7fbffc1f3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:03.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.765+0000 7fc003156640 1 --2- 172.21.15.55:0/281355093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbffc0c73c0 0x7fbffc1563e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:03.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.765+0000 7fc003156640 1 -- 172.21.15.55:0/281355093 >> 172.21.15.55:0/281355093 conn(0x7fbffc0cc9c0 msgr2=0x7fbffc159a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:03.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.765+0000 7fc003156640 1 -- 172.21.15.55:0/281355093 shutdown_connections 2024-03-19T01:52:03.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.765+0000 7fc003156640 1 -- 172.21.15.55:0/281355093 wait complete. 2024-03-19T01:52:03.783 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-03-19T01:52:02.330077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":599080,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":293,"num_write_kb":2462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236504,"kb_used_data":3736,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738216,"statfs":{"total":773094113280,"available":772851933184,"internally_reserved":0,"allocated":3825664,"data_stored":2189166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4072,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005212"},"pg_stats":[{"pgid":"2.1f","version":"29'9","reported_seq":38,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:57.562495+0000","last_change":"2024-03-19T01:51:06.982533+0000","last_active":"2024-03-19T01:51:57.562495+0000","last_peered":"2024-03-19T01:51:57.562495+0000","last_clean":"2024-03-19T01:51:57.562495+0000","last_became_active":"2024-03-19T01:51:06.981802+0000","last_became_peered":"2024-03-19T01:51:06.981802+0000","last_unstale":"2024-03-19T01:51:57.562495+0000","last_undegraded":"2024-03-19T01:51:57.562495+0000","last_fullsized":"2024-03-19T01:51:57.562495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:10:01.431003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086436+0000","last_change":"2024-03-19T01:51:09.172057+0000","last_active":"2024-03-19T01:51:18.086436+0000","last_peered":"2024-03-19T01:51:18.086436+0000","last_clean":"2024-03-19T01:51:18.086436+0000","last_became_active":"2024-03-19T01:51:09.171753+0000","last_became_peered":"2024-03-19T01:51:09.171753+0000","last_unstale":"2024-03-19T01:51:18.086436+0000","last_undegraded":"2024-03-19T01:51:18.086436+0000","last_fullsized":"2024-03-19T01:51:18.086436+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:50.575804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087409+0000","last_change":"2024-03-19T01:51:16.537753+0000","last_active":"2024-03-19T01:51:18.087409+0000","last_peered":"2024-03-19T01:51:18.087409+0000","last_clean":"2024-03-19T01:51:18.087409+0000","last_became_active":"2024-03-19T01:51:16.534200+0000","last_became_peered":"2024-03-19T01:51:16.534200+0000","last_unstale":"2024-03-19T01:51:18.087409+0000","last_undegraded":"2024-03-19T01:51:18.087409+0000","last_fullsized":"2024-03-19T01:51:18.087409+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:18:34.208854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086695+0000","last_change":"2024-03-19T01:51:06.962968+0000","last_active":"2024-03-19T01:51:18.086695+0000","last_peered":"2024-03-19T01:51:18.086695+0000","last_clean":"2024-03-19T01:51:18.086695+0000","last_became_active":"2024-03-19T01:51:06.961856+0000","last_became_peered":"2024-03-19T01:51:06.961856+0000","last_unstale":"2024-03-19T01:51:18.086695+0000","last_undegraded":"2024-03-19T01:51:18.086695+0000","last_fullsized":"2024-03-19T01:51:18.086695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:49.676651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746193+0000","last_change":"2024-03-19T01:51:09.160338+0000","last_active":"2024-03-19T01:51:17.746193+0000","last_peered":"2024-03-19T01:51:17.746193+0000","last_clean":"2024-03-19T01:51:17.746193+0000","last_became_active":"2024-03-19T01:51:09.158682+0000","last_became_peered":"2024-03-19T01:51:09.158682+0000","last_unstale":"2024-03-19T01:51:17.746193+0000","last_undegraded":"2024-03-19T01:51:17.746193+0000","last_fullsized":"2024-03-19T01:51:17.746193+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521163+0000","last_change":"2024-03-19T01:51:16.592390+0000","last_active":"2024-03-19T01:51:17.521163+0000","last_peered":"2024-03-19T01:51:17.521163+0000","last_clean":"2024-03-19T01:51:17.521163+0000","last_became_active":"2024-03-19T01:51:16.591093+0000","last_became_peered":"2024-03-19T01:51:16.591093+0000","last_unstale":"2024-03-19T01:51:17.521163+0000","last_undegraded":"2024-03-19T01:51:17.521163+0000","last_fullsized":"2024-03-19T01:51:17.521163+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:45.300462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:57.445250+0000","last_change":"2024-03-19T01:51:07.044049+0000","last_active":"2024-03-19T01:51:57.445250+0000","last_peered":"2024-03-19T01:51:57.445250+0000","last_clean":"2024-03-19T01:51:57.445250+0000","last_became_active":"2024-03-19T01:51:07.043209+0000","last_became_peered":"2024-03-19T01:51:07.043209+0000","last_unstale":"2024-03-19T01:51:57.445250+0000","last_undegraded":"2024-03-19T01:51:57.445250+0000","last_fullsized":"2024-03-19T01:51:57.445250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:42.992419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088027+0000","last_change":"2024-03-19T01:51:09.179615+0000","last_active":"2024-03-19T01:51:18.088027+0000","last_peered":"2024-03-19T01:51:18.088027+0000","last_clean":"2024-03-19T01:51:18.088027+0000","last_became_active":"2024-03-19T01:51:09.173962+0000","last_became_peered":"2024-03-19T01:51:09.173962+0000","last_unstale":"2024-03-19T01:51:18.088027+0000","last_undegraded":"2024-03-19T01:51:18.088027+0000","last_fullsized":"2024-03-19T01:51:18.088027+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:06:12.676868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522548+0000","last_change":"2024-03-19T01:51:16.593210+0000","last_active":"2024-03-19T01:51:17.522548+0000","last_peered":"2024-03-19T01:51:17.522548+0000","last_clean":"2024-03-19T01:51:17.522548+0000","last_became_active":"2024-03-19T01:51:16.591235+0000","last_became_peered":"2024-03-19T01:51:16.591235+0000","last_unstale":"2024-03-19T01:51:17.522548+0000","last_undegraded":"2024-03-19T01:51:17.522548+0000","last_fullsized":"2024-03-19T01:51:17.522548+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:13.248793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"25'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088902+0000","last_change":"2024-03-19T01:51:06.983291+0000","last_active":"2024-03-19T01:51:18.088902+0000","last_peered":"2024-03-19T01:51:18.088902+0000","last_clean":"2024-03-19T01:51:18.088902+0000","last_became_active":"2024-03-19T01:51:06.982309+0000","last_became_peered":"2024-03-19T01:51:06.982309+0000","last_unstale":"2024-03-19T01:51:18.088902+0000","last_undegraded":"2024-03-19T01:51:18.088902+0000","last_fullsized":"2024-03-19T01:51:18.088902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:27.305286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089984+0000","last_change":"2024-03-19T01:51:09.160738+0000","last_active":"2024-03-19T01:51:18.089984+0000","last_peered":"2024-03-19T01:51:18.089984+0000","last_clean":"2024-03-19T01:51:18.089984+0000","last_became_active":"2024-03-19T01:51:09.159951+0000","last_became_peered":"2024-03-19T01:51:09.159951+0000","last_unstale":"2024-03-19T01:51:18.089984+0000","last_undegraded":"2024-03-19T01:51:18.089984+0000","last_fullsized":"2024-03-19T01:51:18.089984+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:30.620041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521585+0000","last_change":"2024-03-19T01:51:16.577378+0000","last_active":"2024-03-19T01:51:17.521585+0000","last_peered":"2024-03-19T01:51:17.521585+0000","last_clean":"2024-03-19T01:51:17.521585+0000","last_became_active":"2024-03-19T01:51:16.576256+0000","last_became_peered":"2024-03-19T01:51:16.576256+0000","last_unstale":"2024-03-19T01:51:17.521585+0000","last_undegraded":"2024-03-19T01:51:17.521585+0000","last_fullsized":"2024-03-19T01:51:17.521585+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:16.096547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426589+0000","last_change":"2024-03-19T01:51:09.185969+0000","last_active":"2024-03-19T01:51:17.426589+0000","last_peered":"2024-03-19T01:51:17.426589+0000","last_clean":"2024-03-19T01:51:17.426589+0000","last_became_active":"2024-03-19T01:51:09.185378+0000","last_became_peered":"2024-03-19T01:51:09.185378+0000","last_unstale":"2024-03-19T01:51:17.426589+0000","last_undegraded":"2024-03-19T01:51:17.426589+0000","last_fullsized":"2024-03-19T01:51:17.426589+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.338974+0000","last_change":"2024-03-19T01:51:06.988067+0000","last_active":"2024-03-19T01:51:50.338974+0000","last_peered":"2024-03-19T01:51:50.338974+0000","last_clean":"2024-03-19T01:51:50.338974+0000","last_became_active":"2024-03-19T01:51:06.987450+0000","last_became_peered":"2024-03-19T01:51:06.987450+0000","last_unstale":"2024-03-19T01:51:50.338974+0000","last_undegraded":"2024-03-19T01:51:50.338974+0000","last_fullsized":"2024-03-19T01:51:50.338974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:56.962922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522867+0000","last_change":"2024-03-19T01:51:16.583190+0000","last_active":"2024-03-19T01:51:17.522867+0000","last_peered":"2024-03-19T01:51:17.522867+0000","last_clean":"2024-03-19T01:51:17.522867+0000","last_became_active":"2024-03-19T01:51:16.582532+0000","last_became_peered":"2024-03-19T01:51:16.582532+0000","last_unstale":"2024-03-19T01:51:17.522867+0000","last_undegraded":"2024-03-19T01:51:17.522867+0000","last_fullsized":"2024-03-19T01:51:17.522867+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:16:49.817940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085997+0000","last_change":"2024-03-19T01:51:09.174687+0000","last_active":"2024-03-19T01:51:18.085997+0000","last_peered":"2024-03-19T01:51:18.085997+0000","last_clean":"2024-03-19T01:51:18.085997+0000","last_became_active":"2024-03-19T01:51:09.174409+0000","last_became_peered":"2024-03-19T01:51:09.174409+0000","last_unstale":"2024-03-19T01:51:18.085997+0000","last_undegraded":"2024-03-19T01:51:18.085997+0000","last_fullsized":"2024-03-19T01:51:18.085997+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:33:07.160350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"29'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472423+0000","last_change":"2024-03-19T01:51:06.955434+0000","last_active":"2024-03-19T01:51:17.472423+0000","last_peered":"2024-03-19T01:51:17.472423+0000","last_clean":"2024-03-19T01:51:17.472423+0000","last_became_active":"2024-03-19T01:51:06.954635+0000","last_became_peered":"2024-03-19T01:51:06.954635+0000","last_unstale":"2024-03-19T01:51:17.472423+0000","last_undegraded":"2024-03-19T01:51:17.472423+0000","last_fullsized":"2024-03-19T01:51:17.472423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:41:39.176905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521820+0000","last_change":"2024-03-19T01:51:16.588604+0000","last_active":"2024-03-19T01:51:17.521820+0000","last_peered":"2024-03-19T01:51:17.521820+0000","last_clean":"2024-03-19T01:51:17.521820+0000","last_became_active":"2024-03-19T01:51:16.587483+0000","last_became_peered":"2024-03-19T01:51:16.587483+0000","last_unstale":"2024-03-19T01:51:17.521820+0000","last_undegraded":"2024-03-19T01:51:17.521820+0000","last_fullsized":"2024-03-19T01:51:17.521820+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:51.658547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086030+0000","last_change":"2024-03-19T01:51:09.173987+0000","last_active":"2024-03-19T01:51:18.086030+0000","last_peered":"2024-03-19T01:51:18.086030+0000","last_clean":"2024-03-19T01:51:18.086030+0000","last_became_active":"2024-03-19T01:51:09.173648+0000","last_became_peered":"2024-03-19T01:51:09.173648+0000","last_unstale":"2024-03-19T01:51:18.086030+0000","last_undegraded":"2024-03-19T01:51:18.086030+0000","last_fullsized":"2024-03-19T01:51:18.086030+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:13:17.277014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.374560+0000","last_change":"2024-03-19T01:51:06.981820+0000","last_active":"2024-03-19T01:51:50.374560+0000","last_peered":"2024-03-19T01:51:50.374560+0000","last_clean":"2024-03-19T01:51:50.374560+0000","last_became_active":"2024-03-19T01:51:06.981087+0000","last_became_peered":"2024-03-19T01:51:06.981087+0000","last_unstale":"2024-03-19T01:51:50.374560+0000","last_undegraded":"2024-03-19T01:51:50.374560+0000","last_fullsized":"2024-03-19T01:51:50.374560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:25:59.116138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.473071+0000","last_change":"2024-03-19T01:51:16.579017+0000","last_active":"2024-03-19T01:51:17.473071+0000","last_peered":"2024-03-19T01:51:17.473071+0000","last_clean":"2024-03-19T01:51:17.473071+0000","last_became_active":"2024-03-19T01:51:16.573845+0000","last_became_peered":"2024-03-19T01:51:16.573845+0000","last_unstale":"2024-03-19T01:51:17.473071+0000","last_undegraded":"2024-03-19T01:51:17.473071+0000","last_fullsized":"2024-03-19T01:51:17.473071+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:42:30.142766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522501+0000","last_change":"2024-03-19T01:51:09.192886+0000","last_active":"2024-03-19T01:51:17.522501+0000","last_peered":"2024-03-19T01:51:17.522501+0000","last_clean":"2024-03-19T01:51:17.522501+0000","last_became_active":"2024-03-19T01:51:09.185970+0000","last_became_peered":"2024-03-19T01:51:09.185970+0000","last_unstale":"2024-03-19T01:51:17.522501+0000","last_undegraded":"2024-03-19T01:51:17.522501+0000","last_fullsized":"2024-03-19T01:51:17.522501+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:05.196985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.339562+0000","last_change":"2024-03-19T01:51:06.984875+0000","last_active":"2024-03-19T01:51:50.339562+0000","last_peered":"2024-03-19T01:51:50.339562+0000","last_clean":"2024-03-19T01:51:50.339562+0000","last_became_active":"2024-03-19T01:51:06.983858+0000","last_became_peered":"2024-03-19T01:51:06.983858+0000","last_unstale":"2024-03-19T01:51:50.339562+0000","last_undegraded":"2024-03-19T01:51:50.339562+0000","last_fullsized":"2024-03-19T01:51:50.339562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:14.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522293+0000","last_change":"2024-03-19T01:51:16.591614+0000","last_active":"2024-03-19T01:51:17.522293+0000","last_peered":"2024-03-19T01:51:17.522293+0000","last_clean":"2024-03-19T01:51:17.522293+0000","last_became_active":"2024-03-19T01:51:16.590968+0000","last_became_peered":"2024-03-19T01:51:16.590968+0000","last_unstale":"2024-03-19T01:51:17.522293+0000","last_undegraded":"2024-03-19T01:51:17.522293+0000","last_fullsized":"2024-03-19T01:51:17.522293+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:50.144136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746685+0000","last_change":"2024-03-19T01:51:09.168941+0000","last_active":"2024-03-19T01:51:17.746685+0000","last_peered":"2024-03-19T01:51:17.746685+0000","last_clean":"2024-03-19T01:51:17.746685+0000","last_became_active":"2024-03-19T01:51:09.168243+0000","last_became_peered":"2024-03-19T01:51:09.168243+0000","last_unstale":"2024-03-19T01:51:17.746685+0000","last_undegraded":"2024-03-19T01:51:17.746685+0000","last_fullsized":"2024-03-19T01:51:17.746685+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.369887+0000","last_change":"2024-03-19T01:51:07.042870+0000","last_active":"2024-03-19T01:51:50.369887+0000","last_peered":"2024-03-19T01:51:50.369887+0000","last_clean":"2024-03-19T01:51:50.369887+0000","last_became_active":"2024-03-19T01:51:07.042327+0000","last_became_peered":"2024-03-19T01:51:07.042327+0000","last_unstale":"2024-03-19T01:51:50.369887+0000","last_undegraded":"2024-03-19T01:51:50.369887+0000","last_fullsized":"2024-03-19T01:51:50.369887+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:13.137673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521677+0000","last_change":"2024-03-19T01:51:16.587730+0000","last_active":"2024-03-19T01:51:17.521677+0000","last_peered":"2024-03-19T01:51:17.521677+0000","last_clean":"2024-03-19T01:51:17.521677+0000","last_became_active":"2024-03-19T01:51:16.586614+0000","last_became_peered":"2024-03-19T01:51:16.586614+0000","last_unstale":"2024-03-19T01:51:17.521677+0000","last_undegraded":"2024-03-19T01:51:17.521677+0000","last_fullsized":"2024-03-19T01:51:17.521677+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:44.830637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087909+0000","last_change":"2024-03-19T01:51:09.186397+0000","last_active":"2024-03-19T01:51:18.087909+0000","last_peered":"2024-03-19T01:51:18.087909+0000","last_clean":"2024-03-19T01:51:18.087909+0000","last_became_active":"2024-03-19T01:51:09.186033+0000","last_became_peered":"2024-03-19T01:51:09.186033+0000","last_unstale":"2024-03-19T01:51:18.087909+0000","last_undegraded":"2024-03-19T01:51:18.087909+0000","last_fullsized":"2024-03-19T01:51:18.087909+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:03:04.997711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:57.570619+0000","last_change":"2024-03-19T01:51:07.043697+0000","last_active":"2024-03-19T01:51:57.570619+0000","last_peered":"2024-03-19T01:51:57.570619+0000","last_clean":"2024-03-19T01:51:57.570619+0000","last_became_active":"2024-03-19T01:51:07.043185+0000","last_became_peered":"2024-03-19T01:51:07.043185+0000","last_unstale":"2024-03-19T01:51:57.570619+0000","last_undegraded":"2024-03-19T01:51:57.570619+0000","last_fullsized":"2024-03-19T01:51:57.570619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:19.016579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085498+0000","last_change":"2024-03-19T01:51:16.535771+0000","last_active":"2024-03-19T01:51:18.085498+0000","last_peered":"2024-03-19T01:51:18.085498+0000","last_clean":"2024-03-19T01:51:18.085498+0000","last_became_active":"2024-03-19T01:51:16.533661+0000","last_became_peered":"2024-03-19T01:51:16.533661+0000","last_unstale":"2024-03-19T01:51:18.085498+0000","last_undegraded":"2024-03-19T01:51:18.085498+0000","last_fullsized":"2024-03-19T01:51:18.085498+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:55.283782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472472+0000","last_change":"2024-03-19T01:51:09.157526+0000","last_active":"2024-03-19T01:51:17.472472+0000","last_peered":"2024-03-19T01:51:17.472472+0000","last_clean":"2024-03-19T01:51:17.472472+0000","last_became_active":"2024-03-19T01:51:09.156336+0000","last_became_peered":"2024-03-19T01:51:09.156336+0000","last_unstale":"2024-03-19T01:51:17.472472+0000","last_undegraded":"2024-03-19T01:51:17.472472+0000","last_fullsized":"2024-03-19T01:51:17.472472+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090045+0000","last_change":"2024-03-19T01:51:06.982994+0000","last_active":"2024-03-19T01:51:18.090045+0000","last_peered":"2024-03-19T01:51:18.090045+0000","last_clean":"2024-03-19T01:51:18.090045+0000","last_became_active":"2024-03-19T01:51:06.982133+0000","last_became_peered":"2024-03-19T01:51:06.982133+0000","last_unstale":"2024-03-19T01:51:18.090045+0000","last_undegraded":"2024-03-19T01:51:18.090045+0000","last_fullsized":"2024-03-19T01:51:18.090045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:03.574098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746076+0000","last_change":"2024-03-19T01:51:16.578493+0000","last_active":"2024-03-19T01:51:17.746076+0000","last_peered":"2024-03-19T01:51:17.746076+0000","last_clean":"2024-03-19T01:51:17.746076+0000","last_became_active":"2024-03-19T01:51:16.578040+0000","last_became_peered":"2024-03-19T01:51:16.578040+0000","last_unstale":"2024-03-19T01:51:17.746076+0000","last_undegraded":"2024-03-19T01:51:17.746076+0000","last_fullsized":"2024-03-19T01:51:17.746076+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:45.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090208+0000","last_change":"2024-03-19T01:51:09.180758+0000","last_active":"2024-03-19T01:51:18.090208+0000","last_peered":"2024-03-19T01:51:18.090208+0000","last_clean":"2024-03-19T01:51:18.090208+0000","last_became_active":"2024-03-19T01:51:09.180326+0000","last_became_peered":"2024-03-19T01:51:09.180326+0000","last_unstale":"2024-03-19T01:51:18.090208+0000","last_undegraded":"2024-03-19T01:51:18.090208+0000","last_fullsized":"2024-03-19T01:51:18.090208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:56:07.046285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472576+0000","last_change":"2024-03-19T01:51:06.962643+0000","last_active":"2024-03-19T01:51:17.472576+0000","last_peered":"2024-03-19T01:51:17.472576+0000","last_clean":"2024-03-19T01:51:17.472576+0000","last_became_active":"2024-03-19T01:51:06.961862+0000","last_became_peered":"2024-03-19T01:51:06.961862+0000","last_unstale":"2024-03-19T01:51:17.472576+0000","last_undegraded":"2024-03-19T01:51:17.472576+0000","last_fullsized":"2024-03-19T01:51:17.472576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.471619+0000","last_change":"2024-03-19T01:51:16.577730+0000","last_active":"2024-03-19T01:51:17.471619+0000","last_peered":"2024-03-19T01:51:17.471619+0000","last_clean":"2024-03-19T01:51:17.471619+0000","last_became_active":"2024-03-19T01:51:16.576955+0000","last_became_peered":"2024-03-19T01:51:16.576955+0000","last_unstale":"2024-03-19T01:51:17.471619+0000","last_undegraded":"2024-03-19T01:51:17.471619+0000","last_fullsized":"2024-03-19T01:51:17.471619+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:35.409504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522368+0000","last_change":"2024-03-19T01:51:09.183678+0000","last_active":"2024-03-19T01:51:17.522368+0000","last_peered":"2024-03-19T01:51:17.522368+0000","last_clean":"2024-03-19T01:51:17.522368+0000","last_became_active":"2024-03-19T01:51:09.176780+0000","last_became_peered":"2024-03-19T01:51:09.176780+0000","last_unstale":"2024-03-19T01:51:17.522368+0000","last_undegraded":"2024-03-19T01:51:17.522368+0000","last_fullsized":"2024-03-19T01:51:17.522368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:11:02.502333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089066+0000","last_change":"2024-03-19T01:51:06.956307+0000","last_active":"2024-03-19T01:51:18.089066+0000","last_peered":"2024-03-19T01:51:18.089066+0000","last_clean":"2024-03-19T01:51:18.089066+0000","last_became_active":"2024-03-19T01:51:06.955529+0000","last_became_peered":"2024-03-19T01:51:06.955529+0000","last_unstale":"2024-03-19T01:51:18.089066+0000","last_undegraded":"2024-03-19T01:51:18.089066+0000","last_fullsized":"2024-03-19T01:51:18.089066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:10.486012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472717+0000","last_change":"2024-03-19T01:51:16.550718+0000","last_active":"2024-03-19T01:51:17.472717+0000","last_peered":"2024-03-19T01:51:17.472717+0000","last_clean":"2024-03-19T01:51:17.472717+0000","last_became_active":"2024-03-19T01:51:16.550115+0000","last_became_peered":"2024-03-19T01:51:16.550115+0000","last_unstale":"2024-03-19T01:51:17.472717+0000","last_undegraded":"2024-03-19T01:51:17.472717+0000","last_fullsized":"2024-03-19T01:51:17.472717+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:43.172796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.471788+0000","last_change":"2024-03-19T01:51:09.155718+0000","last_active":"2024-03-19T01:51:17.471788+0000","last_peered":"2024-03-19T01:51:17.471788+0000","last_clean":"2024-03-19T01:51:17.471788+0000","last_became_active":"2024-03-19T01:51:09.155014+0000","last_became_peered":"2024-03-19T01:51:09.155014+0000","last_unstale":"2024-03-19T01:51:17.471788+0000","last_undegraded":"2024-03-19T01:51:17.471788+0000","last_fullsized":"2024-03-19T01:51:17.471788+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:57.450335+0000","last_change":"2024-03-19T01:51:06.955372+0000","last_active":"2024-03-19T01:51:57.450335+0000","last_peered":"2024-03-19T01:51:57.450335+0000","last_clean":"2024-03-19T01:51:57.450335+0000","last_became_active":"2024-03-19T01:51:06.954637+0000","last_became_peered":"2024-03-19T01:51:06.954637+0000","last_unstale":"2024-03-19T01:51:57.450335+0000","last_undegraded":"2024-03-19T01:51:57.450335+0000","last_fullsized":"2024-03-19T01:51:57.450335+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:38:20.129564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.473243+0000","last_change":"2024-03-19T01:51:16.575157+0000","last_active":"2024-03-19T01:51:17.473243+0000","last_peered":"2024-03-19T01:51:17.473243+0000","last_clean":"2024-03-19T01:51:17.473243+0000","last_became_active":"2024-03-19T01:51:16.574575+0000","last_became_peered":"2024-03-19T01:51:16.574575+0000","last_unstale":"2024-03-19T01:51:17.473243+0000","last_undegraded":"2024-03-19T01:51:17.473243+0000","last_fullsized":"2024-03-19T01:51:17.473243+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:15:17.718630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.745221+0000","last_change":"2024-03-19T01:51:09.192610+0000","last_active":"2024-03-19T01:51:17.745221+0000","last_peered":"2024-03-19T01:51:17.745221+0000","last_clean":"2024-03-19T01:51:17.745221+0000","last_became_active":"2024-03-19T01:51:09.192280+0000","last_became_peered":"2024-03-19T01:51:09.192280+0000","last_unstale":"2024-03-19T01:51:17.745221+0000","last_undegraded":"2024-03-19T01:51:17.745221+0000","last_fullsized":"2024-03-19T01:51:17.745221+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:15.454555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:57.446239+0000","last_change":"2024-03-19T01:51:06.986895+0000","last_active":"2024-03-19T01:51:57.446239+0000","last_peered":"2024-03-19T01:51:57.446239+0000","last_clean":"2024-03-19T01:51:57.446239+0000","last_became_active":"2024-03-19T01:51:06.985965+0000","last_became_peered":"2024-03-19T01:51:06.985965+0000","last_unstale":"2024-03-19T01:51:57.446239+0000","last_undegraded":"2024-03-19T01:51:57.446239+0000","last_fullsized":"2024-03-19T01:51:57.446239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746886+0000","last_change":"2024-03-19T01:51:16.505278+0000","last_active":"2024-03-19T01:51:17.746886+0000","last_peered":"2024-03-19T01:51:17.746886+0000","last_clean":"2024-03-19T01:51:17.746886+0000","last_became_active":"2024-03-19T01:51:16.504908+0000","last_became_peered":"2024-03-19T01:51:16.504908+0000","last_unstale":"2024-03-19T01:51:17.746886+0000","last_undegraded":"2024-03-19T01:51:17.746886+0000","last_fullsized":"2024-03-19T01:51:17.746886+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:42:30.142766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085144+0000","last_change":"2024-03-19T01:51:09.179725+0000","last_active":"2024-03-19T01:51:18.085144+0000","last_peered":"2024-03-19T01:51:18.085144+0000","last_clean":"2024-03-19T01:51:18.085144+0000","last_became_active":"2024-03-19T01:51:09.179235+0000","last_became_peered":"2024-03-19T01:51:09.179235+0000","last_unstale":"2024-03-19T01:51:18.085144+0000","last_undegraded":"2024-03-19T01:51:18.085144+0000","last_fullsized":"2024-03-19T01:51:18.085144+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:31.395833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:52:00.726787+0000","last_change":"2024-03-19T01:51:06.953474+0000","last_active":"2024-03-19T01:52:00.726787+0000","last_peered":"2024-03-19T01:52:00.726787+0000","last_clean":"2024-03-19T01:52:00.726787+0000","last_became_active":"2024-03-19T01:51:06.952898+0000","last_became_peered":"2024-03-19T01:51:06.952898+0000","last_unstale":"2024-03-19T01:52:00.726787+0000","last_undegraded":"2024-03-19T01:52:00.726787+0000","last_fullsized":"2024-03-19T01:52:00.726787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:38.808918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":184,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088303+0000","last_change":"2024-03-19T01:50:26.097484+0000","last_active":"2024-03-19T01:51:18.088303+0000","last_peered":"2024-03-19T01:51:18.088303+0000","last_clean":"2024-03-19T01:51:18.088303+0000","last_became_active":"2024-03-19T01:50:26.095496+0000","last_became_peered":"2024-03-19T01:50:26.095496+0000","last_unstale":"2024-03-19T01:51:18.088303+0000","last_undegraded":"2024-03-19T01:51:18.088303+0000","last_fullsized":"2024-03-19T01:51:18.088303+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:50:24.112070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:50:24.112070+0000","last_clean_scrub_stamp":"2024-03-19T01:50:24.112070+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:05:00.843775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426122+0000","last_change":"2024-03-19T01:51:16.580516+0000","last_active":"2024-03-19T01:51:17.426122+0000","last_peered":"2024-03-19T01:51:17.426122+0000","last_clean":"2024-03-19T01:51:17.426122+0000","last_became_active":"2024-03-19T01:51:16.579906+0000","last_became_peered":"2024-03-19T01:51:16.579906+0000","last_unstale":"2024-03-19T01:51:17.426122+0000","last_undegraded":"2024-03-19T01:51:17.426122+0000","last_fullsized":"2024-03-19T01:51:17.426122+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:53:57.897649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089446+0000","last_change":"2024-03-19T01:51:09.196648+0000","last_active":"2024-03-19T01:51:18.089446+0000","last_peered":"2024-03-19T01:51:18.089446+0000","last_clean":"2024-03-19T01:51:18.089446+0000","last_became_active":"2024-03-19T01:51:09.196093+0000","last_became_peered":"2024-03-19T01:51:09.196093+0000","last_unstale":"2024-03-19T01:51:18.089446+0000","last_undegraded":"2024-03-19T01:51:18.089446+0000","last_fullsized":"2024-03-19T01:51:18.089446+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:54:09.655994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:52:00.748884+0000","last_change":"2024-03-19T01:51:06.950023+0000","last_active":"2024-03-19T01:52:00.748884+0000","last_peered":"2024-03-19T01:52:00.748884+0000","last_clean":"2024-03-19T01:52:00.748884+0000","last_became_active":"2024-03-19T01:51:06.949234+0000","last_became_peered":"2024-03-19T01:51:06.949234+0000","last_unstale":"2024-03-19T01:52:00.748884+0000","last_undegraded":"2024-03-19T01:52:00.748884+0000","last_fullsized":"2024-03-19T01:52:00.748884+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:54:13.131325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425731+0000","last_change":"2024-03-19T01:51:16.554046+0000","last_active":"2024-03-19T01:51:17.425731+0000","last_peered":"2024-03-19T01:51:17.425731+0000","last_clean":"2024-03-19T01:51:17.425731+0000","last_became_active":"2024-03-19T01:51:16.553595+0000","last_became_peered":"2024-03-19T01:51:16.553595+0000","last_unstale":"2024-03-19T01:51:17.425731+0000","last_undegraded":"2024-03-19T01:51:17.425731+0000","last_fullsized":"2024-03-19T01:51:17.425731+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:25.684226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088279+0000","last_change":"2024-03-19T01:51:09.181518+0000","last_active":"2024-03-19T01:51:18.088279+0000","last_peered":"2024-03-19T01:51:18.088279+0000","last_clean":"2024-03-19T01:51:18.088279+0000","last_became_active":"2024-03-19T01:51:09.174122+0000","last_became_peered":"2024-03-19T01:51:09.174122+0000","last_unstale":"2024-03-19T01:51:18.088279+0000","last_undegraded":"2024-03-19T01:51:18.088279+0000","last_fullsized":"2024-03-19T01:51:18.088279+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:18:53.079890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"24'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472911+0000","last_change":"2024-03-19T01:51:06.938396+0000","last_active":"2024-03-19T01:51:17.472911+0000","last_peered":"2024-03-19T01:51:17.472911+0000","last_clean":"2024-03-19T01:51:17.472911+0000","last_became_active":"2024-03-19T01:51:06.937580+0000","last_became_peered":"2024-03-19T01:51:06.937580+0000","last_unstale":"2024-03-19T01:51:17.472911+0000","last_undegraded":"2024-03-19T01:51:17.472911+0000","last_fullsized":"2024-03-19T01:51:17.472911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:08.307376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746471+0000","last_change":"2024-03-19T01:51:16.582111+0000","last_active":"2024-03-19T01:51:17.746471+0000","last_peered":"2024-03-19T01:51:17.746471+0000","last_clean":"2024-03-19T01:51:17.746471+0000","last_became_active":"2024-03-19T01:51:16.581746+0000","last_became_peered":"2024-03-19T01:51:16.581746+0000","last_unstale":"2024-03-19T01:51:17.746471+0000","last_undegraded":"2024-03-19T01:51:17.746471+0000","last_fullsized":"2024-03-19T01:51:17.746471+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:19.442287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090581+0000","last_change":"2024-03-19T01:51:09.195225+0000","last_active":"2024-03-19T01:51:18.090581+0000","last_peered":"2024-03-19T01:51:18.090581+0000","last_clean":"2024-03-19T01:51:18.090581+0000","last_became_active":"2024-03-19T01:51:09.194645+0000","last_became_peered":"2024-03-19T01:51:09.194645+0000","last_unstale":"2024-03-19T01:51:18.090581+0000","last_undegraded":"2024-03-19T01:51:18.090581+0000","last_fullsized":"2024-03-19T01:51:18.090581+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:50:04.024767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086792+0000","last_change":"2024-03-19T01:51:06.955976+0000","last_active":"2024-03-19T01:51:18.086792+0000","last_peered":"2024-03-19T01:51:18.086792+0000","last_clean":"2024-03-19T01:51:18.086792+0000","last_became_active":"2024-03-19T01:51:06.954951+0000","last_became_peered":"2024-03-19T01:51:06.954951+0000","last_unstale":"2024-03-19T01:51:18.086792+0000","last_undegraded":"2024-03-19T01:51:18.086792+0000","last_fullsized":"2024-03-19T01:51:18.086792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:47.536586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087418+0000","last_change":"2024-03-19T01:51:16.586904+0000","last_active":"2024-03-19T01:51:18.087418+0000","last_peered":"2024-03-19T01:51:18.087418+0000","last_clean":"2024-03-19T01:51:18.087418+0000","last_became_active":"2024-03-19T01:51:16.586298+0000","last_became_peered":"2024-03-19T01:51:16.586298+0000","last_unstale":"2024-03-19T01:51:18.087418+0000","last_undegraded":"2024-03-19T01:51:18.087418+0000","last_fullsized":"2024-03-19T01:51:18.087418+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:42.703415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089106+0000","last_change":"2024-03-19T01:51:09.195400+0000","last_active":"2024-03-19T01:51:18.089106+0000","last_peered":"2024-03-19T01:51:18.089106+0000","last_clean":"2024-03-19T01:51:18.089106+0000","last_became_active":"2024-03-19T01:51:09.195111+0000","last_became_peered":"2024-03-19T01:51:09.195111+0000","last_unstale":"2024-03-19T01:51:18.089106+0000","last_undegraded":"2024-03-19T01:51:18.089106+0000","last_fullsized":"2024-03-19T01:51:18.089106+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:12.084475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"24'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522397+0000","last_change":"2024-03-19T01:51:06.963650+0000","last_active":"2024-03-19T01:51:17.522397+0000","last_peered":"2024-03-19T01:51:17.522397+0000","last_clean":"2024-03-19T01:51:17.522397+0000","last_became_active":"2024-03-19T01:51:06.963028+0000","last_became_peered":"2024-03-19T01:51:06.963028+0000","last_unstale":"2024-03-19T01:51:17.522397+0000","last_undegraded":"2024-03-19T01:51:17.522397+0000","last_fullsized":"2024-03-19T01:51:17.522397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:59:26.369856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521307+0000","last_change":"2024-03-19T01:51:16.588077+0000","last_active":"2024-03-19T01:51:17.521307+0000","last_peered":"2024-03-19T01:51:17.521307+0000","last_clean":"2024-03-19T01:51:17.521307+0000","last_became_active":"2024-03-19T01:51:16.587459+0000","last_became_peered":"2024-03-19T01:51:16.587459+0000","last_unstale":"2024-03-19T01:51:17.521307+0000","last_undegraded":"2024-03-19T01:51:17.521307+0000","last_fullsized":"2024-03-19T01:51:17.521307+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:52.667114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089490+0000","last_change":"2024-03-19T01:51:09.195758+0000","last_active":"2024-03-19T01:51:18.089490+0000","last_peered":"2024-03-19T01:51:18.089490+0000","last_clean":"2024-03-19T01:51:18.089490+0000","last_became_active":"2024-03-19T01:51:09.195139+0000","last_became_peered":"2024-03-19T01:51:09.195139+0000","last_unstale":"2024-03-19T01:51:18.089490+0000","last_undegraded":"2024-03-19T01:51:18.089490+0000","last_fullsized":"2024-03-19T01:51:18.089490+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:18.326414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"29'16","reported_seq":45,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:57.467794+0000","last_change":"2024-03-19T01:51:06.941200+0000","last_active":"2024-03-19T01:51:57.467794+0000","last_peered":"2024-03-19T01:51:57.467794+0000","last_clean":"2024-03-19T01:51:57.467794+0000","last_became_active":"2024-03-19T01:51:06.940594+0000","last_became_peered":"2024-03-19T01:51:06.940594+0000","last_unstale":"2024-03-19T01:51:57.467794+0000","last_undegraded":"2024-03-19T01:51:57.467794+0000","last_fullsized":"2024-03-19T01:51:57.467794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:25:05.442423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746500+0000","last_change":"2024-03-19T01:51:16.548804+0000","last_active":"2024-03-19T01:51:17.746500+0000","last_peered":"2024-03-19T01:51:17.746500+0000","last_clean":"2024-03-19T01:51:17.746500+0000","last_became_active":"2024-03-19T01:51:16.547800+0000","last_became_peered":"2024-03-19T01:51:16.547800+0000","last_unstale":"2024-03-19T01:51:17.746500+0000","last_undegraded":"2024-03-19T01:51:17.746500+0000","last_fullsized":"2024-03-19T01:51:17.746500+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:43.172796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425380+0000","last_change":"2024-03-19T01:51:09.167987+0000","last_active":"2024-03-19T01:51:17.425380+0000","last_peered":"2024-03-19T01:51:17.425380+0000","last_clean":"2024-03-19T01:51:17.425380+0000","last_became_active":"2024-03-19T01:51:09.167670+0000","last_became_peered":"2024-03-19T01:51:09.167670+0000","last_unstale":"2024-03-19T01:51:17.425380+0000","last_undegraded":"2024-03-19T01:51:17.425380+0000","last_fullsized":"2024-03-19T01:51:17.425380+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:57.426586+0000","last_change":"2024-03-19T01:51:07.048197+0000","last_active":"2024-03-19T01:51:57.426586+0000","last_peered":"2024-03-19T01:51:57.426586+0000","last_clean":"2024-03-19T01:51:57.426586+0000","last_became_active":"2024-03-19T01:51:07.047785+0000","last_became_peered":"2024-03-19T01:51:07.047785+0000","last_unstale":"2024-03-19T01:51:57.426586+0000","last_undegraded":"2024-03-19T01:51:57.426586+0000","last_fullsized":"2024-03-19T01:51:57.426586+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:52.323249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087324+0000","last_change":"2024-03-19T01:51:16.583394+0000","last_active":"2024-03-19T01:51:18.087324+0000","last_peered":"2024-03-19T01:51:18.087324+0000","last_clean":"2024-03-19T01:51:18.087324+0000","last_became_active":"2024-03-19T01:51:16.582860+0000","last_became_peered":"2024-03-19T01:51:16.582860+0000","last_unstale":"2024-03-19T01:51:18.087324+0000","last_undegraded":"2024-03-19T01:51:18.087324+0000","last_fullsized":"2024-03-19T01:51:18.087324+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:37.669591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.091680+0000","last_change":"2024-03-19T01:51:09.195008+0000","last_active":"2024-03-19T01:51:18.091680+0000","last_peered":"2024-03-19T01:51:18.091680+0000","last_clean":"2024-03-19T01:51:18.091680+0000","last_became_active":"2024-03-19T01:51:09.194728+0000","last_became_peered":"2024-03-19T01:51:09.194728+0000","last_unstale":"2024-03-19T01:51:18.091680+0000","last_undegraded":"2024-03-19T01:51:18.091680+0000","last_fullsized":"2024-03-19T01:51:18.091680+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:15:10.360817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.490405+0000","last_change":"2024-03-19T01:51:06.956905+0000","last_active":"2024-03-19T01:51:50.490405+0000","last_peered":"2024-03-19T01:51:50.490405+0000","last_clean":"2024-03-19T01:51:50.490405+0000","last_became_active":"2024-03-19T01:51:06.955991+0000","last_became_peered":"2024-03-19T01:51:06.955991+0000","last_unstale":"2024-03-19T01:51:50.490405+0000","last_undegraded":"2024-03-19T01:51:50.490405+0000","last_fullsized":"2024-03-19T01:51:50.490405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:02:45.970853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087664+0000","last_change":"2024-03-19T01:51:16.539701+0000","last_active":"2024-03-19T01:51:18.087664+0000","last_peered":"2024-03-19T01:51:18.087664+0000","last_clean":"2024-03-19T01:51:18.087664+0000","last_became_active":"2024-03-19T01:51:16.539254+0000","last_became_peered":"2024-03-19T01:51:16.539254+0000","last_unstale":"2024-03-19T01:51:18.087664+0000","last_undegraded":"2024-03-19T01:51:18.087664+0000","last_fullsized":"2024-03-19T01:51:18.087664+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:40:24.244131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.745703+0000","last_change":"2024-03-19T01:51:09.188003+0000","last_active":"2024-03-19T01:51:17.745703+0000","last_peered":"2024-03-19T01:51:17.745703+0000","last_clean":"2024-03-19T01:51:17.745703+0000","last_became_active":"2024-03-19T01:51:09.187275+0000","last_became_peered":"2024-03-19T01:51:09.187275+0000","last_unstale":"2024-03-19T01:51:17.745703+0000","last_undegraded":"2024-03-19T01:51:17.745703+0000","last_fullsized":"2024-03-19T01:51:17.745703+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:10.624259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"29'6","reported_seq":35,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:57.548834+0000","last_change":"2024-03-19T01:51:06.964413+0000","last_active":"2024-03-19T01:51:57.548834+0000","last_peered":"2024-03-19T01:51:57.548834+0000","last_clean":"2024-03-19T01:51:57.548834+0000","last_became_active":"2024-03-19T01:51:06.963763+0000","last_became_peered":"2024-03-19T01:51:06.963763+0000","last_unstale":"2024-03-19T01:51:57.548834+0000","last_undegraded":"2024-03-19T01:51:57.548834+0000","last_fullsized":"2024-03-19T01:51:57.548834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089726+0000","last_change":"2024-03-19T01:51:16.574401+0000","last_active":"2024-03-19T01:51:18.089726+0000","last_peered":"2024-03-19T01:51:18.089726+0000","last_clean":"2024-03-19T01:51:18.089726+0000","last_became_active":"2024-03-19T01:51:16.573744+0000","last_became_peered":"2024-03-19T01:51:16.573744+0000","last_unstale":"2024-03-19T01:51:18.089726+0000","last_undegraded":"2024-03-19T01:51:18.089726+0000","last_fullsized":"2024-03-19T01:51:18.089726+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:41.414631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089315+0000","last_change":"2024-03-19T01:51:09.187207+0000","last_active":"2024-03-19T01:51:18.089315+0000","last_peered":"2024-03-19T01:51:18.089315+0000","last_clean":"2024-03-19T01:51:18.089315+0000","last_became_active":"2024-03-19T01:51:09.175898+0000","last_became_peered":"2024-03-19T01:51:09.175898+0000","last_unstale":"2024-03-19T01:51:18.089315+0000","last_undegraded":"2024-03-19T01:51:18.089315+0000","last_fullsized":"2024-03-19T01:51:18.089315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:38.387069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'12","reported_seq":41,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.370011+0000","last_change":"2024-03-19T01:51:07.042663+0000","last_active":"2024-03-19T01:51:50.370011+0000","last_peered":"2024-03-19T01:51:50.370011+0000","last_clean":"2024-03-19T01:51:50.370011+0000","last_became_active":"2024-03-19T01:51:07.041262+0000","last_became_peered":"2024-03-19T01:51:07.041262+0000","last_unstale":"2024-03-19T01:51:50.370011+0000","last_undegraded":"2024-03-19T01:51:50.370011+0000","last_fullsized":"2024-03-19T01:51:50.370011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.471923+0000","last_change":"2024-03-19T01:51:16.549744+0000","last_active":"2024-03-19T01:51:17.471923+0000","last_peered":"2024-03-19T01:51:17.471923+0000","last_clean":"2024-03-19T01:51:17.471923+0000","last_became_active":"2024-03-19T01:51:16.549347+0000","last_became_peered":"2024-03-19T01:51:16.549347+0000","last_unstale":"2024-03-19T01:51:17.471923+0000","last_undegraded":"2024-03-19T01:51:17.471923+0000","last_fullsized":"2024-03-19T01:51:17.471923+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:19.442287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522393+0000","last_change":"2024-03-19T01:51:09.194165+0000","last_active":"2024-03-19T01:51:17.522393+0000","last_peered":"2024-03-19T01:51:17.522393+0000","last_clean":"2024-03-19T01:51:17.522393+0000","last_became_active":"2024-03-19T01:51:09.183607+0000","last_became_peered":"2024-03-19T01:51:09.183607+0000","last_unstale":"2024-03-19T01:51:17.522393+0000","last_undegraded":"2024-03-19T01:51:17.522393+0000","last_fullsized":"2024-03-19T01:51:17.522393+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:09.211794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.473060+0000","last_change":"2024-03-19T01:51:06.952515+0000","last_active":"2024-03-19T01:51:17.473060+0000","last_peered":"2024-03-19T01:51:17.473060+0000","last_clean":"2024-03-19T01:51:17.473060+0000","last_became_active":"2024-03-19T01:51:06.951700+0000","last_became_peered":"2024-03-19T01:51:06.951700+0000","last_unstale":"2024-03-19T01:51:17.473060+0000","last_undegraded":"2024-03-19T01:51:17.473060+0000","last_fullsized":"2024-03-19T01:51:17.473060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:13.137673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522152+0000","last_change":"2024-03-19T01:51:16.587027+0000","last_active":"2024-03-19T01:51:17.522152+0000","last_peered":"2024-03-19T01:51:17.522152+0000","last_clean":"2024-03-19T01:51:17.522152+0000","last_became_active":"2024-03-19T01:51:16.586514+0000","last_became_peered":"2024-03-19T01:51:16.586514+0000","last_unstale":"2024-03-19T01:51:17.522152+0000","last_undegraded":"2024-03-19T01:51:17.522152+0000","last_fullsized":"2024-03-19T01:51:17.522152+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:06.637617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746586+0000","last_change":"2024-03-19T01:51:09.172554+0000","last_active":"2024-03-19T01:51:17.746586+0000","last_peered":"2024-03-19T01:51:17.746586+0000","last_clean":"2024-03-19T01:51:17.746586+0000","last_became_active":"2024-03-19T01:51:09.171814+0000","last_became_peered":"2024-03-19T01:51:09.171814+0000","last_unstale":"2024-03-19T01:51:17.746586+0000","last_undegraded":"2024-03-19T01:51:17.746586+0000","last_fullsized":"2024-03-19T01:51:17.746586+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.488757+0000","last_change":"2024-03-19T01:51:07.043404+0000","last_active":"2024-03-19T01:51:50.488757+0000","last_peered":"2024-03-19T01:51:50.488757+0000","last_clean":"2024-03-19T01:51:50.488757+0000","last_became_active":"2024-03-19T01:51:07.043069+0000","last_became_peered":"2024-03-19T01:51:07.043069+0000","last_unstale":"2024-03-19T01:51:50.488757+0000","last_undegraded":"2024-03-19T01:51:50.488757+0000","last_fullsized":"2024-03-19T01:51:50.488757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:13:14.960132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085834+0000","last_change":"2024-03-19T01:51:16.581797+0000","last_active":"2024-03-19T01:51:18.085834+0000","last_peered":"2024-03-19T01:51:18.085834+0000","last_clean":"2024-03-19T01:51:18.085834+0000","last_became_active":"2024-03-19T01:51:16.581174+0000","last_became_peered":"2024-03-19T01:51:16.581174+0000","last_unstale":"2024-03-19T01:51:18.085834+0000","last_undegraded":"2024-03-19T01:51:18.085834+0000","last_fullsized":"2024-03-19T01:51:18.085834+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:29:34.911347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090870+0000","last_change":"2024-03-19T01:51:09.194372+0000","last_active":"2024-03-19T01:51:18.090870+0000","last_peered":"2024-03-19T01:51:18.090870+0000","last_clean":"2024-03-19T01:51:18.090870+0000","last_became_active":"2024-03-19T01:51:09.193656+0000","last_became_peered":"2024-03-19T01:51:09.193656+0000","last_unstale":"2024-03-19T01:51:18.090870+0000","last_undegraded":"2024-03-19T01:51:18.090870+0000","last_fullsized":"2024-03-19T01:51:18.090870+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:23.464803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.371819+0000","last_change":"2024-03-19T01:51:07.046089+0000","last_active":"2024-03-19T01:51:50.371819+0000","last_peered":"2024-03-19T01:51:50.371819+0000","last_clean":"2024-03-19T01:51:50.371819+0000","last_became_active":"2024-03-19T01:51:07.045342+0000","last_became_peered":"2024-03-19T01:51:07.045342+0000","last_unstale":"2024-03-19T01:51:50.371819+0000","last_undegraded":"2024-03-19T01:51:50.371819+0000","last_fullsized":"2024-03-19T01:51:50.371819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:26.105915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087835+0000","last_change":"2024-03-19T01:51:16.572559+0000","last_active":"2024-03-19T01:51:18.087835+0000","last_peered":"2024-03-19T01:51:18.087835+0000","last_clean":"2024-03-19T01:51:18.087835+0000","last_became_active":"2024-03-19T01:51:16.572283+0000","last_became_peered":"2024-03-19T01:51:16.572283+0000","last_unstale":"2024-03-19T01:51:18.087835+0000","last_undegraded":"2024-03-19T01:51:18.087835+0000","last_fullsized":"2024-03-19T01:51:18.087835+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:42.703415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746448+0000","last_change":"2024-03-19T01:51:16.534088+0000","last_active":"2024-03-19T01:51:17.746448+0000","last_peered":"2024-03-19T01:51:17.746448+0000","last_clean":"2024-03-19T01:51:17.746448+0000","last_became_active":"2024-03-19T01:51:16.533665+0000","last_became_peered":"2024-03-19T01:51:16.533665+0000","last_unstale":"2024-03-19T01:51:17.746448+0000","last_undegraded":"2024-03-19T01:51:17.746448+0000","last_fullsized":"2024-03-19T01:51:17.746448+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:35.409504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.471346+0000","last_change":"2024-03-19T01:51:09.168372+0000","last_active":"2024-03-19T01:51:17.471346+0000","last_peered":"2024-03-19T01:51:17.471346+0000","last_clean":"2024-03-19T01:51:17.471346+0000","last_became_active":"2024-03-19T01:51:09.167626+0000","last_became_peered":"2024-03-19T01:51:09.167626+0000","last_unstale":"2024-03-19T01:51:17.471346+0000","last_undegraded":"2024-03-19T01:51:17.471346+0000","last_fullsized":"2024-03-19T01:51:17.471346+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.489850+0000","last_change":"2024-03-19T01:51:06.951225+0000","last_active":"2024-03-19T01:51:50.489850+0000","last_peered":"2024-03-19T01:51:50.489850+0000","last_clean":"2024-03-19T01:51:50.489850+0000","last_became_active":"2024-03-19T01:51:06.950826+0000","last_became_peered":"2024-03-19T01:51:06.950826+0000","last_unstale":"2024-03-19T01:51:50.489850+0000","last_undegraded":"2024-03-19T01:51:50.489850+0000","last_fullsized":"2024-03-19T01:51:50.489850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:24.949925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426878+0000","last_change":"2024-03-19T01:51:16.581997+0000","last_active":"2024-03-19T01:51:17.426878+0000","last_peered":"2024-03-19T01:51:17.426878+0000","last_clean":"2024-03-19T01:51:17.426878+0000","last_became_active":"2024-03-19T01:51:16.581412+0000","last_became_peered":"2024-03-19T01:51:16.581412+0000","last_unstale":"2024-03-19T01:51:17.426878+0000","last_undegraded":"2024-03-19T01:51:17.426878+0000","last_fullsized":"2024-03-19T01:51:17.426878+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:45.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085931+0000","last_change":"2024-03-19T01:51:09.161686+0000","last_active":"2024-03-19T01:51:18.085931+0000","last_peered":"2024-03-19T01:51:18.085931+0000","last_clean":"2024-03-19T01:51:18.085931+0000","last_became_active":"2024-03-19T01:51:09.160978+0000","last_became_peered":"2024-03-19T01:51:09.160978+0000","last_unstale":"2024-03-19T01:51:18.085931+0000","last_undegraded":"2024-03-19T01:51:18.085931+0000","last_fullsized":"2024-03-19T01:51:18.085931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:27.266807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:57.449677+0000","last_change":"2024-03-19T01:51:07.047363+0000","last_active":"2024-03-19T01:51:57.449677+0000","last_peered":"2024-03-19T01:51:57.449677+0000","last_clean":"2024-03-19T01:51:57.449677+0000","last_became_active":"2024-03-19T01:51:07.046886+0000","last_became_peered":"2024-03-19T01:51:07.046886+0000","last_unstale":"2024-03-19T01:51:57.449677+0000","last_undegraded":"2024-03-19T01:51:57.449677+0000","last_fullsized":"2024-03-19T01:51:57.449677+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:19.418603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:57.425977+0000","last_change":"2024-03-19T01:51:07.045496+0000","last_active":"2024-03-19T01:51:57.425977+0000","last_peered":"2024-03-19T01:51:57.425977+0000","last_clean":"2024-03-19T01:51:57.425977+0000","last_became_active":"2024-03-19T01:51:07.044554+0000","last_became_peered":"2024-03-19T01:51:07.044554+0000","last_unstale":"2024-03-19T01:51:57.425977+0000","last_undegraded":"2024-03-19T01:51:57.425977+0000","last_fullsized":"2024-03-19T01:51:57.425977+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:14.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746492+0000","last_change":"2024-03-19T01:51:09.184934+0000","last_active":"2024-03-19T01:51:17.746492+0000","last_peered":"2024-03-19T01:51:17.746492+0000","last_clean":"2024-03-19T01:51:17.746492+0000","last_became_active":"2024-03-19T01:51:09.184666+0000","last_became_peered":"2024-03-19T01:51:09.184666+0000","last_unstale":"2024-03-19T01:51:17.746492+0000","last_undegraded":"2024-03-19T01:51:17.746492+0000","last_fullsized":"2024-03-19T01:51:17.746492+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:42.786323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085344+0000","last_change":"2024-03-19T01:51:16.575084+0000","last_active":"2024-03-19T01:51:18.085344+0000","last_peered":"2024-03-19T01:51:18.085344+0000","last_clean":"2024-03-19T01:51:18.085344+0000","last_became_active":"2024-03-19T01:51:16.574631+0000","last_became_peered":"2024-03-19T01:51:16.574631+0000","last_unstale":"2024-03-19T01:51:18.085344+0000","last_undegraded":"2024-03-19T01:51:18.085344+0000","last_fullsized":"2024-03-19T01:51:18.085344+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:24:35.634113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.503274+0000","last_change":"2024-03-19T01:51:06.982280+0000","last_active":"2024-03-19T01:51:50.503274+0000","last_peered":"2024-03-19T01:51:50.503274+0000","last_clean":"2024-03-19T01:51:50.503274+0000","last_became_active":"2024-03-19T01:51:06.981704+0000","last_became_peered":"2024-03-19T01:51:06.981704+0000","last_unstale":"2024-03-19T01:51:50.503274+0000","last_undegraded":"2024-03-19T01:51:50.503274+0000","last_fullsized":"2024-03-19T01:51:50.503274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:22:12.657985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521885+0000","last_change":"2024-03-19T01:51:09.184624+0000","last_active":"2024-03-19T01:51:17.521885+0000","last_peered":"2024-03-19T01:51:17.521885+0000","last_clean":"2024-03-19T01:51:17.521885+0000","last_became_active":"2024-03-19T01:51:09.183976+0000","last_became_peered":"2024-03-19T01:51:09.183976+0000","last_unstale":"2024-03-19T01:51:17.521885+0000","last_undegraded":"2024-03-19T01:51:17.521885+0000","last_fullsized":"2024-03-19T01:51:17.521885+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:24:29.313729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086581+0000","last_change":"2024-03-19T01:51:16.576775+0000","last_active":"2024-03-19T01:51:18.086581+0000","last_peered":"2024-03-19T01:51:18.086581+0000","last_clean":"2024-03-19T01:51:18.086581+0000","last_became_active":"2024-03-19T01:51:16.576406+0000","last_became_peered":"2024-03-19T01:51:16.576406+0000","last_unstale":"2024-03-19T01:51:18.086581+0000","last_undegraded":"2024-03-19T01:51:18.086581+0000","last_fullsized":"2024-03-19T01:51:18.086581+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:43.725447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425113+0000","last_change":"2024-03-19T01:51:09.169493+0000","last_active":"2024-03-19T01:51:17.425113+0000","last_peered":"2024-03-19T01:51:17.425113+0000","last_clean":"2024-03-19T01:51:17.425113+0000","last_became_active":"2024-03-19T01:51:09.168231+0000","last_became_peered":"2024-03-19T01:51:09.168231+0000","last_unstale":"2024-03-19T01:51:17.425113+0000","last_undegraded":"2024-03-19T01:51:17.425113+0000","last_fullsized":"2024-03-19T01:51:17.425113+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:10.624259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085344+0000","last_change":"2024-03-19T01:51:09.155483+0000","last_active":"2024-03-19T01:51:18.085344+0000","last_peered":"2024-03-19T01:51:18.085344+0000","last_clean":"2024-03-19T01:51:18.085344+0000","last_became_active":"2024-03-19T01:51:09.154545+0000","last_became_peered":"2024-03-19T01:51:09.154545+0000","last_unstale":"2024-03-19T01:51:18.085344+0000","last_undegraded":"2024-03-19T01:51:18.085344+0000","last_fullsized":"2024-03-19T01:51:18.085344+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:25:09.517323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088404+0000","last_change":"2024-03-19T01:51:09.180498+0000","last_active":"2024-03-19T01:51:18.088404+0000","last_peered":"2024-03-19T01:51:18.088404+0000","last_clean":"2024-03-19T01:51:18.088404+0000","last_became_active":"2024-03-19T01:51:09.179840+0000","last_became_peered":"2024-03-19T01:51:09.179840+0000","last_unstale":"2024-03-19T01:51:18.088404+0000","last_undegraded":"2024-03-19T01:51:18.088404+0000","last_fullsized":"2024-03-19T01:51:18.088404+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:01:21.716571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746226+0000","last_change":"2024-03-19T01:51:09.170389+0000","last_active":"2024-03-19T01:51:17.746226+0000","last_peered":"2024-03-19T01:51:17.746226+0000","last_clean":"2024-03-19T01:51:17.746226+0000","last_became_active":"2024-03-19T01:51:09.163974+0000","last_became_peered":"2024-03-19T01:51:09.163974+0000","last_unstale":"2024-03-19T01:51:17.746226+0000","last_undegraded":"2024-03-19T01:51:17.746226+0000","last_fullsized":"2024-03-19T01:51:17.746226+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087052+0000","last_change":"2024-03-19T01:51:09.177060+0000","last_active":"2024-03-19T01:51:18.087052+0000","last_peered":"2024-03-19T01:51:18.087052+0000","last_clean":"2024-03-19T01:51:18.087052+0000","last_became_active":"2024-03-19T01:51:09.176474+0000","last_became_peered":"2024-03-19T01:51:09.176474+0000","last_unstale":"2024-03-19T01:51:18.087052+0000","last_undegraded":"2024-03-19T01:51:18.087052+0000","last_fullsized":"2024-03-19T01:51:18.087052+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:56.318137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472633+0000","last_change":"2024-03-19T01:51:09.168741+0000","last_active":"2024-03-19T01:51:17.472633+0000","last_peered":"2024-03-19T01:51:17.472633+0000","last_clean":"2024-03-19T01:51:17.472633+0000","last_became_active":"2024-03-19T01:51:09.167982+0000","last_became_peered":"2024-03-19T01:51:09.167982+0000","last_unstale":"2024-03-19T01:51:17.472633+0000","last_undegraded":"2024-03-19T01:51:17.472633+0000","last_fullsized":"2024-03-19T01:51:17.472633+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425915+0000","last_change":"2024-03-19T01:51:09.189814+0000","last_active":"2024-03-19T01:51:17.425915+0000","last_peered":"2024-03-19T01:51:17.425915+0000","last_clean":"2024-03-19T01:51:17.425915+0000","last_became_active":"2024-03-19T01:51:09.189128+0000","last_became_peered":"2024-03-19T01:51:09.189128+0000","last_unstale":"2024-03-19T01:51:17.425915+0000","last_undegraded":"2024-03-19T01:51:17.425915+0000","last_fullsized":"2024-03-19T01:51:17.425915+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087688+0000","last_change":"2024-03-19T01:51:09.183989+0000","last_active":"2024-03-19T01:51:18.087688+0000","last_peered":"2024-03-19T01:51:18.087688+0000","last_clean":"2024-03-19T01:51:18.087688+0000","last_became_active":"2024-03-19T01:51:09.183417+0000","last_became_peered":"2024-03-19T01:51:09.183417+0000","last_unstale":"2024-03-19T01:51:18.087688+0000","last_undegraded":"2024-03-19T01:51:18.087688+0000","last_fullsized":"2024-03-19T01:51:18.087688+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:31.405618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087352+0000","last_change":"2024-03-19T01:51:09.172717+0000","last_active":"2024-03-19T01:51:18.087352+0000","last_peered":"2024-03-19T01:51:18.087352+0000","last_clean":"2024-03-19T01:51:18.087352+0000","last_became_active":"2024-03-19T01:51:09.165872+0000","last_became_peered":"2024-03-19T01:51:09.165872+0000","last_unstale":"2024-03-19T01:51:18.087352+0000","last_undegraded":"2024-03-19T01:51:18.087352+0000","last_fullsized":"2024-03-19T01:51:18.087352+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:10.995492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088202+0000","last_change":"2024-03-19T01:51:09.195116+0000","last_active":"2024-03-19T01:51:18.088202+0000","last_peered":"2024-03-19T01:51:18.088202+0000","last_clean":"2024-03-19T01:51:18.088202+0000","last_became_active":"2024-03-19T01:51:09.194558+0000","last_became_peered":"2024-03-19T01:51:09.194558+0000","last_unstale":"2024-03-19T01:51:18.088202+0000","last_undegraded":"2024-03-19T01:51:18.088202+0000","last_fullsized":"2024-03-19T01:51:18.088202+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:41:53.624234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088160+0000","last_change":"2024-03-19T01:51:09.195141+0000","last_active":"2024-03-19T01:51:18.088160+0000","last_peered":"2024-03-19T01:51:18.088160+0000","last_clean":"2024-03-19T01:51:18.088160+0000","last_became_active":"2024-03-19T01:51:09.194536+0000","last_became_peered":"2024-03-19T01:51:09.194536+0000","last_unstale":"2024-03-19T01:51:18.088160+0000","last_undegraded":"2024-03-19T01:51:18.088160+0000","last_fullsized":"2024-03-19T01:51:18.088160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:31.395833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.745890+0000","last_change":"2024-03-19T01:51:09.189197+0000","last_active":"2024-03-19T01:51:17.745890+0000","last_peered":"2024-03-19T01:51:17.745890+0000","last_clean":"2024-03-19T01:51:17.745890+0000","last_became_active":"2024-03-19T01:51:09.188565+0000","last_became_peered":"2024-03-19T01:51:09.188565+0000","last_unstale":"2024-03-19T01:51:17.745890+0000","last_undegraded":"2024-03-19T01:51:17.745890+0000","last_fullsized":"2024-03-19T01:51:17.745890+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426243+0000","last_change":"2024-03-19T01:51:09.168581+0000","last_active":"2024-03-19T01:51:17.426243+0000","last_peered":"2024-03-19T01:51:17.426243+0000","last_clean":"2024-03-19T01:51:17.426243+0000","last_became_active":"2024-03-19T01:51:09.167812+0000","last_became_peered":"2024-03-19T01:51:09.167812+0000","last_unstale":"2024-03-19T01:51:17.426243+0000","last_undegraded":"2024-03-19T01:51:17.426243+0000","last_fullsized":"2024-03-19T01:51:17.426243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089822+0000","last_change":"2024-03-19T01:51:09.188390+0000","last_active":"2024-03-19T01:51:18.089822+0000","last_peered":"2024-03-19T01:51:18.089822+0000","last_clean":"2024-03-19T01:51:18.089822+0000","last_became_active":"2024-03-19T01:51:09.187104+0000","last_became_peered":"2024-03-19T01:51:09.187104+0000","last_unstale":"2024-03-19T01:51:18.089822+0000","last_undegraded":"2024-03-19T01:51:18.089822+0000","last_fullsized":"2024-03-19T01:51:18.089822+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:37.257686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086435+0000","last_change":"2024-03-19T01:51:09.168451+0000","last_active":"2024-03-19T01:51:18.086435+0000","last_peered":"2024-03-19T01:51:18.086435+0000","last_clean":"2024-03-19T01:51:18.086435+0000","last_became_active":"2024-03-19T01:51:09.167744+0000","last_became_peered":"2024-03-19T01:51:09.167744+0000","last_unstale":"2024-03-19T01:51:18.086435+0000","last_undegraded":"2024-03-19T01:51:18.086435+0000","last_fullsized":"2024-03-19T01:51:18.086435+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:21.735485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089493+0000","last_change":"2024-03-19T01:51:09.196067+0000","last_active":"2024-03-19T01:51:18.089493+0000","last_peered":"2024-03-19T01:51:18.089493+0000","last_clean":"2024-03-19T01:51:18.089493+0000","last_became_active":"2024-03-19T01:51:09.195488+0000","last_became_peered":"2024-03-19T01:51:09.195488+0000","last_unstale":"2024-03-19T01:51:18.089493+0000","last_undegraded":"2024-03-19T01:51:18.089493+0000","last_fullsized":"2024-03-19T01:51:18.089493+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:07:44.120161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087081+0000","last_change":"2024-03-19T01:51:09.193831+0000","last_active":"2024-03-19T01:51:18.087081+0000","last_peered":"2024-03-19T01:51:18.087081+0000","last_clean":"2024-03-19T01:51:18.087081+0000","last_became_active":"2024-03-19T01:51:09.192777+0000","last_became_peered":"2024-03-19T01:51:09.192777+0000","last_unstale":"2024-03-19T01:51:18.087081+0000","last_undegraded":"2024-03-19T01:51:18.087081+0000","last_fullsized":"2024-03-19T01:51:18.087081+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:56.318137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.471571+0000","last_change":"2024-03-19T01:51:09.193581+0000","last_active":"2024-03-19T01:51:17.471571+0000","last_peered":"2024-03-19T01:51:17.471571+0000","last_clean":"2024-03-19T01:51:17.471571+0000","last_became_active":"2024-03-19T01:51:09.192677+0000","last_became_peered":"2024-03-19T01:51:09.192677+0000","last_unstale":"2024-03-19T01:51:17.471571+0000","last_undegraded":"2024-03-19T01:51:17.471571+0000","last_fullsized":"2024-03-19T01:51:17.471571+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.745598+0000","last_change":"2024-03-19T01:51:09.165568+0000","last_active":"2024-03-19T01:51:17.745598+0000","last_peered":"2024-03-19T01:51:17.745598+0000","last_clean":"2024-03-19T01:51:17.745598+0000","last_became_active":"2024-03-19T01:51:09.164753+0000","last_became_peered":"2024-03-19T01:51:09.164753+0000","last_unstale":"2024-03-19T01:51:17.745598+0000","last_undegraded":"2024-03-19T01:51:17.745598+0000","last_fullsized":"2024-03-19T01:51:17.745598+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:28.422798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086531+0000","last_change":"2024-03-19T01:51:09.170898+0000","last_active":"2024-03-19T01:51:18.086531+0000","last_peered":"2024-03-19T01:51:18.086531+0000","last_clean":"2024-03-19T01:51:18.086531+0000","last_became_active":"2024-03-19T01:51:09.170153+0000","last_became_peered":"2024-03-19T01:51:09.170153+0000","last_unstale":"2024-03-19T01:51:18.086531+0000","last_undegraded":"2024-03-19T01:51:18.086531+0000","last_fullsized":"2024-03-19T01:51:18.086531+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:41.125800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522152+0000","last_change":"2024-03-19T01:51:09.185953+0000","last_active":"2024-03-19T01:51:17.522152+0000","last_peered":"2024-03-19T01:51:17.522152+0000","last_clean":"2024-03-19T01:51:17.522152+0000","last_became_active":"2024-03-19T01:51:09.177996+0000","last_became_peered":"2024-03-19T01:51:09.177996+0000","last_unstale":"2024-03-19T01:51:17.522152+0000","last_undegraded":"2024-03-19T01:51:17.522152+0000","last_fullsized":"2024-03-19T01:51:17.522152+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:39:06.045413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426250+0000","last_change":"2024-03-19T01:51:09.183841+0000","last_active":"2024-03-19T01:51:17.426250+0000","last_peered":"2024-03-19T01:51:17.426250+0000","last_clean":"2024-03-19T01:51:17.426250+0000","last_became_active":"2024-03-19T01:51:09.183045+0000","last_became_peered":"2024-03-19T01:51:09.183045+0000","last_unstale":"2024-03-19T01:51:17.426250+0000","last_undegraded":"2024-03-19T01:51:17.426250+0000","last_fullsized":"2024-03-19T01:51:17.426250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425940+0000","last_change":"2024-03-19T01:51:09.184022+0000","last_active":"2024-03-19T01:51:17.425940+0000","last_peered":"2024-03-19T01:51:17.425940+0000","last_clean":"2024-03-19T01:51:17.425940+0000","last_became_active":"2024-03-19T01:51:09.183351+0000","last_became_peered":"2024-03-19T01:51:09.183351+0000","last_unstale":"2024-03-19T01:51:17.425940+0000","last_undegraded":"2024-03-19T01:51:17.425940+0000","last_fullsized":"2024-03-19T01:51:17.425940+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090466+0000","last_change":"2024-03-19T01:51:09.204005+0000","last_active":"2024-03-19T01:51:18.090466+0000","last_peered":"2024-03-19T01:51:18.090466+0000","last_clean":"2024-03-19T01:51:18.090466+0000","last_became_active":"2024-03-19T01:51:09.203545+0000","last_became_peered":"2024-03-19T01:51:09.203545+0000","last_unstale":"2024-03-19T01:51:18.090466+0000","last_undegraded":"2024-03-19T01:51:18.090466+0000","last_fullsized":"2024-03-19T01:51:18.090466+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:42:22.784953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085666+0000","last_change":"2024-03-19T01:51:09.180013+0000","last_active":"2024-03-19T01:51:18.085666+0000","last_peered":"2024-03-19T01:51:18.085666+0000","last_clean":"2024-03-19T01:51:18.085666+0000","last_became_active":"2024-03-19T01:51:09.179541+0000","last_became_peered":"2024-03-19T01:51:09.179541+0000","last_unstale":"2024-03-19T01:51:18.085666+0000","last_undegraded":"2024-03-19T01:51:18.085666+0000","last_fullsized":"2024-03-19T01:51:18.085666+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:19:25.933028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089242+0000","last_change":"2024-03-19T01:51:09.188671+0000","last_active":"2024-03-19T01:51:18.089242+0000","last_peered":"2024-03-19T01:51:18.089242+0000","last_clean":"2024-03-19T01:51:18.089242+0000","last_became_active":"2024-03-19T01:51:09.177627+0000","last_became_peered":"2024-03-19T01:51:09.177627+0000","last_unstale":"2024-03-19T01:51:18.089242+0000","last_undegraded":"2024-03-19T01:51:18.089242+0000","last_fullsized":"2024-03-19T01:51:18.089242+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:53:50.539837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087926+0000","last_change":"2024-03-19T01:51:09.180735+0000","last_active":"2024-03-19T01:51:18.087926+0000","last_peered":"2024-03-19T01:51:18.087926+0000","last_clean":"2024-03-19T01:51:18.087926+0000","last_became_active":"2024-03-19T01:51:09.173235+0000","last_became_peered":"2024-03-19T01:51:09.173235+0000","last_unstale":"2024-03-19T01:51:18.087926+0000","last_undegraded":"2024-03-19T01:51:18.087926+0000","last_fullsized":"2024-03-19T01:51:18.087926+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:31.405618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.091424+0000","last_change":"2024-03-19T01:51:09.188346+0000","last_active":"2024-03-19T01:51:18.091424+0000","last_peered":"2024-03-19T01:51:18.091424+0000","last_clean":"2024-03-19T01:51:18.091424+0000","last_became_active":"2024-03-19T01:51:09.187703+0000","last_became_peered":"2024-03-19T01:51:09.187703+0000","last_unstale":"2024-03-19T01:51:18.091424+0000","last_undegraded":"2024-03-19T01:51:18.091424+0000","last_fullsized":"2024-03-19T01:51:18.091424+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:28.051691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087787+0000","last_change":"2024-03-19T01:51:09.182114+0000","last_active":"2024-03-19T01:51:18.087787+0000","last_peered":"2024-03-19T01:51:18.087787+0000","last_clean":"2024-03-19T01:51:18.087787+0000","last_became_active":"2024-03-19T01:51:09.181056+0000","last_became_peered":"2024-03-19T01:51:09.181056+0000","last_unstale":"2024-03-19T01:51:18.087787+0000","last_undegraded":"2024-03-19T01:51:18.087787+0000","last_fullsized":"2024-03-19T01:51:18.087787+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:03:04.997711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746015+0000","last_change":"2024-03-19T01:51:09.161476+0000","last_active":"2024-03-19T01:51:17.746015+0000","last_peered":"2024-03-19T01:51:17.746015+0000","last_clean":"2024-03-19T01:51:17.746015+0000","last_became_active":"2024-03-19T01:51:09.158609+0000","last_became_peered":"2024-03-19T01:51:09.158609+0000","last_unstale":"2024-03-19T01:51:17.746015+0000","last_undegraded":"2024-03-19T01:51:17.746015+0000","last_fullsized":"2024-03-19T01:51:17.746015+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.473414+0000","last_change":"2024-03-19T01:51:09.165938+0000","last_active":"2024-03-19T01:51:17.473414+0000","last_peered":"2024-03-19T01:51:17.473414+0000","last_clean":"2024-03-19T01:51:17.473414+0000","last_became_active":"2024-03-19T01:51:09.165207+0000","last_became_peered":"2024-03-19T01:51:09.165207+0000","last_unstale":"2024-03-19T01:51:17.473414+0000","last_undegraded":"2024-03-19T01:51:17.473414+0000","last_fullsized":"2024-03-19T01:51:17.473414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090233+0000","last_change":"2024-03-19T01:51:09.176023+0000","last_active":"2024-03-19T01:51:18.090233+0000","last_peered":"2024-03-19T01:51:18.090233+0000","last_clean":"2024-03-19T01:51:18.090233+0000","last_became_active":"2024-03-19T01:51:09.175682+0000","last_became_peered":"2024-03-19T01:51:09.175682+0000","last_unstale":"2024-03-19T01:51:18.090233+0000","last_undegraded":"2024-03-19T01:51:18.090233+0000","last_fullsized":"2024-03-19T01:51:18.090233+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:35.814983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8712,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":134,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":16932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509463,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":921600,"data_stored":715706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509465,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":327680,"data_stored":128542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509464,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":323584,"data_stored":124736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509465,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31500,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340340,"statfs":{"total":96636764160,"available":96604508160,"internally_reserved":0,"allocated":331776,"data_stored":124966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509464,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339768,"statfs":{"total":96636764160,"available":96603922432,"internally_reserved":0,"allocated":917504,"data_stored":715050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542168,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":344064,"data_stored":126750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542169,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31500,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340340,"statfs":{"total":96636764160,"available":96604508160,"internally_reserved":0,"allocated":331776,"data_stored":128576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542168,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31496,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340344,"statfs":{"total":96636764160,"available":96604512256,"internally_reserved":0,"allocated":327680,"data_stored":124840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T01:52:03.787 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T01:52:03.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.929+0000 7fc7c103a640 1 Processor -- start 2024-03-19T01:52:03.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.929+0000 7fc7c103a640 1 -- start start 2024-03-19T01:52:03.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.929+0000 7fc7c103a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7bc154420 0x7fc7bc15dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:03.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.929+0000 7fc7c103a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc7bc15a1b0 0x7fc7bc0708a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:03.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.929+0000 7fc7c103a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7bc15a660 con 0x7fc7bc15e320 2024-03-19T01:52:03.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.929+0000 7fc7c103a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fc7bc15d950 con 0x7fc7bc15a1b0 2024-03-19T01:52:03.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.929+0000 7fc7c103a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc7bc15a940 con 0x7fc7bc154420 2024-03-19T01:52:03.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.929+0000 7fc7bad9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7bc154420 0x7fc7bc15dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:03.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.929+0000 7fc7ba59c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc7bc15a1b0 0x7fc7bc0708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:03.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.929+0000 7fc7bad9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7bc154420 0x7fc7bc15dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:42920/0 (socket says 172.21.15.55:42920) 2024-03-19T01:52:03.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.929+0000 7fc7bad9d640 1 -- 172.21.15.55:0/128051409 learned_addr learned my addr 172.21.15.55:0/128051409 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:03.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.933+0000 7fc7bad9d640 1 -- 172.21.15.55:0/128051409 >> v2:172.21.15.55:3300/0 conn(0x7fc7bc15a1b0 msgr2=0x7fc7bc0708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:03.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.933+0000 7fc7bad9d640 1 --2- 172.21.15.55:0/128051409 >> v2:172.21.15.55:3300/0 conn(0x7fc7bc15a1b0 0x7fc7bc0708a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:03.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.933+0000 7fc7bad9d640 1 -- 172.21.15.55:0/128051409 >> v1:172.21.15.55:6789/0 conn(0x7fc7bc15e320 legacy=0x7fc7bc0728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:03.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.933+0000 7fc7bad9d640 1 -- 172.21.15.55:0/128051409 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7bc15ac20 con 0x7fc7bc154420 2024-03-19T01:52:03.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.937+0000 7fc7bad9d640 1 --2- 172.21.15.55:0/128051409 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7bc154420 0x7fc7bc15dde0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009870 tx=0x7fc7ac01ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ef27fdc098939e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:03.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.941+0000 7fc7b9d9b640 1 -- 172.21.15.55:0/128051409 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac0b5070 con 0x7fc7bc154420 2024-03-19T01:52:03.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.945+0000 7fc7b9d9b640 1 -- 172.21.15.55:0/128051409 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7ac0a8a50 con 0x7fc7bc154420 2024-03-19T01:52:03.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.945+0000 7fc7c103a640 1 -- 172.21.15.55:0/128051409 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7bc154420 msgr2=0x7fc7bc15dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:03.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.945+0000 7fc7c103a640 1 --2- 172.21.15.55:0/128051409 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7bc154420 0x7fc7bc15dde0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009870 tx=0x7fc7ac01ed50 comp rx=0 tx=0).stop 2024-03-19T01:52:03.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.945+0000 7fc7c103a640 1 -- 172.21.15.55:0/128051409 shutdown_connections 2024-03-19T01:52:03.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.945+0000 7fc7c103a640 1 --2- 172.21.15.55:0/128051409 >> v2:172.21.15.55:3300/0 conn(0x7fc7bc15a1b0 0x7fc7bc0708a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:03.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.945+0000 7fc7c103a640 1 --2- 172.21.15.55:0/128051409 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7bc154420 0x7fc7bc15dde0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:03.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.945+0000 7fc7c103a640 1 -- 172.21.15.55:0/128051409 >> 172.21.15.55:0/128051409 conn(0x7fc7bc06c3b0 msgr2=0x7fc7bc06c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:03.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.945+0000 7fc7c103a640 1 -- 172.21.15.55:0/128051409 shutdown_connections 2024-03-19T01:52:03.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.945+0000 7fc7c103a640 1 -- 172.21.15.55:0/128051409 wait complete. 2024-03-19T01:52:03.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.949+0000 7fc7c103a640 1 Processor -- start 2024-03-19T01:52:03.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.949+0000 7fc7c103a640 1 -- start start 2024-03-19T01:52:03.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.949+0000 7fc7c103a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc7bc154420 0x7fc7bc146f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:03.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.949+0000 7fc7c103a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7bc15a1b0 0x7fc7bc14b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:03.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.949+0000 7fc7c103a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7bc15e320 0x7fc7bc14b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:03.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.949+0000 7fc7c103a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc154980 con 0x7fc7bc15e320 2024-03-19T01:52:03.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.949+0000 7fc7c103a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc154800 con 0x7fc7bc154420 2024-03-19T01:52:03.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.949+0000 7fc7c103a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc7bc070de0 con 0x7fc7bc15a1b0 2024-03-19T01:52:03.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.949+0000 7fc7bb59e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7bc15e320 0x7fc7bc14b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:03.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.949+0000 7fc7bb59e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7bc15e320 0x7fc7bc14b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53278/0 (socket says 172.21.15.55:53278) 2024-03-19T01:52:03.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.949+0000 7fc7bb59e640 1 -- 172.21.15.55:0/3433654042 learned_addr learned my addr 172.21.15.55:0/3433654042 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:03.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.949+0000 7fc7bad9d640 1 --2- 172.21.15.55:0/3433654042 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc7bc154420 0x7fc7bc146f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:03.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.949+0000 7fc7ba59c640 1 --2- 172.21.15.55:0/3433654042 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7bc15a1b0 0x7fc7bc14b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:03.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.953+0000 7fc7bb59e640 1 -- 172.21.15.55:0/3433654042 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7bc15a1b0 msgr2=0x7fc7bc14b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:03.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.953+0000 7fc7bb59e640 1 --2- 172.21.15.55:0/3433654042 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7bc15a1b0 0x7fc7bc14b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:03.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.953+0000 7fc7bb59e640 1 -- 172.21.15.55:0/3433654042 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc7bc154420 msgr2=0x7fc7bc146f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:03.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.953+0000 7fc7bb59e640 1 --2- 172.21.15.55:0/3433654042 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc7bc154420 0x7fc7bc146f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:03.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.953+0000 7fc7bb59e640 1 -- 172.21.15.55:0/3433654042 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7bc1474c0 con 0x7fc7bc15e320 2024-03-19T01:52:03.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.953+0000 7fc7bb59e640 1 --2- 172.21.15.55:0/3433654042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7bc15e320 0x7fc7bc14b8c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc7b400c730 tx=0x7fc7b400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:03.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.957+0000 7fc7a37fe640 1 -- 172.21.15.55:0/3433654042 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4003b80 con 0x7fc7bc15e320 2024-03-19T01:52:03.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.957+0000 7fc7c103a640 1 -- 172.21.15.55:0/3433654042 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7bc1f3b30 con 0x7fc7bc15e320 2024-03-19T01:52:03.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.957+0000 7fc7c103a640 1 -- 172.21.15.55:0/3433654042 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7bc1efda0 con 0x7fc7bc15e320 2024-03-19T01:52:03.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:03.957+0000 7fc7c103a640 1 -- 172.21.15.55:0/3433654042 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7bc068a40 con 0x7fc7bc15e320 2024-03-19T01:52:04.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.169+0000 7fc7a37fe640 1 -- 172.21.15.55:0/3433654042 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7b400ce40 con 0x7fc7bc15e320 2024-03-19T01:52:04.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.301+0000 7fc7a37fe640 1 -- 172.21.15.55:0/3433654042 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b400a7f0 con 0x7fc7bc15e320 2024-03-19T01:52:04.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.301+0000 7fc7a37fe640 1 -- 172.21.15.55:0/3433654042 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc7b400aa50 con 0x7fc7bc15e320 2024-03-19T01:52:04.303 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.301+0000 7fc7a37fe640 1 --2- 172.21.15.55:0/3433654042 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc7a8067660 0x7fc7a8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:04.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.301+0000 7fc7bad9d640 1 --2- 172.21.15.55:0/3433654042 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc7a8067660 0x7fc7a8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:04.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.305+0000 7fc7bad9d640 1 --2- 172.21.15.55:0/3433654042 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc7a8067660 0x7fc7a8069b20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac0b2000 tx=0x7fc7ac0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:04.342 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.341+0000 7fc7a37fe640 1 -- 172.21.15.55:0/3433654042 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fc7b4054800 con 0x7fc7bc15e320 2024-03-19T01:52:04.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.341+0000 7fc7a37fe640 1 -- 172.21.15.55:0/3433654042 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc7b4056760 con 0x7fc7bc15e320 2024-03-19T01:52:04.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.517+0000 7fc7c103a640 1 -- 172.21.15.55:0/3433654042 --> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc7bc1f40b0 con 0x7fc7a8067660 2024-03-19T01:52:04.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.533+0000 7fc7a37fe640 1 -- 172.21.15.55:0/3433654042 <== mgr.4105 v2:172.21.15.84:6800/998533783 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332707 (secure 0 0 0) 0x7fc7bc1f40b0 con 0x7fc7a8067660 2024-03-19T01:52:04.538 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:52:04.540 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-19T01:52:04.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.537+0000 7fc7c103a640 1 -- 172.21.15.55:0/3433654042 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc7a8067660 msgr2=0x7fc7a8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:04.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.537+0000 7fc7c103a640 1 --2- 172.21.15.55:0/3433654042 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc7a8067660 0x7fc7a8069b20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac0b2000 tx=0x7fc7ac0aa070 comp rx=0 tx=0).stop 2024-03-19T01:52:04.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.537+0000 7fc7c103a640 1 -- 172.21.15.55:0/3433654042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7bc15e320 msgr2=0x7fc7bc14b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:04.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.537+0000 7fc7c103a640 1 --2- 172.21.15.55:0/3433654042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7bc15e320 0x7fc7bc14b8c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc7b400c730 tx=0x7fc7b400cbf0 comp rx=0 tx=0).stop 2024-03-19T01:52:04.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.537+0000 7fc7c103a640 1 -- 172.21.15.55:0/3433654042 shutdown_connections 2024-03-19T01:52:04.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.537+0000 7fc7c103a640 1 --2- 172.21.15.55:0/3433654042 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc7a8067660 0x7fc7a8069b20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:04.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.537+0000 7fc7c103a640 1 --2- 172.21.15.55:0/3433654042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7bc15e320 0x7fc7bc14b8c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:04.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.537+0000 7fc7c103a640 1 --2- 172.21.15.55:0/3433654042 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7bc15a1b0 0x7fc7bc14b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:04.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.537+0000 7fc7c103a640 1 --2- 172.21.15.55:0/3433654042 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc7bc154420 0x7fc7bc146f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:04.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.537+0000 7fc7c103a640 1 -- 172.21.15.55:0/3433654042 >> 172.21.15.55:0/3433654042 conn(0x7fc7bc06c3b0 msgr2=0x7fc7bc153290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:04.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.537+0000 7fc7c103a640 1 -- 172.21.15.55:0/3433654042 shutdown_connections 2024-03-19T01:52:04.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.537+0000 7fc7c103a640 1 -- 172.21.15.55:0/3433654042 wait complete. 2024-03-19T01:52:04.558 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-03-19T01:52:04.330791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":599226,"num_objects":79,"num_object_clones":0,"num_object_copies":158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":326,"num_write_kb":2471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":263,"ondisk_log_size":263,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236528,"kb_used_data":3760,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738192,"statfs":{"total":773094113280,"available":772851908608,"internally_reserved":0,"allocated":3850240,"data_stored":2189950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3596,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004920"},"pg_stats":[{"pgid":"2.1f","version":"29'11","reported_seq":40,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:52:00.679264+0000","last_change":"2024-03-19T01:51:06.982533+0000","last_active":"2024-03-19T01:52:00.679264+0000","last_peered":"2024-03-19T01:52:00.679264+0000","last_clean":"2024-03-19T01:52:00.679264+0000","last_became_active":"2024-03-19T01:51:06.981802+0000","last_became_peered":"2024-03-19T01:51:06.981802+0000","last_unstale":"2024-03-19T01:52:00.679264+0000","last_undegraded":"2024-03-19T01:52:00.679264+0000","last_fullsized":"2024-03-19T01:52:00.679264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:10:01.431003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086436+0000","last_change":"2024-03-19T01:51:09.172057+0000","last_active":"2024-03-19T01:51:18.086436+0000","last_peered":"2024-03-19T01:51:18.086436+0000","last_clean":"2024-03-19T01:51:18.086436+0000","last_became_active":"2024-03-19T01:51:09.171753+0000","last_became_peered":"2024-03-19T01:51:09.171753+0000","last_unstale":"2024-03-19T01:51:18.086436+0000","last_undegraded":"2024-03-19T01:51:18.086436+0000","last_fullsized":"2024-03-19T01:51:18.086436+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:50.575804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087409+0000","last_change":"2024-03-19T01:51:16.537753+0000","last_active":"2024-03-19T01:51:18.087409+0000","last_peered":"2024-03-19T01:51:18.087409+0000","last_clean":"2024-03-19T01:51:18.087409+0000","last_became_active":"2024-03-19T01:51:16.534200+0000","last_became_peered":"2024-03-19T01:51:16.534200+0000","last_unstale":"2024-03-19T01:51:18.087409+0000","last_undegraded":"2024-03-19T01:51:18.087409+0000","last_fullsized":"2024-03-19T01:51:18.087409+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:18:34.208854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086695+0000","last_change":"2024-03-19T01:51:06.962968+0000","last_active":"2024-03-19T01:51:18.086695+0000","last_peered":"2024-03-19T01:51:18.086695+0000","last_clean":"2024-03-19T01:51:18.086695+0000","last_became_active":"2024-03-19T01:51:06.961856+0000","last_became_peered":"2024-03-19T01:51:06.961856+0000","last_unstale":"2024-03-19T01:51:18.086695+0000","last_undegraded":"2024-03-19T01:51:18.086695+0000","last_fullsized":"2024-03-19T01:51:18.086695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:49.676651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746193+0000","last_change":"2024-03-19T01:51:09.160338+0000","last_active":"2024-03-19T01:51:17.746193+0000","last_peered":"2024-03-19T01:51:17.746193+0000","last_clean":"2024-03-19T01:51:17.746193+0000","last_became_active":"2024-03-19T01:51:09.158682+0000","last_became_peered":"2024-03-19T01:51:09.158682+0000","last_unstale":"2024-03-19T01:51:17.746193+0000","last_undegraded":"2024-03-19T01:51:17.746193+0000","last_fullsized":"2024-03-19T01:51:17.746193+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521163+0000","last_change":"2024-03-19T01:51:16.592390+0000","last_active":"2024-03-19T01:51:17.521163+0000","last_peered":"2024-03-19T01:51:17.521163+0000","last_clean":"2024-03-19T01:51:17.521163+0000","last_became_active":"2024-03-19T01:51:16.591093+0000","last_became_peered":"2024-03-19T01:51:16.591093+0000","last_unstale":"2024-03-19T01:51:17.521163+0000","last_undegraded":"2024-03-19T01:51:17.521163+0000","last_fullsized":"2024-03-19T01:51:17.521163+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:45.300462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"29'7","reported_seq":36,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:52:00.681660+0000","last_change":"2024-03-19T01:51:07.044049+0000","last_active":"2024-03-19T01:52:00.681660+0000","last_peered":"2024-03-19T01:52:00.681660+0000","last_clean":"2024-03-19T01:52:00.681660+0000","last_became_active":"2024-03-19T01:51:07.043209+0000","last_became_peered":"2024-03-19T01:51:07.043209+0000","last_unstale":"2024-03-19T01:52:00.681660+0000","last_undegraded":"2024-03-19T01:52:00.681660+0000","last_fullsized":"2024-03-19T01:52:00.681660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:42.992419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088027+0000","last_change":"2024-03-19T01:51:09.179615+0000","last_active":"2024-03-19T01:51:18.088027+0000","last_peered":"2024-03-19T01:51:18.088027+0000","last_clean":"2024-03-19T01:51:18.088027+0000","last_became_active":"2024-03-19T01:51:09.173962+0000","last_became_peered":"2024-03-19T01:51:09.173962+0000","last_unstale":"2024-03-19T01:51:18.088027+0000","last_undegraded":"2024-03-19T01:51:18.088027+0000","last_fullsized":"2024-03-19T01:51:18.088027+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:06:12.676868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522548+0000","last_change":"2024-03-19T01:51:16.593210+0000","last_active":"2024-03-19T01:51:17.522548+0000","last_peered":"2024-03-19T01:51:17.522548+0000","last_clean":"2024-03-19T01:51:17.522548+0000","last_became_active":"2024-03-19T01:51:16.591235+0000","last_became_peered":"2024-03-19T01:51:16.591235+0000","last_unstale":"2024-03-19T01:51:17.522548+0000","last_undegraded":"2024-03-19T01:51:17.522548+0000","last_fullsized":"2024-03-19T01:51:17.522548+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:13.248793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"25'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088902+0000","last_change":"2024-03-19T01:51:06.983291+0000","last_active":"2024-03-19T01:51:18.088902+0000","last_peered":"2024-03-19T01:51:18.088902+0000","last_clean":"2024-03-19T01:51:18.088902+0000","last_became_active":"2024-03-19T01:51:06.982309+0000","last_became_peered":"2024-03-19T01:51:06.982309+0000","last_unstale":"2024-03-19T01:51:18.088902+0000","last_undegraded":"2024-03-19T01:51:18.088902+0000","last_fullsized":"2024-03-19T01:51:18.088902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:27.305286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089984+0000","last_change":"2024-03-19T01:51:09.160738+0000","last_active":"2024-03-19T01:51:18.089984+0000","last_peered":"2024-03-19T01:51:18.089984+0000","last_clean":"2024-03-19T01:51:18.089984+0000","last_became_active":"2024-03-19T01:51:09.159951+0000","last_became_peered":"2024-03-19T01:51:09.159951+0000","last_unstale":"2024-03-19T01:51:18.089984+0000","last_undegraded":"2024-03-19T01:51:18.089984+0000","last_fullsized":"2024-03-19T01:51:18.089984+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:30.620041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521585+0000","last_change":"2024-03-19T01:51:16.577378+0000","last_active":"2024-03-19T01:51:17.521585+0000","last_peered":"2024-03-19T01:51:17.521585+0000","last_clean":"2024-03-19T01:51:17.521585+0000","last_became_active":"2024-03-19T01:51:16.576256+0000","last_became_peered":"2024-03-19T01:51:16.576256+0000","last_unstale":"2024-03-19T01:51:17.521585+0000","last_undegraded":"2024-03-19T01:51:17.521585+0000","last_fullsized":"2024-03-19T01:51:17.521585+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:16.096547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426589+0000","last_change":"2024-03-19T01:51:09.185969+0000","last_active":"2024-03-19T01:51:17.426589+0000","last_peered":"2024-03-19T01:51:17.426589+0000","last_clean":"2024-03-19T01:51:17.426589+0000","last_became_active":"2024-03-19T01:51:09.185378+0000","last_became_peered":"2024-03-19T01:51:09.185378+0000","last_unstale":"2024-03-19T01:51:17.426589+0000","last_undegraded":"2024-03-19T01:51:17.426589+0000","last_fullsized":"2024-03-19T01:51:17.426589+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.338974+0000","last_change":"2024-03-19T01:51:06.988067+0000","last_active":"2024-03-19T01:51:50.338974+0000","last_peered":"2024-03-19T01:51:50.338974+0000","last_clean":"2024-03-19T01:51:50.338974+0000","last_became_active":"2024-03-19T01:51:06.987450+0000","last_became_peered":"2024-03-19T01:51:06.987450+0000","last_unstale":"2024-03-19T01:51:50.338974+0000","last_undegraded":"2024-03-19T01:51:50.338974+0000","last_fullsized":"2024-03-19T01:51:50.338974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:56.962922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522867+0000","last_change":"2024-03-19T01:51:16.583190+0000","last_active":"2024-03-19T01:51:17.522867+0000","last_peered":"2024-03-19T01:51:17.522867+0000","last_clean":"2024-03-19T01:51:17.522867+0000","last_became_active":"2024-03-19T01:51:16.582532+0000","last_became_peered":"2024-03-19T01:51:16.582532+0000","last_unstale":"2024-03-19T01:51:17.522867+0000","last_undegraded":"2024-03-19T01:51:17.522867+0000","last_fullsized":"2024-03-19T01:51:17.522867+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:16:49.817940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085997+0000","last_change":"2024-03-19T01:51:09.174687+0000","last_active":"2024-03-19T01:51:18.085997+0000","last_peered":"2024-03-19T01:51:18.085997+0000","last_clean":"2024-03-19T01:51:18.085997+0000","last_became_active":"2024-03-19T01:51:09.174409+0000","last_became_peered":"2024-03-19T01:51:09.174409+0000","last_unstale":"2024-03-19T01:51:18.085997+0000","last_undegraded":"2024-03-19T01:51:18.085997+0000","last_fullsized":"2024-03-19T01:51:18.085997+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:33:07.160350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:57.469468+0000","last_change":"2024-03-19T01:51:06.955434+0000","last_active":"2024-03-19T01:51:57.469468+0000","last_peered":"2024-03-19T01:51:57.469468+0000","last_clean":"2024-03-19T01:51:57.469468+0000","last_became_active":"2024-03-19T01:51:06.954635+0000","last_became_peered":"2024-03-19T01:51:06.954635+0000","last_unstale":"2024-03-19T01:51:57.469468+0000","last_undegraded":"2024-03-19T01:51:57.469468+0000","last_fullsized":"2024-03-19T01:51:57.469468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:41:39.176905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521820+0000","last_change":"2024-03-19T01:51:16.588604+0000","last_active":"2024-03-19T01:51:17.521820+0000","last_peered":"2024-03-19T01:51:17.521820+0000","last_clean":"2024-03-19T01:51:17.521820+0000","last_became_active":"2024-03-19T01:51:16.587483+0000","last_became_peered":"2024-03-19T01:51:16.587483+0000","last_unstale":"2024-03-19T01:51:17.521820+0000","last_undegraded":"2024-03-19T01:51:17.521820+0000","last_fullsized":"2024-03-19T01:51:17.521820+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:51.658547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086030+0000","last_change":"2024-03-19T01:51:09.173987+0000","last_active":"2024-03-19T01:51:18.086030+0000","last_peered":"2024-03-19T01:51:18.086030+0000","last_clean":"2024-03-19T01:51:18.086030+0000","last_became_active":"2024-03-19T01:51:09.173648+0000","last_became_peered":"2024-03-19T01:51:09.173648+0000","last_unstale":"2024-03-19T01:51:18.086030+0000","last_undegraded":"2024-03-19T01:51:18.086030+0000","last_fullsized":"2024-03-19T01:51:18.086030+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:13:17.277014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:52:00.667772+0000","last_change":"2024-03-19T01:51:06.981820+0000","last_active":"2024-03-19T01:52:00.667772+0000","last_peered":"2024-03-19T01:52:00.667772+0000","last_clean":"2024-03-19T01:52:00.667772+0000","last_became_active":"2024-03-19T01:51:06.981087+0000","last_became_peered":"2024-03-19T01:51:06.981087+0000","last_unstale":"2024-03-19T01:52:00.667772+0000","last_undegraded":"2024-03-19T01:52:00.667772+0000","last_fullsized":"2024-03-19T01:52:00.667772+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:25:59.116138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.473071+0000","last_change":"2024-03-19T01:51:16.579017+0000","last_active":"2024-03-19T01:51:17.473071+0000","last_peered":"2024-03-19T01:51:17.473071+0000","last_clean":"2024-03-19T01:51:17.473071+0000","last_became_active":"2024-03-19T01:51:16.573845+0000","last_became_peered":"2024-03-19T01:51:16.573845+0000","last_unstale":"2024-03-19T01:51:17.473071+0000","last_undegraded":"2024-03-19T01:51:17.473071+0000","last_fullsized":"2024-03-19T01:51:17.473071+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:42:30.142766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522501+0000","last_change":"2024-03-19T01:51:09.192886+0000","last_active":"2024-03-19T01:51:17.522501+0000","last_peered":"2024-03-19T01:51:17.522501+0000","last_clean":"2024-03-19T01:51:17.522501+0000","last_became_active":"2024-03-19T01:51:09.185970+0000","last_became_peered":"2024-03-19T01:51:09.185970+0000","last_unstale":"2024-03-19T01:51:17.522501+0000","last_undegraded":"2024-03-19T01:51:17.522501+0000","last_fullsized":"2024-03-19T01:51:17.522501+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:05.196985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.339562+0000","last_change":"2024-03-19T01:51:06.984875+0000","last_active":"2024-03-19T01:51:50.339562+0000","last_peered":"2024-03-19T01:51:50.339562+0000","last_clean":"2024-03-19T01:51:50.339562+0000","last_became_active":"2024-03-19T01:51:06.983858+0000","last_became_peered":"2024-03-19T01:51:06.983858+0000","last_unstale":"2024-03-19T01:51:50.339562+0000","last_undegraded":"2024-03-19T01:51:50.339562+0000","last_fullsized":"2024-03-19T01:51:50.339562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:14.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522293+0000","last_change":"2024-03-19T01:51:16.591614+0000","last_active":"2024-03-19T01:51:17.522293+0000","last_peered":"2024-03-19T01:51:17.522293+0000","last_clean":"2024-03-19T01:51:17.522293+0000","last_became_active":"2024-03-19T01:51:16.590968+0000","last_became_peered":"2024-03-19T01:51:16.590968+0000","last_unstale":"2024-03-19T01:51:17.522293+0000","last_undegraded":"2024-03-19T01:51:17.522293+0000","last_fullsized":"2024-03-19T01:51:17.522293+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:50.144136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746685+0000","last_change":"2024-03-19T01:51:09.168941+0000","last_active":"2024-03-19T01:51:17.746685+0000","last_peered":"2024-03-19T01:51:17.746685+0000","last_clean":"2024-03-19T01:51:17.746685+0000","last_became_active":"2024-03-19T01:51:09.168243+0000","last_became_peered":"2024-03-19T01:51:09.168243+0000","last_unstale":"2024-03-19T01:51:17.746685+0000","last_undegraded":"2024-03-19T01:51:17.746685+0000","last_fullsized":"2024-03-19T01:51:17.746685+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.370018+0000","last_change":"2024-03-19T01:51:07.042870+0000","last_active":"2024-03-19T01:51:50.370018+0000","last_peered":"2024-03-19T01:51:50.370018+0000","last_clean":"2024-03-19T01:51:50.370018+0000","last_became_active":"2024-03-19T01:51:07.042327+0000","last_became_peered":"2024-03-19T01:51:07.042327+0000","last_unstale":"2024-03-19T01:51:50.370018+0000","last_undegraded":"2024-03-19T01:51:50.370018+0000","last_fullsized":"2024-03-19T01:51:50.370018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:13.137673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521677+0000","last_change":"2024-03-19T01:51:16.587730+0000","last_active":"2024-03-19T01:51:17.521677+0000","last_peered":"2024-03-19T01:51:17.521677+0000","last_clean":"2024-03-19T01:51:17.521677+0000","last_became_active":"2024-03-19T01:51:16.586614+0000","last_became_peered":"2024-03-19T01:51:16.586614+0000","last_unstale":"2024-03-19T01:51:17.521677+0000","last_undegraded":"2024-03-19T01:51:17.521677+0000","last_fullsized":"2024-03-19T01:51:17.521677+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:44.830637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087909+0000","last_change":"2024-03-19T01:51:09.186397+0000","last_active":"2024-03-19T01:51:18.087909+0000","last_peered":"2024-03-19T01:51:18.087909+0000","last_clean":"2024-03-19T01:51:18.087909+0000","last_became_active":"2024-03-19T01:51:09.186033+0000","last_became_peered":"2024-03-19T01:51:09.186033+0000","last_unstale":"2024-03-19T01:51:18.087909+0000","last_undegraded":"2024-03-19T01:51:18.087909+0000","last_fullsized":"2024-03-19T01:51:18.087909+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:03:04.997711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"29'7","reported_seq":36,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:52:00.678421+0000","last_change":"2024-03-19T01:51:07.043697+0000","last_active":"2024-03-19T01:52:00.678421+0000","last_peered":"2024-03-19T01:52:00.678421+0000","last_clean":"2024-03-19T01:52:00.678421+0000","last_became_active":"2024-03-19T01:51:07.043185+0000","last_became_peered":"2024-03-19T01:51:07.043185+0000","last_unstale":"2024-03-19T01:52:00.678421+0000","last_undegraded":"2024-03-19T01:52:00.678421+0000","last_fullsized":"2024-03-19T01:52:00.678421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:19.016579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085498+0000","last_change":"2024-03-19T01:51:16.535771+0000","last_active":"2024-03-19T01:51:18.085498+0000","last_peered":"2024-03-19T01:51:18.085498+0000","last_clean":"2024-03-19T01:51:18.085498+0000","last_became_active":"2024-03-19T01:51:16.533661+0000","last_became_peered":"2024-03-19T01:51:16.533661+0000","last_unstale":"2024-03-19T01:51:18.085498+0000","last_undegraded":"2024-03-19T01:51:18.085498+0000","last_fullsized":"2024-03-19T01:51:18.085498+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:55.283782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472472+0000","last_change":"2024-03-19T01:51:09.157526+0000","last_active":"2024-03-19T01:51:17.472472+0000","last_peered":"2024-03-19T01:51:17.472472+0000","last_clean":"2024-03-19T01:51:17.472472+0000","last_became_active":"2024-03-19T01:51:09.156336+0000","last_became_peered":"2024-03-19T01:51:09.156336+0000","last_unstale":"2024-03-19T01:51:17.472472+0000","last_undegraded":"2024-03-19T01:51:17.472472+0000","last_fullsized":"2024-03-19T01:51:17.472472+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:52:00.690141+0000","last_change":"2024-03-19T01:51:06.982994+0000","last_active":"2024-03-19T01:52:00.690141+0000","last_peered":"2024-03-19T01:52:00.690141+0000","last_clean":"2024-03-19T01:52:00.690141+0000","last_became_active":"2024-03-19T01:51:06.982133+0000","last_became_peered":"2024-03-19T01:51:06.982133+0000","last_unstale":"2024-03-19T01:52:00.690141+0000","last_undegraded":"2024-03-19T01:52:00.690141+0000","last_fullsized":"2024-03-19T01:52:00.690141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:03.574098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746076+0000","last_change":"2024-03-19T01:51:16.578493+0000","last_active":"2024-03-19T01:51:17.746076+0000","last_peered":"2024-03-19T01:51:17.746076+0000","last_clean":"2024-03-19T01:51:17.746076+0000","last_became_active":"2024-03-19T01:51:16.578040+0000","last_became_peered":"2024-03-19T01:51:16.578040+0000","last_unstale":"2024-03-19T01:51:17.746076+0000","last_undegraded":"2024-03-19T01:51:17.746076+0000","last_fullsized":"2024-03-19T01:51:17.746076+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:45.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090208+0000","last_change":"2024-03-19T01:51:09.180758+0000","last_active":"2024-03-19T01:51:18.090208+0000","last_peered":"2024-03-19T01:51:18.090208+0000","last_clean":"2024-03-19T01:51:18.090208+0000","last_became_active":"2024-03-19T01:51:09.180326+0000","last_became_peered":"2024-03-19T01:51:09.180326+0000","last_unstale":"2024-03-19T01:51:18.090208+0000","last_undegraded":"2024-03-19T01:51:18.090208+0000","last_fullsized":"2024-03-19T01:51:18.090208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:56:07.046285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"29'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472624+0000","last_change":"2024-03-19T01:51:06.962643+0000","last_active":"2024-03-19T01:51:17.472624+0000","last_peered":"2024-03-19T01:51:17.472624+0000","last_clean":"2024-03-19T01:51:17.472624+0000","last_became_active":"2024-03-19T01:51:06.961862+0000","last_became_peered":"2024-03-19T01:51:06.961862+0000","last_unstale":"2024-03-19T01:51:17.472624+0000","last_undegraded":"2024-03-19T01:51:17.472624+0000","last_fullsized":"2024-03-19T01:51:17.472624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.471619+0000","last_change":"2024-03-19T01:51:16.577730+0000","last_active":"2024-03-19T01:51:17.471619+0000","last_peered":"2024-03-19T01:51:17.471619+0000","last_clean":"2024-03-19T01:51:17.471619+0000","last_became_active":"2024-03-19T01:51:16.576955+0000","last_became_peered":"2024-03-19T01:51:16.576955+0000","last_unstale":"2024-03-19T01:51:17.471619+0000","last_undegraded":"2024-03-19T01:51:17.471619+0000","last_fullsized":"2024-03-19T01:51:17.471619+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:35.409504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522368+0000","last_change":"2024-03-19T01:51:09.183678+0000","last_active":"2024-03-19T01:51:17.522368+0000","last_peered":"2024-03-19T01:51:17.522368+0000","last_clean":"2024-03-19T01:51:17.522368+0000","last_became_active":"2024-03-19T01:51:09.176780+0000","last_became_peered":"2024-03-19T01:51:09.176780+0000","last_unstale":"2024-03-19T01:51:17.522368+0000","last_undegraded":"2024-03-19T01:51:17.522368+0000","last_fullsized":"2024-03-19T01:51:17.522368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:11:02.502333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089066+0000","last_change":"2024-03-19T01:51:06.956307+0000","last_active":"2024-03-19T01:51:18.089066+0000","last_peered":"2024-03-19T01:51:18.089066+0000","last_clean":"2024-03-19T01:51:18.089066+0000","last_became_active":"2024-03-19T01:51:06.955529+0000","last_became_peered":"2024-03-19T01:51:06.955529+0000","last_unstale":"2024-03-19T01:51:18.089066+0000","last_undegraded":"2024-03-19T01:51:18.089066+0000","last_fullsized":"2024-03-19T01:51:18.089066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:10.486012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472717+0000","last_change":"2024-03-19T01:51:16.550718+0000","last_active":"2024-03-19T01:51:17.472717+0000","last_peered":"2024-03-19T01:51:17.472717+0000","last_clean":"2024-03-19T01:51:17.472717+0000","last_became_active":"2024-03-19T01:51:16.550115+0000","last_became_peered":"2024-03-19T01:51:16.550115+0000","last_unstale":"2024-03-19T01:51:17.472717+0000","last_undegraded":"2024-03-19T01:51:17.472717+0000","last_fullsized":"2024-03-19T01:51:17.472717+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:43.172796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.471788+0000","last_change":"2024-03-19T01:51:09.155718+0000","last_active":"2024-03-19T01:51:17.471788+0000","last_peered":"2024-03-19T01:51:17.471788+0000","last_clean":"2024-03-19T01:51:17.471788+0000","last_became_active":"2024-03-19T01:51:09.155014+0000","last_became_peered":"2024-03-19T01:51:09.155014+0000","last_unstale":"2024-03-19T01:51:17.471788+0000","last_undegraded":"2024-03-19T01:51:17.471788+0000","last_fullsized":"2024-03-19T01:51:17.471788+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:57.450335+0000","last_change":"2024-03-19T01:51:06.955372+0000","last_active":"2024-03-19T01:51:57.450335+0000","last_peered":"2024-03-19T01:51:57.450335+0000","last_clean":"2024-03-19T01:51:57.450335+0000","last_became_active":"2024-03-19T01:51:06.954637+0000","last_became_peered":"2024-03-19T01:51:06.954637+0000","last_unstale":"2024-03-19T01:51:57.450335+0000","last_undegraded":"2024-03-19T01:51:57.450335+0000","last_fullsized":"2024-03-19T01:51:57.450335+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:38:20.129564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.473243+0000","last_change":"2024-03-19T01:51:16.575157+0000","last_active":"2024-03-19T01:51:17.473243+0000","last_peered":"2024-03-19T01:51:17.473243+0000","last_clean":"2024-03-19T01:51:17.473243+0000","last_became_active":"2024-03-19T01:51:16.574575+0000","last_became_peered":"2024-03-19T01:51:16.574575+0000","last_unstale":"2024-03-19T01:51:17.473243+0000","last_undegraded":"2024-03-19T01:51:17.473243+0000","last_fullsized":"2024-03-19T01:51:17.473243+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:15:17.718630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.745221+0000","last_change":"2024-03-19T01:51:09.192610+0000","last_active":"2024-03-19T01:51:17.745221+0000","last_peered":"2024-03-19T01:51:17.745221+0000","last_clean":"2024-03-19T01:51:17.745221+0000","last_became_active":"2024-03-19T01:51:09.192280+0000","last_became_peered":"2024-03-19T01:51:09.192280+0000","last_unstale":"2024-03-19T01:51:17.745221+0000","last_undegraded":"2024-03-19T01:51:17.745221+0000","last_fullsized":"2024-03-19T01:51:17.745221+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:15.454555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:57.446239+0000","last_change":"2024-03-19T01:51:06.986895+0000","last_active":"2024-03-19T01:51:57.446239+0000","last_peered":"2024-03-19T01:51:57.446239+0000","last_clean":"2024-03-19T01:51:57.446239+0000","last_became_active":"2024-03-19T01:51:06.985965+0000","last_became_peered":"2024-03-19T01:51:06.985965+0000","last_unstale":"2024-03-19T01:51:57.446239+0000","last_undegraded":"2024-03-19T01:51:57.446239+0000","last_fullsized":"2024-03-19T01:51:57.446239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746886+0000","last_change":"2024-03-19T01:51:16.505278+0000","last_active":"2024-03-19T01:51:17.746886+0000","last_peered":"2024-03-19T01:51:17.746886+0000","last_clean":"2024-03-19T01:51:17.746886+0000","last_became_active":"2024-03-19T01:51:16.504908+0000","last_became_peered":"2024-03-19T01:51:16.504908+0000","last_unstale":"2024-03-19T01:51:17.746886+0000","last_undegraded":"2024-03-19T01:51:17.746886+0000","last_fullsized":"2024-03-19T01:51:17.746886+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:42:30.142766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085144+0000","last_change":"2024-03-19T01:51:09.179725+0000","last_active":"2024-03-19T01:51:18.085144+0000","last_peered":"2024-03-19T01:51:18.085144+0000","last_clean":"2024-03-19T01:51:18.085144+0000","last_became_active":"2024-03-19T01:51:09.179235+0000","last_became_peered":"2024-03-19T01:51:09.179235+0000","last_unstale":"2024-03-19T01:51:18.085144+0000","last_undegraded":"2024-03-19T01:51:18.085144+0000","last_fullsized":"2024-03-19T01:51:18.085144+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:31.395833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:52:00.726787+0000","last_change":"2024-03-19T01:51:06.953474+0000","last_active":"2024-03-19T01:52:00.726787+0000","last_peered":"2024-03-19T01:52:00.726787+0000","last_clean":"2024-03-19T01:52:00.726787+0000","last_became_active":"2024-03-19T01:51:06.952898+0000","last_became_peered":"2024-03-19T01:51:06.952898+0000","last_unstale":"2024-03-19T01:52:00.726787+0000","last_undegraded":"2024-03-19T01:52:00.726787+0000","last_fullsized":"2024-03-19T01:52:00.726787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:38.808918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":184,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088303+0000","last_change":"2024-03-19T01:50:26.097484+0000","last_active":"2024-03-19T01:51:18.088303+0000","last_peered":"2024-03-19T01:51:18.088303+0000","last_clean":"2024-03-19T01:51:18.088303+0000","last_became_active":"2024-03-19T01:50:26.095496+0000","last_became_peered":"2024-03-19T01:50:26.095496+0000","last_unstale":"2024-03-19T01:51:18.088303+0000","last_undegraded":"2024-03-19T01:51:18.088303+0000","last_fullsized":"2024-03-19T01:51:18.088303+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:50:24.112070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:50:24.112070+0000","last_clean_scrub_stamp":"2024-03-19T01:50:24.112070+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:05:00.843775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426122+0000","last_change":"2024-03-19T01:51:16.580516+0000","last_active":"2024-03-19T01:51:17.426122+0000","last_peered":"2024-03-19T01:51:17.426122+0000","last_clean":"2024-03-19T01:51:17.426122+0000","last_became_active":"2024-03-19T01:51:16.579906+0000","last_became_peered":"2024-03-19T01:51:16.579906+0000","last_unstale":"2024-03-19T01:51:17.426122+0000","last_undegraded":"2024-03-19T01:51:17.426122+0000","last_fullsized":"2024-03-19T01:51:17.426122+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:53:57.897649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089446+0000","last_change":"2024-03-19T01:51:09.196648+0000","last_active":"2024-03-19T01:51:18.089446+0000","last_peered":"2024-03-19T01:51:18.089446+0000","last_clean":"2024-03-19T01:51:18.089446+0000","last_became_active":"2024-03-19T01:51:09.196093+0000","last_became_peered":"2024-03-19T01:51:09.196093+0000","last_unstale":"2024-03-19T01:51:18.089446+0000","last_undegraded":"2024-03-19T01:51:18.089446+0000","last_fullsized":"2024-03-19T01:51:18.089446+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:54:09.655994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:52:00.748884+0000","last_change":"2024-03-19T01:51:06.950023+0000","last_active":"2024-03-19T01:52:00.748884+0000","last_peered":"2024-03-19T01:52:00.748884+0000","last_clean":"2024-03-19T01:52:00.748884+0000","last_became_active":"2024-03-19T01:51:06.949234+0000","last_became_peered":"2024-03-19T01:51:06.949234+0000","last_unstale":"2024-03-19T01:52:00.748884+0000","last_undegraded":"2024-03-19T01:52:00.748884+0000","last_fullsized":"2024-03-19T01:52:00.748884+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:54:13.131325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425731+0000","last_change":"2024-03-19T01:51:16.554046+0000","last_active":"2024-03-19T01:51:17.425731+0000","last_peered":"2024-03-19T01:51:17.425731+0000","last_clean":"2024-03-19T01:51:17.425731+0000","last_became_active":"2024-03-19T01:51:16.553595+0000","last_became_peered":"2024-03-19T01:51:16.553595+0000","last_unstale":"2024-03-19T01:51:17.425731+0000","last_undegraded":"2024-03-19T01:51:17.425731+0000","last_fullsized":"2024-03-19T01:51:17.425731+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:25.684226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088279+0000","last_change":"2024-03-19T01:51:09.181518+0000","last_active":"2024-03-19T01:51:18.088279+0000","last_peered":"2024-03-19T01:51:18.088279+0000","last_clean":"2024-03-19T01:51:18.088279+0000","last_became_active":"2024-03-19T01:51:09.174122+0000","last_became_peered":"2024-03-19T01:51:09.174122+0000","last_unstale":"2024-03-19T01:51:18.088279+0000","last_undegraded":"2024-03-19T01:51:18.088279+0000","last_fullsized":"2024-03-19T01:51:18.088279+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:18:53.079890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472978+0000","last_change":"2024-03-19T01:51:06.938396+0000","last_active":"2024-03-19T01:51:17.472978+0000","last_peered":"2024-03-19T01:51:17.472978+0000","last_clean":"2024-03-19T01:51:17.472978+0000","last_became_active":"2024-03-19T01:51:06.937580+0000","last_became_peered":"2024-03-19T01:51:06.937580+0000","last_unstale":"2024-03-19T01:51:17.472978+0000","last_undegraded":"2024-03-19T01:51:17.472978+0000","last_fullsized":"2024-03-19T01:51:17.472978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:08.307376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746471+0000","last_change":"2024-03-19T01:51:16.582111+0000","last_active":"2024-03-19T01:51:17.746471+0000","last_peered":"2024-03-19T01:51:17.746471+0000","last_clean":"2024-03-19T01:51:17.746471+0000","last_became_active":"2024-03-19T01:51:16.581746+0000","last_became_peered":"2024-03-19T01:51:16.581746+0000","last_unstale":"2024-03-19T01:51:17.746471+0000","last_undegraded":"2024-03-19T01:51:17.746471+0000","last_fullsized":"2024-03-19T01:51:17.746471+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:19.442287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090581+0000","last_change":"2024-03-19T01:51:09.195225+0000","last_active":"2024-03-19T01:51:18.090581+0000","last_peered":"2024-03-19T01:51:18.090581+0000","last_clean":"2024-03-19T01:51:18.090581+0000","last_became_active":"2024-03-19T01:51:09.194645+0000","last_became_peered":"2024-03-19T01:51:09.194645+0000","last_unstale":"2024-03-19T01:51:18.090581+0000","last_undegraded":"2024-03-19T01:51:18.090581+0000","last_fullsized":"2024-03-19T01:51:18.090581+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:50:04.024767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086792+0000","last_change":"2024-03-19T01:51:06.955976+0000","last_active":"2024-03-19T01:51:18.086792+0000","last_peered":"2024-03-19T01:51:18.086792+0000","last_clean":"2024-03-19T01:51:18.086792+0000","last_became_active":"2024-03-19T01:51:06.954951+0000","last_became_peered":"2024-03-19T01:51:06.954951+0000","last_unstale":"2024-03-19T01:51:18.086792+0000","last_undegraded":"2024-03-19T01:51:18.086792+0000","last_fullsized":"2024-03-19T01:51:18.086792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:47.536586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087418+0000","last_change":"2024-03-19T01:51:16.586904+0000","last_active":"2024-03-19T01:51:18.087418+0000","last_peered":"2024-03-19T01:51:18.087418+0000","last_clean":"2024-03-19T01:51:18.087418+0000","last_became_active":"2024-03-19T01:51:16.586298+0000","last_became_peered":"2024-03-19T01:51:16.586298+0000","last_unstale":"2024-03-19T01:51:18.087418+0000","last_undegraded":"2024-03-19T01:51:18.087418+0000","last_fullsized":"2024-03-19T01:51:18.087418+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:42.703415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089106+0000","last_change":"2024-03-19T01:51:09.195400+0000","last_active":"2024-03-19T01:51:18.089106+0000","last_peered":"2024-03-19T01:51:18.089106+0000","last_clean":"2024-03-19T01:51:18.089106+0000","last_became_active":"2024-03-19T01:51:09.195111+0000","last_became_peered":"2024-03-19T01:51:09.195111+0000","last_unstale":"2024-03-19T01:51:18.089106+0000","last_undegraded":"2024-03-19T01:51:18.089106+0000","last_fullsized":"2024-03-19T01:51:18.089106+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:12.084475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"24'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522397+0000","last_change":"2024-03-19T01:51:06.963650+0000","last_active":"2024-03-19T01:51:17.522397+0000","last_peered":"2024-03-19T01:51:17.522397+0000","last_clean":"2024-03-19T01:51:17.522397+0000","last_became_active":"2024-03-19T01:51:06.963028+0000","last_became_peered":"2024-03-19T01:51:06.963028+0000","last_unstale":"2024-03-19T01:51:17.522397+0000","last_undegraded":"2024-03-19T01:51:17.522397+0000","last_fullsized":"2024-03-19T01:51:17.522397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:59:26.369856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521307+0000","last_change":"2024-03-19T01:51:16.588077+0000","last_active":"2024-03-19T01:51:17.521307+0000","last_peered":"2024-03-19T01:51:17.521307+0000","last_clean":"2024-03-19T01:51:17.521307+0000","last_became_active":"2024-03-19T01:51:16.587459+0000","last_became_peered":"2024-03-19T01:51:16.587459+0000","last_unstale":"2024-03-19T01:51:17.521307+0000","last_undegraded":"2024-03-19T01:51:17.521307+0000","last_fullsized":"2024-03-19T01:51:17.521307+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:52.667114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089490+0000","last_change":"2024-03-19T01:51:09.195758+0000","last_active":"2024-03-19T01:51:18.089490+0000","last_peered":"2024-03-19T01:51:18.089490+0000","last_clean":"2024-03-19T01:51:18.089490+0000","last_became_active":"2024-03-19T01:51:09.195139+0000","last_became_peered":"2024-03-19T01:51:09.195139+0000","last_unstale":"2024-03-19T01:51:18.089490+0000","last_undegraded":"2024-03-19T01:51:18.089490+0000","last_fullsized":"2024-03-19T01:51:18.089490+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:18.326414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"29'16","reported_seq":45,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:57.467794+0000","last_change":"2024-03-19T01:51:06.941200+0000","last_active":"2024-03-19T01:51:57.467794+0000","last_peered":"2024-03-19T01:51:57.467794+0000","last_clean":"2024-03-19T01:51:57.467794+0000","last_became_active":"2024-03-19T01:51:06.940594+0000","last_became_peered":"2024-03-19T01:51:06.940594+0000","last_unstale":"2024-03-19T01:51:57.467794+0000","last_undegraded":"2024-03-19T01:51:57.467794+0000","last_fullsized":"2024-03-19T01:51:57.467794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:25:05.442423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746500+0000","last_change":"2024-03-19T01:51:16.548804+0000","last_active":"2024-03-19T01:51:17.746500+0000","last_peered":"2024-03-19T01:51:17.746500+0000","last_clean":"2024-03-19T01:51:17.746500+0000","last_became_active":"2024-03-19T01:51:16.547800+0000","last_became_peered":"2024-03-19T01:51:16.547800+0000","last_unstale":"2024-03-19T01:51:17.746500+0000","last_undegraded":"2024-03-19T01:51:17.746500+0000","last_fullsized":"2024-03-19T01:51:17.746500+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:43.172796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425380+0000","last_change":"2024-03-19T01:51:09.167987+0000","last_active":"2024-03-19T01:51:17.425380+0000","last_peered":"2024-03-19T01:51:17.425380+0000","last_clean":"2024-03-19T01:51:17.425380+0000","last_became_active":"2024-03-19T01:51:09.167670+0000","last_became_peered":"2024-03-19T01:51:09.167670+0000","last_unstale":"2024-03-19T01:51:17.425380+0000","last_undegraded":"2024-03-19T01:51:17.425380+0000","last_fullsized":"2024-03-19T01:51:17.425380+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:52:00.774300+0000","last_change":"2024-03-19T01:51:07.048197+0000","last_active":"2024-03-19T01:52:00.774300+0000","last_peered":"2024-03-19T01:52:00.774300+0000","last_clean":"2024-03-19T01:52:00.774300+0000","last_became_active":"2024-03-19T01:51:07.047785+0000","last_became_peered":"2024-03-19T01:51:07.047785+0000","last_unstale":"2024-03-19T01:52:00.774300+0000","last_undegraded":"2024-03-19T01:52:00.774300+0000","last_fullsized":"2024-03-19T01:52:00.774300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:52.323249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087324+0000","last_change":"2024-03-19T01:51:16.583394+0000","last_active":"2024-03-19T01:51:18.087324+0000","last_peered":"2024-03-19T01:51:18.087324+0000","last_clean":"2024-03-19T01:51:18.087324+0000","last_became_active":"2024-03-19T01:51:16.582860+0000","last_became_peered":"2024-03-19T01:51:16.582860+0000","last_unstale":"2024-03-19T01:51:18.087324+0000","last_undegraded":"2024-03-19T01:51:18.087324+0000","last_fullsized":"2024-03-19T01:51:18.087324+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:37.669591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.091680+0000","last_change":"2024-03-19T01:51:09.195008+0000","last_active":"2024-03-19T01:51:18.091680+0000","last_peered":"2024-03-19T01:51:18.091680+0000","last_clean":"2024-03-19T01:51:18.091680+0000","last_became_active":"2024-03-19T01:51:09.194728+0000","last_became_peered":"2024-03-19T01:51:09.194728+0000","last_unstale":"2024-03-19T01:51:18.091680+0000","last_undegraded":"2024-03-19T01:51:18.091680+0000","last_fullsized":"2024-03-19T01:51:18.091680+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:15:10.360817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.490405+0000","last_change":"2024-03-19T01:51:06.956905+0000","last_active":"2024-03-19T01:51:50.490405+0000","last_peered":"2024-03-19T01:51:50.490405+0000","last_clean":"2024-03-19T01:51:50.490405+0000","last_became_active":"2024-03-19T01:51:06.955991+0000","last_became_peered":"2024-03-19T01:51:06.955991+0000","last_unstale":"2024-03-19T01:51:50.490405+0000","last_undegraded":"2024-03-19T01:51:50.490405+0000","last_fullsized":"2024-03-19T01:51:50.490405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:02:45.970853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087664+0000","last_change":"2024-03-19T01:51:16.539701+0000","last_active":"2024-03-19T01:51:18.087664+0000","last_peered":"2024-03-19T01:51:18.087664+0000","last_clean":"2024-03-19T01:51:18.087664+0000","last_became_active":"2024-03-19T01:51:16.539254+0000","last_became_peered":"2024-03-19T01:51:16.539254+0000","last_unstale":"2024-03-19T01:51:18.087664+0000","last_undegraded":"2024-03-19T01:51:18.087664+0000","last_fullsized":"2024-03-19T01:51:18.087664+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:40:24.244131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.745703+0000","last_change":"2024-03-19T01:51:09.188003+0000","last_active":"2024-03-19T01:51:17.745703+0000","last_peered":"2024-03-19T01:51:17.745703+0000","last_clean":"2024-03-19T01:51:17.745703+0000","last_became_active":"2024-03-19T01:51:09.187275+0000","last_became_peered":"2024-03-19T01:51:09.187275+0000","last_unstale":"2024-03-19T01:51:17.745703+0000","last_undegraded":"2024-03-19T01:51:17.745703+0000","last_fullsized":"2024-03-19T01:51:17.745703+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:10.624259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"29'7","reported_seq":36,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:57.560487+0000","last_change":"2024-03-19T01:51:06.964413+0000","last_active":"2024-03-19T01:51:57.560487+0000","last_peered":"2024-03-19T01:51:57.560487+0000","last_clean":"2024-03-19T01:51:57.560487+0000","last_became_active":"2024-03-19T01:51:06.963763+0000","last_became_peered":"2024-03-19T01:51:06.963763+0000","last_unstale":"2024-03-19T01:51:57.560487+0000","last_undegraded":"2024-03-19T01:51:57.560487+0000","last_fullsized":"2024-03-19T01:51:57.560487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089726+0000","last_change":"2024-03-19T01:51:16.574401+0000","last_active":"2024-03-19T01:51:18.089726+0000","last_peered":"2024-03-19T01:51:18.089726+0000","last_clean":"2024-03-19T01:51:18.089726+0000","last_became_active":"2024-03-19T01:51:16.573744+0000","last_became_peered":"2024-03-19T01:51:16.573744+0000","last_unstale":"2024-03-19T01:51:18.089726+0000","last_undegraded":"2024-03-19T01:51:18.089726+0000","last_fullsized":"2024-03-19T01:51:18.089726+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:41.414631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089315+0000","last_change":"2024-03-19T01:51:09.187207+0000","last_active":"2024-03-19T01:51:18.089315+0000","last_peered":"2024-03-19T01:51:18.089315+0000","last_clean":"2024-03-19T01:51:18.089315+0000","last_became_active":"2024-03-19T01:51:09.175898+0000","last_became_peered":"2024-03-19T01:51:09.175898+0000","last_unstale":"2024-03-19T01:51:18.089315+0000","last_undegraded":"2024-03-19T01:51:18.089315+0000","last_fullsized":"2024-03-19T01:51:18.089315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:38.387069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'12","reported_seq":41,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.370011+0000","last_change":"2024-03-19T01:51:07.042663+0000","last_active":"2024-03-19T01:51:50.370011+0000","last_peered":"2024-03-19T01:51:50.370011+0000","last_clean":"2024-03-19T01:51:50.370011+0000","last_became_active":"2024-03-19T01:51:07.041262+0000","last_became_peered":"2024-03-19T01:51:07.041262+0000","last_unstale":"2024-03-19T01:51:50.370011+0000","last_undegraded":"2024-03-19T01:51:50.370011+0000","last_fullsized":"2024-03-19T01:51:50.370011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.471923+0000","last_change":"2024-03-19T01:51:16.549744+0000","last_active":"2024-03-19T01:51:17.471923+0000","last_peered":"2024-03-19T01:51:17.471923+0000","last_clean":"2024-03-19T01:51:17.471923+0000","last_became_active":"2024-03-19T01:51:16.549347+0000","last_became_peered":"2024-03-19T01:51:16.549347+0000","last_unstale":"2024-03-19T01:51:17.471923+0000","last_undegraded":"2024-03-19T01:51:17.471923+0000","last_fullsized":"2024-03-19T01:51:17.471923+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:19.442287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522393+0000","last_change":"2024-03-19T01:51:09.194165+0000","last_active":"2024-03-19T01:51:17.522393+0000","last_peered":"2024-03-19T01:51:17.522393+0000","last_clean":"2024-03-19T01:51:17.522393+0000","last_became_active":"2024-03-19T01:51:09.183607+0000","last_became_peered":"2024-03-19T01:51:09.183607+0000","last_unstale":"2024-03-19T01:51:17.522393+0000","last_undegraded":"2024-03-19T01:51:17.522393+0000","last_fullsized":"2024-03-19T01:51:17.522393+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:09.211794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.374380+0000","last_change":"2024-03-19T01:51:06.952515+0000","last_active":"2024-03-19T01:51:50.374380+0000","last_peered":"2024-03-19T01:51:50.374380+0000","last_clean":"2024-03-19T01:51:50.374380+0000","last_became_active":"2024-03-19T01:51:06.951700+0000","last_became_peered":"2024-03-19T01:51:06.951700+0000","last_unstale":"2024-03-19T01:51:50.374380+0000","last_undegraded":"2024-03-19T01:51:50.374380+0000","last_fullsized":"2024-03-19T01:51:50.374380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:13.137673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522152+0000","last_change":"2024-03-19T01:51:16.587027+0000","last_active":"2024-03-19T01:51:17.522152+0000","last_peered":"2024-03-19T01:51:17.522152+0000","last_clean":"2024-03-19T01:51:17.522152+0000","last_became_active":"2024-03-19T01:51:16.586514+0000","last_became_peered":"2024-03-19T01:51:16.586514+0000","last_unstale":"2024-03-19T01:51:17.522152+0000","last_undegraded":"2024-03-19T01:51:17.522152+0000","last_fullsized":"2024-03-19T01:51:17.522152+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:06.637617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746586+0000","last_change":"2024-03-19T01:51:09.172554+0000","last_active":"2024-03-19T01:51:17.746586+0000","last_peered":"2024-03-19T01:51:17.746586+0000","last_clean":"2024-03-19T01:51:17.746586+0000","last_became_active":"2024-03-19T01:51:09.171814+0000","last_became_peered":"2024-03-19T01:51:09.171814+0000","last_unstale":"2024-03-19T01:51:17.746586+0000","last_undegraded":"2024-03-19T01:51:17.746586+0000","last_fullsized":"2024-03-19T01:51:17.746586+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.488757+0000","last_change":"2024-03-19T01:51:07.043404+0000","last_active":"2024-03-19T01:51:50.488757+0000","last_peered":"2024-03-19T01:51:50.488757+0000","last_clean":"2024-03-19T01:51:50.488757+0000","last_became_active":"2024-03-19T01:51:07.043069+0000","last_became_peered":"2024-03-19T01:51:07.043069+0000","last_unstale":"2024-03-19T01:51:50.488757+0000","last_undegraded":"2024-03-19T01:51:50.488757+0000","last_fullsized":"2024-03-19T01:51:50.488757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:13:14.960132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085834+0000","last_change":"2024-03-19T01:51:16.581797+0000","last_active":"2024-03-19T01:51:18.085834+0000","last_peered":"2024-03-19T01:51:18.085834+0000","last_clean":"2024-03-19T01:51:18.085834+0000","last_became_active":"2024-03-19T01:51:16.581174+0000","last_became_peered":"2024-03-19T01:51:16.581174+0000","last_unstale":"2024-03-19T01:51:18.085834+0000","last_undegraded":"2024-03-19T01:51:18.085834+0000","last_fullsized":"2024-03-19T01:51:18.085834+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:29:34.911347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090870+0000","last_change":"2024-03-19T01:51:09.194372+0000","last_active":"2024-03-19T01:51:18.090870+0000","last_peered":"2024-03-19T01:51:18.090870+0000","last_clean":"2024-03-19T01:51:18.090870+0000","last_became_active":"2024-03-19T01:51:09.193656+0000","last_became_peered":"2024-03-19T01:51:09.193656+0000","last_unstale":"2024-03-19T01:51:18.090870+0000","last_undegraded":"2024-03-19T01:51:18.090870+0000","last_fullsized":"2024-03-19T01:51:18.090870+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:23.464803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.375628+0000","last_change":"2024-03-19T01:51:07.046089+0000","last_active":"2024-03-19T01:51:50.375628+0000","last_peered":"2024-03-19T01:51:50.375628+0000","last_clean":"2024-03-19T01:51:50.375628+0000","last_became_active":"2024-03-19T01:51:07.045342+0000","last_became_peered":"2024-03-19T01:51:07.045342+0000","last_unstale":"2024-03-19T01:51:50.375628+0000","last_undegraded":"2024-03-19T01:51:50.375628+0000","last_fullsized":"2024-03-19T01:51:50.375628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:26.105915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087835+0000","last_change":"2024-03-19T01:51:16.572559+0000","last_active":"2024-03-19T01:51:18.087835+0000","last_peered":"2024-03-19T01:51:18.087835+0000","last_clean":"2024-03-19T01:51:18.087835+0000","last_became_active":"2024-03-19T01:51:16.572283+0000","last_became_peered":"2024-03-19T01:51:16.572283+0000","last_unstale":"2024-03-19T01:51:18.087835+0000","last_undegraded":"2024-03-19T01:51:18.087835+0000","last_fullsized":"2024-03-19T01:51:18.087835+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:42.703415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746448+0000","last_change":"2024-03-19T01:51:16.534088+0000","last_active":"2024-03-19T01:51:17.746448+0000","last_peered":"2024-03-19T01:51:17.746448+0000","last_clean":"2024-03-19T01:51:17.746448+0000","last_became_active":"2024-03-19T01:51:16.533665+0000","last_became_peered":"2024-03-19T01:51:16.533665+0000","last_unstale":"2024-03-19T01:51:17.746448+0000","last_undegraded":"2024-03-19T01:51:17.746448+0000","last_fullsized":"2024-03-19T01:51:17.746448+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:35.409504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.471346+0000","last_change":"2024-03-19T01:51:09.168372+0000","last_active":"2024-03-19T01:51:17.471346+0000","last_peered":"2024-03-19T01:51:17.471346+0000","last_clean":"2024-03-19T01:51:17.471346+0000","last_became_active":"2024-03-19T01:51:09.167626+0000","last_became_peered":"2024-03-19T01:51:09.167626+0000","last_unstale":"2024-03-19T01:51:17.471346+0000","last_undegraded":"2024-03-19T01:51:17.471346+0000","last_fullsized":"2024-03-19T01:51:17.471346+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.489850+0000","last_change":"2024-03-19T01:51:06.951225+0000","last_active":"2024-03-19T01:51:50.489850+0000","last_peered":"2024-03-19T01:51:50.489850+0000","last_clean":"2024-03-19T01:51:50.489850+0000","last_became_active":"2024-03-19T01:51:06.950826+0000","last_became_peered":"2024-03-19T01:51:06.950826+0000","last_unstale":"2024-03-19T01:51:50.489850+0000","last_undegraded":"2024-03-19T01:51:50.489850+0000","last_fullsized":"2024-03-19T01:51:50.489850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:24.949925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426878+0000","last_change":"2024-03-19T01:51:16.581997+0000","last_active":"2024-03-19T01:51:17.426878+0000","last_peered":"2024-03-19T01:51:17.426878+0000","last_clean":"2024-03-19T01:51:17.426878+0000","last_became_active":"2024-03-19T01:51:16.581412+0000","last_became_peered":"2024-03-19T01:51:16.581412+0000","last_unstale":"2024-03-19T01:51:17.426878+0000","last_undegraded":"2024-03-19T01:51:17.426878+0000","last_fullsized":"2024-03-19T01:51:17.426878+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:45.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085931+0000","last_change":"2024-03-19T01:51:09.161686+0000","last_active":"2024-03-19T01:51:18.085931+0000","last_peered":"2024-03-19T01:51:18.085931+0000","last_clean":"2024-03-19T01:51:18.085931+0000","last_became_active":"2024-03-19T01:51:09.160978+0000","last_became_peered":"2024-03-19T01:51:09.160978+0000","last_unstale":"2024-03-19T01:51:18.085931+0000","last_undegraded":"2024-03-19T01:51:18.085931+0000","last_fullsized":"2024-03-19T01:51:18.085931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:27.266807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:57.449677+0000","last_change":"2024-03-19T01:51:07.047363+0000","last_active":"2024-03-19T01:51:57.449677+0000","last_peered":"2024-03-19T01:51:57.449677+0000","last_clean":"2024-03-19T01:51:57.449677+0000","last_became_active":"2024-03-19T01:51:07.046886+0000","last_became_peered":"2024-03-19T01:51:07.046886+0000","last_unstale":"2024-03-19T01:51:57.449677+0000","last_undegraded":"2024-03-19T01:51:57.449677+0000","last_fullsized":"2024-03-19T01:51:57.449677+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:19.418603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"29'7","reported_seq":36,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:52:00.775087+0000","last_change":"2024-03-19T01:51:07.045496+0000","last_active":"2024-03-19T01:52:00.775087+0000","last_peered":"2024-03-19T01:52:00.775087+0000","last_clean":"2024-03-19T01:52:00.775087+0000","last_became_active":"2024-03-19T01:51:07.044554+0000","last_became_peered":"2024-03-19T01:51:07.044554+0000","last_unstale":"2024-03-19T01:52:00.775087+0000","last_undegraded":"2024-03-19T01:52:00.775087+0000","last_fullsized":"2024-03-19T01:52:00.775087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:14.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746492+0000","last_change":"2024-03-19T01:51:09.184934+0000","last_active":"2024-03-19T01:51:17.746492+0000","last_peered":"2024-03-19T01:51:17.746492+0000","last_clean":"2024-03-19T01:51:17.746492+0000","last_became_active":"2024-03-19T01:51:09.184666+0000","last_became_peered":"2024-03-19T01:51:09.184666+0000","last_unstale":"2024-03-19T01:51:17.746492+0000","last_undegraded":"2024-03-19T01:51:17.746492+0000","last_fullsized":"2024-03-19T01:51:17.746492+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:42.786323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085344+0000","last_change":"2024-03-19T01:51:16.575084+0000","last_active":"2024-03-19T01:51:18.085344+0000","last_peered":"2024-03-19T01:51:18.085344+0000","last_clean":"2024-03-19T01:51:18.085344+0000","last_became_active":"2024-03-19T01:51:16.574631+0000","last_became_peered":"2024-03-19T01:51:16.574631+0000","last_unstale":"2024-03-19T01:51:18.085344+0000","last_undegraded":"2024-03-19T01:51:18.085344+0000","last_fullsized":"2024-03-19T01:51:18.085344+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:24:35.634113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:50.503274+0000","last_change":"2024-03-19T01:51:06.982280+0000","last_active":"2024-03-19T01:51:50.503274+0000","last_peered":"2024-03-19T01:51:50.503274+0000","last_clean":"2024-03-19T01:51:50.503274+0000","last_became_active":"2024-03-19T01:51:06.981704+0000","last_became_peered":"2024-03-19T01:51:06.981704+0000","last_unstale":"2024-03-19T01:51:50.503274+0000","last_undegraded":"2024-03-19T01:51:50.503274+0000","last_fullsized":"2024-03-19T01:51:50.503274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:22:12.657985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.521885+0000","last_change":"2024-03-19T01:51:09.184624+0000","last_active":"2024-03-19T01:51:17.521885+0000","last_peered":"2024-03-19T01:51:17.521885+0000","last_clean":"2024-03-19T01:51:17.521885+0000","last_became_active":"2024-03-19T01:51:09.183976+0000","last_became_peered":"2024-03-19T01:51:09.183976+0000","last_unstale":"2024-03-19T01:51:17.521885+0000","last_undegraded":"2024-03-19T01:51:17.521885+0000","last_fullsized":"2024-03-19T01:51:17.521885+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:24:29.313729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086581+0000","last_change":"2024-03-19T01:51:16.576775+0000","last_active":"2024-03-19T01:51:18.086581+0000","last_peered":"2024-03-19T01:51:18.086581+0000","last_clean":"2024-03-19T01:51:18.086581+0000","last_became_active":"2024-03-19T01:51:16.576406+0000","last_became_peered":"2024-03-19T01:51:16.576406+0000","last_unstale":"2024-03-19T01:51:18.086581+0000","last_undegraded":"2024-03-19T01:51:18.086581+0000","last_fullsized":"2024-03-19T01:51:18.086581+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:43.725447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425113+0000","last_change":"2024-03-19T01:51:09.169493+0000","last_active":"2024-03-19T01:51:17.425113+0000","last_peered":"2024-03-19T01:51:17.425113+0000","last_clean":"2024-03-19T01:51:17.425113+0000","last_became_active":"2024-03-19T01:51:09.168231+0000","last_became_peered":"2024-03-19T01:51:09.168231+0000","last_unstale":"2024-03-19T01:51:17.425113+0000","last_undegraded":"2024-03-19T01:51:17.425113+0000","last_fullsized":"2024-03-19T01:51:17.425113+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:10.624259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085344+0000","last_change":"2024-03-19T01:51:09.155483+0000","last_active":"2024-03-19T01:51:18.085344+0000","last_peered":"2024-03-19T01:51:18.085344+0000","last_clean":"2024-03-19T01:51:18.085344+0000","last_became_active":"2024-03-19T01:51:09.154545+0000","last_became_peered":"2024-03-19T01:51:09.154545+0000","last_unstale":"2024-03-19T01:51:18.085344+0000","last_undegraded":"2024-03-19T01:51:18.085344+0000","last_fullsized":"2024-03-19T01:51:18.085344+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:25:09.517323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088404+0000","last_change":"2024-03-19T01:51:09.180498+0000","last_active":"2024-03-19T01:51:18.088404+0000","last_peered":"2024-03-19T01:51:18.088404+0000","last_clean":"2024-03-19T01:51:18.088404+0000","last_became_active":"2024-03-19T01:51:09.179840+0000","last_became_peered":"2024-03-19T01:51:09.179840+0000","last_unstale":"2024-03-19T01:51:18.088404+0000","last_undegraded":"2024-03-19T01:51:18.088404+0000","last_fullsized":"2024-03-19T01:51:18.088404+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:01:21.716571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746226+0000","last_change":"2024-03-19T01:51:09.170389+0000","last_active":"2024-03-19T01:51:17.746226+0000","last_peered":"2024-03-19T01:51:17.746226+0000","last_clean":"2024-03-19T01:51:17.746226+0000","last_became_active":"2024-03-19T01:51:09.163974+0000","last_became_peered":"2024-03-19T01:51:09.163974+0000","last_unstale":"2024-03-19T01:51:17.746226+0000","last_undegraded":"2024-03-19T01:51:17.746226+0000","last_fullsized":"2024-03-19T01:51:17.746226+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087052+0000","last_change":"2024-03-19T01:51:09.177060+0000","last_active":"2024-03-19T01:51:18.087052+0000","last_peered":"2024-03-19T01:51:18.087052+0000","last_clean":"2024-03-19T01:51:18.087052+0000","last_became_active":"2024-03-19T01:51:09.176474+0000","last_became_peered":"2024-03-19T01:51:09.176474+0000","last_unstale":"2024-03-19T01:51:18.087052+0000","last_undegraded":"2024-03-19T01:51:18.087052+0000","last_fullsized":"2024-03-19T01:51:18.087052+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:56.318137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.472633+0000","last_change":"2024-03-19T01:51:09.168741+0000","last_active":"2024-03-19T01:51:17.472633+0000","last_peered":"2024-03-19T01:51:17.472633+0000","last_clean":"2024-03-19T01:51:17.472633+0000","last_became_active":"2024-03-19T01:51:09.167982+0000","last_became_peered":"2024-03-19T01:51:09.167982+0000","last_unstale":"2024-03-19T01:51:17.472633+0000","last_undegraded":"2024-03-19T01:51:17.472633+0000","last_fullsized":"2024-03-19T01:51:17.472633+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425915+0000","last_change":"2024-03-19T01:51:09.189814+0000","last_active":"2024-03-19T01:51:17.425915+0000","last_peered":"2024-03-19T01:51:17.425915+0000","last_clean":"2024-03-19T01:51:17.425915+0000","last_became_active":"2024-03-19T01:51:09.189128+0000","last_became_peered":"2024-03-19T01:51:09.189128+0000","last_unstale":"2024-03-19T01:51:17.425915+0000","last_undegraded":"2024-03-19T01:51:17.425915+0000","last_fullsized":"2024-03-19T01:51:17.425915+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087688+0000","last_change":"2024-03-19T01:51:09.183989+0000","last_active":"2024-03-19T01:51:18.087688+0000","last_peered":"2024-03-19T01:51:18.087688+0000","last_clean":"2024-03-19T01:51:18.087688+0000","last_became_active":"2024-03-19T01:51:09.183417+0000","last_became_peered":"2024-03-19T01:51:09.183417+0000","last_unstale":"2024-03-19T01:51:18.087688+0000","last_undegraded":"2024-03-19T01:51:18.087688+0000","last_fullsized":"2024-03-19T01:51:18.087688+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:31.405618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087352+0000","last_change":"2024-03-19T01:51:09.172717+0000","last_active":"2024-03-19T01:51:18.087352+0000","last_peered":"2024-03-19T01:51:18.087352+0000","last_clean":"2024-03-19T01:51:18.087352+0000","last_became_active":"2024-03-19T01:51:09.165872+0000","last_became_peered":"2024-03-19T01:51:09.165872+0000","last_unstale":"2024-03-19T01:51:18.087352+0000","last_undegraded":"2024-03-19T01:51:18.087352+0000","last_fullsized":"2024-03-19T01:51:18.087352+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:10.995492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088202+0000","last_change":"2024-03-19T01:51:09.195116+0000","last_active":"2024-03-19T01:51:18.088202+0000","last_peered":"2024-03-19T01:51:18.088202+0000","last_clean":"2024-03-19T01:51:18.088202+0000","last_became_active":"2024-03-19T01:51:09.194558+0000","last_became_peered":"2024-03-19T01:51:09.194558+0000","last_unstale":"2024-03-19T01:51:18.088202+0000","last_undegraded":"2024-03-19T01:51:18.088202+0000","last_fullsized":"2024-03-19T01:51:18.088202+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:41:53.624234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.088160+0000","last_change":"2024-03-19T01:51:09.195141+0000","last_active":"2024-03-19T01:51:18.088160+0000","last_peered":"2024-03-19T01:51:18.088160+0000","last_clean":"2024-03-19T01:51:18.088160+0000","last_became_active":"2024-03-19T01:51:09.194536+0000","last_became_peered":"2024-03-19T01:51:09.194536+0000","last_unstale":"2024-03-19T01:51:18.088160+0000","last_undegraded":"2024-03-19T01:51:18.088160+0000","last_fullsized":"2024-03-19T01:51:18.088160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:31.395833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.745890+0000","last_change":"2024-03-19T01:51:09.189197+0000","last_active":"2024-03-19T01:51:17.745890+0000","last_peered":"2024-03-19T01:51:17.745890+0000","last_clean":"2024-03-19T01:51:17.745890+0000","last_became_active":"2024-03-19T01:51:09.188565+0000","last_became_peered":"2024-03-19T01:51:09.188565+0000","last_unstale":"2024-03-19T01:51:17.745890+0000","last_undegraded":"2024-03-19T01:51:17.745890+0000","last_fullsized":"2024-03-19T01:51:17.745890+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426243+0000","last_change":"2024-03-19T01:51:09.168581+0000","last_active":"2024-03-19T01:51:17.426243+0000","last_peered":"2024-03-19T01:51:17.426243+0000","last_clean":"2024-03-19T01:51:17.426243+0000","last_became_active":"2024-03-19T01:51:09.167812+0000","last_became_peered":"2024-03-19T01:51:09.167812+0000","last_unstale":"2024-03-19T01:51:17.426243+0000","last_undegraded":"2024-03-19T01:51:17.426243+0000","last_fullsized":"2024-03-19T01:51:17.426243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089822+0000","last_change":"2024-03-19T01:51:09.188390+0000","last_active":"2024-03-19T01:51:18.089822+0000","last_peered":"2024-03-19T01:51:18.089822+0000","last_clean":"2024-03-19T01:51:18.089822+0000","last_became_active":"2024-03-19T01:51:09.187104+0000","last_became_peered":"2024-03-19T01:51:09.187104+0000","last_unstale":"2024-03-19T01:51:18.089822+0000","last_undegraded":"2024-03-19T01:51:18.089822+0000","last_fullsized":"2024-03-19T01:51:18.089822+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:37.257686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086435+0000","last_change":"2024-03-19T01:51:09.168451+0000","last_active":"2024-03-19T01:51:18.086435+0000","last_peered":"2024-03-19T01:51:18.086435+0000","last_clean":"2024-03-19T01:51:18.086435+0000","last_became_active":"2024-03-19T01:51:09.167744+0000","last_became_peered":"2024-03-19T01:51:09.167744+0000","last_unstale":"2024-03-19T01:51:18.086435+0000","last_undegraded":"2024-03-19T01:51:18.086435+0000","last_fullsized":"2024-03-19T01:51:18.086435+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:21.735485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089493+0000","last_change":"2024-03-19T01:51:09.196067+0000","last_active":"2024-03-19T01:51:18.089493+0000","last_peered":"2024-03-19T01:51:18.089493+0000","last_clean":"2024-03-19T01:51:18.089493+0000","last_became_active":"2024-03-19T01:51:09.195488+0000","last_became_peered":"2024-03-19T01:51:09.195488+0000","last_unstale":"2024-03-19T01:51:18.089493+0000","last_undegraded":"2024-03-19T01:51:18.089493+0000","last_fullsized":"2024-03-19T01:51:18.089493+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:07:44.120161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087081+0000","last_change":"2024-03-19T01:51:09.193831+0000","last_active":"2024-03-19T01:51:18.087081+0000","last_peered":"2024-03-19T01:51:18.087081+0000","last_clean":"2024-03-19T01:51:18.087081+0000","last_became_active":"2024-03-19T01:51:09.192777+0000","last_became_peered":"2024-03-19T01:51:09.192777+0000","last_unstale":"2024-03-19T01:51:18.087081+0000","last_undegraded":"2024-03-19T01:51:18.087081+0000","last_fullsized":"2024-03-19T01:51:18.087081+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:56.318137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.471571+0000","last_change":"2024-03-19T01:51:09.193581+0000","last_active":"2024-03-19T01:51:17.471571+0000","last_peered":"2024-03-19T01:51:17.471571+0000","last_clean":"2024-03-19T01:51:17.471571+0000","last_became_active":"2024-03-19T01:51:09.192677+0000","last_became_peered":"2024-03-19T01:51:09.192677+0000","last_unstale":"2024-03-19T01:51:17.471571+0000","last_undegraded":"2024-03-19T01:51:17.471571+0000","last_fullsized":"2024-03-19T01:51:17.471571+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.745598+0000","last_change":"2024-03-19T01:51:09.165568+0000","last_active":"2024-03-19T01:51:17.745598+0000","last_peered":"2024-03-19T01:51:17.745598+0000","last_clean":"2024-03-19T01:51:17.745598+0000","last_became_active":"2024-03-19T01:51:09.164753+0000","last_became_peered":"2024-03-19T01:51:09.164753+0000","last_unstale":"2024-03-19T01:51:17.745598+0000","last_undegraded":"2024-03-19T01:51:17.745598+0000","last_fullsized":"2024-03-19T01:51:17.745598+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:28.422798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.086531+0000","last_change":"2024-03-19T01:51:09.170898+0000","last_active":"2024-03-19T01:51:18.086531+0000","last_peered":"2024-03-19T01:51:18.086531+0000","last_clean":"2024-03-19T01:51:18.086531+0000","last_became_active":"2024-03-19T01:51:09.170153+0000","last_became_peered":"2024-03-19T01:51:09.170153+0000","last_unstale":"2024-03-19T01:51:18.086531+0000","last_undegraded":"2024-03-19T01:51:18.086531+0000","last_fullsized":"2024-03-19T01:51:18.086531+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:41.125800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.522152+0000","last_change":"2024-03-19T01:51:09.185953+0000","last_active":"2024-03-19T01:51:17.522152+0000","last_peered":"2024-03-19T01:51:17.522152+0000","last_clean":"2024-03-19T01:51:17.522152+0000","last_became_active":"2024-03-19T01:51:09.177996+0000","last_became_peered":"2024-03-19T01:51:09.177996+0000","last_unstale":"2024-03-19T01:51:17.522152+0000","last_undegraded":"2024-03-19T01:51:17.522152+0000","last_fullsized":"2024-03-19T01:51:17.522152+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:39:06.045413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.426250+0000","last_change":"2024-03-19T01:51:09.183841+0000","last_active":"2024-03-19T01:51:17.426250+0000","last_peered":"2024-03-19T01:51:17.426250+0000","last_clean":"2024-03-19T01:51:17.426250+0000","last_became_active":"2024-03-19T01:51:09.183045+0000","last_became_peered":"2024-03-19T01:51:09.183045+0000","last_unstale":"2024-03-19T01:51:17.426250+0000","last_undegraded":"2024-03-19T01:51:17.426250+0000","last_fullsized":"2024-03-19T01:51:17.426250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.425940+0000","last_change":"2024-03-19T01:51:09.184022+0000","last_active":"2024-03-19T01:51:17.425940+0000","last_peered":"2024-03-19T01:51:17.425940+0000","last_clean":"2024-03-19T01:51:17.425940+0000","last_became_active":"2024-03-19T01:51:09.183351+0000","last_became_peered":"2024-03-19T01:51:09.183351+0000","last_unstale":"2024-03-19T01:51:17.425940+0000","last_undegraded":"2024-03-19T01:51:17.425940+0000","last_fullsized":"2024-03-19T01:51:17.425940+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090466+0000","last_change":"2024-03-19T01:51:09.204005+0000","last_active":"2024-03-19T01:51:18.090466+0000","last_peered":"2024-03-19T01:51:18.090466+0000","last_clean":"2024-03-19T01:51:18.090466+0000","last_became_active":"2024-03-19T01:51:09.203545+0000","last_became_peered":"2024-03-19T01:51:09.203545+0000","last_unstale":"2024-03-19T01:51:18.090466+0000","last_undegraded":"2024-03-19T01:51:18.090466+0000","last_fullsized":"2024-03-19T01:51:18.090466+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:42:22.784953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.085666+0000","last_change":"2024-03-19T01:51:09.180013+0000","last_active":"2024-03-19T01:51:18.085666+0000","last_peered":"2024-03-19T01:51:18.085666+0000","last_clean":"2024-03-19T01:51:18.085666+0000","last_became_active":"2024-03-19T01:51:09.179541+0000","last_became_peered":"2024-03-19T01:51:09.179541+0000","last_unstale":"2024-03-19T01:51:18.085666+0000","last_undegraded":"2024-03-19T01:51:18.085666+0000","last_fullsized":"2024-03-19T01:51:18.085666+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:19:25.933028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.089242+0000","last_change":"2024-03-19T01:51:09.188671+0000","last_active":"2024-03-19T01:51:18.089242+0000","last_peered":"2024-03-19T01:51:18.089242+0000","last_clean":"2024-03-19T01:51:18.089242+0000","last_became_active":"2024-03-19T01:51:09.177627+0000","last_became_peered":"2024-03-19T01:51:09.177627+0000","last_unstale":"2024-03-19T01:51:18.089242+0000","last_undegraded":"2024-03-19T01:51:18.089242+0000","last_fullsized":"2024-03-19T01:51:18.089242+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:53:50.539837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087926+0000","last_change":"2024-03-19T01:51:09.180735+0000","last_active":"2024-03-19T01:51:18.087926+0000","last_peered":"2024-03-19T01:51:18.087926+0000","last_clean":"2024-03-19T01:51:18.087926+0000","last_became_active":"2024-03-19T01:51:09.173235+0000","last_became_peered":"2024-03-19T01:51:09.173235+0000","last_unstale":"2024-03-19T01:51:18.087926+0000","last_undegraded":"2024-03-19T01:51:18.087926+0000","last_fullsized":"2024-03-19T01:51:18.087926+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:31.405618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.091424+0000","last_change":"2024-03-19T01:51:09.188346+0000","last_active":"2024-03-19T01:51:18.091424+0000","last_peered":"2024-03-19T01:51:18.091424+0000","last_clean":"2024-03-19T01:51:18.091424+0000","last_became_active":"2024-03-19T01:51:09.187703+0000","last_became_peered":"2024-03-19T01:51:09.187703+0000","last_unstale":"2024-03-19T01:51:18.091424+0000","last_undegraded":"2024-03-19T01:51:18.091424+0000","last_fullsized":"2024-03-19T01:51:18.091424+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:28.051691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.087787+0000","last_change":"2024-03-19T01:51:09.182114+0000","last_active":"2024-03-19T01:51:18.087787+0000","last_peered":"2024-03-19T01:51:18.087787+0000","last_clean":"2024-03-19T01:51:18.087787+0000","last_became_active":"2024-03-19T01:51:09.181056+0000","last_became_peered":"2024-03-19T01:51:09.181056+0000","last_unstale":"2024-03-19T01:51:18.087787+0000","last_undegraded":"2024-03-19T01:51:18.087787+0000","last_fullsized":"2024-03-19T01:51:18.087787+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:03:04.997711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.746015+0000","last_change":"2024-03-19T01:51:09.161476+0000","last_active":"2024-03-19T01:51:17.746015+0000","last_peered":"2024-03-19T01:51:17.746015+0000","last_clean":"2024-03-19T01:51:17.746015+0000","last_became_active":"2024-03-19T01:51:09.158609+0000","last_became_peered":"2024-03-19T01:51:09.158609+0000","last_unstale":"2024-03-19T01:51:17.746015+0000","last_undegraded":"2024-03-19T01:51:17.746015+0000","last_fullsized":"2024-03-19T01:51:17.746015+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:17.473414+0000","last_change":"2024-03-19T01:51:09.165938+0000","last_active":"2024-03-19T01:51:17.473414+0000","last_peered":"2024-03-19T01:51:17.473414+0000","last_clean":"2024-03-19T01:51:17.473414+0000","last_became_active":"2024-03-19T01:51:09.165207+0000","last_became_peered":"2024-03-19T01:51:09.165207+0000","last_unstale":"2024-03-19T01:51:17.473414+0000","last_undegraded":"2024-03-19T01:51:17.473414+0000","last_fullsized":"2024-03-19T01:51:17.473414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-19T01:51:18.090233+0000","last_change":"2024-03-19T01:51:09.176023+0000","last_active":"2024-03-19T01:51:18.090233+0000","last_peered":"2024-03-19T01:51:18.090233+0000","last_clean":"2024-03-19T01:51:18.090233+0000","last_became_active":"2024-03-19T01:51:09.175682+0000","last_became_peered":"2024-03-19T01:51:09.175682+0000","last_unstale":"2024-03-19T01:51:18.090233+0000","last_undegraded":"2024-03-19T01:51:18.090233+0000","last_fullsized":"2024-03-19T01:51:18.090233+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:35.814983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8858,"num_objects":77,"num_object_clones":0,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":167,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":17716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509464,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343920,"statfs":{"total":96636764160,"available":96608174080,"internally_reserved":0,"allocated":925696,"data_stored":715728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509466,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":128654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509465,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":331776,"data_stored":125352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509466,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31500,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340340,"statfs":{"total":96636764160,"available":96604508160,"internally_reserved":0,"allocated":331776,"data_stored":124966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509465,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32076,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339764,"statfs":{"total":96636764160,"available":96603918336,"internally_reserved":0,"allocated":921600,"data_stored":715084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542168,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":344064,"data_stored":126750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542169,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31500,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340340,"statfs":{"total":96636764160,"available":96604508160,"internally_reserved":0,"allocated":331776,"data_stored":128576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542168,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31496,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340344,"statfs":{"total":96636764160,"available":96604512256,"internally_reserved":0,"allocated":327680,"data_stored":124840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T01:52:04.563 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-19T01:52:04.563 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-19T01:52:04.564 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-19T01:52:04.564 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-19T01:52:04.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.725+0000 7f4e524f9640 1 Processor -- start 2024-03-19T01:52:04.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.725+0000 7f4e524f9640 1 -- start start 2024-03-19T01:52:04.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.725+0000 7f4e524f9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4e4c13dd00 0x7f4e4c0c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:04.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.725+0000 7f4e524f9640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f4e4c0c9780 0x7f4e4c0c8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:04.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.725+0000 7f4e524f9640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4e4c15b550 0x7f4e4c0c5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:04.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.725+0000 7f4e524f9640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f4e4c0c8f70 con 0x7f4e4c15b550 2024-03-19T01:52:04.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.725+0000 7f4e524f9640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f4e4c0c9250 con 0x7f4e4c0c9780 2024-03-19T01:52:04.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.725+0000 7f4e524f9640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4e4c1582b0 con 0x7f4e4c13dd00 2024-03-19T01:52:04.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.725+0000 7f4e4bfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4e4c13dd00 0x7f4e4c0c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:04.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.725+0000 7f4e4b7fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f4e4c0c9780 0x7f4e4c0c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:04.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.729+0000 7f4e4bfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4e4c13dd00 0x7f4e4c0c7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:42930/0 (socket says 172.21.15.55:42930) 2024-03-19T01:52:04.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.729+0000 7f4e4bfff640 1 -- 172.21.15.55:0/2848539114 learned_addr learned my addr 172.21.15.55:0/2848539114 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:04.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.729+0000 7f4e4bfff640 1 -- 172.21.15.55:0/2848539114 >> v2:172.21.15.84:3300/0 conn(0x7f4e4c0c9780 msgr2=0x7f4e4c0c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:04.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.729+0000 7f4e4bfff640 1 --2- 172.21.15.55:0/2848539114 >> v2:172.21.15.84:3300/0 conn(0x7f4e4c0c9780 0x7f4e4c0c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:04.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.729+0000 7f4e4bfff640 1 -- 172.21.15.55:0/2848539114 >> v2:172.21.15.55:3300/0 conn(0x7f4e4c15b550 msgr2=0x7f4e4c0c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:04.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.729+0000 7f4e4bfff640 1 --2- 172.21.15.55:0/2848539114 >> v2:172.21.15.55:3300/0 conn(0x7f4e4c15b550 0x7f4e4c0c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:04.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.729+0000 7f4e4bfff640 1 -- 172.21.15.55:0/2848539114 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e4c158590 con 0x7f4e4c13dd00 2024-03-19T01:52:04.736 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.733+0000 7f4e4bfff640 1 --2- 172.21.15.55:0/2848539114 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4e4c13dd00 0x7f4e4c0c7430 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c072a00 tx=0x7f4e3c01c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a707bc413d94664f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:04.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.741+0000 7f4e4affd640 1 -- 172.21.15.55:0/2848539114 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c0b1070 con 0x7f4e4c13dd00 2024-03-19T01:52:04.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.741+0000 7f4e4affd640 1 -- 172.21.15.55:0/2848539114 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e3c0ac410 con 0x7f4e4c13dd00 2024-03-19T01:52:04.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.741+0000 7f4e4affd640 1 -- 172.21.15.55:0/2848539114 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c0ac7e0 con 0x7f4e4c13dd00 2024-03-19T01:52:04.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.745+0000 7f4e524f9640 1 -- 172.21.15.55:0/2848539114 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4e4c13dd00 msgr2=0x7f4e4c0c7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:04.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.745+0000 7f4e524f9640 1 --2- 172.21.15.55:0/2848539114 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4e4c13dd00 0x7f4e4c0c7430 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c072a00 tx=0x7f4e3c01c520 comp rx=0 tx=0).stop 2024-03-19T01:52:04.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.745+0000 7f4e524f9640 1 -- 172.21.15.55:0/2848539114 shutdown_connections 2024-03-19T01:52:04.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.745+0000 7f4e524f9640 1 --2- 172.21.15.55:0/2848539114 >> v2:172.21.15.55:3300/0 conn(0x7f4e4c15b550 0x7f4e4c0c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:04.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.745+0000 7f4e524f9640 1 --2- 172.21.15.55:0/2848539114 >> v2:172.21.15.84:3300/0 conn(0x7f4e4c0c9780 0x7f4e4c0c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:04.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.745+0000 7f4e524f9640 1 --2- 172.21.15.55:0/2848539114 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4e4c13dd00 0x7f4e4c0c7430 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:04.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.745+0000 7f4e524f9640 1 -- 172.21.15.55:0/2848539114 >> 172.21.15.55:0/2848539114 conn(0x7f4e4c06e5a0 msgr2=0x7f4e4c072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:04.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.745+0000 7f4e524f9640 1 -- 172.21.15.55:0/2848539114 shutdown_connections 2024-03-19T01:52:04.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.745+0000 7f4e524f9640 1 -- 172.21.15.55:0/2848539114 wait complete. 2024-03-19T01:52:04.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.745+0000 7f4e524f9640 1 Processor -- start 2024-03-19T01:52:04.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.745+0000 7f4e524f9640 1 -- start start 2024-03-19T01:52:04.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.745+0000 7f4e524f9640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e4c0c9780 0x7f4e4c1f16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:04.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.745+0000 7f4e524f9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4e4c13dd00 0x7f4e4c156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:04.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.745+0000 7f4e524f9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e4c15b550 0x7f4e4c1f4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:04.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.745+0000 7f4e524f9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4e4c158f10 con 0x7f4e4c15b550 2024-03-19T01:52:04.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.745+0000 7f4e524f9640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4e4c158d90 con 0x7f4e4c0c9780 2024-03-19T01:52:04.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.745+0000 7f4e524f9640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4e4c0cc7c0 con 0x7f4e4c13dd00 2024-03-19T01:52:04.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.749+0000 7f4e4b7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4e4c13dd00 0x7f4e4c156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:04.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.749+0000 7f4e4b7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4e4c13dd00 0x7f4e4c156960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:42934/0 (socket says 172.21.15.55:42934) 2024-03-19T01:52:04.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.749+0000 7f4e4b7fe640 1 -- 172.21.15.55:0/1471122192 learned_addr learned my addr 172.21.15.55:0/1471122192 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:04.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.749+0000 7f4e4bfff640 1 --2- 172.21.15.55:0/1471122192 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e4c0c9780 0x7f4e4c1f16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:04.754 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.753+0000 7f4e4b7fe640 1 -- 172.21.15.55:0/1471122192 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e4c0c9780 msgr2=0x7f4e4c1f16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:04.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.753+0000 7f4e4b7fe640 1 --2- 172.21.15.55:0/1471122192 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e4c0c9780 0x7f4e4c1f16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:04.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.753+0000 7f4e4b7fe640 1 -- 172.21.15.55:0/1471122192 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e4c15b550 msgr2=0x7f4e4c1f4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:04.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.753+0000 7f4e4b7fe640 1 --2- 172.21.15.55:0/1471122192 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e4c15b550 0x7f4e4c1f4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:04.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.753+0000 7f4e4b7fe640 1 -- 172.21.15.55:0/1471122192 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e4c1f41b0 con 0x7f4e4c13dd00 2024-03-19T01:52:04.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.757+0000 7f4e4b7fe640 1 --2- 172.21.15.55:0/1471122192 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4e4c13dd00 0x7f4e4c156960 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4e4000e930 tx=0x7f4e4000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:04.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.765+0000 7f4e48ff9640 1 -- 172.21.15.55:0/1471122192 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e40017070 con 0x7f4e4c13dd00 2024-03-19T01:52:04.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.765+0000 7f4e48ff9640 1 -- 172.21.15.55:0/1471122192 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e4000ca00 con 0x7f4e4c13dd00 2024-03-19T01:52:04.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.765+0000 7f4e48ff9640 1 -- 172.21.15.55:0/1471122192 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4000cdd0 con 0x7f4e4c13dd00 2024-03-19T01:52:04.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.765+0000 7f4e524f9640 1 -- 172.21.15.55:0/1471122192 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e4c152aa0 con 0x7f4e4c13dd00 2024-03-19T01:52:04.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.765+0000 7f4e524f9640 1 -- 172.21.15.55:0/1471122192 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e4c153060 con 0x7f4e4c13dd00 2024-03-19T01:52:04.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:04.765+0000 7f4e524f9640 1 -- 172.21.15.55:0/1471122192 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e4c068a40 con 0x7f4e4c13dd00 2024-03-19T01:52:05.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.109+0000 7f4e48ff9640 1 -- 172.21.15.55:0/1471122192 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f4e4001a070 con 0x7f4e4c13dd00 2024-03-19T01:52:05.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.113+0000 7f4e48ff9640 1 --2- 172.21.15.55:0/1471122192 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4e380676b0 0x7f4e38069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:05.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.113+0000 7f4e4bfff640 1 --2- 172.21.15.55:0/1471122192 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4e380676b0 0x7f4e38069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:05.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.113+0000 7f4e4bfff640 1 --2- 172.21.15.55:0/1471122192 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4e380676b0 0x7f4e38069b70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c0ae000 tx=0x7f4e3c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:05.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.117+0000 7f4e48ff9640 1 -- 172.21.15.55:0/1471122192 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f4e40054710 con 0x7f4e4c13dd00 2024-03-19T01:52:05.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.137+0000 7f4e48ff9640 1 -- 172.21.15.55:0/1471122192 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4e40056670 con 0x7f4e4c13dd00 2024-03-19T01:52:05.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.345+0000 7f4e524f9640 1 -- 172.21.15.55:0/1471122192 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f4e4c0c9560 con 0x7f4e4c13dd00 2024-03-19T01:52:05.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.349+0000 7f4e48ff9640 1 -- 172.21.15.55:0/1471122192 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f4e4005a710 con 0x7f4e4c13dd00 2024-03-19T01:52:05.352 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:52:05.352 INFO:teuthology.orchestra.run.smithi055.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-19T01:52:05.354 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.353+0000 7f4e524f9640 1 -- 172.21.15.55:0/1471122192 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4e380676b0 msgr2=0x7f4e38069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:05.354 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.353+0000 7f4e524f9640 1 --2- 172.21.15.55:0/1471122192 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4e380676b0 0x7f4e38069b70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c0ae000 tx=0x7f4e3c009040 comp rx=0 tx=0).stop 2024-03-19T01:52:05.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.353+0000 7f4e524f9640 1 -- 172.21.15.55:0/1471122192 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4e4c13dd00 msgr2=0x7f4e4c156960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:05.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.353+0000 7f4e524f9640 1 --2- 172.21.15.55:0/1471122192 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4e4c13dd00 0x7f4e4c156960 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4e4000e930 tx=0x7f4e4000edf0 comp rx=0 tx=0).stop 2024-03-19T01:52:05.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.353+0000 7f4e524f9640 1 -- 172.21.15.55:0/1471122192 shutdown_connections 2024-03-19T01:52:05.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.353+0000 7f4e524f9640 1 --2- 172.21.15.55:0/1471122192 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4e380676b0 0x7f4e38069b70 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:05.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.353+0000 7f4e524f9640 1 --2- 172.21.15.55:0/1471122192 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e4c15b550 0x7f4e4c1f4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:05.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.353+0000 7f4e524f9640 1 --2- 172.21.15.55:0/1471122192 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4e4c13dd00 0x7f4e4c156960 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:05.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.353+0000 7f4e524f9640 1 --2- 172.21.15.55:0/1471122192 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e4c0c9780 0x7f4e4c1f16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:05.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.353+0000 7f4e524f9640 1 -- 172.21.15.55:0/1471122192 >> 172.21.15.55:0/1471122192 conn(0x7f4e4c06e5a0 msgr2=0x7f4e4c155dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:05.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.353+0000 7f4e524f9640 1 -- 172.21.15.55:0/1471122192 shutdown_connections 2024-03-19T01:52:05.356 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.353+0000 7f4e524f9640 1 -- 172.21.15.55:0/1471122192 wait complete. 2024-03-19T01:52:05.372 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-19T01:52:05.372 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:52:05.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.489+0000 7f93d357a640 1 Processor -- start 2024-03-19T01:52:05.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.493+0000 7f93d357a640 1 -- start start 2024-03-19T01:52:05.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.493+0000 7f93d357a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f93cc150ae0 0x7f93cc0c6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:05.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.493+0000 7f93d357a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f93cc06c630 0x7f93cc160f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:05.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.493+0000 7f93d357a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93cc06c230 con 0x7f93cc0c3520 2024-03-19T01:52:05.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.493+0000 7f93d357a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f93cc0c3a40 con 0x7f93cc150ae0 2024-03-19T01:52:05.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.493+0000 7f93d357a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f93cc0c8200 con 0x7f93cc06c630 2024-03-19T01:52:05.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.493+0000 7f93d1316640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f93cc150ae0 0x7f93cc0c6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:05.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.493+0000 7f93d1b17640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f93cc06c630 0x7f93cc160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:05.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.493+0000 7f93d1316640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f93cc150ae0 0x7f93cc0c6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53310/0 (socket says 172.21.15.55:53310) 2024-03-19T01:52:05.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.493+0000 7f93d1316640 1 -- 172.21.15.55:0/170485013 learned_addr learned my addr 172.21.15.55:0/170485013 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:05.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.497+0000 7f93d1316640 1 -- 172.21.15.55:0/170485013 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f93cc06c630 msgr2=0x7f93cc160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:05.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.497+0000 7f93d1316640 1 --2- 172.21.15.55:0/170485013 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f93cc06c630 0x7f93cc160f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:05.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.497+0000 7f93d1316640 1 -- 172.21.15.55:0/170485013 >> v1:172.21.15.84:6789/0 conn(0x7f93cc0c3520 legacy=0x7f93cc0cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:05.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.497+0000 7f93d1316640 1 -- 172.21.15.55:0/170485013 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93cc06bc70 con 0x7f93cc150ae0 2024-03-19T01:52:05.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.497+0000 7f93d1316640 1 --2- 172.21.15.55:0/170485013 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93cc150ae0 0x7f93cc0c6e80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009870 tx=0x7f93c801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=417f9cef3c368b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:05.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.505+0000 7f93c3fff640 1 -- 172.21.15.55:0/170485013 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c80b0070 con 0x7f93cc150ae0 2024-03-19T01:52:05.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.505+0000 7f93c3fff640 1 -- 172.21.15.55:0/170485013 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93c80ab600 con 0x7f93cc150ae0 2024-03-19T01:52:05.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.505+0000 7f93c3fff640 1 -- 172.21.15.55:0/170485013 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c80ab9d0 con 0x7f93cc150ae0 2024-03-19T01:52:05.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.505+0000 7f93d357a640 1 -- 172.21.15.55:0/170485013 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93cc150ae0 msgr2=0x7f93cc0c6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:05.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.505+0000 7f93d357a640 1 --2- 172.21.15.55:0/170485013 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93cc150ae0 0x7f93cc0c6e80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009870 tx=0x7f93c801c560 comp rx=0 tx=0).stop 2024-03-19T01:52:05.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.505+0000 7f93d357a640 1 -- 172.21.15.55:0/170485013 shutdown_connections 2024-03-19T01:52:05.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.505+0000 7f93d357a640 1 --2- 172.21.15.55:0/170485013 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f93cc06c630 0x7f93cc160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:05.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.505+0000 7f93d357a640 1 --2- 172.21.15.55:0/170485013 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93cc150ae0 0x7f93cc0c6e80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:05.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.505+0000 7f93d357a640 1 -- 172.21.15.55:0/170485013 >> 172.21.15.55:0/170485013 conn(0x7f93cc06e5a0 msgr2=0x7f93cc072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:05.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.505+0000 7f93d357a640 1 -- 172.21.15.55:0/170485013 shutdown_connections 2024-03-19T01:52:05.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.505+0000 7f93d357a640 1 -- 172.21.15.55:0/170485013 wait complete. 2024-03-19T01:52:05.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.505+0000 7f93d357a640 1 Processor -- start 2024-03-19T01:52:05.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.505+0000 7f93d357a640 1 -- start start 2024-03-19T01:52:05.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.505+0000 7f93d357a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93cc06c630 0x7f93cc1f3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:05.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.505+0000 7f93d357a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93cc0c3520 0x7f93cc14c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:05.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.505+0000 7f93d357a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f93cc150ae0 0x7f93cc1f4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:05.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.505+0000 7f93d357a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f93cc0c6a20 con 0x7f93cc06c630 2024-03-19T01:52:05.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.505+0000 7f93d357a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f93cc0c6380 con 0x7f93cc0c3520 2024-03-19T01:52:05.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.505+0000 7f93d357a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f93cc0c6ba0 con 0x7f93cc150ae0 2024-03-19T01:52:05.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.505+0000 7f93d1316640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93cc06c630 0x7f93cc1f3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:05.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.509+0000 7f93d1b17640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f93cc150ae0 0x7f93cc1f4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:05.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.509+0000 7f93d0b15640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93cc0c3520 0x7f93cc14c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:05.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.509+0000 7f93d1316640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93cc06c630 0x7f93cc1f3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53322/0 (socket says 172.21.15.55:53322) 2024-03-19T01:52:05.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.509+0000 7f93d1316640 1 -- 172.21.15.55:0/1285945254 learned_addr learned my addr 172.21.15.55:0/1285945254 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:05.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.509+0000 7f93d1316640 1 -- 172.21.15.55:0/1285945254 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f93cc150ae0 msgr2=0x7f93cc1f4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:05.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.509+0000 7f93d1316640 1 --2- 172.21.15.55:0/1285945254 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f93cc150ae0 0x7f93cc1f4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:05.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.509+0000 7f93d1316640 1 -- 172.21.15.55:0/1285945254 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93cc0c3520 msgr2=0x7f93cc14c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:05.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.509+0000 7f93d1316640 1 --2- 172.21.15.55:0/1285945254 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93cc0c3520 0x7f93cc14c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:05.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.509+0000 7f93d1316640 1 -- 172.21.15.55:0/1285945254 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93cc0c5cb0 con 0x7f93cc06c630 2024-03-19T01:52:05.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.513+0000 7f93d1316640 1 --2- 172.21.15.55:0/1285945254 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93cc06c630 0x7f93cc1f3960 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f93c80ae040 tx=0x7f93c8019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:05.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.517+0000 7f93c1ffb640 1 -- 172.21.15.55:0/1285945254 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c80b0040 con 0x7f93cc06c630 2024-03-19T01:52:05.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.517+0000 7f93c1ffb640 1 -- 172.21.15.55:0/1285945254 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93c80b9070 con 0x7f93cc06c630 2024-03-19T01:52:05.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.517+0000 7f93d357a640 1 -- 172.21.15.55:0/1285945254 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93cc153e70 con 0x7f93cc06c630 2024-03-19T01:52:05.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.517+0000 7f93c1ffb640 1 -- 172.21.15.55:0/1285945254 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c80b4680 con 0x7f93cc06c630 2024-03-19T01:52:05.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.517+0000 7f93d357a640 1 -- 172.21.15.55:0/1285945254 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93cc1556a0 con 0x7f93cc06c630 2024-03-19T01:52:05.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.517+0000 7f93d357a640 1 -- 172.21.15.55:0/1285945254 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93cc068a40 con 0x7f93cc06c630 2024-03-19T01:52:05.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.537+0000 7f93c1ffb640 1 -- 172.21.15.55:0/1285945254 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f93c80b4820 con 0x7f93cc06c630 2024-03-19T01:52:05.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.537+0000 7f93c1ffb640 1 --2- 172.21.15.55:0/1285945254 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f93b00676e0 0x7f93b0069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:05.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.537+0000 7f93d0b15640 1 --2- 172.21.15.55:0/1285945254 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f93b00676e0 0x7f93b0069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:05.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.537+0000 7f93d0b15640 1 --2- 172.21.15.55:0/1285945254 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f93b00676e0 0x7f93b0069ba0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f93cc14dc90 tx=0x7f93bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:05.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.541+0000 7f93c1ffb640 1 -- 172.21.15.55:0/1285945254 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f93c80ee410 con 0x7f93cc06c630 2024-03-19T01:52:05.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:05.837+0000 7f93c1ffb640 1 -- 172.21.15.55:0/1285945254 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f93cc0c5cb0 con 0x7f93cc06c630 2024-03-19T01:52:06.026 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.025+0000 7f93d357a640 1 -- 172.21.15.55:0/1285945254 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f93cc06c410 con 0x7f93cc06c630 2024-03-19T01:52:06.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.085+0000 7f93c1ffb640 1 -- 172.21.15.55:0/1285945254 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+18430 (secure 0 0 0) 0x7f93c8016150 con 0x7f93cc06c630 2024-03-19T01:52:06.085 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:52:06.085 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":29,"fsid":"def9d4e5-0bc7-449d-b893-2490cbd10e25","created":"2024-03-19T01:50:02.482954+0000","modified":"2024-03-19T01:51:17.228291+0000","last_up_change":"2024-03-19T01:50:21.735508+0000","last_in_change":"2024-03-19T01:50:12.168944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:50:23.456904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:51:04.578583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:51:06.871659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T01:51:14.274442+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"35d5e3c5-d089-4f38-8961-98d064188f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6825","nonce":2757675043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6827","nonce":2757675043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6831","nonce":2757675043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6829","nonce":2757675043}]},"public_addr":"172.21.15.55:6825/2757675043","cluster_addr":"172.21.15.55:6827/2757675043","heartbeat_back_addr":"172.21.15.55:6831/2757675043","heartbeat_front_addr":"172.21.15.55:6829/2757675043","state":["exists","up"]},{"osd":1,"uuid":"8e58f791-001a-44d0-a464-ca68a479f70c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6817","nonce":543460007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6819","nonce":543460007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6823","nonce":543460007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6821","nonce":543460007}]},"public_addr":"172.21.15.55:6817/543460007","cluster_addr":"172.21.15.55:6819/543460007","heartbeat_back_addr":"172.21.15.55:6823/543460007","heartbeat_front_addr":"172.21.15.55:6821/543460007","state":["exists","up"]},{"osd":2,"uuid":"d44f38b8-67f4-43de-98a6-f3c520acaf10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6801","nonce":1760253550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6803","nonce":1760253550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6807","nonce":1760253550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6805","nonce":1760253550}]},"public_addr":"172.21.15.55:6801/1760253550","cluster_addr":"172.21.15.55:6803/1760253550","heartbeat_back_addr":"172.21.15.55:6807/1760253550","heartbeat_front_addr":"172.21.15.55:6805/1760253550","state":["exists","up"]},{"osd":3,"uuid":"850a7316-2d74-43f0-8649-b5f8d8f6692e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6809","nonce":2797616448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6811","nonce":2797616448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6815","nonce":2797616448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6813","nonce":2797616448}]},"public_addr":"172.21.15.55:6809/2797616448","cluster_addr":"172.21.15.55:6811/2797616448","heartbeat_back_addr":"172.21.15.55:6815/2797616448","heartbeat_front_addr":"172.21.15.55:6813/2797616448","state":["exists","up"]},{"osd":4,"uuid":"4e6152ae-a77d-4bb6-a855-5782ad5411a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6819","nonce":2938436826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6821","nonce":2938436826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6825","nonce":2938436826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6823","nonce":2938436826}]},"public_addr":"172.21.15.84:6819/2938436826","cluster_addr":"172.21.15.84:6821/2938436826","heartbeat_back_addr":"172.21.15.84:6825/2938436826","heartbeat_front_addr":"172.21.15.84:6823/2938436826","state":["exists","up"]},{"osd":5,"uuid":"1217ceba-dd31-4615-96f1-dfaf57160f3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6803","nonce":460717295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6805","nonce":460717295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6809","nonce":460717295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6807","nonce":460717295}]},"public_addr":"172.21.15.84:6803/460717295","cluster_addr":"172.21.15.84:6805/460717295","heartbeat_back_addr":"172.21.15.84:6809/460717295","heartbeat_front_addr":"172.21.15.84:6807/460717295","state":["exists","up"]},{"osd":6,"uuid":"3d9ed78a-2284-4c28-a24a-70eb812c7836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6827","nonce":416123135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6829","nonce":416123135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6833","nonce":416123135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6831","nonce":416123135}]},"public_addr":"172.21.15.84:6827/416123135","cluster_addr":"172.21.15.84:6829/416123135","heartbeat_back_addr":"172.21.15.84:6833/416123135","heartbeat_front_addr":"172.21.15.84:6831/416123135","state":["exists","up"]},{"osd":7,"uuid":"c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6811","nonce":460083528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6813","nonce":460083528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6817","nonce":460083528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6815","nonce":460083528}]},"public_addr":"172.21.15.84:6811/460083528","cluster_addr":"172.21.15.84:6813/460083528","heartbeat_back_addr":"172.21.15.84:6817/460083528","heartbeat_front_addr":"172.21.15.84:6815/460083528","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.269126+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.121001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.075714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.092461+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.194676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.062833+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.316515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.237111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:52:06.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.085+0000 7f93d357a640 1 -- 172.21.15.55:0/1285945254 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f93b00676e0 msgr2=0x7f93b0069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:06.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.085+0000 7f93d357a640 1 --2- 172.21.15.55:0/1285945254 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f93b00676e0 0x7f93b0069ba0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f93cc14dc90 tx=0x7f93bc009040 comp rx=0 tx=0).stop 2024-03-19T01:52:06.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.085+0000 7f93d357a640 1 -- 172.21.15.55:0/1285945254 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93cc06c630 msgr2=0x7f93cc1f3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:06.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.085+0000 7f93d357a640 1 --2- 172.21.15.55:0/1285945254 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93cc06c630 0x7f93cc1f3960 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f93c80ae040 tx=0x7f93c8019300 comp rx=0 tx=0).stop 2024-03-19T01:52:06.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.085+0000 7f93d357a640 1 -- 172.21.15.55:0/1285945254 shutdown_connections 2024-03-19T01:52:06.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.085+0000 7f93d357a640 1 --2- 172.21.15.55:0/1285945254 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f93b00676e0 0x7f93b0069ba0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.085+0000 7f93d357a640 1 --2- 172.21.15.55:0/1285945254 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f93cc150ae0 0x7f93cc1f4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.085+0000 7f93d357a640 1 --2- 172.21.15.55:0/1285945254 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93cc0c3520 0x7f93cc14c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.085+0000 7f93d357a640 1 --2- 172.21.15.55:0/1285945254 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93cc06c630 0x7f93cc1f3960 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.085+0000 7f93d357a640 1 -- 172.21.15.55:0/1285945254 >> 172.21.15.55:0/1285945254 conn(0x7f93cc06e5a0 msgr2=0x7f93cc0c9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:06.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.085+0000 7f93d357a640 1 -- 172.21.15.55:0/1285945254 shutdown_connections 2024-03-19T01:52:06.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.085+0000 7f93d357a640 1 -- 172.21.15.55:0/1285945254 wait complete. 2024-03-19T01:52:06.104 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:50:23.456904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:51:04.578583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:51:06.871659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T01:51:14.274442+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T01:52:06.105 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:52:06.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.221+0000 7fc7072d6640 1 Processor -- start 2024-03-19T01:52:06.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.221+0000 7fc7072d6640 1 -- start start 2024-03-19T01:52:06.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.221+0000 7fc7072d6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc700152120 0x7fc7000c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:06.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.221+0000 7fc7072d6640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fc7000c9610 0x7fc700157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:06.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.221+0000 7fc7072d6640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc700070780 0x7fc700158510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:06.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.221+0000 7fc7072d6640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fc700152500 con 0x7fc700070780 2024-03-19T01:52:06.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.221+0000 7fc7072d6640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fc700070590 con 0x7fc7000c9610 2024-03-19T01:52:06.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.221+0000 7fc7072d6640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc7000c3530 con 0x7fc700152120 2024-03-19T01:52:06.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.221+0000 7fc704871640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fc7000c9610 0x7fc700157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:06.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.221+0000 7fc705873640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc700070780 0x7fc700158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:06.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.225+0000 7fc705072640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc700152120 0x7fc7000c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:06.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.225+0000 7fc705072640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc700152120 0x7fc7000c90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:42954/0 (socket says 172.21.15.55:42954) 2024-03-19T01:52:06.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.225+0000 7fc705072640 1 -- 172.21.15.55:0/3707656733 learned_addr learned my addr 172.21.15.55:0/3707656733 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:06.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.225+0000 7fc705072640 1 -- 172.21.15.55:0/3707656733 >> v2:172.21.15.84:3300/0 conn(0x7fc7000c9610 msgr2=0x7fc700157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:06.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.225+0000 7fc705072640 1 --2- 172.21.15.55:0/3707656733 >> v2:172.21.15.84:3300/0 conn(0x7fc7000c9610 0x7fc700157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.225+0000 7fc705072640 1 -- 172.21.15.55:0/3707656733 >> v2:172.21.15.55:3300/0 conn(0x7fc700070780 msgr2=0x7fc700158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:06.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.225+0000 7fc705072640 1 --2- 172.21.15.55:0/3707656733 >> v2:172.21.15.55:3300/0 conn(0x7fc700070780 0x7fc700158510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.225+0000 7fc705072640 1 -- 172.21.15.55:0/3707656733 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc70006fff0 con 0x7fc700152120 2024-03-19T01:52:06.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.225+0000 7fc704871640 1 --2- 172.21.15.55:0/3707656733 >> v2:172.21.15.84:3300/0 conn(0x7fc7000c9610 0x7fc700157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:52:06.232 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.229+0000 7fc705072640 1 --2- 172.21.15.55:0/3707656733 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc700152120 0x7fc7000c90d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc7000cb040 tx=0x7fc6fc01c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9deb3779038c27a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:06.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.237+0000 7fc6f7fff640 1 -- 172.21.15.55:0/3707656733 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc0b0070 con 0x7fc700152120 2024-03-19T01:52:06.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.237+0000 7fc6f7fff640 1 -- 172.21.15.55:0/3707656733 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6fc0ab600 con 0x7fc700152120 2024-03-19T01:52:06.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.237+0000 7fc6f7fff640 1 -- 172.21.15.55:0/3707656733 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc0ab9d0 con 0x7fc700152120 2024-03-19T01:52:06.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.241+0000 7fc7072d6640 1 -- 172.21.15.55:0/3707656733 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc700152120 msgr2=0x7fc7000c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:06.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.241+0000 7fc7072d6640 1 --2- 172.21.15.55:0/3707656733 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc700152120 0x7fc7000c90d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc7000cb040 tx=0x7fc6fc01c520 comp rx=0 tx=0).stop 2024-03-19T01:52:06.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.241+0000 7fc7072d6640 1 -- 172.21.15.55:0/3707656733 shutdown_connections 2024-03-19T01:52:06.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.241+0000 7fc7072d6640 1 --2- 172.21.15.55:0/3707656733 >> v2:172.21.15.55:3300/0 conn(0x7fc700070780 0x7fc700158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.241+0000 7fc7072d6640 1 --2- 172.21.15.55:0/3707656733 >> v2:172.21.15.84:3300/0 conn(0x7fc7000c9610 0x7fc700157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.241+0000 7fc7072d6640 1 --2- 172.21.15.55:0/3707656733 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc700152120 0x7fc7000c90d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.241+0000 7fc7072d6640 1 -- 172.21.15.55:0/3707656733 >> 172.21.15.55:0/3707656733 conn(0x7fc700067600 msgr2=0x7fc700071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:06.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.241+0000 7fc7072d6640 1 -- 172.21.15.55:0/3707656733 shutdown_connections 2024-03-19T01:52:06.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.241+0000 7fc7072d6640 1 -- 172.21.15.55:0/3707656733 wait complete. 2024-03-19T01:52:06.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.241+0000 7fc7072d6640 1 Processor -- start 2024-03-19T01:52:06.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.241+0000 7fc7072d6640 1 -- start start 2024-03-19T01:52:06.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.245+0000 7fc7072d6640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc700070780 0x7fc7001f74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:06.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.245+0000 7fc7072d6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7000c9610 0x7fc70015aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:06.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.245+0000 7fc7072d6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc700152120 0x7fc7000cc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:06.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.245+0000 7fc7072d6640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc7000c2ee0 con 0x7fc700152120 2024-03-19T01:52:06.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.245+0000 7fc7072d6640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc70006e3c0 con 0x7fc700070780 2024-03-19T01:52:06.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.245+0000 7fc7072d6640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc7000c31c0 con 0x7fc7000c9610 2024-03-19T01:52:06.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.245+0000 7fc705072640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc700070780 0x7fc7001f74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:06.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.245+0000 7fc705072640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc700070780 0x7fc7001f74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:56086/0 (socket says 172.21.15.55:56086) 2024-03-19T01:52:06.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.245+0000 7fc705072640 1 -- 172.21.15.55:0/394424128 learned_addr learned my addr 172.21.15.55:0/394424128 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:06.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.245+0000 7fc704871640 1 --2- 172.21.15.55:0/394424128 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7000c9610 0x7fc70015aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:06.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.249+0000 7fc705873640 1 --2- 172.21.15.55:0/394424128 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc700152120 0x7fc7000cc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:06.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.249+0000 7fc705072640 1 -- 172.21.15.55:0/394424128 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7000c9610 msgr2=0x7fc70015aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:06.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.249+0000 7fc705072640 1 --2- 172.21.15.55:0/394424128 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7000c9610 0x7fc70015aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.249+0000 7fc705072640 1 -- 172.21.15.55:0/394424128 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc700152120 msgr2=0x7fc7000cc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:06.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.249+0000 7fc705072640 1 --2- 172.21.15.55:0/394424128 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc700152120 0x7fc7000cc510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.249+0000 7fc705072640 1 -- 172.21.15.55:0/394424128 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7000ce270 con 0x7fc700070780 2024-03-19T01:52:06.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.257+0000 7fc705072640 1 --2- 172.21.15.55:0/394424128 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc700070780 0x7fc7001f74e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc01cdc0 tx=0x7fc6fc0192c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:06.268 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.265+0000 7fc6f5ffb640 1 -- 172.21.15.55:0/394424128 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc0bb020 con 0x7fc700070780 2024-03-19T01:52:06.268 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.265+0000 7fc6f5ffb640 1 -- 172.21.15.55:0/394424128 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6fc0acdb0 con 0x7fc700070780 2024-03-19T01:52:06.268 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.265+0000 7fc6f5ffb640 1 -- 172.21.15.55:0/394424128 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc0b0040 con 0x7fc700070780 2024-03-19T01:52:06.269 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.265+0000 7fc7072d6640 1 -- 172.21.15.55:0/394424128 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7000ce550 con 0x7fc700070780 2024-03-19T01:52:06.269 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.269+0000 7fc7072d6640 1 -- 172.21.15.55:0/394424128 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7000cba30 con 0x7fc700070780 2024-03-19T01:52:06.269 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.269+0000 7fc7072d6640 1 -- 172.21.15.55:0/394424128 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc700068a40 con 0x7fc700070780 2024-03-19T01:52:06.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.305+0000 7fc6f5ffb640 1 -- 172.21.15.55:0/394424128 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc6fc0be020 con 0x7fc700070780 2024-03-19T01:52:06.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.309+0000 7fc6f5ffb640 1 --2- 172.21.15.55:0/394424128 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc6e4067660 0x7fc6e4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:06.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.309+0000 7fc704871640 1 --2- 172.21.15.55:0/394424128 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc6e4067660 0x7fc6e4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:06.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.309+0000 7fc704871640 1 --2- 172.21.15.55:0/394424128 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc6e4067660 0x7fc6e4069b20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0009580 tx=0x7fc6f0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:06.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.601+0000 7fc6f5ffb640 1 -- 172.21.15.55:0/394424128 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fc6fc0b8e10 con 0x7fc700070780 2024-03-19T01:52:06.621 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.621+0000 7fc6f5ffb640 1 -- 172.21.15.55:0/394424128 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc6fc016110 con 0x7fc700070780 2024-03-19T01:52:06.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.793+0000 7fc7072d6640 1 -- 172.21.15.55:0/394424128 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc7000c84b0 con 0x7fc700070780 2024-03-19T01:52:06.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.805+0000 7fc6f5ffb640 1 -- 172.21.15.55:0/394424128 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) ==== 93+0+10 (secure 0 0 0) 0x7fc6fc0f0a40 con 0x7fc700070780 2024-03-19T01:52:06.808 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 1 2024-03-19T01:52:06.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.809+0000 7fc7072d6640 1 -- 172.21.15.55:0/394424128 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc6e4067660 msgr2=0x7fc6e4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:06.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.809+0000 7fc7072d6640 1 --2- 172.21.15.55:0/394424128 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc6e4067660 0x7fc6e4069b20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0009580 tx=0x7fc6f0009190 comp rx=0 tx=0).stop 2024-03-19T01:52:06.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.809+0000 7fc7072d6640 1 -- 172.21.15.55:0/394424128 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc700070780 msgr2=0x7fc7001f74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:06.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.809+0000 7fc7072d6640 1 --2- 172.21.15.55:0/394424128 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc700070780 0x7fc7001f74e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc01cdc0 tx=0x7fc6fc0192c0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.809+0000 7fc7072d6640 1 -- 172.21.15.55:0/394424128 shutdown_connections 2024-03-19T01:52:06.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.809+0000 7fc7072d6640 1 --2- 172.21.15.55:0/394424128 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc6e4067660 0x7fc6e4069b20 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.809+0000 7fc7072d6640 1 --2- 172.21.15.55:0/394424128 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc700152120 0x7fc7000cc510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.809+0000 7fc7072d6640 1 --2- 172.21.15.55:0/394424128 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7000c9610 0x7fc70015aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.809+0000 7fc7072d6640 1 --2- 172.21.15.55:0/394424128 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc700070780 0x7fc7001f74e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.809+0000 7fc7072d6640 1 -- 172.21.15.55:0/394424128 >> 172.21.15.55:0/394424128 conn(0x7fc700067600 msgr2=0x7fc70014cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:06.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.809+0000 7fc7072d6640 1 -- 172.21.15.55:0/394424128 shutdown_connections 2024-03-19T01:52:06.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.809+0000 7fc7072d6640 1 -- 172.21.15.55:0/394424128 wait complete. 2024-03-19T01:52:06.827 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:52:06.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.937+0000 7f170e051640 1 Processor -- start 2024-03-19T01:52:06.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.937+0000 7f170e051640 1 -- start start 2024-03-19T01:52:06.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.937+0000 7f170e051640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1708152250 0x7f17080c8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:06.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.937+0000 7f170e051640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f1708152630 0x7f17080cad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:06.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.937+0000 7f170e051640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17080cb290 0x7f17081580c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:06.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.937+0000 7f170e051640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c9190 con 0x7f1708152250 2024-03-19T01:52:06.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.937+0000 7f170e051640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c2df0 con 0x7f1708152630 2024-03-19T01:52:06.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.937+0000 7f170e051640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1708158b70 con 0x7f17080cb290 2024-03-19T01:52:06.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.941+0000 7f17077fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1708152250 0x7f17080c8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:06.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.941+0000 7f1706ffd640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f1708152630 0x7f17080cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:06.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.941+0000 7f1707fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17080cb290 0x7f17081580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:06.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.941+0000 7f17077fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1708152250 0x7f17080c8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53344/0 (socket says 172.21.15.55:53344) 2024-03-19T01:52:06.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.941+0000 7f17077fe640 1 -- 172.21.15.55:0/1557480561 learned_addr learned my addr 172.21.15.55:0/1557480561 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:06.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.941+0000 7f17077fe640 1 -- 172.21.15.55:0/1557480561 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17080cb290 msgr2=0x7f17081580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:06.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.941+0000 7f17077fe640 1 --2- 172.21.15.55:0/1557480561 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17080cb290 0x7f17081580c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.941+0000 7f17077fe640 1 -- 172.21.15.55:0/1557480561 >> v2:172.21.15.84:3300/0 conn(0x7f1708152630 msgr2=0x7f17080cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:06.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.941+0000 7f17077fe640 1 --2- 172.21.15.55:0/1557480561 >> v2:172.21.15.84:3300/0 conn(0x7f1708152630 0x7f17080cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.941+0000 7f17077fe640 1 -- 172.21.15.55:0/1557480561 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1708158e50 con 0x7f1708152250 2024-03-19T01:52:06.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.945+0000 7f17077fe640 1 --2- 172.21.15.55:0/1557480561 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1708152250 0x7f17080c8c50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009870 tx=0x7f16f801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3b4a8456456fa1d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:06.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.953+0000 7f17067fc640 1 -- 172.21.15.55:0/1557480561 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f80b5070 con 0x7f1708152250 2024-03-19T01:52:06.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.953+0000 7f17067fc640 1 -- 172.21.15.55:0/1557480561 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16f80a89d0 con 0x7f1708152250 2024-03-19T01:52:06.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.953+0000 7f17067fc640 1 -- 172.21.15.55:0/1557480561 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f80a8da0 con 0x7f1708152250 2024-03-19T01:52:06.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.953+0000 7f170e051640 1 -- 172.21.15.55:0/1557480561 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1708152250 msgr2=0x7f17080c8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:06.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.953+0000 7f170e051640 1 --2- 172.21.15.55:0/1557480561 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1708152250 0x7f17080c8c50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009870 tx=0x7f16f801ed10 comp rx=0 tx=0).stop 2024-03-19T01:52:06.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.953+0000 7f170e051640 1 -- 172.21.15.55:0/1557480561 shutdown_connections 2024-03-19T01:52:06.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.953+0000 7f170e051640 1 --2- 172.21.15.55:0/1557480561 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17080cb290 0x7f17081580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.953+0000 7f170e051640 1 --2- 172.21.15.55:0/1557480561 >> v2:172.21.15.84:3300/0 conn(0x7f1708152630 0x7f17080cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.953+0000 7f170e051640 1 --2- 172.21.15.55:0/1557480561 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1708152250 0x7f17080c8c50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.953+0000 7f170e051640 1 -- 172.21.15.55:0/1557480561 >> 172.21.15.55:0/1557480561 conn(0x7f1708067000 msgr2=0x7f1708067410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:06.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.953+0000 7f170e051640 1 -- 172.21.15.55:0/1557480561 shutdown_connections 2024-03-19T01:52:06.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.953+0000 7f170e051640 1 -- 172.21.15.55:0/1557480561 wait complete. 2024-03-19T01:52:06.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.953+0000 7f170e051640 1 Processor -- start 2024-03-19T01:52:06.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.953+0000 7f170e051640 1 -- start start 2024-03-19T01:52:06.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.957+0000 7f170e051640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17080cb290 0x7f17080c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:06.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.957+0000 7f170e051640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1708152250 0x7f170814a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:06.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.957+0000 7f170e051640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1708152630 0x7f17081f0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:06.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.957+0000 7f170e051640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f17080ca830 con 0x7f17080cb290 2024-03-19T01:52:06.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.957+0000 7f170e051640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f17080ca550 con 0x7f1708152250 2024-03-19T01:52:06.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.957+0000 7f170e051640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1708158600 con 0x7f1708152630 2024-03-19T01:52:06.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.957+0000 7f17077fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17080cb290 0x7f17080c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:06.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.957+0000 7f1706ffd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1708152250 0x7f170814a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:06.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.957+0000 7f1707fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1708152630 0x7f17081f0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:06.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.957+0000 7f17077fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17080cb290 0x7f17080c3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53346/0 (socket says 172.21.15.55:53346) 2024-03-19T01:52:06.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.957+0000 7f17077fe640 1 -- 172.21.15.55:0/2591417764 learned_addr learned my addr 172.21.15.55:0/2591417764 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:06.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.961+0000 7f17077fe640 1 -- 172.21.15.55:0/2591417764 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1708152630 msgr2=0x7f17081f0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:06.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.961+0000 7f17077fe640 1 --2- 172.21.15.55:0/2591417764 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1708152630 0x7f17081f0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.961+0000 7f17077fe640 1 -- 172.21.15.55:0/2591417764 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1708152250 msgr2=0x7f170814a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:06.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.961+0000 7f17077fe640 1 --2- 172.21.15.55:0/2591417764 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1708152250 0x7f170814a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:06.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.961+0000 7f17077fe640 1 -- 172.21.15.55:0/2591417764 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17081ee7a0 con 0x7f17080cb290 2024-03-19T01:52:06.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.965+0000 7f17077fe640 1 --2- 172.21.15.55:0/2591417764 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17080cb290 0x7f17080c3c80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f16f801d3c0 tx=0x7f16f801bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:06.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.969+0000 7f16effff640 1 -- 172.21.15.55:0/2591417764 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f80be070 con 0x7f17080cb290 2024-03-19T01:52:06.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.969+0000 7f16effff640 1 -- 172.21.15.55:0/2591417764 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16f80b0660 con 0x7f17080cb290 2024-03-19T01:52:06.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.969+0000 7f16effff640 1 -- 172.21.15.55:0/2591417764 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f80b5040 con 0x7f17080cb290 2024-03-19T01:52:06.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.973+0000 7f170e051640 1 -- 172.21.15.55:0/2591417764 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f170814b460 con 0x7f17080cb290 2024-03-19T01:52:06.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.973+0000 7f170e051640 1 -- 172.21.15.55:0/2591417764 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1708149ea0 con 0x7f17080cb290 2024-03-19T01:52:06.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.973+0000 7f170e051640 1 -- 172.21.15.55:0/2591417764 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1708068a40 con 0x7f17080cb290 2024-03-19T01:52:06.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.993+0000 7f16effff640 1 -- 172.21.15.55:0/2591417764 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f16f80b0a50 con 0x7f17080cb290 2024-03-19T01:52:06.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.993+0000 7f16effff640 1 --2- 172.21.15.55:0/2591417764 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f16f4067690 0x7f16f4069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:06.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.993+0000 7f1706ffd640 1 --2- 172.21.15.55:0/2591417764 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f16f4067690 0x7f16f4069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:06.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.993+0000 7f1706ffd640 1 --2- 172.21.15.55:0/2591417764 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f16f4067690 0x7f16f4069b50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1708068e60 tx=0x7f16fc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:06.997 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:06.997+0000 7f16effff640 1 -- 172.21.15.55:0/2591417764 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f16f80f07e0 con 0x7f17080cb290 2024-03-19T01:52:07.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.253+0000 7f16effff640 1 -- 172.21.15.55:0/2591417764 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f16f8018870 con 0x7f17080cb290 2024-03-19T01:52:07.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.457+0000 7f170e051640 1 -- 172.21.15.55:0/2591417764 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1708148770 con 0x7f17080cb290 2024-03-19T01:52:07.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.565+0000 7f16effff640 1 -- 172.21.15.55:0/2591417764 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) ==== 104+0+11 (secure 0 0 0) 0x7f16f80f2740 con 0x7f17080cb290 2024-03-19T01:52:07.568 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 32 2024-03-19T01:52:07.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.569+0000 7f170e051640 1 -- 172.21.15.55:0/2591417764 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f16f4067690 msgr2=0x7f16f4069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:07.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.569+0000 7f170e051640 1 --2- 172.21.15.55:0/2591417764 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f16f4067690 0x7f16f4069b50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1708068e60 tx=0x7f16fc009190 comp rx=0 tx=0).stop 2024-03-19T01:52:07.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.569+0000 7f170e051640 1 -- 172.21.15.55:0/2591417764 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17080cb290 msgr2=0x7f17080c3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:07.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.569+0000 7f170e051640 1 --2- 172.21.15.55:0/2591417764 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17080cb290 0x7f17080c3c80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f16f801d3c0 tx=0x7f16f801bab0 comp rx=0 tx=0).stop 2024-03-19T01:52:07.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.569+0000 7f170e051640 1 -- 172.21.15.55:0/2591417764 shutdown_connections 2024-03-19T01:52:07.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.569+0000 7f170e051640 1 --2- 172.21.15.55:0/2591417764 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f16f4067690 0x7f16f4069b50 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:07.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.569+0000 7f170e051640 1 --2- 172.21.15.55:0/2591417764 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1708152630 0x7f17081f0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:07.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.569+0000 7f170e051640 1 --2- 172.21.15.55:0/2591417764 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1708152250 0x7f170814a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:07.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.569+0000 7f170e051640 1 --2- 172.21.15.55:0/2591417764 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17080cb290 0x7f17080c3c80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:07.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.569+0000 7f170e051640 1 -- 172.21.15.55:0/2591417764 >> 172.21.15.55:0/2591417764 conn(0x7f1708067000 msgr2=0x7f17080673e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:07.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.569+0000 7f170e051640 1 -- 172.21.15.55:0/2591417764 shutdown_connections 2024-03-19T01:52:07.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.569+0000 7f170e051640 1 -- 172.21.15.55:0/2591417764 wait complete. 2024-03-19T01:52:07.586 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:52:07.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.713+0000 7fcb49240640 1 Processor -- start 2024-03-19T01:52:07.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.713+0000 7fcb49240640 1 -- start start 2024-03-19T01:52:07.716 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.713+0000 7fcb49240640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fcb4415d2d0 0x7fcb4415aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:07.716 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.713+0000 7fcb49240640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcb4415bf60 0x7fcb4415c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:07.716 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.713+0000 7fcb49240640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb4415c910 con 0x7fcb440ceb10 2024-03-19T01:52:07.716 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.713+0000 7fcb49240640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fcb4415b3f0 con 0x7fcb4415d2d0 2024-03-19T01:52:07.716 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.713+0000 7fcb49240640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fcb4415bad0 con 0x7fcb4415bf60 2024-03-19T01:52:07.717 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.717+0000 7fcb4259c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fcb440ceb10 0x7fcb4406c3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:38640/0 (socket says 172.21.15.55:38640) 2024-03-19T01:52:07.717 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.717+0000 7fcb4259c640 1 -- 172.21.15.55:0/76091686 learned_addr learned my addr 172.21.15.55:0/76091686 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:52:07.717 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.717+0000 7fcb42d9d640 1 --2- 172.21.15.55:0/76091686 >> v2:172.21.15.84:3300/0 conn(0x7fcb4415d2d0 0x7fcb4415aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:07.717 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.717+0000 7fcb4359e640 1 --2- 172.21.15.55:0/76091686 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcb4415bf60 0x7fcb4415c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:07.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.717+0000 7fcb41d9b640 1 -- 172.21.15.55:0/76091686 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2001960380 0 0) 0x7fcb4415c910 con 0x7fcb440ceb10 2024-03-19T01:52:07.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.717+0000 7fcb41d9b640 1 -- 172.21.15.55:0/76091686 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb2c003530 con 0x7fcb440ceb10 2024-03-19T01:52:07.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.721+0000 7fcb42d9d640 1 -- 172.21.15.55:0/76091686 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcb4415bf60 msgr2=0x7fcb4415c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:07.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.721+0000 7fcb42d9d640 1 --2- 172.21.15.55:0/76091686 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcb4415bf60 0x7fcb4415c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:07.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.721+0000 7fcb42d9d640 1 -- 172.21.15.55:0/76091686 >> v1:172.21.15.55:6789/0 conn(0x7fcb440ceb10 legacy=0x7fcb4406c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:07.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.721+0000 7fcb42d9d640 1 -- 172.21.15.55:0/76091686 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb440cdde0 con 0x7fcb4415d2d0 2024-03-19T01:52:07.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.725+0000 7fcb42d9d640 1 --2- 172.21.15.55:0/76091686 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcb4415d2d0 0x7fcb4415aeb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcb340097c0 tx=0x7fcb3401ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb54707bdbfbcbc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:07.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.729+0000 7fcb41d9b640 1 -- 172.21.15.55:0/76091686 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb340b5070 con 0x7fcb4415d2d0 2024-03-19T01:52:07.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.729+0000 7fcb41d9b640 1 -- 172.21.15.55:0/76091686 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb340a8990 con 0x7fcb4415d2d0 2024-03-19T01:52:07.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.729+0000 7fcb41d9b640 1 -- 172.21.15.55:0/76091686 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb340a8d60 con 0x7fcb4415d2d0 2024-03-19T01:52:07.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.729+0000 7fcb49240640 1 -- 172.21.15.55:0/76091686 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcb4415d2d0 msgr2=0x7fcb4415aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:07.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.729+0000 7fcb49240640 1 --2- 172.21.15.55:0/76091686 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcb4415d2d0 0x7fcb4415aeb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcb340097c0 tx=0x7fcb3401ec90 comp rx=0 tx=0).stop 2024-03-19T01:52:07.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.729+0000 7fcb49240640 1 -- 172.21.15.55:0/76091686 shutdown_connections 2024-03-19T01:52:07.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.729+0000 7fcb49240640 1 --2- 172.21.15.55:0/76091686 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcb4415bf60 0x7fcb4415c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:07.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.729+0000 7fcb49240640 1 --2- 172.21.15.55:0/76091686 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcb4415d2d0 0x7fcb4415aeb0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:07.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.729+0000 7fcb49240640 1 -- 172.21.15.55:0/76091686 >> 172.21.15.55:0/76091686 conn(0x7fcb440cfd90 msgr2=0x7fcb440d01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:07.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.729+0000 7fcb49240640 1 -- 172.21.15.55:0/76091686 shutdown_connections 2024-03-19T01:52:07.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.729+0000 7fcb49240640 1 -- 172.21.15.55:0/76091686 wait complete. 2024-03-19T01:52:07.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.729+0000 7fcb49240640 1 Processor -- start 2024-03-19T01:52:07.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.729+0000 7fcb49240640 1 -- start start 2024-03-19T01:52:07.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.733+0000 7fcb49240640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcb440ceb10 0x7fcb440c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:07.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.733+0000 7fcb49240640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcb4415bf60 0x7fcb441539c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:07.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.733+0000 7fcb49240640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcb4415d2d0 0x7fcb441f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:07.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.733+0000 7fcb49240640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcb440ce680 con 0x7fcb440ceb10 2024-03-19T01:52:07.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.733+0000 7fcb49240640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fcb440ce3a0 con 0x7fcb4415bf60 2024-03-19T01:52:07.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.733+0000 7fcb49240640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fcb4415d6b0 con 0x7fcb4415d2d0 2024-03-19T01:52:07.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.733+0000 7fcb42d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcb440ceb10 0x7fcb440c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:07.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.733+0000 7fcb4359e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcb4415d2d0 0x7fcb441f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:07.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.733+0000 7fcb42d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcb440ceb10 0x7fcb440c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53348/0 (socket says 172.21.15.55:53348) 2024-03-19T01:52:07.735 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.733+0000 7fcb42d9d640 1 -- 172.21.15.55:0/2348325152 learned_addr learned my addr 172.21.15.55:0/2348325152 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:07.735 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.733+0000 7fcb4259c640 1 --2- 172.21.15.55:0/2348325152 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcb4415bf60 0x7fcb441539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:07.737 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.737+0000 7fcb42d9d640 1 -- 172.21.15.55:0/2348325152 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcb4415d2d0 msgr2=0x7fcb441f3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:07.737 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.737+0000 7fcb42d9d640 1 --2- 172.21.15.55:0/2348325152 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcb4415d2d0 0x7fcb441f3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:07.737 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.737+0000 7fcb42d9d640 1 -- 172.21.15.55:0/2348325152 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcb4415bf60 msgr2=0x7fcb441539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:07.737 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.737+0000 7fcb42d9d640 1 --2- 172.21.15.55:0/2348325152 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcb4415bf60 0x7fcb441539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:07.737 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.737+0000 7fcb42d9d640 1 -- 172.21.15.55:0/2348325152 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb44068540 con 0x7fcb440ceb10 2024-03-19T01:52:07.739 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.737+0000 7fcb42d9d640 1 --2- 172.21.15.55:0/2348325152 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcb440ceb10 0x7fcb440c5a80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcb3401c8c0 tx=0x7fcb3401ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:07.743 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.741+0000 7fcb2b7fe640 1 -- 172.21.15.55:0/2348325152 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb340bf070 con 0x7fcb440ceb10 2024-03-19T01:52:07.743 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.741+0000 7fcb2b7fe640 1 -- 172.21.15.55:0/2348325152 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb340b0660 con 0x7fcb440ceb10 2024-03-19T01:52:07.743 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.741+0000 7fcb2b7fe640 1 -- 172.21.15.55:0/2348325152 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb340b5040 con 0x7fcb440ceb10 2024-03-19T01:52:07.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.745+0000 7fcb49240640 1 -- 172.21.15.55:0/2348325152 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb440c86a0 con 0x7fcb440ceb10 2024-03-19T01:52:07.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.745+0000 7fcb49240640 1 -- 172.21.15.55:0/2348325152 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb440c6780 con 0x7fcb440ceb10 2024-03-19T01:52:07.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.745+0000 7fcb49240640 1 -- 172.21.15.55:0/2348325152 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb44068a40 con 0x7fcb440ceb10 2024-03-19T01:52:07.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.765+0000 7fcb2b7fe640 1 -- 172.21.15.55:0/2348325152 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fcb340c2070 con 0x7fcb440ceb10 2024-03-19T01:52:07.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.765+0000 7fcb2b7fe640 1 --2- 172.21.15.55:0/2348325152 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcb300676b0 0x7fcb30069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:07.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.765+0000 7fcb4259c640 1 --2- 172.21.15.55:0/2348325152 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcb300676b0 0x7fcb30069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:07.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:07.765+0000 7fcb4259c640 1 --2- 172.21.15.55:0/2348325152 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcb300676b0 0x7fcb30069b70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcb38007fd0 tx=0x7fcb38009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:08.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.045+0000 7fcb2b7fe640 1 -- 172.21.15.55:0/2348325152 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fcb340f0940 con 0x7fcb440ceb10 2024-03-19T01:52:08.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.045+0000 7fcb2b7fe640 1 -- 172.21.15.55:0/2348325152 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fcb340c9080 con 0x7fcb440ceb10 2024-03-19T01:52:08.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.225+0000 7fcb49240640 1 -- 172.21.15.55:0/2348325152 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcb44154820 con 0x7fcb440ceb10 2024-03-19T01:52:08.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.241+0000 7fcb2b7fe640 1 -- 172.21.15.55:0/2348325152 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) ==== 100+0+11 (secure 0 0 0) 0x7fcb340187f0 con 0x7fcb440ceb10 2024-03-19T01:52:08.242 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 64 2024-03-19T01:52:08.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.241+0000 7fcb49240640 1 -- 172.21.15.55:0/2348325152 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcb300676b0 msgr2=0x7fcb30069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:08.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.241+0000 7fcb49240640 1 --2- 172.21.15.55:0/2348325152 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcb300676b0 0x7fcb30069b70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcb38007fd0 tx=0x7fcb38009040 comp rx=0 tx=0).stop 2024-03-19T01:52:08.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.241+0000 7fcb49240640 1 -- 172.21.15.55:0/2348325152 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcb440ceb10 msgr2=0x7fcb440c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:08.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.241+0000 7fcb49240640 1 --2- 172.21.15.55:0/2348325152 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcb440ceb10 0x7fcb440c5a80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcb3401c8c0 tx=0x7fcb3401ba30 comp rx=0 tx=0).stop 2024-03-19T01:52:08.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.241+0000 7fcb49240640 1 -- 172.21.15.55:0/2348325152 shutdown_connections 2024-03-19T01:52:08.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.241+0000 7fcb49240640 1 --2- 172.21.15.55:0/2348325152 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcb300676b0 0x7fcb30069b70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:08.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.241+0000 7fcb49240640 1 --2- 172.21.15.55:0/2348325152 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcb4415d2d0 0x7fcb441f3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:08.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.241+0000 7fcb49240640 1 --2- 172.21.15.55:0/2348325152 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcb4415bf60 0x7fcb441539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:08.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.241+0000 7fcb49240640 1 --2- 172.21.15.55:0/2348325152 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcb440ceb10 0x7fcb440c5a80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:08.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.241+0000 7fcb49240640 1 -- 172.21.15.55:0/2348325152 >> 172.21.15.55:0/2348325152 conn(0x7fcb440cfd90 msgr2=0x7fcb440cbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:08.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.245+0000 7fcb49240640 1 -- 172.21.15.55:0/2348325152 shutdown_connections 2024-03-19T01:52:08.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.245+0000 7fcb49240640 1 -- 172.21.15.55:0/2348325152 wait complete. 2024-03-19T01:52:08.261 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T01:52:08.388 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.385+0000 7fbd90c4c640 1 Processor -- start 2024-03-19T01:52:08.388 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.385+0000 7fbd90c4c640 1 -- start start 2024-03-19T01:52:08.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.389+0000 7fbd90c4c640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fbd8c072c60 0x7fbd8c158a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:08.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.389+0000 7fbd90c4c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd8c15b540 0x7fbd8c0c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:08.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.389+0000 7fbd90c4c640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd8c152660 con 0x7fbd8c0c39c0 2024-03-19T01:52:08.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.389+0000 7fbd90c4c640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fbd8c15b990 con 0x7fbd8c072c60 2024-03-19T01:52:08.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.389+0000 7fbd90c4c640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbd8c073130 con 0x7fbd8c15b540 2024-03-19T01:52:08.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.389+0000 7fbd8ad9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd8c15b540 0x7fbd8c0c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:08.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.389+0000 7fbd89d9b640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fbd8c072c60 0x7fbd8c158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:08.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.389+0000 7fbd8ad9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd8c15b540 0x7fbd8c0c60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:43016/0 (socket says 172.21.15.55:43016) 2024-03-19T01:52:08.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.389+0000 7fbd8ad9d640 1 -- 172.21.15.55:0/2676363390 learned_addr learned my addr 172.21.15.55:0/2676363390 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:08.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.393+0000 7fbd8ad9d640 1 -- 172.21.15.55:0/2676363390 >> v2:172.21.15.84:3300/0 conn(0x7fbd8c072c60 msgr2=0x7fbd8c158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:08.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.393+0000 7fbd8ad9d640 1 --2- 172.21.15.55:0/2676363390 >> v2:172.21.15.84:3300/0 conn(0x7fbd8c072c60 0x7fbd8c158a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:08.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.393+0000 7fbd8ad9d640 1 -- 172.21.15.55:0/2676363390 >> v1:172.21.15.84:6789/0 conn(0x7fbd8c0c39c0 legacy=0x7fbd8c06b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:08.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.393+0000 7fbd8ad9d640 1 -- 172.21.15.55:0/2676363390 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd8c0733f0 con 0x7fbd8c15b540 2024-03-19T01:52:08.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.397+0000 7fbd8ad9d640 1 --2- 172.21.15.55:0/2676363390 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd8c15b540 0x7fbd8c0c60d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbd84009e50 tx=0x7fbd8401f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ec1793344f3791f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:08.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.405+0000 7fbd8959a640 1 -- 172.21.15.55:0/2676363390 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd840a8bd0 con 0x7fbd8c15b540 2024-03-19T01:52:08.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.405+0000 7fbd8959a640 1 -- 172.21.15.55:0/2676363390 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd840a8d70 con 0x7fbd8c15b540 2024-03-19T01:52:08.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.405+0000 7fbd8959a640 1 -- 172.21.15.55:0/2676363390 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd840b0610 con 0x7fbd8c15b540 2024-03-19T01:52:08.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.405+0000 7fbd90c4c640 1 -- 172.21.15.55:0/2676363390 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd8c15b540 msgr2=0x7fbd8c0c60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:08.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.405+0000 7fbd90c4c640 1 --2- 172.21.15.55:0/2676363390 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd8c15b540 0x7fbd8c0c60d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbd84009e50 tx=0x7fbd8401f000 comp rx=0 tx=0).stop 2024-03-19T01:52:08.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.405+0000 7fbd90c4c640 1 -- 172.21.15.55:0/2676363390 shutdown_connections 2024-03-19T01:52:08.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.405+0000 7fbd90c4c640 1 --2- 172.21.15.55:0/2676363390 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd8c15b540 0x7fbd8c0c60d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:08.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.405+0000 7fbd90c4c640 1 --2- 172.21.15.55:0/2676363390 >> v2:172.21.15.84:3300/0 conn(0x7fbd8c072c60 0x7fbd8c158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:08.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.405+0000 7fbd90c4c640 1 -- 172.21.15.55:0/2676363390 >> 172.21.15.55:0/2676363390 conn(0x7fbd8c0cc9c0 msgr2=0x7fbd8c0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:08.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.405+0000 7fbd90c4c640 1 -- 172.21.15.55:0/2676363390 shutdown_connections 2024-03-19T01:52:08.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.405+0000 7fbd90c4c640 1 -- 172.21.15.55:0/2676363390 wait complete. 2024-03-19T01:52:08.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.409+0000 7fbd90c4c640 1 Processor -- start 2024-03-19T01:52:08.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.409+0000 7fbd90c4c640 1 -- start start 2024-03-19T01:52:08.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.409+0000 7fbd90c4c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd8c072c60 0x7fbd8c0c58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:08.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.409+0000 7fbd90c4c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd8c0c39c0 0x7fbd8c15ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:08.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.409+0000 7fbd90c4c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd8c15b540 0x7fbd8c0c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:08.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.409+0000 7fbd90c4c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8c158560 con 0x7fbd8c072c60 2024-03-19T01:52:08.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.409+0000 7fbd90c4c640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8c158280 con 0x7fbd8c0c39c0 2024-03-19T01:52:08.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.409+0000 7fbd90c4c640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbd8c157f50 con 0x7fbd8c15b540 2024-03-19T01:52:08.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.409+0000 7fbd8a59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd8c072c60 0x7fbd8c0c58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:08.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.409+0000 7fbd8a59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd8c072c60 0x7fbd8c0c58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53352/0 (socket says 172.21.15.55:53352) 2024-03-19T01:52:08.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.409+0000 7fbd8a59c640 1 -- 172.21.15.55:0/741151159 learned_addr learned my addr 172.21.15.55:0/741151159 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:08.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.409+0000 7fbd89d9b640 1 --2- 172.21.15.55:0/741151159 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd8c0c39c0 0x7fbd8c15ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:08.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.409+0000 7fbd8ad9d640 1 --2- 172.21.15.55:0/741151159 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd8c15b540 0x7fbd8c0c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:08.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.413+0000 7fbd8a59c640 1 -- 172.21.15.55:0/741151159 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd8c15b540 msgr2=0x7fbd8c0c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:08.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.413+0000 7fbd8a59c640 1 --2- 172.21.15.55:0/741151159 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd8c15b540 0x7fbd8c0c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:08.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.413+0000 7fbd8a59c640 1 -- 172.21.15.55:0/741151159 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd8c0c39c0 msgr2=0x7fbd8c15ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:08.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.413+0000 7fbd8a59c640 1 --2- 172.21.15.55:0/741151159 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd8c0c39c0 0x7fbd8c15ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:08.415 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.413+0000 7fbd8a59c640 1 -- 172.21.15.55:0/741151159 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd8c15a6e0 con 0x7fbd8c072c60 2024-03-19T01:52:08.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.417+0000 7fbd8a59c640 1 --2- 172.21.15.55:0/741151159 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd8c072c60 0x7fbd8c0c58a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c00c490 tx=0x7fbd7c001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:08.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.417+0000 7fbd7affd640 1 -- 172.21.15.55:0/741151159 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c017070 con 0x7fbd8c072c60 2024-03-19T01:52:08.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.417+0000 7fbd90c4c640 1 -- 172.21.15.55:0/741151159 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd8c0c26d0 con 0x7fbd8c072c60 2024-03-19T01:52:08.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.417+0000 7fbd90c4c640 1 -- 172.21.15.55:0/741151159 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd8c0c2c90 con 0x7fbd8c072c60 2024-03-19T01:52:08.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.421+0000 7fbd90c4c640 1 -- 172.21.15.55:0/741151159 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd8c155870 con 0x7fbd8c072c60 2024-03-19T01:52:08.422 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.421+0000 7fbd7affd640 1 -- 172.21.15.55:0/741151159 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd7c007030 con 0x7fbd8c072c60 2024-03-19T01:52:08.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.421+0000 7fbd7affd640 1 -- 172.21.15.55:0/741151159 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c021060 con 0x7fbd8c072c60 2024-03-19T01:52:08.539 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:52:08.541 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:52:08.717 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.717+0000 7fbd7affd640 1 -- 172.21.15.55:0/741151159 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fbd7c01a070 con 0x7fbd8c072c60 2024-03-19T01:52:08.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.717+0000 7fbd7affd640 1 --2- 172.21.15.55:0/741151159 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbd6c067660 0x7fbd6c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:08.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.717+0000 7fbd89d9b640 1 --2- 172.21.15.55:0/741151159 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbd6c067660 0x7fbd6c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:08.722 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.717+0000 7fbd89d9b640 1 --2- 172.21.15.55:0/741151159 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbd6c067660 0x7fbd6c069b20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbd80009ac0 tx=0x7fbd80009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:08.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.785+0000 7fbd7affd640 1 -- 172.21.15.55:0/741151159 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fbd7c0010d0 con 0x7fbd8c072c60 2024-03-19T01:52:08.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.785+0000 7fbd7affd640 1 -- 172.21.15.55:0/741151159 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fbd8c15a6e0 con 0x7fbd8c072c60 2024-03-19T01:52:08.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.965+0000 7fbd90c4c640 1 -- 172.21.15.55:0/741151159 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fbd8c13aae0 con 0x7fbd8c072c60 2024-03-19T01:52:08.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.985+0000 7fbd7affd640 1 -- 172.21.15.55:0/741151159 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) ==== 103+0+11 (secure 0 0 0) 0x7fbd7c0556a0 con 0x7fbd8c072c60 2024-03-19T01:52:08.985 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 32 2024-03-19T01:52:08.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.985+0000 7fbd90c4c640 1 -- 172.21.15.55:0/741151159 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbd6c067660 msgr2=0x7fbd6c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:08.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.985+0000 7fbd90c4c640 1 --2- 172.21.15.55:0/741151159 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbd6c067660 0x7fbd6c069b20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbd80009ac0 tx=0x7fbd80009190 comp rx=0 tx=0).stop 2024-03-19T01:52:08.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.985+0000 7fbd90c4c640 1 -- 172.21.15.55:0/741151159 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd8c072c60 msgr2=0x7fbd8c0c58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:08.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.985+0000 7fbd90c4c640 1 --2- 172.21.15.55:0/741151159 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd8c072c60 0x7fbd8c0c58a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c00c490 tx=0x7fbd7c001e80 comp rx=0 tx=0).stop 2024-03-19T01:52:08.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.985+0000 7fbd90c4c640 1 -- 172.21.15.55:0/741151159 shutdown_connections 2024-03-19T01:52:08.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.985+0000 7fbd90c4c640 1 --2- 172.21.15.55:0/741151159 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbd6c067660 0x7fbd6c069b20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:08.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.985+0000 7fbd90c4c640 1 --2- 172.21.15.55:0/741151159 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd8c15b540 0x7fbd8c0c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:08.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.985+0000 7fbd90c4c640 1 --2- 172.21.15.55:0/741151159 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd8c0c39c0 0x7fbd8c15ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:08.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.985+0000 7fbd90c4c640 1 --2- 172.21.15.55:0/741151159 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd8c072c60 0x7fbd8c0c58a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:08.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.985+0000 7fbd90c4c640 1 -- 172.21.15.55:0/741151159 >> 172.21.15.55:0/741151159 conn(0x7fbd8c0cc9c0 msgr2=0x7fbd8c06b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:08.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.985+0000 7fbd90c4c640 1 -- 172.21.15.55:0/741151159 shutdown_connections 2024-03-19T01:52:08.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:08.985+0000 7fbd90c4c640 1 -- 172.21.15.55:0/741151159 wait complete. 2024-03-19T01:52:09.008 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:52:09.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.121+0000 7fa8d90ae640 1 Processor -- start 2024-03-19T01:52:09.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.121+0000 7fa8d90ae640 1 -- start start 2024-03-19T01:52:09.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.121+0000 7fa8d90ae640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa8d41544f0 0x7fa8d4073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:09.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.121+0000 7fa8d90ae640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa8d406b2a0 0x7fa8d415e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:09.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.121+0000 7fa8d90ae640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8d40c7fa0 con 0x7fa8d415acb0 2024-03-19T01:52:09.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.121+0000 7fa8d90ae640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fa8d415b210 con 0x7fa8d406b2a0 2024-03-19T01:52:09.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.121+0000 7fa8d90ae640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa8d40c8240 con 0x7fa8d41544f0 2024-03-19T01:52:09.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.137+0000 7fa8d259c640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa8d406b2a0 0x7fa8d415e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:09.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.141+0000 7fa8d259c640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa8d406b2a0 0x7fa8d415e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:38578/0 (socket says 172.21.15.55:38578) 2024-03-19T01:52:09.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.141+0000 7fa8d259c640 1 -- 172.21.15.55:0/53076326 learned_addr learned my addr 172.21.15.55:0/53076326 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:09.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.145+0000 7fa8d259c640 1 -- 172.21.15.55:0/53076326 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa8d41544f0 msgr2=0x7fa8d4073540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:09.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.145+0000 7fa8d259c640 1 --2- 172.21.15.55:0/53076326 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa8d41544f0 0x7fa8d4073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:09.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.145+0000 7fa8d259c640 1 -- 172.21.15.55:0/53076326 >> v1:172.21.15.84:6789/0 conn(0x7fa8d415acb0 legacy=0x7fa8d40cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:09.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.145+0000 7fa8d259c640 1 -- 172.21.15.55:0/53076326 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8d415d920 con 0x7fa8d406b2a0 2024-03-19T01:52:09.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.149+0000 7fa8d259c640 1 --2- 172.21.15.55:0/53076326 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8d406b2a0 0x7fa8d415e120 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa8d40ca7c0 tx=0x7fa8c801ecf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a12c2028ab38b7b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:09.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.153+0000 7fa8d1d9b640 1 -- 172.21.15.55:0/53076326 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c80b5070 con 0x7fa8d406b2a0 2024-03-19T01:52:09.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.153+0000 7fa8d1d9b640 1 -- 172.21.15.55:0/53076326 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8c80a89b0 con 0x7fa8d406b2a0 2024-03-19T01:52:09.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.153+0000 7fa8d1d9b640 1 -- 172.21.15.55:0/53076326 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c80a8d80 con 0x7fa8d406b2a0 2024-03-19T01:52:09.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.157+0000 7fa8d90ae640 1 -- 172.21.15.55:0/53076326 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8d406b2a0 msgr2=0x7fa8d415e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:09.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.157+0000 7fa8d90ae640 1 --2- 172.21.15.55:0/53076326 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8d406b2a0 0x7fa8d415e120 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa8d40ca7c0 tx=0x7fa8c801ecf0 comp rx=0 tx=0).stop 2024-03-19T01:52:09.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.157+0000 7fa8d90ae640 1 -- 172.21.15.55:0/53076326 shutdown_connections 2024-03-19T01:52:09.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.157+0000 7fa8d90ae640 1 --2- 172.21.15.55:0/53076326 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8d406b2a0 0x7fa8d415e120 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:09.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.157+0000 7fa8d90ae640 1 --2- 172.21.15.55:0/53076326 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa8d41544f0 0x7fa8d4073540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:09.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.157+0000 7fa8d90ae640 1 -- 172.21.15.55:0/53076326 >> 172.21.15.55:0/53076326 conn(0x7fa8d406e670 msgr2=0x7fa8d406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:09.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.157+0000 7fa8d90ae640 1 -- 172.21.15.55:0/53076326 shutdown_connections 2024-03-19T01:52:09.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.157+0000 7fa8d90ae640 1 -- 172.21.15.55:0/53076326 wait complete. 2024-03-19T01:52:09.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.157+0000 7fa8d90ae640 1 Processor -- start 2024-03-19T01:52:09.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.157+0000 7fa8d90ae640 1 -- start start 2024-03-19T01:52:09.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.157+0000 7fa8d90ae640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8d406b2a0 0x7fa8d41f0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:09.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.157+0000 7fa8d90ae640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8d41544f0 0x7fa8d40c9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:09.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.157+0000 7fa8d90ae640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa8d415acb0 0x7fa8d40c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:09.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.157+0000 7fa8d90ae640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d41548d0 con 0x7fa8d41544f0 2024-03-19T01:52:09.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.157+0000 7fa8d90ae640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d415e660 con 0x7fa8d406b2a0 2024-03-19T01:52:09.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.157+0000 7fa8d90ae640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa8d40cd540 con 0x7fa8d415acb0 2024-03-19T01:52:09.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.157+0000 7fa8d2d9d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8d406b2a0 0x7fa8d41f0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:09.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.157+0000 7fa8d259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8d41544f0 0x7fa8d40c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:09.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.161+0000 7fa8d259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8d41544f0 0x7fa8d40c9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:32948/0 (socket says 172.21.15.55:32948) 2024-03-19T01:52:09.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.161+0000 7fa8d259c640 1 -- 172.21.15.55:0/1878031763 learned_addr learned my addr 172.21.15.55:0/1878031763 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:09.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.165+0000 7fa8d359e640 1 --2- 172.21.15.55:0/1878031763 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa8d415acb0 0x7fa8d40c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:09.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.165+0000 7fa8d259c640 1 -- 172.21.15.55:0/1878031763 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa8d415acb0 msgr2=0x7fa8d40c9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:09.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.165+0000 7fa8d259c640 1 --2- 172.21.15.55:0/1878031763 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa8d415acb0 0x7fa8d40c9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:09.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.165+0000 7fa8d259c640 1 -- 172.21.15.55:0/1878031763 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8d406b2a0 msgr2=0x7fa8d41f0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:09.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.165+0000 7fa8d259c640 1 --2- 172.21.15.55:0/1878031763 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8d406b2a0 0x7fa8d41f0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:09.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.165+0000 7fa8d259c640 1 -- 172.21.15.55:0/1878031763 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8d41f0290 con 0x7fa8d41544f0 2024-03-19T01:52:09.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.165+0000 7fa8d359e640 1 --2- 172.21.15.55:0/1878031763 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa8d415acb0 0x7fa8d40c9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:52:09.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.173+0000 7fa8d259c640 1 --2- 172.21.15.55:0/1878031763 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8d41544f0 0x7fa8d40c9960 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa8c801c920 tx=0x7fa8c801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:09.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.177+0000 7fa8bb7fe640 1 -- 172.21.15.55:0/1878031763 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c80c0070 con 0x7fa8d41544f0 2024-03-19T01:52:09.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.177+0000 7fa8bb7fe640 1 -- 172.21.15.55:0/1878031763 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8c80b1790 con 0x7fa8d41544f0 2024-03-19T01:52:09.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.177+0000 7fa8bb7fe640 1 -- 172.21.15.55:0/1878031763 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c80b5040 con 0x7fa8d41544f0 2024-03-19T01:52:09.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.177+0000 7fa8d90ae640 1 -- 172.21.15.55:0/1878031763 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8d4154fc0 con 0x7fa8d41544f0 2024-03-19T01:52:09.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.177+0000 7fa8d90ae640 1 -- 172.21.15.55:0/1878031763 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8d4155580 con 0x7fa8d41544f0 2024-03-19T01:52:09.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.177+0000 7fa8d90ae640 1 -- 172.21.15.55:0/1878031763 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8d4068a40 con 0x7fa8d41544f0 2024-03-19T01:52:09.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.457+0000 7fa8bb7fe640 1 -- 172.21.15.55:0/1878031763 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa8c80c3020 con 0x7fa8d41544f0 2024-03-19T01:52:09.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.461+0000 7fa8bb7fe640 1 --2- 172.21.15.55:0/1878031763 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa8c00676b0 0x7fa8c0069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:09.461 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.461+0000 7fa8d2d9d640 1 --2- 172.21.15.55:0/1878031763 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa8c00676b0 0x7fa8c0069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:09.461 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.461+0000 7fa8d2d9d640 1 --2- 172.21.15.55:0/1878031763 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa8c00676b0 0x7fa8c0069b70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4009990 tx=0x7fa8c4009110 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:09.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.485+0000 7fa8bb7fe640 1 -- 172.21.15.55:0/1878031763 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7fa8c80f0f50 con 0x7fa8d41544f0 2024-03-19T01:52:09.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.485+0000 7fa8bb7fe640 1 -- 172.21.15.55:0/1878031763 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa8c8018850 con 0x7fa8d41544f0 2024-03-19T01:52:09.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.685+0000 7fa8d90ae640 1 -- 172.21.15.55:0/1878031763 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8d4072a90 con 0x7fa8d41544f0 2024-03-19T01:52:09.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.705+0000 7fa8bb7fe640 1 -- 172.21.15.55:0/1878031763 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6909 (secure 0 0 0) 0x7fa8c80f2eb0 con 0x7fa8d41544f0 2024-03-19T01:52:09.706 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:52:09.707 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4200,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:52:05.800317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4249,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4249":{"gid":4249,"name":"f","rank":1,"incarnation":11,"state":"up:active","state_seq":20,"addr":"172.21.15.84:6839/354880008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6838","nonce":354880008},{"type":"v1","addr":"172.21.15.84:6839","nonce":354880008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4249,4237,4255]},"id":1}]} 2024-03-19T01:52:09.707 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 20 2024-03-19T01:52:09.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.709+0000 7fa8d90ae640 1 -- 172.21.15.55:0/1878031763 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa8c00676b0 msgr2=0x7fa8c0069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:09.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.709+0000 7fa8d90ae640 1 --2- 172.21.15.55:0/1878031763 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa8c00676b0 0x7fa8c0069b70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4009990 tx=0x7fa8c4009110 comp rx=0 tx=0).stop 2024-03-19T01:52:09.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.709+0000 7fa8d90ae640 1 -- 172.21.15.55:0/1878031763 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8d41544f0 msgr2=0x7fa8d40c9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:09.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.709+0000 7fa8d90ae640 1 --2- 172.21.15.55:0/1878031763 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8d41544f0 0x7fa8d40c9960 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa8c801c920 tx=0x7fa8c801ba90 comp rx=0 tx=0).stop 2024-03-19T01:52:09.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.709+0000 7fa8d90ae640 1 -- 172.21.15.55:0/1878031763 shutdown_connections 2024-03-19T01:52:09.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.709+0000 7fa8d90ae640 1 --2- 172.21.15.55:0/1878031763 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa8c00676b0 0x7fa8c0069b70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:09.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.709+0000 7fa8d90ae640 1 --2- 172.21.15.55:0/1878031763 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa8d415acb0 0x7fa8d40c9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:09.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.709+0000 7fa8d90ae640 1 --2- 172.21.15.55:0/1878031763 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8d41544f0 0x7fa8d40c9960 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:09.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.709+0000 7fa8d90ae640 1 --2- 172.21.15.55:0/1878031763 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8d406b2a0 0x7fa8d41f0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:09.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.709+0000 7fa8d90ae640 1 -- 172.21.15.55:0/1878031763 >> 172.21.15.55:0/1878031763 conn(0x7fa8d406e670 msgr2=0x7fa8d40c8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:09.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.709+0000 7fa8d90ae640 1 -- 172.21.15.55:0/1878031763 shutdown_connections 2024-03-19T01:52:09.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.709+0000 7fa8d90ae640 1 -- 172.21.15.55:0/1878031763 wait complete. 2024-03-19T01:52:09.725 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:52:09.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.833+0000 7f0fb7725640 1 Processor -- start 2024-03-19T01:52:09.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.833+0000 7f0fb7725640 1 -- start start 2024-03-19T01:52:09.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.833+0000 7f0fb7725640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0fb015e2e0 0x7f0fb015a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:09.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.833+0000 7f0fb7725640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f0fb00d0210 0x7f0fb0164b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:09.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.833+0000 7f0fb7725640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fb015d870 con 0x7f0fb00cfe30 2024-03-19T01:52:09.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.833+0000 7f0fb7725640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f0fb015e740 con 0x7f0fb00d0210 2024-03-19T01:52:09.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.833+0000 7f0fb7725640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f0fb00ccf80 con 0x7f0fb015e2e0 2024-03-19T01:52:09.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.837+0000 7f0fb4cc0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0fb015e2e0 0x7f0fb015a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:09.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.837+0000 7f0fb54c1640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f0fb00cfe30 0x7f0fb00cc870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:58418/0 (socket says 172.21.15.55:58418) 2024-03-19T01:52:09.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.837+0000 7f0fb54c1640 1 -- 172.21.15.55:0/4262004578 learned_addr learned my addr 172.21.15.55:0/4262004578 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:52:09.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.837+0000 7f0fb5cc2640 1 --2- 172.21.15.55:0/4262004578 >> v2:172.21.15.84:3300/0 conn(0x7f0fb00d0210 0x7f0fb0164b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:09.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.837+0000 7f0fa7fff640 1 -- 172.21.15.55:0/4262004578 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1984537946 0 0) 0x7f0fb015d870 con 0x7f0fb00cfe30 2024-03-19T01:52:09.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.837+0000 7f0fa7fff640 1 -- 172.21.15.55:0/4262004578 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f9c003530 con 0x7f0fb00cfe30 2024-03-19T01:52:09.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.841+0000 7f0fb4cc0640 1 -- 172.21.15.55:0/4262004578 >> v2:172.21.15.84:3300/0 conn(0x7f0fb00d0210 msgr2=0x7f0fb0164b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:09.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.841+0000 7f0fb4cc0640 1 --2- 172.21.15.55:0/4262004578 >> v2:172.21.15.84:3300/0 conn(0x7f0fb00d0210 0x7f0fb0164b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:09.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.841+0000 7f0fb4cc0640 1 -- 172.21.15.55:0/4262004578 >> v1:172.21.15.55:6789/0 conn(0x7f0fb00cfe30 legacy=0x7f0fb00cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:09.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.841+0000 7f0fb4cc0640 1 -- 172.21.15.55:0/4262004578 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb00cf170 con 0x7f0fb015e2e0 2024-03-19T01:52:09.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.845+0000 7f0fb4cc0640 1 --2- 172.21.15.55:0/4262004578 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0fb015e2e0 0x7f0fb015a260 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0fa00097c0 tx=0x7f0fa001c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29fb31062cb2ec18 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:09.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.853+0000 7f0fa7fff640 1 -- 172.21.15.55:0/4262004578 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa00b0070 con 0x7f0fb015e2e0 2024-03-19T01:52:09.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.853+0000 7f0fa7fff640 1 -- 172.21.15.55:0/4262004578 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fa00ab600 con 0x7f0fb015e2e0 2024-03-19T01:52:09.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.853+0000 7f0fa7fff640 1 -- 172.21.15.55:0/4262004578 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa00ab9d0 con 0x7f0fb015e2e0 2024-03-19T01:52:09.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.853+0000 7f0fb7725640 1 -- 172.21.15.55:0/4262004578 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0fb015e2e0 msgr2=0x7f0fb015a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:09.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.853+0000 7f0fb7725640 1 --2- 172.21.15.55:0/4262004578 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0fb015e2e0 0x7f0fb015a260 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0fa00097c0 tx=0x7f0fa001c410 comp rx=0 tx=0).stop 2024-03-19T01:52:09.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.853+0000 7f0fb7725640 1 -- 172.21.15.55:0/4262004578 shutdown_connections 2024-03-19T01:52:09.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.853+0000 7f0fb7725640 1 --2- 172.21.15.55:0/4262004578 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0fb015e2e0 0x7f0fb015a260 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:09.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.853+0000 7f0fb7725640 1 --2- 172.21.15.55:0/4262004578 >> v2:172.21.15.84:3300/0 conn(0x7f0fb00d0210 0x7f0fb0164b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:09.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.853+0000 7f0fb7725640 1 -- 172.21.15.55:0/4262004578 >> 172.21.15.55:0/4262004578 conn(0x7f0fb00671d0 msgr2=0x7f0fb00675b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:09.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.853+0000 7f0fb7725640 1 -- 172.21.15.55:0/4262004578 shutdown_connections 2024-03-19T01:52:09.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.853+0000 7f0fb7725640 1 -- 172.21.15.55:0/4262004578 wait complete. 2024-03-19T01:52:09.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.853+0000 7f0fb7725640 1 Processor -- start 2024-03-19T01:52:09.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.853+0000 7f0fb7725640 1 -- start start 2024-03-19T01:52:09.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.857+0000 7f0fb7725640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0fb00cfe30 0x7f0fb014ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:09.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.857+0000 7f0fb7725640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fb00d0210 0x7f0fb0153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:09.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.857+0000 7f0fb7725640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0fb015e2e0 0x7f0fb01f61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:09.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.857+0000 7f0fb7725640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb00c9d90 con 0x7f0fb00d0210 2024-03-19T01:52:09.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.857+0000 7f0fb7725640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb00cf730 con 0x7f0fb015e2e0 2024-03-19T01:52:09.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.857+0000 7f0fb7725640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f0fb00c9f10 con 0x7f0fb00cfe30 2024-03-19T01:52:09.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.857+0000 7f0fb4cc0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fb00d0210 0x7f0fb0153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:09.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.857+0000 7f0fb4cc0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fb00d0210 0x7f0fb0153ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:32950/0 (socket says 172.21.15.55:32950) 2024-03-19T01:52:09.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.857+0000 7f0fb4cc0640 1 -- 172.21.15.55:0/192373001 learned_addr learned my addr 172.21.15.55:0/192373001 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:09.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.857+0000 7f0fb5cc2640 1 --2- 172.21.15.55:0/192373001 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0fb015e2e0 0x7f0fb01f61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:09.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.857+0000 7f0fb54c1640 1 --2- 172.21.15.55:0/192373001 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0fb00cfe30 0x7f0fb014ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:09.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.861+0000 7f0fb4cc0640 1 -- 172.21.15.55:0/192373001 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0fb00cfe30 msgr2=0x7f0fb014ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:09.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.861+0000 7f0fb4cc0640 1 --2- 172.21.15.55:0/192373001 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0fb00cfe30 0x7f0fb014ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:09.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.861+0000 7f0fb4cc0640 1 -- 172.21.15.55:0/192373001 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0fb015e2e0 msgr2=0x7f0fb01f61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:09.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.861+0000 7f0fb4cc0640 1 --2- 172.21.15.55:0/192373001 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0fb015e2e0 0x7f0fb01f61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:09.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.861+0000 7f0fb4cc0640 1 -- 172.21.15.55:0/192373001 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb0155660 con 0x7f0fb00d0210 2024-03-19T01:52:09.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.869+0000 7f0fb4cc0640 1 --2- 172.21.15.55:0/192373001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fb00d0210 0x7f0fb0153ff0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0fa001a040 tx=0x7f0fa00191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:09.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.873+0000 7f0fa5ffb640 1 -- 172.21.15.55:0/192373001 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa00ba020 con 0x7f0fb00d0210 2024-03-19T01:52:09.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.873+0000 7f0fa5ffb640 1 -- 172.21.15.55:0/192373001 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fa00abe50 con 0x7f0fb00d0210 2024-03-19T01:52:09.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.873+0000 7f0fb7725640 1 -- 172.21.15.55:0/192373001 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fb01f5cc0 con 0x7f0fb00d0210 2024-03-19T01:52:09.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.873+0000 7f0fa5ffb640 1 -- 172.21.15.55:0/192373001 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa00b0040 con 0x7f0fb00d0210 2024-03-19T01:52:09.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.873+0000 7f0fb7725640 1 -- 172.21.15.55:0/192373001 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fb014cc70 con 0x7f0fb00d0210 2024-03-19T01:52:09.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.873+0000 7f0fb7725640 1 -- 172.21.15.55:0/192373001 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fb0068330 con 0x7f0fb00d0210 2024-03-19T01:52:09.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.893+0000 7f0fa5ffb640 1 -- 172.21.15.55:0/192373001 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f0fa00bd020 con 0x7f0fb00d0210 2024-03-19T01:52:09.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.897+0000 7f0fa5ffb640 1 --2- 172.21.15.55:0/192373001 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0f940676b0 0x7f0f94069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:09.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.897+0000 7f0fb54c1640 1 --2- 172.21.15.55:0/192373001 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0f940676b0 0x7f0f94069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:09.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.897+0000 7f0fb54c1640 1 --2- 172.21.15.55:0/192373001 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0f940676b0 0x7f0f94069b70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0fac007fd0 tx=0x7f0fac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:09.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:09.897+0000 7f0fa5ffb640 1 -- 172.21.15.55:0/192373001 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f0fa00ee410 con 0x7f0fb00d0210 2024-03-19T01:52:10.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.225+0000 7f0fa5ffb640 1 -- 172.21.15.55:0/192373001 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f0fa0016090 con 0x7f0fb00d0210 2024-03-19T01:52:10.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.417+0000 7f0fb7725640 1 -- 172.21.15.55:0/192373001 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0fb00cd1d0 con 0x7f0fb00d0210 2024-03-19T01:52:10.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.485+0000 7f0fa5ffb640 1 -- 172.21.15.55:0/192373001 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+18430 (secure 0 0 0) 0x7f0fa00f0370 con 0x7f0fb00d0210 2024-03-19T01:52:10.488 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:52:10.489 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":29,"fsid":"def9d4e5-0bc7-449d-b893-2490cbd10e25","created":"2024-03-19T01:50:02.482954+0000","modified":"2024-03-19T01:51:17.228291+0000","last_up_change":"2024-03-19T01:50:21.735508+0000","last_in_change":"2024-03-19T01:50:12.168944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:50:23.456904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:51:04.578583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:51:06.871659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T01:51:14.274442+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"35d5e3c5-d089-4f38-8961-98d064188f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6825","nonce":2757675043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6827","nonce":2757675043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6831","nonce":2757675043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6829","nonce":2757675043}]},"public_addr":"172.21.15.55:6825/2757675043","cluster_addr":"172.21.15.55:6827/2757675043","heartbeat_back_addr":"172.21.15.55:6831/2757675043","heartbeat_front_addr":"172.21.15.55:6829/2757675043","state":["exists","up"]},{"osd":1,"uuid":"8e58f791-001a-44d0-a464-ca68a479f70c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6817","nonce":543460007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6819","nonce":543460007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6823","nonce":543460007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6821","nonce":543460007}]},"public_addr":"172.21.15.55:6817/543460007","cluster_addr":"172.21.15.55:6819/543460007","heartbeat_back_addr":"172.21.15.55:6823/543460007","heartbeat_front_addr":"172.21.15.55:6821/543460007","state":["exists","up"]},{"osd":2,"uuid":"d44f38b8-67f4-43de-98a6-f3c520acaf10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6801","nonce":1760253550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6803","nonce":1760253550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6807","nonce":1760253550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6805","nonce":1760253550}]},"public_addr":"172.21.15.55:6801/1760253550","cluster_addr":"172.21.15.55:6803/1760253550","heartbeat_back_addr":"172.21.15.55:6807/1760253550","heartbeat_front_addr":"172.21.15.55:6805/1760253550","state":["exists","up"]},{"osd":3,"uuid":"850a7316-2d74-43f0-8649-b5f8d8f6692e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6809","nonce":2797616448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6811","nonce":2797616448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6815","nonce":2797616448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6813","nonce":2797616448}]},"public_addr":"172.21.15.55:6809/2797616448","cluster_addr":"172.21.15.55:6811/2797616448","heartbeat_back_addr":"172.21.15.55:6815/2797616448","heartbeat_front_addr":"172.21.15.55:6813/2797616448","state":["exists","up"]},{"osd":4,"uuid":"4e6152ae-a77d-4bb6-a855-5782ad5411a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6819","nonce":2938436826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6821","nonce":2938436826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6825","nonce":2938436826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6823","nonce":2938436826}]},"public_addr":"172.21.15.84:6819/2938436826","cluster_addr":"172.21.15.84:6821/2938436826","heartbeat_back_addr":"172.21.15.84:6825/2938436826","heartbeat_front_addr":"172.21.15.84:6823/2938436826","state":["exists","up"]},{"osd":5,"uuid":"1217ceba-dd31-4615-96f1-dfaf57160f3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6803","nonce":460717295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6805","nonce":460717295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6809","nonce":460717295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6807","nonce":460717295}]},"public_addr":"172.21.15.84:6803/460717295","cluster_addr":"172.21.15.84:6805/460717295","heartbeat_back_addr":"172.21.15.84:6809/460717295","heartbeat_front_addr":"172.21.15.84:6807/460717295","state":["exists","up"]},{"osd":6,"uuid":"3d9ed78a-2284-4c28-a24a-70eb812c7836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6827","nonce":416123135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6829","nonce":416123135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6833","nonce":416123135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6831","nonce":416123135}]},"public_addr":"172.21.15.84:6827/416123135","cluster_addr":"172.21.15.84:6829/416123135","heartbeat_back_addr":"172.21.15.84:6833/416123135","heartbeat_front_addr":"172.21.15.84:6831/416123135","state":["exists","up"]},{"osd":7,"uuid":"c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6811","nonce":460083528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6813","nonce":460083528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6817","nonce":460083528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6815","nonce":460083528}]},"public_addr":"172.21.15.84:6811/460083528","cluster_addr":"172.21.15.84:6813/460083528","heartbeat_back_addr":"172.21.15.84:6817/460083528","heartbeat_front_addr":"172.21.15.84:6815/460083528","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.269126+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.121001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.075714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.092461+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.194676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.062833+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.316515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.237111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:52:10.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.489+0000 7f0fb7725640 1 -- 172.21.15.55:0/192373001 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0f940676b0 msgr2=0x7f0f94069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:10.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.489+0000 7f0fb7725640 1 --2- 172.21.15.55:0/192373001 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0f940676b0 0x7f0f94069b70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0fac007fd0 tx=0x7f0fac009040 comp rx=0 tx=0).stop 2024-03-19T01:52:10.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.489+0000 7f0fb7725640 1 -- 172.21.15.55:0/192373001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fb00d0210 msgr2=0x7f0fb0153ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:10.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.489+0000 7f0fb7725640 1 --2- 172.21.15.55:0/192373001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fb00d0210 0x7f0fb0153ff0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0fa001a040 tx=0x7f0fa00191b0 comp rx=0 tx=0).stop 2024-03-19T01:52:10.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.489+0000 7f0fb7725640 1 -- 172.21.15.55:0/192373001 shutdown_connections 2024-03-19T01:52:10.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.489+0000 7f0fb7725640 1 --2- 172.21.15.55:0/192373001 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0f940676b0 0x7f0f94069b70 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:10.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.489+0000 7f0fb7725640 1 --2- 172.21.15.55:0/192373001 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0fb015e2e0 0x7f0fb01f61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:10.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.489+0000 7f0fb7725640 1 --2- 172.21.15.55:0/192373001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0fb00d0210 0x7f0fb0153ff0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:10.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.489+0000 7f0fb7725640 1 --2- 172.21.15.55:0/192373001 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0fb00cfe30 0x7f0fb014ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:10.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.489+0000 7f0fb7725640 1 -- 172.21.15.55:0/192373001 >> 172.21.15.55:0/192373001 conn(0x7f0fb00671d0 msgr2=0x7f0fb015ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:10.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.489+0000 7f0fb7725640 1 -- 172.21.15.55:0/192373001 shutdown_connections 2024-03-19T01:52:10.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.489+0000 7f0fb7725640 1 -- 172.21.15.55:0/192373001 wait complete. 2024-03-19T01:52:10.507 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:50:23.456904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:51:04.578583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:51:06.871659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T01:51:14.274442+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T01:52:10.508 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:52:10.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.629+0000 7f6149cf7640 1 Processor -- start 2024-03-19T01:52:10.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.629+0000 7f6149cf7640 1 -- start start 2024-03-19T01:52:10.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.629+0000 7f6149cf7640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f61440c2e70 0x7f6144073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:10.631 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.629+0000 7f6149cf7640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61440c22b0 con 0x7f6144154460 2024-03-19T01:52:10.631 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.629+0000 7f6149cf7640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f614415a490 con 0x7f614406b2a0 2024-03-19T01:52:10.631 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.629+0000 7f6149cf7640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f614415a230 con 0x7f61440c2e70 2024-03-19T01:52:10.631 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.629+0000 7f6143fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f61440c2e70 0x7f6144073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:10.631 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.629+0000 7f6142ffd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f614406b2a0 0x7f61440c2760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:48402/0 (socket says 172.21.15.55:48402) 2024-03-19T01:52:10.632 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.629+0000 7f6142ffd640 1 -- 172.21.15.55:0/1973460094 learned_addr learned my addr 172.21.15.55:0/1973460094 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:52:10.636 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.633+0000 7f6143fff640 1 -- 172.21.15.55:0/1973460094 >> v1:172.21.15.84:6789/0 conn(0x7f614406b2a0 legacy=0x7f61440c2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:10.636 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.633+0000 7f6143fff640 1 -- 172.21.15.55:0/1973460094 >> v1:172.21.15.55:6789/0 conn(0x7f6144154460 legacy=0x7f61440cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:10.636 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.633+0000 7f6143fff640 1 -- 172.21.15.55:0/1973460094 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f614415a770 con 0x7f61440c2e70 2024-03-19T01:52:10.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.641+0000 7f6143fff640 1 --2- 172.21.15.55:0/1973460094 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f61440c2e70 0x7f6144073540 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f613c009dc0 tx=0x7f613c01f140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8554c1f2e9c17be7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:10.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.645+0000 7f61427fc640 1 -- 172.21.15.55:0/1973460094 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f613c0b4070 con 0x7f61440c2e70 2024-03-19T01:52:10.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.645+0000 7f61427fc640 1 -- 172.21.15.55:0/1973460094 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f613c0af600 con 0x7f61440c2e70 2024-03-19T01:52:10.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.645+0000 7f61427fc640 1 -- 172.21.15.55:0/1973460094 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f613c0af9d0 con 0x7f61440c2e70 2024-03-19T01:52:10.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.645+0000 7f6149cf7640 1 -- 172.21.15.55:0/1973460094 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f61440c2e70 msgr2=0x7f6144073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:10.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.645+0000 7f6149cf7640 1 --2- 172.21.15.55:0/1973460094 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f61440c2e70 0x7f6144073540 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f613c009dc0 tx=0x7f613c01f140 comp rx=0 tx=0).stop 2024-03-19T01:52:10.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.645+0000 7f6149cf7640 1 -- 172.21.15.55:0/1973460094 shutdown_connections 2024-03-19T01:52:10.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.645+0000 7f6149cf7640 1 --2- 172.21.15.55:0/1973460094 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f61440c2e70 0x7f6144073540 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:10.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.645+0000 7f6149cf7640 1 -- 172.21.15.55:0/1973460094 >> 172.21.15.55:0/1973460094 conn(0x7f614406e670 msgr2=0x7f614406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:10.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.645+0000 7f6149cf7640 1 -- 172.21.15.55:0/1973460094 shutdown_connections 2024-03-19T01:52:10.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.645+0000 7f6149cf7640 1 -- 172.21.15.55:0/1973460094 wait complete. 2024-03-19T01:52:10.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.649+0000 7f6149cf7640 1 Processor -- start 2024-03-19T01:52:10.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.649+0000 7f6149cf7640 1 -- start start 2024-03-19T01:52:10.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.649+0000 7f6149cf7640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f614406b2a0 0x7f61440caf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:10.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.649+0000 7f6149cf7640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61440c2e70 0x7f61441f3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:10.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.649+0000 7f6149cf7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6144154460 0x7f61441f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:10.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.649+0000 7f6149cf7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f614415b010 con 0x7f6144154460 2024-03-19T01:52:10.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.649+0000 7f6149cf7640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f614415ad30 con 0x7f61440c2e70 2024-03-19T01:52:10.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.649+0000 7f6149cf7640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f61440c7600 con 0x7f614406b2a0 2024-03-19T01:52:10.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.649+0000 7f6143fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6144154460 0x7f61441f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:10.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.649+0000 7f6143fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6144154460 0x7f61441f3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:32962/0 (socket says 172.21.15.55:32962) 2024-03-19T01:52:10.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.649+0000 7f6143fff640 1 -- 172.21.15.55:0/4030125242 learned_addr learned my addr 172.21.15.55:0/4030125242 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:10.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.649+0000 7f6142ffd640 1 --2- 172.21.15.55:0/4030125242 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61440c2e70 0x7f61441f3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:10.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.649+0000 7f61437fe640 1 --2- 172.21.15.55:0/4030125242 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f614406b2a0 0x7f61440caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:10.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.653+0000 7f6143fff640 1 -- 172.21.15.55:0/4030125242 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f614406b2a0 msgr2=0x7f61440caf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:10.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.653+0000 7f6143fff640 1 --2- 172.21.15.55:0/4030125242 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f614406b2a0 0x7f61440caf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:10.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.653+0000 7f6143fff640 1 -- 172.21.15.55:0/4030125242 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61440c2e70 msgr2=0x7f61441f3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:10.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.653+0000 7f6143fff640 1 --2- 172.21.15.55:0/4030125242 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61440c2e70 0x7f61441f3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:10.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.653+0000 7f6143fff640 1 -- 172.21.15.55:0/4030125242 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f614414d3a0 con 0x7f6144154460 2024-03-19T01:52:10.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.653+0000 7f6143fff640 1 --2- 172.21.15.55:0/4030125242 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6144154460 0x7f61441f3d50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f613c009eb0 tx=0x7f613c01bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:10.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.657+0000 7f612bfff640 1 -- 172.21.15.55:0/4030125242 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f613c0b4040 con 0x7f6144154460 2024-03-19T01:52:10.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.657+0000 7f612bfff640 1 -- 172.21.15.55:0/4030125242 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f613c0c2070 con 0x7f6144154460 2024-03-19T01:52:10.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.657+0000 7f6149cf7640 1 -- 172.21.15.55:0/4030125242 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f614414d640 con 0x7f6144154460 2024-03-19T01:52:10.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.657+0000 7f612bfff640 1 -- 172.21.15.55:0/4030125242 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f613c0ba660 con 0x7f6144154460 2024-03-19T01:52:10.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.657+0000 7f6149cf7640 1 -- 172.21.15.55:0/4030125242 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f614414f310 con 0x7f6144154460 2024-03-19T01:52:10.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.661+0000 7f6149cf7640 1 -- 172.21.15.55:0/4030125242 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6144068a40 con 0x7f6144154460 2024-03-19T01:52:10.924 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.925+0000 7f612bfff640 1 -- 172.21.15.55:0/4030125242 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f613c0ba800 con 0x7f6144154460 2024-03-19T01:52:10.925 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.925+0000 7f612bfff640 1 --2- 172.21.15.55:0/4030125242 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6130067690 0x7f6130069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:10.925 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.925+0000 7f61437fe640 1 --2- 172.21.15.55:0/4030125242 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6130067690 0x7f6130069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:10.926 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.925+0000 7f61437fe640 1 --2- 172.21.15.55:0/4030125242 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6130067690 0x7f6130069b50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f61380098e0 tx=0x7f6138009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:10.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:10.965+0000 7f612bfff640 1 -- 172.21.15.55:0/4030125242 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6532+0+0 (secure 0 0 0) 0x7f613c0f2fd0 con 0x7f6144154460 2024-03-19T01:52:11.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.021+0000 7f612bfff640 1 -- 172.21.15.55:0/4030125242 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f613c018ca0 con 0x7f6144154460 2024-03-19T01:52:11.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.241+0000 7f6149cf7640 1 -- 172.21.15.55:0/4030125242 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f614406c5c0 con 0x7f6144154460 2024-03-19T01:52:11.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.305+0000 7f612bfff640 1 -- 172.21.15.55:0/4030125242 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) ==== 93+0+10 (secure 0 0 0) 0x7f613c0f70a0 con 0x7f6144154460 2024-03-19T01:52:11.307 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 1 2024-03-19T01:52:11.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.309+0000 7f6149cf7640 1 -- 172.21.15.55:0/4030125242 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6130067690 msgr2=0x7f6130069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:11.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.309+0000 7f6149cf7640 1 --2- 172.21.15.55:0/4030125242 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6130067690 0x7f6130069b50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f61380098e0 tx=0x7f6138009060 comp rx=0 tx=0).stop 2024-03-19T01:52:11.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.309+0000 7f6149cf7640 1 -- 172.21.15.55:0/4030125242 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6144154460 msgr2=0x7f61441f3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:11.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.309+0000 7f6149cf7640 1 --2- 172.21.15.55:0/4030125242 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6144154460 0x7f61441f3d50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f613c009eb0 tx=0x7f613c01bee0 comp rx=0 tx=0).stop 2024-03-19T01:52:11.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.309+0000 7f6149cf7640 1 -- 172.21.15.55:0/4030125242 shutdown_connections 2024-03-19T01:52:11.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.309+0000 7f6149cf7640 1 --2- 172.21.15.55:0/4030125242 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6130067690 0x7f6130069b50 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:11.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.309+0000 7f6149cf7640 1 --2- 172.21.15.55:0/4030125242 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6144154460 0x7f61441f3d50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:11.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.309+0000 7f6149cf7640 1 --2- 172.21.15.55:0/4030125242 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61440c2e70 0x7f61441f3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:11.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.309+0000 7f6149cf7640 1 --2- 172.21.15.55:0/4030125242 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f614406b2a0 0x7f61440caf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:11.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.309+0000 7f6149cf7640 1 -- 172.21.15.55:0/4030125242 >> 172.21.15.55:0/4030125242 conn(0x7f614406e670 msgr2=0x7f614406b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:11.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.309+0000 7f6149cf7640 1 -- 172.21.15.55:0/4030125242 shutdown_connections 2024-03-19T01:52:11.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.309+0000 7f6149cf7640 1 -- 172.21.15.55:0/4030125242 wait complete. 2024-03-19T01:52:11.330 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:52:11.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.441+0000 7fdee4ea0640 1 Processor -- start 2024-03-19T01:52:11.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.441+0000 7fdee4ea0640 1 -- start start 2024-03-19T01:52:11.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.441+0000 7fdee4ea0640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fdee00c96b0 0x7fdee015aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:11.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.441+0000 7fdee4ea0640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdee0073350 con 0x7fdee00c9a90 2024-03-19T01:52:11.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.441+0000 7fdee4ea0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdee0073630 con 0x7fdee015e4d0 2024-03-19T01:52:11.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.441+0000 7fdee4ea0640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fdee015e040 con 0x7fdee00c96b0 2024-03-19T01:52:11.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.485+0000 7fdede59c640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fdee00c96b0 0x7fdee015aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:11.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.485+0000 7fdede59c640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fdee00c96b0 0x7fdee015aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:38624/0 (socket says 172.21.15.55:38624) 2024-03-19T01:52:11.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.485+0000 7fdede59c640 1 -- 172.21.15.55:0/526530751 learned_addr learned my addr 172.21.15.55:0/526530751 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:11.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.489+0000 7fdede59c640 1 -- 172.21.15.55:0/526530751 >> v1:172.21.15.84:6789/0 conn(0x7fdee015e4d0 legacy=0x7fdee0153f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:11.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.489+0000 7fdede59c640 1 -- 172.21.15.55:0/526530751 >> v1:172.21.15.55:6789/0 conn(0x7fdee00c9a90 legacy=0x7fdee0070c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:11.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.489+0000 7fdede59c640 1 -- 172.21.15.55:0/526530751 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee015d9f0 con 0x7fdee00c96b0 2024-03-19T01:52:11.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.493+0000 7fdede59c640 1 --2- 172.21.15.55:0/526530751 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdee00c96b0 0x7fdee015aea0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fded0009870 tx=0x7fded001c4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6b9b6c0da365f69 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:11.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.497+0000 7fdedd59a640 1 -- 172.21.15.55:0/526530751 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded00b0070 con 0x7fdee00c96b0 2024-03-19T01:52:11.499 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.497+0000 7fdedd59a640 1 -- 172.21.15.55:0/526530751 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fded00ab600 con 0x7fdee00c96b0 2024-03-19T01:52:11.499 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.497+0000 7fdedd59a640 1 -- 172.21.15.55:0/526530751 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded00ab9d0 con 0x7fdee00c96b0 2024-03-19T01:52:11.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.497+0000 7fdee4ea0640 1 -- 172.21.15.55:0/526530751 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdee00c96b0 msgr2=0x7fdee015aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:11.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.497+0000 7fdee4ea0640 1 --2- 172.21.15.55:0/526530751 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdee00c96b0 0x7fdee015aea0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fded0009870 tx=0x7fded001c4e0 comp rx=0 tx=0).stop 2024-03-19T01:52:11.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.497+0000 7fdee4ea0640 1 -- 172.21.15.55:0/526530751 shutdown_connections 2024-03-19T01:52:11.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.497+0000 7fdee4ea0640 1 --2- 172.21.15.55:0/526530751 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdee00c96b0 0x7fdee015aea0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:11.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.497+0000 7fdee4ea0640 1 -- 172.21.15.55:0/526530751 >> 172.21.15.55:0/526530751 conn(0x7fdee00cc9c0 msgr2=0x7fdee00ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:11.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.497+0000 7fdee4ea0640 1 -- 172.21.15.55:0/526530751 shutdown_connections 2024-03-19T01:52:11.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.497+0000 7fdee4ea0640 1 -- 172.21.15.55:0/526530751 wait complete. 2024-03-19T01:52:11.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.501+0000 7fdee4ea0640 1 Processor -- start 2024-03-19T01:52:11.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.501+0000 7fdee4ea0640 1 -- start start 2024-03-19T01:52:11.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.501+0000 7fdee4ea0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdee00c96b0 0x7fdee0159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:11.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.501+0000 7fdee4ea0640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdee00c9a90 0x7fdee015ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:11.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.501+0000 7fdee4ea0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdee015e4d0 0x7fdee01f3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:11.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.501+0000 7fdee4ea0640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdee0168590 con 0x7fdee015e4d0 2024-03-19T01:52:11.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.501+0000 7fdee4ea0640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fdee0168410 con 0x7fdee00c9a90 2024-03-19T01:52:11.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.501+0000 7fdee4ea0640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fdee0168710 con 0x7fdee00c96b0 2024-03-19T01:52:11.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.505+0000 7fdeddd9b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdee00c9a90 0x7fdee015ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:11.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.505+0000 7fdeddd9b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdee00c9a90 0x7fdee015ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:38628/0 (socket says 172.21.15.55:38628) 2024-03-19T01:52:11.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.505+0000 7fdeddd9b640 1 -- 172.21.15.55:0/2670905235 learned_addr learned my addr 172.21.15.55:0/2670905235 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:11.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.509+0000 7fdeddd9b640 1 -- 172.21.15.55:0/2670905235 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdee00c96b0 msgr2=0x7fdee0159f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:11.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.509+0000 7fdeddd9b640 1 --2- 172.21.15.55:0/2670905235 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdee00c96b0 0x7fdee0159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:11.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.509+0000 7fdeddd9b640 1 -- 172.21.15.55:0/2670905235 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdee015e4d0 msgr2=0x7fdee01f3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:11.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.509+0000 7fdeddd9b640 1 --2- 172.21.15.55:0/2670905235 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdee015e4d0 0x7fdee01f3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:11.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.509+0000 7fdeddd9b640 1 -- 172.21.15.55:0/2670905235 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee01f7ba0 con 0x7fdee00c9a90 2024-03-19T01:52:11.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.513+0000 7fdeddd9b640 1 --2- 172.21.15.55:0/2670905235 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdee00c9a90 0x7fdee015ca10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fded400c410 tx=0x7fded400c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:11.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.517+0000 7fdeceffd640 1 -- 172.21.15.55:0/2670905235 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded4017070 con 0x7fdee00c9a90 2024-03-19T01:52:11.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.517+0000 7fdeceffd640 1 -- 172.21.15.55:0/2670905235 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fded4003600 con 0x7fdee00c9a90 2024-03-19T01:52:11.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.517+0000 7fdee4ea0640 1 -- 172.21.15.55:0/2670905235 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdee01f7020 con 0x7fdee00c9a90 2024-03-19T01:52:11.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.517+0000 7fdeceffd640 1 -- 172.21.15.55:0/2670905235 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded40039f0 con 0x7fdee00c9a90 2024-03-19T01:52:11.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.517+0000 7fdee4ea0640 1 -- 172.21.15.55:0/2670905235 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdee01f75e0 con 0x7fdee00c9a90 2024-03-19T01:52:11.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.517+0000 7fdee4ea0640 1 -- 172.21.15.55:0/2670905235 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdee0068a40 con 0x7fdee00c9a90 2024-03-19T01:52:11.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.549+0000 7fdeceffd640 1 -- 172.21.15.55:0/2670905235 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fded401a070 con 0x7fdee00c9a90 2024-03-19T01:52:11.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.549+0000 7fdeceffd640 1 --2- 172.21.15.55:0/2670905235 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdec0067690 0x7fdec0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:11.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.549+0000 7fdede59c640 1 --2- 172.21.15.55:0/2670905235 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdec0067690 0x7fdec0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:11.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.549+0000 7fdede59c640 1 --2- 172.21.15.55:0/2670905235 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdec0067690 0x7fdec0069b50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fded00ad000 tx=0x7fded0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:11.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.557+0000 7fdeceffd640 1 -- 172.21.15.55:0/2670905235 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6677+0+0 (secure 0 0 0) 0x7fded4053a70 con 0x7fdee00c9a90 2024-03-19T01:52:11.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.821+0000 7fdeceffd640 1 -- 172.21.15.55:0/2670905235 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fded4021070 con 0x7fdee00c9a90 2024-03-19T01:52:11.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:11.993+0000 7fdee4ea0640 1 -- 172.21.15.55:0/2670905235 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdee0072db0 con 0x7fdee00c9a90 2024-03-19T01:52:12.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.001+0000 7fdeceffd640 1 -- 172.21.15.55:0/2670905235 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v30) ==== 104+0+11 (secure 0 0 0) 0x7fded40560d0 con 0x7fdee00c9a90 2024-03-19T01:52:12.004 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 32 2024-03-19T01:52:12.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.005+0000 7fdee4ea0640 1 -- 172.21.15.55:0/2670905235 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdec0067690 msgr2=0x7fdec0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:12.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.005+0000 7fdee4ea0640 1 --2- 172.21.15.55:0/2670905235 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdec0067690 0x7fdec0069b50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fded00ad000 tx=0x7fded0008040 comp rx=0 tx=0).stop 2024-03-19T01:52:12.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.005+0000 7fdee4ea0640 1 -- 172.21.15.55:0/2670905235 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdee00c9a90 msgr2=0x7fdee015ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:12.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.005+0000 7fdee4ea0640 1 --2- 172.21.15.55:0/2670905235 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdee00c9a90 0x7fdee015ca10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fded400c410 tx=0x7fded400c8d0 comp rx=0 tx=0).stop 2024-03-19T01:52:12.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.005+0000 7fdee4ea0640 1 -- 172.21.15.55:0/2670905235 shutdown_connections 2024-03-19T01:52:12.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.005+0000 7fdee4ea0640 1 --2- 172.21.15.55:0/2670905235 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdec0067690 0x7fdec0069b50 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:12.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.005+0000 7fdee4ea0640 1 --2- 172.21.15.55:0/2670905235 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdee015e4d0 0x7fdee01f3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:12.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.005+0000 7fdee4ea0640 1 --2- 172.21.15.55:0/2670905235 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdee00c9a90 0x7fdee015ca10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:12.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.005+0000 7fdee4ea0640 1 --2- 172.21.15.55:0/2670905235 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdee00c96b0 0x7fdee0159f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:12.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.005+0000 7fdee4ea0640 1 -- 172.21.15.55:0/2670905235 >> 172.21.15.55:0/2670905235 conn(0x7fdee00cc9c0 msgr2=0x7fdee015a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:12.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.005+0000 7fdee4ea0640 1 -- 172.21.15.55:0/2670905235 shutdown_connections 2024-03-19T01:52:12.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.005+0000 7fdee4ea0640 1 -- 172.21.15.55:0/2670905235 wait complete. 2024-03-19T01:52:12.023 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:52:12.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.133+0000 7f60b5952640 1 Processor -- start 2024-03-19T01:52:12.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.133+0000 7f60b5952640 1 -- start start 2024-03-19T01:52:12.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.137+0000 7f60b5952640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f60b01510f0 0x7f60b00c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:12.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.137+0000 7f60b5952640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f60b015a410 0x7f60b00708a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:12.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.137+0000 7f60b5952640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f60b00c73c0 0x7f60b00c7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:12.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.137+0000 7f60b5952640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f60b00c3f60 con 0x7f60b015a410 2024-03-19T01:52:12.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.137+0000 7f60b5952640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f60b01514d0 con 0x7f60b01510f0 2024-03-19T01:52:12.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.137+0000 7f60b5952640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f60b00c6870 con 0x7f60b00c73c0 2024-03-19T01:52:12.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.137+0000 7f60ae7fc640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f60b015a410 0x7f60b00708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:12.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.137+0000 7f60aeffd640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f60b01510f0 0x7f60b00c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:12.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.137+0000 7f60af7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f60b00c73c0 0x7f60b00c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:12.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.137+0000 7f60ae7fc640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f60b015a410 0x7f60b00708a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:32990/0 (socket says 172.21.15.55:32990) 2024-03-19T01:52:12.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.137+0000 7f60ae7fc640 1 -- 172.21.15.55:0/1567756297 learned_addr learned my addr 172.21.15.55:0/1567756297 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:12.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.141+0000 7f60ae7fc640 1 -- 172.21.15.55:0/1567756297 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f60b00c73c0 msgr2=0x7f60b00c7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:12.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.141+0000 7f60ae7fc640 1 --2- 172.21.15.55:0/1567756297 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f60b00c73c0 0x7f60b00c7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:12.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.141+0000 7f60ae7fc640 1 -- 172.21.15.55:0/1567756297 >> v2:172.21.15.84:3300/0 conn(0x7f60b01510f0 msgr2=0x7f60b00c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:12.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.141+0000 7f60ae7fc640 1 --2- 172.21.15.55:0/1567756297 >> v2:172.21.15.84:3300/0 conn(0x7f60b01510f0 0x7f60b00c61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:12.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.141+0000 7f60ae7fc640 1 -- 172.21.15.55:0/1567756297 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b00c6b50 con 0x7f60b015a410 2024-03-19T01:52:12.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.145+0000 7f60ae7fc640 1 --2- 172.21.15.55:0/1567756297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60b015a410 0x7f60b00708a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f60b00cd800 tx=0x7f60a401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c884b153dd37cb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:12.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.149+0000 7f60adffb640 1 -- 172.21.15.55:0/1567756297 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a40b5070 con 0x7f60b015a410 2024-03-19T01:52:12.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.149+0000 7f60adffb640 1 -- 172.21.15.55:0/1567756297 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60a40a89d0 con 0x7f60b015a410 2024-03-19T01:52:12.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.149+0000 7f60adffb640 1 -- 172.21.15.55:0/1567756297 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a40a8da0 con 0x7f60b015a410 2024-03-19T01:52:12.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.149+0000 7f60b5952640 1 -- 172.21.15.55:0/1567756297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60b015a410 msgr2=0x7f60b00708a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:12.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.149+0000 7f60b5952640 1 --2- 172.21.15.55:0/1567756297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60b015a410 0x7f60b00708a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f60b00cd800 tx=0x7f60a401ed10 comp rx=0 tx=0).stop 2024-03-19T01:52:12.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.149+0000 7f60b5952640 1 -- 172.21.15.55:0/1567756297 shutdown_connections 2024-03-19T01:52:12.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.149+0000 7f60b5952640 1 --2- 172.21.15.55:0/1567756297 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f60b00c73c0 0x7f60b00c7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:12.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.149+0000 7f60b5952640 1 --2- 172.21.15.55:0/1567756297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60b015a410 0x7f60b00708a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:12.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.149+0000 7f60b5952640 1 --2- 172.21.15.55:0/1567756297 >> v2:172.21.15.84:3300/0 conn(0x7f60b01510f0 0x7f60b00c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:12.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.149+0000 7f60b5952640 1 -- 172.21.15.55:0/1567756297 >> 172.21.15.55:0/1567756297 conn(0x7f60b00cc9c0 msgr2=0x7f60b00ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:12.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.149+0000 7f60b5952640 1 -- 172.21.15.55:0/1567756297 shutdown_connections 2024-03-19T01:52:12.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.149+0000 7f60b5952640 1 -- 172.21.15.55:0/1567756297 wait complete. 2024-03-19T01:52:12.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.149+0000 7f60b5952640 1 Processor -- start 2024-03-19T01:52:12.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.149+0000 7f60b5952640 1 -- start start 2024-03-19T01:52:12.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.153+0000 7f60b5952640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f60b00c73c0 0x7f60b01e4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:12.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.153+0000 7f60b5952640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60b01510f0 0x7f60b01ee3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:12.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.153+0000 7f60b5952640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60b015a410 0x7f60b01e35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:12.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.153+0000 7f60b5952640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f60b00c6e30 con 0x7f60b01510f0 2024-03-19T01:52:12.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.153+0000 7f60b5952640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f60b00c44e0 con 0x7f60b015a410 2024-03-19T01:52:12.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.153+0000 7f60b5952640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f60b00c4a00 con 0x7f60b00c73c0 2024-03-19T01:52:12.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.153+0000 7f60ae7fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60b01510f0 0x7f60b01ee3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:12.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.153+0000 7f60af7fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60b015a410 0x7f60b01e35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:12.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.153+0000 7f60ae7fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60b01510f0 0x7f60b01ee3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:32994/0 (socket says 172.21.15.55:32994) 2024-03-19T01:52:12.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.153+0000 7f60ae7fc640 1 -- 172.21.15.55:0/3659393436 learned_addr learned my addr 172.21.15.55:0/3659393436 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:12.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.153+0000 7f60aeffd640 1 --2- 172.21.15.55:0/3659393436 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f60b00c73c0 0x7f60b01e4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:12.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.157+0000 7f60ae7fc640 1 -- 172.21.15.55:0/3659393436 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f60b00c73c0 msgr2=0x7f60b01e4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:12.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.157+0000 7f60ae7fc640 1 --2- 172.21.15.55:0/3659393436 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f60b00c73c0 0x7f60b01e4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:12.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.157+0000 7f60ae7fc640 1 -- 172.21.15.55:0/3659393436 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60b015a410 msgr2=0x7f60b01e35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:12.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.157+0000 7f60ae7fc640 1 --2- 172.21.15.55:0/3659393436 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60b015a410 0x7f60b01e35f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:12.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.157+0000 7f60ae7fc640 1 -- 172.21.15.55:0/3659393436 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b01f1650 con 0x7f60b01510f0 2024-03-19T01:52:12.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.161+0000 7f60ae7fc640 1 --2- 172.21.15.55:0/3659393436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60b01510f0 0x7f60b01ee3b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f60a40b3040 tx=0x7f60a401bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:12.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.165+0000 7f6097fff640 1 -- 172.21.15.55:0/3659393436 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a40c0070 con 0x7f60b01510f0 2024-03-19T01:52:12.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.165+0000 7f6097fff640 1 -- 172.21.15.55:0/3659393436 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60a40b0740 con 0x7f60b01510f0 2024-03-19T01:52:12.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.165+0000 7f60b5952640 1 -- 172.21.15.55:0/3659393436 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60b01f1930 con 0x7f60b01510f0 2024-03-19T01:52:12.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.165+0000 7f6097fff640 1 -- 172.21.15.55:0/3659393436 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a40b5040 con 0x7f60b01510f0 2024-03-19T01:52:12.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.165+0000 7f60b5952640 1 -- 172.21.15.55:0/3659393436 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60b01f1ef0 con 0x7f60b01510f0 2024-03-19T01:52:12.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.169+0000 7f60b5952640 1 -- 172.21.15.55:0/3659393436 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60b0068a40 con 0x7f60b01510f0 2024-03-19T01:52:12.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.197+0000 7f6097fff640 1 -- 172.21.15.55:0/3659393436 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f60a40c3070 con 0x7f60b01510f0 2024-03-19T01:52:12.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.197+0000 7f6097fff640 1 --2- 172.21.15.55:0/3659393436 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f609c0676b0 0x7f609c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:12.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.197+0000 7f60aeffd640 1 --2- 172.21.15.55:0/3659393436 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f609c0676b0 0x7f609c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:12.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.197+0000 7f60aeffd640 1 --2- 172.21.15.55:0/3659393436 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f609c0676b0 0x7f609c069b70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009b60 tx=0x7f60a0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:12.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.201+0000 7f6097fff640 1 -- 172.21.15.55:0/3659393436 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6677+0+0 (secure 0 0 0) 0x7f60a40f10e0 con 0x7f60b01510f0 2024-03-19T01:52:12.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.493+0000 7f6097fff640 1 -- 172.21.15.55:0/3659393436 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f60a40ca080 con 0x7f60b01510f0 2024-03-19T01:52:12.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.685+0000 7f60b5952640 1 -- 172.21.15.55:0/3659393436 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f60b00c92c0 con 0x7f60b01510f0 2024-03-19T01:52:12.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.865+0000 7f6097fff640 1 -- 172.21.15.55:0/3659393436 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v30) ==== 100+0+11 (secure 0 0 0) 0x7f60b00c92c0 con 0x7f60b01510f0 2024-03-19T01:52:12.866 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 64 2024-03-19T01:52:12.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.865+0000 7f60b5952640 1 -- 172.21.15.55:0/3659393436 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f609c0676b0 msgr2=0x7f609c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:12.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.865+0000 7f60b5952640 1 --2- 172.21.15.55:0/3659393436 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f609c0676b0 0x7f609c069b70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009b60 tx=0x7f60a0009190 comp rx=0 tx=0).stop 2024-03-19T01:52:12.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.865+0000 7f60b5952640 1 -- 172.21.15.55:0/3659393436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60b01510f0 msgr2=0x7f60b01ee3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:12.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.865+0000 7f60b5952640 1 --2- 172.21.15.55:0/3659393436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60b01510f0 0x7f60b01ee3b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f60a40b3040 tx=0x7f60a401bab0 comp rx=0 tx=0).stop 2024-03-19T01:52:12.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.865+0000 7f60b5952640 1 -- 172.21.15.55:0/3659393436 shutdown_connections 2024-03-19T01:52:12.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.865+0000 7f60b5952640 1 --2- 172.21.15.55:0/3659393436 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f609c0676b0 0x7f609c069b70 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:12.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.865+0000 7f60b5952640 1 --2- 172.21.15.55:0/3659393436 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60b015a410 0x7f60b01e35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:12.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.865+0000 7f60b5952640 1 --2- 172.21.15.55:0/3659393436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60b01510f0 0x7f60b01ee3b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:12.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.865+0000 7f60b5952640 1 --2- 172.21.15.55:0/3659393436 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f60b00c73c0 0x7f60b01e4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:12.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.865+0000 7f60b5952640 1 -- 172.21.15.55:0/3659393436 >> 172.21.15.55:0/3659393436 conn(0x7f60b00cc9c0 msgr2=0x7f60b00c7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:12.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.865+0000 7f60b5952640 1 -- 172.21.15.55:0/3659393436 shutdown_connections 2024-03-19T01:52:12.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:12.865+0000 7f60b5952640 1 -- 172.21.15.55:0/3659393436 wait complete. 2024-03-19T01:52:12.886 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T01:52:13.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.017+0000 7f650aa6a640 1 Processor -- start 2024-03-19T01:52:13.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.017+0000 7f650aa6a640 1 -- start start 2024-03-19T01:52:13.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.021+0000 7f650aa6a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6504154020 0x7f650406c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:13.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.021+0000 7f650aa6a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f6504067390 0x7f650406e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:13.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.021+0000 7f650aa6a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f650415b0e0 con 0x7f6504154400 2024-03-19T01:52:13.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.021+0000 7f650aa6a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f650406ee60 con 0x7f6504067390 2024-03-19T01:52:13.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.021+0000 7f650aa6a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f650406b340 con 0x7f6504154020 2024-03-19T01:52:13.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.021+0000 7f6509007640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f6504067390 0x7f650406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:13.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.021+0000 7f6503fff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f6504154400 0x7f650406e210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:48410/0 (socket says 172.21.15.55:48410) 2024-03-19T01:52:13.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.021+0000 7f6503fff640 1 -- 172.21.15.55:0/3302629879 learned_addr learned my addr 172.21.15.55:0/3302629879 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:52:13.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.025+0000 7f6509007640 1 -- 172.21.15.55:0/3302629879 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6504154020 msgr2=0x7f650406c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:13.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.025+0000 7f6509007640 1 --2- 172.21.15.55:0/3302629879 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6504154020 0x7f650406c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:13.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.025+0000 7f6509007640 1 -- 172.21.15.55:0/3302629879 >> v1:172.21.15.84:6789/0 conn(0x7f6504154400 legacy=0x7f650406e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:13.025 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.025+0000 7f6509007640 1 -- 172.21.15.55:0/3302629879 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f650406b620 con 0x7f6504067390 2024-03-19T01:52:13.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.029+0000 7f6509007640 1 --2- 172.21.15.55:0/3302629879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6504067390 0x7f650406e920 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f64fc009d90 tx=0x7f64fc01c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9988207489f76834 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:13.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.037+0000 7f65037fe640 1 -- 172.21.15.55:0/3302629879 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64fc0b2070 con 0x7f6504067390 2024-03-19T01:52:13.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.037+0000 7f65037fe640 1 -- 172.21.15.55:0/3302629879 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64fc0a5a00 con 0x7f6504067390 2024-03-19T01:52:13.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.037+0000 7f65037fe640 1 -- 172.21.15.55:0/3302629879 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64fc0a5dd0 con 0x7f6504067390 2024-03-19T01:52:13.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.037+0000 7f650aa6a640 1 -- 172.21.15.55:0/3302629879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6504067390 msgr2=0x7f650406e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:13.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.037+0000 7f650aa6a640 1 --2- 172.21.15.55:0/3302629879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6504067390 0x7f650406e920 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f64fc009d90 tx=0x7f64fc01c940 comp rx=0 tx=0).stop 2024-03-19T01:52:13.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.037+0000 7f650aa6a640 1 -- 172.21.15.55:0/3302629879 shutdown_connections 2024-03-19T01:52:13.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.037+0000 7f650aa6a640 1 --2- 172.21.15.55:0/3302629879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6504067390 0x7f650406e920 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:13.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.037+0000 7f650aa6a640 1 --2- 172.21.15.55:0/3302629879 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6504154020 0x7f650406c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:13.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.037+0000 7f650aa6a640 1 -- 172.21.15.55:0/3302629879 >> 172.21.15.55:0/3302629879 conn(0x7f65040cc270 msgr2=0x7f65040cc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:13.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.037+0000 7f650aa6a640 1 -- 172.21.15.55:0/3302629879 shutdown_connections 2024-03-19T01:52:13.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.037+0000 7f650aa6a640 1 -- 172.21.15.55:0/3302629879 wait complete. 2024-03-19T01:52:13.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.037+0000 7f650aa6a640 1 Processor -- start 2024-03-19T01:52:13.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.037+0000 7f650aa6a640 1 -- start start 2024-03-19T01:52:13.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.041+0000 7f650aa6a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6504067390 0x7f650414bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:13.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.041+0000 7f650aa6a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6504154020 0x7f65041509f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:13.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.041+0000 7f650aa6a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6504154400 0x7f6504150f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:13.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.041+0000 7f650aa6a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f65040ccd90 con 0x7f6504154400 2024-03-19T01:52:13.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.041+0000 7f650aa6a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f650406c920 con 0x7f6504067390 2024-03-19T01:52:13.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.041+0000 7f650aa6a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f65040ccf10 con 0x7f6504154020 2024-03-19T01:52:13.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.041+0000 7f6508806640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6504067390 0x7f650414bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:13.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.041+0000 7f6503fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6504154020 0x7f65041509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:13.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.041+0000 7f6508806640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6504067390 0x7f650414bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:38662/0 (socket says 172.21.15.55:38662) 2024-03-19T01:52:13.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.041+0000 7f6508806640 1 -- 172.21.15.55:0/2746860659 learned_addr learned my addr 172.21.15.55:0/2746860659 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:13.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.045+0000 7f6503fff640 1 -- 172.21.15.55:0/2746860659 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6504067390 msgr2=0x7f650414bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:13.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.045+0000 7f6503fff640 1 --2- 172.21.15.55:0/2746860659 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6504067390 0x7f650414bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:13.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.045+0000 7f6503fff640 1 -- 172.21.15.55:0/2746860659 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6504154400 msgr2=0x7f6504150f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:13.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.045+0000 7f6503fff640 1 --2- 172.21.15.55:0/2746860659 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6504154400 0x7f6504150f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:13.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.045+0000 7f6503fff640 1 -- 172.21.15.55:0/2746860659 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f650414adf0 con 0x7f6504154020 2024-03-19T01:52:13.050 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.049+0000 7f6503fff640 1 --2- 172.21.15.55:0/2746860659 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6504154020 0x7f65041509f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f64f000c430 tx=0x7f64f000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:13.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.053+0000 7f65017fa640 1 -- 172.21.15.55:0/2746860659 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f0017070 con 0x7f6504154020 2024-03-19T01:52:13.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.053+0000 7f65017fa640 1 -- 172.21.15.55:0/2746860659 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64f0005cf0 con 0x7f6504154020 2024-03-19T01:52:13.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.053+0000 7f650aa6a640 1 -- 172.21.15.55:0/2746860659 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f650414eff0 con 0x7f6504154020 2024-03-19T01:52:13.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.053+0000 7f65017fa640 1 -- 172.21.15.55:0/2746860659 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f00072d0 con 0x7f6504154020 2024-03-19T01:52:13.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.053+0000 7f650aa6a640 1 -- 172.21.15.55:0/2746860659 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f650414daf0 con 0x7f6504154020 2024-03-19T01:52:13.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.057+0000 7f650aa6a640 1 -- 172.21.15.55:0/2746860659 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6504068a40 con 0x7f6504154020 2024-03-19T01:52:13.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.089+0000 7f65017fa640 1 -- 172.21.15.55:0/2746860659 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f64f001a070 con 0x7f6504154020 2024-03-19T01:52:13.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.089+0000 7f65017fa640 1 --2- 172.21.15.55:0/2746860659 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f64e80676b0 0x7f64e8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:13.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.089+0000 7f6508806640 1 --2- 172.21.15.55:0/2746860659 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f64e80676b0 0x7f64e8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:13.092 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.089+0000 7f6508806640 1 --2- 172.21.15.55:0/2746860659 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f64e80676b0 0x7f64e8069b70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f64f8009a50 tx=0x7f64f8009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:13.356 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.353+0000 7f65017fa640 1 -- 172.21.15.55:0/2746860659 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(31..31 src has 1..31) ==== 6822+0+0 (secure 0 0 0) 0x7f64f0053720 con 0x7f6504154020 2024-03-19T01:52:13.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.385+0000 7f65017fa640 1 -- 172.21.15.55:0/2746860659 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f64f0055680 con 0x7f6504154020 2024-03-19T01:52:13.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.565+0000 7f650aa6a640 1 -- 172.21.15.55:0/2746860659 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f650414a760 con 0x7f6504154020 2024-03-19T01:52:13.576 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.573+0000 7f65017fa640 1 -- 172.21.15.55:0/2746860659 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v31) ==== 103+0+11 (secure 0 0 0) 0x7f64f0059720 con 0x7f6504154020 2024-03-19T01:52:13.576 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 32 2024-03-19T01:52:13.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.577+0000 7f650aa6a640 1 -- 172.21.15.55:0/2746860659 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f64e80676b0 msgr2=0x7f64e8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:13.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.577+0000 7f650aa6a640 1 --2- 172.21.15.55:0/2746860659 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f64e80676b0 0x7f64e8069b70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f64f8009a50 tx=0x7f64f8009210 comp rx=0 tx=0).stop 2024-03-19T01:52:13.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.577+0000 7f650aa6a640 1 -- 172.21.15.55:0/2746860659 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6504154020 msgr2=0x7f65041509f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:13.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.577+0000 7f650aa6a640 1 --2- 172.21.15.55:0/2746860659 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6504154020 0x7f65041509f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f64f000c430 tx=0x7f64f000c8f0 comp rx=0 tx=0).stop 2024-03-19T01:52:13.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.577+0000 7f650aa6a640 1 -- 172.21.15.55:0/2746860659 shutdown_connections 2024-03-19T01:52:13.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.577+0000 7f650aa6a640 1 --2- 172.21.15.55:0/2746860659 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f64e80676b0 0x7f64e8069b70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:13.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.577+0000 7f650aa6a640 1 --2- 172.21.15.55:0/2746860659 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6504154400 0x7f6504150f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:13.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.577+0000 7f650aa6a640 1 --2- 172.21.15.55:0/2746860659 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6504154020 0x7f65041509f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:13.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.577+0000 7f650aa6a640 1 --2- 172.21.15.55:0/2746860659 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6504067390 0x7f650414bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:13.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.577+0000 7f650aa6a640 1 -- 172.21.15.55:0/2746860659 >> 172.21.15.55:0/2746860659 conn(0x7f65040cc270 msgr2=0x7f65040c5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:13.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.577+0000 7f650aa6a640 1 -- 172.21.15.55:0/2746860659 shutdown_connections 2024-03-19T01:52:13.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.577+0000 7f650aa6a640 1 -- 172.21.15.55:0/2746860659 wait complete. 2024-03-19T01:52:13.595 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:52:13.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.709+0000 7fa6965e3640 1 Processor -- start 2024-03-19T01:52:13.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.709+0000 7fa6965e3640 1 -- start start 2024-03-19T01:52:13.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.709+0000 7fa6965e3640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa6900c9000 0x7fa6900c9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:13.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.709+0000 7fa6965e3640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa6900c2ad0 0x7fa6900c2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:13.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.709+0000 7fa6965e3640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa690158d50 con 0x7fa6901520b0 2024-03-19T01:52:13.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.709+0000 7fa6965e3640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fa69015c4e0 con 0x7fa6900c9000 2024-03-19T01:52:13.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.709+0000 7fa6965e3640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa69015c050 con 0x7fa6900c2ad0 2024-03-19T01:52:13.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.761+0000 7fa68ffff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fa6901520b0 0x7fa6900709a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:58440/0 (socket says 172.21.15.55:58440) 2024-03-19T01:52:13.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.761+0000 7fa68ffff640 1 -- 172.21.15.55:0/2231702359 learned_addr learned my addr 172.21.15.55:0/2231702359 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:52:13.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.765+0000 7fa68effd640 1 -- 172.21.15.55:0/2231702359 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3780867676 0 0) 0x7fa690158d50 con 0x7fa6901520b0 2024-03-19T01:52:13.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.765+0000 7fa68effd640 1 -- 172.21.15.55:0/2231702359 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa678003530 con 0x7fa6901520b0 2024-03-19T01:52:13.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.769+0000 7fa68effd640 1 -- 172.21.15.55:0/2231702359 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4226248612 0 0) 0x7fa678003530 con 0x7fa6901520b0 2024-03-19T01:52:13.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.769+0000 7fa68effd640 1 -- 172.21.15.55:0/2231702359 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa6900c2ad0 msgr2=0x7fa6900c2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:13.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.769+0000 7fa68effd640 1 --2- 172.21.15.55:0/2231702359 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa6900c2ad0 0x7fa6900c2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:13.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.769+0000 7fa68effd640 1 -- 172.21.15.55:0/2231702359 >> v2:172.21.15.84:3300/0 conn(0x7fa6900c9000 msgr2=0x7fa6900c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:13.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.769+0000 7fa68effd640 1 --2- 172.21.15.55:0/2231702359 >> v2:172.21.15.84:3300/0 conn(0x7fa6900c9000 0x7fa6900c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:13.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.769+0000 7fa68effd640 1 -- 172.21.15.55:0/2231702359 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa69015b720 con 0x7fa6901520b0 2024-03-19T01:52:13.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.769+0000 7fa68effd640 1 -- 172.21.15.55:0/2231702359 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7fa680001680 con 0x7fa6901520b0 2024-03-19T01:52:13.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.769+0000 7fa68effd640 1 -- 172.21.15.55:0/2231702359 >> v1:172.21.15.55:6789/0 conn(0x7fa6901520b0 legacy=0x7fa6900709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:13.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.769+0000 7fa68effd640 1 --2- 172.21.15.55:0/2231702359 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6780039c0 0x7fa678003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:13.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.769+0000 7fa68effd640 1 --2- 172.21.15.55:0/2231702359 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa6900c2ad0 0x7fa678004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:13.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.769+0000 7fa68effd640 1 --2- 172.21.15.55:0/2231702359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa678004e70 0x7fa67800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:13.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.769+0000 7fa68effd640 1 -- 172.21.15.55:0/2231702359 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa69015c4e0 con 0x7fa678004e70 2024-03-19T01:52:13.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.769+0000 7fa68effd640 1 -- 172.21.15.55:0/2231702359 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa69015c050 con 0x7fa6780039c0 2024-03-19T01:52:13.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.769+0000 7fa68effd640 1 -- 172.21.15.55:0/2231702359 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa6780055a0 con 0x7fa6900c2ad0 2024-03-19T01:52:13.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.773+0000 7fa68ffff640 1 --2- 172.21.15.55:0/2231702359 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6780039c0 0x7fa678003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:13.776 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.773+0000 7fa694b80640 1 --2- 172.21.15.55:0/2231702359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa678004e70 0x7fa67800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:13.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.777+0000 7fa68ffff640 1 -- 172.21.15.55:0/2231702359 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa6900c2ad0 msgr2=0x7fa678004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:13.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.777+0000 7fa68ffff640 1 --2- 172.21.15.55:0/2231702359 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa6900c2ad0 0x7fa678004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:13.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.777+0000 7fa68ffff640 1 -- 172.21.15.55:0/2231702359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa678004e70 msgr2=0x7fa67800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:13.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.777+0000 7fa68ffff640 1 --2- 172.21.15.55:0/2231702359 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa678004e70 0x7fa67800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:13.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.777+0000 7fa68ffff640 1 -- 172.21.15.55:0/2231702359 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa678003530 con 0x7fa6780039c0 2024-03-19T01:52:13.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.781+0000 7fa68ffff640 1 --2- 172.21.15.55:0/2231702359 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6780039c0 0x7fa678003de0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa68000c190 tx=0x7fa68001cbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddab63614769128d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:13.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.789+0000 7fa68effd640 1 -- 172.21.15.55:0/2231702359 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa680007c70 con 0x7fa6780039c0 2024-03-19T01:52:13.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.789+0000 7fa68effd640 1 -- 172.21.15.55:0/2231702359 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6800a9020 con 0x7fa6780039c0 2024-03-19T01:52:13.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.789+0000 7fa68effd640 1 -- 172.21.15.55:0/2231702359 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa680005ae0 con 0x7fa6780039c0 2024-03-19T01:52:13.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.789+0000 7fa6965e3640 1 -- 172.21.15.55:0/2231702359 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6780039c0 msgr2=0x7fa678003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:13.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.789+0000 7fa6965e3640 1 --2- 172.21.15.55:0/2231702359 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6780039c0 0x7fa678003de0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa68000c190 tx=0x7fa68001cbe0 comp rx=0 tx=0).stop 2024-03-19T01:52:13.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.789+0000 7fa694b80640 1 -- 172.21.15.55:0/2231702359 reap_dead start 2024-03-19T01:52:13.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.789+0000 7fa6965e3640 1 -- 172.21.15.55:0/2231702359 shutdown_connections 2024-03-19T01:52:13.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.789+0000 7fa6965e3640 1 -- 172.21.15.55:0/2231702359 >> 172.21.15.55:0/2231702359 conn(0x7fa690067600 msgr2=0x7fa690071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:13.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.789+0000 7fa6965e3640 1 -- 172.21.15.55:0/2231702359 shutdown_connections 2024-03-19T01:52:13.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.789+0000 7fa6965e3640 1 -- 172.21.15.55:0/2231702359 wait complete. 2024-03-19T01:52:13.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.789+0000 7fa6965e3640 1 Processor -- start 2024-03-19T01:52:13.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.793+0000 7fa6965e3640 1 -- start start 2024-03-19T01:52:13.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.793+0000 7fa6965e3640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa6780039c0 0x7fa69014b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:13.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.793+0000 7fa6965e3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa690150e70 0x7fa6900cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:13.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.793+0000 7fa6965e3640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6900ce440 0x7fa6900cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:13.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.793+0000 7fa6965e3640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa69015bce0 con 0x7fa690150e70 2024-03-19T01:52:13.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.793+0000 7fa6965e3640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa6900710b0 con 0x7fa6900ce440 2024-03-19T01:52:13.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.793+0000 7fa6965e3640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa6901589a0 con 0x7fa6780039c0 2024-03-19T01:52:13.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.793+0000 7fa694b80640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6900ce440 0x7fa6900cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:13.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.793+0000 7fa68f7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa690150e70 0x7fa6900cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:13.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.793+0000 7fa68ffff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa6780039c0 0x7fa69014b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:13.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.793+0000 7fa694b80640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6900ce440 0x7fa6900cc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:38676/0 (socket says 172.21.15.55:38676) 2024-03-19T01:52:13.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.793+0000 7fa694b80640 1 -- 172.21.15.55:0/3182525202 learned_addr learned my addr 172.21.15.55:0/3182525202 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:13.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.797+0000 7fa694b80640 1 -- 172.21.15.55:0/3182525202 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa6780039c0 msgr2=0x7fa69014b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:13.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.797+0000 7fa694b80640 1 --2- 172.21.15.55:0/3182525202 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa6780039c0 0x7fa69014b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:13.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.797+0000 7fa694b80640 1 -- 172.21.15.55:0/3182525202 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa690150e70 msgr2=0x7fa6900cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:13.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.797+0000 7fa694b80640 1 --2- 172.21.15.55:0/3182525202 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa690150e70 0x7fa6900cc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:13.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.797+0000 7fa694b80640 1 -- 172.21.15.55:0/3182525202 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa690150520 con 0x7fa6900ce440 2024-03-19T01:52:13.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.801+0000 7fa694b80640 1 --2- 172.21.15.55:0/3182525202 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6900ce440 0x7fa6900cc580 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa690067fc0 tx=0x7fa68800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:13.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.805+0000 7fa68cff9640 1 -- 172.21.15.55:0/3182525202 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa688014070 con 0x7fa6900ce440 2024-03-19T01:52:13.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.805+0000 7fa68cff9640 1 -- 172.21.15.55:0/3182525202 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa68800be80 con 0x7fa6900ce440 2024-03-19T01:52:13.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.805+0000 7fa68cff9640 1 -- 172.21.15.55:0/3182525202 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa688008040 con 0x7fa6900ce440 2024-03-19T01:52:13.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.805+0000 7fa6965e3640 1 -- 172.21.15.55:0/3182525202 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6901ee810 con 0x7fa6900ce440 2024-03-19T01:52:13.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.805+0000 7fa6965e3640 1 -- 172.21.15.55:0/3182525202 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa690151670 con 0x7fa6900ce440 2024-03-19T01:52:13.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:13.805+0000 7fa6965e3640 1 -- 172.21.15.55:0/3182525202 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa690068a40 con 0x7fa6900ce440 2024-03-19T01:52:14.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.069+0000 7fa68cff9640 1 -- 172.21.15.55:0/3182525202 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa68800e660 con 0x7fa6900ce440 2024-03-19T01:52:14.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.069+0000 7fa68cff9640 1 --2- 172.21.15.55:0/3182525202 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa67c067690 0x7fa67c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:14.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.069+0000 7fa68ffff640 1 --2- 172.21.15.55:0/3182525202 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa67c067690 0x7fa67c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:14.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.073+0000 7fa68ffff640 1 --2- 172.21.15.55:0/3182525202 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa67c067690 0x7fa67c069b50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa6800aa000 tx=0x7fa690157f20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:14.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.073+0000 7fa68cff9640 1 -- 172.21.15.55:0/3182525202 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6822+0+0 (secure 0 0 0) 0x7fa688054450 con 0x7fa6900ce440 2024-03-19T01:52:14.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.085+0000 7fa68cff9640 1 -- 172.21.15.55:0/3182525202 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa68801e060 con 0x7fa6900ce440 2024-03-19T01:52:14.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.317+0000 7fa6965e3640 1 -- 172.21.15.55:0/3182525202 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6900cabf0 con 0x7fa6900ce440 2024-03-19T01:52:14.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.353+0000 7fa68cff9640 1 -- 172.21.15.55:0/3182525202 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6909 (secure 0 0 0) 0x7fa6880563b0 con 0x7fa6900ce440 2024-03-19T01:52:14.356 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:52:14.356 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4200,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:52:05.800317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4249,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4249":{"gid":4249,"name":"f","rank":1,"incarnation":11,"state":"up:active","state_seq":20,"addr":"172.21.15.84:6839/354880008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6838","nonce":354880008},{"type":"v1","addr":"172.21.15.84:6839","nonce":354880008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4249,4237,4255]},"id":1}]} 2024-03-19T01:52:14.356 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 20 2024-03-19T01:52:14.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.357+0000 7fa6965e3640 1 -- 172.21.15.55:0/3182525202 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa67c067690 msgr2=0x7fa67c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:14.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.357+0000 7fa6965e3640 1 --2- 172.21.15.55:0/3182525202 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa67c067690 0x7fa67c069b50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa6800aa000 tx=0x7fa690157f20 comp rx=0 tx=0).stop 2024-03-19T01:52:14.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.357+0000 7fa6965e3640 1 -- 172.21.15.55:0/3182525202 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6900ce440 msgr2=0x7fa6900cc580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:14.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.357+0000 7fa6965e3640 1 --2- 172.21.15.55:0/3182525202 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6900ce440 0x7fa6900cc580 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa690067fc0 tx=0x7fa68800b330 comp rx=0 tx=0).stop 2024-03-19T01:52:14.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.357+0000 7fa6965e3640 1 -- 172.21.15.55:0/3182525202 shutdown_connections 2024-03-19T01:52:14.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.357+0000 7fa6965e3640 1 --2- 172.21.15.55:0/3182525202 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa67c067690 0x7fa67c069b50 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:14.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.357+0000 7fa6965e3640 1 --2- 172.21.15.55:0/3182525202 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6900ce440 0x7fa6900cc580 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:14.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.357+0000 7fa6965e3640 1 --2- 172.21.15.55:0/3182525202 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa690150e70 0x7fa6900cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:14.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.357+0000 7fa6965e3640 1 --2- 172.21.15.55:0/3182525202 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa6780039c0 0x7fa69014b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:14.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.357+0000 7fa6965e3640 1 -- 172.21.15.55:0/3182525202 >> 172.21.15.55:0/3182525202 conn(0x7fa690067600 msgr2=0x7fa690156870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:14.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.357+0000 7fa6965e3640 1 -- 172.21.15.55:0/3182525202 shutdown_connections 2024-03-19T01:52:14.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.357+0000 7fa6965e3640 1 -- 172.21.15.55:0/3182525202 wait complete. 2024-03-19T01:52:14.375 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:52:14.499 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.497+0000 7f26eb21e640 1 Processor -- start 2024-03-19T01:52:14.499 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.497+0000 7f26eb21e640 1 -- start start 2024-03-19T01:52:14.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.497+0000 7f26eb21e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f26e414ec70 0x7f26e40c8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:14.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.497+0000 7f26eb21e640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f26e4073700 0x7f26e406b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:14.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.497+0000 7f26eb21e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26e4158360 con 0x7f26e40732b0 2024-03-19T01:52:14.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.497+0000 7f26eb21e640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f26e4158600 con 0x7f26e414ec70 2024-03-19T01:52:14.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.497+0000 7f26eb21e640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f26e41588c0 con 0x7f26e4073700 2024-03-19T01:52:14.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.501+0000 7f26e97bb640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f26e4073700 0x7f26e406b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:14.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.501+0000 7f26e8fba640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f26e414ec70 0x7f26e40c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:14.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.501+0000 7f26e97bb640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f26e4073700 0x7f26e406b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:38678/0 (socket says 172.21.15.55:38678) 2024-03-19T01:52:14.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.501+0000 7f26e97bb640 1 -- 172.21.15.55:0/1594415556 learned_addr learned my addr 172.21.15.55:0/1594415556 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:14.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.505+0000 7f26e8fba640 1 -- 172.21.15.55:0/1594415556 >> v2:172.21.15.84:3300/0 conn(0x7f26e4073700 msgr2=0x7f26e406b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:14.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.505+0000 7f26e8fba640 1 --2- 172.21.15.55:0/1594415556 >> v2:172.21.15.84:3300/0 conn(0x7f26e4073700 0x7f26e406b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:14.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.505+0000 7f26e8fba640 1 -- 172.21.15.55:0/1594415556 >> v1:172.21.15.84:6789/0 conn(0x7f26e40732b0 legacy=0x7f26e40ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:14.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.505+0000 7f26e8fba640 1 -- 172.21.15.55:0/1594415556 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26e4158ba0 con 0x7f26e414ec70 2024-03-19T01:52:14.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.509+0000 7f26e8fba640 1 --2- 172.21.15.55:0/1594415556 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26e414ec70 0x7f26e40c8010 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f26e40ca7c0 tx=0x7f26d401ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a08d603cc13ae94 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:14.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.533+0000 7f26e37fe640 1 -- 172.21.15.55:0/1594415556 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d40b5070 con 0x7f26e414ec70 2024-03-19T01:52:14.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.533+0000 7f26e37fe640 1 -- 172.21.15.55:0/1594415556 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26d40a8910 con 0x7f26e414ec70 2024-03-19T01:52:14.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.533+0000 7f26e37fe640 1 -- 172.21.15.55:0/1594415556 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d40a8ce0 con 0x7f26e414ec70 2024-03-19T01:52:14.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.533+0000 7f26eb21e640 1 -- 172.21.15.55:0/1594415556 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26e414ec70 msgr2=0x7f26e40c8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:14.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.533+0000 7f26eb21e640 1 --2- 172.21.15.55:0/1594415556 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26e414ec70 0x7f26e40c8010 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f26e40ca7c0 tx=0x7f26d401ec10 comp rx=0 tx=0).stop 2024-03-19T01:52:14.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.533+0000 7f26eb21e640 1 -- 172.21.15.55:0/1594415556 shutdown_connections 2024-03-19T01:52:14.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.533+0000 7f26eb21e640 1 --2- 172.21.15.55:0/1594415556 >> v2:172.21.15.84:3300/0 conn(0x7f26e4073700 0x7f26e406b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:14.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.533+0000 7f26eb21e640 1 --2- 172.21.15.55:0/1594415556 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26e414ec70 0x7f26e40c8010 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:14.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.533+0000 7f26eb21e640 1 -- 172.21.15.55:0/1594415556 >> 172.21.15.55:0/1594415556 conn(0x7f26e406e670 msgr2=0x7f26e406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:14.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.533+0000 7f26eb21e640 1 -- 172.21.15.55:0/1594415556 shutdown_connections 2024-03-19T01:52:14.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.533+0000 7f26eb21e640 1 -- 172.21.15.55:0/1594415556 wait complete. 2024-03-19T01:52:14.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.533+0000 7f26eb21e640 1 Processor -- start 2024-03-19T01:52:14.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.533+0000 7f26eb21e640 1 -- start start 2024-03-19T01:52:14.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.533+0000 7f26eb21e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f26e40732b0 0x7f26e40c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:14.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.533+0000 7f26eb21e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26e4073700 0x7f26e40c2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:14.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.533+0000 7f26eb21e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26e414ec70 0x7f26e40c30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:14.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.533+0000 7f26eb21e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f26e414f050 con 0x7f26e414ec70 2024-03-19T01:52:14.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.533+0000 7f26eb21e640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f26e4158050 con 0x7f26e4073700 2024-03-19T01:52:14.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.533+0000 7f26eb21e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f26e414f1d0 con 0x7f26e40732b0 2024-03-19T01:52:14.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.537+0000 7f26e8fba640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f26e40732b0 0x7f26e40c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:14.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.537+0000 7f26e3fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26e4073700 0x7f26e40c2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:14.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.537+0000 7f26e3fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26e4073700 0x7f26e40c2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:38688/0 (socket says 172.21.15.55:38688) 2024-03-19T01:52:14.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.537+0000 7f26e3fff640 1 -- 172.21.15.55:0/747538689 learned_addr learned my addr 172.21.15.55:0/747538689 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:14.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.541+0000 7f26e97bb640 1 --2- 172.21.15.55:0/747538689 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26e414ec70 0x7f26e40c30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:14.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.541+0000 7f26e3fff640 1 -- 172.21.15.55:0/747538689 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f26e40732b0 msgr2=0x7f26e40c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:14.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.541+0000 7f26e3fff640 1 --2- 172.21.15.55:0/747538689 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f26e40732b0 0x7f26e40c9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:14.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.541+0000 7f26e3fff640 1 -- 172.21.15.55:0/747538689 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26e414ec70 msgr2=0x7f26e40c30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:14.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.541+0000 7f26e3fff640 1 --2- 172.21.15.55:0/747538689 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26e414ec70 0x7f26e40c30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:14.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.541+0000 7f26e3fff640 1 -- 172.21.15.55:0/747538689 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26e40c4c60 con 0x7f26e4073700 2024-03-19T01:52:14.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.549+0000 7f26e3fff640 1 --2- 172.21.15.55:0/747538689 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26e4073700 0x7f26e40c2bb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f26d800b3e0 tx=0x7f26d800b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:14.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.553+0000 7f26e17fa640 1 -- 172.21.15.55:0/747538689 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d8015020 con 0x7f26e4073700 2024-03-19T01:52:14.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.553+0000 7f26e17fa640 1 -- 172.21.15.55:0/747538689 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26d8011850 con 0x7f26e4073700 2024-03-19T01:52:14.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.553+0000 7f26e17fa640 1 -- 172.21.15.55:0/747538689 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d8011c40 con 0x7f26e4073700 2024-03-19T01:52:14.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.553+0000 7f26eb21e640 1 -- 172.21.15.55:0/747538689 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26e41f3d50 con 0x7f26e4073700 2024-03-19T01:52:14.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.553+0000 7f26eb21e640 1 -- 172.21.15.55:0/747538689 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26e40c4f00 con 0x7f26e4073700 2024-03-19T01:52:14.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.553+0000 7f26eb21e640 1 -- 172.21.15.55:0/747538689 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26e4068a40 con 0x7f26e4073700 2024-03-19T01:52:14.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.849+0000 7f26e17fa640 1 -- 172.21.15.55:0/747538689 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f26d8018020 con 0x7f26e4073700 2024-03-19T01:52:14.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.849+0000 7f26e17fa640 1 --2- 172.21.15.55:0/747538689 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f26d00676e0 0x7f26d0069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:14.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.849+0000 7f26e8fba640 1 --2- 172.21.15.55:0/747538689 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f26d00676e0 0x7f26d0069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:14.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.853+0000 7f26e8fba640 1 --2- 172.21.15.55:0/747538689 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f26d00676e0 0x7f26d0069ba0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f26d40b2000 tx=0x7f26d40aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:14.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.853+0000 7f26e17fa640 1 -- 172.21.15.55:0/747538689 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6822+0+0 (secure 0 0 0) 0x7f26d8011de0 con 0x7f26e4073700 2024-03-19T01:52:14.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:14.873+0000 7f26e17fa640 1 -- 172.21.15.55:0/747538689 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f26d801f070 con 0x7f26e4073700 2024-03-19T01:52:15.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.057+0000 7f26eb21e640 1 -- 172.21.15.55:0/747538689 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f26e414bfc0 con 0x7f26e4073700 2024-03-19T01:52:15.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.161+0000 7f26e17fa640 1 -- 172.21.15.55:0/747538689 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+18904 (secure 0 0 0) 0x7f26d8055ea0 con 0x7f26e4073700 2024-03-19T01:52:15.164 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:52:15.164 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":32,"fsid":"def9d4e5-0bc7-449d-b893-2490cbd10e25","created":"2024-03-19T01:50:02.482954+0000","modified":"2024-03-19T01:52:13.876079+0000","last_up_change":"2024-03-19T01:50:21.735508+0000","last_in_change":"2024-03-19T01:50:12.168944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:50:23.456904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:51:04.578583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:51:06.871659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T01:51:14.274442+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"35d5e3c5-d089-4f38-8961-98d064188f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6825","nonce":2757675043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6827","nonce":2757675043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6831","nonce":2757675043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6829","nonce":2757675043}]},"public_addr":"172.21.15.55:6825/2757675043","cluster_addr":"172.21.15.55:6827/2757675043","heartbeat_back_addr":"172.21.15.55:6831/2757675043","heartbeat_front_addr":"172.21.15.55:6829/2757675043","state":["exists","up"]},{"osd":1,"uuid":"8e58f791-001a-44d0-a464-ca68a479f70c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6817","nonce":543460007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6819","nonce":543460007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6823","nonce":543460007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6821","nonce":543460007}]},"public_addr":"172.21.15.55:6817/543460007","cluster_addr":"172.21.15.55:6819/543460007","heartbeat_back_addr":"172.21.15.55:6823/543460007","heartbeat_front_addr":"172.21.15.55:6821/543460007","state":["exists","up"]},{"osd":2,"uuid":"d44f38b8-67f4-43de-98a6-f3c520acaf10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6801","nonce":1760253550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6803","nonce":1760253550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6807","nonce":1760253550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6805","nonce":1760253550}]},"public_addr":"172.21.15.55:6801/1760253550","cluster_addr":"172.21.15.55:6803/1760253550","heartbeat_back_addr":"172.21.15.55:6807/1760253550","heartbeat_front_addr":"172.21.15.55:6805/1760253550","state":["exists","up"]},{"osd":3,"uuid":"850a7316-2d74-43f0-8649-b5f8d8f6692e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6809","nonce":2797616448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6811","nonce":2797616448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6815","nonce":2797616448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6813","nonce":2797616448}]},"public_addr":"172.21.15.55:6809/2797616448","cluster_addr":"172.21.15.55:6811/2797616448","heartbeat_back_addr":"172.21.15.55:6815/2797616448","heartbeat_front_addr":"172.21.15.55:6813/2797616448","state":["exists","up"]},{"osd":4,"uuid":"4e6152ae-a77d-4bb6-a855-5782ad5411a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6819","nonce":2938436826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6821","nonce":2938436826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6825","nonce":2938436826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6823","nonce":2938436826}]},"public_addr":"172.21.15.84:6819/2938436826","cluster_addr":"172.21.15.84:6821/2938436826","heartbeat_back_addr":"172.21.15.84:6825/2938436826","heartbeat_front_addr":"172.21.15.84:6823/2938436826","state":["exists","up"]},{"osd":5,"uuid":"1217ceba-dd31-4615-96f1-dfaf57160f3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6803","nonce":460717295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6805","nonce":460717295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6809","nonce":460717295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6807","nonce":460717295}]},"public_addr":"172.21.15.84:6803/460717295","cluster_addr":"172.21.15.84:6805/460717295","heartbeat_back_addr":"172.21.15.84:6809/460717295","heartbeat_front_addr":"172.21.15.84:6807/460717295","state":["exists","up"]},{"osd":6,"uuid":"3d9ed78a-2284-4c28-a24a-70eb812c7836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6827","nonce":416123135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6829","nonce":416123135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6833","nonce":416123135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6831","nonce":416123135}]},"public_addr":"172.21.15.84:6827/416123135","cluster_addr":"172.21.15.84:6829/416123135","heartbeat_back_addr":"172.21.15.84:6833/416123135","heartbeat_front_addr":"172.21.15.84:6831/416123135","state":["exists","up"]},{"osd":7,"uuid":"c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6811","nonce":460083528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6813","nonce":460083528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6817","nonce":460083528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6815","nonce":460083528}]},"public_addr":"172.21.15.84:6811/460083528","cluster_addr":"172.21.15.84:6813/460083528","heartbeat_back_addr":"172.21.15.84:6817/460083528","heartbeat_front_addr":"172.21.15.84:6815/460083528","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.269126+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.121001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.075714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.092461+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.194676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.062833+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.316515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.237111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.5","mappings":[{"from":3,"to":0}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":5}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:52:15.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.165+0000 7f26eb21e640 1 -- 172.21.15.55:0/747538689 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f26d00676e0 msgr2=0x7f26d0069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:15.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.165+0000 7f26eb21e640 1 --2- 172.21.15.55:0/747538689 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f26d00676e0 0x7f26d0069ba0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f26d40b2000 tx=0x7f26d40aa070 comp rx=0 tx=0).stop 2024-03-19T01:52:15.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.165+0000 7f26eb21e640 1 -- 172.21.15.55:0/747538689 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26e4073700 msgr2=0x7f26e40c2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:15.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.165+0000 7f26eb21e640 1 --2- 172.21.15.55:0/747538689 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26e4073700 0x7f26e40c2bb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f26d800b3e0 tx=0x7f26d800b8a0 comp rx=0 tx=0).stop 2024-03-19T01:52:15.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.165+0000 7f26eb21e640 1 -- 172.21.15.55:0/747538689 shutdown_connections 2024-03-19T01:52:15.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.165+0000 7f26eb21e640 1 --2- 172.21.15.55:0/747538689 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f26d00676e0 0x7f26d0069ba0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:15.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.165+0000 7f26eb21e640 1 --2- 172.21.15.55:0/747538689 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26e414ec70 0x7f26e40c30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:15.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.165+0000 7f26eb21e640 1 --2- 172.21.15.55:0/747538689 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26e4073700 0x7f26e40c2bb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:15.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.165+0000 7f26eb21e640 1 --2- 172.21.15.55:0/747538689 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f26e40732b0 0x7f26e40c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:15.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.165+0000 7f26eb21e640 1 -- 172.21.15.55:0/747538689 >> 172.21.15.55:0/747538689 conn(0x7f26e406e670 msgr2=0x7f26e406b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:15.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.165+0000 7f26eb21e640 1 -- 172.21.15.55:0/747538689 shutdown_connections 2024-03-19T01:52:15.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.165+0000 7f26eb21e640 1 -- 172.21.15.55:0/747538689 wait complete. 2024-03-19T01:52:15.184 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:52:15.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.297+0000 7f9b6ca35640 1 Processor -- start 2024-03-19T01:52:15.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.297+0000 7f9b6ca35640 1 -- start start 2024-03-19T01:52:15.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.297+0000 7f9b6ca35640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f9b680c96b0 0x7f9b68070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:15.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.297+0000 7f9b6ca35640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b6815e3a0 con 0x7f9b680c9a90 2024-03-19T01:52:15.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.297+0000 7f9b6ca35640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b6815e680 con 0x7f9b681540c0 2024-03-19T01:52:15.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.297+0000 7f9b6ca35640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f9b680c7ad0 con 0x7f9b680c96b0 2024-03-19T01:52:15.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.297+0000 7f9b6659c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f9b680c96b0 0x7f9b68070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:15.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.297+0000 7f9b65d9b640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f9b680c9a90 0x7f9b680c73c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:58446/0 (socket says 172.21.15.55:58446) 2024-03-19T01:52:15.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.297+0000 7f9b65d9b640 1 -- 172.21.15.55:0/3577108531 learned_addr learned my addr 172.21.15.55:0/3577108531 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:52:15.303 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.301+0000 7f9b6659c640 1 -- 172.21.15.55:0/3577108531 >> v1:172.21.15.84:6789/0 conn(0x7f9b681540c0 legacy=0x7f9b6815d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:15.303 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.301+0000 7f9b6659c640 1 -- 172.21.15.55:0/3577108531 >> v1:172.21.15.55:6789/0 conn(0x7f9b680c9a90 legacy=0x7f9b680c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:15.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.301+0000 7f9b6659c640 1 -- 172.21.15.55:0/3577108531 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b6815a330 con 0x7f9b680c96b0 2024-03-19T01:52:15.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.309+0000 7f9b6659c640 1 --2- 172.21.15.55:0/3577108531 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b680c96b0 0x7f9b68070f20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9b580097c0 tx=0x7f9b5801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a151618546ae02 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:15.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.313+0000 7f9b6559a640 1 -- 172.21.15.55:0/3577108531 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b580b0070 con 0x7f9b680c96b0 2024-03-19T01:52:15.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.313+0000 7f9b6559a640 1 -- 172.21.15.55:0/3577108531 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b580ab600 con 0x7f9b680c96b0 2024-03-19T01:52:15.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.313+0000 7f9b6559a640 1 -- 172.21.15.55:0/3577108531 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b580ab9d0 con 0x7f9b680c96b0 2024-03-19T01:52:15.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.313+0000 7f9b6ca35640 1 -- 172.21.15.55:0/3577108531 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b680c96b0 msgr2=0x7f9b68070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:15.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.313+0000 7f9b6ca35640 1 --2- 172.21.15.55:0/3577108531 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b680c96b0 0x7f9b68070f20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9b580097c0 tx=0x7f9b5801c3d0 comp rx=0 tx=0).stop 2024-03-19T01:52:15.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.313+0000 7f9b6ca35640 1 -- 172.21.15.55:0/3577108531 shutdown_connections 2024-03-19T01:52:15.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.313+0000 7f9b6ca35640 1 --2- 172.21.15.55:0/3577108531 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b680c96b0 0x7f9b68070f20 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:15.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.313+0000 7f9b6ca35640 1 -- 172.21.15.55:0/3577108531 >> 172.21.15.55:0/3577108531 conn(0x7f9b680cc9c0 msgr2=0x7f9b680ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:15.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.313+0000 7f9b6ca35640 1 -- 172.21.15.55:0/3577108531 shutdown_connections 2024-03-19T01:52:15.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.313+0000 7f9b6ca35640 1 -- 172.21.15.55:0/3577108531 wait complete. 2024-03-19T01:52:15.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.317+0000 7f9b6ca35640 1 Processor -- start 2024-03-19T01:52:15.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.317+0000 7f9b6ca35640 1 -- start start 2024-03-19T01:52:15.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.317+0000 7f9b6ca35640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b680c96b0 0x7f9b68141640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:15.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.317+0000 7f9b6ca35640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9b680c9a90 0x7f9b681f68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:15.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.317+0000 7f9b6ca35640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b681540c0 0x7f9b68144150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:15.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.317+0000 7f9b6ca35640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6815b0f0 con 0x7f9b681540c0 2024-03-19T01:52:15.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.317+0000 7f9b6ca35640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6815abd0 con 0x7f9b680c96b0 2024-03-19T01:52:15.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.317+0000 7f9b6ca35640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f9b6815df20 con 0x7f9b680c9a90 2024-03-19T01:52:15.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.317+0000 7f9b66d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b681540c0 0x7f9b68144150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:15.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.317+0000 7f9b66d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b681540c0 0x7f9b68144150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33066/0 (socket says 172.21.15.55:33066) 2024-03-19T01:52:15.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.317+0000 7f9b66d9d640 1 -- 172.21.15.55:0/2202444492 learned_addr learned my addr 172.21.15.55:0/2202444492 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:15.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.317+0000 7f9b65d9b640 1 --2- 172.21.15.55:0/2202444492 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9b680c9a90 0x7f9b681f68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:15.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.321+0000 7f9b6659c640 1 --2- 172.21.15.55:0/2202444492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b680c96b0 0x7f9b68141640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:15.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.321+0000 7f9b66d9d640 1 -- 172.21.15.55:0/2202444492 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9b680c9a90 msgr2=0x7f9b681f68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:15.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.321+0000 7f9b66d9d640 1 --2- 172.21.15.55:0/2202444492 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9b680c9a90 0x7f9b681f68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:15.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.321+0000 7f9b66d9d640 1 -- 172.21.15.55:0/2202444492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b680c96b0 msgr2=0x7f9b68141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:15.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.321+0000 7f9b66d9d640 1 --2- 172.21.15.55:0/2202444492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b680c96b0 0x7f9b68141640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:15.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.321+0000 7f9b66d9d640 1 -- 172.21.15.55:0/2202444492 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b68145fc0 con 0x7f9b681540c0 2024-03-19T01:52:15.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.325+0000 7f9b66d9d640 1 --2- 172.21.15.55:0/2202444492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b681540c0 0x7f9b68144150 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9b6000c780 tx=0x7f9b6000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:15.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.385+0000 7f9b56ffd640 1 -- 172.21.15.55:0/2202444492 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b60017070 con 0x7f9b681540c0 2024-03-19T01:52:15.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.385+0000 7f9b56ffd640 1 -- 172.21.15.55:0/2202444492 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b60003db0 con 0x7f9b681540c0 2024-03-19T01:52:15.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.385+0000 7f9b6ca35640 1 -- 172.21.15.55:0/2202444492 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b68147130 con 0x7f9b681540c0 2024-03-19T01:52:15.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.385+0000 7f9b56ffd640 1 -- 172.21.15.55:0/2202444492 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b600052b0 con 0x7f9b681540c0 2024-03-19T01:52:15.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.385+0000 7f9b6ca35640 1 -- 172.21.15.55:0/2202444492 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b681457c0 con 0x7f9b681540c0 2024-03-19T01:52:15.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.385+0000 7f9b6ca35640 1 -- 172.21.15.55:0/2202444492 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b68068a40 con 0x7f9b681540c0 2024-03-19T01:52:15.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.701+0000 7f9b56ffd640 1 -- 172.21.15.55:0/2202444492 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f9b6001a070 con 0x7f9b681540c0 2024-03-19T01:52:15.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.705+0000 7f9b56ffd640 1 --2- 172.21.15.55:0/2202444492 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9b480676b0 0x7f9b48069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:15.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.705+0000 7f9b6659c640 1 --2- 172.21.15.55:0/2202444492 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9b480676b0 0x7f9b48069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:15.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.705+0000 7f9b6659c640 1 --2- 172.21.15.55:0/2202444492 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9b480676b0 0x7f9b48069b70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9b580ad000 tx=0x7f9b58008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:15.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.725+0000 7f9b56ffd640 1 -- 172.21.15.55:0/2202444492 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6822+0+0 (secure 0 0 0) 0x7f9b600539c0 con 0x7f9b681540c0 2024-03-19T01:52:15.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.725+0000 7f9b56ffd640 1 -- 172.21.15.55:0/2202444492 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f9b60021020 con 0x7f9b681540c0 2024-03-19T01:52:15.924 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.921+0000 7f9b6ca35640 1 -- 172.21.15.55:0/2202444492 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b681505d0 con 0x7f9b681540c0 2024-03-19T01:52:15.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.937+0000 7f9b56ffd640 1 -- 172.21.15.55:0/2202444492 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6909 (secure 0 0 0) 0x7f9b60055920 con 0x7f9b681540c0 2024-03-19T01:52:15.940 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:52:15.940 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4200,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:52:05.800317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4249,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4249":{"gid":4249,"name":"f","rank":1,"incarnation":11,"state":"up:active","state_seq":20,"addr":"172.21.15.84:6839/354880008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6838","nonce":354880008},{"type":"v1","addr":"172.21.15.84:6839","nonce":354880008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4249,4237,4255]},"id":1}]} 2024-03-19T01:52:15.940 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 20 2024-03-19T01:52:15.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.941+0000 7f9b6ca35640 1 -- 172.21.15.55:0/2202444492 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9b480676b0 msgr2=0x7f9b48069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:15.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.941+0000 7f9b6ca35640 1 --2- 172.21.15.55:0/2202444492 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9b480676b0 0x7f9b48069b70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9b580ad000 tx=0x7f9b58008040 comp rx=0 tx=0).stop 2024-03-19T01:52:15.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.941+0000 7f9b6ca35640 1 -- 172.21.15.55:0/2202444492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b681540c0 msgr2=0x7f9b68144150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:15.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.941+0000 7f9b6ca35640 1 --2- 172.21.15.55:0/2202444492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b681540c0 0x7f9b68144150 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9b6000c780 tx=0x7f9b6000cc40 comp rx=0 tx=0).stop 2024-03-19T01:52:15.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.941+0000 7f9b6ca35640 1 -- 172.21.15.55:0/2202444492 shutdown_connections 2024-03-19T01:52:15.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.941+0000 7f9b6ca35640 1 --2- 172.21.15.55:0/2202444492 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9b480676b0 0x7f9b48069b70 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:15.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.941+0000 7f9b6ca35640 1 --2- 172.21.15.55:0/2202444492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b681540c0 0x7f9b68144150 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:15.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.941+0000 7f9b6ca35640 1 --2- 172.21.15.55:0/2202444492 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9b680c9a90 0x7f9b681f68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:15.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.941+0000 7f9b6ca35640 1 --2- 172.21.15.55:0/2202444492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b680c96b0 0x7f9b68141640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:15.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.941+0000 7f9b6ca35640 1 -- 172.21.15.55:0/2202444492 >> 172.21.15.55:0/2202444492 conn(0x7f9b680cc9c0 msgr2=0x7f9b680c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:15.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.941+0000 7f9b6ca35640 1 -- 172.21.15.55:0/2202444492 shutdown_connections 2024-03-19T01:52:15.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:15.941+0000 7f9b6ca35640 1 -- 172.21.15.55:0/2202444492 wait complete. 2024-03-19T01:52:16.563 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:52:05.800317+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4249, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4249': {'gid': 4249, 'name': 'f', 'rank': 1, 'incarnation': 11, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.84:6839/354880008', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6838', 'nonce': 354880008}, {'type': 'v1', 'addr': '172.21.15.84:6839', 'nonce': 354880008}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4249, 4237, 4255]} 2024-03-19T01:52:16.564 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-19T01:52:16.564 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-19T01:52:16.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.689+0000 7f0750287640 1 Processor -- start 2024-03-19T01:52:16.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.689+0000 7f0750287640 1 -- start start 2024-03-19T01:52:16.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.693+0000 7f0750287640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f07480ca310 0x7f074815e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:16.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.693+0000 7f0750287640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f074815ad40 0x7f0748154670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:16.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.693+0000 7f0750287640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f074806bc90 con 0x7f0748154290 2024-03-19T01:52:16.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.693+0000 7f0750287640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f074815b2c0 con 0x7f07480ca310 2024-03-19T01:52:16.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.693+0000 7f0750287640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f074806c1d0 con 0x7f074815ad40 2024-03-19T01:52:16.697 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.697+0000 7f074e824640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f074815ad40 0x7f0748154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:16.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.697+0000 7f074e824640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f074815ad40 0x7f0748154670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:38700/0 (socket says 172.21.15.55:38700) 2024-03-19T01:52:16.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.697+0000 7f074e824640 1 -- 172.21.15.55:0/125920430 learned_addr learned my addr 172.21.15.55:0/125920430 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:16.703 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.701+0000 7f074e824640 1 -- 172.21.15.55:0/125920430 >> v2:172.21.15.55:3300/0 conn(0x7f07480ca310 msgr2=0x7f074815e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:16.703 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.701+0000 7f074e824640 1 --2- 172.21.15.55:0/125920430 >> v2:172.21.15.55:3300/0 conn(0x7f07480ca310 0x7f074815e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:16.703 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.701+0000 7f074e824640 1 -- 172.21.15.55:0/125920430 >> v1:172.21.15.55:6789/0 conn(0x7f0748154290 legacy=0x7f0748070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:16.703 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.701+0000 7f074e824640 1 -- 172.21.15.55:0/125920430 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f074806bf30 con 0x7f074815ad40 2024-03-19T01:52:16.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.705+0000 7f074e824640 1 --2- 172.21.15.55:0/125920430 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f074815ad40 0x7f0748154670 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f07480b19f0 tx=0x7f074001c9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=672f35ea7c3e475f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:16.714 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.713+0000 7f074d021640 1 -- 172.21.15.55:0/125920430 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07400a5780 con 0x7f074815ad40 2024-03-19T01:52:16.714 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.713+0000 7f074d021640 1 -- 172.21.15.55:0/125920430 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07400a5920 con 0x7f074815ad40 2024-03-19T01:52:16.714 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.713+0000 7f074d021640 1 -- 172.21.15.55:0/125920430 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07400a5cf0 con 0x7f074815ad40 2024-03-19T01:52:16.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.713+0000 7f0750287640 1 -- 172.21.15.55:0/125920430 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f074815ad40 msgr2=0x7f0748154670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:16.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.713+0000 7f0750287640 1 --2- 172.21.15.55:0/125920430 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f074815ad40 0x7f0748154670 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f07480b19f0 tx=0x7f074001c9b0 comp rx=0 tx=0).stop 2024-03-19T01:52:16.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.713+0000 7f0750287640 1 -- 172.21.15.55:0/125920430 shutdown_connections 2024-03-19T01:52:16.716 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.713+0000 7f0750287640 1 --2- 172.21.15.55:0/125920430 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f074815ad40 0x7f0748154670 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:16.716 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.713+0000 7f0750287640 1 --2- 172.21.15.55:0/125920430 >> v2:172.21.15.55:3300/0 conn(0x7f07480ca310 0x7f074815e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:16.716 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.713+0000 7f0750287640 1 -- 172.21.15.55:0/125920430 >> 172.21.15.55:0/125920430 conn(0x7f074806c580 msgr2=0x7f074806c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:16.716 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.713+0000 7f0750287640 1 -- 172.21.15.55:0/125920430 shutdown_connections 2024-03-19T01:52:16.716 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.713+0000 7f0750287640 1 -- 172.21.15.55:0/125920430 wait complete. 2024-03-19T01:52:16.717 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.717+0000 7f0750287640 1 Processor -- start 2024-03-19T01:52:16.717 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.717+0000 7f0750287640 1 -- start start 2024-03-19T01:52:16.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.717+0000 7f0750287640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07480ca310 0x7f074814e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:16.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.717+0000 7f0750287640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0748154290 0x7f0748152f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:16.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.717+0000 7f0750287640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f074815ad40 0x7f07481503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:16.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.717+0000 7f0750287640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f07480b2460 con 0x7f074815ad40 2024-03-19T01:52:16.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.717+0000 7f0750287640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f07480b22e0 con 0x7f07480ca310 2024-03-19T01:52:16.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.717+0000 7f0750287640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f0748070d50 con 0x7f0748154290 2024-03-19T01:52:16.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.717+0000 7f074d822640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0748154290 0x7f0748152f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:16.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.717+0000 7f074d822640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0748154290 0x7f0748152f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:52652/0 (socket says 172.21.15.55:52652) 2024-03-19T01:52:16.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.717+0000 7f074d822640 1 -- 172.21.15.55:0/919657755 learned_addr learned my addr 172.21.15.55:0/919657755 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:16.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.721+0000 7f074e824640 1 --2- 172.21.15.55:0/919657755 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f074815ad40 0x7f07481503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:16.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.721+0000 7f074e023640 1 --2- 172.21.15.55:0/919657755 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07480ca310 0x7f074814e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:16.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.725+0000 7f074e824640 1 -- 172.21.15.55:0/919657755 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0748154290 msgr2=0x7f0748152f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:16.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.725+0000 7f074e824640 1 --2- 172.21.15.55:0/919657755 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0748154290 0x7f0748152f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:16.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.725+0000 7f074e824640 1 -- 172.21.15.55:0/919657755 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07480ca310 msgr2=0x7f074814e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:16.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.725+0000 7f074e824640 1 --2- 172.21.15.55:0/919657755 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07480ca310 0x7f074814e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:16.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.725+0000 7f074e824640 1 -- 172.21.15.55:0/919657755 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07481515c0 con 0x7f074815ad40 2024-03-19T01:52:16.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.725+0000 7f074e824640 1 --2- 172.21.15.55:0/919657755 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f074815ad40 0x7f07481503c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f07400ab000 tx=0x7f0740019750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:16.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.733+0000 7f073effd640 1 -- 172.21.15.55:0/919657755 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07400b8470 con 0x7f074815ad40 2024-03-19T01:52:16.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.733+0000 7f073effd640 1 -- 172.21.15.55:0/919657755 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07400b8630 con 0x7f074815ad40 2024-03-19T01:52:16.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.733+0000 7f073effd640 1 -- 172.21.15.55:0/919657755 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07400b8a20 con 0x7f074815ad40 2024-03-19T01:52:16.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.733+0000 7f0750287640 1 -- 172.21.15.55:0/919657755 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0748151320 con 0x7f074815ad40 2024-03-19T01:52:16.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.733+0000 7f0750287640 1 -- 172.21.15.55:0/919657755 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07481f3640 con 0x7f074815ad40 2024-03-19T01:52:16.736 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.733+0000 7f0753409000 1 Processor -- start 2024-03-19T01:52:16.736 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.737+0000 7f0753409000 1 -- start start 2024-03-19T01:52:16.737 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.737+0000 7f0753409000 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x559a10d39710 0x559a10d39af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:16.737 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.737+0000 7f0753409000 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559a10d74dd0 0x559a10d751b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:16.737 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.737+0000 7f0753409000 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x559a10d756f0 0x559a10d80060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:16.737 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.737+0000 7f0753409000 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x559a10c1b280 con 0x559a10d74dd0 2024-03-19T01:52:16.737 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.737+0000 7f0753409000 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x559a10b83690 con 0x559a10d756f0 2024-03-19T01:52:16.737 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.737+0000 7f0753409000 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x559a10b0ac50 con 0x559a10d39710 2024-03-19T01:52:16.739 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.737+0000 7f074d822640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x559a10d39710 0x559a10d39af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:16.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.741+0000 7f074d822640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x559a10d39710 0x559a10d39af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:52668/0 (socket says 172.21.15.55:52668) 2024-03-19T01:52:16.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.741+0000 7f074d822640 1 -- 172.21.15.55:0/2996149156 learned_addr learned my addr 172.21.15.55:0/2996149156 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:16.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.741+0000 7f074d822640 1 --2- 172.21.15.55:0/2996149156 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x559a10d756f0 0x559a10d80060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:16.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.745+0000 7f074d822640 1 -- 172.21.15.55:0/2996149156 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x559a10d756f0 msgr2=0x559a10d80060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:16.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.745+0000 7f074d822640 1 --2- 172.21.15.55:0/2996149156 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x559a10d756f0 0x559a10d80060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:16.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.745+0000 7f074d822640 1 -- 172.21.15.55:0/2996149156 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559a10d74dd0 msgr2=0x559a10d751b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:16.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.745+0000 7f074d822640 1 --2- 172.21.15.55:0/2996149156 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559a10d74dd0 0x559a10d751b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:16.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.745+0000 7f074d822640 1 -- 172.21.15.55:0/2996149156 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a10b0aea0 con 0x559a10d39710 2024-03-19T01:52:16.752 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.749+0000 7f074d822640 1 --2- 172.21.15.55:0/2996149156 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x559a10d39710 0x559a10d39af0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0738009f10 tx=0x7f07380095a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46978bb5744d91f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:16.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f072bfff640 1 -- 172.21.15.55:0/2996149156 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0738017470 con 0x559a10d39710 2024-03-19T01:52:16.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f072bfff640 1 -- 172.21.15.55:0/2996149156 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f073801c070 con 0x559a10d39710 2024-03-19T01:52:16.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f072bfff640 1 -- 172.21.15.55:0/2996149156 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0738017830 con 0x559a10d39710 2024-03-19T01:52:16.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f0753409000 1 -- 172.21.15.55:0/2996149156 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x559a10d39710 msgr2=0x559a10d39af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:52:16.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f0753409000 1 --2- 172.21.15.55:0/2996149156 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x559a10d39710 0x559a10d39af0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0738009f10 tx=0x7f07380095a0 comp rx=0 tx=0).stop 2024-03-19T01:52:16.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f0753409000 1 -- 172.21.15.55:0/2996149156 shutdown_connections 2024-03-19T01:52:16.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f0753409000 1 --2- 172.21.15.55:0/2996149156 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x559a10d756f0 0x559a10d80060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:16.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f0753409000 1 --2- 172.21.15.55:0/2996149156 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559a10d74dd0 0x559a10d751b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:16.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f0753409000 1 --2- 172.21.15.55:0/2996149156 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x559a10d39710 0x559a10d39af0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:16.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f0753409000 1 -- 172.21.15.55:0/2996149156 >> 172.21.15.55:0/2996149156 conn(0x559a10d2aed0 msgr2=0x559a10d72b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:52:16.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f0753409000 1 -- 172.21.15.55:0/2996149156 shutdown_connections 2024-03-19T01:52:16.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f0753409000 1 -- 172.21.15.55:0/2996149156 wait complete. 2024-03-19T01:52:16.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f0753409000 1 -- start start 2024-03-19T01:52:16.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f0753409000 1 Processor -- start 2024-03-19T01:52:16.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f0753409000 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x559a10d39710 0x559a10d6e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:16.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f0753409000 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x559a10d74dd0 0x559a10d6eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:16.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f0753409000 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559a10d756f0 0x559a10d34290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:16.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f0753409000 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x559a10b3db40 con 0x559a10d756f0 2024-03-19T01:52:16.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f0753409000 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x559a10b7f730 con 0x559a10d39710 2024-03-19T01:52:16.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f0753409000 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x559a10b4b750 con 0x559a10d74dd0 2024-03-19T01:52:16.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f0753409000 10 client.0 _ll_register_callbacks cb 0x559a10d1f740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:52:16.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.761+0000 7f074d822640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x559a10d39710 0x559a10d6e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:16.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.765+0000 7f074d822640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x559a10d39710 0x559a10d6e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:38716/0 (socket says 172.21.15.55:38716) 2024-03-19T01:52:16.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.765+0000 7f074d822640 1 -- 172.21.15.55:0/2307513310 learned_addr learned my addr 172.21.15.55:0/2307513310 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:52:16.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.765+0000 7f073effd640 1 -- 172.21.15.55:0/919657755 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f07400c0030 con 0x7f074815ad40 2024-03-19T01:52:16.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.765+0000 7f074e023640 1 --2- 172.21.15.55:0/2307513310 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x559a10d74dd0 0x559a10d6eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:16.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.765+0000 7f073effd640 1 --2- 172.21.15.55:0/919657755 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f072c067620 0x7f072c069ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:16.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.765+0000 7f074e824640 1 --2- 172.21.15.55:0/919657755 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f072c067620 0x7f072c069ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:52:16.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.765+0000 7f074e824640 1 --2- 172.21.15.55:0/919657755 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f072c067620 0x7f072c069ae0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f07400f0960 tx=0x7f0740009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:16.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.769+0000 7f074d822640 1 -- 172.21.15.55:0/2307513310 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x559a10d74dd0 msgr2=0x559a10d6eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:16.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.769+0000 7f074d822640 1 --2- 172.21.15.55:0/2307513310 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x559a10d74dd0 0x559a10d6eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:16.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.769+0000 7f074d822640 1 -- 172.21.15.55:0/2307513310 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559a10d756f0 msgr2=0x559a10d34290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:52:16.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.769+0000 7f074d822640 1 --2- 172.21.15.55:0/2307513310 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559a10d756f0 0x559a10d34290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:52:16.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.769+0000 7f074d822640 1 -- 172.21.15.55:0/2307513310 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a10b0b5b0 con 0x559a10d39710 2024-03-19T01:52:16.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.773+0000 7f074d822640 1 --2- 172.21.15.55:0/2307513310 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x559a10d39710 0x559a10d6e570 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x559a10b5e420 tx=0x7f07380046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b2158ac68054b70 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:52:16.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.773+0000 7f07297fa640 10 client.0 ms_handle_connect on v2:172.21.15.84:3300/0 2024-03-19T01:52:16.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.773+0000 7f073effd640 1 -- 172.21.15.55:0/919657755 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6822+0+0 (secure 0 0 0) 0x7f07400f8ba0 con 0x7f074815ad40 2024-03-19T01:52:16.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.777+0000 7f07297fa640 1 -- 172.21.15.55:0/2307513310 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073802c470 con 0x559a10d39710 2024-03-19T01:52:16.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.777+0000 7f07297fa640 1 -- 172.21.15.55:0/2307513310 <== mon.1 v2:172.21.15.84:3300/0 2 ==== osd_map(32..32 src has 1..32) ==== 6822+0+0 (secure 0 0 0) 0x7f073802c610 con 0x559a10d39710 2024-03-19T01:52:16.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.777+0000 7f0753409000 1 -- 172.21.15.55:0/2307513310 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f073801c070 con 0x559a10d39710 2024-03-19T01:52:16.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.781+0000 7f07297fa640 1 -- 172.21.15.55:0/2307513310 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f0738034050 con 0x559a10d39710 2024-03-19T01:52:16.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.781+0000 7f0753409000 10 client.4648 fetch_fsmap learned FSMap version 20 2024-03-19T01:52:16.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.781+0000 7f0753409000 1 -- 172.21.15.55:0/2307513310 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=2+}) -- 0x559a10c048d0 con 0x559a10d39710 2024-03-19T01:52:16.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.805+0000 7f07297fa640 1 -- 172.21.15.55:0/2307513310 <== mon.1 v2:172.21.15.84:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f073801c070 con 0x559a10d39710 2024-03-19T01:52:16.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.805+0000 7f07297fa640 1 -- 172.21.15.55:0/2307513310 <== mon.1 v2:172.21.15.84:3300/0 5 ==== fsmap(e 20) ==== 4792+0+0 (secure 0 0 0) 0x7f0738037020 con 0x559a10d39710 2024-03-19T01:52:16.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.805+0000 7f0753409000 10 client.4648 fetch_fsmap finished waiting for FSMap version 20 2024-03-19T01:52:16.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.805+0000 7f0753409000 10 client.4648 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-19T01:52:16.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.805+0000 7f0753409000 20 client.4648 populate_metadata read hostname 'smithi055' 2024-03-19T01:52:16.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.805+0000 7f0753409000 1 --2- 172.21.15.55:0/2307513310 >> [v2:172.21.15.55:6834/3445165511,v1:172.21.15.55:6835/3445165511] conn(0x559a10d35000 0x559a10d353e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:52:16.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.805+0000 7f0753409000 4 client.4648 mds_command: new command op to 4209 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:52:16.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:16.805+0000 7f0753409000 1 -- 172.21.15.55:0/2307513310 --> [v2:172.21.15.55:6834/3445165511,v1:172.21.15.55:6835/3445165511] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a10b1ebc0 con 0x559a10d35000 2024-03-19T01:52:21.328 INFO:tasks.ceph.mds.c.smithi055.stderr:2024-03-19T01:52:21.321+0000 f9f5640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-03-19T01:52:26.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:26.813+0000 7f074e023640 1 -- 172.21.15.55:0/2307513310 >> [v2:172.21.15.55:6834/3445165511,v1:172.21.15.55:6835/3445165511] conn(0x559a10d35000 msgr2=0x559a10d353e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.55:6834/3445165511, fault. 2024-03-19T01:52:26.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:26.813+0000 7f074e023640 1 --2- 172.21.15.55:0/2307513310 >> [v2:172.21.15.55:6834/3445165511,v1:172.21.15.55:6835/3445165511] conn(0x559a10d35000 0x559a10d353e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T01:52:37.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:37.025+0000 7f074e023640 1 -- 172.21.15.55:0/2307513310 >> [v2:172.21.15.55:6834/3445165511,v1:172.21.15.55:6835/3445165511] conn(0x559a10d35000 msgr2=0x559a10d353e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.55:6834/3445165511, fault. 2024-03-19T01:52:37.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:37.025+0000 7f074e023640 1 --2- 172.21.15.55:0/2307513310 >> [v2:172.21.15.55:6834/3445165511,v1:172.21.15.55:6835/3445165511] conn(0x559a10d35000 0x559a10d353e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T01:52:38.640 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:52:38.643 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:52:47.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:47.437+0000 7f074e023640 1 -- 172.21.15.55:0/2307513310 >> [v2:172.21.15.55:6834/3445165511,v1:172.21.15.55:6835/3445165511] conn(0x559a10d35000 msgr2=0x559a10d353e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.55:6834/3445165511, fault. 2024-03-19T01:52:47.437 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:47.437+0000 7f074e023640 1 --2- 172.21.15.55:0/2307513310 >> [v2:172.21.15.55:6834/3445165511,v1:172.21.15.55:6835/3445165511] conn(0x559a10d35000 0x559a10d353e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T01:52:58.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:58.249+0000 7f074e023640 1 -- 172.21.15.55:0/2307513310 >> [v2:172.21.15.55:6834/3445165511,v1:172.21.15.55:6835/3445165511] conn(0x559a10d35000 msgr2=0x559a10d353e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.55:6834/3445165511, fault. 2024-03-19T01:52:58.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:52:58.249+0000 7f074e023640 1 --2- 172.21.15.55:0/2307513310 >> [v2:172.21.15.55:6834/3445165511,v1:172.21.15.55:6835/3445165511] conn(0x559a10d35000 0x559a10d353e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T01:53:08.703 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:53:08.706 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:53:09.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:53:09.860+0000 7f074e023640 1 -- 172.21.15.55:0/2307513310 >> [v2:172.21.15.55:6834/3445165511,v1:172.21.15.55:6835/3445165511] conn(0x559a10d35000 msgr2=0x559a10d353e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.55:6834/3445165511, fault. 2024-03-19T01:53:09.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:53:09.860+0000 7f074e023640 1 --2- 172.21.15.55:0/2307513310 >> [v2:172.21.15.55:6834/3445165511,v1:172.21.15.55:6835/3445165511] conn(0x559a10d35000 0x559a10d353e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-19T01:53:23.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:53:23.068+0000 7f074e023640 1 -- 172.21.15.55:0/2307513310 >> [v2:172.21.15.55:6834/3445165511,v1:172.21.15.55:6835/3445165511] conn(0x559a10d35000 msgr2=0x559a10d353e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.55:6834/3445165511, fault. 2024-03-19T01:53:23.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:53:23.068+0000 7f074e023640 1 --2- 172.21.15.55:0/2307513310 >> [v2:172.21.15.55:6834/3445165511,v1:172.21.15.55:6835/3445165511] conn(0x559a10d35000 0x559a10d353e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-19T01:53:38.822 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:53:38.825 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:53:39.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:53:39.480+0000 7f074e023640 1 -- 172.21.15.55:0/2307513310 >> [v2:172.21.15.55:6834/3445165511,v1:172.21.15.55:6835/3445165511] conn(0x559a10d35000 msgr2=0x559a10d353e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.55:6834/3445165511, fault. 2024-03-19T01:53:39.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:53:39.480+0000 7f074e023640 1 --2- 172.21.15.55:0/2307513310 >> [v2:172.21.15.55:6834/3445165511,v1:172.21.15.55:6835/3445165511] conn(0x559a10d35000 0x559a10d353e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-19T01:54:02.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:02.300+0000 7f074e023640 1 -- 172.21.15.55:0/2307513310 >> [v2:172.21.15.55:6834/3445165511,v1:172.21.15.55:6835/3445165511] conn(0x559a10d35000 msgr2=0x559a10d353e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.55:6834/3445165511, fault. 2024-03-19T01:54:02.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:02.300+0000 7f074e023640 1 --2- 172.21.15.55:0/2307513310 >> [v2:172.21.15.55:6834/3445165511,v1:172.21.15.55:6835/3445165511] conn(0x559a10d35000 0x559a10d353e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-19T01:54:08.868 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:54:08.871 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:54:16.581 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-19T01:54:17.583 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:54:17.697 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.696+0000 7f46307fe640 1 Processor -- start 2024-03-19T01:54:17.697 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.696+0000 7f46307fe640 1 -- start start 2024-03-19T01:54:17.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.696+0000 7f46307fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f46280c2420 0x7f4628156770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:17.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.696+0000 7f46307fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4628150c70 0x7f4628158510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:17.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.696+0000 7f46307fe640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4628156e70 con 0x7f46280c2800 2024-03-19T01:54:17.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.696+0000 7f46307fe640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f46281576f0 con 0x7f46280c2420 2024-03-19T01:54:17.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.696+0000 7f46307fe640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4628157130 con 0x7f4628150c70 2024-03-19T01:54:17.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.700+0000 7f462e59a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f46280c2420 0x7f4628156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:17.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.700+0000 7f462e59a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f46280c2420 0x7f4628156770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:53766/0 (socket says 172.21.15.55:53766) 2024-03-19T01:54:17.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.700+0000 7f462e59a640 1 -- 172.21.15.55:0/440340590 learned_addr learned my addr 172.21.15.55:0/440340590 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:17.703 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.704+0000 7f462e59a640 1 -- 172.21.15.55:0/440340590 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4628150c70 msgr2=0x7f4628158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:54:17.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.704+0000 7f462e59a640 1 --2- 172.21.15.55:0/440340590 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4628150c70 0x7f4628158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:17.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.704+0000 7f462e59a640 1 -- 172.21.15.55:0/440340590 >> v1:172.21.15.84:6789/0 conn(0x7f46280c2800 legacy=0x7f4628150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:17.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.704+0000 7f462e59a640 1 -- 172.21.15.55:0/440340590 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4628157410 con 0x7f46280c2420 2024-03-19T01:54:17.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.712+0000 7f462e59a640 1 --2- 172.21.15.55:0/440340590 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46280c2420 0x7f4628156770 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4624009870 tx=0x7f462401c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1724c04c7f4523f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:17.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.716+0000 7f462d598640 1 -- 172.21.15.55:0/440340590 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46240a7070 con 0x7f46280c2420 2024-03-19T01:54:17.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.716+0000 7f462d598640 1 -- 172.21.15.55:0/440340590 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46240b0070 con 0x7f46280c2420 2024-03-19T01:54:17.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.716+0000 7f462d598640 1 -- 172.21.15.55:0/440340590 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46240ab890 con 0x7f46280c2420 2024-03-19T01:54:17.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.720+0000 7f46307fe640 1 -- 172.21.15.55:0/440340590 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46280c2420 msgr2=0x7f4628156770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:17.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.720+0000 7f46307fe640 1 --2- 172.21.15.55:0/440340590 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46280c2420 0x7f4628156770 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4624009870 tx=0x7f462401c560 comp rx=0 tx=0).stop 2024-03-19T01:54:17.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.720+0000 7f46307fe640 1 -- 172.21.15.55:0/440340590 shutdown_connections 2024-03-19T01:54:17.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.720+0000 7f46307fe640 1 --2- 172.21.15.55:0/440340590 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4628150c70 0x7f4628158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:17.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.720+0000 7f46307fe640 1 --2- 172.21.15.55:0/440340590 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46280c2420 0x7f4628156770 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:17.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.720+0000 7f46307fe640 1 -- 172.21.15.55:0/440340590 >> 172.21.15.55:0/440340590 conn(0x7f462806f3b0 msgr2=0x7f462806f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:17.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.720+0000 7f46307fe640 1 -- 172.21.15.55:0/440340590 shutdown_connections 2024-03-19T01:54:17.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.720+0000 7f46307fe640 1 -- 172.21.15.55:0/440340590 wait complete. 2024-03-19T01:54:17.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.720+0000 7f46307fe640 1 Processor -- start 2024-03-19T01:54:17.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.720+0000 7f46307fe640 1 -- start start 2024-03-19T01:54:17.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.720+0000 7f46307fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46280c2420 0x7f46280ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:17.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.720+0000 7f46307fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46280c2800 0x7f4628152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:17.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.720+0000 7f46307fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4628150c70 0x7f4628152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:17.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.720+0000 7f46307fe640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4628158e20 con 0x7f46280c2800 2024-03-19T01:54:17.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.720+0000 7f46307fe640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4628158ca0 con 0x7f46280c2420 2024-03-19T01:54:17.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.720+0000 7f46307fe640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4628161010 con 0x7f4628150c70 2024-03-19T01:54:17.722 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.720+0000 7f462e59a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46280c2420 0x7f46280ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:17.722 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.720+0000 7f462dd99640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46280c2800 0x7f4628152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:17.722 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.720+0000 7f462e59a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46280c2420 0x7f46280ca9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:53768/0 (socket says 172.21.15.55:53768) 2024-03-19T01:54:17.722 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.720+0000 7f462e59a640 1 -- 172.21.15.55:0/3397433225 learned_addr learned my addr 172.21.15.55:0/3397433225 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:17.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.724+0000 7f462ed9b640 1 --2- 172.21.15.55:0/3397433225 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4628150c70 0x7f4628152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:17.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.728+0000 7f462e59a640 1 -- 172.21.15.55:0/3397433225 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4628150c70 msgr2=0x7f4628152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:17.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.728+0000 7f462e59a640 1 --2- 172.21.15.55:0/3397433225 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4628150c70 0x7f4628152da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:17.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.728+0000 7f462e59a640 1 -- 172.21.15.55:0/3397433225 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46280c2800 msgr2=0x7f4628152860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:17.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.728+0000 7f462e59a640 1 --2- 172.21.15.55:0/3397433225 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46280c2800 0x7f4628152860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:17.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.728+0000 7f462e59a640 1 -- 172.21.15.55:0/3397433225 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46280c65e0 con 0x7f46280c2420 2024-03-19T01:54:17.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.732+0000 7f462e59a640 1 --2- 172.21.15.55:0/3397433225 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46280c2420 0x7f46280ca9f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f46240ad040 tx=0x7f462401ada0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:17.735 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.736+0000 7f461effd640 1 -- 172.21.15.55:0/3397433225 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46240abe60 con 0x7f46280c2420 2024-03-19T01:54:17.735 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.736+0000 7f46307fe640 1 -- 172.21.15.55:0/3397433225 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46280c68c0 con 0x7f46280c2420 2024-03-19T01:54:17.735 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.736+0000 7f46307fe640 1 -- 172.21.15.55:0/3397433225 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46280c6e80 con 0x7f46280c2420 2024-03-19T01:54:17.736 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.736+0000 7f46307fe640 1 -- 172.21.15.55:0/3397433225 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4628068330 con 0x7f46280c2420 2024-03-19T01:54:17.737 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.736+0000 7f461effd640 1 -- 172.21.15.55:0/3397433225 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46240b0070 con 0x7f46280c2420 2024-03-19T01:54:17.737 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:17.736+0000 7f461effd640 1 -- 172.21.15.55:0/3397433225 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46240b4a20 con 0x7f46280c2420 2024-03-19T01:54:18.026 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:18.024+0000 7f461effd640 1 -- 172.21.15.55:0/3397433225 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f46240b4bc0 con 0x7f46280c2420 2024-03-19T01:54:18.026 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:18.024+0000 7f461effd640 1 --2- 172.21.15.55:0/3397433225 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f460c067660 0x7f460c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:18.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:18.028+0000 7f462dd99640 1 --2- 172.21.15.55:0/3397433225 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f460c067660 0x7f460c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:18.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:18.028+0000 7f462dd99640 1 --2- 172.21.15.55:0/3397433225 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f460c067660 0x7f460c069b20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f462806fba0 tx=0x7f4618009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:18.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:18.032+0000 7f461effd640 1 -- 172.21.15.55:0/3397433225 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f46240ee770 con 0x7f46280c2420 2024-03-19T01:54:18.050 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:18.048+0000 7f461effd640 1 -- 172.21.15.55:0/3397433225 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f46240c4060 con 0x7f46280c2420 2024-03-19T01:54:18.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:18.260+0000 7f46307fe640 1 -- 172.21.15.55:0/3397433225 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4628073480 con 0x7f46280c2420 2024-03-19T01:54:18.284 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:18.284+0000 7f461effd640 1 -- 172.21.15.55:0/3397433225 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7f4624016150 con 0x7f46280c2420 2024-03-19T01:54:18.284 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:54:18.284 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:54:18.285 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:54:18.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:18.284+0000 7f46307fe640 1 -- 172.21.15.55:0/3397433225 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f460c067660 msgr2=0x7f460c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:18.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:18.284+0000 7f46307fe640 1 --2- 172.21.15.55:0/3397433225 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f460c067660 0x7f460c069b20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f462806fba0 tx=0x7f4618009040 comp rx=0 tx=0).stop 2024-03-19T01:54:18.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:18.284+0000 7f46307fe640 1 -- 172.21.15.55:0/3397433225 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46280c2420 msgr2=0x7f46280ca9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:18.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:18.284+0000 7f46307fe640 1 --2- 172.21.15.55:0/3397433225 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46280c2420 0x7f46280ca9f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f46240ad040 tx=0x7f462401ada0 comp rx=0 tx=0).stop 2024-03-19T01:54:18.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:18.284+0000 7f46307fe640 1 -- 172.21.15.55:0/3397433225 shutdown_connections 2024-03-19T01:54:18.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:18.284+0000 7f46307fe640 1 --2- 172.21.15.55:0/3397433225 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f460c067660 0x7f460c069b20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:18.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:18.284+0000 7f46307fe640 1 --2- 172.21.15.55:0/3397433225 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4628150c70 0x7f4628152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:18.288 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:18.284+0000 7f46307fe640 1 --2- 172.21.15.55:0/3397433225 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46280c2800 0x7f4628152860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:18.288 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:18.284+0000 7f46307fe640 1 --2- 172.21.15.55:0/3397433225 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46280c2420 0x7f46280ca9f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:18.288 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:18.284+0000 7f46307fe640 1 -- 172.21.15.55:0/3397433225 >> 172.21.15.55:0/3397433225 conn(0x7f462806f3b0 msgr2=0x7f46280c57f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:18.288 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:18.284+0000 7f46307fe640 1 -- 172.21.15.55:0/3397433225 shutdown_connections 2024-03-19T01:54:18.288 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:18.284+0000 7f46307fe640 1 -- 172.21.15.55:0/3397433225 wait complete. 2024-03-19T01:54:18.908 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:54:18.908 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:54:19.910 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:54:20.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.020+0000 7ff067b8a640 1 Processor -- start 2024-03-19T01:54:20.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.020+0000 7ff067b8a640 1 -- start start 2024-03-19T01:54:20.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.024+0000 7ff067b8a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff060070970 0x7ff0600c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:20.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.024+0000 7ff067b8a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff0600c9600 0x7ff0600c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:20.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.024+0000 7ff067b8a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff06006f6d0 con 0x7ff060152150 2024-03-19T01:54:20.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.024+0000 7ff067b8a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7ff060071130 con 0x7ff0600c9600 2024-03-19T01:54:20.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.024+0000 7ff067b8a640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7ff060070450 con 0x7ff060070970 2024-03-19T01:54:20.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.024+0000 7ff066127640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff0600c9600 0x7ff0600c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:20.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.024+0000 7ff065125640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff060070970 0x7ff0600c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:20.025 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.024+0000 7ff065125640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff060070970 0x7ff0600c90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:60586/0 (socket says 172.21.15.55:60586) 2024-03-19T01:54:20.025 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.024+0000 7ff065125640 1 -- 172.21.15.55:0/1773573181 learned_addr learned my addr 172.21.15.55:0/1773573181 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:20.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.028+0000 7ff065125640 1 -- 172.21.15.55:0/1773573181 >> v2:172.21.15.55:3300/0 conn(0x7ff0600c9600 msgr2=0x7ff0600c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:54:20.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.028+0000 7ff065125640 1 --2- 172.21.15.55:0/1773573181 >> v2:172.21.15.55:3300/0 conn(0x7ff0600c9600 0x7ff0600c2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:20.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.028+0000 7ff065125640 1 -- 172.21.15.55:0/1773573181 >> v1:172.21.15.55:6789/0 conn(0x7ff060152150 legacy=0x7ff060158a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:20.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.028+0000 7ff065125640 1 -- 172.21.15.55:0/1773573181 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff060070e50 con 0x7ff060070970 2024-03-19T01:54:20.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.032+0000 7ff065125640 1 --2- 172.21.15.55:0/1773573181 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff060070970 0x7ff0600c90c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff0600b0d00 tx=0x7ff05001ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50bcac36e2ab4650 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:20.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.040+0000 7ff064924640 1 -- 172.21.15.55:0/1773573181 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0500b5070 con 0x7ff060070970 2024-03-19T01:54:20.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.040+0000 7ff064924640 1 -- 172.21.15.55:0/1773573181 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0500a8a10 con 0x7ff060070970 2024-03-19T01:54:20.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.040+0000 7ff064924640 1 -- 172.21.15.55:0/1773573181 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0500a8de0 con 0x7ff060070970 2024-03-19T01:54:20.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.040+0000 7ff067b8a640 1 -- 172.21.15.55:0/1773573181 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff060070970 msgr2=0x7ff0600c90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:20.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.040+0000 7ff067b8a640 1 --2- 172.21.15.55:0/1773573181 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff060070970 0x7ff0600c90c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff0600b0d00 tx=0x7ff05001ed10 comp rx=0 tx=0).stop 2024-03-19T01:54:20.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.040+0000 7ff067b8a640 1 -- 172.21.15.55:0/1773573181 shutdown_connections 2024-03-19T01:54:20.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.040+0000 7ff067b8a640 1 --2- 172.21.15.55:0/1773573181 >> v2:172.21.15.55:3300/0 conn(0x7ff0600c9600 0x7ff0600c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:20.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.040+0000 7ff067b8a640 1 --2- 172.21.15.55:0/1773573181 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff060070970 0x7ff0600c90c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:20.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.040+0000 7ff067b8a640 1 -- 172.21.15.55:0/1773573181 >> 172.21.15.55:0/1773573181 conn(0x7ff060067600 msgr2=0x7ff0600714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:20.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.040+0000 7ff067b8a640 1 -- 172.21.15.55:0/1773573181 shutdown_connections 2024-03-19T01:54:20.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.040+0000 7ff067b8a640 1 -- 172.21.15.55:0/1773573181 wait complete. 2024-03-19T01:54:20.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.044+0000 7ff067b8a640 1 Processor -- start 2024-03-19T01:54:20.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.044+0000 7ff067b8a640 1 -- start start 2024-03-19T01:54:20.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.044+0000 7ff067b8a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff060070970 0x7ff060154b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:20.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.044+0000 7ff067b8a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0600c9600 0x7ff0600b1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:20.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.044+0000 7ff067b8a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff060152150 0x7ff0601470e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:20.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.044+0000 7ff067b8a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff06006ff30 con 0x7ff060152150 2024-03-19T01:54:20.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.044+0000 7ff067b8a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff06006fc50 con 0x7ff060070970 2024-03-19T01:54:20.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.044+0000 7ff067b8a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff060152530 con 0x7ff0600c9600 2024-03-19T01:54:20.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.044+0000 7ff066127640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff060152150 0x7ff0601470e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:20.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.044+0000 7ff065926640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff060070970 0x7ff060154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:20.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.044+0000 7ff065125640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0600c9600 0x7ff0600b1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:20.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.044+0000 7ff066127640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff060152150 0x7ff0601470e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60538/0 (socket says 172.21.15.55:60538) 2024-03-19T01:54:20.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.044+0000 7ff066127640 1 -- 172.21.15.55:0/1238534854 learned_addr learned my addr 172.21.15.55:0/1238534854 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:20.050 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.048+0000 7ff066127640 1 -- 172.21.15.55:0/1238534854 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0600c9600 msgr2=0x7ff0600b1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:20.050 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.048+0000 7ff066127640 1 --2- 172.21.15.55:0/1238534854 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0600c9600 0x7ff0600b1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:20.050 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.048+0000 7ff066127640 1 -- 172.21.15.55:0/1238534854 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff060070970 msgr2=0x7ff060154b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:20.050 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.048+0000 7ff066127640 1 --2- 172.21.15.55:0/1238534854 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff060070970 0x7ff060154b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:20.050 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.048+0000 7ff066127640 1 -- 172.21.15.55:0/1238534854 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0601550c0 con 0x7ff060152150 2024-03-19T01:54:20.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.052+0000 7ff066127640 1 --2- 172.21.15.55:0/1238534854 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff060152150 0x7ff0601470e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff05800ef90 tx=0x7ff05800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:20.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.056+0000 7ff0567fc640 1 -- 172.21.15.55:0/1238534854 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff058018020 con 0x7ff060152150 2024-03-19T01:54:20.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.056+0000 7ff0567fc640 1 -- 172.21.15.55:0/1238534854 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff058014480 con 0x7ff060152150 2024-03-19T01:54:20.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.056+0000 7ff0567fc640 1 -- 172.21.15.55:0/1238534854 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff058014870 con 0x7ff060152150 2024-03-19T01:54:20.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.056+0000 7ff067b8a640 1 -- 172.21.15.55:0/1238534854 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0601540a0 con 0x7ff060152150 2024-03-19T01:54:20.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.056+0000 7ff067b8a640 1 -- 172.21.15.55:0/1238534854 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff060154660 con 0x7ff060152150 2024-03-19T01:54:20.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.056+0000 7ff067b8a640 1 -- 172.21.15.55:0/1238534854 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff060154860 con 0x7ff060152150 2024-03-19T01:54:20.347 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.348+0000 7ff0567fc640 1 -- 172.21.15.55:0/1238534854 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ff05801b020 con 0x7ff060152150 2024-03-19T01:54:20.347 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.348+0000 7ff0567fc640 1 --2- 172.21.15.55:0/1238534854 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff0440676b0 0x7ff044069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:20.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.348+0000 7ff065926640 1 --2- 172.21.15.55:0/1238534854 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff0440676b0 0x7ff044069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:20.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.348+0000 7ff065926640 1 --2- 172.21.15.55:0/1238534854 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff0440676b0 0x7ff044069b70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff05c007fd0 tx=0x7ff05c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:20.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.384+0000 7ff0567fc640 1 -- 172.21.15.55:0/1238534854 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7ff0580551f0 con 0x7ff060152150 2024-03-19T01:54:20.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.384+0000 7ff0567fc640 1 -- 172.21.15.55:0/1238534854 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ff060154860 con 0x7ff060152150 2024-03-19T01:54:20.591 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.592+0000 7ff067b8a640 1 -- 172.21.15.55:0/1238534854 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff060068a40 con 0x7ff060152150 2024-03-19T01:54:20.621 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.620+0000 7ff0567fc640 1 -- 172.21.15.55:0/1238534854 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7ff058022030 con 0x7ff060152150 2024-03-19T01:54:20.622 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:54:20.622 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:54:20.622 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:54:20.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.624+0000 7ff067b8a640 1 -- 172.21.15.55:0/1238534854 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff0440676b0 msgr2=0x7ff044069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:20.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.624+0000 7ff067b8a640 1 --2- 172.21.15.55:0/1238534854 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff0440676b0 0x7ff044069b70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff05c007fd0 tx=0x7ff05c009040 comp rx=0 tx=0).stop 2024-03-19T01:54:20.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.624+0000 7ff067b8a640 1 -- 172.21.15.55:0/1238534854 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff060152150 msgr2=0x7ff0601470e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:20.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.624+0000 7ff067b8a640 1 --2- 172.21.15.55:0/1238534854 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff060152150 0x7ff0601470e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff05800ef90 tx=0x7ff05800c490 comp rx=0 tx=0).stop 2024-03-19T01:54:20.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.624+0000 7ff067b8a640 1 -- 172.21.15.55:0/1238534854 shutdown_connections 2024-03-19T01:54:20.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.624+0000 7ff067b8a640 1 --2- 172.21.15.55:0/1238534854 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff0440676b0 0x7ff044069b70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:20.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.624+0000 7ff067b8a640 1 --2- 172.21.15.55:0/1238534854 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff060152150 0x7ff0601470e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:20.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.624+0000 7ff067b8a640 1 --2- 172.21.15.55:0/1238534854 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0600c9600 0x7ff0600b1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:20.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.624+0000 7ff067b8a640 1 --2- 172.21.15.55:0/1238534854 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff060070970 0x7ff060154b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:20.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.624+0000 7ff067b8a640 1 -- 172.21.15.55:0/1238534854 >> 172.21.15.55:0/1238534854 conn(0x7ff060067600 msgr2=0x7ff060157090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:20.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.624+0000 7ff067b8a640 1 -- 172.21.15.55:0/1238534854 shutdown_connections 2024-03-19T01:54:20.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:20.624+0000 7ff067b8a640 1 -- 172.21.15.55:0/1238534854 wait complete. 2024-03-19T01:54:21.251 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:54:21.251 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:54:22.252 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:54:22.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.384+0000 7f444b912640 1 Processor -- start 2024-03-19T01:54:22.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.384+0000 7f444b912640 1 -- start start 2024-03-19T01:54:22.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.388+0000 7f444b912640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f44440c98e0 0x7f444415e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:22.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.388+0000 7f444b912640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4444071280 con 0x7f4444154720 2024-03-19T01:54:22.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.388+0000 7f444b912640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f444415d900 con 0x7f4444154340 2024-03-19T01:54:22.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.388+0000 7f444b912640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f4444073840 con 0x7f44440c98e0 2024-03-19T01:54:22.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.388+0000 7f44496ae640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f4444154340 0x7f4444072e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:45948/0 (socket says 172.21.15.55:45948) 2024-03-19T01:54:22.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.388+0000 7f44496ae640 1 -- 172.21.15.55:0/2190358919 learned_addr learned my addr 172.21.15.55:0/2190358919 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:54:22.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.392+0000 7f443bfff640 1 -- 172.21.15.55:0/2190358919 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4201860298 0 0) 0x7f444415d900 con 0x7f4444154340 2024-03-19T01:54:22.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.392+0000 7f443bfff640 1 -- 172.21.15.55:0/2190358919 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4430003530 con 0x7f4444154340 2024-03-19T01:54:22.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.400+0000 7f443bfff640 1 -- 172.21.15.55:0/2190358919 <== mon.1 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2086986123 0 0) 0x7f4430003530 con 0x7f4444154340 2024-03-19T01:54:22.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.400+0000 7f443bfff640 1 -- 172.21.15.55:0/2190358919 >> v2:172.21.15.55:3300/0 conn(0x7f44440c98e0 msgr2=0x7f444415e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:54:22.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.400+0000 7f443bfff640 1 --2- 172.21.15.55:0/2190358919 >> v2:172.21.15.55:3300/0 conn(0x7f44440c98e0 0x7f444415e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:22.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.400+0000 7f443bfff640 1 -- 172.21.15.55:0/2190358919 >> v1:172.21.15.55:6789/0 conn(0x7f4444154720 legacy=0x7f44440c91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:54:22.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.400+0000 7f443bfff640 1 -- 172.21.15.55:0/2190358919 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4444070cc0 con 0x7f4444154340 2024-03-19T01:54:22.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.400+0000 7f443bfff640 1 -- 172.21.15.55:0/2190358919 <== mon.1 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f4440001680 con 0x7f4444154340 2024-03-19T01:54:22.398 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.400+0000 7f443bfff640 1 -- 172.21.15.55:0/2190358919 >> v1:172.21.15.84:6789/0 conn(0x7f4444154340 legacy=0x7f4444072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:22.398 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.400+0000 7f443bfff640 1 --2- 172.21.15.55:0/2190358919 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4430003a10 0x7f4430003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:22.398 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.400+0000 7f443bfff640 1 --2- 172.21.15.55:0/2190358919 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4430004810 0x7f4430004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:22.398 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.400+0000 7f443bfff640 1 --2- 172.21.15.55:0/2190358919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44300052f0 0x7f443000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:22.398 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.400+0000 7f443bfff640 1 -- 172.21.15.55:0/2190358919 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f443000c210 con 0x7f44300052f0 2024-03-19T01:54:22.398 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.400+0000 7f443bfff640 1 -- 172.21.15.55:0/2190358919 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f443000c3c0 con 0x7f4430003a10 2024-03-19T01:54:22.398 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.400+0000 7f443bfff640 1 -- 172.21.15.55:0/2190358919 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f443000c5c0 con 0x7f4430004810 2024-03-19T01:54:22.398 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.400+0000 7f4448ead640 1 --2- 172.21.15.55:0/2190358919 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4430004810 0x7f4430004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:22.401 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.404+0000 7f44496ae640 1 --2- 172.21.15.55:0/2190358919 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4430003a10 0x7f4430003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:22.402 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.404+0000 7f4448ead640 1 -- 172.21.15.55:0/2190358919 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4430003a10 msgr2=0x7f4430003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:54:22.402 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.404+0000 7f4448ead640 1 --2- 172.21.15.55:0/2190358919 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4430003a10 0x7f4430003e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:22.402 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.404+0000 7f4448ead640 1 -- 172.21.15.55:0/2190358919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44300052f0 msgr2=0x7f443000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:54:22.402 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.404+0000 7f4448ead640 1 --2- 172.21.15.55:0/2190358919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44300052f0 0x7f443000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:22.402 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.404+0000 7f4448ead640 1 -- 172.21.15.55:0/2190358919 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4430003530 con 0x7f4430004810 2024-03-19T01:54:22.402 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.404+0000 7f4449eaf640 1 -- 172.21.15.55:0/2190358919 reap_dead start 2024-03-19T01:54:22.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.408+0000 7f4448ead640 1 --2- 172.21.15.55:0/2190358919 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4430004810 0x7f4430004c90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4434007fd0 tx=0x7f443401c2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=846033ac7e4288b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:22.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.416+0000 7f443bfff640 1 -- 172.21.15.55:0/2190358919 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4434001250 con 0x7f4430004810 2024-03-19T01:54:22.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.416+0000 7f443bfff640 1 -- 172.21.15.55:0/2190358919 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4434005270 con 0x7f4430004810 2024-03-19T01:54:22.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.416+0000 7f443bfff640 1 -- 172.21.15.55:0/2190358919 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44340054a0 con 0x7f4430004810 2024-03-19T01:54:22.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.416+0000 7f444b912640 1 -- 172.21.15.55:0/2190358919 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4430004810 msgr2=0x7f4430004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:22.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.416+0000 7f444b912640 1 --2- 172.21.15.55:0/2190358919 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4430004810 0x7f4430004c90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4434007fd0 tx=0x7f443401c2e0 comp rx=0 tx=0).stop 2024-03-19T01:54:22.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.416+0000 7f444b912640 1 -- 172.21.15.55:0/2190358919 shutdown_connections 2024-03-19T01:54:22.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.416+0000 7f444b912640 1 --2- 172.21.15.55:0/2190358919 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4430004810 0x7f4430004c90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:22.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.416+0000 7f444b912640 1 -- 172.21.15.55:0/2190358919 >> 172.21.15.55:0/2190358919 conn(0x7f44440cc9c0 msgr2=0x7f44440ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:22.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.416+0000 7f444b912640 1 -- 172.21.15.55:0/2190358919 shutdown_connections 2024-03-19T01:54:22.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.416+0000 7f444b912640 1 -- 172.21.15.55:0/2190358919 wait complete. 2024-03-19T01:54:22.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.420+0000 7f444b912640 1 Processor -- start 2024-03-19T01:54:22.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.420+0000 7f444b912640 1 -- start start 2024-03-19T01:54:22.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.420+0000 7f444b912640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4430004810 0x7f444415b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:22.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.420+0000 7f444b912640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44441f3de0 0x7f44441efc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:22.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.420+0000 7f444b912640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4444151340 0x7f4444148d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:22.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.420+0000 7f444b912640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f444415dba0 con 0x7f4430004810 2024-03-19T01:54:22.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.420+0000 7f444b912640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f444415df10 con 0x7f44441f3de0 2024-03-19T01:54:22.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.420+0000 7f444b912640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f444415e3a0 con 0x7f4444151340 2024-03-19T01:54:22.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.420+0000 7f44496ae640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4430004810 0x7f444415b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:22.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.424+0000 7f44496ae640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4430004810 0x7f444415b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60562/0 (socket says 172.21.15.55:60562) 2024-03-19T01:54:22.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.424+0000 7f44496ae640 1 -- 172.21.15.55:0/3010989277 learned_addr learned my addr 172.21.15.55:0/3010989277 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:22.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.424+0000 7f4448ead640 1 --2- 172.21.15.55:0/3010989277 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44441f3de0 0x7f44441efc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:22.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.424+0000 7f4449eaf640 1 --2- 172.21.15.55:0/3010989277 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4444151340 0x7f4444148d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:22.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.428+0000 7f44496ae640 1 -- 172.21.15.55:0/3010989277 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4444151340 msgr2=0x7f4444148d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:22.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.428+0000 7f44496ae640 1 --2- 172.21.15.55:0/3010989277 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4444151340 0x7f4444148d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:22.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.428+0000 7f44496ae640 1 -- 172.21.15.55:0/3010989277 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44441f3de0 msgr2=0x7f44441efc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:22.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.428+0000 7f44496ae640 1 --2- 172.21.15.55:0/3010989277 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44441f3de0 0x7f44441efc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:22.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.428+0000 7f44496ae640 1 -- 172.21.15.55:0/3010989277 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44441f3340 con 0x7f4430004810 2024-03-19T01:54:22.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.432+0000 7f44496ae640 1 --2- 172.21.15.55:0/3010989277 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4430004810 0x7f444415b6b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f444000d910 tx=0x7f444000ddd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:22.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.436+0000 7f4439ffb640 1 -- 172.21.15.55:0/3010989277 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4440019070 con 0x7f4430004810 2024-03-19T01:54:22.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.436+0000 7f4439ffb640 1 -- 172.21.15.55:0/3010989277 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44400074e0 con 0x7f4430004810 2024-03-19T01:54:22.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.436+0000 7f444b912640 1 -- 172.21.15.55:0/3010989277 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44441f0170 con 0x7f4430004810 2024-03-19T01:54:22.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.436+0000 7f4439ffb640 1 -- 172.21.15.55:0/3010989277 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44400049f0 con 0x7f4430004810 2024-03-19T01:54:22.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.436+0000 7f444b912640 1 -- 172.21.15.55:0/3010989277 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44441e7360 con 0x7f4430004810 2024-03-19T01:54:22.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.436+0000 7f444b912640 1 -- 172.21.15.55:0/3010989277 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44441e7640 con 0x7f4430004810 2024-03-19T01:54:22.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.468+0000 7f4439ffb640 1 -- 172.21.15.55:0/3010989277 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f444001c070 con 0x7f4430004810 2024-03-19T01:54:22.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.468+0000 7f4439ffb640 1 --2- 172.21.15.55:0/3010989277 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4428067690 0x7f4428069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:22.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.468+0000 7f4448ead640 1 --2- 172.21.15.55:0/3010989277 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4428067690 0x7f4428069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:22.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.468+0000 7f4448ead640 1 --2- 172.21.15.55:0/3010989277 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4428067690 0x7f4428069b50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f44340aa000 tx=0x7f4434001030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:22.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.472+0000 7f4439ffb640 1 -- 172.21.15.55:0/3010989277 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f4440053e00 con 0x7f4430004810 2024-03-19T01:54:22.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:22.768+0000 7f4439ffb640 1 -- 172.21.15.55:0/3010989277 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f44441e7640 con 0x7f4430004810 2024-03-19T01:54:23.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:23.028+0000 7f444b912640 1 -- 172.21.15.55:0/3010989277 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4444068a40 con 0x7f4430004810 2024-03-19T01:54:23.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:23.044+0000 7f4439ffb640 1 -- 172.21.15.55:0/3010989277 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7f4440023020 con 0x7f4430004810 2024-03-19T01:54:23.041 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:54:23.041 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:54:23.041 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:54:23.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:23.044+0000 7f444b912640 1 -- 172.21.15.55:0/3010989277 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4428067690 msgr2=0x7f4428069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:23.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:23.044+0000 7f444b912640 1 --2- 172.21.15.55:0/3010989277 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4428067690 0x7f4428069b50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f44340aa000 tx=0x7f4434001030 comp rx=0 tx=0).stop 2024-03-19T01:54:23.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:23.044+0000 7f444b912640 1 -- 172.21.15.55:0/3010989277 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4430004810 msgr2=0x7f444415b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:23.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:23.044+0000 7f444b912640 1 --2- 172.21.15.55:0/3010989277 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4430004810 0x7f444415b6b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f444000d910 tx=0x7f444000ddd0 comp rx=0 tx=0).stop 2024-03-19T01:54:23.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:23.044+0000 7f444b912640 1 -- 172.21.15.55:0/3010989277 shutdown_connections 2024-03-19T01:54:23.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:23.044+0000 7f444b912640 1 --2- 172.21.15.55:0/3010989277 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4428067690 0x7f4428069b50 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:23.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:23.044+0000 7f444b912640 1 --2- 172.21.15.55:0/3010989277 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4444151340 0x7f4444148d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:23.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:23.044+0000 7f444b912640 1 --2- 172.21.15.55:0/3010989277 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44441f3de0 0x7f44441efc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:23.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:23.044+0000 7f444b912640 1 --2- 172.21.15.55:0/3010989277 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4430004810 0x7f444415b6b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:23.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:23.044+0000 7f444b912640 1 -- 172.21.15.55:0/3010989277 >> 172.21.15.55:0/3010989277 conn(0x7f44440cc9c0 msgr2=0x7f44440ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:23.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:23.044+0000 7f444b912640 1 -- 172.21.15.55:0/3010989277 shutdown_connections 2024-03-19T01:54:23.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:23.044+0000 7f444b912640 1 -- 172.21.15.55:0/3010989277 wait complete. 2024-03-19T01:54:23.663 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:54:23.663 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:54:24.664 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:54:24.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.796+0000 7f2f4e509640 1 Processor -- start 2024-03-19T01:54:24.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.796+0000 7f2f4e509640 1 -- start start 2024-03-19T01:54:24.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.796+0000 7f2f4e509640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f4806c610 0x7f2f4815ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:24.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.796+0000 7f2f4e509640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2f4806c9f0 0x7f2f480c92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:24.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.796+0000 7f2f4e509640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f48070e80 con 0x7f2f480c9800 2024-03-19T01:54:24.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.796+0000 7f2f4e509640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f2f4815b250 con 0x7f2f4806c9f0 2024-03-19T01:54:24.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.796+0000 7f2f4e509640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2f4806fb00 con 0x7f2f4806c610 2024-03-19T01:54:24.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.796+0000 7f2f477fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2f4806c9f0 0x7f2f480c92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:24.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.800+0000 7f2f47fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f4806c610 0x7f2f4815ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:24.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.800+0000 7f2f477fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2f4806c9f0 0x7f2f480c92c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60576/0 (socket says 172.21.15.55:60576) 2024-03-19T01:54:24.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.800+0000 7f2f477fe640 1 -- 172.21.15.55:0/1359512780 learned_addr learned my addr 172.21.15.55:0/1359512780 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:24.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.800+0000 7f2f477fe640 1 -- 172.21.15.55:0/1359512780 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f4806c610 msgr2=0x7f2f4815ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:54:24.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.800+0000 7f2f477fe640 1 --2- 172.21.15.55:0/1359512780 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f4806c610 0x7f2f4815ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:24.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.800+0000 7f2f477fe640 1 -- 172.21.15.55:0/1359512780 >> v1:172.21.15.84:6789/0 conn(0x7f2f480c9800 legacy=0x7f2f481427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:24.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.800+0000 7f2f477fe640 1 -- 172.21.15.55:0/1359512780 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f4806fde0 con 0x7f2f4806c9f0 2024-03-19T01:54:24.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.804+0000 7f2f477fe640 1 --2- 172.21.15.55:0/1359512780 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f4806c9f0 0x7f2f480c92c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c009870 tx=0x7f2f3c01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52d7a4104b4b39d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:24.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.812+0000 7f2f46ffd640 1 -- 172.21.15.55:0/1359512780 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c0b5070 con 0x7f2f4806c9f0 2024-03-19T01:54:24.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.812+0000 7f2f46ffd640 1 -- 172.21.15.55:0/1359512780 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f3c0a8a50 con 0x7f2f4806c9f0 2024-03-19T01:54:24.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.812+0000 7f2f46ffd640 1 -- 172.21.15.55:0/1359512780 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c0a8e20 con 0x7f2f4806c9f0 2024-03-19T01:54:24.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.812+0000 7f2f4e509640 1 -- 172.21.15.55:0/1359512780 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f4806c9f0 msgr2=0x7f2f480c92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:24.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.812+0000 7f2f4e509640 1 --2- 172.21.15.55:0/1359512780 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f4806c9f0 0x7f2f480c92c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c009870 tx=0x7f2f3c01ed50 comp rx=0 tx=0).stop 2024-03-19T01:54:24.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.812+0000 7f2f4e509640 1 -- 172.21.15.55:0/1359512780 shutdown_connections 2024-03-19T01:54:24.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.812+0000 7f2f4e509640 1 --2- 172.21.15.55:0/1359512780 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f4806c9f0 0x7f2f480c92c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:24.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.812+0000 7f2f4e509640 1 --2- 172.21.15.55:0/1359512780 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f4806c610 0x7f2f4815ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:24.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.812+0000 7f2f4e509640 1 -- 172.21.15.55:0/1359512780 >> 172.21.15.55:0/1359512780 conn(0x7f2f48067600 msgr2=0x7f2f48071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:24.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.812+0000 7f2f4e509640 1 -- 172.21.15.55:0/1359512780 shutdown_connections 2024-03-19T01:54:24.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.812+0000 7f2f4e509640 1 -- 172.21.15.55:0/1359512780 wait complete. 2024-03-19T01:54:24.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.812+0000 7f2f4e509640 1 Processor -- start 2024-03-19T01:54:24.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.812+0000 7f2f4e509640 1 -- start start 2024-03-19T01:54:24.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.812+0000 7f2f4e509640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f4806c610 0x7f2f480c5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:24.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.812+0000 7f2f4e509640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f4806c9f0 0x7f2f4815bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:24.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.812+0000 7f2f4e509640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f480c9800 0x7f2f4815c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:24.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.812+0000 7f2f4e509640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2f48070680 con 0x7f2f480c9800 2024-03-19T01:54:24.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.812+0000 7f2f4e509640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2f480703a0 con 0x7f2f4806c610 2024-03-19T01:54:24.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.812+0000 7f2f4e509640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2f48070960 con 0x7f2f4806c9f0 2024-03-19T01:54:24.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.816+0000 7f2f47fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f4806c610 0x7f2f480c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:24.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.816+0000 7f2f4caa6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f480c9800 0x7f2f4815c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:24.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.816+0000 7f2f477fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f4806c9f0 0x7f2f4815bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:24.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.816+0000 7f2f4caa6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f480c9800 0x7f2f4815c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60582/0 (socket says 172.21.15.55:60582) 2024-03-19T01:54:24.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.816+0000 7f2f4caa6640 1 -- 172.21.15.55:0/1918453149 learned_addr learned my addr 172.21.15.55:0/1918453149 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:24.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.820+0000 7f2f4caa6640 1 -- 172.21.15.55:0/1918453149 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f4806c9f0 msgr2=0x7f2f4815bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:24.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.820+0000 7f2f4caa6640 1 --2- 172.21.15.55:0/1918453149 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f4806c9f0 0x7f2f4815bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:24.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.820+0000 7f2f4caa6640 1 -- 172.21.15.55:0/1918453149 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f4806c610 msgr2=0x7f2f480c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:24.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.820+0000 7f2f4caa6640 1 --2- 172.21.15.55:0/1918453149 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f4806c610 0x7f2f480c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:24.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.820+0000 7f2f4caa6640 1 -- 172.21.15.55:0/1918453149 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f4815de90 con 0x7f2f480c9800 2024-03-19T01:54:24.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.824+0000 7f2f4caa6640 1 --2- 172.21.15.55:0/1918453149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f480c9800 0x7f2f4815c220 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2f4000c730 tx=0x7f2f4000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:24.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.828+0000 7f2f44ff9640 1 -- 172.21.15.55:0/1918453149 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f40003b80 con 0x7f2f480c9800 2024-03-19T01:54:24.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.828+0000 7f2f44ff9640 1 -- 172.21.15.55:0/1918453149 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f40003d20 con 0x7f2f480c9800 2024-03-19T01:54:24.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.828+0000 7f2f4e509640 1 -- 172.21.15.55:0/1918453149 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f48158dc0 con 0x7f2f480c9800 2024-03-19T01:54:24.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.828+0000 7f2f44ff9640 1 -- 172.21.15.55:0/1918453149 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f40005310 con 0x7f2f480c9800 2024-03-19T01:54:24.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.828+0000 7f2f4e509640 1 -- 172.21.15.55:0/1918453149 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f4815dbb0 con 0x7f2f480c9800 2024-03-19T01:54:24.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.828+0000 7f2f4e509640 1 -- 172.21.15.55:0/1918453149 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f4813b280 con 0x7f2f480c9800 2024-03-19T01:54:24.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.860+0000 7f2f44ff9640 1 -- 172.21.15.55:0/1918453149 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f2f400056c0 con 0x7f2f480c9800 2024-03-19T01:54:24.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.860+0000 7f2f44ff9640 1 --2- 172.21.15.55:0/1918453149 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f340676b0 0x7f2f34069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:24.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.860+0000 7f2f47fff640 1 --2- 172.21.15.55:0/1918453149 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f340676b0 0x7f2f34069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:24.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:24.864+0000 7f2f47fff640 1 --2- 172.21.15.55:0/1918453149 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f340676b0 0x7f2f34069b70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2f38009b60 tx=0x7f2f38009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:25.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:25.156+0000 7f2f44ff9640 1 -- 172.21.15.55:0/1918453149 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f2f40054a20 con 0x7f2f480c9800 2024-03-19T01:54:25.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:25.156+0000 7f2f44ff9640 1 -- 172.21.15.55:0/1918453149 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2f40056980 con 0x7f2f480c9800 2024-03-19T01:54:25.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:25.376+0000 7f2f4e509640 1 -- 172.21.15.55:0/1918453149 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f4815e4e0 con 0x7f2f480c9800 2024-03-19T01:54:25.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:25.492+0000 7f2f44ff9640 1 -- 172.21.15.55:0/1918453149 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7f2f4005aa20 con 0x7f2f480c9800 2024-03-19T01:54:25.491 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:54:25.491 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:54:25.491 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:54:25.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:25.496+0000 7f2f4e509640 1 -- 172.21.15.55:0/1918453149 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f340676b0 msgr2=0x7f2f34069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:25.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:25.496+0000 7f2f4e509640 1 --2- 172.21.15.55:0/1918453149 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f340676b0 0x7f2f34069b70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2f38009b60 tx=0x7f2f38009190 comp rx=0 tx=0).stop 2024-03-19T01:54:25.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:25.496+0000 7f2f4e509640 1 -- 172.21.15.55:0/1918453149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f480c9800 msgr2=0x7f2f4815c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:25.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:25.496+0000 7f2f4e509640 1 --2- 172.21.15.55:0/1918453149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f480c9800 0x7f2f4815c220 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2f4000c730 tx=0x7f2f4000cbf0 comp rx=0 tx=0).stop 2024-03-19T01:54:25.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:25.496+0000 7f2f4e509640 1 -- 172.21.15.55:0/1918453149 shutdown_connections 2024-03-19T01:54:25.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:25.496+0000 7f2f4e509640 1 --2- 172.21.15.55:0/1918453149 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f340676b0 0x7f2f34069b70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:25.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:25.496+0000 7f2f4e509640 1 --2- 172.21.15.55:0/1918453149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f480c9800 0x7f2f4815c220 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:25.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:25.496+0000 7f2f4e509640 1 --2- 172.21.15.55:0/1918453149 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f4806c9f0 0x7f2f4815bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:25.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:25.496+0000 7f2f4e509640 1 --2- 172.21.15.55:0/1918453149 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f4806c610 0x7f2f480c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:25.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:25.496+0000 7f2f4e509640 1 -- 172.21.15.55:0/1918453149 >> 172.21.15.55:0/1918453149 conn(0x7f2f48067600 msgr2=0x7f2f48159260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:25.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:25.496+0000 7f2f4e509640 1 -- 172.21.15.55:0/1918453149 shutdown_connections 2024-03-19T01:54:25.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:25.496+0000 7f2f4e509640 1 -- 172.21.15.55:0/1918453149 wait complete. 2024-03-19T01:54:26.116 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:54:26.116 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:54:27.117 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:54:27.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.240+0000 7fbb958ec640 1 Processor -- start 2024-03-19T01:54:27.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.240+0000 7fbb958ec640 1 -- start start 2024-03-19T01:54:27.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.240+0000 7fbb958ec640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbb9006ceb0 0x7fbb901427f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:27.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.240+0000 7fbb958ec640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb90070e80 con 0x7fbb9006c9f0 2024-03-19T01:54:27.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.240+0000 7fbb958ec640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb9006fb00 con 0x7fbb9006c610 2024-03-19T01:54:27.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.240+0000 7fbb958ec640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbb9006fde0 con 0x7fbb9006ceb0 2024-03-19T01:54:27.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.240+0000 7fbb8e7fc640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fbb9006c9f0 0x7fbb900c92c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:37002/0 (socket says 172.21.15.55:37002) 2024-03-19T01:54:27.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.240+0000 7fbb8e7fc640 1 -- 172.21.15.55:0/3308284806 learned_addr learned my addr 172.21.15.55:0/3308284806 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:54:27.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.240+0000 7fbb8f7fe640 1 --2- 172.21.15.55:0/3308284806 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbb9006ceb0 0x7fbb901427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:27.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.244+0000 7fbb8dffb640 1 -- 172.21.15.55:0/3308284806 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3047757690 0 0) 0x7fbb90070e80 con 0x7fbb9006c9f0 2024-03-19T01:54:27.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.244+0000 7fbb8dffb640 1 -- 172.21.15.55:0/3308284806 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb78003530 con 0x7fbb9006c9f0 2024-03-19T01:54:27.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.244+0000 7fbb8f7fe640 1 -- 172.21.15.55:0/3308284806 >> v1:172.21.15.84:6789/0 conn(0x7fbb9006c610 legacy=0x7fbb9015ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:27.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.244+0000 7fbb8f7fe640 1 -- 172.21.15.55:0/3308284806 >> v1:172.21.15.55:6789/0 conn(0x7fbb9006c9f0 legacy=0x7fbb900c92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:27.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.244+0000 7fbb8f7fe640 1 -- 172.21.15.55:0/3308284806 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb900700c0 con 0x7fbb9006ceb0 2024-03-19T01:54:27.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.252+0000 7fbb8f7fe640 1 --2- 172.21.15.55:0/3308284806 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbb9006ceb0 0x7fbb901427f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbb88009dc0 tx=0x7fbb8801f170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=886ddf3828fe85ac server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:27.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.260+0000 7fbb8dffb640 1 -- 172.21.15.55:0/3308284806 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb880b4070 con 0x7fbb9006ceb0 2024-03-19T01:54:27.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.260+0000 7fbb8dffb640 1 -- 172.21.15.55:0/3308284806 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb880af600 con 0x7fbb9006ceb0 2024-03-19T01:54:27.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.260+0000 7fbb8dffb640 1 -- 172.21.15.55:0/3308284806 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb880af9d0 con 0x7fbb9006ceb0 2024-03-19T01:54:27.257 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.260+0000 7fbb958ec640 1 -- 172.21.15.55:0/3308284806 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbb9006ceb0 msgr2=0x7fbb901427f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:27.257 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.260+0000 7fbb958ec640 1 --2- 172.21.15.55:0/3308284806 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbb9006ceb0 0x7fbb901427f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbb88009dc0 tx=0x7fbb8801f170 comp rx=0 tx=0).stop 2024-03-19T01:54:27.257 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.260+0000 7fbb958ec640 1 -- 172.21.15.55:0/3308284806 shutdown_connections 2024-03-19T01:54:27.257 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.260+0000 7fbb958ec640 1 --2- 172.21.15.55:0/3308284806 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbb9006ceb0 0x7fbb901427f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:27.257 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.260+0000 7fbb958ec640 1 -- 172.21.15.55:0/3308284806 >> 172.21.15.55:0/3308284806 conn(0x7fbb90067600 msgr2=0x7fbb90071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:27.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.260+0000 7fbb958ec640 1 -- 172.21.15.55:0/3308284806 shutdown_connections 2024-03-19T01:54:27.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.260+0000 7fbb958ec640 1 -- 172.21.15.55:0/3308284806 wait complete. 2024-03-19T01:54:27.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.260+0000 7fbb958ec640 1 Processor -- start 2024-03-19T01:54:27.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.260+0000 7fbb958ec640 1 -- start start 2024-03-19T01:54:27.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.260+0000 7fbb958ec640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb9006c610 0x7fbb901f0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:27.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.260+0000 7fbb958ec640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbb9006c9f0 0x7fbb9013a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:27.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.260+0000 7fbb958ec640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbb9006ceb0 0x7fbb900cb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:27.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.260+0000 7fbb958ec640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbb90070960 con 0x7fbb9006c610 2024-03-19T01:54:27.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.260+0000 7fbb958ec640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fbb90070680 con 0x7fbb9006c9f0 2024-03-19T01:54:27.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.260+0000 7fbb958ec640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbb900c99d0 con 0x7fbb9006ceb0 2024-03-19T01:54:27.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.264+0000 7fbb8effd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb9006c610 0x7fbb901f0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:27.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.264+0000 7fbb8effd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb9006c610 0x7fbb901f0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60596/0 (socket says 172.21.15.55:60596) 2024-03-19T01:54:27.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.264+0000 7fbb8effd640 1 -- 172.21.15.55:0/3041874428 learned_addr learned my addr 172.21.15.55:0/3041874428 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:27.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.264+0000 7fbb8e7fc640 1 --2- 172.21.15.55:0/3041874428 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbb9006c9f0 0x7fbb9013a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:27.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.264+0000 7fbb8f7fe640 1 --2- 172.21.15.55:0/3041874428 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbb9006ceb0 0x7fbb900cb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:27.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.268+0000 7fbb8effd640 1 -- 172.21.15.55:0/3041874428 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbb9006ceb0 msgr2=0x7fbb900cb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:27.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.268+0000 7fbb8effd640 1 --2- 172.21.15.55:0/3041874428 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbb9006ceb0 0x7fbb900cb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:27.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.268+0000 7fbb8effd640 1 -- 172.21.15.55:0/3041874428 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbb9006c9f0 msgr2=0x7fbb9013a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:27.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.268+0000 7fbb8effd640 1 --2- 172.21.15.55:0/3041874428 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbb9006c9f0 0x7fbb9013a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:27.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.268+0000 7fbb8effd640 1 -- 172.21.15.55:0/3041874428 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb901f0e40 con 0x7fbb9006c610 2024-03-19T01:54:27.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.268+0000 7fbb8effd640 1 --2- 172.21.15.55:0/3041874428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb9006c610 0x7fbb901f0900 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000c500 tx=0x7fbb8000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:27.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.272+0000 7fbb77fff640 1 -- 172.21.15.55:0/3041874428 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb80016020 con 0x7fbb9006c610 2024-03-19T01:54:27.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.272+0000 7fbb77fff640 1 -- 172.21.15.55:0/3041874428 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb80005220 con 0x7fbb9006c610 2024-03-19T01:54:27.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.272+0000 7fbb77fff640 1 -- 172.21.15.55:0/3041874428 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb80007030 con 0x7fbb9006c610 2024-03-19T01:54:27.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.272+0000 7fbb958ec640 1 -- 172.21.15.55:0/3041874428 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb9013ab80 con 0x7fbb9006c610 2024-03-19T01:54:27.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.276+0000 7fbb958ec640 1 -- 172.21.15.55:0/3041874428 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb9015e2d0 con 0x7fbb9006c610 2024-03-19T01:54:27.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.276+0000 7fbb958ec640 1 -- 172.21.15.55:0/3041874428 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb90068a40 con 0x7fbb9006c610 2024-03-19T01:54:27.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.560+0000 7fbb77fff640 1 -- 172.21.15.55:0/3041874428 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fbb800071f0 con 0x7fbb9006c610 2024-03-19T01:54:27.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.560+0000 7fbb77fff640 1 --2- 172.21.15.55:0/3041874428 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbb7c0676b0 0x7fbb7c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:27.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.560+0000 7fbb8e7fc640 1 --2- 172.21.15.55:0/3041874428 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbb7c0676b0 0x7fbb7c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:27.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.560+0000 7fbb8e7fc640 1 --2- 172.21.15.55:0/3041874428 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbb7c0676b0 0x7fbb7c069b70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbb9015de80 tx=0x7fbb84007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:27.584 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.588+0000 7fbb77fff640 1 -- 172.21.15.55:0/3041874428 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fbb80054670 con 0x7fbb9006c610 2024-03-19T01:54:27.584 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.588+0000 7fbb77fff640 1 -- 172.21.15.55:0/3041874428 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fbb80020020 con 0x7fbb9006c610 2024-03-19T01:54:27.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.788+0000 7fbb958ec640 1 -- 172.21.15.55:0/3041874428 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb9013b260 con 0x7fbb9006c610 2024-03-19T01:54:27.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.904+0000 7fbb77fff640 1 -- 172.21.15.55:0/3041874428 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7fbb800565d0 con 0x7fbb9006c610 2024-03-19T01:54:27.902 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:54:27.902 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:54:27.902 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:54:27.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.908+0000 7fbb958ec640 1 -- 172.21.15.55:0/3041874428 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbb7c0676b0 msgr2=0x7fbb7c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:27.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.908+0000 7fbb958ec640 1 --2- 172.21.15.55:0/3041874428 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbb7c0676b0 0x7fbb7c069b70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbb9015de80 tx=0x7fbb84007d40 comp rx=0 tx=0).stop 2024-03-19T01:54:27.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.908+0000 7fbb958ec640 1 -- 172.21.15.55:0/3041874428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb9006c610 msgr2=0x7fbb901f0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:27.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.908+0000 7fbb958ec640 1 --2- 172.21.15.55:0/3041874428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb9006c610 0x7fbb901f0900 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000c500 tx=0x7fbb8000c9c0 comp rx=0 tx=0).stop 2024-03-19T01:54:27.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.908+0000 7fbb958ec640 1 -- 172.21.15.55:0/3041874428 shutdown_connections 2024-03-19T01:54:27.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.908+0000 7fbb958ec640 1 --2- 172.21.15.55:0/3041874428 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbb7c0676b0 0x7fbb7c069b70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:27.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.908+0000 7fbb958ec640 1 --2- 172.21.15.55:0/3041874428 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbb9006ceb0 0x7fbb900cb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:27.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.908+0000 7fbb958ec640 1 --2- 172.21.15.55:0/3041874428 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbb9006c9f0 0x7fbb9013a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:27.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.908+0000 7fbb958ec640 1 --2- 172.21.15.55:0/3041874428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb9006c610 0x7fbb901f0900 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:27.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.908+0000 7fbb958ec640 1 -- 172.21.15.55:0/3041874428 >> 172.21.15.55:0/3041874428 conn(0x7fbb90067600 msgr2=0x7fbb9013ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:27.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.908+0000 7fbb958ec640 1 -- 172.21.15.55:0/3041874428 shutdown_connections 2024-03-19T01:54:27.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:27.908+0000 7fbb958ec640 1 -- 172.21.15.55:0/3041874428 wait complete. 2024-03-19T01:54:28.531 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:54:28.531 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:54:29.533 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:54:29.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.664+0000 7fdfdf942640 1 Processor -- start 2024-03-19T01:54:29.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.664+0000 7fdfdf942640 1 -- start start 2024-03-19T01:54:29.661 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.664+0000 7fdfdf942640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdfd81586f0 0x7fdfd8158b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:29.661 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.664+0000 7fdfdf942640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfd8073700 con 0x7fdfd8073270 2024-03-19T01:54:29.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.664+0000 7fdfdf942640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfd8072950 con 0x7fdfd81522e0 2024-03-19T01:54:29.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.664+0000 7fdfdf942640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fdfd8072c30 con 0x7fdfd81586f0 2024-03-19T01:54:29.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.664+0000 7fdfddedf640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fdfd8073270 0x7fdfd806e4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:34060/0 (socket says 172.21.15.55:34060) 2024-03-19T01:54:29.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.664+0000 7fdfddedf640 1 -- 172.21.15.55:0/1400265720 learned_addr learned my addr 172.21.15.55:0/1400265720 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:54:29.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.664+0000 7fdfdcedd640 1 --2- 172.21.15.55:0/1400265720 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdfd81586f0 0x7fdfd8158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:29.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.668+0000 7fdfcffff640 1 -- 172.21.15.55:0/1400265720 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2567748468 0 0) 0x7fdfd8073700 con 0x7fdfd8073270 2024-03-19T01:54:29.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.668+0000 7fdfcffff640 1 -- 172.21.15.55:0/1400265720 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfc4003530 con 0x7fdfd8073270 2024-03-19T01:54:29.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.668+0000 7fdfdcedd640 1 -- 172.21.15.55:0/1400265720 >> v1:172.21.15.84:6789/0 conn(0x7fdfd81522e0 legacy=0x7fdfd8157fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:29.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.668+0000 7fdfdcedd640 1 -- 172.21.15.55:0/1400265720 >> v1:172.21.15.55:6789/0 conn(0x7fdfd8073270 legacy=0x7fdfd806e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:29.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.668+0000 7fdfdcedd640 1 -- 172.21.15.55:0/1400265720 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd8072f10 con 0x7fdfd81586f0 2024-03-19T01:54:29.671 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.676+0000 7fdfdcedd640 1 --2- 172.21.15.55:0/1400265720 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdfd81586f0 0x7fdfd8158b50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdfc80097f0 tx=0x7fdfc801c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c20d920b09c6e1c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:29.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.680+0000 7fdfcffff640 1 -- 172.21.15.55:0/1400265720 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc80ab650 con 0x7fdfd81586f0 2024-03-19T01:54:29.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.684+0000 7fdfcffff640 1 -- 172.21.15.55:0/1400265720 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfc80b0070 con 0x7fdfd81586f0 2024-03-19T01:54:29.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.684+0000 7fdfcffff640 1 -- 172.21.15.55:0/1400265720 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc80ab9f0 con 0x7fdfd81586f0 2024-03-19T01:54:29.680 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.684+0000 7fdfdf942640 1 -- 172.21.15.55:0/1400265720 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdfd81586f0 msgr2=0x7fdfd8158b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:29.680 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.684+0000 7fdfdf942640 1 --2- 172.21.15.55:0/1400265720 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdfd81586f0 0x7fdfd8158b50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdfc80097f0 tx=0x7fdfc801c450 comp rx=0 tx=0).stop 2024-03-19T01:54:29.680 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.684+0000 7fdfdf942640 1 -- 172.21.15.55:0/1400265720 shutdown_connections 2024-03-19T01:54:29.680 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.684+0000 7fdfdf942640 1 --2- 172.21.15.55:0/1400265720 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdfd81586f0 0x7fdfd8158b50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:29.680 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.684+0000 7fdfdf942640 1 -- 172.21.15.55:0/1400265720 >> 172.21.15.55:0/1400265720 conn(0x7fdfd80c8990 msgr2=0x7fdfd80c8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:29.680 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.684+0000 7fdfdf942640 1 -- 172.21.15.55:0/1400265720 shutdown_connections 2024-03-19T01:54:29.680 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.684+0000 7fdfdf942640 1 -- 172.21.15.55:0/1400265720 wait complete. 2024-03-19T01:54:29.681 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.684+0000 7fdfdf942640 1 Processor -- start 2024-03-19T01:54:29.681 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.684+0000 7fdfdf942640 1 -- start start 2024-03-19T01:54:29.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.684+0000 7fdfdf942640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdfd8073270 0x7fdfd8156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:29.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.684+0000 7fdfdf942640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfd81522e0 0x7fdfd813e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:29.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.684+0000 7fdfdf942640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdfd813ef30 0x7fdfd8140710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:29.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.684+0000 7fdfdf942640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd80725c0 con 0x7fdfd81522e0 2024-03-19T01:54:29.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.684+0000 7fdfdf942640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd81526c0 con 0x7fdfd813ef30 2024-03-19T01:54:29.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.684+0000 7fdfdf942640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fdfd8166520 con 0x7fdfd8073270 2024-03-19T01:54:29.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.684+0000 7fdfdcedd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfd81522e0 0x7fdfd813e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:29.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.688+0000 7fdfdcedd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfd81522e0 0x7fdfd813e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:32870/0 (socket says 172.21.15.55:32870) 2024-03-19T01:54:29.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.688+0000 7fdfdcedd640 1 -- 172.21.15.55:0/2116620557 learned_addr learned my addr 172.21.15.55:0/2116620557 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:29.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.688+0000 7fdfddedf640 1 --2- 172.21.15.55:0/2116620557 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdfd813ef30 0x7fdfd8140710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:29.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.688+0000 7fdfdd6de640 1 --2- 172.21.15.55:0/2116620557 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdfd8073270 0x7fdfd8156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:29.686 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.688+0000 7fdfdcedd640 1 -- 172.21.15.55:0/2116620557 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdfd8073270 msgr2=0x7fdfd8156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:29.686 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.688+0000 7fdfdcedd640 1 --2- 172.21.15.55:0/2116620557 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdfd8073270 0x7fdfd8156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:29.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.688+0000 7fdfdcedd640 1 -- 172.21.15.55:0/2116620557 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdfd813ef30 msgr2=0x7fdfd8140710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:29.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.688+0000 7fdfdcedd640 1 --2- 172.21.15.55:0/2116620557 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdfd813ef30 0x7fdfd8140710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:29.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.688+0000 7fdfdcedd640 1 -- 172.21.15.55:0/2116620557 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd813f3c0 con 0x7fdfd81522e0 2024-03-19T01:54:29.689 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.692+0000 7fdfdcedd640 1 --2- 172.21.15.55:0/2116620557 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfd81522e0 0x7fdfd813e9f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdfc80ad040 tx=0x7fdfc80191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:29.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.696+0000 7fdfcdffb640 1 -- 172.21.15.55:0/2116620557 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc80c1070 con 0x7fdfd81522e0 2024-03-19T01:54:29.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.696+0000 7fdfcdffb640 1 -- 172.21.15.55:0/2116620557 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfc8003820 con 0x7fdfd81522e0 2024-03-19T01:54:29.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.696+0000 7fdfdf942640 1 -- 172.21.15.55:0/2116620557 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd814ba70 con 0x7fdfd81522e0 2024-03-19T01:54:29.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.696+0000 7fdfcdffb640 1 -- 172.21.15.55:0/2116620557 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8003bf0 con 0x7fdfd81522e0 2024-03-19T01:54:29.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.696+0000 7fdfdf942640 1 -- 172.21.15.55:0/2116620557 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd814c030 con 0x7fdfd81522e0 2024-03-19T01:54:29.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.696+0000 7fdfdf942640 1 -- 172.21.15.55:0/2116620557 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfd806c230 con 0x7fdfd81522e0 2024-03-19T01:54:29.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.724+0000 7fdfcdffb640 1 -- 172.21.15.55:0/2116620557 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fdfc80c4070 con 0x7fdfd81522e0 2024-03-19T01:54:29.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.724+0000 7fdfcdffb640 1 --2- 172.21.15.55:0/2116620557 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdfbc067660 0x7fdfbc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:29.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.724+0000 7fdfdd6de640 1 --2- 172.21.15.55:0/2116620557 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdfbc067660 0x7fdfbc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:29.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:29.724+0000 7fdfdd6de640 1 --2- 172.21.15.55:0/2116620557 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdfbc067660 0x7fdfbc069b20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4007fd0 tx=0x7fdfd4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:29.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:30.000+0000 7fdfcdffb640 1 -- 172.21.15.55:0/2116620557 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fdfc80edfc0 con 0x7fdfd81522e0 2024-03-19T01:54:29.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:30.000+0000 7fdfcdffb640 1 -- 172.21.15.55:0/2116620557 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fdfc80160d0 con 0x7fdfd81522e0 2024-03-19T01:54:30.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:30.244+0000 7fdfdf942640 1 -- 172.21.15.55:0/2116620557 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfd8068330 con 0x7fdfd81522e0 2024-03-19T01:54:30.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:30.276+0000 7fdfcdffb640 1 -- 172.21.15.55:0/2116620557 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7fdfc80eff20 con 0x7fdfd81522e0 2024-03-19T01:54:30.275 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:54:30.275 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:54:30.275 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:54:30.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:30.284+0000 7fdfdf942640 1 -- 172.21.15.55:0/2116620557 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdfbc067660 msgr2=0x7fdfbc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:30.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:30.284+0000 7fdfdf942640 1 --2- 172.21.15.55:0/2116620557 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdfbc067660 0x7fdfbc069b20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4007fd0 tx=0x7fdfd4009040 comp rx=0 tx=0).stop 2024-03-19T01:54:30.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:30.284+0000 7fdfdf942640 1 -- 172.21.15.55:0/2116620557 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfd81522e0 msgr2=0x7fdfd813e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:30.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:30.284+0000 7fdfdf942640 1 --2- 172.21.15.55:0/2116620557 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfd81522e0 0x7fdfd813e9f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdfc80ad040 tx=0x7fdfc80191f0 comp rx=0 tx=0).stop 2024-03-19T01:54:30.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:30.284+0000 7fdfdf942640 1 -- 172.21.15.55:0/2116620557 shutdown_connections 2024-03-19T01:54:30.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:30.284+0000 7fdfdf942640 1 --2- 172.21.15.55:0/2116620557 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdfbc067660 0x7fdfbc069b20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:30.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:30.284+0000 7fdfdf942640 1 --2- 172.21.15.55:0/2116620557 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdfd813ef30 0x7fdfd8140710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:30.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:30.284+0000 7fdfdf942640 1 --2- 172.21.15.55:0/2116620557 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfd81522e0 0x7fdfd813e9f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:30.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:30.284+0000 7fdfdf942640 1 --2- 172.21.15.55:0/2116620557 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdfd8073270 0x7fdfd8156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:30.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:30.284+0000 7fdfdf942640 1 -- 172.21.15.55:0/2116620557 >> 172.21.15.55:0/2116620557 conn(0x7fdfd80c8990 msgr2=0x7fdfd80c8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:30.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:30.284+0000 7fdfdf942640 1 -- 172.21.15.55:0/2116620557 shutdown_connections 2024-03-19T01:54:30.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:30.284+0000 7fdfdf942640 1 -- 172.21.15.55:0/2116620557 wait complete. 2024-03-19T01:54:30.900 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:54:30.900 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:54:31.901 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:54:32.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.016+0000 7f3eb659f640 1 Processor -- start 2024-03-19T01:54:32.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.016+0000 7f3eb659f640 1 -- start start 2024-03-19T01:54:32.015 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.016+0000 7f3eb659f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3eb0152120 0x7f3eb00c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:32.015 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.016+0000 7f3eb659f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3eb00c9610 0x7f3eb0157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:32.015 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.020+0000 7f3eb659f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3eb006fff0 con 0x7f3eb0070780 2024-03-19T01:54:32.015 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.020+0000 7f3eb659f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f3eb0152500 con 0x7f3eb0152120 2024-03-19T01:54:32.015 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.020+0000 7f3eb659f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3eb0070590 con 0x7f3eb00c9610 2024-03-19T01:54:32.016 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.020+0000 7f3eaf7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3eb00c9610 0x7f3eb0157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:32.016 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.020+0000 7f3eaf7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3eb00c9610 0x7f3eb0157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:58272/0 (socket says 172.21.15.55:58272) 2024-03-19T01:54:32.016 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.020+0000 7f3eaf7fe640 1 -- 172.21.15.55:0/1028814132 learned_addr learned my addr 172.21.15.55:0/1028814132 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:32.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.020+0000 7f3eaf7fe640 1 -- 172.21.15.55:0/1028814132 >> v2:172.21.15.55:3300/0 conn(0x7f3eb0152120 msgr2=0x7f3eb00c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:54:32.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.020+0000 7f3eaf7fe640 1 --2- 172.21.15.55:0/1028814132 >> v2:172.21.15.55:3300/0 conn(0x7f3eb0152120 0x7f3eb00c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:32.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.020+0000 7f3eaf7fe640 1 -- 172.21.15.55:0/1028814132 >> v1:172.21.15.55:6789/0 conn(0x7f3eb0070780 legacy=0x7f3eb0158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:54:32.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.024+0000 7f3eaf7fe640 1 -- 172.21.15.55:0/1028814132 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb00c39c0 con 0x7f3eb00c9610 2024-03-19T01:54:32.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.028+0000 7f3eaf7fe640 1 --2- 172.21.15.55:0/1028814132 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3eb00c9610 0x7f3eb0157fd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4009870 tx=0x7f3ea401c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a66c9b49aeefe97 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:32.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.032+0000 7f3eaeffd640 1 -- 172.21.15.55:0/1028814132 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea40b0070 con 0x7f3eb00c9610 2024-03-19T01:54:32.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.032+0000 7f3eaeffd640 1 -- 172.21.15.55:0/1028814132 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ea40ab600 con 0x7f3eb00c9610 2024-03-19T01:54:32.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.032+0000 7f3eaeffd640 1 -- 172.21.15.55:0/1028814132 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea40ab9d0 con 0x7f3eb00c9610 2024-03-19T01:54:32.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.032+0000 7f3eb659f640 1 -- 172.21.15.55:0/1028814132 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3eb00c9610 msgr2=0x7f3eb0157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:32.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.032+0000 7f3eb659f640 1 --2- 172.21.15.55:0/1028814132 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3eb00c9610 0x7f3eb0157fd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4009870 tx=0x7f3ea401c520 comp rx=0 tx=0).stop 2024-03-19T01:54:32.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.032+0000 7f3eb659f640 1 -- 172.21.15.55:0/1028814132 shutdown_connections 2024-03-19T01:54:32.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.032+0000 7f3eb659f640 1 --2- 172.21.15.55:0/1028814132 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3eb00c9610 0x7f3eb0157fd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:32.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.032+0000 7f3eb659f640 1 --2- 172.21.15.55:0/1028814132 >> v2:172.21.15.55:3300/0 conn(0x7f3eb0152120 0x7f3eb00c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:32.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.032+0000 7f3eb659f640 1 -- 172.21.15.55:0/1028814132 >> 172.21.15.55:0/1028814132 conn(0x7f3eb0067600 msgr2=0x7f3eb0071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:32.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.032+0000 7f3eb659f640 1 -- 172.21.15.55:0/1028814132 shutdown_connections 2024-03-19T01:54:32.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.032+0000 7f3eb659f640 1 -- 172.21.15.55:0/1028814132 wait complete. 2024-03-19T01:54:32.031 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.036+0000 7f3eb659f640 1 Processor -- start 2024-03-19T01:54:32.031 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.036+0000 7f3eb659f640 1 -- start start 2024-03-19T01:54:32.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.036+0000 7f3eb659f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3eb0070780 0x7f3eb006d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:32.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.036+0000 7f3eb659f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3eb00c9610 0x7f3eb0156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:32.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.036+0000 7f3eb659f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3eb0152120 0x7f3eb00cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:32.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.036+0000 7f3eb659f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb00c86d0 con 0x7f3eb0152120 2024-03-19T01:54:32.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.036+0000 7f3eb659f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb00c31c0 con 0x7f3eb0070780 2024-03-19T01:54:32.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.036+0000 7f3eb659f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3eb0165170 con 0x7f3eb00c9610 2024-03-19T01:54:32.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.036+0000 7f3eaf7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3eb00c9610 0x7f3eb0156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:32.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.036+0000 7f3eaffff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3eb0070780 0x7f3eb006d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:32.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.036+0000 7f3eaf7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3eb00c9610 0x7f3eb0156f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:58288/0 (socket says 172.21.15.55:58288) 2024-03-19T01:54:32.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.036+0000 7f3eaf7fe640 1 -- 172.21.15.55:0/381076997 learned_addr learned my addr 172.21.15.55:0/381076997 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:32.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.044+0000 7f3eaf7fe640 1 -- 172.21.15.55:0/381076997 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3eb0070780 msgr2=0x7f3eb006d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:32.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.044+0000 7f3eaf7fe640 1 --2- 172.21.15.55:0/381076997 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3eb0070780 0x7f3eb006d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:32.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.044+0000 7f3eaf7fe640 1 -- 172.21.15.55:0/381076997 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3eb0152120 msgr2=0x7f3eb00cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:32.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.044+0000 7f3eaf7fe640 1 --2- 172.21.15.55:0/381076997 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3eb0152120 0x7f3eb00cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:32.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.044+0000 7f3eaf7fe640 1 -- 172.21.15.55:0/381076997 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb014bea0 con 0x7f3eb00c9610 2024-03-19T01:54:32.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.044+0000 7f3eaf7fe640 1 --2- 172.21.15.55:0/381076997 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3eb00c9610 0x7f3eb0156f30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40ae040 tx=0x7f3ea401ad60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:32.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.052+0000 7f3eacff9640 1 -- 172.21.15.55:0/381076997 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea40b0040 con 0x7f3eb00c9610 2024-03-19T01:54:32.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.052+0000 7f3eacff9640 1 -- 172.21.15.55:0/381076997 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ea40bc070 con 0x7f3eb00c9610 2024-03-19T01:54:32.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.052+0000 7f3eb659f640 1 -- 172.21.15.55:0/381076997 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3eb00cdfa0 con 0x7f3eb00c9610 2024-03-19T01:54:32.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.052+0000 7f3eacff9640 1 -- 172.21.15.55:0/381076997 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea40b6860 con 0x7f3eb00c9610 2024-03-19T01:54:32.050 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.052+0000 7f3eb659f640 1 -- 172.21.15.55:0/381076997 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3eb006c550 con 0x7f3eb00c9610 2024-03-19T01:54:32.050 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.052+0000 7f3eb659f640 1 -- 172.21.15.55:0/381076997 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3eb0068a40 con 0x7f3eb00c9610 2024-03-19T01:54:32.356 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.360+0000 7f3eacff9640 1 -- 172.21.15.55:0/381076997 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f3ea40b6a00 con 0x7f3eb00c9610 2024-03-19T01:54:32.356 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.360+0000 7f3eacff9640 1 --2- 172.21.15.55:0/381076997 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3e9c067690 0x7f3e9c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:32.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.360+0000 7f3eaffff640 1 --2- 172.21.15.55:0/381076997 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3e9c067690 0x7f3e9c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:32.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.360+0000 7f3eaffff640 1 --2- 172.21.15.55:0/381076997 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3e9c067690 0x7f3e9c069b50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3ea00098f0 tx=0x7f3ea0006f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:32.365 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.368+0000 7f3eacff9640 1 -- 172.21.15.55:0/381076997 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f3ea40eeca0 con 0x7f3eb00c9610 2024-03-19T01:54:32.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.388+0000 7f3eacff9640 1 -- 172.21.15.55:0/381076997 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f3eb014bea0 con 0x7f3eb00c9610 2024-03-19T01:54:32.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.608+0000 7f3eb659f640 1 -- 172.21.15.55:0/381076997 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3eb0070db0 con 0x7f3eb00c9610 2024-03-19T01:54:32.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.788+0000 7f3eacff9640 1 -- 172.21.15.55:0/381076997 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7f3ea4016110 con 0x7f3eb00c9610 2024-03-19T01:54:32.785 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:54:32.785 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:54:32.785 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:54:32.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.792+0000 7f3eb659f640 1 -- 172.21.15.55:0/381076997 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3e9c067690 msgr2=0x7f3e9c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:32.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.792+0000 7f3eb659f640 1 --2- 172.21.15.55:0/381076997 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3e9c067690 0x7f3e9c069b50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3ea00098f0 tx=0x7f3ea0006f90 comp rx=0 tx=0).stop 2024-03-19T01:54:32.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.792+0000 7f3eb659f640 1 -- 172.21.15.55:0/381076997 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3eb00c9610 msgr2=0x7f3eb0156f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:32.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.792+0000 7f3eb659f640 1 --2- 172.21.15.55:0/381076997 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3eb00c9610 0x7f3eb0156f30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40ae040 tx=0x7f3ea401ad60 comp rx=0 tx=0).stop 2024-03-19T01:54:32.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.792+0000 7f3eb659f640 1 -- 172.21.15.55:0/381076997 shutdown_connections 2024-03-19T01:54:32.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.792+0000 7f3eb659f640 1 --2- 172.21.15.55:0/381076997 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3e9c067690 0x7f3e9c069b50 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:32.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.792+0000 7f3eb659f640 1 --2- 172.21.15.55:0/381076997 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3eb0152120 0x7f3eb00cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:32.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.792+0000 7f3eb659f640 1 --2- 172.21.15.55:0/381076997 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3eb00c9610 0x7f3eb0156f30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:32.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.792+0000 7f3eb659f640 1 --2- 172.21.15.55:0/381076997 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3eb0070780 0x7f3eb006d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:32.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.792+0000 7f3eb659f640 1 -- 172.21.15.55:0/381076997 >> 172.21.15.55:0/381076997 conn(0x7f3eb0067600 msgr2=0x7f3eb014d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:32.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.792+0000 7f3eb659f640 1 -- 172.21.15.55:0/381076997 shutdown_connections 2024-03-19T01:54:32.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:32.792+0000 7f3eb659f640 1 -- 172.21.15.55:0/381076997 wait complete. 2024-03-19T01:54:33.415 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:54:33.415 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:54:34.416 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:54:34.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.544+0000 7f56e9db1640 1 Processor -- start 2024-03-19T01:54:34.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.544+0000 7f56e9db1640 1 -- start start 2024-03-19T01:54:34.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.548+0000 7f56e9db1640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f56e40c9a90 0x7f56e4070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:34.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.548+0000 7f56e9db1640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f56e41540c0 0x7f56e4154550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:34.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.548+0000 7f56e9db1640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56e415e3a0 con 0x7f56e40c96b0 2024-03-19T01:54:34.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.548+0000 7f56e9db1640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f56e40c7ad0 con 0x7f56e40c9a90 2024-03-19T01:54:34.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.548+0000 7f56e9db1640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f56e415e680 con 0x7f56e41540c0 2024-03-19T01:54:34.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.548+0000 7f56e2ffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f56e40c9a90 0x7f56e4070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:34.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.548+0000 7f56e2ffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f56e40c9a90 0x7f56e4070f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:32886/0 (socket says 172.21.15.55:32886) 2024-03-19T01:54:34.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.548+0000 7f56e2ffd640 1 -- 172.21.15.55:0/916905282 learned_addr learned my addr 172.21.15.55:0/916905282 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:34.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.548+0000 7f56e3fff640 1 --2- 172.21.15.55:0/916905282 >> v2:172.21.15.84:3300/0 conn(0x7f56e41540c0 0x7f56e4154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:34.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.552+0000 7f56e2ffd640 1 -- 172.21.15.55:0/916905282 >> v2:172.21.15.84:3300/0 conn(0x7f56e41540c0 msgr2=0x7f56e4154550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:54:34.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.552+0000 7f56e2ffd640 1 --2- 172.21.15.55:0/916905282 >> v2:172.21.15.84:3300/0 conn(0x7f56e41540c0 0x7f56e4154550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:34.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.552+0000 7f56e2ffd640 1 -- 172.21.15.55:0/916905282 >> v1:172.21.15.55:6789/0 conn(0x7f56e40c96b0 legacy=0x7f56e40c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:34.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.552+0000 7f56e2ffd640 1 -- 172.21.15.55:0/916905282 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56e415a330 con 0x7f56e40c9a90 2024-03-19T01:54:34.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.556+0000 7f56e2ffd640 1 --2- 172.21.15.55:0/916905282 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56e40c9a90 0x7f56e4070f20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f56e4072890 tx=0x7f56d801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b18e482ca21265ae server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:34.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.560+0000 7f56e27fc640 1 -- 172.21.15.55:0/916905282 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d80b2070 con 0x7f56e40c9a90 2024-03-19T01:54:34.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.560+0000 7f56e27fc640 1 -- 172.21.15.55:0/916905282 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56d80ad680 con 0x7f56e40c9a90 2024-03-19T01:54:34.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.560+0000 7f56e27fc640 1 -- 172.21.15.55:0/916905282 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d80ada50 con 0x7f56e40c9a90 2024-03-19T01:54:34.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.560+0000 7f56e9db1640 1 -- 172.21.15.55:0/916905282 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56e40c9a90 msgr2=0x7f56e4070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:34.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.560+0000 7f56e9db1640 1 --2- 172.21.15.55:0/916905282 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56e40c9a90 0x7f56e4070f20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f56e4072890 tx=0x7f56d801c490 comp rx=0 tx=0).stop 2024-03-19T01:54:34.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.560+0000 7f56e9db1640 1 -- 172.21.15.55:0/916905282 shutdown_connections 2024-03-19T01:54:34.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.560+0000 7f56e9db1640 1 --2- 172.21.15.55:0/916905282 >> v2:172.21.15.84:3300/0 conn(0x7f56e41540c0 0x7f56e4154550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:34.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.560+0000 7f56e9db1640 1 --2- 172.21.15.55:0/916905282 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56e40c9a90 0x7f56e4070f20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:34.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.560+0000 7f56e9db1640 1 -- 172.21.15.55:0/916905282 >> 172.21.15.55:0/916905282 conn(0x7f56e40cc9c0 msgr2=0x7f56e40ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:34.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.560+0000 7f56e9db1640 1 -- 172.21.15.55:0/916905282 shutdown_connections 2024-03-19T01:54:34.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.560+0000 7f56e9db1640 1 -- 172.21.15.55:0/916905282 wait complete. 2024-03-19T01:54:34.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.564+0000 7f56e9db1640 1 Processor -- start 2024-03-19T01:54:34.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.564+0000 7f56e9db1640 1 -- start start 2024-03-19T01:54:34.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.564+0000 7f56e9db1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56e40c96b0 0x7f56e4149d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:34.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.564+0000 7f56e9db1640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56e40c9a90 0x7f56e40cba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:34.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.564+0000 7f56e9db1640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f56e41540c0 0x7f56e41489c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:34.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.564+0000 7f56e9db1640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f56e4166b90 con 0x7f56e40c96b0 2024-03-19T01:54:34.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.564+0000 7f56e9db1640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f56e415b0f0 con 0x7f56e41540c0 2024-03-19T01:54:34.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.564+0000 7f56e9db1640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f56e4166d10 con 0x7f56e40c9a90 2024-03-19T01:54:34.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.564+0000 7f56e2ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56e40c9a90 0x7f56e40cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:34.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.564+0000 7f56e3fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f56e41540c0 0x7f56e41489c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:34.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.564+0000 7f56e2ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56e40c9a90 0x7f56e40cba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:58290/0 (socket says 172.21.15.55:58290) 2024-03-19T01:54:34.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.564+0000 7f56e2ffd640 1 -- 172.21.15.55:0/1634496698 learned_addr learned my addr 172.21.15.55:0/1634496698 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:34.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.564+0000 7f56e37fe640 1 --2- 172.21.15.55:0/1634496698 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56e40c96b0 0x7f56e4149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:34.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.568+0000 7f56e2ffd640 1 -- 172.21.15.55:0/1634496698 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f56e41540c0 msgr2=0x7f56e41489c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:34.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.568+0000 7f56e2ffd640 1 --2- 172.21.15.55:0/1634496698 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f56e41540c0 0x7f56e41489c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:34.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.568+0000 7f56e2ffd640 1 -- 172.21.15.55:0/1634496698 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56e40c96b0 msgr2=0x7f56e4149d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:34.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.568+0000 7f56e2ffd640 1 --2- 172.21.15.55:0/1634496698 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56e40c96b0 0x7f56e4149d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:34.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.568+0000 7f56e2ffd640 1 -- 172.21.15.55:0/1634496698 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56e41f3700 con 0x7f56e40c9a90 2024-03-19T01:54:34.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.572+0000 7f56e2ffd640 1 --2- 172.21.15.55:0/1634496698 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56e40c9a90 0x7f56e40cba70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f56d801a0c0 tx=0x7f56d801acd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:34.576 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.580+0000 7f56cbfff640 1 -- 172.21.15.55:0/1634496698 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d80b2040 con 0x7f56e40c9a90 2024-03-19T01:54:34.576 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.580+0000 7f56cbfff640 1 -- 172.21.15.55:0/1634496698 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56d80ad9b0 con 0x7f56e40c9a90 2024-03-19T01:54:34.576 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.580+0000 7f56cbfff640 1 -- 172.21.15.55:0/1634496698 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d80add80 con 0x7f56e40c9a90 2024-03-19T01:54:34.577 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.580+0000 7f56e9db1640 1 -- 172.21.15.55:0/1634496698 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56e41f39c0 con 0x7f56e40c9a90 2024-03-19T01:54:34.577 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.580+0000 7f56e9db1640 1 -- 172.21.15.55:0/1634496698 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f56e4150380 con 0x7f56e40c9a90 2024-03-19T01:54:34.577 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.580+0000 7f56e9db1640 1 -- 172.21.15.55:0/1634496698 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56e4068a40 con 0x7f56e40c9a90 2024-03-19T01:54:34.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.620+0000 7f56cbfff640 1 -- 172.21.15.55:0/1634496698 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f56d80bf030 con 0x7f56e40c9a90 2024-03-19T01:54:34.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.620+0000 7f56cbfff640 1 --2- 172.21.15.55:0/1634496698 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f56d00676b0 0x7f56d0069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:34.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.620+0000 7f56e37fe640 1 --2- 172.21.15.55:0/1634496698 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f56d00676b0 0x7f56d0069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:34.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.620+0000 7f56e37fe640 1 --2- 172.21.15.55:0/1634496698 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f56d00676b0 0x7f56d0069b70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f56d4001770 tx=0x7f56d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:34.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.888+0000 7f56cbfff640 1 -- 172.21.15.55:0/1634496698 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f56d80ad630 con 0x7f56e40c9a90 2024-03-19T01:54:34.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:34.908+0000 7f56cbfff640 1 -- 172.21.15.55:0/1634496698 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f56d80c6060 con 0x7f56e40c9a90 2024-03-19T01:54:35.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:35.148+0000 7f56e9db1640 1 -- 172.21.15.55:0/1634496698 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56e415d5f0 con 0x7f56e40c9a90 2024-03-19T01:54:35.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:35.184+0000 7f56cbfff640 1 -- 172.21.15.55:0/1634496698 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7f56d8016110 con 0x7f56e40c9a90 2024-03-19T01:54:35.183 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:54:35.183 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:54:35.183 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:54:35.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:35.188+0000 7f56e9db1640 1 -- 172.21.15.55:0/1634496698 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f56d00676b0 msgr2=0x7f56d0069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:35.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:35.188+0000 7f56e9db1640 1 --2- 172.21.15.55:0/1634496698 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f56d00676b0 0x7f56d0069b70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f56d4001770 tx=0x7f56d4009040 comp rx=0 tx=0).stop 2024-03-19T01:54:35.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:35.188+0000 7f56e9db1640 1 -- 172.21.15.55:0/1634496698 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56e40c9a90 msgr2=0x7f56e40cba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:35.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:35.188+0000 7f56e9db1640 1 --2- 172.21.15.55:0/1634496698 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56e40c9a90 0x7f56e40cba70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f56d801a0c0 tx=0x7f56d801acd0 comp rx=0 tx=0).stop 2024-03-19T01:54:35.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:35.188+0000 7f56e9db1640 1 -- 172.21.15.55:0/1634496698 shutdown_connections 2024-03-19T01:54:35.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:35.188+0000 7f56e9db1640 1 --2- 172.21.15.55:0/1634496698 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f56d00676b0 0x7f56d0069b70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:35.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:35.188+0000 7f56e9db1640 1 --2- 172.21.15.55:0/1634496698 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f56e41540c0 0x7f56e41489c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:35.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:35.188+0000 7f56e9db1640 1 --2- 172.21.15.55:0/1634496698 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56e40c9a90 0x7f56e40cba70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:35.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:35.188+0000 7f56e9db1640 1 --2- 172.21.15.55:0/1634496698 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56e40c96b0 0x7f56e4149d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:35.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:35.188+0000 7f56e9db1640 1 -- 172.21.15.55:0/1634496698 >> 172.21.15.55:0/1634496698 conn(0x7f56e40cc9c0 msgr2=0x7f56e40c5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:35.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:35.188+0000 7f56e9db1640 1 -- 172.21.15.55:0/1634496698 shutdown_connections 2024-03-19T01:54:35.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:35.188+0000 7f56e9db1640 1 -- 172.21.15.55:0/1634496698 wait complete. 2024-03-19T01:54:35.805 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:54:35.806 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:54:36.807 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:54:36.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.940+0000 7f3d4f3cd640 1 Processor -- start 2024-03-19T01:54:36.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.940+0000 7f3d4f3cd640 1 -- start start 2024-03-19T01:54:36.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.940+0000 7f3d4f3cd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3d48158420 0x7f3d48158800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:36.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.940+0000 7f3d4f3cd640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d4815b740 con 0x7f3d4813de70 2024-03-19T01:54:36.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.940+0000 7f3d4f3cd640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d4815ba20 con 0x7f3d4806f1b0 2024-03-19T01:54:36.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.940+0000 7f3d4f3cd640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f3d4813e250 con 0x7f3d48158420 2024-03-19T01:54:36.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.940+0000 7f3d4d96a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3d48158420 0x7f3d48158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:36.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.940+0000 7f3d4c968640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f3d4806f1b0 0x7f3d480c5810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:42184/0 (socket says 172.21.15.55:42184) 2024-03-19T01:54:36.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.940+0000 7f3d4c968640 1 -- 172.21.15.55:0/2459036015 learned_addr learned my addr 172.21.15.55:0/2459036015 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:54:36.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.944+0000 7f3d4d96a640 1 -- 172.21.15.55:0/2459036015 >> v1:172.21.15.84:6789/0 conn(0x7f3d4806f1b0 legacy=0x7f3d480c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:36.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.944+0000 7f3d4d96a640 1 -- 172.21.15.55:0/2459036015 >> v1:172.21.15.55:6789/0 conn(0x7f3d4813de70 legacy=0x7f3d4806e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:36.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.944+0000 7f3d4d96a640 1 -- 172.21.15.55:0/2459036015 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d4815bd00 con 0x7f3d48158420 2024-03-19T01:54:36.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.948+0000 7f3d4d96a640 1 --2- 172.21.15.55:0/2459036015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3d48158420 0x7f3d48158800 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3d40009d90 tx=0x7f3d4001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a98313fd133c27f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:36.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.956+0000 7f3d3ffff640 1 -- 172.21.15.55:0/2459036015 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d400b2070 con 0x7f3d48158420 2024-03-19T01:54:36.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.956+0000 7f3d3ffff640 1 -- 172.21.15.55:0/2459036015 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d400a5830 con 0x7f3d48158420 2024-03-19T01:54:36.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.956+0000 7f3d3ffff640 1 -- 172.21.15.55:0/2459036015 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d400a5c00 con 0x7f3d48158420 2024-03-19T01:54:36.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.956+0000 7f3d4f3cd640 1 -- 172.21.15.55:0/2459036015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3d48158420 msgr2=0x7f3d48158800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:36.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.956+0000 7f3d4f3cd640 1 --2- 172.21.15.55:0/2459036015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3d48158420 0x7f3d48158800 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3d40009d90 tx=0x7f3d4001c970 comp rx=0 tx=0).stop 2024-03-19T01:54:36.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.956+0000 7f3d4f3cd640 1 -- 172.21.15.55:0/2459036015 shutdown_connections 2024-03-19T01:54:36.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.956+0000 7f3d4f3cd640 1 --2- 172.21.15.55:0/2459036015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3d48158420 0x7f3d48158800 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:36.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.956+0000 7f3d4f3cd640 1 -- 172.21.15.55:0/2459036015 >> 172.21.15.55:0/2459036015 conn(0x7f3d4806fb40 msgr2=0x7f3d4806ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:36.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.956+0000 7f3d4f3cd640 1 -- 172.21.15.55:0/2459036015 shutdown_connections 2024-03-19T01:54:36.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.956+0000 7f3d4f3cd640 1 -- 172.21.15.55:0/2459036015 wait complete. 2024-03-19T01:54:36.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.956+0000 7f3d4f3cd640 1 Processor -- start 2024-03-19T01:54:36.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.956+0000 7f3d4f3cd640 1 -- start start 2024-03-19T01:54:36.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.956+0000 7f3d4f3cd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3d4806f1b0 0x7f3d48162470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:36.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.956+0000 7f3d4f3cd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3d4813de70 0x7f3d481f6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:36.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.956+0000 7f3d4f3cd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d48158420 0x7f3d481f70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:36.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.956+0000 7f3d4f3cd640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3d480c5f20 con 0x7f3d4813de70 2024-03-19T01:54:36.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.956+0000 7f3d4f3cd640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3d48158d40 con 0x7f3d48158420 2024-03-19T01:54:36.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.956+0000 7f3d4f3cd640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3d480c60a0 con 0x7f3d4806f1b0 2024-03-19T01:54:36.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.956+0000 7f3d4c968640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3d4813de70 0x7f3d481f6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:36.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.956+0000 7f3d4d96a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d48158420 0x7f3d481f70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:36.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.960+0000 7f3d4c968640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3d4813de70 0x7f3d481f6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:32920/0 (socket says 172.21.15.55:32920) 2024-03-19T01:54:36.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.960+0000 7f3d4c968640 1 -- 172.21.15.55:0/2296058879 learned_addr learned my addr 172.21.15.55:0/2296058879 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:36.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.960+0000 7f3d4d169640 1 --2- 172.21.15.55:0/2296058879 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3d4806f1b0 0x7f3d48162470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:36.960 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.964+0000 7f3d4c968640 1 -- 172.21.15.55:0/2296058879 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3d4806f1b0 msgr2=0x7f3d48162470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:36.960 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.964+0000 7f3d4c968640 1 --2- 172.21.15.55:0/2296058879 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3d4806f1b0 0x7f3d48162470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:36.960 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.964+0000 7f3d4c968640 1 -- 172.21.15.55:0/2296058879 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d48158420 msgr2=0x7f3d481f70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:36.960 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.964+0000 7f3d4c968640 1 --2- 172.21.15.55:0/2296058879 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d48158420 0x7f3d481f70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:36.960 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.964+0000 7f3d4c968640 1 -- 172.21.15.55:0/2296058879 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d48154520 con 0x7f3d4813de70 2024-03-19T01:54:36.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.964+0000 7f3d4c968640 1 --2- 172.21.15.55:0/2296058879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3d4813de70 0x7f3d481f6bb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3d3800c430 tx=0x7f3d3800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:36.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.968+0000 7f3d3dffb640 1 -- 172.21.15.55:0/2296058879 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d38017070 con 0x7f3d4813de70 2024-03-19T01:54:36.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.968+0000 7f3d3dffb640 1 -- 172.21.15.55:0/2296058879 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d38005cf0 con 0x7f3d4813de70 2024-03-19T01:54:36.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.968+0000 7f3d4f3cd640 1 -- 172.21.15.55:0/2296058879 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d480c3a40 con 0x7f3d4813de70 2024-03-19T01:54:36.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.968+0000 7f3d3dffb640 1 -- 172.21.15.55:0/2296058879 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d380072d0 con 0x7f3d4813de70 2024-03-19T01:54:36.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.968+0000 7f3d4f3cd640 1 -- 172.21.15.55:0/2296058879 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d481529a0 con 0x7f3d4813de70 2024-03-19T01:54:36.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:36.972+0000 7f3d4f3cd640 1 -- 172.21.15.55:0/2296058879 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d48068a40 con 0x7f3d4813de70 2024-03-19T01:54:37.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:37.008+0000 7f3d3dffb640 1 -- 172.21.15.55:0/2296058879 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f3d3801a070 con 0x7f3d4813de70 2024-03-19T01:54:37.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:37.008+0000 7f3d3dffb640 1 --2- 172.21.15.55:0/2296058879 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3d2c0676b0 0x7f3d2c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:37.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:37.008+0000 7f3d4d169640 1 --2- 172.21.15.55:0/2296058879 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3d2c0676b0 0x7f3d2c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:37.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:37.012+0000 7f3d4d169640 1 --2- 172.21.15.55:0/2296058879 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3d2c0676b0 0x7f3d2c069b70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3d44001770 tx=0x7f3d44009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:37.282 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:37.284+0000 7f3d3dffb640 1 -- 172.21.15.55:0/2296058879 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f3d380537c0 con 0x7f3d4813de70 2024-03-19T01:54:37.282 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:37.284+0000 7f3d3dffb640 1 -- 172.21.15.55:0/2296058879 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f3d38055720 con 0x7f3d4813de70 2024-03-19T01:54:37.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:37.528+0000 7f3d4f3cd640 1 -- 172.21.15.55:0/2296058879 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d481526e0 con 0x7f3d4813de70 2024-03-19T01:54:37.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:37.544+0000 7f3d3dffb640 1 -- 172.21.15.55:0/2296058879 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7f3d380597c0 con 0x7f3d4813de70 2024-03-19T01:54:37.541 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:54:37.541 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:54:37.542 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:54:37.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:37.544+0000 7f3d4f3cd640 1 -- 172.21.15.55:0/2296058879 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3d2c0676b0 msgr2=0x7f3d2c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:37.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:37.544+0000 7f3d4f3cd640 1 --2- 172.21.15.55:0/2296058879 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3d2c0676b0 0x7f3d2c069b70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3d44001770 tx=0x7f3d44009040 comp rx=0 tx=0).stop 2024-03-19T01:54:37.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:37.544+0000 7f3d4f3cd640 1 -- 172.21.15.55:0/2296058879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3d4813de70 msgr2=0x7f3d481f6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:37.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:37.544+0000 7f3d4f3cd640 1 --2- 172.21.15.55:0/2296058879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3d4813de70 0x7f3d481f6bb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3d3800c430 tx=0x7f3d3800c8f0 comp rx=0 tx=0).stop 2024-03-19T01:54:37.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:37.544+0000 7f3d4f3cd640 1 -- 172.21.15.55:0/2296058879 shutdown_connections 2024-03-19T01:54:37.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:37.544+0000 7f3d4f3cd640 1 --2- 172.21.15.55:0/2296058879 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3d2c0676b0 0x7f3d2c069b70 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:37.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:37.544+0000 7f3d4f3cd640 1 --2- 172.21.15.55:0/2296058879 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d48158420 0x7f3d481f70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:37.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:37.544+0000 7f3d4f3cd640 1 --2- 172.21.15.55:0/2296058879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3d4813de70 0x7f3d481f6bb0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:37.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:37.544+0000 7f3d4f3cd640 1 --2- 172.21.15.55:0/2296058879 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3d4806f1b0 0x7f3d48162470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:37.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:37.544+0000 7f3d4f3cd640 1 -- 172.21.15.55:0/2296058879 >> 172.21.15.55:0/2296058879 conn(0x7f3d4806fb40 msgr2=0x7f3d48156720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:37.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:37.544+0000 7f3d4f3cd640 1 -- 172.21.15.55:0/2296058879 shutdown_connections 2024-03-19T01:54:37.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:37.544+0000 7f3d4f3cd640 1 -- 172.21.15.55:0/2296058879 wait complete. 2024-03-19T01:54:38.163 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:54:38.163 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:54:38.912 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:54:38.915 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:54:39.164 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:54:39.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.296+0000 7f02d08fb640 1 Processor -- start 2024-03-19T01:54:39.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.296+0000 7f02d08fb640 1 -- start start 2024-03-19T01:54:39.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.300+0000 7f02d08fb640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f02cc0c9000 0x7f02cc0c9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:39.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.300+0000 7f02d08fb640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02cc158d50 con 0x7f02cc1520b0 2024-03-19T01:54:39.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.300+0000 7f02d08fb640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02cc15b720 con 0x7f02cc0c2ad0 2024-03-19T01:54:39.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.300+0000 7f02d08fb640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f02cc15c4e0 con 0x7f02cc0c9000 2024-03-19T01:54:39.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.300+0000 7f02ca59c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f02cc1520b0 0x7f02cc0709a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:58102/0 (socket says 172.21.15.55:58102) 2024-03-19T01:54:39.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.300+0000 7f02ca59c640 1 -- 172.21.15.55:0/3126579345 learned_addr learned my addr 172.21.15.55:0/3126579345 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:54:39.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.304+0000 7f02c1d9b640 1 --2- 172.21.15.55:0/3126579345 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f02cc0c9000 0x7f02cc0c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:39.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.304+0000 7f02c9d9b640 1 -- 172.21.15.55:0/3126579345 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1877266163 0 0) 0x7f02cc158d50 con 0x7f02cc1520b0 2024-03-19T01:54:39.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.304+0000 7f02c9d9b640 1 -- 172.21.15.55:0/3126579345 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02b8003530 con 0x7f02cc1520b0 2024-03-19T01:54:39.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.308+0000 7f02c9d9b640 1 -- 172.21.15.55:0/3126579345 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 367116543 0 0) 0x7f02cc15b720 con 0x7f02cc0c2ad0 2024-03-19T01:54:39.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.308+0000 7f02c9d9b640 1 -- 172.21.15.55:0/3126579345 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02cc158d50 con 0x7f02cc0c2ad0 2024-03-19T01:54:39.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.308+0000 7f02c9d9b640 1 -- 172.21.15.55:0/3126579345 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 895931391 0 0) 0x7f02b8003530 con 0x7f02cc1520b0 2024-03-19T01:54:39.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.308+0000 7f02c9d9b640 1 -- 172.21.15.55:0/3126579345 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f02cc0c9000 msgr2=0x7f02cc0c9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:54:39.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.308+0000 7f02c9d9b640 1 --2- 172.21.15.55:0/3126579345 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f02cc0c9000 0x7f02cc0c9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:39.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.308+0000 7f02c9d9b640 1 -- 172.21.15.55:0/3126579345 >> v1:172.21.15.84:6789/0 conn(0x7f02cc0c2ad0 legacy=0x7f02cc0c2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:39.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.308+0000 7f02c9d9b640 1 -- 172.21.15.55:0/3126579345 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02cc15ba00 con 0x7f02cc1520b0 2024-03-19T01:54:39.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.308+0000 7f02c9d9b640 1 -- 172.21.15.55:0/3126579345 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f02bc001680 con 0x7f02cc1520b0 2024-03-19T01:54:39.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.308+0000 7f02c9d9b640 1 -- 172.21.15.55:0/3126579345 >> v1:172.21.15.55:6789/0 conn(0x7f02cc1520b0 legacy=0x7f02cc0709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:39.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.308+0000 7f02c9d9b640 1 --2- 172.21.15.55:0/3126579345 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02b8003b80 0x7f02b8003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:39.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.308+0000 7f02c9d9b640 1 --2- 172.21.15.55:0/3126579345 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02b8004810 0x7f02b8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:39.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.308+0000 7f02c9d9b640 1 --2- 172.21.15.55:0/3126579345 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f02cc0c9000 0x7f02b80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:39.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.308+0000 7f02c9d9b640 1 -- 172.21.15.55:0/3126579345 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc15c4e0 con 0x7f02b8004810 2024-03-19T01:54:39.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.308+0000 7f02c9d9b640 1 -- 172.21.15.55:0/3126579345 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f02b800ba80 con 0x7f02b8003b80 2024-03-19T01:54:39.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.308+0000 7f02c9d9b640 1 -- 172.21.15.55:0/3126579345 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f02b800bc80 con 0x7f02cc0c9000 2024-03-19T01:54:39.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.312+0000 7f02c1d9b640 1 --2- 172.21.15.55:0/3126579345 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02b8004810 0x7f02b8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:39.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.312+0000 7f02cad9d640 1 --2- 172.21.15.55:0/3126579345 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f02cc0c9000 0x7f02b80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:39.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.316+0000 7f02c1d9b640 1 -- 172.21.15.55:0/3126579345 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f02cc0c9000 msgr2=0x7f02b80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:54:39.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.316+0000 7f02c1d9b640 1 --2- 172.21.15.55:0/3126579345 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f02cc0c9000 0x7f02b80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:39.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.316+0000 7f02c1d9b640 1 -- 172.21.15.55:0/3126579345 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02b8003b80 msgr2=0x7f02b8003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:54:39.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.316+0000 7f02c1d9b640 1 --2- 172.21.15.55:0/3126579345 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02b8003b80 0x7f02b8003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:39.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.316+0000 7f02c1d9b640 1 -- 172.21.15.55:0/3126579345 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b8003530 con 0x7f02b8004810 2024-03-19T01:54:39.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.320+0000 7f02c1d9b640 1 --2- 172.21.15.55:0/3126579345 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02b8004810 0x7f02b8004c90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f02b4009ae0 tx=0x7f02b401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b0c33742fdbeafc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:39.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.324+0000 7f02c9d9b640 1 -- 172.21.15.55:0/3126579345 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b40b2070 con 0x7f02b8004810 2024-03-19T01:54:39.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.324+0000 7f02c9d9b640 1 -- 172.21.15.55:0/3126579345 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02b40a5660 con 0x7f02b8004810 2024-03-19T01:54:39.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.324+0000 7f02c9d9b640 1 -- 172.21.15.55:0/3126579345 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b40096b0 con 0x7f02b8004810 2024-03-19T01:54:39.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.324+0000 7f02d08fb640 1 -- 172.21.15.55:0/3126579345 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02b8004810 msgr2=0x7f02b8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:39.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.324+0000 7f02d08fb640 1 --2- 172.21.15.55:0/3126579345 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02b8004810 0x7f02b8004c90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f02b4009ae0 tx=0x7f02b401ca70 comp rx=0 tx=0).stop 2024-03-19T01:54:39.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.324+0000 7f02cad9d640 1 -- 172.21.15.55:0/3126579345 reap_dead start 2024-03-19T01:54:39.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.324+0000 7f02d08fb640 1 -- 172.21.15.55:0/3126579345 shutdown_connections 2024-03-19T01:54:39.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.324+0000 7f02d08fb640 1 -- 172.21.15.55:0/3126579345 >> 172.21.15.55:0/3126579345 conn(0x7f02cc067600 msgr2=0x7f02cc071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:39.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.324+0000 7f02d08fb640 1 -- 172.21.15.55:0/3126579345 shutdown_connections 2024-03-19T01:54:39.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.324+0000 7f02d08fb640 1 -- 172.21.15.55:0/3126579345 wait complete. 2024-03-19T01:54:39.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.324+0000 7f02d08fb640 1 Processor -- start 2024-03-19T01:54:39.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.324+0000 7f02d08fb640 1 -- start start 2024-03-19T01:54:39.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.324+0000 7f02d08fb640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02b8004810 0x7f02cc14fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:39.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.324+0000 7f02d08fb640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f02cc14cea0 0x7f02cc144410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:39.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.324+0000 7f02d08fb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02cc14d2e0 0x7f02cc144950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:39.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.324+0000 7f02d08fb640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc0710b0 con 0x7f02cc14d2e0 2024-03-19T01:54:39.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.324+0000 7f02d08fb640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc15c050 con 0x7f02b8004810 2024-03-19T01:54:39.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.324+0000 7f02d08fb640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f02cc1589a0 con 0x7f02cc14cea0 2024-03-19T01:54:39.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.324+0000 7f02cad9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02cc14d2e0 0x7f02cc144950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:39.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.324+0000 7f02c1d9b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f02cc14cea0 0x7f02cc144410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:39.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.328+0000 7f02ca59c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02b8004810 0x7f02cc14fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:39.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.328+0000 7f02cad9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02cc14d2e0 0x7f02cc144950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:46486/0 (socket says 172.21.15.55:46486) 2024-03-19T01:54:39.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.328+0000 7f02cad9d640 1 -- 172.21.15.55:0/3798010460 learned_addr learned my addr 172.21.15.55:0/3798010460 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:39.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.328+0000 7f02cad9d640 1 -- 172.21.15.55:0/3798010460 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f02cc14cea0 msgr2=0x7f02cc144410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:39.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.328+0000 7f02cad9d640 1 --2- 172.21.15.55:0/3798010460 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f02cc14cea0 0x7f02cc144410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:39.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.328+0000 7f02cad9d640 1 -- 172.21.15.55:0/3798010460 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02b8004810 msgr2=0x7f02cc14fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:39.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.328+0000 7f02cad9d640 1 --2- 172.21.15.55:0/3798010460 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02b8004810 0x7f02cc14fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:39.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.328+0000 7f02cad9d640 1 -- 172.21.15.55:0/3798010460 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02cc1eff20 con 0x7f02cc14d2e0 2024-03-19T01:54:39.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.332+0000 7f02cad9d640 1 --2- 172.21.15.55:0/3798010460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02cc14d2e0 0x7f02cc144950 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f02c400fae0 tx=0x7f02c40020b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:39.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.336+0000 7f02c37fe640 1 -- 172.21.15.55:0/3798010460 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c4019070 con 0x7f02cc14d2e0 2024-03-19T01:54:39.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.336+0000 7f02c37fe640 1 -- 172.21.15.55:0/3798010460 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02c4006d70 con 0x7f02cc14d2e0 2024-03-19T01:54:39.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.336+0000 7f02d08fb640 1 -- 172.21.15.55:0/3798010460 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02cc1f0200 con 0x7f02cc14d2e0 2024-03-19T01:54:39.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.336+0000 7f02c37fe640 1 -- 172.21.15.55:0/3798010460 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c4003bc0 con 0x7f02cc14d2e0 2024-03-19T01:54:39.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.336+0000 7f02d08fb640 1 -- 172.21.15.55:0/3798010460 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02cc1f07c0 con 0x7f02cc14d2e0 2024-03-19T01:54:39.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.336+0000 7f02d08fb640 1 -- 172.21.15.55:0/3798010460 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02cc068a40 con 0x7f02cc14d2e0 2024-03-19T01:54:39.354 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.356+0000 7f02c37fe640 1 -- 172.21.15.55:0/3798010460 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f02c401c070 con 0x7f02cc14d2e0 2024-03-19T01:54:39.354 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.356+0000 7f02c37fe640 1 --2- 172.21.15.55:0/3798010460 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f02ac0676b0 0x7f02ac069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:39.354 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.360+0000 7f02ca59c640 1 --2- 172.21.15.55:0/3798010460 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f02ac0676b0 0x7f02ac069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:39.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.360+0000 7f02ca59c640 1 --2- 172.21.15.55:0/3798010460 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f02ac0676b0 0x7f02ac069b70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f02bc004880 tx=0x7f02bc00a920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:39.641 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.644+0000 7f02c37fe640 1 -- 172.21.15.55:0/3798010460 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f02c4053570 con 0x7f02cc14d2e0 2024-03-19T01:54:39.641 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.644+0000 7f02c37fe640 1 -- 172.21.15.55:0/3798010460 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f02c4023080 con 0x7f02cc14d2e0 2024-03-19T01:54:39.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.888+0000 7f02d08fb640 1 -- 172.21.15.55:0/3798010460 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02cc1476e0 con 0x7f02cc14d2e0 2024-03-19T01:54:39.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.904+0000 7f02c37fe640 1 -- 172.21.15.55:0/3798010460 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7f02c40554d0 con 0x7f02cc14d2e0 2024-03-19T01:54:39.900 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:54:39.900 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:54:39.901 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:54:39.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.904+0000 7f02d08fb640 1 -- 172.21.15.55:0/3798010460 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f02ac0676b0 msgr2=0x7f02ac069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:39.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.904+0000 7f02d08fb640 1 --2- 172.21.15.55:0/3798010460 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f02ac0676b0 0x7f02ac069b70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f02bc004880 tx=0x7f02bc00a920 comp rx=0 tx=0).stop 2024-03-19T01:54:39.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.904+0000 7f02d08fb640 1 -- 172.21.15.55:0/3798010460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02cc14d2e0 msgr2=0x7f02cc144950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:39.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.904+0000 7f02d08fb640 1 --2- 172.21.15.55:0/3798010460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02cc14d2e0 0x7f02cc144950 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f02c400fae0 tx=0x7f02c40020b0 comp rx=0 tx=0).stop 2024-03-19T01:54:39.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.904+0000 7f02d08fb640 1 -- 172.21.15.55:0/3798010460 shutdown_connections 2024-03-19T01:54:39.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.904+0000 7f02d08fb640 1 --2- 172.21.15.55:0/3798010460 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f02ac0676b0 0x7f02ac069b70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:39.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.904+0000 7f02d08fb640 1 --2- 172.21.15.55:0/3798010460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02cc14d2e0 0x7f02cc144950 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:39.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.904+0000 7f02d08fb640 1 --2- 172.21.15.55:0/3798010460 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f02cc14cea0 0x7f02cc144410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:39.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.904+0000 7f02d08fb640 1 --2- 172.21.15.55:0/3798010460 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02b8004810 0x7f02cc14fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:39.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.904+0000 7f02d08fb640 1 -- 172.21.15.55:0/3798010460 >> 172.21.15.55:0/3798010460 conn(0x7f02cc067600 msgr2=0x7f02cc1572b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:39.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.904+0000 7f02d08fb640 1 -- 172.21.15.55:0/3798010460 shutdown_connections 2024-03-19T01:54:39.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:39.904+0000 7f02d08fb640 1 -- 172.21.15.55:0/3798010460 wait complete. 2024-03-19T01:54:40.522 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:54:40.522 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:54:41.523 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:54:41.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.656+0000 7fa1426ed640 1 Processor -- start 2024-03-19T01:54:41.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.656+0000 7fa1426ed640 1 -- start start 2024-03-19T01:54:41.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.656+0000 7fa1426ed640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa13c154300 0x7fa13c15ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:41.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.656+0000 7fa1426ed640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa13c06b2a0 0x7fa13c15e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:41.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.656+0000 7fa1426ed640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa13c0c8840 con 0x7fa13c15e600 2024-03-19T01:54:41.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.656+0000 7fa1426ed640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fa13c15b260 con 0x7fa13c154300 2024-03-19T01:54:41.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.656+0000 7fa1426ed640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa13c0c8ab0 con 0x7fa13c06b2a0 2024-03-19T01:54:41.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.656+0000 7fa13bfff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa13c154300 0x7fa13c15ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:41.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.660+0000 7fa133fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa13c06b2a0 0x7fa13c15e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:41.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.660+0000 7fa13bfff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa13c154300 0x7fa13c15ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:46502/0 (socket says 172.21.15.55:46502) 2024-03-19T01:54:41.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.660+0000 7fa13bfff640 1 -- 172.21.15.55:0/3889839127 learned_addr learned my addr 172.21.15.55:0/3889839127 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:41.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.660+0000 7fa13bfff640 1 -- 172.21.15.55:0/3889839127 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa13c06b2a0 msgr2=0x7fa13c15e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:54:41.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.660+0000 7fa13bfff640 1 --2- 172.21.15.55:0/3889839127 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa13c06b2a0 0x7fa13c15e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:41.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.660+0000 7fa13bfff640 1 -- 172.21.15.55:0/3889839127 >> v1:172.21.15.84:6789/0 conn(0x7fa13c15e600 legacy=0x7fa13c06c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:41.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.660+0000 7fa13bfff640 1 -- 172.21.15.55:0/3889839127 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa13c0c8d90 con 0x7fa13c154300 2024-03-19T01:54:41.661 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.664+0000 7fa13bfff640 1 --2- 172.21.15.55:0/3889839127 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa13c154300 0x7fa13c15ad20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa12c009870 tx=0x7fa12c01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=418c289dab7f1f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:41.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.668+0000 7fa13b7fe640 1 -- 172.21.15.55:0/3889839127 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c0b5070 con 0x7fa13c154300 2024-03-19T01:54:41.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.668+0000 7fa13b7fe640 1 -- 172.21.15.55:0/3889839127 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa12c0a8a50 con 0x7fa13c154300 2024-03-19T01:54:41.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.668+0000 7fa13b7fe640 1 -- 172.21.15.55:0/3889839127 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c0a8e20 con 0x7fa13c154300 2024-03-19T01:54:41.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.668+0000 7fa1426ed640 1 -- 172.21.15.55:0/3889839127 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa13c154300 msgr2=0x7fa13c15ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:41.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.668+0000 7fa1426ed640 1 --2- 172.21.15.55:0/3889839127 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa13c154300 0x7fa13c15ad20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa12c009870 tx=0x7fa12c01ed50 comp rx=0 tx=0).stop 2024-03-19T01:54:41.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.668+0000 7fa1426ed640 1 -- 172.21.15.55:0/3889839127 shutdown_connections 2024-03-19T01:54:41.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.668+0000 7fa1426ed640 1 --2- 172.21.15.55:0/3889839127 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa13c06b2a0 0x7fa13c15e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:41.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.668+0000 7fa1426ed640 1 --2- 172.21.15.55:0/3889839127 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa13c154300 0x7fa13c15ad20 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:41.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.668+0000 7fa1426ed640 1 -- 172.21.15.55:0/3889839127 >> 172.21.15.55:0/3889839127 conn(0x7fa13c06fb40 msgr2=0x7fa13c06ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:41.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.668+0000 7fa1426ed640 1 -- 172.21.15.55:0/3889839127 shutdown_connections 2024-03-19T01:54:41.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.668+0000 7fa1426ed640 1 -- 172.21.15.55:0/3889839127 wait complete. 2024-03-19T01:54:41.668 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.672+0000 7fa1426ed640 1 Processor -- start 2024-03-19T01:54:41.668 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.672+0000 7fa1426ed640 1 -- start start 2024-03-19T01:54:41.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.672+0000 7fa1426ed640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa13c06b2a0 0x7fa13c1528e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:41.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.672+0000 7fa1426ed640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa13c154300 0x7fa13c1e6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:41.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.672+0000 7fa1426ed640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa13c15e600 0x7fa13c1f3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:41.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.672+0000 7fa1426ed640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa13c15d8c0 con 0x7fa13c154300 2024-03-19T01:54:41.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.672+0000 7fa1426ed640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa13c0c9350 con 0x7fa13c06b2a0 2024-03-19T01:54:41.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.672+0000 7fa1426ed640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa13c15dba0 con 0x7fa13c15e600 2024-03-19T01:54:41.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.672+0000 7fa133fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa13c154300 0x7fa13c1e6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:41.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.672+0000 7fa140c8a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa13c15e600 0x7fa13c1f3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:41.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.672+0000 7fa133fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa13c154300 0x7fa13c1e6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:46508/0 (socket says 172.21.15.55:46508) 2024-03-19T01:54:41.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.672+0000 7fa133fff640 1 -- 172.21.15.55:0/2813310609 learned_addr learned my addr 172.21.15.55:0/2813310609 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:41.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.672+0000 7fa13bfff640 1 --2- 172.21.15.55:0/2813310609 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa13c06b2a0 0x7fa13c1528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:41.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.676+0000 7fa133fff640 1 -- 172.21.15.55:0/2813310609 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa13c15e600 msgr2=0x7fa13c1f3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:41.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.676+0000 7fa133fff640 1 --2- 172.21.15.55:0/2813310609 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa13c15e600 0x7fa13c1f3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:41.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.676+0000 7fa133fff640 1 -- 172.21.15.55:0/2813310609 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa13c06b2a0 msgr2=0x7fa13c1528e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:41.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.676+0000 7fa133fff640 1 --2- 172.21.15.55:0/2813310609 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa13c06b2a0 0x7fa13c1528e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:41.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.676+0000 7fa133fff640 1 -- 172.21.15.55:0/2813310609 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa13c1521a0 con 0x7fa13c154300 2024-03-19T01:54:41.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.680+0000 7fa133fff640 1 --2- 172.21.15.55:0/2813310609 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa13c154300 0x7fa13c1e6ab0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa12400e960 tx=0x7fa12400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:41.680 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.684+0000 7fa1397fa640 1 -- 172.21.15.55:0/2813310609 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa124017070 con 0x7fa13c154300 2024-03-19T01:54:41.680 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.684+0000 7fa1397fa640 1 -- 172.21.15.55:0/2813310609 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa12400ca00 con 0x7fa13c154300 2024-03-19T01:54:41.680 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.684+0000 7fa1397fa640 1 -- 172.21.15.55:0/2813310609 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12400cdb0 con 0x7fa13c154300 2024-03-19T01:54:41.680 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.684+0000 7fa1426ed640 1 -- 172.21.15.55:0/2813310609 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa13c152460 con 0x7fa13c154300 2024-03-19T01:54:41.680 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.684+0000 7fa1426ed640 1 -- 172.21.15.55:0/2813310609 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa13c1512e0 con 0x7fa13c154300 2024-03-19T01:54:41.681 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.684+0000 7fa1426ed640 1 -- 172.21.15.55:0/2813310609 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa13c068a40 con 0x7fa13c154300 2024-03-19T01:54:41.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.704+0000 7fa1397fa640 1 -- 172.21.15.55:0/2813310609 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa12401a070 con 0x7fa13c154300 2024-03-19T01:54:41.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.704+0000 7fa1397fa640 1 --2- 172.21.15.55:0/2813310609 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa11c0676b0 0x7fa11c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:41.701 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.704+0000 7fa13bfff640 1 --2- 172.21.15.55:0/2813310609 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa11c0676b0 0x7fa11c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:41.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.704+0000 7fa13bfff640 1 --2- 172.21.15.55:0/2813310609 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa11c0676b0 0x7fa11c069b70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0b2000 tx=0x7fa12c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:41.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:41.708+0000 7fa1397fa640 1 -- 172.21.15.55:0/2813310609 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fa1240546b0 con 0x7fa13c154300 2024-03-19T01:54:42.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:42.060+0000 7fa1397fa640 1 -- 172.21.15.55:0/2813310609 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa124056610 con 0x7fa13c154300 2024-03-19T01:54:42.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:42.288+0000 7fa1426ed640 1 -- 172.21.15.55:0/2813310609 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa13c153b30 con 0x7fa13c154300 2024-03-19T01:54:42.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:42.308+0000 7fa1397fa640 1 -- 172.21.15.55:0/2813310609 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7fa12405a6b0 con 0x7fa13c154300 2024-03-19T01:54:42.306 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:54:42.306 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:54:42.307 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:54:42.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:42.312+0000 7fa1426ed640 1 -- 172.21.15.55:0/2813310609 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa11c0676b0 msgr2=0x7fa11c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:42.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:42.312+0000 7fa1426ed640 1 --2- 172.21.15.55:0/2813310609 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa11c0676b0 0x7fa11c069b70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0b2000 tx=0x7fa12c009040 comp rx=0 tx=0).stop 2024-03-19T01:54:42.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:42.312+0000 7fa1426ed640 1 -- 172.21.15.55:0/2813310609 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa13c154300 msgr2=0x7fa13c1e6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:42.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:42.312+0000 7fa1426ed640 1 --2- 172.21.15.55:0/2813310609 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa13c154300 0x7fa13c1e6ab0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa12400e960 tx=0x7fa12400ee20 comp rx=0 tx=0).stop 2024-03-19T01:54:42.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:42.312+0000 7fa1426ed640 1 -- 172.21.15.55:0/2813310609 shutdown_connections 2024-03-19T01:54:42.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:42.312+0000 7fa1426ed640 1 --2- 172.21.15.55:0/2813310609 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa11c0676b0 0x7fa11c069b70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:42.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:42.312+0000 7fa1426ed640 1 --2- 172.21.15.55:0/2813310609 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa13c15e600 0x7fa13c1f3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:42.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:42.312+0000 7fa1426ed640 1 --2- 172.21.15.55:0/2813310609 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa13c154300 0x7fa13c1e6ab0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:42.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:42.312+0000 7fa1426ed640 1 --2- 172.21.15.55:0/2813310609 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa13c06b2a0 0x7fa13c1528e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:42.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:42.312+0000 7fa1426ed640 1 -- 172.21.15.55:0/2813310609 >> 172.21.15.55:0/2813310609 conn(0x7fa13c06fb40 msgr2=0x7fa13c0c5200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:42.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:42.312+0000 7fa1426ed640 1 -- 172.21.15.55:0/2813310609 shutdown_connections 2024-03-19T01:54:42.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:42.312+0000 7fa1426ed640 1 -- 172.21.15.55:0/2813310609 wait complete. 2024-03-19T01:54:42.931 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:54:42.931 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:54:43.932 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:54:44.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.071+0000 7f06124a0640 1 Processor -- start 2024-03-19T01:54:44.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.071+0000 7f06124a0640 1 -- start start 2024-03-19T01:54:44.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.071+0000 7f06124a0640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f060c154410 0x7f060c0cae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:44.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.071+0000 7f06124a0640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f060c15d8b0 0x7f060c15aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:44.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.071+0000 7f06124a0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f060c15afe0 0x7f060c06f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:44.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.071+0000 7f06124a0640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f060c0c9460 con 0x7f060c15d8b0 2024-03-19T01:54:44.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.071+0000 7f06124a0640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f060c15a2c0 con 0x7f060c154410 2024-03-19T01:54:44.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.071+0000 7f06124a0640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f060c15dda0 con 0x7f060c15afe0 2024-03-19T01:54:44.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.071+0000 7f060b7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f060c15d8b0 0x7f060c15aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:44.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.075+0000 7f0610a3d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f060c15afe0 0x7f060c06f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:44.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.075+0000 7f060b7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f060c15d8b0 0x7f060c15aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:46520/0 (socket says 172.21.15.55:46520) 2024-03-19T01:54:44.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.075+0000 7f060b7fe640 1 -- 172.21.15.55:0/1181321785 learned_addr learned my addr 172.21.15.55:0/1181321785 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:44.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.075+0000 7f060bfff640 1 --2- 172.21.15.55:0/1181321785 >> v2:172.21.15.84:3300/0 conn(0x7f060c154410 0x7f060c0cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:44.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.075+0000 7f060b7fe640 1 -- 172.21.15.55:0/1181321785 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f060c15afe0 msgr2=0x7f060c06f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:54:44.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.075+0000 7f060b7fe640 1 --2- 172.21.15.55:0/1181321785 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f060c15afe0 0x7f060c06f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:44.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.075+0000 7f060b7fe640 1 -- 172.21.15.55:0/1181321785 >> v2:172.21.15.84:3300/0 conn(0x7f060c154410 msgr2=0x7f060c0cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:54:44.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.075+0000 7f060b7fe640 1 --2- 172.21.15.55:0/1181321785 >> v2:172.21.15.84:3300/0 conn(0x7f060c154410 0x7f060c0cae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:44.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.075+0000 7f060b7fe640 1 -- 172.21.15.55:0/1181321785 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f060c15e080 con 0x7f060c15d8b0 2024-03-19T01:54:44.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.079+0000 7f060b7fe640 1 --2- 172.21.15.55:0/1181321785 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f060c15d8b0 0x7f060c15aaa0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f060c067c70 tx=0x7f060001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d868795c04509710 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:44.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.087+0000 7f060affd640 1 -- 172.21.15.55:0/1181321785 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06000b0070 con 0x7f060c15d8b0 2024-03-19T01:54:44.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.087+0000 7f060affd640 1 -- 172.21.15.55:0/1181321785 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06000ab600 con 0x7f060c15d8b0 2024-03-19T01:54:44.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.087+0000 7f060affd640 1 -- 172.21.15.55:0/1181321785 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06000ab9d0 con 0x7f060c15d8b0 2024-03-19T01:54:44.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.087+0000 7f06124a0640 1 -- 172.21.15.55:0/1181321785 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f060c15d8b0 msgr2=0x7f060c15aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:44.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.087+0000 7f06124a0640 1 --2- 172.21.15.55:0/1181321785 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f060c15d8b0 0x7f060c15aaa0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f060c067c70 tx=0x7f060001c460 comp rx=0 tx=0).stop 2024-03-19T01:54:44.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.087+0000 7f06124a0640 1 -- 172.21.15.55:0/1181321785 shutdown_connections 2024-03-19T01:54:44.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.087+0000 7f06124a0640 1 --2- 172.21.15.55:0/1181321785 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f060c15afe0 0x7f060c06f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:44.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.087+0000 7f06124a0640 1 --2- 172.21.15.55:0/1181321785 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f060c15d8b0 0x7f060c15aaa0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:44.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.087+0000 7f06124a0640 1 --2- 172.21.15.55:0/1181321785 >> v2:172.21.15.84:3300/0 conn(0x7f060c154410 0x7f060c0cae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:44.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.087+0000 7f06124a0640 1 -- 172.21.15.55:0/1181321785 >> 172.21.15.55:0/1181321785 conn(0x7f060c070cc0 msgr2=0x7f060c067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:44.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.087+0000 7f06124a0640 1 -- 172.21.15.55:0/1181321785 shutdown_connections 2024-03-19T01:54:44.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.087+0000 7f06124a0640 1 -- 172.21.15.55:0/1181321785 wait complete. 2024-03-19T01:54:44.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.091+0000 7f06124a0640 1 Processor -- start 2024-03-19T01:54:44.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.091+0000 7f06124a0640 1 -- start start 2024-03-19T01:54:44.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.091+0000 7f06124a0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f060c154410 0x7f060c0ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:44.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.091+0000 7f06124a0640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f060c15afe0 0x7f060c15cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:44.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.091+0000 7f06124a0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f060c15d8b0 0x7f060c15d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:44.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.091+0000 7f06124a0640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f060c0cea20 con 0x7f060c154410 2024-03-19T01:54:44.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.091+0000 7f06124a0640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f060c0ce8a0 con 0x7f060c15afe0 2024-03-19T01:54:44.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.091+0000 7f06124a0640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f060c1547f0 con 0x7f060c15d8b0 2024-03-19T01:54:44.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.091+0000 7f060bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f060c154410 0x7f060c0ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:44.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.091+0000 7f060bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f060c154410 0x7f060c0ccc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:46524/0 (socket says 172.21.15.55:46524) 2024-03-19T01:54:44.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.091+0000 7f060bfff640 1 -- 172.21.15.55:0/842806297 learned_addr learned my addr 172.21.15.55:0/842806297 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:44.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.091+0000 7f0610a3d640 1 --2- 172.21.15.55:0/842806297 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f060c15d8b0 0x7f060c15d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:44.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.091+0000 7f060b7fe640 1 --2- 172.21.15.55:0/842806297 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f060c15afe0 0x7f060c15cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:44.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.095+0000 7f060bfff640 1 -- 172.21.15.55:0/842806297 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f060c15d8b0 msgr2=0x7f060c15d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:44.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.095+0000 7f060bfff640 1 --2- 172.21.15.55:0/842806297 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f060c15d8b0 0x7f060c15d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:44.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.095+0000 7f060bfff640 1 -- 172.21.15.55:0/842806297 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f060c15afe0 msgr2=0x7f060c15cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:44.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.095+0000 7f060bfff640 1 --2- 172.21.15.55:0/842806297 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f060c15afe0 0x7f060c15cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:44.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.095+0000 7f060bfff640 1 -- 172.21.15.55:0/842806297 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f060c150710 con 0x7f060c154410 2024-03-19T01:54:44.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.099+0000 7f060bfff640 1 --2- 172.21.15.55:0/842806297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f060c154410 0x7f060c0ccc70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00e870 tx=0x7f05fc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:44.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.103+0000 7f0608ff9640 1 -- 172.21.15.55:0/842806297 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05fc017070 con 0x7f060c154410 2024-03-19T01:54:44.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.103+0000 7f0608ff9640 1 -- 172.21.15.55:0/842806297 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05fc00ee60 con 0x7f060c154410 2024-03-19T01:54:44.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.103+0000 7f06124a0640 1 -- 172.21.15.55:0/842806297 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f060c150170 con 0x7f060c154410 2024-03-19T01:54:44.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.103+0000 7f0608ff9640 1 -- 172.21.15.55:0/842806297 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05fc00cb80 con 0x7f060c154410 2024-03-19T01:54:44.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.103+0000 7f06124a0640 1 -- 172.21.15.55:0/842806297 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f060c154cc0 con 0x7f060c154410 2024-03-19T01:54:44.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.103+0000 7f06124a0640 1 -- 172.21.15.55:0/842806297 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f060c155380 con 0x7f060c154410 2024-03-19T01:54:44.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.391+0000 7f0608ff9640 1 -- 172.21.15.55:0/842806297 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f05fc01a070 con 0x7f060c154410 2024-03-19T01:54:44.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.391+0000 7f0608ff9640 1 --2- 172.21.15.55:0/842806297 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f05f80676b0 0x7f05f8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:44.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.391+0000 7f060b7fe640 1 --2- 172.21.15.55:0/842806297 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f05f80676b0 0x7f05f8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:44.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.391+0000 7f060b7fe640 1 --2- 172.21.15.55:0/842806297 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f05f80676b0 0x7f05f8069b70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f06000ad000 tx=0x7f06000a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:44.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.415+0000 7f0608ff9640 1 -- 172.21.15.55:0/842806297 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f05fc054560 con 0x7f060c154410 2024-03-19T01:54:44.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.415+0000 7f0608ff9640 1 -- 172.21.15.55:0/842806297 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f05fc0564c0 con 0x7f060c154410 2024-03-19T01:54:44.637 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.639+0000 7f06124a0640 1 -- 172.21.15.55:0/842806297 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f060c068330 con 0x7f060c154410 2024-03-19T01:54:44.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.731+0000 7f0608ff9640 1 -- 172.21.15.55:0/842806297 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7f05fc05a560 con 0x7f060c154410 2024-03-19T01:54:44.728 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:54:44.729 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:54:44.729 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:54:44.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.735+0000 7f06124a0640 1 -- 172.21.15.55:0/842806297 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f05f80676b0 msgr2=0x7f05f8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:44.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.735+0000 7f06124a0640 1 --2- 172.21.15.55:0/842806297 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f05f80676b0 0x7f05f8069b70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f06000ad000 tx=0x7f06000a6000 comp rx=0 tx=0).stop 2024-03-19T01:54:44.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.735+0000 7f06124a0640 1 -- 172.21.15.55:0/842806297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f060c154410 msgr2=0x7f060c0ccc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:44.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.735+0000 7f06124a0640 1 --2- 172.21.15.55:0/842806297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f060c154410 0x7f060c0ccc70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00e870 tx=0x7f05fc00ed30 comp rx=0 tx=0).stop 2024-03-19T01:54:44.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.735+0000 7f06124a0640 1 -- 172.21.15.55:0/842806297 shutdown_connections 2024-03-19T01:54:44.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.735+0000 7f06124a0640 1 --2- 172.21.15.55:0/842806297 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f05f80676b0 0x7f05f8069b70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:44.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.735+0000 7f06124a0640 1 --2- 172.21.15.55:0/842806297 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f060c15d8b0 0x7f060c15d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:44.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.735+0000 7f06124a0640 1 --2- 172.21.15.55:0/842806297 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f060c15afe0 0x7f060c15cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:44.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.735+0000 7f06124a0640 1 --2- 172.21.15.55:0/842806297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f060c154410 0x7f060c0ccc70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:44.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.735+0000 7f06124a0640 1 -- 172.21.15.55:0/842806297 >> 172.21.15.55:0/842806297 conn(0x7f060c070cc0 msgr2=0x7f060c152400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:44.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.735+0000 7f06124a0640 1 -- 172.21.15.55:0/842806297 shutdown_connections 2024-03-19T01:54:44.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:44.735+0000 7f06124a0640 1 -- 172.21.15.55:0/842806297 wait complete. 2024-03-19T01:54:45.354 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:54:45.355 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:54:46.356 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:54:46.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.471+0000 7f9b5357f640 1 Processor -- start 2024-03-19T01:54:46.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.475+0000 7f9b5357f640 1 -- start start 2024-03-19T01:54:46.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.475+0000 7f9b5357f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f9b4c15a0e0 0x7f9b4c0cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:46.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.475+0000 7f9b5357f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b4c15a6b0 con 0x7f9b4c0cc4f0 2024-03-19T01:54:46.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.475+0000 7f9b5357f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b4c15a990 con 0x7f9b4c1544b0 2024-03-19T01:54:46.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.475+0000 7f9b5357f640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f9b4c15ac70 con 0x7f9b4c15a0e0 2024-03-19T01:54:46.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.475+0000 7f9b51b1c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f9b4c0cc4f0 0x7f9b4c0c9050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:58116/0 (socket says 172.21.15.55:58116) 2024-03-19T01:54:46.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.475+0000 7f9b51b1c640 1 -- 172.21.15.55:0/479862285 learned_addr learned my addr 172.21.15.55:0/479862285 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:54:46.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.475+0000 7f9b50b1a640 1 --2- 172.21.15.55:0/479862285 >> v2:172.21.15.84:3300/0 conn(0x7f9b4c15a0e0 0x7f9b4c0cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:46.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.479+0000 7f9b43fff640 1 -- 172.21.15.55:0/479862285 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2345079681 0 0) 0x7f9b4c15a6b0 con 0x7f9b4c0cc4f0 2024-03-19T01:54:46.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.479+0000 7f9b43fff640 1 -- 172.21.15.55:0/479862285 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b38003530 con 0x7f9b4c0cc4f0 2024-03-19T01:54:46.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.479+0000 7f9b50b1a640 1 -- 172.21.15.55:0/479862285 >> v1:172.21.15.84:6789/0 conn(0x7f9b4c1544b0 legacy=0x7f9b4c06b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:46.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.479+0000 7f9b50b1a640 1 -- 172.21.15.55:0/479862285 >> v1:172.21.15.55:6789/0 conn(0x7f9b4c0cc4f0 legacy=0x7f9b4c0c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:46.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.479+0000 7f9b50b1a640 1 -- 172.21.15.55:0/479862285 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4c15af50 con 0x7f9b4c15a0e0 2024-03-19T01:54:46.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.487+0000 7f9b50b1a640 1 --2- 172.21.15.55:0/479862285 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b4c15a0e0 0x7f9b4c0cbfb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0097c0 tx=0x7f9b3c01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ec8f90dfd8bf257 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:46.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.491+0000 7f9b43fff640 1 -- 172.21.15.55:0/479862285 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c0b5070 con 0x7f9b4c15a0e0 2024-03-19T01:54:46.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.491+0000 7f9b43fff640 1 -- 172.21.15.55:0/479862285 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b3c0a8990 con 0x7f9b4c15a0e0 2024-03-19T01:54:46.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.491+0000 7f9b43fff640 1 -- 172.21.15.55:0/479862285 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c0a8d60 con 0x7f9b4c15a0e0 2024-03-19T01:54:46.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.495+0000 7f9b5357f640 1 -- 172.21.15.55:0/479862285 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b4c15a0e0 msgr2=0x7f9b4c0cbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:46.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.495+0000 7f9b5357f640 1 --2- 172.21.15.55:0/479862285 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b4c15a0e0 0x7f9b4c0cbfb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0097c0 tx=0x7f9b3c01ec90 comp rx=0 tx=0).stop 2024-03-19T01:54:46.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.495+0000 7f9b5357f640 1 -- 172.21.15.55:0/479862285 shutdown_connections 2024-03-19T01:54:46.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.495+0000 7f9b5357f640 1 --2- 172.21.15.55:0/479862285 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b4c15a0e0 0x7f9b4c0cbfb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:46.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.495+0000 7f9b5357f640 1 -- 172.21.15.55:0/479862285 >> 172.21.15.55:0/479862285 conn(0x7f9b4c06e5a0 msgr2=0x7f9b4c072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:46.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.495+0000 7f9b5357f640 1 -- 172.21.15.55:0/479862285 shutdown_connections 2024-03-19T01:54:46.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.495+0000 7f9b5357f640 1 -- 172.21.15.55:0/479862285 wait complete. 2024-03-19T01:54:46.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.495+0000 7f9b5357f640 1 Processor -- start 2024-03-19T01:54:46.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.495+0000 7f9b5357f640 1 -- start start 2024-03-19T01:54:46.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.499+0000 7f9b5357f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c0cc4f0 0x7f9b4c14bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:46.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.499+0000 7f9b5357f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b4c1544b0 0x7f9b4c15c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:46.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.499+0000 7f9b5357f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9b4c15a0e0 0x7f9b4c0ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:46.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.499+0000 7f9b5357f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c0c8760 con 0x7f9b4c0cc4f0 2024-03-19T01:54:46.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.499+0000 7f9b5357f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c0c6be0 con 0x7f9b4c1544b0 2024-03-19T01:54:46.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.499+0000 7f9b5357f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f9b4c0c88e0 con 0x7f9b4c15a0e0 2024-03-19T01:54:46.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.499+0000 7f9b5131b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c0cc4f0 0x7f9b4c14bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:46.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.499+0000 7f9b50b1a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b4c1544b0 0x7f9b4c15c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:46.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.499+0000 7f9b51b1c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9b4c15a0e0 0x7f9b4c0ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:46.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.499+0000 7f9b5131b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c0cc4f0 0x7f9b4c14bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:46540/0 (socket says 172.21.15.55:46540) 2024-03-19T01:54:46.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.499+0000 7f9b5131b640 1 -- 172.21.15.55:0/263788029 learned_addr learned my addr 172.21.15.55:0/263788029 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:46.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.503+0000 7f9b5131b640 1 -- 172.21.15.55:0/263788029 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9b4c15a0e0 msgr2=0x7f9b4c0ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:46.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.503+0000 7f9b5131b640 1 --2- 172.21.15.55:0/263788029 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9b4c15a0e0 0x7f9b4c0ca180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:46.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.503+0000 7f9b5131b640 1 -- 172.21.15.55:0/263788029 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b4c1544b0 msgr2=0x7f9b4c15c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:46.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.503+0000 7f9b5131b640 1 --2- 172.21.15.55:0/263788029 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b4c1544b0 0x7f9b4c15c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:46.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.503+0000 7f9b5131b640 1 -- 172.21.15.55:0/263788029 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4c14aa00 con 0x7f9b4c0cc4f0 2024-03-19T01:54:46.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.503+0000 7f9b5131b640 1 --2- 172.21.15.55:0/263788029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c0cc4f0 0x7f9b4c14bba0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800c450 tx=0x7f9b4800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:46.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.511+0000 7f9b41ffb640 1 -- 172.21.15.55:0/263788029 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b48016020 con 0x7f9b4c0cc4f0 2024-03-19T01:54:46.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.511+0000 7f9b41ffb640 1 -- 172.21.15.55:0/263788029 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b480052d0 con 0x7f9b4c0cc4f0 2024-03-19T01:54:46.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.511+0000 7f9b41ffb640 1 -- 172.21.15.55:0/263788029 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b48007260 con 0x7f9b4c0cc4f0 2024-03-19T01:54:46.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.511+0000 7f9b5357f640 1 -- 172.21.15.55:0/263788029 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b4c14aca0 con 0x7f9b4c0cc4f0 2024-03-19T01:54:46.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.511+0000 7f9b5357f640 1 -- 172.21.15.55:0/263788029 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b4c15b580 con 0x7f9b4c0cc4f0 2024-03-19T01:54:46.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.511+0000 7f9b5357f640 1 -- 172.21.15.55:0/263788029 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b4c068a40 con 0x7f9b4c0cc4f0 2024-03-19T01:54:46.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.543+0000 7f9b41ffb640 1 -- 172.21.15.55:0/263788029 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f9b48007400 con 0x7f9b4c0cc4f0 2024-03-19T01:54:46.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.543+0000 7f9b41ffb640 1 --2- 172.21.15.55:0/263788029 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9b300676e0 0x7f9b30069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:46.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.543+0000 7f9b50b1a640 1 --2- 172.21.15.55:0/263788029 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9b300676e0 0x7f9b30069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:46.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.543+0000 7f9b50b1a640 1 --2- 172.21.15.55:0/263788029 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9b300676e0 0x7f9b30069ba0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0b2000 tx=0x7f9b3c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:46.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.839+0000 7f9b41ffb640 1 -- 172.21.15.55:0/263788029 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f9b48054620 con 0x7f9b4c0cc4f0 2024-03-19T01:54:46.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:46.843+0000 7f9b41ffb640 1 -- 172.21.15.55:0/263788029 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f9b48020020 con 0x7f9b4c0cc4f0 2024-03-19T01:54:47.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:47.055+0000 7f9b5357f640 1 -- 172.21.15.55:0/263788029 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b4c0cb1c0 con 0x7f9b4c0cc4f0 2024-03-19T01:54:47.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:47.179+0000 7f9b41ffb640 1 -- 172.21.15.55:0/263788029 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7f9b48005480 con 0x7f9b4c0cc4f0 2024-03-19T01:54:47.177 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:54:47.177 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:54:47.177 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:54:47.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:47.183+0000 7f9b5357f640 1 -- 172.21.15.55:0/263788029 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9b300676e0 msgr2=0x7f9b30069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:47.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:47.183+0000 7f9b5357f640 1 --2- 172.21.15.55:0/263788029 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9b300676e0 0x7f9b30069ba0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0b2000 tx=0x7f9b3c008040 comp rx=0 tx=0).stop 2024-03-19T01:54:47.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:47.183+0000 7f9b5357f640 1 -- 172.21.15.55:0/263788029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c0cc4f0 msgr2=0x7f9b4c14bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:47.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:47.183+0000 7f9b5357f640 1 --2- 172.21.15.55:0/263788029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c0cc4f0 0x7f9b4c14bba0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800c450 tx=0x7f9b4800c910 comp rx=0 tx=0).stop 2024-03-19T01:54:47.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:47.183+0000 7f9b5357f640 1 -- 172.21.15.55:0/263788029 shutdown_connections 2024-03-19T01:54:47.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:47.183+0000 7f9b5357f640 1 --2- 172.21.15.55:0/263788029 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9b300676e0 0x7f9b30069ba0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:47.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:47.183+0000 7f9b5357f640 1 --2- 172.21.15.55:0/263788029 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9b4c15a0e0 0x7f9b4c0ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:47.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:47.183+0000 7f9b5357f640 1 --2- 172.21.15.55:0/263788029 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b4c1544b0 0x7f9b4c15c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:47.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:47.183+0000 7f9b5357f640 1 --2- 172.21.15.55:0/263788029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b4c0cc4f0 0x7f9b4c14bba0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:47.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:47.183+0000 7f9b5357f640 1 -- 172.21.15.55:0/263788029 >> 172.21.15.55:0/263788029 conn(0x7f9b4c06e5a0 msgr2=0x7f9b4c0c4f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:47.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:47.183+0000 7f9b5357f640 1 -- 172.21.15.55:0/263788029 shutdown_connections 2024-03-19T01:54:47.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:47.183+0000 7f9b5357f640 1 -- 172.21.15.55:0/263788029 wait complete. 2024-03-19T01:54:47.803 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:54:47.804 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:54:48.804 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:54:48.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.935+0000 7fd73630b640 1 Processor -- start 2024-03-19T01:54:48.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.935+0000 7fd73630b640 1 -- start start 2024-03-19T01:54:48.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.939+0000 7fd73630b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd7300c60d0 0x7fd7300c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:48.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.939+0000 7fd73630b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd73014ee80 con 0x7fd7301585f0 2024-03-19T01:54:48.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.939+0000 7fd73630b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7300c7540 con 0x7fd73014eaa0 2024-03-19T01:54:48.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.939+0000 7fd73630b640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fd7300c77c0 con 0x7fd7300c60d0 2024-03-19T01:54:48.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.939+0000 7fd72f7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd7300c60d0 0x7fd7300c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:48.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.939+0000 7fd7348a8640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fd7301585f0 0x7fd730162940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:33752/0 (socket says 172.21.15.55:33752) 2024-03-19T01:54:48.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.939+0000 7fd7348a8640 1 -- 172.21.15.55:0/4273986997 learned_addr learned my addr 172.21.15.55:0/4273986997 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:54:48.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.943+0000 7fd72f7fe640 1 -- 172.21.15.55:0/4273986997 >> v1:172.21.15.84:6789/0 conn(0x7fd73014eaa0 legacy=0x7fd73006b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:48.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.943+0000 7fd72f7fe640 1 -- 172.21.15.55:0/4273986997 >> v1:172.21.15.55:6789/0 conn(0x7fd7301585f0 legacy=0x7fd730162940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:48.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.943+0000 7fd72f7fe640 1 -- 172.21.15.55:0/4273986997 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7300c7aa0 con 0x7fd7300c60d0 2024-03-19T01:54:48.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.947+0000 7fd72f7fe640 1 --2- 172.21.15.55:0/4273986997 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7300c60d0 0x7fd7300c6540 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd7240097c0 tx=0x7fd72401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ca6a4e26e8ce632 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:48.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.951+0000 7fd72effd640 1 -- 172.21.15.55:0/4273986997 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7240ab610 con 0x7fd7300c60d0 2024-03-19T01:54:48.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.951+0000 7fd72effd640 1 -- 172.21.15.55:0/4273986997 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7240b0070 con 0x7fd7300c60d0 2024-03-19T01:54:48.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.951+0000 7fd72effd640 1 -- 172.21.15.55:0/4273986997 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7240ab9d0 con 0x7fd7300c60d0 2024-03-19T01:54:48.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.951+0000 7fd73630b640 1 -- 172.21.15.55:0/4273986997 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7300c60d0 msgr2=0x7fd7300c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:48.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.951+0000 7fd73630b640 1 --2- 172.21.15.55:0/4273986997 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7300c60d0 0x7fd7300c6540 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd7240097c0 tx=0x7fd72401c410 comp rx=0 tx=0).stop 2024-03-19T01:54:48.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.951+0000 7fd73630b640 1 -- 172.21.15.55:0/4273986997 shutdown_connections 2024-03-19T01:54:48.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.951+0000 7fd73630b640 1 --2- 172.21.15.55:0/4273986997 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7300c60d0 0x7fd7300c6540 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:48.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.951+0000 7fd73630b640 1 -- 172.21.15.55:0/4273986997 >> 172.21.15.55:0/4273986997 conn(0x7fd7300cc9c0 msgr2=0x7fd7300ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:48.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.951+0000 7fd73630b640 1 -- 172.21.15.55:0/4273986997 shutdown_connections 2024-03-19T01:54:48.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.951+0000 7fd73630b640 1 -- 172.21.15.55:0/4273986997 wait complete. 2024-03-19T01:54:48.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.955+0000 7fd73630b640 1 Processor -- start 2024-03-19T01:54:48.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.955+0000 7fd73630b640 1 -- start start 2024-03-19T01:54:48.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.955+0000 7fd73630b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7300c60d0 0x7fd730144f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:48.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.955+0000 7fd73630b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd73014eaa0 0x7fd7300c5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:48.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.955+0000 7fd73630b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd7301585f0 0x7fd7301f6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:48.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.955+0000 7fd73630b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd73006c090 con 0x7fd7300c60d0 2024-03-19T01:54:48.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.955+0000 7fd73630b640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd730158e10 con 0x7fd73014eaa0 2024-03-19T01:54:48.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.955+0000 7fd73630b640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd73006c210 con 0x7fd7301585f0 2024-03-19T01:54:48.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.955+0000 7fd72ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7300c60d0 0x7fd730144f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:48.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.955+0000 7fd72ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7300c60d0 0x7fd730144f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57982/0 (socket says 172.21.15.55:57982) 2024-03-19T01:54:48.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.955+0000 7fd72ffff640 1 -- 172.21.15.55:0/3283333857 learned_addr learned my addr 172.21.15.55:0/3283333857 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:48.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.955+0000 7fd7348a8640 1 --2- 172.21.15.55:0/3283333857 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd7301585f0 0x7fd7301f6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:48.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.955+0000 7fd72f7fe640 1 --2- 172.21.15.55:0/3283333857 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd73014eaa0 0x7fd7300c5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:48.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.963+0000 7fd72ffff640 1 -- 172.21.15.55:0/3283333857 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd7301585f0 msgr2=0x7fd7301f6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:48.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.963+0000 7fd72ffff640 1 --2- 172.21.15.55:0/3283333857 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd7301585f0 0x7fd7301f6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:48.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.963+0000 7fd72ffff640 1 -- 172.21.15.55:0/3283333857 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd73014eaa0 msgr2=0x7fd7300c5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:48.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.963+0000 7fd72ffff640 1 --2- 172.21.15.55:0/3283333857 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd73014eaa0 0x7fd7300c5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:48.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.963+0000 7fd72ffff640 1 -- 172.21.15.55:0/3283333857 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7300c4970 con 0x7fd7300c60d0 2024-03-19T01:54:48.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.963+0000 7fd72ffff640 1 --2- 172.21.15.55:0/3283333857 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7300c60d0 0x7fd730144f40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd72000c760 tx=0x7fd72000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:48.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.971+0000 7fd72cff9640 1 -- 172.21.15.55:0/3283333857 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd720016020 con 0x7fd7300c60d0 2024-03-19T01:54:48.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.971+0000 7fd72cff9640 1 -- 172.21.15.55:0/3283333857 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd720006d60 con 0x7fd7300c60d0 2024-03-19T01:54:48.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.971+0000 7fd72cff9640 1 -- 172.21.15.55:0/3283333857 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd720003bc0 con 0x7fd7300c60d0 2024-03-19T01:54:48.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.971+0000 7fd73630b640 1 -- 172.21.15.55:0/3283333857 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7300c4c50 con 0x7fd7300c60d0 2024-03-19T01:54:48.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.971+0000 7fd73630b640 1 -- 172.21.15.55:0/3283333857 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7301f76a0 con 0x7fd7300c60d0 2024-03-19T01:54:48.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:48.971+0000 7fd73630b640 1 -- 172.21.15.55:0/3283333857 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd730068a40 con 0x7fd7300c60d0 2024-03-19T01:54:49.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:49.003+0000 7fd72cff9640 1 -- 172.21.15.55:0/3283333857 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fd72000a850 con 0x7fd7300c60d0 2024-03-19T01:54:49.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:49.003+0000 7fd72cff9640 1 --2- 172.21.15.55:0/3283333857 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd71c0676e0 0x7fd71c069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:49.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:49.003+0000 7fd72f7fe640 1 --2- 172.21.15.55:0/3283333857 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd71c0676e0 0x7fd71c069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:49.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:49.007+0000 7fd72f7fe640 1 --2- 172.21.15.55:0/3283333857 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd71c0676e0 0x7fd71c069ba0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd7240af000 tx=0x7fd7240a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:49.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:49.011+0000 7fd72cff9640 1 -- 172.21.15.55:0/3283333857 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fd720054910 con 0x7fd7300c60d0 2024-03-19T01:54:49.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:49.319+0000 7fd72cff9640 1 -- 172.21.15.55:0/3283333857 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fd720020020 con 0x7fd7300c60d0 2024-03-19T01:54:49.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:49.547+0000 7fd73630b640 1 -- 172.21.15.55:0/3283333857 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7300729d0 con 0x7fd7300c60d0 2024-03-19T01:54:49.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:49.591+0000 7fd72cff9640 1 -- 172.21.15.55:0/3283333857 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7fd7300c7540 con 0x7fd7300c60d0 2024-03-19T01:54:49.588 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:54:49.588 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:54:49.589 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:54:49.591 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:49.595+0000 7fd73630b640 1 -- 172.21.15.55:0/3283333857 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd71c0676e0 msgr2=0x7fd71c069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:49.591 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:49.595+0000 7fd73630b640 1 --2- 172.21.15.55:0/3283333857 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd71c0676e0 0x7fd71c069ba0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd7240af000 tx=0x7fd7240a7000 comp rx=0 tx=0).stop 2024-03-19T01:54:49.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:49.595+0000 7fd73630b640 1 -- 172.21.15.55:0/3283333857 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7300c60d0 msgr2=0x7fd730144f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:49.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:49.595+0000 7fd73630b640 1 --2- 172.21.15.55:0/3283333857 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7300c60d0 0x7fd730144f40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd72000c760 tx=0x7fd72000cc20 comp rx=0 tx=0).stop 2024-03-19T01:54:49.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:49.595+0000 7fd73630b640 1 -- 172.21.15.55:0/3283333857 shutdown_connections 2024-03-19T01:54:49.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:49.595+0000 7fd73630b640 1 --2- 172.21.15.55:0/3283333857 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd71c0676e0 0x7fd71c069ba0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:49.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:49.595+0000 7fd73630b640 1 --2- 172.21.15.55:0/3283333857 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd7301585f0 0x7fd7301f6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:49.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:49.595+0000 7fd73630b640 1 --2- 172.21.15.55:0/3283333857 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd73014eaa0 0x7fd7300c5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:49.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:49.595+0000 7fd73630b640 1 --2- 172.21.15.55:0/3283333857 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7300c60d0 0x7fd730144f40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:49.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:49.595+0000 7fd73630b640 1 -- 172.21.15.55:0/3283333857 >> 172.21.15.55:0/3283333857 conn(0x7fd7300cc9c0 msgr2=0x7fd730148560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:49.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:49.595+0000 7fd73630b640 1 -- 172.21.15.55:0/3283333857 shutdown_connections 2024-03-19T01:54:49.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:49.595+0000 7fd73630b640 1 -- 172.21.15.55:0/3283333857 wait complete. 2024-03-19T01:54:50.213 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:54:50.213 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:54:51.214 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:54:51.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.339+0000 7fabf95be640 1 Processor -- start 2024-03-19T01:54:51.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.339+0000 7fabf95be640 1 -- start start 2024-03-19T01:54:51.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.339+0000 7fabf95be640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fabf40708a0 0x7fabf40c8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:51.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.339+0000 7fabf95be640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fabf406b2a0 0x7fabf40ca010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:51.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.339+0000 7fabf95be640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabf406bbd0 con 0x7fabf4070c80 2024-03-19T01:54:51.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.339+0000 7fabf95be640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fabf40c7dd0 con 0x7fabf40708a0 2024-03-19T01:54:51.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.339+0000 7fabf95be640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fabf406beb0 con 0x7fabf406b2a0 2024-03-19T01:54:51.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.339+0000 7fabf2ffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fabf40708a0 0x7fabf40c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:51.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.339+0000 7fabf27fc640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fabf4070c80 0x7fabf4157f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:53474/0 (socket says 172.21.15.55:53474) 2024-03-19T01:54:51.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.339+0000 7fabf27fc640 1 -- 172.21.15.55:0/681214237 learned_addr learned my addr 172.21.15.55:0/681214237 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:54:51.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.339+0000 7fabf37fe640 1 --2- 172.21.15.55:0/681214237 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fabf406b2a0 0x7fabf40ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:51.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.343+0000 7fabf2ffd640 1 -- 172.21.15.55:0/681214237 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fabf406b2a0 msgr2=0x7fabf40ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:54:51.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.343+0000 7fabf2ffd640 1 --2- 172.21.15.55:0/681214237 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fabf406b2a0 0x7fabf40ca010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:51.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.343+0000 7fabf2ffd640 1 -- 172.21.15.55:0/681214237 >> v1:172.21.15.84:6789/0 conn(0x7fabf4070c80 legacy=0x7fabf4157f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:51.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.343+0000 7fabf2ffd640 1 -- 172.21.15.55:0/681214237 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabf40c7780 con 0x7fabf40708a0 2024-03-19T01:54:51.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.343+0000 7fabf37fe640 1 --2- 172.21.15.55:0/681214237 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fabf406b2a0 0x7fabf40ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:54:51.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.347+0000 7fabf2ffd640 1 --2- 172.21.15.55:0/681214237 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabf40708a0 0x7fabf40c8260 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fabe40097c0 tx=0x7fabe401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff9169a1c91a471d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:51.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.351+0000 7fabf1ffb640 1 -- 172.21.15.55:0/681214237 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe40b0070 con 0x7fabf40708a0 2024-03-19T01:54:51.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.351+0000 7fabf1ffb640 1 -- 172.21.15.55:0/681214237 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabe40ab600 con 0x7fabf40708a0 2024-03-19T01:54:51.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.351+0000 7fabf1ffb640 1 -- 172.21.15.55:0/681214237 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe40ab9d0 con 0x7fabf40708a0 2024-03-19T01:54:51.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.351+0000 7fabf95be640 1 -- 172.21.15.55:0/681214237 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabf40708a0 msgr2=0x7fabf40c8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:51.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.351+0000 7fabf95be640 1 --2- 172.21.15.55:0/681214237 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabf40708a0 0x7fabf40c8260 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fabe40097c0 tx=0x7fabe401c4a0 comp rx=0 tx=0).stop 2024-03-19T01:54:51.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.351+0000 7fabf95be640 1 -- 172.21.15.55:0/681214237 shutdown_connections 2024-03-19T01:54:51.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.351+0000 7fabf95be640 1 --2- 172.21.15.55:0/681214237 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fabf406b2a0 0x7fabf40ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:51.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.351+0000 7fabf95be640 1 --2- 172.21.15.55:0/681214237 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabf40708a0 0x7fabf40c8260 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:51.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.351+0000 7fabf95be640 1 -- 172.21.15.55:0/681214237 >> 172.21.15.55:0/681214237 conn(0x7fabf406c580 msgr2=0x7fabf406c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:51.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.351+0000 7fabf95be640 1 -- 172.21.15.55:0/681214237 shutdown_connections 2024-03-19T01:54:51.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.351+0000 7fabf95be640 1 -- 172.21.15.55:0/681214237 wait complete. 2024-03-19T01:54:51.351 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.355+0000 7fabf95be640 1 Processor -- start 2024-03-19T01:54:51.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.355+0000 7fabf95be640 1 -- start start 2024-03-19T01:54:51.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.355+0000 7fabf95be640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fabf406b2a0 0x7fabf41548f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:51.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.355+0000 7fabf95be640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fabf40708a0 0x7fabf4134d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:51.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.355+0000 7fabf95be640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabf4070c80 0x7fabf413fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:51.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.355+0000 7fabf95be640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fabf4164aa0 con 0x7fabf4070c80 2024-03-19T01:54:51.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.355+0000 7fabf95be640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fabf4164920 con 0x7fabf40708a0 2024-03-19T01:54:51.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.355+0000 7fabf95be640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fabf4164c20 con 0x7fabf406b2a0 2024-03-19T01:54:51.354 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.355+0000 7fabf37fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabf4070c80 0x7fabf413fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:51.354 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.355+0000 7fabf27fc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fabf40708a0 0x7fabf4134d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:51.354 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.355+0000 7fabf2ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fabf406b2a0 0x7fabf41548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:51.354 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.355+0000 7fabf37fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabf4070c80 0x7fabf413fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57998/0 (socket says 172.21.15.55:57998) 2024-03-19T01:54:51.354 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.355+0000 7fabf37fe640 1 -- 172.21.15.55:0/1022579112 learned_addr learned my addr 172.21.15.55:0/1022579112 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:51.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.359+0000 7fabf37fe640 1 -- 172.21.15.55:0/1022579112 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fabf406b2a0 msgr2=0x7fabf41548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:51.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.359+0000 7fabf37fe640 1 --2- 172.21.15.55:0/1022579112 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fabf406b2a0 0x7fabf41548f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:51.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.359+0000 7fabf37fe640 1 -- 172.21.15.55:0/1022579112 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fabf40708a0 msgr2=0x7fabf4134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:51.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.359+0000 7fabf37fe640 1 --2- 172.21.15.55:0/1022579112 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fabf40708a0 0x7fabf4134d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:51.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.359+0000 7fabf37fe640 1 -- 172.21.15.55:0/1022579112 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabf41364d0 con 0x7fabf4070c80 2024-03-19T01:54:51.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.363+0000 7fabf37fe640 1 --2- 172.21.15.55:0/1022579112 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabf4070c80 0x7fabf413fe70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fabec007ef0 tx=0x7fabec00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:51.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.367+0000 7fabdb7fe640 1 -- 172.21.15.55:0/1022579112 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabec01a070 con 0x7fabf4070c80 2024-03-19T01:54:51.365 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.367+0000 7fabdb7fe640 1 -- 172.21.15.55:0/1022579112 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabec003ba0 con 0x7fabf4070c80 2024-03-19T01:54:51.365 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.367+0000 7fabf95be640 1 -- 172.21.15.55:0/1022579112 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabf41367b0 con 0x7fabf4070c80 2024-03-19T01:54:51.365 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.367+0000 7fabdb7fe640 1 -- 172.21.15.55:0/1022579112 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabec004030 con 0x7fabf4070c80 2024-03-19T01:54:51.365 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.367+0000 7fabf95be640 1 -- 172.21.15.55:0/1022579112 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabf413e520 con 0x7fabf4070c80 2024-03-19T01:54:51.365 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.367+0000 7fabf95be640 1 -- 172.21.15.55:0/1022579112 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabf4068a40 con 0x7fabf4070c80 2024-03-19T01:54:51.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.651+0000 7fabdb7fe640 1 -- 172.21.15.55:0/1022579112 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fabec01d070 con 0x7fabf4070c80 2024-03-19T01:54:51.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.651+0000 7fabdb7fe640 1 --2- 172.21.15.55:0/1022579112 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fabe00676e0 0x7fabe0069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:51.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.651+0000 7fabf2ffd640 1 --2- 172.21.15.55:0/1022579112 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fabe00676e0 0x7fabe0069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:51.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.651+0000 7fabf2ffd640 1 --2- 172.21.15.55:0/1022579112 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fabe00676e0 0x7fabe0069ba0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fabe40ad000 tx=0x7fabe4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:51.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.667+0000 7fabdb7fe640 1 -- 172.21.15.55:0/1022579112 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fabec0546b0 con 0x7fabf4070c80 2024-03-19T01:54:51.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.667+0000 7fabdb7fe640 1 -- 172.21.15.55:0/1022579112 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fabec010070 con 0x7fabf4070c80 2024-03-19T01:54:51.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.879+0000 7fabf95be640 1 -- 172.21.15.55:0/1022579112 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fabf406cce0 con 0x7fabf4070c80 2024-03-19T01:54:51.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.903+0000 7fabdb7fe640 1 -- 172.21.15.55:0/1022579112 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7fabec056610 con 0x7fabf4070c80 2024-03-19T01:54:51.901 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:54:51.901 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:54:51.901 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:54:51.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.907+0000 7fabf95be640 1 -- 172.21.15.55:0/1022579112 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fabe00676e0 msgr2=0x7fabe0069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:51.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.907+0000 7fabf95be640 1 --2- 172.21.15.55:0/1022579112 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fabe00676e0 0x7fabe0069ba0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fabe40ad000 tx=0x7fabe4008040 comp rx=0 tx=0).stop 2024-03-19T01:54:51.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.907+0000 7fabf95be640 1 -- 172.21.15.55:0/1022579112 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabf4070c80 msgr2=0x7fabf413fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:51.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.907+0000 7fabf95be640 1 --2- 172.21.15.55:0/1022579112 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabf4070c80 0x7fabf413fe70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fabec007ef0 tx=0x7fabec00d4a0 comp rx=0 tx=0).stop 2024-03-19T01:54:51.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.907+0000 7fabf95be640 1 -- 172.21.15.55:0/1022579112 shutdown_connections 2024-03-19T01:54:51.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.907+0000 7fabf95be640 1 --2- 172.21.15.55:0/1022579112 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fabe00676e0 0x7fabe0069ba0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:51.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.907+0000 7fabf95be640 1 --2- 172.21.15.55:0/1022579112 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabf4070c80 0x7fabf413fe70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:51.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.907+0000 7fabf95be640 1 --2- 172.21.15.55:0/1022579112 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fabf40708a0 0x7fabf4134d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:51.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.907+0000 7fabf95be640 1 --2- 172.21.15.55:0/1022579112 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fabf406b2a0 0x7fabf41548f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:51.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.907+0000 7fabf95be640 1 -- 172.21.15.55:0/1022579112 >> 172.21.15.55:0/1022579112 conn(0x7fabf406c580 msgr2=0x7fabf413ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:51.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.907+0000 7fabf95be640 1 -- 172.21.15.55:0/1022579112 shutdown_connections 2024-03-19T01:54:51.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:51.907+0000 7fabf95be640 1 -- 172.21.15.55:0/1022579112 wait complete. 2024-03-19T01:54:52.523 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:54:52.524 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:54:53.525 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:54:53.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.647+0000 7efc7d613640 1 Processor -- start 2024-03-19T01:54:53.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.647+0000 7efc7d613640 1 -- start start 2024-03-19T01:54:53.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.647+0000 7efc7d613640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7efc781510f0 0x7efc780c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:53.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.647+0000 7efc7d613640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7efc7815a410 0x7efc780708a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:53.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.647+0000 7efc7d613640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7efc780c73c0 0x7efc780c7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:53.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.647+0000 7efc7d613640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7efc780c3f60 con 0x7efc781510f0 2024-03-19T01:54:53.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.647+0000 7efc7d613640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7efc781514d0 con 0x7efc780c73c0 2024-03-19T01:54:53.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.647+0000 7efc7d613640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7efc780c6870 con 0x7efc7815a410 2024-03-19T01:54:53.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.647+0000 7efc76ffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7efc781510f0 0x7efc780c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:53.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.647+0000 7efc777fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7efc780c73c0 0x7efc780c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:53.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.651+0000 7efc767fc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7efc7815a410 0x7efc780708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:53.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.651+0000 7efc76ffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7efc781510f0 0x7efc780c61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58012/0 (socket says 172.21.15.55:58012) 2024-03-19T01:54:53.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.651+0000 7efc76ffd640 1 -- 172.21.15.55:0/956257665 learned_addr learned my addr 172.21.15.55:0/956257665 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:53.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.651+0000 7efc76ffd640 1 -- 172.21.15.55:0/956257665 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7efc7815a410 msgr2=0x7efc780708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:54:53.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.651+0000 7efc76ffd640 1 --2- 172.21.15.55:0/956257665 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7efc7815a410 0x7efc780708a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:53.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.651+0000 7efc76ffd640 1 -- 172.21.15.55:0/956257665 >> v2:172.21.15.84:3300/0 conn(0x7efc780c73c0 msgr2=0x7efc780c7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:54:53.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.651+0000 7efc76ffd640 1 --2- 172.21.15.55:0/956257665 >> v2:172.21.15.84:3300/0 conn(0x7efc780c73c0 0x7efc780c7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:53.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.651+0000 7efc76ffd640 1 -- 172.21.15.55:0/956257665 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc780c6b50 con 0x7efc781510f0 2024-03-19T01:54:53.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.655+0000 7efc76ffd640 1 --2- 172.21.15.55:0/956257665 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc781510f0 0x7efc780c61c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efc780c7f50 tx=0x7efc6801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91515b7c23574672 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:53.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.659+0000 7efc75ffb640 1 -- 172.21.15.55:0/956257665 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc680b5070 con 0x7efc781510f0 2024-03-19T01:54:53.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.659+0000 7efc75ffb640 1 -- 172.21.15.55:0/956257665 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc680a89d0 con 0x7efc781510f0 2024-03-19T01:54:53.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.659+0000 7efc75ffb640 1 -- 172.21.15.55:0/956257665 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc680a8da0 con 0x7efc781510f0 2024-03-19T01:54:53.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.663+0000 7efc7d613640 1 -- 172.21.15.55:0/956257665 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc781510f0 msgr2=0x7efc780c61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:53.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.663+0000 7efc7d613640 1 --2- 172.21.15.55:0/956257665 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc781510f0 0x7efc780c61c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efc780c7f50 tx=0x7efc6801ed10 comp rx=0 tx=0).stop 2024-03-19T01:54:53.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.663+0000 7efc7d613640 1 -- 172.21.15.55:0/956257665 shutdown_connections 2024-03-19T01:54:53.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.663+0000 7efc7d613640 1 --2- 172.21.15.55:0/956257665 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7efc7815a410 0x7efc780708a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:53.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.663+0000 7efc7d613640 1 --2- 172.21.15.55:0/956257665 >> v2:172.21.15.84:3300/0 conn(0x7efc780c73c0 0x7efc780c7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:53.661 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.663+0000 7efc7d613640 1 --2- 172.21.15.55:0/956257665 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc781510f0 0x7efc780c61c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:53.661 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.663+0000 7efc7d613640 1 -- 172.21.15.55:0/956257665 >> 172.21.15.55:0/956257665 conn(0x7efc780cc9c0 msgr2=0x7efc780ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:53.661 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.663+0000 7efc7d613640 1 -- 172.21.15.55:0/956257665 shutdown_connections 2024-03-19T01:54:53.661 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.663+0000 7efc7d613640 1 -- 172.21.15.55:0/956257665 wait complete. 2024-03-19T01:54:53.661 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.663+0000 7efc7d613640 1 Processor -- start 2024-03-19T01:54:53.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.663+0000 7efc7d613640 1 -- start start 2024-03-19T01:54:53.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.667+0000 7efc7d613640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc780c73c0 0x7efc78151b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:53.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.667+0000 7efc7d613640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efc781510f0 0x7efc78143580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:53.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.667+0000 7efc7d613640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7efc7815a410 0x7efc781f03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:53.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.667+0000 7efc7d613640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7efc780c6e30 con 0x7efc780c73c0 2024-03-19T01:54:53.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.667+0000 7efc7d613640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7efc780c44e0 con 0x7efc781510f0 2024-03-19T01:54:53.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.667+0000 7efc7d613640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7efc780c4a00 con 0x7efc7815a410 2024-03-19T01:54:53.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.667+0000 7efc76ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc780c73c0 0x7efc78151b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:53.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.667+0000 7efc76ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc780c73c0 0x7efc78151b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58016/0 (socket says 172.21.15.55:58016) 2024-03-19T01:54:53.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.667+0000 7efc76ffd640 1 -- 172.21.15.55:0/1138516837 learned_addr learned my addr 172.21.15.55:0/1138516837 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:53.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.667+0000 7efc777fe640 1 --2- 172.21.15.55:0/1138516837 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7efc7815a410 0x7efc781f03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:53.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.667+0000 7efc767fc640 1 --2- 172.21.15.55:0/1138516837 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efc781510f0 0x7efc78143580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:53.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.671+0000 7efc76ffd640 1 -- 172.21.15.55:0/1138516837 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7efc7815a410 msgr2=0x7efc781f03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:53.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.671+0000 7efc76ffd640 1 --2- 172.21.15.55:0/1138516837 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7efc7815a410 0x7efc781f03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:53.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.671+0000 7efc76ffd640 1 -- 172.21.15.55:0/1138516837 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efc781510f0 msgr2=0x7efc78143580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:53.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.671+0000 7efc76ffd640 1 --2- 172.21.15.55:0/1138516837 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efc781510f0 0x7efc78143580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:53.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.671+0000 7efc76ffd640 1 -- 172.21.15.55:0/1138516837 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc78147700 con 0x7efc780c73c0 2024-03-19T01:54:53.672 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.675+0000 7efc76ffd640 1 --2- 172.21.15.55:0/1138516837 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc780c73c0 0x7efc78151b10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7efc6801d3c0 tx=0x7efc6801bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:53.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.679+0000 7efc5f7fe640 1 -- 172.21.15.55:0/1138516837 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc680b5070 con 0x7efc780c73c0 2024-03-19T01:54:53.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.679+0000 7efc5f7fe640 1 -- 172.21.15.55:0/1138516837 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc680b0950 con 0x7efc780c73c0 2024-03-19T01:54:53.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.679+0000 7efc5f7fe640 1 -- 172.21.15.55:0/1138516837 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc680b0d40 con 0x7efc780c73c0 2024-03-19T01:54:53.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.679+0000 7efc7d613640 1 -- 172.21.15.55:0/1138516837 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc781521c0 con 0x7efc780c73c0 2024-03-19T01:54:53.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.679+0000 7efc7d613640 1 -- 172.21.15.55:0/1138516837 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc781479a0 con 0x7efc780c73c0 2024-03-19T01:54:53.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.679+0000 7efc7d613640 1 -- 172.21.15.55:0/1138516837 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc78068a40 con 0x7efc780c73c0 2024-03-19T01:54:53.956 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.959+0000 7efc5f7fe640 1 -- 172.21.15.55:0/1138516837 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7efc680c2060 con 0x7efc780c73c0 2024-03-19T01:54:53.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.959+0000 7efc5f7fe640 1 --2- 172.21.15.55:0/1138516837 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7efc640676b0 0x7efc64069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:53.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.959+0000 7efc767fc640 1 --2- 172.21.15.55:0/1138516837 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7efc640676b0 0x7efc64069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:53.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.959+0000 7efc767fc640 1 --2- 172.21.15.55:0/1138516837 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7efc640676b0 0x7efc64069b70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efc7815b3b0 tx=0x7efc6c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:53.983 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.987+0000 7efc5f7fe640 1 -- 172.21.15.55:0/1138516837 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7efc680f11f0 con 0x7efc780c73c0 2024-03-19T01:54:53.984 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:53.987+0000 7efc5f7fe640 1 -- 172.21.15.55:0/1138516837 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7efc68018870 con 0x7efc780c73c0 2024-03-19T01:54:54.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:54.195+0000 7efc7d613640 1 -- 172.21.15.55:0/1138516837 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efc78143c00 con 0x7efc780c73c0 2024-03-19T01:54:54.209 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:54.211+0000 7efc5f7fe640 1 -- 172.21.15.55:0/1138516837 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7efc680f3150 con 0x7efc780c73c0 2024-03-19T01:54:54.210 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:54:54.210 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:54:54.210 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:54:54.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:54.215+0000 7efc7d613640 1 -- 172.21.15.55:0/1138516837 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7efc640676b0 msgr2=0x7efc64069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:54.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:54.215+0000 7efc7d613640 1 --2- 172.21.15.55:0/1138516837 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7efc640676b0 0x7efc64069b70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efc7815b3b0 tx=0x7efc6c009190 comp rx=0 tx=0).stop 2024-03-19T01:54:54.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:54.215+0000 7efc7d613640 1 -- 172.21.15.55:0/1138516837 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc780c73c0 msgr2=0x7efc78151b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:54.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:54.215+0000 7efc7d613640 1 --2- 172.21.15.55:0/1138516837 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc780c73c0 0x7efc78151b10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7efc6801d3c0 tx=0x7efc6801bab0 comp rx=0 tx=0).stop 2024-03-19T01:54:54.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:54.215+0000 7efc7d613640 1 -- 172.21.15.55:0/1138516837 shutdown_connections 2024-03-19T01:54:54.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:54.215+0000 7efc7d613640 1 --2- 172.21.15.55:0/1138516837 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7efc640676b0 0x7efc64069b70 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:54.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:54.215+0000 7efc7d613640 1 --2- 172.21.15.55:0/1138516837 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7efc7815a410 0x7efc781f03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:54.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:54.215+0000 7efc7d613640 1 --2- 172.21.15.55:0/1138516837 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efc781510f0 0x7efc78143580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:54.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:54.215+0000 7efc7d613640 1 --2- 172.21.15.55:0/1138516837 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc780c73c0 0x7efc78151b10 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:54.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:54.215+0000 7efc7d613640 1 -- 172.21.15.55:0/1138516837 >> 172.21.15.55:0/1138516837 conn(0x7efc780cc9c0 msgr2=0x7efc780c8b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:54.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:54.215+0000 7efc7d613640 1 -- 172.21.15.55:0/1138516837 shutdown_connections 2024-03-19T01:54:54.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:54.215+0000 7efc7d613640 1 -- 172.21.15.55:0/1138516837 wait complete. 2024-03-19T01:54:54.832 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:54:54.832 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:54:55.833 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:54:55.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.963+0000 7f5ab09ea640 1 Processor -- start 2024-03-19T01:54:55.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.963+0000 7f5ab09ea640 1 -- start start 2024-03-19T01:54:55.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.967+0000 7f5ab09ea640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f5aac0c52b0 0x7f5aac151d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:55.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.967+0000 7f5ab09ea640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f5aac0c89f0 0x7f5aac152260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:55.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.967+0000 7f5ab09ea640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aac0724c0 con 0x7f5aac0c5690 2024-03-19T01:54:55.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.967+0000 7f5ab09ea640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f5aac13d290 con 0x7f5aac0c89f0 2024-03-19T01:54:55.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.967+0000 7f5ab09ea640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f5aac13d570 con 0x7f5aac0c52b0 2024-03-19T01:54:55.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.967+0000 7f5aa9d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f5aac0c89f0 0x7f5aac152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:55.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.967+0000 7f5aaa59c640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f5aac0c52b0 0x7f5aac151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:55.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.967+0000 7f5aa9d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f5aac0c89f0 0x7f5aac152260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58030/0 (socket says 172.21.15.55:58030) 2024-03-19T01:54:55.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.967+0000 7f5aa9d9b640 1 -- 172.21.15.55:0/2218848726 learned_addr learned my addr 172.21.15.55:0/2218848726 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:55.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.971+0000 7f5aa9d9b640 1 -- 172.21.15.55:0/2218848726 >> v2:172.21.15.84:3300/0 conn(0x7f5aac0c52b0 msgr2=0x7f5aac151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:54:55.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.971+0000 7f5aa9d9b640 1 --2- 172.21.15.55:0/2218848726 >> v2:172.21.15.84:3300/0 conn(0x7f5aac0c52b0 0x7f5aac151d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:55.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.971+0000 7f5aa9d9b640 1 -- 172.21.15.55:0/2218848726 >> v1:172.21.15.55:6789/0 conn(0x7f5aac0c5690 legacy=0x7f5aac158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:55.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.971+0000 7f5aa9d9b640 1 -- 172.21.15.55:0/2218848726 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aac13d850 con 0x7f5aac0c89f0 2024-03-19T01:54:55.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.975+0000 7f5aa9d9b640 1 --2- 172.21.15.55:0/2218848726 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5aac0c89f0 0x7f5aac152260 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5aac0cca00 tx=0x7f5aa001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=addebf763e5f79a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:55.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.979+0000 7f5aa959a640 1 -- 172.21.15.55:0/2218848726 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa00b5070 con 0x7f5aac0c89f0 2024-03-19T01:54:55.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.979+0000 7f5aa959a640 1 -- 172.21.15.55:0/2218848726 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5aa00a8a10 con 0x7f5aac0c89f0 2024-03-19T01:54:55.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.979+0000 7f5aa959a640 1 -- 172.21.15.55:0/2218848726 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa00a8de0 con 0x7f5aac0c89f0 2024-03-19T01:54:55.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.979+0000 7f5ab09ea640 1 -- 172.21.15.55:0/2218848726 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5aac0c89f0 msgr2=0x7f5aac152260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:55.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.979+0000 7f5ab09ea640 1 --2- 172.21.15.55:0/2218848726 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5aac0c89f0 0x7f5aac152260 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5aac0cca00 tx=0x7f5aa001ed10 comp rx=0 tx=0).stop 2024-03-19T01:54:55.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.979+0000 7f5ab09ea640 1 -- 172.21.15.55:0/2218848726 shutdown_connections 2024-03-19T01:54:55.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.979+0000 7f5ab09ea640 1 --2- 172.21.15.55:0/2218848726 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5aac0c89f0 0x7f5aac152260 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:55.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.979+0000 7f5ab09ea640 1 --2- 172.21.15.55:0/2218848726 >> v2:172.21.15.84:3300/0 conn(0x7f5aac0c52b0 0x7f5aac151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:55.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.979+0000 7f5ab09ea640 1 -- 172.21.15.55:0/2218848726 >> 172.21.15.55:0/2218848726 conn(0x7f5aac06f3b0 msgr2=0x7f5aac06f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:55.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.979+0000 7f5ab09ea640 1 -- 172.21.15.55:0/2218848726 shutdown_connections 2024-03-19T01:54:55.979 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.979+0000 7f5ab09ea640 1 -- 172.21.15.55:0/2218848726 wait complete. 2024-03-19T01:54:55.979 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.983+0000 7f5ab09ea640 1 Processor -- start 2024-03-19T01:54:55.980 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.983+0000 7f5ab09ea640 1 -- start start 2024-03-19T01:54:55.981 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.983+0000 7f5ab09ea640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5aac0c52b0 0x7f5aac157090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:55.981 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.983+0000 7f5ab09ea640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5aac0c5690 0x7f5aac155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:55.981 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.983+0000 7f5ab09ea640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5aac0c89f0 0x7f5aac138550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:55.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.983+0000 7f5ab09ea640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5aac158780 con 0x7f5aac0c89f0 2024-03-19T01:54:55.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.983+0000 7f5ab09ea640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f5aac13de10 con 0x7f5aac0c5690 2024-03-19T01:54:55.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.983+0000 7f5ab09ea640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f5aac158900 con 0x7f5aac0c52b0 2024-03-19T01:54:55.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.983+0000 7f5aaad9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5aac0c89f0 0x7f5aac138550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:55.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.983+0000 7f5aaad9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5aac0c89f0 0x7f5aac138550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58042/0 (socket says 172.21.15.55:58042) 2024-03-19T01:54:55.983 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.983+0000 7f5aaad9d640 1 -- 172.21.15.55:0/4150644570 learned_addr learned my addr 172.21.15.55:0/4150644570 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:55.983 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.983+0000 7f5aaa59c640 1 --2- 172.21.15.55:0/4150644570 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5aac0c52b0 0x7f5aac157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:55.983 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.983+0000 7f5aa9d9b640 1 --2- 172.21.15.55:0/4150644570 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5aac0c5690 0x7f5aac155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:55.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.987+0000 7f5aaad9d640 1 -- 172.21.15.55:0/4150644570 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5aac0c52b0 msgr2=0x7f5aac157090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:55.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.987+0000 7f5aaad9d640 1 --2- 172.21.15.55:0/4150644570 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5aac0c52b0 0x7f5aac157090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:55.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.987+0000 7f5aaad9d640 1 -- 172.21.15.55:0/4150644570 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5aac0c5690 msgr2=0x7f5aac155300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:55.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.987+0000 7f5aaad9d640 1 --2- 172.21.15.55:0/4150644570 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5aac0c5690 0x7f5aac155300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:55.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.987+0000 7f5aaad9d640 1 -- 172.21.15.55:0/4150644570 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aac1f3590 con 0x7f5aac0c89f0 2024-03-19T01:54:55.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.991+0000 7f5aaad9d640 1 --2- 172.21.15.55:0/4150644570 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5aac0c89f0 0x7f5aac138550 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5aa400c6f0 tx=0x7f5aa400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:55.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.995+0000 7f5a9affd640 1 -- 172.21.15.55:0/4150644570 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4003980 con 0x7f5aac0c89f0 2024-03-19T01:54:55.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.995+0000 7f5a9affd640 1 -- 172.21.15.55:0/4150644570 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5aa4003b20 con 0x7f5aac0c89f0 2024-03-19T01:54:55.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.995+0000 7f5ab09ea640 1 -- 172.21.15.55:0/4150644570 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aac1f3830 con 0x7f5aac0c89f0 2024-03-19T01:54:55.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.995+0000 7f5a9affd640 1 -- 172.21.15.55:0/4150644570 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa400a850 con 0x7f5aac0c89f0 2024-03-19T01:54:55.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.995+0000 7f5ab09ea640 1 -- 172.21.15.55:0/4150644570 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aac1f3df0 con 0x7f5aac0c89f0 2024-03-19T01:54:55.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:55.995+0000 7f5ab09ea640 1 -- 172.21.15.55:0/4150644570 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5aac068330 con 0x7f5aac0c89f0 2024-03-19T01:54:56.270 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:56.271+0000 7f5a9affd640 1 -- 172.21.15.55:0/4150644570 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f5aa400a9f0 con 0x7f5aac0c89f0 2024-03-19T01:54:56.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:56.275+0000 7f5a9affd640 1 --2- 172.21.15.55:0/4150644570 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5a8c0676e0 0x7f5a8c069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:56.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:56.275+0000 7f5aaa59c640 1 --2- 172.21.15.55:0/4150644570 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5a8c0676e0 0x7f5a8c069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:56.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:56.275+0000 7f5aaa59c640 1 --2- 172.21.15.55:0/4150644570 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5a8c0676e0 0x7f5a8c069ba0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c009ac0 tx=0x7f5a9c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:56.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:56.295+0000 7f5a9affd640 1 -- 172.21.15.55:0/4150644570 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f5aa4054770 con 0x7f5aac0c89f0 2024-03-19T01:54:56.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:56.295+0000 7f5a9affd640 1 -- 172.21.15.55:0/4150644570 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f5aa40566d0 con 0x7f5aac0c89f0 2024-03-19T01:54:56.499 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:56.503+0000 7f5ab09ea640 1 -- 172.21.15.55:0/4150644570 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5aac13cb20 con 0x7f5aac0c89f0 2024-03-19T01:54:56.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:56.527+0000 7f5a9affd640 1 -- 172.21.15.55:0/4150644570 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7f5aa405a770 con 0x7f5aac0c89f0 2024-03-19T01:54:56.525 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:54:56.525 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:54:56.525 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:54:56.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:56.531+0000 7f5ab09ea640 1 -- 172.21.15.55:0/4150644570 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5a8c0676e0 msgr2=0x7f5a8c069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:56.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:56.531+0000 7f5ab09ea640 1 --2- 172.21.15.55:0/4150644570 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5a8c0676e0 0x7f5a8c069ba0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c009ac0 tx=0x7f5a9c009190 comp rx=0 tx=0).stop 2024-03-19T01:54:56.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:56.531+0000 7f5ab09ea640 1 -- 172.21.15.55:0/4150644570 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5aac0c89f0 msgr2=0x7f5aac138550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:56.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:56.531+0000 7f5ab09ea640 1 --2- 172.21.15.55:0/4150644570 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5aac0c89f0 0x7f5aac138550 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5aa400c6f0 tx=0x7f5aa400cbb0 comp rx=0 tx=0).stop 2024-03-19T01:54:56.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:56.531+0000 7f5ab09ea640 1 -- 172.21.15.55:0/4150644570 shutdown_connections 2024-03-19T01:54:56.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:56.531+0000 7f5ab09ea640 1 --2- 172.21.15.55:0/4150644570 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5a8c0676e0 0x7f5a8c069ba0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:56.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:56.531+0000 7f5ab09ea640 1 --2- 172.21.15.55:0/4150644570 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5aac0c89f0 0x7f5aac138550 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:56.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:56.531+0000 7f5ab09ea640 1 --2- 172.21.15.55:0/4150644570 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5aac0c5690 0x7f5aac155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:56.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:56.531+0000 7f5ab09ea640 1 --2- 172.21.15.55:0/4150644570 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5aac0c52b0 0x7f5aac157090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:56.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:56.531+0000 7f5ab09ea640 1 -- 172.21.15.55:0/4150644570 >> 172.21.15.55:0/4150644570 conn(0x7f5aac06f3b0 msgr2=0x7f5aac13b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:56.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:56.531+0000 7f5ab09ea640 1 -- 172.21.15.55:0/4150644570 shutdown_connections 2024-03-19T01:54:56.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:56.531+0000 7f5ab09ea640 1 -- 172.21.15.55:0/4150644570 wait complete. 2024-03-19T01:54:57.154 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:54:57.154 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:54:58.155 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:54:58.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.275+0000 7f069795e640 1 Processor -- start 2024-03-19T01:54:58.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.275+0000 7f069795e640 1 -- start start 2024-03-19T01:54:58.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.275+0000 7f069795e640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f06900c98e0 0x7f069015e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:58.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.275+0000 7f069795e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0690071280 con 0x7f0690154340 2024-03-19T01:54:58.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.275+0000 7f069795e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f069015d900 con 0x7f0690154720 2024-03-19T01:54:58.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.275+0000 7f069795e640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f0690073840 con 0x7f06900c98e0 2024-03-19T01:54:58.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.279+0000 7f06956fa640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f0690154340 0x7f0690072e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:33778/0 (socket says 172.21.15.55:33778) 2024-03-19T01:54:58.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.279+0000 7f06956fa640 1 -- 172.21.15.55:0/1938529841 learned_addr learned my addr 172.21.15.55:0/1938529841 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:54:58.276 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.279+0000 7f0695efb640 1 --2- 172.21.15.55:0/1938529841 >> v2:172.21.15.84:3300/0 conn(0x7f06900c98e0 0x7f069015e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:58.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.283+0000 7f0687fff640 1 -- 172.21.15.55:0/1938529841 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2388474132 0 0) 0x7f0690071280 con 0x7f0690154340 2024-03-19T01:54:58.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.283+0000 7f0687fff640 1 -- 172.21.15.55:0/1938529841 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f067c003530 con 0x7f0690154340 2024-03-19T01:54:58.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.283+0000 7f0695efb640 1 -- 172.21.15.55:0/1938529841 >> v1:172.21.15.84:6789/0 conn(0x7f0690154720 legacy=0x7f06900c91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:54:58.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.283+0000 7f0695efb640 1 -- 172.21.15.55:0/1938529841 >> v1:172.21.15.55:6789/0 conn(0x7f0690154340 legacy=0x7f0690072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:58.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.283+0000 7f0695efb640 1 -- 172.21.15.55:0/1938529841 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0690070cc0 con 0x7f06900c98e0 2024-03-19T01:54:58.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.291+0000 7f0695efb640 1 --2- 172.21.15.55:0/1938529841 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f06900c98e0 0x7f069015e3a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0688009d90 tx=0x7f068801c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d785d314c7b630a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:58.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.295+0000 7f0687fff640 1 -- 172.21.15.55:0/1938529841 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06880b2070 con 0x7f06900c98e0 2024-03-19T01:54:58.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.299+0000 7f0687fff640 1 -- 172.21.15.55:0/1938529841 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06880a57f0 con 0x7f06900c98e0 2024-03-19T01:54:58.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.299+0000 7f0687fff640 1 -- 172.21.15.55:0/1938529841 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06880a5ba0 con 0x7f06900c98e0 2024-03-19T01:54:58.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.299+0000 7f069795e640 1 -- 172.21.15.55:0/1938529841 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f06900c98e0 msgr2=0x7f069015e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:58.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.299+0000 7f069795e640 1 --2- 172.21.15.55:0/1938529841 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f06900c98e0 0x7f069015e3a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0688009d90 tx=0x7f068801c930 comp rx=0 tx=0).stop 2024-03-19T01:54:58.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.299+0000 7f069795e640 1 -- 172.21.15.55:0/1938529841 shutdown_connections 2024-03-19T01:54:58.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.299+0000 7f069795e640 1 --2- 172.21.15.55:0/1938529841 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f06900c98e0 0x7f069015e3a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:58.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.299+0000 7f069795e640 1 -- 172.21.15.55:0/1938529841 >> 172.21.15.55:0/1938529841 conn(0x7f06900cc9c0 msgr2=0x7f06900ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:58.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.299+0000 7f069795e640 1 -- 172.21.15.55:0/1938529841 shutdown_connections 2024-03-19T01:54:58.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.299+0000 7f069795e640 1 -- 172.21.15.55:0/1938529841 wait complete. 2024-03-19T01:54:58.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.299+0000 7f069795e640 1 Processor -- start 2024-03-19T01:54:58.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.299+0000 7f069795e640 1 -- start start 2024-03-19T01:54:58.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.303+0000 7f069795e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f06900c98e0 0x7f06901577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:58.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.303+0000 7f069795e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0690154340 0x7f06900c7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:58.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.303+0000 7f069795e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0690154720 0x7f06901f4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:58.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.303+0000 7f069795e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0690168390 con 0x7f0690154720 2024-03-19T01:54:58.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.303+0000 7f069795e640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f069015b0b0 con 0x7f06900c98e0 2024-03-19T01:54:58.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.303+0000 7f069795e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f0690168510 con 0x7f0690154340 2024-03-19T01:54:58.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.303+0000 7f0694ef9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0690154340 0x7f06900c7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:58.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.303+0000 7f0694ef9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0690154340 0x7f06900c7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:37158/0 (socket says 172.21.15.55:37158) 2024-03-19T01:54:58.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.303+0000 7f0694ef9640 1 -- 172.21.15.55:0/1202289348 learned_addr learned my addr 172.21.15.55:0/1202289348 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:54:58.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.307+0000 7f06956fa640 1 --2- 172.21.15.55:0/1202289348 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f06900c98e0 0x7f06901577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:58.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.307+0000 7f0695efb640 1 --2- 172.21.15.55:0/1202289348 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0690154720 0x7f06901f4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:58.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.315+0000 7f0694ef9640 1 -- 172.21.15.55:0/1202289348 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f06900c98e0 msgr2=0x7f06901577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:58.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.315+0000 7f0694ef9640 1 --2- 172.21.15.55:0/1202289348 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f06900c98e0 0x7f06901577a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:58.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.315+0000 7f0694ef9640 1 -- 172.21.15.55:0/1202289348 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0690154720 msgr2=0x7f06901f4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:58.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.315+0000 7f0694ef9640 1 --2- 172.21.15.55:0/1202289348 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0690154720 0x7f06901f4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:58.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.315+0000 7f0694ef9640 1 -- 172.21.15.55:0/1202289348 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06901566c0 con 0x7f0690154340 2024-03-19T01:54:58.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.319+0000 7f0694ef9640 1 --2- 172.21.15.55:0/1202289348 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0690154340 0x7f06900c7c40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f068000b440 tx=0x7f068000b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:58.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.327+0000 7f0685ffb640 1 -- 172.21.15.55:0/1202289348 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0680015020 con 0x7f0690154340 2024-03-19T01:54:58.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.327+0000 7f0685ffb640 1 -- 172.21.15.55:0/1202289348 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0680011890 con 0x7f0690154340 2024-03-19T01:54:58.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.327+0000 7f0685ffb640 1 -- 172.21.15.55:0/1202289348 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0680011c80 con 0x7f0690154340 2024-03-19T01:54:58.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.327+0000 7f069795e640 1 -- 172.21.15.55:0/1202289348 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06901569a0 con 0x7f0690154340 2024-03-19T01:54:58.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.327+0000 7f069795e640 1 -- 172.21.15.55:0/1202289348 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0690156f60 con 0x7f0690154340 2024-03-19T01:54:58.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.331+0000 7f069795e640 1 -- 172.21.15.55:0/1202289348 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0690068a40 con 0x7f0690154340 2024-03-19T01:54:58.613 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.615+0000 7f0685ffb640 1 -- 172.21.15.55:0/1202289348 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f0680018020 con 0x7f0690154340 2024-03-19T01:54:58.614 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.615+0000 7f0685ffb640 1 --2- 172.21.15.55:0/1202289348 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f06740676e0 0x7f0674069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:54:58.614 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.615+0000 7f06956fa640 1 --2- 172.21.15.55:0/1202289348 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f06740676e0 0x7f0674069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:54:58.615 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.619+0000 7f06956fa640 1 --2- 172.21.15.55:0/1202289348 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f06740676e0 0x7f0674069ba0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f068c007fd0 tx=0x7f068c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:54:58.616 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.619+0000 7f0685ffb640 1 -- 172.21.15.55:0/1202289348 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f0680054070 con 0x7f0690154340 2024-03-19T01:54:58.631 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.635+0000 7f0685ffb640 1 -- 172.21.15.55:0/1202289348 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f068001f030 con 0x7f0690154340 2024-03-19T01:54:58.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:58.867+0000 7f069795e640 1 -- 172.21.15.55:0/1202289348 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06900c8710 con 0x7f0690154340 2024-03-19T01:54:59.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:59.071+0000 7f0685ffb640 1 -- 172.21.15.55:0/1202289348 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7f068000d040 con 0x7f0690154340 2024-03-19T01:54:59.067 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:54:59.068 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:54:59.068 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:54:59.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:59.075+0000 7f069795e640 1 -- 172.21.15.55:0/1202289348 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f06740676e0 msgr2=0x7f0674069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:59.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:59.075+0000 7f069795e640 1 --2- 172.21.15.55:0/1202289348 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f06740676e0 0x7f0674069ba0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f068c007fd0 tx=0x7f068c009040 comp rx=0 tx=0).stop 2024-03-19T01:54:59.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:59.075+0000 7f069795e640 1 -- 172.21.15.55:0/1202289348 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0690154340 msgr2=0x7f06900c7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:54:59.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:59.075+0000 7f069795e640 1 --2- 172.21.15.55:0/1202289348 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0690154340 0x7f06900c7c40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f068000b440 tx=0x7f068000b900 comp rx=0 tx=0).stop 2024-03-19T01:54:59.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:59.075+0000 7f069795e640 1 -- 172.21.15.55:0/1202289348 shutdown_connections 2024-03-19T01:54:59.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:59.075+0000 7f069795e640 1 --2- 172.21.15.55:0/1202289348 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f06740676e0 0x7f0674069ba0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:59.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:59.075+0000 7f069795e640 1 --2- 172.21.15.55:0/1202289348 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0690154720 0x7f06901f4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:59.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:59.075+0000 7f069795e640 1 --2- 172.21.15.55:0/1202289348 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0690154340 0x7f06900c7c40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:59.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:59.075+0000 7f069795e640 1 --2- 172.21.15.55:0/1202289348 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f06900c98e0 0x7f06901577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:54:59.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:59.075+0000 7f069795e640 1 -- 172.21.15.55:0/1202289348 >> 172.21.15.55:0/1202289348 conn(0x7f06900cc9c0 msgr2=0x7f069006cdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:54:59.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:59.075+0000 7f069795e640 1 -- 172.21.15.55:0/1202289348 shutdown_connections 2024-03-19T01:54:59.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:54:59.075+0000 7f069795e640 1 -- 172.21.15.55:0/1202289348 wait complete. 2024-03-19T01:54:59.695 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:54:59.695 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:00.696 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:00.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.819+0000 7f234ece7640 1 Processor -- start 2024-03-19T01:55:00.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.819+0000 7f234ece7640 1 -- start start 2024-03-19T01:55:00.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.819+0000 7f234ece7640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f234806b2a0 0x7f2348073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:00.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.819+0000 7f234ece7640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f234815acb0 0x7f23480cce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:00.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.819+0000 7f234ece7640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23480c7fa0 con 0x7f23481544f0 2024-03-19T01:55:00.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.819+0000 7f234ece7640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f234815b210 con 0x7f234815acb0 2024-03-19T01:55:00.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.819+0000 7f234ece7640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f23480c8240 con 0x7f234806b2a0 2024-03-19T01:55:00.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.819+0000 7f234d284640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f234815acb0 0x7f23480cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:00.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.823+0000 7f234ca83640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f23481544f0 0x7f234815e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:56172/0 (socket says 172.21.15.55:56172) 2024-03-19T01:55:00.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.823+0000 7f234ca83640 1 -- 172.21.15.55:0/2964767348 learned_addr learned my addr 172.21.15.55:0/2964767348 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:55:00.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.823+0000 7f2347fff640 1 --2- 172.21.15.55:0/2964767348 >> v2:172.21.15.84:3300/0 conn(0x7f234806b2a0 0x7f2348073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:00.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.827+0000 7f234d284640 1 -- 172.21.15.55:0/2964767348 >> v2:172.21.15.84:3300/0 conn(0x7f234806b2a0 msgr2=0x7f2348073540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:00.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.827+0000 7f234d284640 1 --2- 172.21.15.55:0/2964767348 >> v2:172.21.15.84:3300/0 conn(0x7f234806b2a0 0x7f2348073540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:00.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.827+0000 7f234d284640 1 -- 172.21.15.55:0/2964767348 >> v1:172.21.15.55:6789/0 conn(0x7f23481544f0 legacy=0x7f234815e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:00.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.827+0000 7f234d284640 1 -- 172.21.15.55:0/2964767348 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f234815d920 con 0x7f234815acb0 2024-03-19T01:55:00.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.831+0000 7f234d284640 1 --2- 172.21.15.55:0/2964767348 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f234815acb0 0x7f23480cce30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f23480ca790 tx=0x7f234001f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=734d1970484faa89 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:00.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.835+0000 7f23477fe640 1 -- 172.21.15.55:0/2964767348 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23400b4070 con 0x7f234815acb0 2024-03-19T01:55:00.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.835+0000 7f23477fe640 1 -- 172.21.15.55:0/2964767348 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23400af600 con 0x7f234815acb0 2024-03-19T01:55:00.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.835+0000 7f23477fe640 1 -- 172.21.15.55:0/2964767348 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23400af9d0 con 0x7f234815acb0 2024-03-19T01:55:00.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.835+0000 7f234ece7640 1 -- 172.21.15.55:0/2964767348 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f234815acb0 msgr2=0x7f23480cce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:00.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.835+0000 7f234ece7640 1 --2- 172.21.15.55:0/2964767348 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f234815acb0 0x7f23480cce30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f23480ca790 tx=0x7f234001f110 comp rx=0 tx=0).stop 2024-03-19T01:55:00.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.835+0000 7f234ece7640 1 -- 172.21.15.55:0/2964767348 shutdown_connections 2024-03-19T01:55:00.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.835+0000 7f234ece7640 1 --2- 172.21.15.55:0/2964767348 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f234815acb0 0x7f23480cce30 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:00.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.835+0000 7f234ece7640 1 --2- 172.21.15.55:0/2964767348 >> v2:172.21.15.84:3300/0 conn(0x7f234806b2a0 0x7f2348073540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:00.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.835+0000 7f234ece7640 1 -- 172.21.15.55:0/2964767348 >> 172.21.15.55:0/2964767348 conn(0x7f234806e670 msgr2=0x7f234806ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:00.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.839+0000 7f234ece7640 1 -- 172.21.15.55:0/2964767348 shutdown_connections 2024-03-19T01:55:00.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.839+0000 7f234ece7640 1 -- 172.21.15.55:0/2964767348 wait complete. 2024-03-19T01:55:00.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.839+0000 7f234ece7640 1 Processor -- start 2024-03-19T01:55:00.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.839+0000 7f234ece7640 1 -- start start 2024-03-19T01:55:00.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.839+0000 7f234ece7640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f234806b2a0 0x7f23480c5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:00.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.839+0000 7f234ece7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23481544f0 0x7f2348159e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:00.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.839+0000 7f234ece7640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f234815acb0 0x7f2348152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:00.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.839+0000 7f234ece7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f23480ca120 con 0x7f23481544f0 2024-03-19T01:55:00.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.839+0000 7f234ece7640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f23481548d0 con 0x7f234815acb0 2024-03-19T01:55:00.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.839+0000 7f234ece7640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f23480ca2a0 con 0x7f234806b2a0 2024-03-19T01:55:00.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.839+0000 7f2347fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23481544f0 0x7f2348159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:00.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.839+0000 7f2347fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23481544f0 0x7f2348159e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52980/0 (socket says 172.21.15.55:52980) 2024-03-19T01:55:00.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.839+0000 7f2347fff640 1 -- 172.21.15.55:0/4191970199 learned_addr learned my addr 172.21.15.55:0/4191970199 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:00.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.839+0000 7f234d284640 1 --2- 172.21.15.55:0/4191970199 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f234815acb0 0x7f2348152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:00.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.839+0000 7f234ca83640 1 --2- 172.21.15.55:0/4191970199 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f234806b2a0 0x7f23480c5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:00.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.843+0000 7f2347fff640 1 -- 172.21.15.55:0/4191970199 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f234806b2a0 msgr2=0x7f23480c5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:00.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.843+0000 7f2347fff640 1 --2- 172.21.15.55:0/4191970199 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f234806b2a0 0x7f23480c5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:00.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.843+0000 7f2347fff640 1 -- 172.21.15.55:0/4191970199 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f234815acb0 msgr2=0x7f2348152ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:00.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.843+0000 7f2347fff640 1 --2- 172.21.15.55:0/4191970199 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f234815acb0 0x7f2348152ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:00.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.843+0000 7f2347fff640 1 -- 172.21.15.55:0/4191970199 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23480c40c0 con 0x7f23481544f0 2024-03-19T01:55:00.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.847+0000 7f2347fff640 1 --2- 172.21.15.55:0/4191970199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23481544f0 0x7f2348159e00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f233c00e930 tx=0x7f233c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:00.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.851+0000 7f23457fa640 1 -- 172.21.15.55:0/4191970199 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c017070 con 0x7f23481544f0 2024-03-19T01:55:00.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.851+0000 7f23457fa640 1 -- 172.21.15.55:0/4191970199 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f233c00ca00 con 0x7f23481544f0 2024-03-19T01:55:00.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.851+0000 7f234ece7640 1 -- 172.21.15.55:0/4191970199 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23480c43a0 con 0x7f23481544f0 2024-03-19T01:55:00.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.851+0000 7f23457fa640 1 -- 172.21.15.55:0/4191970199 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c00cdb0 con 0x7f23481544f0 2024-03-19T01:55:00.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.851+0000 7f234ece7640 1 -- 172.21.15.55:0/4191970199 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23480c4960 con 0x7f23481544f0 2024-03-19T01:55:00.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.851+0000 7f234ece7640 1 -- 172.21.15.55:0/4191970199 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2348068a40 con 0x7f23481544f0 2024-03-19T01:55:00.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.871+0000 7f23457fa640 1 -- 172.21.15.55:0/4191970199 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f233c01a070 con 0x7f23481544f0 2024-03-19T01:55:00.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.871+0000 7f23457fa640 1 --2- 172.21.15.55:0/4191970199 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f23340676e0 0x7f2334069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:00.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.871+0000 7f234ca83640 1 --2- 172.21.15.55:0/4191970199 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f23340676e0 0x7f2334069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:00.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.871+0000 7f234ca83640 1 --2- 172.21.15.55:0/4191970199 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f23340676e0 0x7f2334069ba0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2338000f70 tx=0x7f2338007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:00.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:00.875+0000 7f23457fa640 1 -- 172.21.15.55:0/4191970199 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f233c054790 con 0x7f23481544f0 2024-03-19T01:55:01.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:01.135+0000 7f23457fa640 1 -- 172.21.15.55:0/4191970199 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f233c0566f0 con 0x7f23481544f0 2024-03-19T01:55:01.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:01.463+0000 7f234ece7640 1 -- 172.21.15.55:0/4191970199 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f23480729f0 con 0x7f23481544f0 2024-03-19T01:55:01.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:01.487+0000 7f23457fa640 1 -- 172.21.15.55:0/4191970199 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7f233c05a790 con 0x7f23481544f0 2024-03-19T01:55:01.484 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:01.484 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:55:01.484 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:55:01.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:01.487+0000 7f234ece7640 1 -- 172.21.15.55:0/4191970199 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f23340676e0 msgr2=0x7f2334069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:01.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:01.487+0000 7f234ece7640 1 --2- 172.21.15.55:0/4191970199 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f23340676e0 0x7f2334069ba0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2338000f70 tx=0x7f2338007e90 comp rx=0 tx=0).stop 2024-03-19T01:55:01.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:01.487+0000 7f234ece7640 1 -- 172.21.15.55:0/4191970199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23481544f0 msgr2=0x7f2348159e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:01.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:01.487+0000 7f234ece7640 1 --2- 172.21.15.55:0/4191970199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23481544f0 0x7f2348159e00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f233c00e930 tx=0x7f233c00edf0 comp rx=0 tx=0).stop 2024-03-19T01:55:01.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:01.491+0000 7f234ece7640 1 -- 172.21.15.55:0/4191970199 shutdown_connections 2024-03-19T01:55:01.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:01.491+0000 7f234ece7640 1 --2- 172.21.15.55:0/4191970199 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f23340676e0 0x7f2334069ba0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:01.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:01.491+0000 7f234ece7640 1 --2- 172.21.15.55:0/4191970199 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f234815acb0 0x7f2348152ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:01.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:01.491+0000 7f234ece7640 1 --2- 172.21.15.55:0/4191970199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23481544f0 0x7f2348159e00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:01.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:01.491+0000 7f234ece7640 1 --2- 172.21.15.55:0/4191970199 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f234806b2a0 0x7f23480c5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:01.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:01.491+0000 7f234ece7640 1 -- 172.21.15.55:0/4191970199 >> 172.21.15.55:0/4191970199 conn(0x7f234806e670 msgr2=0x7f23480c96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:01.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:01.491+0000 7f234ece7640 1 -- 172.21.15.55:0/4191970199 shutdown_connections 2024-03-19T01:55:01.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:01.491+0000 7f234ece7640 1 -- 172.21.15.55:0/4191970199 wait complete. 2024-03-19T01:55:02.106 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:55:02.106 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:03.108 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:03.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.223+0000 7fe707e80640 1 Processor -- start 2024-03-19T01:55:03.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.223+0000 7fe707e80640 1 -- start start 2024-03-19T01:55:03.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.223+0000 7fe707e80640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fe700152120 0x7fe7000c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:03.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.223+0000 7fe707e80640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7000c9610 0x7fe700157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:03.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.223+0000 7fe707e80640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe700070780 0x7fe700158510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:03.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.223+0000 7fe707e80640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fe700152500 con 0x7fe700070780 2024-03-19T01:55:03.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.223+0000 7fe707e80640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fe700070590 con 0x7fe700152120 2024-03-19T01:55:03.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.223+0000 7fe707e80640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fe7000c3530 con 0x7fe7000c9610 2024-03-19T01:55:03.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.227+0000 7fe70641d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe700070780 0x7fe700158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:03.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.227+0000 7fe70541b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7000c9610 0x7fe700157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:03.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.227+0000 7fe70641d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe700070780 0x7fe700158510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52986/0 (socket says 172.21.15.55:52986) 2024-03-19T01:55:03.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.227+0000 7fe70641d640 1 -- 172.21.15.55:0/1579388143 learned_addr learned my addr 172.21.15.55:0/1579388143 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:03.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.227+0000 7fe705c1c640 1 --2- 172.21.15.55:0/1579388143 >> v2:172.21.15.84:3300/0 conn(0x7fe700152120 0x7fe7000c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:03.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.227+0000 7fe70641d640 1 -- 172.21.15.55:0/1579388143 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7000c9610 msgr2=0x7fe700157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:03.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.227+0000 7fe70641d640 1 --2- 172.21.15.55:0/1579388143 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7000c9610 0x7fe700157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:03.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.227+0000 7fe70641d640 1 -- 172.21.15.55:0/1579388143 >> v2:172.21.15.84:3300/0 conn(0x7fe700152120 msgr2=0x7fe7000c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:03.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.227+0000 7fe70641d640 1 --2- 172.21.15.55:0/1579388143 >> v2:172.21.15.84:3300/0 conn(0x7fe700152120 0x7fe7000c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:03.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.227+0000 7fe70641d640 1 -- 172.21.15.55:0/1579388143 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe70006fff0 con 0x7fe700070780 2024-03-19T01:55:03.230 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.231+0000 7fe70641d640 1 --2- 172.21.15.55:0/1579388143 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe700070780 0x7fe700158510 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe700072430 tx=0x7fe6f801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e55f169bf616769 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:03.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.235+0000 7fe704c1a640 1 -- 172.21.15.55:0/1579388143 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f80a5670 con 0x7fe700070780 2024-03-19T01:55:03.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.239+0000 7fe704c1a640 1 -- 172.21.15.55:0/1579388143 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6f80a5810 con 0x7fe700070780 2024-03-19T01:55:03.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.239+0000 7fe704c1a640 1 -- 172.21.15.55:0/1579388143 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f80a5be0 con 0x7fe700070780 2024-03-19T01:55:03.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.239+0000 7fe707e80640 1 -- 172.21.15.55:0/1579388143 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe700070780 msgr2=0x7fe700158510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:03.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.239+0000 7fe707e80640 1 --2- 172.21.15.55:0/1579388143 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe700070780 0x7fe700158510 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe700072430 tx=0x7fe6f801c9f0 comp rx=0 tx=0).stop 2024-03-19T01:55:03.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.239+0000 7fe707e80640 1 -- 172.21.15.55:0/1579388143 shutdown_connections 2024-03-19T01:55:03.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.239+0000 7fe707e80640 1 --2- 172.21.15.55:0/1579388143 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7000c9610 0x7fe700157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:03.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.239+0000 7fe707e80640 1 --2- 172.21.15.55:0/1579388143 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe700070780 0x7fe700158510 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:03.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.239+0000 7fe707e80640 1 --2- 172.21.15.55:0/1579388143 >> v2:172.21.15.84:3300/0 conn(0x7fe700152120 0x7fe7000c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:03.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.239+0000 7fe707e80640 1 -- 172.21.15.55:0/1579388143 >> 172.21.15.55:0/1579388143 conn(0x7fe700067600 msgr2=0x7fe700071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:03.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.239+0000 7fe707e80640 1 -- 172.21.15.55:0/1579388143 shutdown_connections 2024-03-19T01:55:03.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.239+0000 7fe707e80640 1 -- 172.21.15.55:0/1579388143 wait complete. 2024-03-19T01:55:03.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.243+0000 7fe707e80640 1 Processor -- start 2024-03-19T01:55:03.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.243+0000 7fe707e80640 1 -- start start 2024-03-19T01:55:03.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.243+0000 7fe707e80640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe700070780 0x7fe700160d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:03.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.243+0000 7fe707e80640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7000c9610 0x7fe700161260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:03.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.243+0000 7fe707e80640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe700152120 0x7fe7000c6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:03.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.243+0000 7fe707e80640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe7000c31c0 con 0x7fe700070780 2024-03-19T01:55:03.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.243+0000 7fe707e80640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fe7000c2ee0 con 0x7fe7000c9610 2024-03-19T01:55:03.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.243+0000 7fe707e80640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fe7000c86d0 con 0x7fe700152120 2024-03-19T01:55:03.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.243+0000 7fe705c1c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe700070780 0x7fe700160d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:03.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.243+0000 7fe70641d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe700152120 0x7fe7000c6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:03.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.243+0000 7fe705c1c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe700070780 0x7fe700160d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52998/0 (socket says 172.21.15.55:52998) 2024-03-19T01:55:03.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.243+0000 7fe705c1c640 1 -- 172.21.15.55:0/705341820 learned_addr learned my addr 172.21.15.55:0/705341820 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:03.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.243+0000 7fe70541b640 1 --2- 172.21.15.55:0/705341820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7000c9610 0x7fe700161260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:03.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.247+0000 7fe705c1c640 1 -- 172.21.15.55:0/705341820 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe700152120 msgr2=0x7fe7000c6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:03.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.247+0000 7fe705c1c640 1 --2- 172.21.15.55:0/705341820 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe700152120 0x7fe7000c6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:03.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.247+0000 7fe705c1c640 1 -- 172.21.15.55:0/705341820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7000c9610 msgr2=0x7fe700161260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:03.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.247+0000 7fe705c1c640 1 --2- 172.21.15.55:0/705341820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7000c9610 0x7fe700161260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:03.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.247+0000 7fe705c1c640 1 -- 172.21.15.55:0/705341820 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe700145aa0 con 0x7fe700070780 2024-03-19T01:55:03.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.251+0000 7fe705c1c640 1 --2- 172.21.15.55:0/705341820 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe700070780 0x7fe700160d20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00e870 tx=0x7fe6fc00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:03.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.255+0000 7fe6f67fc640 1 -- 172.21.15.55:0/705341820 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc017070 con 0x7fe700070780 2024-03-19T01:55:03.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.255+0000 7fe707e80640 1 -- 172.21.15.55:0/705341820 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7001457c0 con 0x7fe700070780 2024-03-19T01:55:03.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.255+0000 7fe707e80640 1 -- 172.21.15.55:0/705341820 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe70006dce0 con 0x7fe700070780 2024-03-19T01:55:03.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.255+0000 7fe707e80640 1 -- 172.21.15.55:0/705341820 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe70006e0c0 con 0x7fe700070780 2024-03-19T01:55:03.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.255+0000 7fe6f67fc640 1 -- 172.21.15.55:0/705341820 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6fc0059a0 con 0x7fe700070780 2024-03-19T01:55:03.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.255+0000 7fe6f67fc640 1 -- 172.21.15.55:0/705341820 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc021060 con 0x7fe700070780 2024-03-19T01:55:03.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.551+0000 7fe6f67fc640 1 -- 172.21.15.55:0/705341820 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fe6fc01a070 con 0x7fe700070780 2024-03-19T01:55:03.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.551+0000 7fe6f67fc640 1 --2- 172.21.15.55:0/705341820 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe6e4067660 0x7fe6e4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:03.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.551+0000 7fe70541b640 1 --2- 172.21.15.55:0/705341820 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe6e4067660 0x7fe6e4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:03.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.551+0000 7fe70541b640 1 --2- 172.21.15.55:0/705341820 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe6e4067660 0x7fe6e4069b20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe7000c8fe0 tx=0x7fe6f0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:03.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.595+0000 7fe6f67fc640 1 -- 172.21.15.55:0/705341820 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fe6fc054540 con 0x7fe700070780 2024-03-19T01:55:03.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.595+0000 7fe6f67fc640 1 -- 172.21.15.55:0/705341820 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fe70006e0c0 con 0x7fe700070780 2024-03-19T01:55:03.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.815+0000 7fe707e80640 1 -- 172.21.15.55:0/705341820 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe700068a40 con 0x7fe700070780 2024-03-19T01:55:03.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.835+0000 7fe6f67fc640 1 -- 172.21.15.55:0/705341820 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7fe6fc0564a0 con 0x7fe700070780 2024-03-19T01:55:03.835 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:03.835 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:55:03.835 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:55:03.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.839+0000 7fe707e80640 1 -- 172.21.15.55:0/705341820 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe6e4067660 msgr2=0x7fe6e4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:03.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.839+0000 7fe707e80640 1 --2- 172.21.15.55:0/705341820 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe6e4067660 0x7fe6e4069b20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe7000c8fe0 tx=0x7fe6f0009190 comp rx=0 tx=0).stop 2024-03-19T01:55:03.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.839+0000 7fe707e80640 1 -- 172.21.15.55:0/705341820 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe700070780 msgr2=0x7fe700160d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:03.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.839+0000 7fe707e80640 1 --2- 172.21.15.55:0/705341820 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe700070780 0x7fe700160d20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00e870 tx=0x7fe6fc00ec70 comp rx=0 tx=0).stop 2024-03-19T01:55:03.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.839+0000 7fe707e80640 1 -- 172.21.15.55:0/705341820 shutdown_connections 2024-03-19T01:55:03.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.839+0000 7fe707e80640 1 --2- 172.21.15.55:0/705341820 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe6e4067660 0x7fe6e4069b20 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:03.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.839+0000 7fe707e80640 1 --2- 172.21.15.55:0/705341820 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe700152120 0x7fe7000c6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:03.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.839+0000 7fe707e80640 1 --2- 172.21.15.55:0/705341820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7000c9610 0x7fe700161260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:03.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.839+0000 7fe707e80640 1 --2- 172.21.15.55:0/705341820 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe700070780 0x7fe700160d20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:03.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.839+0000 7fe707e80640 1 -- 172.21.15.55:0/705341820 >> 172.21.15.55:0/705341820 conn(0x7fe700067600 msgr2=0x7fe70014c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:03.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.839+0000 7fe707e80640 1 -- 172.21.15.55:0/705341820 shutdown_connections 2024-03-19T01:55:03.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:03.839+0000 7fe707e80640 1 -- 172.21.15.55:0/705341820 wait complete. 2024-03-19T01:55:04.457 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:55:04.457 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:05.458 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:05.586 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.587+0000 7fc640b2a640 1 Processor -- start 2024-03-19T01:55:05.586 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.587+0000 7fc640b2a640 1 -- start start 2024-03-19T01:55:05.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.591+0000 7fc640b2a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc63c0cd090 0x7fc63c0732b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:05.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.591+0000 7fc640b2a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fc63c0cd470 0x7fc63c14e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:05.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.591+0000 7fc640b2a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc63c0c8010 0x7fc63c14ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:05.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.591+0000 7fc640b2a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fc63c158e90 con 0x7fc63c0cd090 2024-03-19T01:55:05.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.591+0000 7fc640b2a640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fc63c158330 con 0x7fc63c0cd470 2024-03-19T01:55:05.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.591+0000 7fc640b2a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc63c1585f0 con 0x7fc63c0c8010 2024-03-19T01:55:05.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.591+0000 7fc63a59c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc63c0cd090 0x7fc63c0732b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:05.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.591+0000 7fc63ad9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc63c0c8010 0x7fc63c14ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:05.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.591+0000 7fc63a59c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc63c0cd090 0x7fc63c0732b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53014/0 (socket says 172.21.15.55:53014) 2024-03-19T01:55:05.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.591+0000 7fc63a59c640 1 -- 172.21.15.55:0/2209071634 learned_addr learned my addr 172.21.15.55:0/2209071634 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:05.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.591+0000 7fc639d9b640 1 --2- 172.21.15.55:0/2209071634 >> v2:172.21.15.84:3300/0 conn(0x7fc63c0cd470 0x7fc63c14e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:05.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.595+0000 7fc63a59c640 1 -- 172.21.15.55:0/2209071634 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc63c0c8010 msgr2=0x7fc63c14ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:05.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.595+0000 7fc63a59c640 1 --2- 172.21.15.55:0/2209071634 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc63c0c8010 0x7fc63c14ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:05.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.595+0000 7fc63a59c640 1 -- 172.21.15.55:0/2209071634 >> v2:172.21.15.84:3300/0 conn(0x7fc63c0cd470 msgr2=0x7fc63c14e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:05.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.595+0000 7fc63a59c640 1 --2- 172.21.15.55:0/2209071634 >> v2:172.21.15.84:3300/0 conn(0x7fc63c0cd470 0x7fc63c14e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:05.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.595+0000 7fc63a59c640 1 -- 172.21.15.55:0/2209071634 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63c1588d0 con 0x7fc63c0cd090 2024-03-19T01:55:05.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.599+0000 7fc63a59c640 1 --2- 172.21.15.55:0/2209071634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc63c0cd090 0x7fc63c0732b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc63c0ca790 tx=0x7fc62c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffda1aeb10d62327 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:05.601 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.603+0000 7fc63959a640 1 -- 172.21.15.55:0/2209071634 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c0b1070 con 0x7fc63c0cd090 2024-03-19T01:55:05.601 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.603+0000 7fc63959a640 1 -- 172.21.15.55:0/2209071634 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc62c0ac630 con 0x7fc63c0cd090 2024-03-19T01:55:05.601 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.603+0000 7fc63959a640 1 -- 172.21.15.55:0/2209071634 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c0a8050 con 0x7fc63c0cd090 2024-03-19T01:55:05.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.603+0000 7fc640b2a640 1 -- 172.21.15.55:0/2209071634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc63c0cd090 msgr2=0x7fc63c0732b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:05.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.603+0000 7fc640b2a640 1 --2- 172.21.15.55:0/2209071634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc63c0cd090 0x7fc63c0732b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc63c0ca790 tx=0x7fc62c01c520 comp rx=0 tx=0).stop 2024-03-19T01:55:05.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.603+0000 7fc640b2a640 1 -- 172.21.15.55:0/2209071634 shutdown_connections 2024-03-19T01:55:05.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.603+0000 7fc640b2a640 1 --2- 172.21.15.55:0/2209071634 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc63c0c8010 0x7fc63c14ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:05.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.603+0000 7fc640b2a640 1 --2- 172.21.15.55:0/2209071634 >> v2:172.21.15.84:3300/0 conn(0x7fc63c0cd470 0x7fc63c14e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:05.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.603+0000 7fc640b2a640 1 --2- 172.21.15.55:0/2209071634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc63c0cd090 0x7fc63c0732b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:05.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.603+0000 7fc640b2a640 1 -- 172.21.15.55:0/2209071634 >> 172.21.15.55:0/2209071634 conn(0x7fc63c06e670 msgr2=0x7fc63c06ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:05.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.607+0000 7fc640b2a640 1 -- 172.21.15.55:0/2209071634 shutdown_connections 2024-03-19T01:55:05.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.607+0000 7fc640b2a640 1 -- 172.21.15.55:0/2209071634 wait complete. 2024-03-19T01:55:05.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.607+0000 7fc640b2a640 1 Processor -- start 2024-03-19T01:55:05.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.607+0000 7fc640b2a640 1 -- start start 2024-03-19T01:55:05.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.607+0000 7fc640b2a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc63c0c8010 0x7fc63c140900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:05.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.607+0000 7fc640b2a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc63c0cd090 0x7fc63c0c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:05.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.607+0000 7fc640b2a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc63c0cd470 0x7fc63c0c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:05.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.607+0000 7fc640b2a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c0737f0 con 0x7fc63c0cd090 2024-03-19T01:55:05.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.607+0000 7fc640b2a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c15b570 con 0x7fc63c0c8010 2024-03-19T01:55:05.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.607+0000 7fc640b2a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc63c0c64a0 con 0x7fc63c0cd470 2024-03-19T01:55:05.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.607+0000 7fc639d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc63c0cd090 0x7fc63c0c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:05.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.611+0000 7fc63ad9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc63c0cd470 0x7fc63c0c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:05.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.611+0000 7fc63a59c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc63c0c8010 0x7fc63c140900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:05.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.611+0000 7fc639d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc63c0cd090 0x7fc63c0c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53030/0 (socket says 172.21.15.55:53030) 2024-03-19T01:55:05.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.611+0000 7fc639d9b640 1 -- 172.21.15.55:0/3207615910 learned_addr learned my addr 172.21.15.55:0/3207615910 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:05.612 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.615+0000 7fc639d9b640 1 -- 172.21.15.55:0/3207615910 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc63c0cd470 msgr2=0x7fc63c0c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:05.612 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.615+0000 7fc639d9b640 1 --2- 172.21.15.55:0/3207615910 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc63c0cd470 0x7fc63c0c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:05.612 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.615+0000 7fc639d9b640 1 -- 172.21.15.55:0/3207615910 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc63c0c8010 msgr2=0x7fc63c140900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:05.612 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.615+0000 7fc639d9b640 1 --2- 172.21.15.55:0/3207615910 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc63c0c8010 0x7fc63c140900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:05.612 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.615+0000 7fc639d9b640 1 -- 172.21.15.55:0/3207615910 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63c0cb410 con 0x7fc63c0cd090 2024-03-19T01:55:05.615 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.619+0000 7fc639d9b640 1 --2- 172.21.15.55:0/3207615910 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc63c0cd090 0x7fc63c0c8c10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc63000e870 tx=0x7fc63000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:05.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.623+0000 7fc62affd640 1 -- 172.21.15.55:0/3207615910 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc630017070 con 0x7fc63c0cd090 2024-03-19T01:55:05.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.623+0000 7fc62affd640 1 -- 172.21.15.55:0/3207615910 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc63000c8e0 con 0x7fc63c0cd090 2024-03-19T01:55:05.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.623+0000 7fc62affd640 1 -- 172.21.15.55:0/3207615910 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63000cc90 con 0x7fc63c0cd090 2024-03-19T01:55:05.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.623+0000 7fc640b2a640 1 -- 172.21.15.55:0/3207615910 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc63c152910 con 0x7fc63c0cd090 2024-03-19T01:55:05.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.623+0000 7fc640b2a640 1 -- 172.21.15.55:0/3207615910 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc63c152bb0 con 0x7fc63c0cd090 2024-03-19T01:55:05.621 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.623+0000 7fc640b2a640 1 -- 172.21.15.55:0/3207615910 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc63c068a40 con 0x7fc63c0cd090 2024-03-19T01:55:05.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.903+0000 7fc62affd640 1 -- 172.21.15.55:0/3207615910 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc63001a070 con 0x7fc63c0cd090 2024-03-19T01:55:05.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.903+0000 7fc62affd640 1 --2- 172.21.15.55:0/3207615910 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc61c0676b0 0x7fc61c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:05.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.903+0000 7fc63a59c640 1 --2- 172.21.15.55:0/3207615910 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc61c0676b0 0x7fc61c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:05.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.907+0000 7fc63a59c640 1 --2- 172.21.15.55:0/3207615910 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc61c0676b0 0x7fc61c069b70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc62c0ae000 tx=0x7fc62c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:05.927 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.931+0000 7fc62affd640 1 -- 172.21.15.55:0/3207615910 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fc6300545f0 con 0x7fc63c0cd090 2024-03-19T01:55:05.927 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:05.931+0000 7fc62affd640 1 -- 172.21.15.55:0/3207615910 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc630056550 con 0x7fc63c0cd090 2024-03-19T01:55:06.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:06.147+0000 7fc640b2a640 1 -- 172.21.15.55:0/3207615910 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc63c074da0 con 0x7fc63c0cd090 2024-03-19T01:55:06.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:06.175+0000 7fc62affd640 1 -- 172.21.15.55:0/3207615910 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7fc63005a5f0 con 0x7fc63c0cd090 2024-03-19T01:55:06.171 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:06.171 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:55:06.171 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:55:06.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:06.175+0000 7fc640b2a640 1 -- 172.21.15.55:0/3207615910 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc61c0676b0 msgr2=0x7fc61c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:06.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:06.175+0000 7fc640b2a640 1 --2- 172.21.15.55:0/3207615910 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc61c0676b0 0x7fc61c069b70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc62c0ae000 tx=0x7fc62c0a7000 comp rx=0 tx=0).stop 2024-03-19T01:55:06.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:06.175+0000 7fc640b2a640 1 -- 172.21.15.55:0/3207615910 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc63c0cd090 msgr2=0x7fc63c0c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:06.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:06.175+0000 7fc640b2a640 1 --2- 172.21.15.55:0/3207615910 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc63c0cd090 0x7fc63c0c8c10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc63000e870 tx=0x7fc63000ec70 comp rx=0 tx=0).stop 2024-03-19T01:55:06.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:06.175+0000 7fc640b2a640 1 -- 172.21.15.55:0/3207615910 shutdown_connections 2024-03-19T01:55:06.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:06.175+0000 7fc640b2a640 1 --2- 172.21.15.55:0/3207615910 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc61c0676b0 0x7fc61c069b70 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:06.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:06.175+0000 7fc640b2a640 1 --2- 172.21.15.55:0/3207615910 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc63c0cd470 0x7fc63c0c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:06.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:06.175+0000 7fc640b2a640 1 --2- 172.21.15.55:0/3207615910 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc63c0cd090 0x7fc63c0c8c10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:06.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:06.175+0000 7fc640b2a640 1 --2- 172.21.15.55:0/3207615910 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc63c0c8010 0x7fc63c140900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:06.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:06.175+0000 7fc640b2a640 1 -- 172.21.15.55:0/3207615910 >> 172.21.15.55:0/3207615910 conn(0x7fc63c06e670 msgr2=0x7fc63c06ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:06.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:06.175+0000 7fc640b2a640 1 -- 172.21.15.55:0/3207615910 shutdown_connections 2024-03-19T01:55:06.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:06.175+0000 7fc640b2a640 1 -- 172.21.15.55:0/3207615910 wait complete. 2024-03-19T01:55:06.793 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:55:06.793 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:07.794 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:07.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.911+0000 7f6d6923d640 1 Processor -- start 2024-03-19T01:55:07.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.911+0000 7f6d6923d640 1 -- start start 2024-03-19T01:55:07.910 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.911+0000 7f6d6923d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6d641544f0 0x7f6d64073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:07.910 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.911+0000 7f6d6923d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d640c7fa0 con 0x7f6d6415acb0 2024-03-19T01:55:07.910 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.911+0000 7f6d6923d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d640c8240 con 0x7f6d6406b2a0 2024-03-19T01:55:07.910 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.911+0000 7f6d6923d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f6d6415b210 con 0x7f6d641544f0 2024-03-19T01:55:07.910 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.915+0000 7f6d6359e640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f6d6415acb0 0x7f6d640cce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:56178/0 (socket says 172.21.15.55:56178) 2024-03-19T01:55:07.911 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.915+0000 7f6d6359e640 1 -- 172.21.15.55:0/3674757143 learned_addr learned my addr 172.21.15.55:0/3674757143 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:55:07.911 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.915+0000 7f6d62d9d640 1 --2- 172.21.15.55:0/3674757143 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6d641544f0 0x7f6d64073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:07.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.919+0000 7f6d62d9d640 1 -- 172.21.15.55:0/3674757143 >> v1:172.21.15.84:6789/0 conn(0x7f6d6406b2a0 legacy=0x7f6d6415e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:07.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.919+0000 7f6d62d9d640 1 -- 172.21.15.55:0/3674757143 >> v1:172.21.15.55:6789/0 conn(0x7f6d6415acb0 legacy=0x7f6d640cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:07.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.919+0000 7f6d61d9b640 1 -- 172.21.15.55:0/3674757143 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3811776872 0 0) 0x7f6d640c7fa0 con 0x7f6d6415acb0 2024-03-19T01:55:07.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.919+0000 7f6d62d9d640 1 -- 172.21.15.55:0/3674757143 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d6415d920 con 0x7f6d641544f0 2024-03-19T01:55:07.919 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.923+0000 7f6d62d9d640 1 --2- 172.21.15.55:0/3674757143 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6d641544f0 0x7f6d64073540 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6d540097f0 tx=0x7f6d5401ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=266b0293dd40e3d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:07.925 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.927+0000 7f6d61d9b640 1 -- 172.21.15.55:0/3674757143 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d540b5070 con 0x7f6d641544f0 2024-03-19T01:55:07.925 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.927+0000 7f6d61d9b640 1 -- 172.21.15.55:0/3674757143 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d540a89d0 con 0x7f6d641544f0 2024-03-19T01:55:07.926 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.927+0000 7f6d61d9b640 1 -- 172.21.15.55:0/3674757143 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d540a8da0 con 0x7f6d641544f0 2024-03-19T01:55:07.926 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.927+0000 7f6d6923d640 1 -- 172.21.15.55:0/3674757143 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6d641544f0 msgr2=0x7f6d64073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:07.927 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.927+0000 7f6d6923d640 1 --2- 172.21.15.55:0/3674757143 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6d641544f0 0x7f6d64073540 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6d540097f0 tx=0x7f6d5401ecd0 comp rx=0 tx=0).stop 2024-03-19T01:55:07.927 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.927+0000 7f6d6923d640 1 -- 172.21.15.55:0/3674757143 shutdown_connections 2024-03-19T01:55:07.927 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.927+0000 7f6d6923d640 1 --2- 172.21.15.55:0/3674757143 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6d641544f0 0x7f6d64073540 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:07.927 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.927+0000 7f6d6923d640 1 -- 172.21.15.55:0/3674757143 >> 172.21.15.55:0/3674757143 conn(0x7f6d6406e670 msgr2=0x7f6d6406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:07.927 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.927+0000 7f6d6923d640 1 -- 172.21.15.55:0/3674757143 shutdown_connections 2024-03-19T01:55:07.927 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.927+0000 7f6d6923d640 1 -- 172.21.15.55:0/3674757143 wait complete. 2024-03-19T01:55:07.928 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.931+0000 7f6d6923d640 1 Processor -- start 2024-03-19T01:55:07.928 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.931+0000 7f6d6923d640 1 -- start start 2024-03-19T01:55:07.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.931+0000 7f6d6923d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6d6406b2a0 0x7f6d64156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:07.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.931+0000 7f6d6923d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d641544f0 0x7f6d641f0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:07.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.931+0000 7f6d6923d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d6415acb0 0x7f6d641f3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:07.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.931+0000 7f6d6923d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6d640cd540 con 0x7f6d641544f0 2024-03-19T01:55:07.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.931+0000 7f6d6923d640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f6d641548d0 con 0x7f6d6415acb0 2024-03-19T01:55:07.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.931+0000 7f6d6923d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f6d640cd6c0 con 0x7f6d6406b2a0 2024-03-19T01:55:07.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.931+0000 7f6d6259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d641544f0 0x7f6d641f0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:07.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.931+0000 7f6d62d9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6d6406b2a0 0x7f6d64156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:07.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.931+0000 7f6d62d9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6d6406b2a0 0x7f6d64156030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:52114/0 (socket says 172.21.15.55:52114) 2024-03-19T01:55:07.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.931+0000 7f6d62d9d640 1 -- 172.21.15.55:0/3631255174 learned_addr learned my addr 172.21.15.55:0/3631255174 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:07.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.931+0000 7f6d6359e640 1 --2- 172.21.15.55:0/3631255174 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d6415acb0 0x7f6d641f3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:07.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.939+0000 7f6d6259c640 1 -- 172.21.15.55:0/3631255174 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6d6406b2a0 msgr2=0x7f6d64156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:07.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.939+0000 7f6d6259c640 1 --2- 172.21.15.55:0/3631255174 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6d6406b2a0 0x7f6d64156030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:07.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.939+0000 7f6d6259c640 1 -- 172.21.15.55:0/3631255174 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d6415acb0 msgr2=0x7f6d641f3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:07.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.939+0000 7f6d6259c640 1 --2- 172.21.15.55:0/3631255174 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d6415acb0 0x7f6d641f3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:07.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.939+0000 7f6d6259c640 1 -- 172.21.15.55:0/3631255174 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d641f3960 con 0x7f6d641544f0 2024-03-19T01:55:07.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.939+0000 7f6d6259c640 1 --2- 172.21.15.55:0/3631255174 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d641544f0 0x7f6d641f0a00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6d6406b680 tx=0x7f6d5800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:07.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.947+0000 7f6d4b7fe640 1 -- 172.21.15.55:0/3631255174 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d58017070 con 0x7f6d641544f0 2024-03-19T01:55:07.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.947+0000 7f6d4b7fe640 1 -- 172.21.15.55:0/3631255174 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d580052d0 con 0x7f6d641544f0 2024-03-19T01:55:07.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.947+0000 7f6d4b7fe640 1 -- 172.21.15.55:0/3631255174 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d58003840 con 0x7f6d641544f0 2024-03-19T01:55:07.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.947+0000 7f6d6923d640 1 -- 172.21.15.55:0/3631255174 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d641f3c20 con 0x7f6d641544f0 2024-03-19T01:55:07.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.947+0000 7f6d6923d640 1 -- 172.21.15.55:0/3631255174 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d641f41e0 con 0x7f6d641544f0 2024-03-19T01:55:07.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:07.947+0000 7f6d6923d640 1 -- 172.21.15.55:0/3631255174 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d64068a40 con 0x7f6d641544f0 2024-03-19T01:55:08.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:08.251+0000 7f6d4b7fe640 1 -- 172.21.15.55:0/3631255174 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f6d5801a070 con 0x7f6d641544f0 2024-03-19T01:55:08.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:08.251+0000 7f6d4b7fe640 1 --2- 172.21.15.55:0/3631255174 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6d500676e0 0x7f6d50069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:08.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:08.251+0000 7f6d62d9d640 1 --2- 172.21.15.55:0/3631255174 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6d500676e0 0x7f6d50069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:08.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:08.251+0000 7f6d62d9d640 1 --2- 172.21.15.55:0/3631255174 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6d500676e0 0x7f6d50069ba0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6d5401c900 tx=0x7f6d54008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:08.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:08.287+0000 7f6d4b7fe640 1 -- 172.21.15.55:0/3631255174 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f6d58053680 con 0x7f6d641544f0 2024-03-19T01:55:08.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:08.291+0000 7f6d4b7fe640 1 -- 172.21.15.55:0/3631255174 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f6d58021020 con 0x7f6d641544f0 2024-03-19T01:55:08.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:08.495+0000 7f6d6923d640 1 -- 172.21.15.55:0/3631255174 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d640c3ab0 con 0x7f6d641544f0 2024-03-19T01:55:08.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:08.535+0000 7f6d4b7fe640 1 -- 172.21.15.55:0/3631255174 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7f6d58005480 con 0x7f6d641544f0 2024-03-19T01:55:08.533 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:08.533 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:55:08.533 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:55:08.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:08.539+0000 7f6d6923d640 1 -- 172.21.15.55:0/3631255174 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6d500676e0 msgr2=0x7f6d50069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:08.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:08.539+0000 7f6d6923d640 1 --2- 172.21.15.55:0/3631255174 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6d500676e0 0x7f6d50069ba0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6d5401c900 tx=0x7f6d54008040 comp rx=0 tx=0).stop 2024-03-19T01:55:08.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:08.539+0000 7f6d6923d640 1 -- 172.21.15.55:0/3631255174 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d641544f0 msgr2=0x7f6d641f0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:08.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:08.539+0000 7f6d6923d640 1 --2- 172.21.15.55:0/3631255174 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d641544f0 0x7f6d641f0a00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6d6406b680 tx=0x7f6d5800c890 comp rx=0 tx=0).stop 2024-03-19T01:55:08.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:08.539+0000 7f6d6923d640 1 -- 172.21.15.55:0/3631255174 shutdown_connections 2024-03-19T01:55:08.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:08.539+0000 7f6d6923d640 1 --2- 172.21.15.55:0/3631255174 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6d500676e0 0x7f6d50069ba0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:08.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:08.539+0000 7f6d6923d640 1 --2- 172.21.15.55:0/3631255174 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d6415acb0 0x7f6d641f3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:08.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:08.539+0000 7f6d6923d640 1 --2- 172.21.15.55:0/3631255174 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d641544f0 0x7f6d641f0a00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:08.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:08.539+0000 7f6d6923d640 1 --2- 172.21.15.55:0/3631255174 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6d6406b2a0 0x7f6d64156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:08.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:08.539+0000 7f6d6923d640 1 -- 172.21.15.55:0/3631255174 >> 172.21.15.55:0/3631255174 conn(0x7f6d6406e670 msgr2=0x7f6d640733d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:08.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:08.539+0000 7f6d6923d640 1 -- 172.21.15.55:0/3631255174 shutdown_connections 2024-03-19T01:55:08.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:08.539+0000 7f6d6923d640 1 -- 172.21.15.55:0/3631255174 wait complete. 2024-03-19T01:55:08.974 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:55:08.976 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:55:09.163 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:55:09.163 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:10.164 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:10.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.295+0000 7f3897e82640 1 Processor -- start 2024-03-19T01:55:10.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.295+0000 7f3897e82640 1 -- start start 2024-03-19T01:55:10.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.299+0000 7f3897e82640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f389013dd00 0x7f38900c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:10.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.299+0000 7f3897e82640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f389015b550 0x7f38900c5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:10.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.299+0000 7f3897e82640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38900c8f70 con 0x7f38900c9780 2024-03-19T01:55:10.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.299+0000 7f3897e82640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f38900c9250 con 0x7f389013dd00 2024-03-19T01:55:10.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.299+0000 7f3897e82640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f38901582b0 con 0x7f389015b550 2024-03-19T01:55:10.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.299+0000 7f389641f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f389015b550 0x7f38900c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:10.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.299+0000 7f3895c1e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f389013dd00 0x7f38900c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:10.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.299+0000 7f389541d640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f38900c9780 0x7f38900c8740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:49784/0 (socket says 172.21.15.55:49784) 2024-03-19T01:55:10.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.299+0000 7f389541d640 1 -- 172.21.15.55:0/2215751133 learned_addr learned my addr 172.21.15.55:0/2215751133 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:55:10.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.303+0000 7f3895c1e640 1 -- 172.21.15.55:0/2215751133 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f389015b550 msgr2=0x7f38900c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:10.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.303+0000 7f3895c1e640 1 --2- 172.21.15.55:0/2215751133 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f389015b550 0x7f38900c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:10.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.303+0000 7f3895c1e640 1 -- 172.21.15.55:0/2215751133 >> v1:172.21.15.84:6789/0 conn(0x7f38900c9780 legacy=0x7f38900c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:10.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.303+0000 7f3895c1e640 1 -- 172.21.15.55:0/2215751133 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3890158590 con 0x7f389013dd00 2024-03-19T01:55:10.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.307+0000 7f3895c1e640 1 --2- 172.21.15.55:0/2215751133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f389013dd00 0x7f38900c7430 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f388c0097c0 tx=0x7f388c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a25dff9e08fc7169 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:10.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.311+0000 7f3894c1c640 1 -- 172.21.15.55:0/2215751133 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f388c0b1070 con 0x7f389013dd00 2024-03-19T01:55:10.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.311+0000 7f3894c1c640 1 -- 172.21.15.55:0/2215751133 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f388c0ac410 con 0x7f389013dd00 2024-03-19T01:55:10.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.311+0000 7f3894c1c640 1 -- 172.21.15.55:0/2215751133 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f388c0ac7e0 con 0x7f389013dd00 2024-03-19T01:55:10.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.311+0000 7f3897e82640 1 -- 172.21.15.55:0/2215751133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f389013dd00 msgr2=0x7f38900c7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:10.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.311+0000 7f3897e82640 1 --2- 172.21.15.55:0/2215751133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f389013dd00 0x7f38900c7430 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f388c0097c0 tx=0x7f388c01c4a0 comp rx=0 tx=0).stop 2024-03-19T01:55:10.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.311+0000 7f3897e82640 1 -- 172.21.15.55:0/2215751133 shutdown_connections 2024-03-19T01:55:10.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.311+0000 7f3897e82640 1 --2- 172.21.15.55:0/2215751133 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f389015b550 0x7f38900c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:10.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.311+0000 7f3897e82640 1 --2- 172.21.15.55:0/2215751133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f389013dd00 0x7f38900c7430 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:10.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.311+0000 7f3897e82640 1 -- 172.21.15.55:0/2215751133 >> 172.21.15.55:0/2215751133 conn(0x7f389006e5a0 msgr2=0x7f3890072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:10.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.311+0000 7f3897e82640 1 -- 172.21.15.55:0/2215751133 shutdown_connections 2024-03-19T01:55:10.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.311+0000 7f3897e82640 1 -- 172.21.15.55:0/2215751133 wait complete. 2024-03-19T01:55:10.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.315+0000 7f3897e82640 1 Processor -- start 2024-03-19T01:55:10.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.315+0000 7f3897e82640 1 -- start start 2024-03-19T01:55:10.313 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.315+0000 7f3897e82640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38900c9780 0x7f3890159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:10.313 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.315+0000 7f3897e82640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f389013dd00 0x7f38900caa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:10.313 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.315+0000 7f3897e82640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f389015b550 0x7f38901f16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:10.313 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.315+0000 7f3897e82640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3890165280 con 0x7f38900c9780 2024-03-19T01:55:10.313 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.315+0000 7f3897e82640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3890165100 con 0x7f389015b550 2024-03-19T01:55:10.313 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.315+0000 7f3897e82640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3890165400 con 0x7f389013dd00 2024-03-19T01:55:10.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.315+0000 7f389641f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f389015b550 0x7f38901f16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:10.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.315+0000 7f389541d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f389013dd00 0x7f38900caa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:10.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.315+0000 7f389641f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f389015b550 0x7f38901f16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:60776/0 (socket says 172.21.15.55:60776) 2024-03-19T01:55:10.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.319+0000 7f389641f640 1 -- 172.21.15.55:0/2573591013 learned_addr learned my addr 172.21.15.55:0/2573591013 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:10.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.319+0000 7f3895c1e640 1 --2- 172.21.15.55:0/2573591013 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38900c9780 0x7f3890159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:10.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.323+0000 7f389641f640 1 -- 172.21.15.55:0/2573591013 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f389013dd00 msgr2=0x7f38900caa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:10.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.323+0000 7f389641f640 1 --2- 172.21.15.55:0/2573591013 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f389013dd00 0x7f38900caa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:10.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.323+0000 7f389641f640 1 -- 172.21.15.55:0/2573591013 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38900c9780 msgr2=0x7f3890159120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:10.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.323+0000 7f389641f640 1 --2- 172.21.15.55:0/2573591013 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38900c9780 0x7f3890159120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:10.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.323+0000 7f389641f640 1 -- 172.21.15.55:0/2573591013 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f389015ab90 con 0x7f389015b550 2024-03-19T01:55:10.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.327+0000 7f389641f640 1 --2- 172.21.15.55:0/2573591013 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f389015b550 0x7f38901f16f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f388800efc0 tx=0x7f388800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:10.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.331+0000 7f38867fc640 1 -- 172.21.15.55:0/2573591013 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3888018020 con 0x7f389015b550 2024-03-19T01:55:10.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.331+0000 7f3897e82640 1 -- 172.21.15.55:0/2573591013 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f389015ae70 con 0x7f389015b550 2024-03-19T01:55:10.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.331+0000 7f3897e82640 1 -- 172.21.15.55:0/2573591013 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38900cb2c0 con 0x7f389015b550 2024-03-19T01:55:10.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.331+0000 7f3897e82640 1 -- 172.21.15.55:0/2573591013 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3890068a40 con 0x7f389015b550 2024-03-19T01:55:10.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.335+0000 7f38867fc640 1 -- 172.21.15.55:0/2573591013 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3888014480 con 0x7f389015b550 2024-03-19T01:55:10.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.335+0000 7f38867fc640 1 -- 172.21.15.55:0/2573591013 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3888014830 con 0x7f389015b550 2024-03-19T01:55:10.369 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.371+0000 7f38867fc640 1 -- 172.21.15.55:0/2573591013 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f388801b020 con 0x7f389015b550 2024-03-19T01:55:10.369 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.371+0000 7f38867fc640 1 --2- 172.21.15.55:0/2573591013 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3874067690 0x7f3874069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:10.369 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.371+0000 7f3895c1e640 1 --2- 172.21.15.55:0/2573591013 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3874067690 0x7f3874069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:10.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.371+0000 7f3895c1e640 1 --2- 172.21.15.55:0/2573591013 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3874067690 0x7f3874069b50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f388c0ae000 tx=0x7f388c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:10.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.667+0000 7f38867fc640 1 -- 172.21.15.55:0/2573591013 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f3888055330 con 0x7f389015b550 2024-03-19T01:55:10.684 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.687+0000 7f38867fc640 1 -- 172.21.15.55:0/2573591013 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f3888022070 con 0x7f389015b550 2024-03-19T01:55:10.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:10.911+0000 7f3897e82640 1 -- 172.21.15.55:0/2573591013 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38901626c0 con 0x7f389015b550 2024-03-19T01:55:11.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:11.079+0000 7f38867fc640 1 -- 172.21.15.55:0/2573591013 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7f3888010040 con 0x7f389015b550 2024-03-19T01:55:11.076 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:11.076 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:55:11.076 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:55:11.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:11.079+0000 7f3897e82640 1 -- 172.21.15.55:0/2573591013 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3874067690 msgr2=0x7f3874069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:11.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:11.079+0000 7f3897e82640 1 --2- 172.21.15.55:0/2573591013 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3874067690 0x7f3874069b50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f388c0ae000 tx=0x7f388c008040 comp rx=0 tx=0).stop 2024-03-19T01:55:11.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:11.079+0000 7f3897e82640 1 -- 172.21.15.55:0/2573591013 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f389015b550 msgr2=0x7f38901f16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:11.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:11.079+0000 7f3897e82640 1 --2- 172.21.15.55:0/2573591013 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f389015b550 0x7f38901f16f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f388800efc0 tx=0x7f388800c490 comp rx=0 tx=0).stop 2024-03-19T01:55:11.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:11.079+0000 7f3897e82640 1 -- 172.21.15.55:0/2573591013 shutdown_connections 2024-03-19T01:55:11.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:11.079+0000 7f3897e82640 1 --2- 172.21.15.55:0/2573591013 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3874067690 0x7f3874069b50 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:11.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:11.079+0000 7f3897e82640 1 --2- 172.21.15.55:0/2573591013 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f389015b550 0x7f38901f16f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:11.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:11.079+0000 7f3897e82640 1 --2- 172.21.15.55:0/2573591013 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f389013dd00 0x7f38900caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:11.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:11.079+0000 7f3897e82640 1 --2- 172.21.15.55:0/2573591013 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38900c9780 0x7f3890159120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:11.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:11.079+0000 7f3897e82640 1 -- 172.21.15.55:0/2573591013 >> 172.21.15.55:0/2573591013 conn(0x7f389006e5a0 msgr2=0x7f389013bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:11.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:11.079+0000 7f3897e82640 1 -- 172.21.15.55:0/2573591013 shutdown_connections 2024-03-19T01:55:11.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:11.079+0000 7f3897e82640 1 -- 172.21.15.55:0/2573591013 wait complete. 2024-03-19T01:55:11.699 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:55:11.699 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:12.701 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:12.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.819+0000 7f8099bc7640 1 Processor -- start 2024-03-19T01:55:12.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.819+0000 7f8099bc7640 1 -- start start 2024-03-19T01:55:12.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.819+0000 7f8099bc7640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f809406c000 0x7f809414e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:12.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.819+0000 7f8099bc7640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8094158330 con 0x7f80940cd090 2024-03-19T01:55:12.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.819+0000 7f8099bc7640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80941585f0 con 0x7f80940cd470 2024-03-19T01:55:12.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.819+0000 7f8099bc7640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f8094158e90 con 0x7f809406c000 2024-03-19T01:55:12.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.819+0000 7f8093fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f809406c000 0x7f809414e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:12.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.819+0000 7f8092ffd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f80940cd470 0x7f809406b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:49790/0 (socket says 172.21.15.55:49790) 2024-03-19T01:55:12.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.819+0000 7f8092ffd640 1 -- 172.21.15.55:0/71594196 learned_addr learned my addr 172.21.15.55:0/71594196 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:55:12.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.823+0000 7f80927fc640 1 -- 172.21.15.55:0/71594196 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2328707515 0 0) 0x7f80941585f0 con 0x7f80940cd470 2024-03-19T01:55:12.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.823+0000 7f80927fc640 1 -- 172.21.15.55:0/71594196 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f807c003530 con 0x7f80940cd470 2024-03-19T01:55:12.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.827+0000 7f8093fff640 1 -- 172.21.15.55:0/71594196 >> v1:172.21.15.84:6789/0 conn(0x7f80940cd470 legacy=0x7f809406b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:12.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.827+0000 7f8093fff640 1 -- 172.21.15.55:0/71594196 >> v1:172.21.15.55:6789/0 conn(0x7f80940cd090 legacy=0x7f80940732b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:12.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.827+0000 7f8093fff640 1 -- 172.21.15.55:0/71594196 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80941588d0 con 0x7f809406c000 2024-03-19T01:55:12.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.831+0000 7f8093fff640 1 --2- 172.21.15.55:0/71594196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f809406c000 0x7f809414e4e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f808c009d90 tx=0x7f808c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c61d8bbc633464c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:12.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.839+0000 7f80927fc640 1 -- 172.21.15.55:0/71594196 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c0b2070 con 0x7f809406c000 2024-03-19T01:55:12.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.839+0000 7f80927fc640 1 -- 172.21.15.55:0/71594196 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f808c0a5830 con 0x7f809406c000 2024-03-19T01:55:12.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.839+0000 7f80927fc640 1 -- 172.21.15.55:0/71594196 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c0a5c00 con 0x7f809406c000 2024-03-19T01:55:12.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.839+0000 7f8099bc7640 1 -- 172.21.15.55:0/71594196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f809406c000 msgr2=0x7f809414e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:12.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.839+0000 7f8099bc7640 1 --2- 172.21.15.55:0/71594196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f809406c000 0x7f809414e4e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f808c009d90 tx=0x7f808c01c970 comp rx=0 tx=0).stop 2024-03-19T01:55:12.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.839+0000 7f8099bc7640 1 -- 172.21.15.55:0/71594196 shutdown_connections 2024-03-19T01:55:12.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.839+0000 7f8099bc7640 1 --2- 172.21.15.55:0/71594196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f809406c000 0x7f809414e4e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:12.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.839+0000 7f8099bc7640 1 -- 172.21.15.55:0/71594196 >> 172.21.15.55:0/71594196 conn(0x7f809406e670 msgr2=0x7f809406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:12.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.839+0000 7f8099bc7640 1 -- 172.21.15.55:0/71594196 shutdown_connections 2024-03-19T01:55:12.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.839+0000 7f8099bc7640 1 -- 172.21.15.55:0/71594196 wait complete. 2024-03-19T01:55:12.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.839+0000 7f8099bc7640 1 Processor -- start 2024-03-19T01:55:12.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.839+0000 7f8099bc7640 1 -- start start 2024-03-19T01:55:12.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.843+0000 7f8099bc7640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f809406c000 0x7f8094143a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:12.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.843+0000 7f8099bc7640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80940cd090 0x7f80941f46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:12.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.843+0000 7f8099bc7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80940cd470 0x7f809413aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:12.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.843+0000 7f8099bc7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f80941662b0 con 0x7f80940cd470 2024-03-19T01:55:12.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.843+0000 7f8099bc7640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f8094166130 con 0x7f80940cd090 2024-03-19T01:55:12.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.843+0000 7f8099bc7640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f8094166430 con 0x7f809406c000 2024-03-19T01:55:12.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.843+0000 7f8093fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80940cd470 0x7f809413aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:12.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.843+0000 7f8093fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80940cd470 0x7f809413aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50092/0 (socket says 172.21.15.55:50092) 2024-03-19T01:55:12.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.843+0000 7f8093fff640 1 -- 172.21.15.55:0/3629232338 learned_addr learned my addr 172.21.15.55:0/3629232338 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:12.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.843+0000 7f8092ffd640 1 --2- 172.21.15.55:0/3629232338 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80940cd090 0x7f80941f46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:12.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.843+0000 7f80937fe640 1 --2- 172.21.15.55:0/3629232338 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f809406c000 0x7f8094143a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:12.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.847+0000 7f8093fff640 1 -- 172.21.15.55:0/3629232338 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f809406c000 msgr2=0x7f8094143a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:12.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.847+0000 7f8093fff640 1 --2- 172.21.15.55:0/3629232338 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f809406c000 0x7f8094143a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:12.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.847+0000 7f8093fff640 1 -- 172.21.15.55:0/3629232338 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80940cd090 msgr2=0x7f80941f46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:12.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.847+0000 7f8093fff640 1 --2- 172.21.15.55:0/3629232338 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80940cd090 0x7f80941f46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:12.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.847+0000 7f8093fff640 1 -- 172.21.15.55:0/3629232338 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8094159710 con 0x7f80940cd470 2024-03-19T01:55:12.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.851+0000 7f8093fff640 1 --2- 172.21.15.55:0/3629232338 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80940cd470 0x7f809413aa30 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f808c019580 tx=0x7f808c019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:12.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.855+0000 7f807bfff640 1 -- 172.21.15.55:0/3629232338 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c0bb020 con 0x7f80940cd470 2024-03-19T01:55:12.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.855+0000 7f807bfff640 1 -- 172.21.15.55:0/3629232338 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f808c0ad440 con 0x7f80940cd470 2024-03-19T01:55:12.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.855+0000 7f8099bc7640 1 -- 172.21.15.55:0/3629232338 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f809415bc10 con 0x7f80940cd470 2024-03-19T01:55:12.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.855+0000 7f807bfff640 1 -- 172.21.15.55:0/3629232338 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c0b2040 con 0x7f80940cd470 2024-03-19T01:55:12.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.855+0000 7f8099bc7640 1 -- 172.21.15.55:0/3629232338 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f809415c1d0 con 0x7f80940cd470 2024-03-19T01:55:12.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.855+0000 7f8099bc7640 1 -- 172.21.15.55:0/3629232338 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8094068a40 con 0x7f80940cd470 2024-03-19T01:55:12.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.879+0000 7f807bfff640 1 -- 172.21.15.55:0/3629232338 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f808c0be020 con 0x7f80940cd470 2024-03-19T01:55:12.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.883+0000 7f807bfff640 1 --2- 172.21.15.55:0/3629232338 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8080067690 0x7f8080069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:12.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.883+0000 7f80937fe640 1 --2- 172.21.15.55:0/3629232338 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8080067690 0x7f8080069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:12.880 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:12.883+0000 7f80937fe640 1 --2- 172.21.15.55:0/3629232338 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8080067690 0x7f8080069b50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8084001770 tx=0x7f8084009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:13.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:13.195+0000 7f807bfff640 1 -- 172.21.15.55:0/3629232338 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f808c0ee810 con 0x7f80940cd470 2024-03-19T01:55:13.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:13.195+0000 7f807bfff640 1 -- 172.21.15.55:0/3629232338 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f808c0c5030 con 0x7f80940cd470 2024-03-19T01:55:13.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:13.431+0000 7f8099bc7640 1 -- 172.21.15.55:0/3629232338 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8094070de0 con 0x7f80940cd470 2024-03-19T01:55:13.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:13.447+0000 7f807bfff640 1 -- 172.21.15.55:0/3629232338 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7f808c016560 con 0x7f80940cd470 2024-03-19T01:55:13.443 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:13.443 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:55:13.444 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:55:13.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:13.447+0000 7f8099bc7640 1 -- 172.21.15.55:0/3629232338 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8080067690 msgr2=0x7f8080069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:13.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:13.447+0000 7f8099bc7640 1 --2- 172.21.15.55:0/3629232338 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8080067690 0x7f8080069b50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8084001770 tx=0x7f8084009040 comp rx=0 tx=0).stop 2024-03-19T01:55:13.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:13.447+0000 7f8099bc7640 1 -- 172.21.15.55:0/3629232338 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80940cd470 msgr2=0x7f809413aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:13.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:13.447+0000 7f8099bc7640 1 --2- 172.21.15.55:0/3629232338 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80940cd470 0x7f809413aa30 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f808c019580 tx=0x7f808c019710 comp rx=0 tx=0).stop 2024-03-19T01:55:13.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:13.447+0000 7f8099bc7640 1 -- 172.21.15.55:0/3629232338 shutdown_connections 2024-03-19T01:55:13.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:13.447+0000 7f8099bc7640 1 --2- 172.21.15.55:0/3629232338 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8080067690 0x7f8080069b50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:13.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:13.447+0000 7f8099bc7640 1 --2- 172.21.15.55:0/3629232338 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80940cd470 0x7f809413aa30 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:13.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:13.447+0000 7f8099bc7640 1 --2- 172.21.15.55:0/3629232338 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80940cd090 0x7f80941f46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:13.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:13.447+0000 7f8099bc7640 1 --2- 172.21.15.55:0/3629232338 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f809406c000 0x7f8094143a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:13.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:13.447+0000 7f8099bc7640 1 -- 172.21.15.55:0/3629232338 >> 172.21.15.55:0/3629232338 conn(0x7f809406e670 msgr2=0x7f80940701b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:13.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:13.447+0000 7f8099bc7640 1 -- 172.21.15.55:0/3629232338 shutdown_connections 2024-03-19T01:55:13.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:13.447+0000 7f8099bc7640 1 -- 172.21.15.55:0/3629232338 wait complete. 2024-03-19T01:55:14.065 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:55:14.065 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:15.066 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:15.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.183+0000 7f998eebb640 1 Processor -- start 2024-03-19T01:55:15.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.183+0000 7f998eebb640 1 -- start start 2024-03-19T01:55:15.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.183+0000 7f998eebb640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f99880cd470 0x7f998814e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:15.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.183+0000 7f998eebb640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f99880c8010 0x7f998814ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:15.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.183+0000 7f998eebb640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9988158330 con 0x7f99880cd090 2024-03-19T01:55:15.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.183+0000 7f998eebb640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f9988158e90 con 0x7f99880cd470 2024-03-19T01:55:15.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.183+0000 7f998eebb640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f99881585f0 con 0x7f99880c8010 2024-03-19T01:55:15.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.183+0000 7f9987fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f99880cd470 0x7f998814e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:15.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.183+0000 7f998d458640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f99880c8010 0x7f998814ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:15.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.183+0000 7f9987fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f99880cd470 0x7f998814e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50106/0 (socket says 172.21.15.55:50106) 2024-03-19T01:55:15.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.183+0000 7f9987fff640 1 -- 172.21.15.55:0/2605450094 learned_addr learned my addr 172.21.15.55:0/2605450094 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:15.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.187+0000 7f9987fff640 1 -- 172.21.15.55:0/2605450094 >> v2:172.21.15.84:3300/0 conn(0x7f99880c8010 msgr2=0x7f998814ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:15.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.187+0000 7f9987fff640 1 --2- 172.21.15.55:0/2605450094 >> v2:172.21.15.84:3300/0 conn(0x7f99880c8010 0x7f998814ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:15.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.187+0000 7f9987fff640 1 -- 172.21.15.55:0/2605450094 >> v1:172.21.15.84:6789/0 conn(0x7f99880cd090 legacy=0x7f99880732b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:15.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.187+0000 7f9987fff640 1 -- 172.21.15.55:0/2605450094 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99881588d0 con 0x7f99880cd470 2024-03-19T01:55:15.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.191+0000 7f9987fff640 1 --2- 172.21.15.55:0/2605450094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99880cd470 0x7f998814e550 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9988159060 tx=0x7f997401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=222660910282c33e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:15.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.195+0000 7f99877fe640 1 -- 172.21.15.55:0/2605450094 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99740b0070 con 0x7f99880cd470 2024-03-19T01:55:15.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.195+0000 7f99877fe640 1 -- 172.21.15.55:0/2605450094 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99740ab600 con 0x7f99880cd470 2024-03-19T01:55:15.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.195+0000 7f99877fe640 1 -- 172.21.15.55:0/2605450094 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99740ab9d0 con 0x7f99880cd470 2024-03-19T01:55:15.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.195+0000 7f998eebb640 1 -- 172.21.15.55:0/2605450094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99880cd470 msgr2=0x7f998814e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:15.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.195+0000 7f998eebb640 1 --2- 172.21.15.55:0/2605450094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99880cd470 0x7f998814e550 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9988159060 tx=0x7f997401c4e0 comp rx=0 tx=0).stop 2024-03-19T01:55:15.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.195+0000 7f998eebb640 1 -- 172.21.15.55:0/2605450094 shutdown_connections 2024-03-19T01:55:15.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.195+0000 7f998eebb640 1 --2- 172.21.15.55:0/2605450094 >> v2:172.21.15.84:3300/0 conn(0x7f99880c8010 0x7f998814ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:15.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.195+0000 7f998eebb640 1 --2- 172.21.15.55:0/2605450094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99880cd470 0x7f998814e550 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:15.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.195+0000 7f998eebb640 1 -- 172.21.15.55:0/2605450094 >> 172.21.15.55:0/2605450094 conn(0x7f998806e670 msgr2=0x7f998806ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:15.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.195+0000 7f998eebb640 1 -- 172.21.15.55:0/2605450094 shutdown_connections 2024-03-19T01:55:15.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.199+0000 7f998eebb640 1 -- 172.21.15.55:0/2605450094 wait complete. 2024-03-19T01:55:15.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.199+0000 7f998eebb640 1 Processor -- start 2024-03-19T01:55:15.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.199+0000 7f998eebb640 1 -- start start 2024-03-19T01:55:15.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.199+0000 7f998eebb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99880c8010 0x7f9988160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:15.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.199+0000 7f998eebb640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f99880cd090 0x7f99880cbe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:15.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.199+0000 7f998eebb640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f99880cd470 0x7f99880cc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:15.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.199+0000 7f998eebb640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9988164a90 con 0x7f99880c8010 2024-03-19T01:55:15.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.199+0000 7f998eebb640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f9988164910 con 0x7f99880cd090 2024-03-19T01:55:15.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.199+0000 7f998eebb640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f9988164c10 con 0x7f99880cd470 2024-03-19T01:55:15.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.199+0000 7f998cc57640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99880c8010 0x7f9988160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:15.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.199+0000 7f998d458640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f99880cd470 0x7f99880cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:15.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.199+0000 7f9987fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f99880cd090 0x7f99880cbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:15.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.199+0000 7f998cc57640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99880c8010 0x7f9988160ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50108/0 (socket says 172.21.15.55:50108) 2024-03-19T01:55:15.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.199+0000 7f998cc57640 1 -- 172.21.15.55:0/3433940576 learned_addr learned my addr 172.21.15.55:0/3433940576 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:15.203 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.207+0000 7f998cc57640 1 -- 172.21.15.55:0/3433940576 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f99880cd470 msgr2=0x7f99880cc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:15.203 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.207+0000 7f998cc57640 1 --2- 172.21.15.55:0/3433940576 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f99880cd470 0x7f99880cc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:15.203 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.207+0000 7f998cc57640 1 -- 172.21.15.55:0/3433940576 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f99880cd090 msgr2=0x7f99880cbe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:15.203 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.207+0000 7f998cc57640 1 --2- 172.21.15.55:0/3433940576 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f99880cd090 0x7f99880cbe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:15.203 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.207+0000 7f998cc57640 1 -- 172.21.15.55:0/3433940576 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99881f2f20 con 0x7f99880c8010 2024-03-19T01:55:15.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.211+0000 7f998cc57640 1 --2- 172.21.15.55:0/3433940576 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99880c8010 0x7f9988160ad0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f997c00c480 tx=0x7f997c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:15.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.215+0000 7f99857fa640 1 -- 172.21.15.55:0/3433940576 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f997c012830 con 0x7f99880c8010 2024-03-19T01:55:15.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.219+0000 7f99857fa640 1 -- 172.21.15.55:0/3433940576 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f997c017070 con 0x7f99880c8010 2024-03-19T01:55:15.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.219+0000 7f998eebb640 1 -- 172.21.15.55:0/3433940576 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99881f6f40 con 0x7f99880c8010 2024-03-19T01:55:15.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.219+0000 7f99857fa640 1 -- 172.21.15.55:0/3433940576 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f997c012c30 con 0x7f99880c8010 2024-03-19T01:55:15.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.219+0000 7f998eebb640 1 -- 172.21.15.55:0/3433940576 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99881f31e0 con 0x7f99880c8010 2024-03-19T01:55:15.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.219+0000 7f998eebb640 1 -- 172.21.15.55:0/3433940576 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9988068a40 con 0x7f99880c8010 2024-03-19T01:55:15.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.495+0000 7f99857fa640 1 -- 172.21.15.55:0/3433940576 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f997c01c460 con 0x7f99880c8010 2024-03-19T01:55:15.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.495+0000 7f99857fa640 1 --2- 172.21.15.55:0/3433940576 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f996c0676b0 0x7f996c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:15.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.495+0000 7f9987fff640 1 --2- 172.21.15.55:0/3433940576 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f996c0676b0 0x7f996c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:15.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.495+0000 7f9987fff640 1 --2- 172.21.15.55:0/3433940576 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f996c0676b0 0x7f996c069b70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f99740ad000 tx=0x7f99740a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:15.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.527+0000 7f99857fa640 1 -- 172.21.15.55:0/3433940576 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f997c054760 con 0x7f99880c8010 2024-03-19T01:55:15.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.527+0000 7f99857fa640 1 -- 172.21.15.55:0/3433940576 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f997c0566c0 con 0x7f99880c8010 2024-03-19T01:55:15.735 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.739+0000 7f998eebb640 1 -- 172.21.15.55:0/3433940576 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99880c79a0 con 0x7f99880c8010 2024-03-19T01:55:15.752 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.755+0000 7f99857fa640 1 -- 172.21.15.55:0/3433940576 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7f997c05a760 con 0x7f99880c8010 2024-03-19T01:55:15.753 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:15.754 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:55:15.754 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:55:15.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.759+0000 7f998eebb640 1 -- 172.21.15.55:0/3433940576 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f996c0676b0 msgr2=0x7f996c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:15.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.759+0000 7f998eebb640 1 --2- 172.21.15.55:0/3433940576 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f996c0676b0 0x7f996c069b70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f99740ad000 tx=0x7f99740a6000 comp rx=0 tx=0).stop 2024-03-19T01:55:15.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.759+0000 7f998eebb640 1 -- 172.21.15.55:0/3433940576 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99880c8010 msgr2=0x7f9988160ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:15.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.759+0000 7f998eebb640 1 --2- 172.21.15.55:0/3433940576 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99880c8010 0x7f9988160ad0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f997c00c480 tx=0x7f997c00c940 comp rx=0 tx=0).stop 2024-03-19T01:55:15.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.759+0000 7f998eebb640 1 -- 172.21.15.55:0/3433940576 shutdown_connections 2024-03-19T01:55:15.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.759+0000 7f998eebb640 1 --2- 172.21.15.55:0/3433940576 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f996c0676b0 0x7f996c069b70 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:15.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.759+0000 7f998eebb640 1 --2- 172.21.15.55:0/3433940576 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f99880cd470 0x7f99880cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:15.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.759+0000 7f998eebb640 1 --2- 172.21.15.55:0/3433940576 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f99880cd090 0x7f99880cbe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:15.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.759+0000 7f998eebb640 1 --2- 172.21.15.55:0/3433940576 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f99880c8010 0x7f9988160ad0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:15.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.759+0000 7f998eebb640 1 -- 172.21.15.55:0/3433940576 >> 172.21.15.55:0/3433940576 conn(0x7f998806e670 msgr2=0x7f9988145180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:15.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.759+0000 7f998eebb640 1 -- 172.21.15.55:0/3433940576 shutdown_connections 2024-03-19T01:55:15.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:15.759+0000 7f998eebb640 1 -- 172.21.15.55:0/3433940576 wait complete. 2024-03-19T01:55:16.378 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:55:16.378 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:17.378 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:17.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.495+0000 7fd7f1c86640 1 Processor -- start 2024-03-19T01:55:17.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.495+0000 7fd7f1c86640 1 -- start start 2024-03-19T01:55:17.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.495+0000 7fd7f1c86640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd7ec152120 0x7fd7ec0c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:17.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.495+0000 7fd7f1c86640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7ec06fff0 con 0x7fd7ec070780 2024-03-19T01:55:17.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.495+0000 7fd7f1c86640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7ec0c39c0 con 0x7fd7ec0c9610 2024-03-19T01:55:17.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.495+0000 7fd7f1c86640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fd7ec152500 con 0x7fd7ec152120 2024-03-19T01:55:17.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.499+0000 7fd7eb7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd7ec152120 0x7fd7ec0c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:17.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.499+0000 7fd7eaffd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fd7ec0c9610 0x7fd7ec157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:49798/0 (socket says 172.21.15.55:49798) 2024-03-19T01:55:17.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.499+0000 7fd7eaffd640 1 -- 172.21.15.55:0/3596800529 learned_addr learned my addr 172.21.15.55:0/3596800529 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:55:17.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.499+0000 7fd7ebfff640 1 --1- 172.21.15.55:0/3596800529 >> v1:172.21.15.55:6789/0 conn(0x7fd7ec070780 0x7fd7ec1586e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:42160/0 (socket says 172.21.15.55:42160) 2024-03-19T01:55:17.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.499+0000 7fd7eb7fe640 1 -- 172.21.15.55:0/3596800529 >> v1:172.21.15.84:6789/0 conn(0x7fd7ec0c9610 legacy=0x7fd7ec157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:17.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.499+0000 7fd7eb7fe640 1 -- 172.21.15.55:0/3596800529 >> v1:172.21.15.55:6789/0 conn(0x7fd7ec070780 legacy=0x7fd7ec1586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:17.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.499+0000 7fd7eb7fe640 1 -- 172.21.15.55:0/3596800529 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ec0702b0 con 0x7fd7ec152120 2024-03-19T01:55:17.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.503+0000 7fd7eb7fe640 1 --2- 172.21.15.55:0/3596800529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7ec152120 0x7fd7ec0c90d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc0097c0 tx=0x7fd7dc01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fcb1dfba11007b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:17.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.511+0000 7fd7ea7fc640 1 -- 172.21.15.55:0/3596800529 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc0b0070 con 0x7fd7ec152120 2024-03-19T01:55:17.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.511+0000 7fd7ea7fc640 1 -- 172.21.15.55:0/3596800529 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7dc0ab600 con 0x7fd7ec152120 2024-03-19T01:55:17.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.511+0000 7fd7ea7fc640 1 -- 172.21.15.55:0/3596800529 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc0ab9d0 con 0x7fd7ec152120 2024-03-19T01:55:17.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.511+0000 7fd7f1c86640 1 -- 172.21.15.55:0/3596800529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7ec152120 msgr2=0x7fd7ec0c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:17.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.511+0000 7fd7f1c86640 1 --2- 172.21.15.55:0/3596800529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7ec152120 0x7fd7ec0c90d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc0097c0 tx=0x7fd7dc01c460 comp rx=0 tx=0).stop 2024-03-19T01:55:17.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.511+0000 7fd7f1c86640 1 -- 172.21.15.55:0/3596800529 shutdown_connections 2024-03-19T01:55:17.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.511+0000 7fd7f1c86640 1 --2- 172.21.15.55:0/3596800529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7ec152120 0x7fd7ec0c90d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:17.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.511+0000 7fd7f1c86640 1 -- 172.21.15.55:0/3596800529 >> 172.21.15.55:0/3596800529 conn(0x7fd7ec067600 msgr2=0x7fd7ec071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:17.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.511+0000 7fd7f1c86640 1 -- 172.21.15.55:0/3596800529 shutdown_connections 2024-03-19T01:55:17.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.511+0000 7fd7f1c86640 1 -- 172.21.15.55:0/3596800529 wait complete. 2024-03-19T01:55:17.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.511+0000 7fd7f1c86640 1 Processor -- start 2024-03-19T01:55:17.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.511+0000 7fd7f1c86640 1 -- start start 2024-03-19T01:55:17.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.511+0000 7fd7f1c86640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd7ec070780 0x7fd7ec0c4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:17.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.511+0000 7fd7f1c86640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7ec0c9610 0x7fd7ec15a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:17.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.511+0000 7fd7f1c86640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7ec152120 0x7fd7ec15ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:17.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.511+0000 7fd7f1c86640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ec158df0 con 0x7fd7ec152120 2024-03-19T01:55:17.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.511+0000 7fd7f1c86640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ec0c3530 con 0x7fd7ec0c9610 2024-03-19T01:55:17.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.511+0000 7fd7f1c86640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd7ec0c86d0 con 0x7fd7ec070780 2024-03-19T01:55:17.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.515+0000 7fd7ebfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7ec152120 0x7fd7ec15ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:17.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.515+0000 7fd7eb7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd7ec070780 0x7fd7ec0c4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:17.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.515+0000 7fd7ebfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7ec152120 0x7fd7ec15ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50120/0 (socket says 172.21.15.55:50120) 2024-03-19T01:55:17.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.515+0000 7fd7ebfff640 1 -- 172.21.15.55:0/2737295211 learned_addr learned my addr 172.21.15.55:0/2737295211 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:17.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.515+0000 7fd7eaffd640 1 --2- 172.21.15.55:0/2737295211 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7ec0c9610 0x7fd7ec15a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:17.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.519+0000 7fd7ebfff640 1 -- 172.21.15.55:0/2737295211 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd7ec070780 msgr2=0x7fd7ec0c4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:17.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.519+0000 7fd7ebfff640 1 --2- 172.21.15.55:0/2737295211 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd7ec070780 0x7fd7ec0c4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:17.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.519+0000 7fd7ebfff640 1 -- 172.21.15.55:0/2737295211 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7ec0c9610 msgr2=0x7fd7ec15a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:17.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.519+0000 7fd7ebfff640 1 --2- 172.21.15.55:0/2737295211 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7ec0c9610 0x7fd7ec15a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:17.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.519+0000 7fd7ebfff640 1 -- 172.21.15.55:0/2737295211 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ec15c1b0 con 0x7fd7ec152120 2024-03-19T01:55:17.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.519+0000 7fd7ebfff640 1 --2- 172.21.15.55:0/2737295211 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7ec152120 0x7fd7ec15ada0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd7e400c760 tx=0x7fd7e400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:17.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.523+0000 7fd7d3fff640 1 -- 172.21.15.55:0/2737295211 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4017070 con 0x7fd7ec152120 2024-03-19T01:55:17.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.523+0000 7fd7d3fff640 1 -- 172.21.15.55:0/2737295211 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7e400a640 con 0x7fd7ec152120 2024-03-19T01:55:17.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.523+0000 7fd7f1c86640 1 -- 172.21.15.55:0/2737295211 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7ec0c7b00 con 0x7fd7ec152120 2024-03-19T01:55:17.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.523+0000 7fd7d3fff640 1 -- 172.21.15.55:0/2737295211 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e400aa30 con 0x7fd7ec152120 2024-03-19T01:55:17.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.527+0000 7fd7f1c86640 1 -- 172.21.15.55:0/2737295211 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7ec0c53f0 con 0x7fd7ec152120 2024-03-19T01:55:17.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.527+0000 7fd7f1c86640 1 -- 172.21.15.55:0/2737295211 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7ec068a40 con 0x7fd7ec152120 2024-03-19T01:55:17.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.547+0000 7fd7d3fff640 1 -- 172.21.15.55:0/2737295211 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fd7e401a070 con 0x7fd7ec152120 2024-03-19T01:55:17.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.547+0000 7fd7d3fff640 1 --2- 172.21.15.55:0/2737295211 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd7d80676b0 0x7fd7d8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:17.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.547+0000 7fd7eb7fe640 1 --2- 172.21.15.55:0/2737295211 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd7d80676b0 0x7fd7d8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:17.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.547+0000 7fd7eb7fe640 1 --2- 172.21.15.55:0/2737295211 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd7d80676b0 0x7fd7d8069b70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc0ad000 tx=0x7fd7dc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:17.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.807+0000 7fd7d3fff640 1 -- 172.21.15.55:0/2737295211 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fd7e40539f0 con 0x7fd7ec152120 2024-03-19T01:55:17.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:17.811+0000 7fd7d3fff640 1 -- 172.21.15.55:0/2737295211 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fd7e4021020 con 0x7fd7ec152120 2024-03-19T01:55:18.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:18.015+0000 7fd7f1c86640 1 -- 172.21.15.55:0/2737295211 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7ec0c84b0 con 0x7fd7ec152120 2024-03-19T01:55:18.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:18.039+0000 7fd7d3fff640 1 -- 172.21.15.55:0/2737295211 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7fd7e4055950 con 0x7fd7ec152120 2024-03-19T01:55:18.039 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:18.039 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:55:18.039 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:55:18.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:18.043+0000 7fd7f1c86640 1 -- 172.21.15.55:0/2737295211 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd7d80676b0 msgr2=0x7fd7d8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:18.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:18.043+0000 7fd7f1c86640 1 --2- 172.21.15.55:0/2737295211 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd7d80676b0 0x7fd7d8069b70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc0ad000 tx=0x7fd7dc008040 comp rx=0 tx=0).stop 2024-03-19T01:55:18.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:18.043+0000 7fd7f1c86640 1 -- 172.21.15.55:0/2737295211 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7ec152120 msgr2=0x7fd7ec15ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:18.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:18.043+0000 7fd7f1c86640 1 --2- 172.21.15.55:0/2737295211 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7ec152120 0x7fd7ec15ada0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd7e400c760 tx=0x7fd7e400cc20 comp rx=0 tx=0).stop 2024-03-19T01:55:18.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:18.043+0000 7fd7f1c86640 1 -- 172.21.15.55:0/2737295211 shutdown_connections 2024-03-19T01:55:18.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:18.043+0000 7fd7f1c86640 1 --2- 172.21.15.55:0/2737295211 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd7d80676b0 0x7fd7d8069b70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:18.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:18.043+0000 7fd7f1c86640 1 --2- 172.21.15.55:0/2737295211 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7ec152120 0x7fd7ec15ada0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:18.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:18.043+0000 7fd7f1c86640 1 --2- 172.21.15.55:0/2737295211 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7ec0c9610 0x7fd7ec15a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:18.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:18.043+0000 7fd7f1c86640 1 --2- 172.21.15.55:0/2737295211 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd7ec070780 0x7fd7ec0c4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:18.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:18.043+0000 7fd7f1c86640 1 -- 172.21.15.55:0/2737295211 >> 172.21.15.55:0/2737295211 conn(0x7fd7ec067600 msgr2=0x7fd7ec1508b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:18.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:18.043+0000 7fd7f1c86640 1 -- 172.21.15.55:0/2737295211 shutdown_connections 2024-03-19T01:55:18.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:18.043+0000 7fd7f1c86640 1 -- 172.21.15.55:0/2737295211 wait complete. 2024-03-19T01:55:18.660 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:55:18.661 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:19.661 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:19.776 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.779+0000 7fb7685a2640 1 Processor -- start 2024-03-19T01:55:19.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.779+0000 7fb7685a2640 1 -- start start 2024-03-19T01:55:19.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.779+0000 7fb7685a2640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fb76013dd00 0x7fb7600c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:19.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.779+0000 7fb7685a2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb76015b550 0x7fb7600c5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:19.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.779+0000 7fb7685a2640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7600c8f70 con 0x7fb7600c9780 2024-03-19T01:55:19.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.779+0000 7fb7685a2640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fb7600c9250 con 0x7fb76013dd00 2024-03-19T01:55:19.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.779+0000 7fb7685a2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fb7601582b0 con 0x7fb76015b550 2024-03-19T01:55:19.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.779+0000 7fb765b3d640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fb7600c9780 0x7fb7600c8740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:41516/0 (socket says 172.21.15.55:41516) 2024-03-19T01:55:19.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.779+0000 7fb765b3d640 1 -- 172.21.15.55:0/3924797050 learned_addr learned my addr 172.21.15.55:0/3924797050 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:55:19.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.779+0000 7fb766b3f640 1 --2- 172.21.15.55:0/3924797050 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb76015b550 0x7fb7600c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:19.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.783+0000 7fb76633e640 1 --2- 172.21.15.55:0/3924797050 >> v2:172.21.15.84:3300/0 conn(0x7fb76013dd00 0x7fb7600c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:19.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.783+0000 7fb766b3f640 1 -- 172.21.15.55:0/3924797050 >> v2:172.21.15.84:3300/0 conn(0x7fb76013dd00 msgr2=0x7fb7600c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:19.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.783+0000 7fb766b3f640 1 --2- 172.21.15.55:0/3924797050 >> v2:172.21.15.84:3300/0 conn(0x7fb76013dd00 0x7fb7600c7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:19.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.783+0000 7fb766b3f640 1 -- 172.21.15.55:0/3924797050 >> v1:172.21.15.55:6789/0 conn(0x7fb7600c9780 legacy=0x7fb7600c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:19.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.783+0000 7fb766b3f640 1 -- 172.21.15.55:0/3924797050 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb760158590 con 0x7fb76015b550 2024-03-19T01:55:19.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.787+0000 7fb766b3f640 1 --2- 172.21.15.55:0/3924797050 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb76015b550 0x7fb7600c5730 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb758009d90 tx=0x7fb75801c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cc599d42ed6f690 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:19.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.795+0000 7fb76533c640 1 -- 172.21.15.55:0/3924797050 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7580b2070 con 0x7fb76015b550 2024-03-19T01:55:19.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.795+0000 7fb76533c640 1 -- 172.21.15.55:0/3924797050 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7580a5940 con 0x7fb76015b550 2024-03-19T01:55:19.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.795+0000 7fb76533c640 1 -- 172.21.15.55:0/3924797050 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7580a5d10 con 0x7fb76015b550 2024-03-19T01:55:19.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.799+0000 7fb7685a2640 1 -- 172.21.15.55:0/3924797050 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb76015b550 msgr2=0x7fb7600c5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:19.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.799+0000 7fb7685a2640 1 --2- 172.21.15.55:0/3924797050 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb76015b550 0x7fb7600c5730 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb758009d90 tx=0x7fb75801c970 comp rx=0 tx=0).stop 2024-03-19T01:55:19.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.799+0000 7fb7685a2640 1 -- 172.21.15.55:0/3924797050 shutdown_connections 2024-03-19T01:55:19.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.799+0000 7fb7685a2640 1 --2- 172.21.15.55:0/3924797050 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb76015b550 0x7fb7600c5730 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:19.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.799+0000 7fb7685a2640 1 --2- 172.21.15.55:0/3924797050 >> v2:172.21.15.84:3300/0 conn(0x7fb76013dd00 0x7fb7600c7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:19.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.799+0000 7fb7685a2640 1 -- 172.21.15.55:0/3924797050 >> 172.21.15.55:0/3924797050 conn(0x7fb76006e5a0 msgr2=0x7fb760072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:19.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.799+0000 7fb7685a2640 1 -- 172.21.15.55:0/3924797050 shutdown_connections 2024-03-19T01:55:19.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.799+0000 7fb7685a2640 1 -- 172.21.15.55:0/3924797050 wait complete. 2024-03-19T01:55:19.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.799+0000 7fb7685a2640 1 Processor -- start 2024-03-19T01:55:19.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.799+0000 7fb7685a2640 1 -- start start 2024-03-19T01:55:19.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.799+0000 7fb7685a2640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb7600c9780 0x7fb76013a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:19.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.799+0000 7fb7685a2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb76013dd00 0x7fb760133a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:19.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.799+0000 7fb7685a2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb76015b550 0x7fb760150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:19.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.799+0000 7fb7685a2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb760165280 con 0x7fb76015b550 2024-03-19T01:55:19.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.799+0000 7fb7685a2640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb760165100 con 0x7fb7600c9780 2024-03-19T01:55:19.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.799+0000 7fb7685a2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fb760165400 con 0x7fb76013dd00 2024-03-19T01:55:19.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.799+0000 7fb766b3f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb76015b550 0x7fb760150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:19.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.799+0000 7fb76633e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb7600c9780 0x7fb76013a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:19.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.799+0000 7fb766b3f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb76015b550 0x7fb760150df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57908/0 (socket says 172.21.15.55:57908) 2024-03-19T01:55:19.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.799+0000 7fb766b3f640 1 -- 172.21.15.55:0/868139637 learned_addr learned my addr 172.21.15.55:0/868139637 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:19.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.803+0000 7fb765b3d640 1 --2- 172.21.15.55:0/868139637 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb76013dd00 0x7fb760133a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:19.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.807+0000 7fb766b3f640 1 -- 172.21.15.55:0/868139637 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb76013dd00 msgr2=0x7fb760133a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:19.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.807+0000 7fb766b3f640 1 --2- 172.21.15.55:0/868139637 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb76013dd00 0x7fb760133a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:19.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.807+0000 7fb766b3f640 1 -- 172.21.15.55:0/868139637 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb7600c9780 msgr2=0x7fb76013a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:19.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.807+0000 7fb766b3f640 1 --2- 172.21.15.55:0/868139637 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb7600c9780 0x7fb76013a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:19.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.807+0000 7fb766b3f640 1 -- 172.21.15.55:0/868139637 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb76015b240 con 0x7fb76015b550 2024-03-19T01:55:19.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.811+0000 7fb766b3f640 1 --2- 172.21.15.55:0/868139637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb76015b550 0x7fb760150df0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb75801a5a0 tx=0x7fb758019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:19.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.815+0000 7fb756ffd640 1 -- 172.21.15.55:0/868139637 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7580b2040 con 0x7fb76015b550 2024-03-19T01:55:19.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.815+0000 7fb756ffd640 1 -- 172.21.15.55:0/868139637 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7580bd070 con 0x7fb76015b550 2024-03-19T01:55:19.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.815+0000 7fb756ffd640 1 -- 172.21.15.55:0/868139637 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7580ad860 con 0x7fb76015b550 2024-03-19T01:55:19.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.815+0000 7fb7685a2640 1 -- 172.21.15.55:0/868139637 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb76013b7d0 con 0x7fb76015b550 2024-03-19T01:55:19.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.815+0000 7fb7685a2640 1 -- 172.21.15.55:0/868139637 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7601592c0 con 0x7fb76015b550 2024-03-19T01:55:19.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:19.815+0000 7fb7685a2640 1 -- 172.21.15.55:0/868139637 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb760068a40 con 0x7fb76015b550 2024-03-19T01:55:20.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:20.091+0000 7fb756ffd640 1 -- 172.21.15.55:0/868139637 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb7580ada00 con 0x7fb76015b550 2024-03-19T01:55:20.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:20.091+0000 7fb756ffd640 1 --2- 172.21.15.55:0/868139637 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb744067660 0x7fb744069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:20.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:20.091+0000 7fb76633e640 1 --2- 172.21.15.55:0/868139637 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb744067660 0x7fb744069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:20.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:20.095+0000 7fb76633e640 1 --2- 172.21.15.55:0/868139637 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb744067660 0x7fb744069b20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb75c009ac0 tx=0x7fb75c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:20.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:20.115+0000 7fb756ffd640 1 -- 172.21.15.55:0/868139637 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fb758002020 con 0x7fb76015b550 2024-03-19T01:55:20.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:20.115+0000 7fb756ffd640 1 -- 172.21.15.55:0/868139637 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fb758016560 con 0x7fb76015b550 2024-03-19T01:55:20.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:20.319+0000 7fb7685a2640 1 -- 172.21.15.55:0/868139637 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7601626c0 con 0x7fb76015b550 2024-03-19T01:55:20.342 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:20.343+0000 7fb756ffd640 1 -- 172.21.15.55:0/868139637 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7fb7580f1190 con 0x7fb76015b550 2024-03-19T01:55:20.342 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:20.342 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:55:20.343 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:55:20.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:20.347+0000 7fb7685a2640 1 -- 172.21.15.55:0/868139637 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb744067660 msgr2=0x7fb744069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:20.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:20.347+0000 7fb7685a2640 1 --2- 172.21.15.55:0/868139637 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb744067660 0x7fb744069b20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb75c009ac0 tx=0x7fb75c009190 comp rx=0 tx=0).stop 2024-03-19T01:55:20.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:20.347+0000 7fb7685a2640 1 -- 172.21.15.55:0/868139637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb76015b550 msgr2=0x7fb760150df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:20.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:20.347+0000 7fb7685a2640 1 --2- 172.21.15.55:0/868139637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb76015b550 0x7fb760150df0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb75801a5a0 tx=0x7fb758019710 comp rx=0 tx=0).stop 2024-03-19T01:55:20.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:20.347+0000 7fb7685a2640 1 -- 172.21.15.55:0/868139637 shutdown_connections 2024-03-19T01:55:20.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:20.347+0000 7fb7685a2640 1 --2- 172.21.15.55:0/868139637 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb744067660 0x7fb744069b20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:20.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:20.347+0000 7fb7685a2640 1 --2- 172.21.15.55:0/868139637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb76015b550 0x7fb760150df0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:20.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:20.347+0000 7fb7685a2640 1 --2- 172.21.15.55:0/868139637 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb76013dd00 0x7fb760133a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:20.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:20.347+0000 7fb7685a2640 1 --2- 172.21.15.55:0/868139637 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb7600c9780 0x7fb76013a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:20.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:20.347+0000 7fb7685a2640 1 -- 172.21.15.55:0/868139637 >> 172.21.15.55:0/868139637 conn(0x7fb76006e5a0 msgr2=0x7fb760155b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:20.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:20.347+0000 7fb7685a2640 1 -- 172.21.15.55:0/868139637 shutdown_connections 2024-03-19T01:55:20.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:20.347+0000 7fb7685a2640 1 -- 172.21.15.55:0/868139637 wait complete. 2024-03-19T01:55:20.965 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:55:20.965 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:21.966 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:22.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.083+0000 7f2dd3b3d640 1 Processor -- start 2024-03-19T01:55:22.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.083+0000 7f2dd3b3d640 1 -- start start 2024-03-19T01:55:22.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.083+0000 7f2dd3b3d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dcc15acb0 0x7f2dcc073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:22.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.083+0000 7f2dd3b3d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dcc0c7fa0 con 0x7f2dcc1544f0 2024-03-19T01:55:22.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.083+0000 7f2dd3b3d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dcc0c8240 con 0x7f2dcc06b2a0 2024-03-19T01:55:22.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.083+0000 7f2dd3b3d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2dcc15b210 con 0x7f2dcc15acb0 2024-03-19T01:55:22.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.083+0000 7f2dd18d9640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f2dcc1544f0 0x7f2dcc15e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:41518/0 (socket says 172.21.15.55:41518) 2024-03-19T01:55:22.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.083+0000 7f2dd18d9640 1 -- 172.21.15.55:0/2344430617 learned_addr learned my addr 172.21.15.55:0/2344430617 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:55:22.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.087+0000 7f2dd20da640 1 --2- 172.21.15.55:0/2344430617 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dcc15acb0 0x7f2dcc073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:22.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.087+0000 7f2dd08d7640 1 -- 172.21.15.55:0/2344430617 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1912704447 0 0) 0x7f2dcc0c7fa0 con 0x7f2dcc1544f0 2024-03-19T01:55:22.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.087+0000 7f2dd08d7640 1 -- 172.21.15.55:0/2344430617 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2db8003530 con 0x7f2dcc1544f0 2024-03-19T01:55:22.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.091+0000 7f2dd08d7640 1 -- 172.21.15.55:0/2344430617 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4157095265 0 0) 0x7f2dcc0c8240 con 0x7f2dcc06b2a0 2024-03-19T01:55:22.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.091+0000 7f2dd08d7640 1 -- 172.21.15.55:0/2344430617 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dcc0c7fa0 con 0x7f2dcc06b2a0 2024-03-19T01:55:22.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.091+0000 7f2dd20da640 1 -- 172.21.15.55:0/2344430617 >> v1:172.21.15.84:6789/0 conn(0x7f2dcc06b2a0 legacy=0x7f2dcc0cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:22.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.091+0000 7f2dd20da640 1 -- 172.21.15.55:0/2344430617 >> v1:172.21.15.55:6789/0 conn(0x7f2dcc1544f0 legacy=0x7f2dcc15e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:22.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.091+0000 7f2dd20da640 1 -- 172.21.15.55:0/2344430617 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dcc15d920 con 0x7f2dcc15acb0 2024-03-19T01:55:22.092 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.095+0000 7f2dd20da640 1 --2- 172.21.15.55:0/2344430617 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dcc15acb0 0x7f2dcc073540 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4009dc0 tx=0x7f2dc401f170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ba6d8f03f3fe02a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:22.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.103+0000 7f2dd08d7640 1 -- 172.21.15.55:0/2344430617 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc40b4070 con 0x7f2dcc15acb0 2024-03-19T01:55:22.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.103+0000 7f2dd08d7640 1 -- 172.21.15.55:0/2344430617 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dc40af600 con 0x7f2dcc15acb0 2024-03-19T01:55:22.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.103+0000 7f2dd08d7640 1 -- 172.21.15.55:0/2344430617 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc40af9d0 con 0x7f2dcc15acb0 2024-03-19T01:55:22.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.103+0000 7f2dd3b3d640 1 -- 172.21.15.55:0/2344430617 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dcc15acb0 msgr2=0x7f2dcc073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:22.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.103+0000 7f2dd3b3d640 1 --2- 172.21.15.55:0/2344430617 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dcc15acb0 0x7f2dcc073540 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4009dc0 tx=0x7f2dc401f170 comp rx=0 tx=0).stop 2024-03-19T01:55:22.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.103+0000 7f2dd3b3d640 1 -- 172.21.15.55:0/2344430617 shutdown_connections 2024-03-19T01:55:22.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.103+0000 7f2dd3b3d640 1 --2- 172.21.15.55:0/2344430617 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dcc15acb0 0x7f2dcc073540 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:22.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.103+0000 7f2dd3b3d640 1 -- 172.21.15.55:0/2344430617 >> 172.21.15.55:0/2344430617 conn(0x7f2dcc06e670 msgr2=0x7f2dcc06ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:22.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.103+0000 7f2dd3b3d640 1 -- 172.21.15.55:0/2344430617 shutdown_connections 2024-03-19T01:55:22.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.103+0000 7f2dd3b3d640 1 -- 172.21.15.55:0/2344430617 wait complete. 2024-03-19T01:55:22.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.103+0000 7f2dd3b3d640 1 Processor -- start 2024-03-19T01:55:22.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.103+0000 7f2dd3b3d640 1 -- start start 2024-03-19T01:55:22.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.107+0000 7f2dd3b3d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2dcc06b2a0 0x7f2dcc070660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:22.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.107+0000 7f2dd3b3d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2dcc1544f0 0x7f2dcc151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:22.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.107+0000 7f2dd3b3d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dcc15acb0 0x7f2dcc1522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:22.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.107+0000 7f2dd3b3d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2dcc0cd540 con 0x7f2dcc06b2a0 2024-03-19T01:55:22.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.107+0000 7f2dd3b3d640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2dcc1548d0 con 0x7f2dcc1544f0 2024-03-19T01:55:22.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.107+0000 7f2dd3b3d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2dcc0cd6c0 con 0x7f2dcc15acb0 2024-03-19T01:55:22.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.107+0000 7f2dd18d9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2dcc06b2a0 0x7f2dcc070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:22.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.107+0000 7f2dd10d8640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2dcc1544f0 0x7f2dcc151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:22.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.107+0000 7f2dd18d9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2dcc06b2a0 0x7f2dcc070660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57914/0 (socket says 172.21.15.55:57914) 2024-03-19T01:55:22.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.107+0000 7f2dd18d9640 1 -- 172.21.15.55:0/356766453 learned_addr learned my addr 172.21.15.55:0/356766453 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:22.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.107+0000 7f2dd20da640 1 --2- 172.21.15.55:0/356766453 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dcc15acb0 0x7f2dcc1522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:22.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.111+0000 7f2dd18d9640 1 -- 172.21.15.55:0/356766453 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dcc15acb0 msgr2=0x7f2dcc1522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:22.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.111+0000 7f2dd18d9640 1 --2- 172.21.15.55:0/356766453 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dcc15acb0 0x7f2dcc1522b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:22.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.111+0000 7f2dd18d9640 1 -- 172.21.15.55:0/356766453 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2dcc1544f0 msgr2=0x7f2dcc151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:22.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.111+0000 7f2dd18d9640 1 --2- 172.21.15.55:0/356766453 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2dcc1544f0 0x7f2dcc151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:22.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.111+0000 7f2dd18d9640 1 -- 172.21.15.55:0/356766453 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dcc15b7d0 con 0x7f2dcc06b2a0 2024-03-19T01:55:22.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.111+0000 7f2dd18d9640 1 --2- 172.21.15.55:0/356766453 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2dcc06b2a0 0x7f2dcc070660 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2dc800c500 tx=0x7f2dc800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:22.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.115+0000 7f2dc27fc640 1 -- 172.21.15.55:0/356766453 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc8017070 con 0x7f2dcc06b2a0 2024-03-19T01:55:22.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.115+0000 7f2dc27fc640 1 -- 172.21.15.55:0/356766453 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dc8005cf0 con 0x7f2dcc06b2a0 2024-03-19T01:55:22.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.115+0000 7f2dd3b3d640 1 -- 172.21.15.55:0/356766453 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dcc1f0160 con 0x7f2dcc06b2a0 2024-03-19T01:55:22.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.115+0000 7f2dc27fc640 1 -- 172.21.15.55:0/356766453 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc8007330 con 0x7f2dcc06b2a0 2024-03-19T01:55:22.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.115+0000 7f2dd3b3d640 1 -- 172.21.15.55:0/356766453 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dcc1f0720 con 0x7f2dcc06b2a0 2024-03-19T01:55:22.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.119+0000 7f2dd3b3d640 1 -- 172.21.15.55:0/356766453 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dcc068a40 con 0x7f2dcc06b2a0 2024-03-19T01:55:22.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.147+0000 7f2dc27fc640 1 -- 172.21.15.55:0/356766453 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f2dc801a070 con 0x7f2dcc06b2a0 2024-03-19T01:55:22.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.147+0000 7f2dc27fc640 1 --2- 172.21.15.55:0/356766453 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2db00676e0 0x7f2db0069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:22.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.147+0000 7f2dd10d8640 1 --2- 172.21.15.55:0/356766453 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2db00676e0 0x7f2db0069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:22.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.147+0000 7f2dd10d8640 1 --2- 172.21.15.55:0/356766453 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2db00676e0 0x7f2db0069ba0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc0c8e30 tx=0x7f2dbc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:22.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.151+0000 7f2dc27fc640 1 -- 172.21.15.55:0/356766453 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f2dc80537d0 con 0x7f2dcc06b2a0 2024-03-19T01:55:22.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.435+0000 7f2dc27fc640 1 -- 172.21.15.55:0/356766453 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2dc8021020 con 0x7f2dcc06b2a0 2024-03-19T01:55:22.688 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.691+0000 7f2dd3b3d640 1 -- 172.21.15.55:0/356766453 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2dcc0c7be0 con 0x7f2dcc06b2a0 2024-03-19T01:55:22.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.747+0000 7f2dc27fc640 1 -- 172.21.15.55:0/356766453 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7f2dc8055730 con 0x7f2dcc06b2a0 2024-03-19T01:55:22.744 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:22.744 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:55:22.744 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:55:22.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.747+0000 7f2dd3b3d640 1 -- 172.21.15.55:0/356766453 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2db00676e0 msgr2=0x7f2db0069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:22.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.747+0000 7f2dd3b3d640 1 --2- 172.21.15.55:0/356766453 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2db00676e0 0x7f2db0069ba0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc0c8e30 tx=0x7f2dbc009040 comp rx=0 tx=0).stop 2024-03-19T01:55:22.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.747+0000 7f2dd3b3d640 1 -- 172.21.15.55:0/356766453 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2dcc06b2a0 msgr2=0x7f2dcc070660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:22.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.747+0000 7f2dd3b3d640 1 --2- 172.21.15.55:0/356766453 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2dcc06b2a0 0x7f2dcc070660 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2dc800c500 tx=0x7f2dc800c9c0 comp rx=0 tx=0).stop 2024-03-19T01:55:22.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.747+0000 7f2dd3b3d640 1 -- 172.21.15.55:0/356766453 shutdown_connections 2024-03-19T01:55:22.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.747+0000 7f2dd3b3d640 1 --2- 172.21.15.55:0/356766453 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2db00676e0 0x7f2db0069ba0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:22.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.747+0000 7f2dd3b3d640 1 --2- 172.21.15.55:0/356766453 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dcc15acb0 0x7f2dcc1522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:22.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.747+0000 7f2dd3b3d640 1 --2- 172.21.15.55:0/356766453 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2dcc1544f0 0x7f2dcc151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:22.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.747+0000 7f2dd3b3d640 1 --2- 172.21.15.55:0/356766453 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2dcc06b2a0 0x7f2dcc070660 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:22.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.747+0000 7f2dd3b3d640 1 -- 172.21.15.55:0/356766453 >> 172.21.15.55:0/356766453 conn(0x7f2dcc06e670 msgr2=0x7f2dcc0cce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:22.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.747+0000 7f2dd3b3d640 1 -- 172.21.15.55:0/356766453 shutdown_connections 2024-03-19T01:55:22.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:22.747+0000 7f2dd3b3d640 1 -- 172.21.15.55:0/356766453 wait complete. 2024-03-19T01:55:23.368 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:55:23.368 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:24.369 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:24.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.503+0000 7f475c9f6640 1 Processor -- start 2024-03-19T01:55:24.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.503+0000 7f475c9f6640 1 -- start start 2024-03-19T01:55:24.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.503+0000 7f475c9f6640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f47581520b0 0x7f47580709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:24.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.503+0000 7f475c9f6640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f47580c9000 0x7f47580c9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:24.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.503+0000 7f475c9f6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4758158d50 con 0x7f47580c2ad0 2024-03-19T01:55:24.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.503+0000 7f475c9f6640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f475815c4e0 con 0x7f47581520b0 2024-03-19T01:55:24.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.503+0000 7f475c9f6640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f475815c050 con 0x7f47580c9000 2024-03-19T01:55:24.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.503+0000 7f475659c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f47581520b0 0x7f47580709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:24.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.503+0000 7f4755d9b640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f47580c9000 0x7f47580c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:24.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.503+0000 7f475659c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f47581520b0 0x7f47580709a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57920/0 (socket says 172.21.15.55:57920) 2024-03-19T01:55:24.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.503+0000 7f475659c640 1 -- 172.21.15.55:0/857751990 learned_addr learned my addr 172.21.15.55:0/857751990 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:24.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.507+0000 7f475659c640 1 -- 172.21.15.55:0/857751990 >> v2:172.21.15.84:3300/0 conn(0x7f47580c9000 msgr2=0x7f47580c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:24.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.507+0000 7f475659c640 1 --2- 172.21.15.55:0/857751990 >> v2:172.21.15.84:3300/0 conn(0x7f47580c9000 0x7f47580c9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:24.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.507+0000 7f475659c640 1 -- 172.21.15.55:0/857751990 >> v1:172.21.15.84:6789/0 conn(0x7f47580c2ad0 legacy=0x7f47580c2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:24.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.507+0000 7f475659c640 1 -- 172.21.15.55:0/857751990 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f475815b720 con 0x7f47581520b0 2024-03-19T01:55:24.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.511+0000 7f475659c640 1 --2- 172.21.15.55:0/857751990 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47581520b0 0x7f47580709a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4758072580 tx=0x7f474801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=961af9fb31bade1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:24.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.515+0000 7f475559a640 1 -- 172.21.15.55:0/857751990 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47480b0070 con 0x7f47581520b0 2024-03-19T01:55:24.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.515+0000 7f475559a640 1 -- 172.21.15.55:0/857751990 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47480ab600 con 0x7f47581520b0 2024-03-19T01:55:24.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.515+0000 7f475559a640 1 -- 172.21.15.55:0/857751990 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47480ab9d0 con 0x7f47581520b0 2024-03-19T01:55:24.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.515+0000 7f475c9f6640 1 -- 172.21.15.55:0/857751990 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47581520b0 msgr2=0x7f47580709a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:24.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.515+0000 7f475c9f6640 1 --2- 172.21.15.55:0/857751990 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47581520b0 0x7f47580709a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4758072580 tx=0x7f474801c520 comp rx=0 tx=0).stop 2024-03-19T01:55:24.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.515+0000 7f475c9f6640 1 -- 172.21.15.55:0/857751990 shutdown_connections 2024-03-19T01:55:24.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.515+0000 7f475c9f6640 1 --2- 172.21.15.55:0/857751990 >> v2:172.21.15.84:3300/0 conn(0x7f47580c9000 0x7f47580c9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:24.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.515+0000 7f475c9f6640 1 --2- 172.21.15.55:0/857751990 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47581520b0 0x7f47580709a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:24.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.515+0000 7f475c9f6640 1 -- 172.21.15.55:0/857751990 >> 172.21.15.55:0/857751990 conn(0x7f4758067600 msgr2=0x7f4758071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:24.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.515+0000 7f475c9f6640 1 -- 172.21.15.55:0/857751990 shutdown_connections 2024-03-19T01:55:24.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.515+0000 7f475c9f6640 1 -- 172.21.15.55:0/857751990 wait complete. 2024-03-19T01:55:24.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.519+0000 7f475c9f6640 1 Processor -- start 2024-03-19T01:55:24.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.519+0000 7f475c9f6640 1 -- start start 2024-03-19T01:55:24.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.519+0000 7f475c9f6640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f47580c2ad0 0x7f47580cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:24.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.519+0000 7f475c9f6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f47580c9000 0x7f47581f69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:24.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.519+0000 7f475c9f6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47581520b0 0x7f47581f6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:24.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.519+0000 7f475c9f6640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4758152610 con 0x7f47581520b0 2024-03-19T01:55:24.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.519+0000 7f475c9f6640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4758152490 con 0x7f47580c2ad0 2024-03-19T01:55:24.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.519+0000 7f475c9f6640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4758070ee0 con 0x7f47580c9000 2024-03-19T01:55:24.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.519+0000 7f4756d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47581520b0 0x7f47581f6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:24.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.519+0000 7f4756d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47581520b0 0x7f47581f6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57930/0 (socket says 172.21.15.55:57930) 2024-03-19T01:55:24.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.519+0000 7f4756d9d640 1 -- 172.21.15.55:0/2601306435 learned_addr learned my addr 172.21.15.55:0/2601306435 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:24.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.523+0000 7f4755d9b640 1 --2- 172.21.15.55:0/2601306435 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f47580c9000 0x7f47581f69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:24.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.527+0000 7f4756d9d640 1 -- 172.21.15.55:0/2601306435 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f47580c9000 msgr2=0x7f47581f69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:24.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.527+0000 7f4756d9d640 1 --2- 172.21.15.55:0/2601306435 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f47580c9000 0x7f47581f69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:24.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.527+0000 7f4756d9d640 1 -- 172.21.15.55:0/2601306435 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f47580c2ad0 msgr2=0x7f47580cdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:24.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.527+0000 7f4756d9d640 1 --2- 172.21.15.55:0/2601306435 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f47580c2ad0 0x7f47580cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:24.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.527+0000 7f4756d9d640 1 -- 172.21.15.55:0/2601306435 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f475806cb40 con 0x7f47581520b0 2024-03-19T01:55:24.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.527+0000 7f4756d9d640 1 --2- 172.21.15.55:0/2601306435 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47581520b0 0x7f47581f6f00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f475000c770 tx=0x7f475000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:24.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.531+0000 7f4746ffd640 1 -- 172.21.15.55:0/2601306435 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4750003b80 con 0x7f47581520b0 2024-03-19T01:55:24.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.535+0000 7f4746ffd640 1 -- 172.21.15.55:0/2601306435 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4750003d20 con 0x7f47581520b0 2024-03-19T01:55:24.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.535+0000 7f4746ffd640 1 -- 172.21.15.55:0/2601306435 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475000aa90 con 0x7f47581520b0 2024-03-19T01:55:24.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.535+0000 7f475c9f6640 1 -- 172.21.15.55:0/2601306435 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f475806c860 con 0x7f47581520b0 2024-03-19T01:55:24.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.535+0000 7f475c9f6640 1 -- 172.21.15.55:0/2601306435 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f475815af10 con 0x7f47581520b0 2024-03-19T01:55:24.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.535+0000 7f475c9f6640 1 -- 172.21.15.55:0/2601306435 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4758068a40 con 0x7f47581520b0 2024-03-19T01:55:24.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.567+0000 7f4746ffd640 1 -- 172.21.15.55:0/2601306435 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f475000ac30 con 0x7f47581520b0 2024-03-19T01:55:24.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.567+0000 7f4746ffd640 1 --2- 172.21.15.55:0/2601306435 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f47380676b0 0x7f4738069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:24.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.567+0000 7f475659c640 1 --2- 172.21.15.55:0/2601306435 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f47380676b0 0x7f4738069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:24.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.567+0000 7f475659c640 1 --2- 172.21.15.55:0/2601306435 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f47380676b0 0x7f4738069b70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f47480ad000 tx=0x7f4748004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:24.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.571+0000 7f4746ffd640 1 -- 172.21.15.55:0/2601306435 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f47500547f0 con 0x7f47581520b0 2024-03-19T01:55:24.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:24.879+0000 7f4746ffd640 1 -- 172.21.15.55:0/2601306435 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4750056e50 con 0x7f47581520b0 2024-03-19T01:55:25.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:25.119+0000 7f475c9f6640 1 -- 172.21.15.55:0/2601306435 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f475814f970 con 0x7f47581520b0 2024-03-19T01:55:25.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:25.179+0000 7f4746ffd640 1 -- 172.21.15.55:0/2601306435 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7f4750056750 con 0x7f47581520b0 2024-03-19T01:55:25.177 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:25.177 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:55:25.177 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:55:25.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:25.183+0000 7f475c9f6640 1 -- 172.21.15.55:0/2601306435 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f47380676b0 msgr2=0x7f4738069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:25.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:25.183+0000 7f475c9f6640 1 --2- 172.21.15.55:0/2601306435 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f47380676b0 0x7f4738069b70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f47480ad000 tx=0x7f4748004740 comp rx=0 tx=0).stop 2024-03-19T01:55:25.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:25.183+0000 7f475c9f6640 1 -- 172.21.15.55:0/2601306435 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47581520b0 msgr2=0x7f47581f6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:25.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:25.183+0000 7f475c9f6640 1 --2- 172.21.15.55:0/2601306435 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47581520b0 0x7f47581f6f00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f475000c770 tx=0x7f475000cc30 comp rx=0 tx=0).stop 2024-03-19T01:55:25.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:25.183+0000 7f475c9f6640 1 -- 172.21.15.55:0/2601306435 shutdown_connections 2024-03-19T01:55:25.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:25.183+0000 7f475c9f6640 1 --2- 172.21.15.55:0/2601306435 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f47380676b0 0x7f4738069b70 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:25.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:25.183+0000 7f475c9f6640 1 --2- 172.21.15.55:0/2601306435 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f47581520b0 0x7f47581f6f00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:25.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:25.183+0000 7f475c9f6640 1 --2- 172.21.15.55:0/2601306435 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f47580c9000 0x7f47581f69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:25.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:25.183+0000 7f475c9f6640 1 --2- 172.21.15.55:0/2601306435 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f47580c2ad0 0x7f47580cdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:25.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:25.183+0000 7f475c9f6640 1 -- 172.21.15.55:0/2601306435 >> 172.21.15.55:0/2601306435 conn(0x7f4758067600 msgr2=0x7f4758151040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:25.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:25.183+0000 7f475c9f6640 1 -- 172.21.15.55:0/2601306435 shutdown_connections 2024-03-19T01:55:25.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:25.183+0000 7f475c9f6640 1 -- 172.21.15.55:0/2601306435 wait complete. 2024-03-19T01:55:25.802 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:55:25.802 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:26.803 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:26.916 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.919+0000 7f37bdd39640 1 Processor -- start 2024-03-19T01:55:26.917 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.919+0000 7f37bdd39640 1 -- start start 2024-03-19T01:55:26.917 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.919+0000 7f37bdd39640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f37b806b2a0 0x7f37b8073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:26.917 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.919+0000 7f37bdd39640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f37b80c2760 0x7f37b80c2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:26.917 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.919+0000 7f37bdd39640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37b80c22b0 con 0x7f37b8154460 2024-03-19T01:55:26.917 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.919+0000 7f37bdd39640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f37b815a230 con 0x7f37b80c2760 2024-03-19T01:55:26.917 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.919+0000 7f37bdd39640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f37b80c3100 con 0x7f37b806b2a0 2024-03-19T01:55:26.918 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.923+0000 7f37b77fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f37b8154460 0x7f37b80cced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:33318/0 (socket says 172.21.15.55:33318) 2024-03-19T01:55:26.918 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.923+0000 7f37b77fe640 1 -- 172.21.15.55:0/3460606330 learned_addr learned my addr 172.21.15.55:0/3460606330 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:55:26.919 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.923+0000 7f37b6ffd640 1 --2- 172.21.15.55:0/3460606330 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f37b806b2a0 0x7f37b8073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:26.919 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.923+0000 7f37b7fff640 1 --2- 172.21.15.55:0/3460606330 >> v2:172.21.15.84:3300/0 conn(0x7f37b80c2760 0x7f37b80c2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:26.922 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.927+0000 7f37b67fc640 1 -- 172.21.15.55:0/3460606330 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1610526850 0 0) 0x7f37b80c22b0 con 0x7f37b8154460 2024-03-19T01:55:26.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.927+0000 7f37b67fc640 1 -- 172.21.15.55:0/3460606330 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37a0003530 con 0x7f37b8154460 2024-03-19T01:55:26.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.927+0000 7f37b6ffd640 1 -- 172.21.15.55:0/3460606330 >> v2:172.21.15.84:3300/0 conn(0x7f37b80c2760 msgr2=0x7f37b80c2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:26.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.927+0000 7f37b6ffd640 1 --2- 172.21.15.55:0/3460606330 >> v2:172.21.15.84:3300/0 conn(0x7f37b80c2760 0x7f37b80c2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:26.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.927+0000 7f37b6ffd640 1 -- 172.21.15.55:0/3460606330 >> v1:172.21.15.84:6789/0 conn(0x7f37b8154460 legacy=0x7f37b80cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:26.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.927+0000 7f37b6ffd640 1 -- 172.21.15.55:0/3460606330 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37b815a490 con 0x7f37b806b2a0 2024-03-19T01:55:26.926 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.931+0000 7f37b6ffd640 1 --2- 172.21.15.55:0/3460606330 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f37b806b2a0 0x7f37b8073540 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f37ac0097c0 tx=0x7f37ac01ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df3bcf62226ccf72 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:26.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.935+0000 7f37b67fc640 1 -- 172.21.15.55:0/3460606330 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ac0b5070 con 0x7f37b806b2a0 2024-03-19T01:55:26.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.935+0000 7f37b67fc640 1 -- 172.21.15.55:0/3460606330 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37ac0a8990 con 0x7f37b806b2a0 2024-03-19T01:55:26.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.935+0000 7f37b67fc640 1 -- 172.21.15.55:0/3460606330 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ac0a8d60 con 0x7f37b806b2a0 2024-03-19T01:55:26.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.935+0000 7f37bdd39640 1 -- 172.21.15.55:0/3460606330 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f37b806b2a0 msgr2=0x7f37b8073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:26.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.935+0000 7f37bdd39640 1 --2- 172.21.15.55:0/3460606330 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f37b806b2a0 0x7f37b8073540 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f37ac0097c0 tx=0x7f37ac01ec90 comp rx=0 tx=0).stop 2024-03-19T01:55:26.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.935+0000 7f37bdd39640 1 -- 172.21.15.55:0/3460606330 shutdown_connections 2024-03-19T01:55:26.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.935+0000 7f37bdd39640 1 --2- 172.21.15.55:0/3460606330 >> v2:172.21.15.84:3300/0 conn(0x7f37b80c2760 0x7f37b80c2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:26.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.935+0000 7f37bdd39640 1 --2- 172.21.15.55:0/3460606330 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f37b806b2a0 0x7f37b8073540 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:26.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.935+0000 7f37bdd39640 1 -- 172.21.15.55:0/3460606330 >> 172.21.15.55:0/3460606330 conn(0x7f37b806e670 msgr2=0x7f37b806ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:26.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.935+0000 7f37bdd39640 1 -- 172.21.15.55:0/3460606330 shutdown_connections 2024-03-19T01:55:26.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.935+0000 7f37bdd39640 1 -- 172.21.15.55:0/3460606330 wait complete. 2024-03-19T01:55:26.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.939+0000 7f37bdd39640 1 Processor -- start 2024-03-19T01:55:26.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.939+0000 7f37bdd39640 1 -- start start 2024-03-19T01:55:26.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.939+0000 7f37bdd39640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f37b806b2a0 0x7f37b81518c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:26.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.939+0000 7f37bdd39640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f37b80c2760 0x7f37b80c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:26.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.939+0000 7f37bdd39640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f37b8154460 0x7f37b80c6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:26.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.939+0000 7f37bdd39640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f37b815ad30 con 0x7f37b8154460 2024-03-19T01:55:26.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.939+0000 7f37bdd39640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f37b815aa50 con 0x7f37b806b2a0 2024-03-19T01:55:26.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.939+0000 7f37bdd39640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f37b815b010 con 0x7f37b80c2760 2024-03-19T01:55:26.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.939+0000 7f37b7fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f37b8154460 0x7f37b80c6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:26.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.939+0000 7f37b77fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f37b806b2a0 0x7f37b81518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:26.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.939+0000 7f37b6ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f37b80c2760 0x7f37b80c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:26.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.939+0000 7f37b7fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f37b8154460 0x7f37b80c6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57940/0 (socket says 172.21.15.55:57940) 2024-03-19T01:55:26.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.939+0000 7f37b7fff640 1 -- 172.21.15.55:0/3132342930 learned_addr learned my addr 172.21.15.55:0/3132342930 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:26.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.943+0000 7f37b7fff640 1 -- 172.21.15.55:0/3132342930 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f37b80c2760 msgr2=0x7f37b80c66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:26.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.943+0000 7f37b7fff640 1 --2- 172.21.15.55:0/3132342930 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f37b80c2760 0x7f37b80c66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:26.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.943+0000 7f37b7fff640 1 -- 172.21.15.55:0/3132342930 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f37b806b2a0 msgr2=0x7f37b81518c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:26.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.943+0000 7f37b7fff640 1 --2- 172.21.15.55:0/3132342930 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f37b806b2a0 0x7f37b81518c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:26.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.943+0000 7f37b7fff640 1 -- 172.21.15.55:0/3132342930 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37b81f3630 con 0x7f37b8154460 2024-03-19T01:55:26.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.947+0000 7f37b7fff640 1 --2- 172.21.15.55:0/3132342930 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f37b8154460 0x7f37b80c6bf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f37b000cd50 tx=0x7f37b000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:26.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.951+0000 7f379ffff640 1 -- 172.21.15.55:0/3132342930 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b0003ba0 con 0x7f37b8154460 2024-03-19T01:55:26.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.951+0000 7f379ffff640 1 -- 172.21.15.55:0/3132342930 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37b0003d40 con 0x7f37b8154460 2024-03-19T01:55:26.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.951+0000 7f379ffff640 1 -- 172.21.15.55:0/3132342930 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b0004240 con 0x7f37b8154460 2024-03-19T01:55:26.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.951+0000 7f37bdd39640 1 -- 172.21.15.55:0/3132342930 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37b80c7f10 con 0x7f37b8154460 2024-03-19T01:55:26.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.951+0000 7f37bdd39640 1 -- 172.21.15.55:0/3132342930 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37b80c84d0 con 0x7f37b8154460 2024-03-19T01:55:26.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:26.951+0000 7f37bdd39640 1 -- 172.21.15.55:0/3132342930 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37b81f3800 con 0x7f37b8154460 2024-03-19T01:55:27.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:27.255+0000 7f379ffff640 1 -- 172.21.15.55:0/3132342930 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f37b00043e0 con 0x7f37b8154460 2024-03-19T01:55:27.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:27.255+0000 7f379ffff640 1 --2- 172.21.15.55:0/3132342930 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f37a40676b0 0x7f37a4069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:27.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:27.259+0000 7f37b77fe640 1 --2- 172.21.15.55:0/3132342930 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f37a40676b0 0x7f37a4069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:27.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:27.259+0000 7f37b77fe640 1 --2- 172.21.15.55:0/3132342930 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f37a40676b0 0x7f37a4069b70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f37a8007fd0 tx=0x7f37a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:27.278 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:27.279+0000 7f379ffff640 1 -- 172.21.15.55:0/3132342930 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f37b0054cb0 con 0x7f37b8154460 2024-03-19T01:55:27.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:27.283+0000 7f379ffff640 1 -- 172.21.15.55:0/3132342930 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f37b0056c10 con 0x7f37b8154460 2024-03-19T01:55:27.605 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:27.607+0000 7f37bdd39640 1 -- 172.21.15.55:0/3132342930 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37b8068a40 con 0x7f37b8154460 2024-03-19T01:55:27.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:27.631+0000 7f379ffff640 1 -- 172.21.15.55:0/3132342930 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7f37b005acb0 con 0x7f37b8154460 2024-03-19T01:55:27.630 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:27.630 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:55:27.630 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:55:27.632 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:27.635+0000 7f37bdd39640 1 -- 172.21.15.55:0/3132342930 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f37a40676b0 msgr2=0x7f37a4069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:27.632 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:27.635+0000 7f37bdd39640 1 --2- 172.21.15.55:0/3132342930 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f37a40676b0 0x7f37a4069b70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f37a8007fd0 tx=0x7f37a8009040 comp rx=0 tx=0).stop 2024-03-19T01:55:27.632 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:27.635+0000 7f37bdd39640 1 -- 172.21.15.55:0/3132342930 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f37b8154460 msgr2=0x7f37b80c6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:27.632 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:27.635+0000 7f37bdd39640 1 --2- 172.21.15.55:0/3132342930 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f37b8154460 0x7f37b80c6bf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f37b000cd50 tx=0x7f37b000a660 comp rx=0 tx=0).stop 2024-03-19T01:55:27.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:27.635+0000 7f37bdd39640 1 -- 172.21.15.55:0/3132342930 shutdown_connections 2024-03-19T01:55:27.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:27.635+0000 7f37bdd39640 1 --2- 172.21.15.55:0/3132342930 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f37a40676b0 0x7f37a4069b70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:27.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:27.635+0000 7f37bdd39640 1 --2- 172.21.15.55:0/3132342930 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f37b8154460 0x7f37b80c6bf0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:27.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:27.635+0000 7f37bdd39640 1 --2- 172.21.15.55:0/3132342930 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f37b80c2760 0x7f37b80c66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:27.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:27.635+0000 7f37bdd39640 1 --2- 172.21.15.55:0/3132342930 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f37b806b2a0 0x7f37b81518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:27.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:27.635+0000 7f37bdd39640 1 -- 172.21.15.55:0/3132342930 >> 172.21.15.55:0/3132342930 conn(0x7f37b806e670 msgr2=0x7f37b80cceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:27.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:27.635+0000 7f37bdd39640 1 -- 172.21.15.55:0/3132342930 shutdown_connections 2024-03-19T01:55:27.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:27.635+0000 7f37bdd39640 1 -- 172.21.15.55:0/3132342930 wait complete. 2024-03-19T01:55:28.251 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:55:28.252 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:29.253 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:29.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.371+0000 7f062656f640 1 Processor -- start 2024-03-19T01:55:29.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.371+0000 7f062656f640 1 -- start start 2024-03-19T01:55:29.369 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.371+0000 7f062656f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f06200c96b0 0x7f0620070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:29.369 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.371+0000 7f062656f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f062015e3a0 con 0x7f06201540c0 2024-03-19T01:55:29.369 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.371+0000 7f062656f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f062015e680 con 0x7f06200c9a90 2024-03-19T01:55:29.369 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.371+0000 7f062656f640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f06200c7ad0 con 0x7f06200c96b0 2024-03-19T01:55:29.369 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.371+0000 7f0624b0c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f06201540c0 0x7f062015d810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:57202/0 (socket says 172.21.15.55:57202) 2024-03-19T01:55:29.369 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.371+0000 7f0624b0c640 1 -- 172.21.15.55:0/3037524692 learned_addr learned my addr 172.21.15.55:0/3037524692 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:55:29.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.371+0000 7f061ffff640 1 --2- 172.21.15.55:0/3037524692 >> v2:172.21.15.84:3300/0 conn(0x7f06200c96b0 0x7f0620070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:29.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.375+0000 7f061effd640 1 -- 172.21.15.55:0/3037524692 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4115733749 0 0) 0x7f062015e3a0 con 0x7f06201540c0 2024-03-19T01:55:29.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.375+0000 7f061effd640 1 -- 172.21.15.55:0/3037524692 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0608003530 con 0x7f06201540c0 2024-03-19T01:55:29.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.379+0000 7f061effd640 1 -- 172.21.15.55:0/3037524692 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1991074223 0 0) 0x7f0608003530 con 0x7f06201540c0 2024-03-19T01:55:29.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.379+0000 7f061effd640 1 -- 172.21.15.55:0/3037524692 >> v2:172.21.15.84:3300/0 conn(0x7f06200c96b0 msgr2=0x7f0620070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:29.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.379+0000 7f061effd640 1 --2- 172.21.15.55:0/3037524692 >> v2:172.21.15.84:3300/0 conn(0x7f06200c96b0 0x7f0620070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:29.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.379+0000 7f061effd640 1 -- 172.21.15.55:0/3037524692 >> v1:172.21.15.84:6789/0 conn(0x7f06200c9a90 legacy=0x7f06200c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:29.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.379+0000 7f061effd640 1 -- 172.21.15.55:0/3037524692 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f062015a330 con 0x7f06201540c0 2024-03-19T01:55:29.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.379+0000 7f061effd640 1 -- 172.21.15.55:0/3037524692 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f0618001a00 con 0x7f06201540c0 2024-03-19T01:55:29.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.379+0000 7f061effd640 1 -- 172.21.15.55:0/3037524692 >> v1:172.21.15.55:6789/0 conn(0x7f06201540c0 legacy=0x7f062015d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:29.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.379+0000 7f061effd640 1 --2- 172.21.15.55:0/3037524692 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0608003a10 0x7f0608003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:29.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.379+0000 7f061effd640 1 --2- 172.21.15.55:0/3037524692 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0608004810 0x7f0608004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:29.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.379+0000 7f061effd640 1 --2- 172.21.15.55:0/3037524692 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0608005220 0x7f060800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:29.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.379+0000 7f061effd640 1 -- 172.21.15.55:0/3037524692 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f06200c7ad0 con 0x7f0608004810 2024-03-19T01:55:29.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.379+0000 7f061effd640 1 -- 172.21.15.55:0/3037524692 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f060800c170 con 0x7f0608003a10 2024-03-19T01:55:29.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.379+0000 7f061effd640 1 -- 172.21.15.55:0/3037524692 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f060800c370 con 0x7f0608005220 2024-03-19T01:55:29.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.383+0000 7f0624b0c640 1 --2- 172.21.15.55:0/3037524692 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0608005220 0x7f060800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:29.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.383+0000 7f061f7fe640 1 --2- 172.21.15.55:0/3037524692 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0608004810 0x7f0608004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:29.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.387+0000 7f0624b0c640 1 -- 172.21.15.55:0/3037524692 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0608003a10 msgr2=0x7f0608003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:29.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.387+0000 7f0624b0c640 1 --2- 172.21.15.55:0/3037524692 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0608003a10 0x7f0608003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:29.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.387+0000 7f0624b0c640 1 -- 172.21.15.55:0/3037524692 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0608004810 msgr2=0x7f0608004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:29.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.387+0000 7f0624b0c640 1 --2- 172.21.15.55:0/3037524692 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0608004810 0x7f0608004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:29.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.387+0000 7f0624b0c640 1 -- 172.21.15.55:0/3037524692 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0608003530 con 0x7f0608005220 2024-03-19T01:55:29.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.387+0000 7f061f7fe640 1 --2- 172.21.15.55:0/3037524692 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0608004810 0x7f0608004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:55:29.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.387+0000 7f0624b0c640 1 -- 172.21.15.55:0/3037524692 reap_dead start 2024-03-19T01:55:29.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.391+0000 7f0624b0c640 1 --2- 172.21.15.55:0/3037524692 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0608005220 0x7f060800b9d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f061800a760 tx=0x7f061801d280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76d58c4520839d15 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:29.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.395+0000 7f061effd640 1 -- 172.21.15.55:0/3037524692 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f061800a3b0 con 0x7f0608005220 2024-03-19T01:55:29.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.395+0000 7f061effd640 1 -- 172.21.15.55:0/3037524692 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06180ac620 con 0x7f0608005220 2024-03-19T01:55:29.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.395+0000 7f061effd640 1 -- 172.21.15.55:0/3037524692 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06180aca10 con 0x7f0608005220 2024-03-19T01:55:29.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.395+0000 7f062656f640 1 -- 172.21.15.55:0/3037524692 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0608005220 msgr2=0x7f060800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:29.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.395+0000 7f062656f640 1 --2- 172.21.15.55:0/3037524692 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0608005220 0x7f060800b9d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f061800a760 tx=0x7f061801d280 comp rx=0 tx=0).stop 2024-03-19T01:55:29.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.399+0000 7f062656f640 1 -- 172.21.15.55:0/3037524692 shutdown_connections 2024-03-19T01:55:29.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.399+0000 7f062656f640 1 --2- 172.21.15.55:0/3037524692 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0608005220 0x7f060800b9d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:29.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.399+0000 7f062656f640 1 -- 172.21.15.55:0/3037524692 >> 172.21.15.55:0/3037524692 conn(0x7f06200cc9c0 msgr2=0x7f06200ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:29.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.399+0000 7f062656f640 1 -- 172.21.15.55:0/3037524692 shutdown_connections 2024-03-19T01:55:29.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.399+0000 7f062656f640 1 -- 172.21.15.55:0/3037524692 wait complete. 2024-03-19T01:55:29.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.399+0000 7f062656f640 1 Processor -- start 2024-03-19T01:55:29.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.399+0000 7f062656f640 1 -- start start 2024-03-19T01:55:29.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.399+0000 7f062656f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0608005220 0x7f06200c36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:29.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.399+0000 7f062656f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f062014efb0 0x7f062014f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:29.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.399+0000 7f062656f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0620141ce0 0x7f0620142160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:29.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.399+0000 7f062656f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f062015b0f0 con 0x7f0608005220 2024-03-19T01:55:29.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.399+0000 7f062656f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f0620154920 con 0x7f062014efb0 2024-03-19T01:55:29.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.399+0000 7f062656f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f0620070f20 con 0x7f0620141ce0 2024-03-19T01:55:29.398 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.399+0000 7f0624b0c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0620141ce0 0x7f0620142160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:29.398 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.399+0000 7f061ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0608005220 0x7f06200c36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:29.398 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.399+0000 7f061f7fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f062014efb0 0x7f062014f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:29.398 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.399+0000 7f0624b0c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0620141ce0 0x7f0620142160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:56990/0 (socket says 172.21.15.55:56990) 2024-03-19T01:55:29.398 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.399+0000 7f0624b0c640 1 -- 172.21.15.55:0/3235753641 learned_addr learned my addr 172.21.15.55:0/3235753641 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:29.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.407+0000 7f0624b0c640 1 -- 172.21.15.55:0/3235753641 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f062014efb0 msgr2=0x7f062014f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:29.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.407+0000 7f0624b0c640 1 --2- 172.21.15.55:0/3235753641 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f062014efb0 0x7f062014f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:29.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.407+0000 7f0624b0c640 1 -- 172.21.15.55:0/3235753641 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0608005220 msgr2=0x7f06200c36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:29.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.407+0000 7f0624b0c640 1 --2- 172.21.15.55:0/3235753641 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0608005220 0x7f06200c36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:29.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.407+0000 7f0624b0c640 1 -- 172.21.15.55:0/3235753641 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06200c9000 con 0x7f0620141ce0 2024-03-19T01:55:29.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.407+0000 7f061ffff640 1 --2- 172.21.15.55:0/3235753641 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0608005220 0x7f06200c36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:55:29.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.411+0000 7f0624b0c640 1 --2- 172.21.15.55:0/3235753641 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0620141ce0 0x7f0620142160 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f061801db20 tx=0x7f0608005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:29.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.419+0000 7f061cff9640 1 -- 172.21.15.55:0/3235753641 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0618001740 con 0x7f0620141ce0 2024-03-19T01:55:29.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.419+0000 7f061cff9640 1 -- 172.21.15.55:0/3235753641 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06180018e0 con 0x7f0620141ce0 2024-03-19T01:55:29.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.419+0000 7f062656f640 1 -- 172.21.15.55:0/3235753641 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f062014f950 con 0x7f0620141ce0 2024-03-19T01:55:29.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.419+0000 7f061cff9640 1 -- 172.21.15.55:0/3235753641 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06180ac9f0 con 0x7f0620141ce0 2024-03-19T01:55:29.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.419+0000 7f062656f640 1 -- 172.21.15.55:0/3235753641 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0620144fe0 con 0x7f0620141ce0 2024-03-19T01:55:29.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.419+0000 7f062656f640 1 -- 172.21.15.55:0/3235753641 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0620068a40 con 0x7f0620141ce0 2024-03-19T01:55:29.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.459+0000 7f061cff9640 1 -- 172.21.15.55:0/3235753641 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f06180a5d10 con 0x7f0620141ce0 2024-03-19T01:55:29.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.459+0000 7f061cff9640 1 --2- 172.21.15.55:0/3235753641 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f060c067690 0x7f060c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:29.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.459+0000 7f061ffff640 1 --2- 172.21.15.55:0/3235753641 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f060c067690 0x7f060c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:29.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.459+0000 7f061ffff640 1 --2- 172.21.15.55:0/3235753641 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f060c067690 0x7f060c069b50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0610009e30 tx=0x7f0610009320 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:29.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.475+0000 7f061cff9640 1 -- 172.21.15.55:0/3235753641 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f06180eddb0 con 0x7f0620141ce0 2024-03-19T01:55:29.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.783+0000 7f061cff9640 1 -- 172.21.15.55:0/3235753641 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f06180be060 con 0x7f0620141ce0 2024-03-19T01:55:29.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:29.995+0000 7f062656f640 1 -- 172.21.15.55:0/3235753641 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06200c7d80 con 0x7f0620141ce0 2024-03-19T01:55:30.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:30.015+0000 7f061cff9640 1 -- 172.21.15.55:0/3235753641 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7f0618016f90 con 0x7f0620141ce0 2024-03-19T01:55:30.014 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:30.014 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:55:30.015 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:55:30.016 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:30.019+0000 7f062656f640 1 -- 172.21.15.55:0/3235753641 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f060c067690 msgr2=0x7f060c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:30.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:30.019+0000 7f062656f640 1 --2- 172.21.15.55:0/3235753641 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f060c067690 0x7f060c069b50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0610009e30 tx=0x7f0610009320 comp rx=0 tx=0).stop 2024-03-19T01:55:30.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:30.019+0000 7f062656f640 1 -- 172.21.15.55:0/3235753641 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0620141ce0 msgr2=0x7f0620142160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:30.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:30.019+0000 7f062656f640 1 --2- 172.21.15.55:0/3235753641 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0620141ce0 0x7f0620142160 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f061801db20 tx=0x7f0608005840 comp rx=0 tx=0).stop 2024-03-19T01:55:30.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:30.019+0000 7f062656f640 1 -- 172.21.15.55:0/3235753641 shutdown_connections 2024-03-19T01:55:30.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:30.019+0000 7f062656f640 1 --2- 172.21.15.55:0/3235753641 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f060c067690 0x7f060c069b50 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:30.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:30.019+0000 7f062656f640 1 --2- 172.21.15.55:0/3235753641 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0620141ce0 0x7f0620142160 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:30.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:30.019+0000 7f062656f640 1 --2- 172.21.15.55:0/3235753641 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f062014efb0 0x7f062014f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:30.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:30.019+0000 7f062656f640 1 --2- 172.21.15.55:0/3235753641 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0608005220 0x7f06200c36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:30.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:30.019+0000 7f062656f640 1 -- 172.21.15.55:0/3235753641 >> 172.21.15.55:0/3235753641 conn(0x7f06200cc9c0 msgr2=0x7f06200c73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:30.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:30.019+0000 7f062656f640 1 -- 172.21.15.55:0/3235753641 shutdown_connections 2024-03-19T01:55:30.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:30.019+0000 7f062656f640 1 -- 172.21.15.55:0/3235753641 wait complete. 2024-03-19T01:55:30.637 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:55:30.637 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:31.639 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:31.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.771+0000 7fe5ca9ad640 1 Processor -- start 2024-03-19T01:55:31.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.771+0000 7fe5ca9ad640 1 -- start start 2024-03-19T01:55:31.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.771+0000 7fe5ca9ad640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe5c4070780 0x7fe5c415b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:31.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.771+0000 7fe5ca9ad640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5c406fff0 con 0x7fe5c4152120 2024-03-19T01:55:31.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.771+0000 7fe5ca9ad640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5c40c39c0 con 0x7fe5c415bb00 2024-03-19T01:55:31.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.771+0000 7fe5ca9ad640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fe5c4152500 con 0x7fe5c4070780 2024-03-19T01:55:31.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.775+0000 7fe5c3fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fe5c4152120 0x7fe5c40c90d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:57204/0 (socket says 172.21.15.55:57204) 2024-03-19T01:55:31.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.775+0000 7fe5c3fff640 1 -- 172.21.15.55:0/2454141237 learned_addr learned my addr 172.21.15.55:0/2454141237 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:55:31.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.775+0000 7fe5c37fe640 1 --2- 172.21.15.55:0/2454141237 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe5c4070780 0x7fe5c415b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:31.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.775+0000 7fe5c2ffd640 1 -- 172.21.15.55:0/2454141237 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2982532985 0 0) 0x7fe5c406fff0 con 0x7fe5c4152120 2024-03-19T01:55:31.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.775+0000 7fe5c2ffd640 1 -- 172.21.15.55:0/2454141237 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5ac003530 con 0x7fe5c4152120 2024-03-19T01:55:31.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.779+0000 7fe5c37fe640 1 -- 172.21.15.55:0/2454141237 >> v1:172.21.15.84:6789/0 conn(0x7fe5c415bb00 legacy=0x7fe5c415bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:31.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.779+0000 7fe5c37fe640 1 -- 172.21.15.55:0/2454141237 >> v1:172.21.15.55:6789/0 conn(0x7fe5c4152120 legacy=0x7fe5c40c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:31.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.779+0000 7fe5c37fe640 1 -- 172.21.15.55:0/2454141237 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c40702b0 con 0x7fe5c4070780 2024-03-19T01:55:31.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.779+0000 7fe5c37fe640 1 --2- 172.21.15.55:0/2454141237 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe5c4070780 0x7fe5c415b5c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe5b80097f0 tx=0x7fe5b801c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=585b126aa4c17529 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:31.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.787+0000 7fe5c2ffd640 1 -- 172.21.15.55:0/2454141237 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b80b0070 con 0x7fe5c4070780 2024-03-19T01:55:31.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.787+0000 7fe5c2ffd640 1 -- 172.21.15.55:0/2454141237 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5b80ab600 con 0x7fe5c4070780 2024-03-19T01:55:31.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.787+0000 7fe5c2ffd640 1 -- 172.21.15.55:0/2454141237 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b80ab9d0 con 0x7fe5c4070780 2024-03-19T01:55:31.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.787+0000 7fe5ca9ad640 1 -- 172.21.15.55:0/2454141237 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe5c4070780 msgr2=0x7fe5c415b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:31.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.787+0000 7fe5ca9ad640 1 --2- 172.21.15.55:0/2454141237 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe5c4070780 0x7fe5c415b5c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe5b80097f0 tx=0x7fe5b801c4e0 comp rx=0 tx=0).stop 2024-03-19T01:55:31.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.787+0000 7fe5ca9ad640 1 -- 172.21.15.55:0/2454141237 shutdown_connections 2024-03-19T01:55:31.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.787+0000 7fe5ca9ad640 1 --2- 172.21.15.55:0/2454141237 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe5c4070780 0x7fe5c415b5c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:31.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.787+0000 7fe5ca9ad640 1 -- 172.21.15.55:0/2454141237 >> 172.21.15.55:0/2454141237 conn(0x7fe5c4067600 msgr2=0x7fe5c4071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:31.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.787+0000 7fe5ca9ad640 1 -- 172.21.15.55:0/2454141237 shutdown_connections 2024-03-19T01:55:31.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.787+0000 7fe5ca9ad640 1 -- 172.21.15.55:0/2454141237 wait complete. 2024-03-19T01:55:31.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.791+0000 7fe5ca9ad640 1 Processor -- start 2024-03-19T01:55:31.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.791+0000 7fe5ca9ad640 1 -- start start 2024-03-19T01:55:31.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.791+0000 7fe5ca9ad640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe5c4070780 0x7fe5c414bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:31.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.791+0000 7fe5ca9ad640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe5c4152120 0x7fe5c41f7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:31.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.791+0000 7fe5ca9ad640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5c415bb00 0x7fe5c4157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:31.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.791+0000 7fe5ca9ad640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c40c97e0 con 0x7fe5c415bb00 2024-03-19T01:55:31.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.791+0000 7fe5ca9ad640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c40c3530 con 0x7fe5c4152120 2024-03-19T01:55:31.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.791+0000 7fe5ca9ad640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fe5c40c86d0 con 0x7fe5c4070780 2024-03-19T01:55:31.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.791+0000 7fe5c8f4a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5c415bb00 0x7fe5c4157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:31.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.791+0000 7fe5c3fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe5c4070780 0x7fe5c414bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:31.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.791+0000 7fe5c8f4a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5c415bb00 0x7fe5c4157fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:36390/0 (socket says 172.21.15.55:36390) 2024-03-19T01:55:31.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.791+0000 7fe5c8f4a640 1 -- 172.21.15.55:0/1158458300 learned_addr learned my addr 172.21.15.55:0/1158458300 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:31.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.791+0000 7fe5c37fe640 1 --2- 172.21.15.55:0/1158458300 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe5c4152120 0x7fe5c41f7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:31.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.795+0000 7fe5c8f4a640 1 -- 172.21.15.55:0/1158458300 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe5c4070780 msgr2=0x7fe5c414bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:31.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.795+0000 7fe5c8f4a640 1 --2- 172.21.15.55:0/1158458300 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe5c4070780 0x7fe5c414bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:31.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.795+0000 7fe5c8f4a640 1 -- 172.21.15.55:0/1158458300 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe5c4152120 msgr2=0x7fe5c41f7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:31.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.795+0000 7fe5c8f4a640 1 --2- 172.21.15.55:0/1158458300 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe5c4152120 0x7fe5c41f7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:31.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.795+0000 7fe5c8f4a640 1 -- 172.21.15.55:0/1158458300 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c4156f20 con 0x7fe5c415bb00 2024-03-19T01:55:31.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.799+0000 7fe5c8f4a640 1 --2- 172.21.15.55:0/1158458300 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5c415bb00 0x7fe5c4157fc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00c730 tx=0x7fe5bc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:31.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.803+0000 7fe5c0ff9640 1 -- 172.21.15.55:0/1158458300 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc003b80 con 0x7fe5c415bb00 2024-03-19T01:55:31.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.803+0000 7fe5c0ff9640 1 -- 172.21.15.55:0/1158458300 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5bc003d20 con 0x7fe5c415bb00 2024-03-19T01:55:31.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.803+0000 7fe5ca9ad640 1 -- 172.21.15.55:0/1158458300 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5c4157200 con 0x7fe5c415bb00 2024-03-19T01:55:31.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.803+0000 7fe5c0ff9640 1 -- 172.21.15.55:0/1158458300 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc00aa30 con 0x7fe5c415bb00 2024-03-19T01:55:31.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.803+0000 7fe5ca9ad640 1 -- 172.21.15.55:0/1158458300 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5c41577c0 con 0x7fe5c415bb00 2024-03-19T01:55:31.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.803+0000 7fe5ca9ad640 1 -- 172.21.15.55:0/1158458300 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5c4068a40 con 0x7fe5c415bb00 2024-03-19T01:55:31.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.835+0000 7fe5c0ff9640 1 -- 172.21.15.55:0/1158458300 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fe5bc00abd0 con 0x7fe5c415bb00 2024-03-19T01:55:31.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.835+0000 7fe5c0ff9640 1 --2- 172.21.15.55:0/1158458300 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe5b00676b0 0x7fe5b0069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:31.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.835+0000 7fe5c3fff640 1 --2- 172.21.15.55:0/1158458300 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe5b00676b0 0x7fe5b0069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:31.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:31.835+0000 7fe5c3fff640 1 --2- 172.21.15.55:0/1158458300 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe5b00676b0 0x7fe5b0069b70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe5b40011f0 tx=0x7fe5b4007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:32.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:32.103+0000 7fe5c0ff9640 1 -- 172.21.15.55:0/1158458300 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fe5bc0547a0 con 0x7fe5c415bb00 2024-03-19T01:55:32.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:32.107+0000 7fe5c0ff9640 1 -- 172.21.15.55:0/1158458300 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fe5bc056700 con 0x7fe5c415bb00 2024-03-19T01:55:32.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:32.323+0000 7fe5ca9ad640 1 -- 172.21.15.55:0/1158458300 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5c40c84b0 con 0x7fe5c415bb00 2024-03-19T01:55:32.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:32.339+0000 7fe5c0ff9640 1 -- 172.21.15.55:0/1158458300 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6230 (secure 0 0 0) 0x7fe5bc05a7a0 con 0x7fe5c415bb00 2024-03-19T01:55:32.335 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:32.335 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:53:06.221236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4209,4223,4237,4255]},"id":1}]} 2024-03-19T01:55:32.335 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 25 2024-03-19T01:55:32.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:32.339+0000 7fe5ca9ad640 1 -- 172.21.15.55:0/1158458300 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe5b00676b0 msgr2=0x7fe5b0069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:32.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:32.339+0000 7fe5ca9ad640 1 --2- 172.21.15.55:0/1158458300 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe5b00676b0 0x7fe5b0069b70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe5b40011f0 tx=0x7fe5b4007d40 comp rx=0 tx=0).stop 2024-03-19T01:55:32.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:32.339+0000 7fe5ca9ad640 1 -- 172.21.15.55:0/1158458300 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5c415bb00 msgr2=0x7fe5c4157fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:32.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:32.339+0000 7fe5ca9ad640 1 --2- 172.21.15.55:0/1158458300 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5c415bb00 0x7fe5c4157fc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00c730 tx=0x7fe5bc00cbf0 comp rx=0 tx=0).stop 2024-03-19T01:55:32.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:32.339+0000 7fe5ca9ad640 1 -- 172.21.15.55:0/1158458300 shutdown_connections 2024-03-19T01:55:32.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:32.339+0000 7fe5ca9ad640 1 --2- 172.21.15.55:0/1158458300 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe5b00676b0 0x7fe5b0069b70 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:32.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:32.339+0000 7fe5ca9ad640 1 --2- 172.21.15.55:0/1158458300 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5c415bb00 0x7fe5c4157fc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:32.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:32.339+0000 7fe5ca9ad640 1 --2- 172.21.15.55:0/1158458300 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe5c4152120 0x7fe5c41f7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:32.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:32.339+0000 7fe5ca9ad640 1 --2- 172.21.15.55:0/1158458300 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe5c4070780 0x7fe5c414bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:32.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:32.339+0000 7fe5ca9ad640 1 -- 172.21.15.55:0/1158458300 >> 172.21.15.55:0/1158458300 conn(0x7fe5c4067600 msgr2=0x7fe5c414c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:32.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:32.343+0000 7fe5ca9ad640 1 -- 172.21.15.55:0/1158458300 shutdown_connections 2024-03-19T01:55:32.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:32.343+0000 7fe5ca9ad640 1 -- 172.21.15.55:0/1158458300 wait complete. 2024-03-19T01:55:32.959 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:53:06.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4209, 4223, 4237, 4255]} 2024-03-19T01:55:32.959 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:33.960 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:34.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.083+0000 7f2c73d14640 1 Processor -- start 2024-03-19T01:55:34.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.083+0000 7f2c73d14640 1 -- start start 2024-03-19T01:55:34.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.083+0000 7f2c73d14640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2c6c14eaa0 0x7f2c6c06b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:34.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.083+0000 7f2c73d14640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6c14ee80 con 0x7f2c6c06bec0 2024-03-19T01:55:34.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.083+0000 7f2c73d14640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6c0c7540 con 0x7f2c6c1585f0 2024-03-19T01:55:34.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.083+0000 7f2c73d14640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2c6c0c77c0 con 0x7f2c6c14eaa0 2024-03-19T01:55:34.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.083+0000 7f2c712af640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f2c6c06bec0 0x7f2c6c0c60d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:57220/0 (socket says 172.21.15.55:57220) 2024-03-19T01:55:34.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.083+0000 7f2c712af640 1 -- 172.21.15.55:0/1523993094 learned_addr learned my addr 172.21.15.55:0/1523993094 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:55:34.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.083+0000 7f2c71ab0640 1 --2- 172.21.15.55:0/1523993094 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2c6c14eaa0 0x7f2c6c06b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:34.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.087+0000 7f2c70aae640 1 -- 172.21.15.55:0/1523993094 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1960921526 0 0) 0x7f2c6c14ee80 con 0x7f2c6c06bec0 2024-03-19T01:55:34.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.087+0000 7f2c70aae640 1 -- 172.21.15.55:0/1523993094 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c58003530 con 0x7f2c6c06bec0 2024-03-19T01:55:34.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.087+0000 7f2c70aae640 1 -- 172.21.15.55:0/1523993094 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 469992646 0 0) 0x7f2c640014a0 con 0x7f2c6c1585f0 2024-03-19T01:55:34.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.087+0000 7f2c70aae640 1 -- 172.21.15.55:0/1523993094 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c6c14ee80 con 0x7f2c6c1585f0 2024-03-19T01:55:34.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.091+0000 7f2c71ab0640 1 -- 172.21.15.55:0/1523993094 >> v1:172.21.15.84:6789/0 conn(0x7f2c6c1585f0 legacy=0x7f2c6c162940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:34.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.091+0000 7f2c71ab0640 1 -- 172.21.15.55:0/1523993094 >> v1:172.21.15.55:6789/0 conn(0x7f2c6c06bec0 legacy=0x7f2c6c0c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:34.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.091+0000 7f2c70aae640 1 -- 172.21.15.55:0/1523993094 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 744350491 0 0) 0x7f2c58003530 con 0x7f2c6c06bec0 2024-03-19T01:55:34.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.091+0000 7f2c71ab0640 1 -- 172.21.15.55:0/1523993094 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6c0c7aa0 con 0x7f2c6c14eaa0 2024-03-19T01:55:34.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.099+0000 7f2c71ab0640 1 --2- 172.21.15.55:0/1523993094 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2c6c14eaa0 0x7f2c6c06b980 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2c680097f0 tx=0x7f2c6801c370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef9c62324978021 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:34.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.107+0000 7f2c70aae640 1 -- 172.21.15.55:0/1523993094 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c680b0070 con 0x7f2c6c14eaa0 2024-03-19T01:55:34.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.107+0000 7f2c70aae640 1 -- 172.21.15.55:0/1523993094 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c680ab400 con 0x7f2c6c14eaa0 2024-03-19T01:55:34.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.107+0000 7f2c70aae640 1 -- 172.21.15.55:0/1523993094 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c001680 con 0x7f2c6c14eaa0 2024-03-19T01:55:34.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.107+0000 7f2c73d14640 1 -- 172.21.15.55:0/1523993094 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2c6c14eaa0 msgr2=0x7f2c6c06b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:34.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.107+0000 7f2c73d14640 1 --2- 172.21.15.55:0/1523993094 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2c6c14eaa0 0x7f2c6c06b980 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2c680097f0 tx=0x7f2c6801c370 comp rx=0 tx=0).stop 2024-03-19T01:55:34.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.107+0000 7f2c73d14640 1 -- 172.21.15.55:0/1523993094 shutdown_connections 2024-03-19T01:55:34.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.107+0000 7f2c73d14640 1 --2- 172.21.15.55:0/1523993094 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2c6c14eaa0 0x7f2c6c06b980 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:34.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.107+0000 7f2c73d14640 1 -- 172.21.15.55:0/1523993094 >> 172.21.15.55:0/1523993094 conn(0x7f2c6c0cc9c0 msgr2=0x7f2c6c0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:34.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.107+0000 7f2c73d14640 1 -- 172.21.15.55:0/1523993094 shutdown_connections 2024-03-19T01:55:34.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.107+0000 7f2c73d14640 1 -- 172.21.15.55:0/1523993094 wait complete. 2024-03-19T01:55:34.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.107+0000 7f2c73d14640 1 Processor -- start 2024-03-19T01:55:34.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.107+0000 7f2c73d14640 1 -- start start 2024-03-19T01:55:34.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.111+0000 7f2c73d14640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2c6c06bec0 0x7f2c6c14d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:34.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.111+0000 7f2c73d14640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2c6c14eaa0 0x7f2c6c0cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:34.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.111+0000 7f2c73d14640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2c6c1585f0 0x7f2c6c0cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:34.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.111+0000 7f2c73d14640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c0c67e0 con 0x7f2c6c06bec0 2024-03-19T01:55:34.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.111+0000 7f2c73d14640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c158e10 con 0x7f2c6c1585f0 2024-03-19T01:55:34.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.111+0000 7f2c73d14640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2c6c0c6960 con 0x7f2c6c14eaa0 2024-03-19T01:55:34.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.111+0000 7f2c71ab0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2c6c06bec0 0x7f2c6c14d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:34.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.111+0000 7f2c722b1640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2c6c1585f0 0x7f2c6c0cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:34.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.111+0000 7f2c712af640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2c6c14eaa0 0x7f2c6c0cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:34.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.111+0000 7f2c71ab0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2c6c06bec0 0x7f2c6c14d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:36396/0 (socket says 172.21.15.55:36396) 2024-03-19T01:55:34.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.111+0000 7f2c71ab0640 1 -- 172.21.15.55:0/30983168 learned_addr learned my addr 172.21.15.55:0/30983168 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:34.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.115+0000 7f2c71ab0640 1 -- 172.21.15.55:0/30983168 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2c6c14eaa0 msgr2=0x7f2c6c0cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:34.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.115+0000 7f2c71ab0640 1 --2- 172.21.15.55:0/30983168 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2c6c14eaa0 0x7f2c6c0cb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:34.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.115+0000 7f2c71ab0640 1 -- 172.21.15.55:0/30983168 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2c6c1585f0 msgr2=0x7f2c6c0cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:34.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.115+0000 7f2c71ab0640 1 --2- 172.21.15.55:0/30983168 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2c6c1585f0 0x7f2c6c0cbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:34.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.115+0000 7f2c71ab0640 1 -- 172.21.15.55:0/30983168 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6c140be0 con 0x7f2c6c06bec0 2024-03-19T01:55:34.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.115+0000 7f2c71ab0640 1 --2- 172.21.15.55:0/30983168 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2c6c06bec0 0x7f2c6c14d630 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2c68019fa0 tx=0x7f2c68019110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:34.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.119+0000 7f2c627fc640 1 -- 172.21.15.55:0/30983168 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c680b0040 con 0x7f2c6c06bec0 2024-03-19T01:55:34.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.119+0000 7f2c627fc640 1 -- 172.21.15.55:0/30983168 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c680bc070 con 0x7f2c6c06bec0 2024-03-19T01:55:34.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.119+0000 7f2c73d14640 1 -- 172.21.15.55:0/30983168 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c6c140ec0 con 0x7f2c6c06bec0 2024-03-19T01:55:34.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.119+0000 7f2c627fc640 1 -- 172.21.15.55:0/30983168 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c680b56a0 con 0x7f2c6c06bec0 2024-03-19T01:55:34.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.119+0000 7f2c73d14640 1 -- 172.21.15.55:0/30983168 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c6c141480 con 0x7f2c6c06bec0 2024-03-19T01:55:34.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.123+0000 7f2c73d14640 1 -- 172.21.15.55:0/30983168 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c6c0cb3e0 con 0x7f2c6c06bec0 2024-03-19T01:55:34.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.143+0000 7f2c627fc640 1 -- 172.21.15.55:0/30983168 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f2c680b3070 con 0x7f2c6c06bec0 2024-03-19T01:55:34.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.143+0000 7f2c627fc640 1 --2- 172.21.15.55:0/30983168 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2c500676b0 0x7f2c50069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:34.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.143+0000 7f2c712af640 1 --2- 172.21.15.55:0/30983168 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2c500676b0 0x7f2c50069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:34.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.143+0000 7f2c712af640 1 --2- 172.21.15.55:0/30983168 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2c500676b0 0x7f2c50069b70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c000fe0 tx=0x7f2c5c0016e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:34.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.427+0000 7f2c627fc640 1 -- 172.21.15.55:0/30983168 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f2c680ee670 con 0x7f2c6c06bec0 2024-03-19T01:55:34.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.427+0000 7f2c627fc640 1 -- 172.21.15.55:0/30983168 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2c680c6020 con 0x7f2c6c06bec0 2024-03-19T01:55:34.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.667+0000 7f2c73d14640 1 -- 172.21.15.55:0/30983168 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c6c068a40 con 0x7f2c6c06bec0 2024-03-19T01:55:34.686 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.687+0000 7f2c627fc640 1 -- 172.21.15.55:0/30983168 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f2c68015ff0 con 0x7f2c6c06bec0 2024-03-19T01:55:34.687 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:34.687 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:55:34.687 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:55:34.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.691+0000 7f2c73d14640 1 -- 172.21.15.55:0/30983168 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2c500676b0 msgr2=0x7f2c50069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:34.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.691+0000 7f2c73d14640 1 --2- 172.21.15.55:0/30983168 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2c500676b0 0x7f2c50069b70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c000fe0 tx=0x7f2c5c0016e0 comp rx=0 tx=0).stop 2024-03-19T01:55:34.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.691+0000 7f2c73d14640 1 -- 172.21.15.55:0/30983168 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2c6c06bec0 msgr2=0x7f2c6c14d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:34.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.691+0000 7f2c73d14640 1 --2- 172.21.15.55:0/30983168 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2c6c06bec0 0x7f2c6c14d630 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2c68019fa0 tx=0x7f2c68019110 comp rx=0 tx=0).stop 2024-03-19T01:55:34.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.691+0000 7f2c73d14640 1 -- 172.21.15.55:0/30983168 shutdown_connections 2024-03-19T01:55:34.691 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.691+0000 7f2c73d14640 1 --2- 172.21.15.55:0/30983168 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2c500676b0 0x7f2c50069b70 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:34.691 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.691+0000 7f2c73d14640 1 --2- 172.21.15.55:0/30983168 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2c6c1585f0 0x7f2c6c0cbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:34.691 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.691+0000 7f2c73d14640 1 --2- 172.21.15.55:0/30983168 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2c6c14eaa0 0x7f2c6c0cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:34.691 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.691+0000 7f2c73d14640 1 --2- 172.21.15.55:0/30983168 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2c6c06bec0 0x7f2c6c14d630 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:34.691 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.691+0000 7f2c73d14640 1 -- 172.21.15.55:0/30983168 >> 172.21.15.55:0/30983168 conn(0x7f2c6c0cc9c0 msgr2=0x7f2c6c14a660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:34.691 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.691+0000 7f2c73d14640 1 -- 172.21.15.55:0/30983168 shutdown_connections 2024-03-19T01:55:34.691 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:34.691+0000 7f2c73d14640 1 -- 172.21.15.55:0/30983168 wait complete. 2024-03-19T01:55:35.310 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:55:35.310 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:36.311 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:36.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.427+0000 7fc59ab29640 1 Processor -- start 2024-03-19T01:55:36.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.427+0000 7fc59ab29640 1 -- start start 2024-03-19T01:55:36.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.427+0000 7fc59ab29640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fc5941544f0 0x7fc594073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:36.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.427+0000 7fc59ab29640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc59415acb0 0x7fc5940cce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:36.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.427+0000 7fc59ab29640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5940c7fa0 con 0x7fc59406b2a0 2024-03-19T01:55:36.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.427+0000 7fc59ab29640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fc59415b210 con 0x7fc5941544f0 2024-03-19T01:55:36.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.427+0000 7fc59ab29640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc5940c8240 con 0x7fc59415acb0 2024-03-19T01:55:36.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.431+0000 7fc5990c6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc59415acb0 0x7fc5940cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:36.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.431+0000 7fc593fff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fc59406b2a0 0x7fc59415e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:43558/0 (socket says 172.21.15.55:43558) 2024-03-19T01:55:36.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.431+0000 7fc593fff640 1 -- 172.21.15.55:0/3276705152 learned_addr learned my addr 172.21.15.55:0/3276705152 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:55:36.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.431+0000 7fc5988c5640 1 --2- 172.21.15.55:0/3276705152 >> v2:172.21.15.84:3300/0 conn(0x7fc5941544f0 0x7fc594073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:36.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.435+0000 7fc5937fe640 1 -- 172.21.15.55:0/3276705152 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3253548309 0 0) 0x7fc5940c7fa0 con 0x7fc59406b2a0 2024-03-19T01:55:36.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.435+0000 7fc5937fe640 1 -- 172.21.15.55:0/3276705152 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc57c003530 con 0x7fc59406b2a0 2024-03-19T01:55:36.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.435+0000 7fc5990c6640 1 -- 172.21.15.55:0/3276705152 >> v2:172.21.15.84:3300/0 conn(0x7fc5941544f0 msgr2=0x7fc594073540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:36.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.435+0000 7fc5990c6640 1 --2- 172.21.15.55:0/3276705152 >> v2:172.21.15.84:3300/0 conn(0x7fc5941544f0 0x7fc594073540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:36.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.435+0000 7fc5990c6640 1 -- 172.21.15.55:0/3276705152 >> v1:172.21.15.84:6789/0 conn(0x7fc59406b2a0 legacy=0x7fc59415e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:36.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.435+0000 7fc5990c6640 1 -- 172.21.15.55:0/3276705152 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc59415d920 con 0x7fc59415acb0 2024-03-19T01:55:36.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.443+0000 7fc5990c6640 1 --2- 172.21.15.55:0/3276705152 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc59415acb0 0x7fc5940cce30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc58c009d90 tx=0x7fc58c01f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75e3a0ff0f30c7e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:36.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.451+0000 7fc5937fe640 1 -- 172.21.15.55:0/3276705152 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58c0b4070 con 0x7fc59415acb0 2024-03-19T01:55:36.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.451+0000 7fc5937fe640 1 -- 172.21.15.55:0/3276705152 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc58c0af600 con 0x7fc59415acb0 2024-03-19T01:55:36.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.451+0000 7fc5937fe640 1 -- 172.21.15.55:0/3276705152 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58c0af9d0 con 0x7fc59415acb0 2024-03-19T01:55:36.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.451+0000 7fc59ab29640 1 -- 172.21.15.55:0/3276705152 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc59415acb0 msgr2=0x7fc5940cce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:36.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.451+0000 7fc59ab29640 1 --2- 172.21.15.55:0/3276705152 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc59415acb0 0x7fc5940cce30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc58c009d90 tx=0x7fc58c01f150 comp rx=0 tx=0).stop 2024-03-19T01:55:36.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.451+0000 7fc59ab29640 1 -- 172.21.15.55:0/3276705152 shutdown_connections 2024-03-19T01:55:36.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.451+0000 7fc59ab29640 1 --2- 172.21.15.55:0/3276705152 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc59415acb0 0x7fc5940cce30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:36.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.451+0000 7fc59ab29640 1 --2- 172.21.15.55:0/3276705152 >> v2:172.21.15.84:3300/0 conn(0x7fc5941544f0 0x7fc594073540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:36.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.451+0000 7fc59ab29640 1 -- 172.21.15.55:0/3276705152 >> 172.21.15.55:0/3276705152 conn(0x7fc59406e670 msgr2=0x7fc59406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:36.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.451+0000 7fc59ab29640 1 -- 172.21.15.55:0/3276705152 shutdown_connections 2024-03-19T01:55:36.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.451+0000 7fc59ab29640 1 -- 172.21.15.55:0/3276705152 wait complete. 2024-03-19T01:55:36.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.455+0000 7fc59ab29640 1 Processor -- start 2024-03-19T01:55:36.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.455+0000 7fc59ab29640 1 -- start start 2024-03-19T01:55:36.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.455+0000 7fc59ab29640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc59406b2a0 0x7fc59415ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:36.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.455+0000 7fc59ab29640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5941544f0 0x7fc5940c53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:36.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.455+0000 7fc59ab29640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc59415acb0 0x7fc5940c5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:36.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.455+0000 7fc59ab29640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc5940ca120 con 0x7fc59406b2a0 2024-03-19T01:55:36.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.455+0000 7fc59ab29640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc5941548d0 con 0x7fc5941544f0 2024-03-19T01:55:36.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.455+0000 7fc59ab29640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc5940ca2a0 con 0x7fc59415acb0 2024-03-19T01:55:36.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.455+0000 7fc5988c5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc59406b2a0 0x7fc59415ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:36.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.455+0000 7fc593fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5941544f0 0x7fc5940c53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:36.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.455+0000 7fc5988c5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc59406b2a0 0x7fc59415ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:36418/0 (socket says 172.21.15.55:36418) 2024-03-19T01:55:36.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.455+0000 7fc5988c5640 1 -- 172.21.15.55:0/198994514 learned_addr learned my addr 172.21.15.55:0/198994514 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:36.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.455+0000 7fc5990c6640 1 --2- 172.21.15.55:0/198994514 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc59415acb0 0x7fc5940c5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:36.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.459+0000 7fc5988c5640 1 -- 172.21.15.55:0/198994514 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc59415acb0 msgr2=0x7fc5940c5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:36.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.459+0000 7fc5988c5640 1 --2- 172.21.15.55:0/198994514 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc59415acb0 0x7fc5940c5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:36.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.459+0000 7fc5988c5640 1 -- 172.21.15.55:0/198994514 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5941544f0 msgr2=0x7fc5940c53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:36.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.459+0000 7fc5988c5640 1 --2- 172.21.15.55:0/198994514 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5941544f0 0x7fc5940c53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:36.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.459+0000 7fc5988c5640 1 -- 172.21.15.55:0/198994514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5941efd90 con 0x7fc59406b2a0 2024-03-19T01:55:36.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.463+0000 7fc5988c5640 1 --2- 172.21.15.55:0/198994514 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc59406b2a0 0x7fc59415ce90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc58400c7f0 tx=0x7fc58400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:36.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.467+0000 7fc5917fa640 1 -- 172.21.15.55:0/198994514 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc584015070 con 0x7fc59406b2a0 2024-03-19T01:55:36.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.467+0000 7fc5917fa640 1 -- 172.21.15.55:0/198994514 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc584004240 con 0x7fc59406b2a0 2024-03-19T01:55:36.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.467+0000 7fc59ab29640 1 -- 172.21.15.55:0/198994514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5941582f0 con 0x7fc59406b2a0 2024-03-19T01:55:36.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.467+0000 7fc5917fa640 1 -- 172.21.15.55:0/198994514 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc584003620 con 0x7fc59406b2a0 2024-03-19T01:55:36.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.467+0000 7fc59ab29640 1 -- 172.21.15.55:0/198994514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5941f0310 con 0x7fc59406b2a0 2024-03-19T01:55:36.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.471+0000 7fc59ab29640 1 -- 172.21.15.55:0/198994514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc594068a40 con 0x7fc59406b2a0 2024-03-19T01:55:36.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.795+0000 7fc5917fa640 1 -- 172.21.15.55:0/198994514 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc5840039d0 con 0x7fc59406b2a0 2024-03-19T01:55:36.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.799+0000 7fc5917fa640 1 --2- 172.21.15.55:0/198994514 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc5800676e0 0x7fc580069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:36.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.799+0000 7fc593fff640 1 --2- 172.21.15.55:0/198994514 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc5800676e0 0x7fc580069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:36.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.799+0000 7fc593fff640 1 --2- 172.21.15.55:0/198994514 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc5800676e0 0x7fc580069ba0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc5940c6c60 tx=0x7fc588000bd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:36.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.839+0000 7fc5917fa640 1 -- 172.21.15.55:0/198994514 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fc5840543b0 con 0x7fc59406b2a0 2024-03-19T01:55:36.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:36.855+0000 7fc5917fa640 1 -- 172.21.15.55:0/198994514 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc584056310 con 0x7fc59406b2a0 2024-03-19T01:55:37.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:37.075+0000 7fc59ab29640 1 -- 172.21.15.55:0/198994514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5940c7be0 con 0x7fc59406b2a0 2024-03-19T01:55:37.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:37.103+0000 7fc5917fa640 1 -- 172.21.15.55:0/198994514 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fc58405a3b0 con 0x7fc59406b2a0 2024-03-19T01:55:37.101 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:37.101 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:55:37.101 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:55:37.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:37.107+0000 7fc59ab29640 1 -- 172.21.15.55:0/198994514 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc5800676e0 msgr2=0x7fc580069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:37.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:37.107+0000 7fc59ab29640 1 --2- 172.21.15.55:0/198994514 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc5800676e0 0x7fc580069ba0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc5940c6c60 tx=0x7fc588000bd0 comp rx=0 tx=0).stop 2024-03-19T01:55:37.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:37.107+0000 7fc59ab29640 1 -- 172.21.15.55:0/198994514 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc59406b2a0 msgr2=0x7fc59415ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:37.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:37.107+0000 7fc59ab29640 1 --2- 172.21.15.55:0/198994514 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc59406b2a0 0x7fc59415ce90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc58400c7f0 tx=0x7fc58400ccb0 comp rx=0 tx=0).stop 2024-03-19T01:55:37.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:37.107+0000 7fc59ab29640 1 -- 172.21.15.55:0/198994514 shutdown_connections 2024-03-19T01:55:37.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:37.107+0000 7fc59ab29640 1 --2- 172.21.15.55:0/198994514 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc5800676e0 0x7fc580069ba0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:37.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:37.107+0000 7fc59ab29640 1 --2- 172.21.15.55:0/198994514 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc59415acb0 0x7fc5940c5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:37.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:37.107+0000 7fc59ab29640 1 --2- 172.21.15.55:0/198994514 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5941544f0 0x7fc5940c53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:37.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:37.107+0000 7fc59ab29640 1 --2- 172.21.15.55:0/198994514 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc59406b2a0 0x7fc59415ce90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:37.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:37.107+0000 7fc59ab29640 1 -- 172.21.15.55:0/198994514 >> 172.21.15.55:0/198994514 conn(0x7fc59406e670 msgr2=0x7fc5941588d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:37.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:37.107+0000 7fc59ab29640 1 -- 172.21.15.55:0/198994514 shutdown_connections 2024-03-19T01:55:37.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:37.107+0000 7fc59ab29640 1 -- 172.21.15.55:0/198994514 wait complete. 2024-03-19T01:55:37.729 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:55:37.729 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:38.730 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:38.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.847+0000 7f0519d53640 1 Processor -- start 2024-03-19T01:55:38.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.847+0000 7f0519d53640 1 -- start start 2024-03-19T01:55:38.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.847+0000 7f0519d53640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f05141546a0 0x7f051415ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:38.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.847+0000 7f0519d53640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f051406cea0 0x7f051406d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:38.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.847+0000 7f0519d53640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f051415b2a0 con 0x7f05140c8750 2024-03-19T01:55:38.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.847+0000 7f0519d53640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f051415e7f0 con 0x7f051406cea0 2024-03-19T01:55:38.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.847+0000 7f0519d53640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f051406d850 con 0x7f05141546a0 2024-03-19T01:55:38.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.847+0000 7f0512ffd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f05140c8750 0x7f05140cdae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:55640/0 (socket says 172.21.15.55:55640) 2024-03-19T01:55:38.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.847+0000 7f0512ffd640 1 -- 172.21.15.55:0/562939452 learned_addr learned my addr 172.21.15.55:0/562939452 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:55:38.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.851+0000 7f05137fe640 1 --2- 172.21.15.55:0/562939452 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f05141546a0 0x7f051415ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:38.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.851+0000 7f0513fff640 1 --2- 172.21.15.55:0/562939452 >> v2:172.21.15.84:3300/0 conn(0x7f051406cea0 0x7f051406d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:38.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.851+0000 7f05127fc640 1 -- 172.21.15.55:0/562939452 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2850287909 0 0) 0x7f051415b2a0 con 0x7f05140c8750 2024-03-19T01:55:38.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.851+0000 7f05127fc640 1 -- 172.21.15.55:0/562939452 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04fc003530 con 0x7f05140c8750 2024-03-19T01:55:38.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.855+0000 7f05137fe640 1 -- 172.21.15.55:0/562939452 >> v2:172.21.15.84:3300/0 conn(0x7f051406cea0 msgr2=0x7f051406d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:38.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.855+0000 7f05137fe640 1 --2- 172.21.15.55:0/562939452 >> v2:172.21.15.84:3300/0 conn(0x7f051406cea0 0x7f051406d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:38.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.855+0000 7f05137fe640 1 -- 172.21.15.55:0/562939452 >> v1:172.21.15.55:6789/0 conn(0x7f05140c8750 legacy=0x7f05140cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:38.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.855+0000 7f05137fe640 1 -- 172.21.15.55:0/562939452 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0514154a80 con 0x7f05141546a0 2024-03-19T01:55:38.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.859+0000 7f05137fe640 1 --2- 172.21.15.55:0/562939452 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f05141546a0 0x7f051415ad60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f05040097c0 tx=0x7f050401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c2cd32af50666c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:38.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.863+0000 7f05127fc640 1 -- 172.21.15.55:0/562939452 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05040b2070 con 0x7f05141546a0 2024-03-19T01:55:38.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.863+0000 7f05127fc640 1 -- 172.21.15.55:0/562939452 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05040ad640 con 0x7f05141546a0 2024-03-19T01:55:38.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.863+0000 7f05127fc640 1 -- 172.21.15.55:0/562939452 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05040ada10 con 0x7f05141546a0 2024-03-19T01:55:38.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.867+0000 7f0519d53640 1 -- 172.21.15.55:0/562939452 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f05141546a0 msgr2=0x7f051415ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:38.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.867+0000 7f0519d53640 1 --2- 172.21.15.55:0/562939452 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f05141546a0 0x7f051415ad60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f05040097c0 tx=0x7f050401c4a0 comp rx=0 tx=0).stop 2024-03-19T01:55:38.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.867+0000 7f0519d53640 1 -- 172.21.15.55:0/562939452 shutdown_connections 2024-03-19T01:55:38.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.867+0000 7f0519d53640 1 --2- 172.21.15.55:0/562939452 >> v2:172.21.15.84:3300/0 conn(0x7f051406cea0 0x7f051406d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:38.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.867+0000 7f0519d53640 1 --2- 172.21.15.55:0/562939452 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f05141546a0 0x7f051415ad60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:38.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.867+0000 7f0519d53640 1 -- 172.21.15.55:0/562939452 >> 172.21.15.55:0/562939452 conn(0x7f05140714d0 msgr2=0x7f05140718e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:38.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.867+0000 7f0519d53640 1 -- 172.21.15.55:0/562939452 shutdown_connections 2024-03-19T01:55:38.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.867+0000 7f0519d53640 1 -- 172.21.15.55:0/562939452 wait complete. 2024-03-19T01:55:38.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.867+0000 7f0519d53640 1 Processor -- start 2024-03-19T01:55:38.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.867+0000 7f0519d53640 1 -- start start 2024-03-19T01:55:38.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.867+0000 7f0519d53640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f051406cea0 0x7f05140ca170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:38.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.867+0000 7f0519d53640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f05140c8750 0x7f05140ca6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:38.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.867+0000 7f0519d53640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f05141546a0 0x7f051413f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:38.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.867+0000 7f0519d53640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f051406c980 con 0x7f051406cea0 2024-03-19T01:55:38.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.867+0000 7f0519d53640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f051415e510 con 0x7f05141546a0 2024-03-19T01:55:38.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.867+0000 7f0519d53640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f051415d8a0 con 0x7f05140c8750 2024-03-19T01:55:38.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.867+0000 7f0513fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f05141546a0 0x7f051413f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:38.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.867+0000 7f0513fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f05141546a0 0x7f051413f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:48832/0 (socket says 172.21.15.55:48832) 2024-03-19T01:55:38.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.867+0000 7f0513fff640 1 -- 172.21.15.55:0/1011730034 learned_addr learned my addr 172.21.15.55:0/1011730034 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:38.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.871+0000 7f05137fe640 1 --2- 172.21.15.55:0/1011730034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f051406cea0 0x7f05140ca170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:38.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.875+0000 7f0512ffd640 1 --2- 172.21.15.55:0/1011730034 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f05140c8750 0x7f05140ca6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:38.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.875+0000 7f0513fff640 1 -- 172.21.15.55:0/1011730034 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f05140c8750 msgr2=0x7f05140ca6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:38.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.875+0000 7f0513fff640 1 --2- 172.21.15.55:0/1011730034 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f05140c8750 0x7f05140ca6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:38.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.875+0000 7f0513fff640 1 -- 172.21.15.55:0/1011730034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f051406cea0 msgr2=0x7f05140ca170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:38.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.875+0000 7f0513fff640 1 --2- 172.21.15.55:0/1011730034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f051406cea0 0x7f05140ca170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:38.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.875+0000 7f0513fff640 1 -- 172.21.15.55:0/1011730034 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f051415c960 con 0x7f05141546a0 2024-03-19T01:55:38.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.879+0000 7f0513fff640 1 --2- 172.21.15.55:0/1011730034 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f05141546a0 0x7f051413f310 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f050c00efc0 tx=0x7f050c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:38.882 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.883+0000 7f04fbfff640 1 -- 172.21.15.55:0/1011730034 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f050c018020 con 0x7f05141546a0 2024-03-19T01:55:38.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.883+0000 7f0519d53640 1 -- 172.21.15.55:0/1011730034 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f051415cc00 con 0x7f05141546a0 2024-03-19T01:55:38.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.883+0000 7f0519d53640 1 -- 172.21.15.55:0/1011730034 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05140c97e0 con 0x7f05141546a0 2024-03-19T01:55:38.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.883+0000 7f0519d53640 1 -- 172.21.15.55:0/1011730034 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0514068330 con 0x7f05141546a0 2024-03-19T01:55:38.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.887+0000 7f04fbfff640 1 -- 172.21.15.55:0/1011730034 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f050c014480 con 0x7f05141546a0 2024-03-19T01:55:38.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:38.887+0000 7f04fbfff640 1 -- 172.21.15.55:0/1011730034 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f050c014850 con 0x7f05141546a0 2024-03-19T01:55:39.096 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:55:39.099 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:55:39.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:39.203+0000 7f04fbfff640 1 -- 172.21.15.55:0/1011730034 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f050c01b020 con 0x7f05141546a0 2024-03-19T01:55:39.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:39.203+0000 7f04fbfff640 1 --2- 172.21.15.55:0/1011730034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0500067690 0x7f0500069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:39.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:39.203+0000 7f05137fe640 1 --2- 172.21.15.55:0/1011730034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0500067690 0x7f0500069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:39.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:39.203+0000 7f05137fe640 1 --2- 172.21.15.55:0/1011730034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0500067690 0x7f0500069b50 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f05040af000 tx=0x7f0504008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:39.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:39.207+0000 7f04fbfff640 1 -- 172.21.15.55:0/1011730034 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f050c055080 con 0x7f05141546a0 2024-03-19T01:55:39.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:39.227+0000 7f04fbfff640 1 -- 172.21.15.55:0/1011730034 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f050c022030 con 0x7f05141546a0 2024-03-19T01:55:39.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:39.435+0000 7f0519d53640 1 -- 172.21.15.55:0/1011730034 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05141535b0 con 0x7f05141546a0 2024-03-19T01:55:39.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:39.531+0000 7f04fbfff640 1 -- 172.21.15.55:0/1011730034 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f050c010040 con 0x7f05141546a0 2024-03-19T01:55:39.531 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:39.531 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:55:39.531 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:55:39.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:39.535+0000 7f0519d53640 1 -- 172.21.15.55:0/1011730034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0500067690 msgr2=0x7f0500069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:39.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:39.535+0000 7f0519d53640 1 --2- 172.21.15.55:0/1011730034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0500067690 0x7f0500069b50 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f05040af000 tx=0x7f0504008040 comp rx=0 tx=0).stop 2024-03-19T01:55:39.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:39.535+0000 7f0519d53640 1 -- 172.21.15.55:0/1011730034 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f05141546a0 msgr2=0x7f051413f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:39.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:39.535+0000 7f0519d53640 1 --2- 172.21.15.55:0/1011730034 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f05141546a0 0x7f051413f310 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f050c00efc0 tx=0x7f050c00c490 comp rx=0 tx=0).stop 2024-03-19T01:55:39.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:39.535+0000 7f0519d53640 1 -- 172.21.15.55:0/1011730034 shutdown_connections 2024-03-19T01:55:39.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:39.535+0000 7f0519d53640 1 --2- 172.21.15.55:0/1011730034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0500067690 0x7f0500069b50 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:39.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:39.535+0000 7f0519d53640 1 --2- 172.21.15.55:0/1011730034 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f05141546a0 0x7f051413f310 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:39.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:39.535+0000 7f0519d53640 1 --2- 172.21.15.55:0/1011730034 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f05140c8750 0x7f05140ca6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:39.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:39.535+0000 7f0519d53640 1 --2- 172.21.15.55:0/1011730034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f051406cea0 0x7f05140ca170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:39.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:39.535+0000 7f0519d53640 1 -- 172.21.15.55:0/1011730034 >> 172.21.15.55:0/1011730034 conn(0x7f05140714d0 msgr2=0x7f0514150950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:39.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:39.535+0000 7f0519d53640 1 -- 172.21.15.55:0/1011730034 shutdown_connections 2024-03-19T01:55:39.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:39.535+0000 7f0519d53640 1 -- 172.21.15.55:0/1011730034 wait complete. 2024-03-19T01:55:40.156 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:55:40.156 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:41.157 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:41.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.275+0000 7f4b9ccb1640 1 Processor -- start 2024-03-19T01:55:41.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.275+0000 7f4b9ccb1640 1 -- start start 2024-03-19T01:55:41.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.275+0000 7f4b9ccb1640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4b981544f0 0x7f4b98073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:41.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.275+0000 7f4b9ccb1640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b980c7fa0 con 0x7f4b9806b2a0 2024-03-19T01:55:41.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.275+0000 7f4b9ccb1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b980c8240 con 0x7f4b9815acb0 2024-03-19T01:55:41.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.275+0000 7f4b9ccb1640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4b9815b210 con 0x7f4b981544f0 2024-03-19T01:55:41.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.275+0000 7f4b96d9d640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f4b9815acb0 0x7f4b980cce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:40630/0 (socket says 172.21.15.55:40630) 2024-03-19T01:55:41.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.275+0000 7f4b96d9d640 1 -- 172.21.15.55:0/1301432851 learned_addr learned my addr 172.21.15.55:0/1301432851 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:55:41.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.275+0000 7f4b9659c640 1 --2- 172.21.15.55:0/1301432851 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4b981544f0 0x7f4b98073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:41.278 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.279+0000 7f4b9559a640 1 -- 172.21.15.55:0/1301432851 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1121699705 0 0) 0x7f4b980c7fa0 con 0x7f4b9806b2a0 2024-03-19T01:55:41.278 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.279+0000 7f4b9559a640 1 -- 172.21.15.55:0/1301432851 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b80003530 con 0x7f4b9806b2a0 2024-03-19T01:55:41.278 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.279+0000 7f4b9559a640 1 -- 172.21.15.55:0/1301432851 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3783684034 0 0) 0x7f4b980c8240 con 0x7f4b9815acb0 2024-03-19T01:55:41.278 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.279+0000 7f4b9559a640 1 -- 172.21.15.55:0/1301432851 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b980c7fa0 con 0x7f4b9815acb0 2024-03-19T01:55:41.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.283+0000 7f4b9659c640 1 -- 172.21.15.55:0/1301432851 >> v1:172.21.15.84:6789/0 conn(0x7f4b9815acb0 legacy=0x7f4b980cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:41.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.283+0000 7f4b9659c640 1 -- 172.21.15.55:0/1301432851 >> v1:172.21.15.55:6789/0 conn(0x7f4b9806b2a0 legacy=0x7f4b9815e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:41.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.283+0000 7f4b9659c640 1 -- 172.21.15.55:0/1301432851 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b9815d920 con 0x7f4b981544f0 2024-03-19T01:55:41.283 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.287+0000 7f4b9659c640 1 --2- 172.21.15.55:0/1301432851 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4b981544f0 0x7f4b98073540 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4b880097f0 tx=0x7f4b8801ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=799f3c62a3efe29e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:41.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.295+0000 7f4b9559a640 1 -- 172.21.15.55:0/1301432851 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b880b5070 con 0x7f4b981544f0 2024-03-19T01:55:41.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.295+0000 7f4b9559a640 1 -- 172.21.15.55:0/1301432851 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b880a89d0 con 0x7f4b981544f0 2024-03-19T01:55:41.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.295+0000 7f4b9559a640 1 -- 172.21.15.55:0/1301432851 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b880a8da0 con 0x7f4b981544f0 2024-03-19T01:55:41.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.295+0000 7f4b9ccb1640 1 -- 172.21.15.55:0/1301432851 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4b981544f0 msgr2=0x7f4b98073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:41.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.295+0000 7f4b9ccb1640 1 --2- 172.21.15.55:0/1301432851 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4b981544f0 0x7f4b98073540 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4b880097f0 tx=0x7f4b8801ecd0 comp rx=0 tx=0).stop 2024-03-19T01:55:41.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.295+0000 7f4b9ccb1640 1 -- 172.21.15.55:0/1301432851 shutdown_connections 2024-03-19T01:55:41.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.295+0000 7f4b9ccb1640 1 --2- 172.21.15.55:0/1301432851 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4b981544f0 0x7f4b98073540 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:41.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.295+0000 7f4b9ccb1640 1 -- 172.21.15.55:0/1301432851 >> 172.21.15.55:0/1301432851 conn(0x7f4b9806e670 msgr2=0x7f4b9806ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:41.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.295+0000 7f4b9ccb1640 1 -- 172.21.15.55:0/1301432851 shutdown_connections 2024-03-19T01:55:41.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.295+0000 7f4b9ccb1640 1 -- 172.21.15.55:0/1301432851 wait complete. 2024-03-19T01:55:41.293 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.295+0000 7f4b9ccb1640 1 Processor -- start 2024-03-19T01:55:41.293 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.295+0000 7f4b9ccb1640 1 -- start start 2024-03-19T01:55:41.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.299+0000 7f4b9ccb1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4b9806b2a0 0x7f4b98157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:41.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.299+0000 7f4b9ccb1640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4b981544f0 0x7f4b981f3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:41.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.299+0000 7f4b9ccb1640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b9815acb0 0x7f4b98152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:41.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.299+0000 7f4b9ccb1640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4b980cd540 con 0x7f4b9806b2a0 2024-03-19T01:55:41.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.299+0000 7f4b9ccb1640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4b981548d0 con 0x7f4b9815acb0 2024-03-19T01:55:41.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.299+0000 7f4b9ccb1640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4b980cd6c0 con 0x7f4b981544f0 2024-03-19T01:55:41.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.299+0000 7f4b9659c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4b9806b2a0 0x7f4b98157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:41.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.299+0000 7f4b96d9d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b9815acb0 0x7f4b98152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:41.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.299+0000 7f4b95d9b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4b981544f0 0x7f4b981f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:41.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.299+0000 7f4b9659c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4b9806b2a0 0x7f4b98157740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:35720/0 (socket says 172.21.15.55:35720) 2024-03-19T01:55:41.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.299+0000 7f4b9659c640 1 -- 172.21.15.55:0/298505491 learned_addr learned my addr 172.21.15.55:0/298505491 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:41.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.303+0000 7f4b9659c640 1 -- 172.21.15.55:0/298505491 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4b981544f0 msgr2=0x7f4b981f3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:41.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.303+0000 7f4b9659c640 1 --2- 172.21.15.55:0/298505491 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4b981544f0 0x7f4b981f3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:41.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.303+0000 7f4b9659c640 1 -- 172.21.15.55:0/298505491 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b9815acb0 msgr2=0x7f4b98152e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:41.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.303+0000 7f4b9659c640 1 --2- 172.21.15.55:0/298505491 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b9815acb0 0x7f4b98152e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:41.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.303+0000 7f4b9659c640 1 -- 172.21.15.55:0/298505491 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b981563a0 con 0x7f4b9806b2a0 2024-03-19T01:55:41.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.303+0000 7f4b9659c640 1 --2- 172.21.15.55:0/298505491 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4b9806b2a0 0x7f4b98157740 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4b880b3040 tx=0x7f4b8801ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:41.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.311+0000 7f4b86ffd640 1 -- 172.21.15.55:0/298505491 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b880bf070 con 0x7f4b9806b2a0 2024-03-19T01:55:41.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.311+0000 7f4b86ffd640 1 -- 172.21.15.55:0/298505491 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b880b0720 con 0x7f4b9806b2a0 2024-03-19T01:55:41.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.311+0000 7f4b86ffd640 1 -- 172.21.15.55:0/298505491 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b880b5040 con 0x7f4b9806b2a0 2024-03-19T01:55:41.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.311+0000 7f4b9ccb1640 1 -- 172.21.15.55:0/298505491 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b98156680 con 0x7f4b9806b2a0 2024-03-19T01:55:41.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.311+0000 7f4b9ccb1640 1 -- 172.21.15.55:0/298505491 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b98156f40 con 0x7f4b9806b2a0 2024-03-19T01:55:41.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.311+0000 7f4b9ccb1640 1 -- 172.21.15.55:0/298505491 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b98068a40 con 0x7f4b9806b2a0 2024-03-19T01:55:41.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.343+0000 7f4b86ffd640 1 -- 172.21.15.55:0/298505491 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f4b880c2070 con 0x7f4b9806b2a0 2024-03-19T01:55:41.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.343+0000 7f4b86ffd640 1 --2- 172.21.15.55:0/298505491 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4b780676e0 0x7f4b78069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:41.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.343+0000 7f4b95d9b640 1 --2- 172.21.15.55:0/298505491 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4b780676e0 0x7f4b78069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:41.342 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.343+0000 7f4b95d9b640 1 --2- 172.21.15.55:0/298505491 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4b780676e0 0x7f4b78069ba0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c007fd0 tx=0x7f4b8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:41.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.347+0000 7f4b86ffd640 1 -- 172.21.15.55:0/298505491 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f4b880f09d0 con 0x7f4b9806b2a0 2024-03-19T01:55:41.632 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.635+0000 7f4b86ffd640 1 -- 172.21.15.55:0/298505491 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4b880c9080 con 0x7f4b9806b2a0 2024-03-19T01:55:41.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.863+0000 7f4b9ccb1640 1 -- 172.21.15.55:0/298505491 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b98071310 con 0x7f4b9806b2a0 2024-03-19T01:55:41.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.887+0000 7f4b86ffd640 1 -- 172.21.15.55:0/298505491 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f4b88018830 con 0x7f4b9806b2a0 2024-03-19T01:55:41.885 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:41.885 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:55:41.886 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:55:41.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.891+0000 7f4b9ccb1640 1 -- 172.21.15.55:0/298505491 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4b780676e0 msgr2=0x7f4b78069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:41.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.891+0000 7f4b9ccb1640 1 --2- 172.21.15.55:0/298505491 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4b780676e0 0x7f4b78069ba0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c007fd0 tx=0x7f4b8c009040 comp rx=0 tx=0).stop 2024-03-19T01:55:41.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.891+0000 7f4b9ccb1640 1 -- 172.21.15.55:0/298505491 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4b9806b2a0 msgr2=0x7f4b98157740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:41.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.891+0000 7f4b9ccb1640 1 --2- 172.21.15.55:0/298505491 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4b9806b2a0 0x7f4b98157740 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4b880b3040 tx=0x7f4b8801ba70 comp rx=0 tx=0).stop 2024-03-19T01:55:41.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.891+0000 7f4b9ccb1640 1 -- 172.21.15.55:0/298505491 shutdown_connections 2024-03-19T01:55:41.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.891+0000 7f4b9ccb1640 1 --2- 172.21.15.55:0/298505491 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4b780676e0 0x7f4b78069ba0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:41.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.891+0000 7f4b9ccb1640 1 --2- 172.21.15.55:0/298505491 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b9815acb0 0x7f4b98152e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:41.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.891+0000 7f4b9ccb1640 1 --2- 172.21.15.55:0/298505491 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4b981544f0 0x7f4b981f3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:41.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.891+0000 7f4b9ccb1640 1 --2- 172.21.15.55:0/298505491 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4b9806b2a0 0x7f4b98157740 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:41.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.891+0000 7f4b9ccb1640 1 -- 172.21.15.55:0/298505491 >> 172.21.15.55:0/298505491 conn(0x7f4b9806e670 msgr2=0x7f4b98070770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:41.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.891+0000 7f4b9ccb1640 1 -- 172.21.15.55:0/298505491 shutdown_connections 2024-03-19T01:55:41.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:41.891+0000 7f4b9ccb1640 1 -- 172.21.15.55:0/298505491 wait complete. 2024-03-19T01:55:42.509 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:55:42.509 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:43.510 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:43.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.643+0000 7f904fa57640 1 Processor -- start 2024-03-19T01:55:43.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.643+0000 7f904fa57640 1 -- start start 2024-03-19T01:55:43.641 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.643+0000 7f904fa57640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f9048154720 0x7f904815e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:43.641 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.643+0000 7f904fa57640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9048071280 con 0x7f90480c73c0 2024-03-19T01:55:43.641 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.643+0000 7f904fa57640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f904815d900 con 0x7f9048154340 2024-03-19T01:55:43.641 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.643+0000 7f904fa57640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f9048073840 con 0x7f9048154720 2024-03-19T01:55:43.642 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.643+0000 7f904dff4640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f90480c73c0 0x7f90480c91d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:55654/0 (socket says 172.21.15.55:55654) 2024-03-19T01:55:43.642 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.643+0000 7f904dff4640 1 -- 172.21.15.55:0/4044058393 learned_addr learned my addr 172.21.15.55:0/4044058393 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:55:43.642 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.647+0000 7f904cff2640 1 --2- 172.21.15.55:0/4044058393 >> v2:172.21.15.84:3300/0 conn(0x7f9048154720 0x7f904815e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:43.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.647+0000 7f903ffff640 1 -- 172.21.15.55:0/4044058393 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2695041322 0 0) 0x7f9048071280 con 0x7f90480c73c0 2024-03-19T01:55:43.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.647+0000 7f903ffff640 1 -- 172.21.15.55:0/4044058393 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9034003530 con 0x7f90480c73c0 2024-03-19T01:55:43.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.651+0000 7f903ffff640 1 -- 172.21.15.55:0/4044058393 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 470371091 0 0) 0x7f904815d900 con 0x7f9048154340 2024-03-19T01:55:43.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.651+0000 7f903ffff640 1 -- 172.21.15.55:0/4044058393 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9048071280 con 0x7f9048154340 2024-03-19T01:55:43.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.651+0000 7f903ffff640 1 -- 172.21.15.55:0/4044058393 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3390503791 0 0) 0x7f9034003530 con 0x7f90480c73c0 2024-03-19T01:55:43.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.651+0000 7f903ffff640 1 -- 172.21.15.55:0/4044058393 >> v2:172.21.15.84:3300/0 conn(0x7f9048154720 msgr2=0x7f904815e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:43.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.651+0000 7f903ffff640 1 --2- 172.21.15.55:0/4044058393 >> v2:172.21.15.84:3300/0 conn(0x7f9048154720 0x7f904815e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:43.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.651+0000 7f903ffff640 1 -- 172.21.15.55:0/4044058393 >> v1:172.21.15.84:6789/0 conn(0x7f9048154340 legacy=0x7f9048072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:43.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.651+0000 7f903ffff640 1 -- 172.21.15.55:0/4044058393 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9048070cc0 con 0x7f90480c73c0 2024-03-19T01:55:43.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.651+0000 7f903ffff640 1 -- 172.21.15.55:0/4044058393 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f9040001a00 con 0x7f90480c73c0 2024-03-19T01:55:43.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.651+0000 7f903ffff640 1 -- 172.21.15.55:0/4044058393 >> v1:172.21.15.55:6789/0 conn(0x7f90480c73c0 legacy=0x7f90480c91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:43.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.651+0000 7f903ffff640 1 --2- 172.21.15.55:0/4044058393 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9034003ba0 0x7f9034003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:43.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.651+0000 7f903ffff640 1 --2- 172.21.15.55:0/4044058393 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9034004810 0x7f9034004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:43.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.651+0000 7f903ffff640 1 --2- 172.21.15.55:0/4044058393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90340051d0 0x7f903400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:43.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.651+0000 7f903ffff640 1 -- 172.21.15.55:0/4044058393 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f903400c0a0 con 0x7f90340051d0 2024-03-19T01:55:43.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.651+0000 7f903ffff640 1 -- 172.21.15.55:0/4044058393 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f903400c220 con 0x7f9034004810 2024-03-19T01:55:43.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.651+0000 7f903ffff640 1 -- 172.21.15.55:0/4044058393 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f903400c420 con 0x7f9034003ba0 2024-03-19T01:55:43.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.651+0000 7f904d7f3640 1 --2- 172.21.15.55:0/4044058393 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9034003ba0 0x7f9034003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:43.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.655+0000 7f904dff4640 1 --2- 172.21.15.55:0/4044058393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90340051d0 0x7f903400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:43.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.659+0000 7f904dff4640 1 -- 172.21.15.55:0/4044058393 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9034003ba0 msgr2=0x7f9034003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:43.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.659+0000 7f904dff4640 1 --2- 172.21.15.55:0/4044058393 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9034003ba0 0x7f9034003fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:43.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.659+0000 7f904dff4640 1 -- 172.21.15.55:0/4044058393 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9034004810 msgr2=0x7f9034004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:43.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.659+0000 7f904dff4640 1 --2- 172.21.15.55:0/4044058393 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9034004810 0x7f9034004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:43.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.659+0000 7f904dff4640 1 -- 172.21.15.55:0/4044058393 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9034003530 con 0x7f90340051d0 2024-03-19T01:55:43.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.659+0000 7f904d7f3640 1 --2- 172.21.15.55:0/4044058393 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9034003ba0 0x7f9034003fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:55:43.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.659+0000 7f904dff4640 1 -- 172.21.15.55:0/4044058393 reap_dead start 2024-03-19T01:55:43.657 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.659+0000 7f904dff4640 1 --2- 172.21.15.55:0/4044058393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90340051d0 0x7f903400b9d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f904000a710 tx=0x7f904001f980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4d17798d4c5240 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:43.661 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.663+0000 7f903ffff640 1 -- 172.21.15.55:0/4044058393 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9044001520 con 0x7f90340051d0 2024-03-19T01:55:43.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.663+0000 7f903ffff640 1 -- 172.21.15.55:0/4044058393 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90400ac650 con 0x7f90340051d0 2024-03-19T01:55:43.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.663+0000 7f903ffff640 1 -- 172.21.15.55:0/4044058393 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904000a330 con 0x7f90340051d0 2024-03-19T01:55:43.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.667+0000 7f904fa57640 1 -- 172.21.15.55:0/4044058393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90340051d0 msgr2=0x7f903400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:43.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.667+0000 7f904fa57640 1 --2- 172.21.15.55:0/4044058393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90340051d0 0x7f903400b9d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f904000a710 tx=0x7f904001f980 comp rx=0 tx=0).stop 2024-03-19T01:55:43.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.667+0000 7f904fa57640 1 -- 172.21.15.55:0/4044058393 shutdown_connections 2024-03-19T01:55:43.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.667+0000 7f904fa57640 1 --2- 172.21.15.55:0/4044058393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90340051d0 0x7f903400b9d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:43.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.667+0000 7f904fa57640 1 -- 172.21.15.55:0/4044058393 >> 172.21.15.55:0/4044058393 conn(0x7f90480cc9c0 msgr2=0x7f90480ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:43.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.667+0000 7f904fa57640 1 -- 172.21.15.55:0/4044058393 shutdown_connections 2024-03-19T01:55:43.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.667+0000 7f904fa57640 1 -- 172.21.15.55:0/4044058393 wait complete. 2024-03-19T01:55:43.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.667+0000 7f904fa57640 1 Processor -- start 2024-03-19T01:55:43.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.667+0000 7f904fa57640 1 -- start start 2024-03-19T01:55:43.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.667+0000 7f904fa57640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f90340051d0 0x7f9048157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:43.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.667+0000 7f904fa57640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f90481efdd0 0x7f904814f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:43.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.667+0000 7f904fa57640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90480c8fe0 0x7f90480c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:43.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.667+0000 7f904fa57640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f904815df10 con 0x7f90480c8fe0 2024-03-19T01:55:43.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.667+0000 7f904fa57640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f9048070e90 con 0x7f90481efdd0 2024-03-19T01:55:43.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.667+0000 7f904fa57640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f9048071010 con 0x7f90340051d0 2024-03-19T01:55:43.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.667+0000 7f904dff4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90480c8fe0 0x7f90480c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:43.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.667+0000 7f904d7f3640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f90340051d0 0x7f9048157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:43.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.667+0000 7f904dff4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90480c8fe0 0x7f90480c9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:35738/0 (socket says 172.21.15.55:35738) 2024-03-19T01:55:43.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.667+0000 7f904dff4640 1 -- 172.21.15.55:0/1517486609 learned_addr learned my addr 172.21.15.55:0/1517486609 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:43.668 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.671+0000 7f904cff2640 1 --2- 172.21.15.55:0/1517486609 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f90481efdd0 0x7f904814f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:43.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.671+0000 7f904dff4640 1 -- 172.21.15.55:0/1517486609 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f90340051d0 msgr2=0x7f9048157990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:43.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.671+0000 7f904dff4640 1 --2- 172.21.15.55:0/1517486609 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f90340051d0 0x7f9048157990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:43.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.671+0000 7f904dff4640 1 -- 172.21.15.55:0/1517486609 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f90481efdd0 msgr2=0x7f904814f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:43.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.671+0000 7f904dff4640 1 --2- 172.21.15.55:0/1517486609 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f90481efdd0 0x7f904814f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:43.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.671+0000 7f904dff4640 1 -- 172.21.15.55:0/1517486609 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90480c79e0 con 0x7f90480c8fe0 2024-03-19T01:55:43.672 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.675+0000 7f904dff4640 1 --2- 172.21.15.55:0/1517486609 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90480c8fe0 0x7f90480c9460 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9040021060 tx=0x7f9034005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:43.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.679+0000 7f903dffb640 1 -- 172.21.15.55:0/1517486609 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90400bc070 con 0x7f90480c8fe0 2024-03-19T01:55:43.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.679+0000 7f904fa57640 1 -- 172.21.15.55:0/1517486609 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90480c7c80 con 0x7f90480c8fe0 2024-03-19T01:55:43.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.679+0000 7f904fa57640 1 -- 172.21.15.55:0/1517486609 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90480c8220 con 0x7f90480c8fe0 2024-03-19T01:55:43.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.679+0000 7f903dffb640 1 -- 172.21.15.55:0/1517486609 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90400ac650 con 0x7f90480c8fe0 2024-03-19T01:55:43.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.679+0000 7f904fa57640 1 -- 172.21.15.55:0/1517486609 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9048068a40 con 0x7f90480c8fe0 2024-03-19T01:55:43.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.679+0000 7f903dffb640 1 -- 172.21.15.55:0/1517486609 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9040001e90 con 0x7f90480c8fe0 2024-03-19T01:55:43.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.955+0000 7f903dffb640 1 -- 172.21.15.55:0/1517486609 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f90400bf070 con 0x7f90480c8fe0 2024-03-19T01:55:43.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.955+0000 7f903dffb640 1 --2- 172.21.15.55:0/1517486609 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f902c067660 0x7f902c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:43.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.959+0000 7f904d7f3640 1 --2- 172.21.15.55:0/1517486609 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f902c067660 0x7f902c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:43.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.959+0000 7f904d7f3640 1 --2- 172.21.15.55:0/1517486609 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f902c067660 0x7f902c069b20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9044000c10 tx=0x7f9044009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:43.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.979+0000 7f903dffb640 1 -- 172.21.15.55:0/1517486609 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f90400f1230 con 0x7f90480c8fe0 2024-03-19T01:55:43.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:43.979+0000 7f903dffb640 1 -- 172.21.15.55:0/1517486609 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f90400c7080 con 0x7f90480c8fe0 2024-03-19T01:55:44.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:44.195+0000 7f904fa57640 1 -- 172.21.15.55:0/1517486609 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90480c3980 con 0x7f90480c8fe0 2024-03-19T01:55:44.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:44.219+0000 7f903dffb640 1 -- 172.21.15.55:0/1517486609 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f9040019690 con 0x7f90480c8fe0 2024-03-19T01:55:44.216 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:44.216 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:55:44.217 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:55:44.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:44.223+0000 7f904fa57640 1 -- 172.21.15.55:0/1517486609 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f902c067660 msgr2=0x7f902c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:44.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:44.223+0000 7f904fa57640 1 --2- 172.21.15.55:0/1517486609 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f902c067660 0x7f902c069b20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9044000c10 tx=0x7f9044009040 comp rx=0 tx=0).stop 2024-03-19T01:55:44.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:44.223+0000 7f904fa57640 1 -- 172.21.15.55:0/1517486609 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90480c8fe0 msgr2=0x7f90480c9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:44.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:44.223+0000 7f904fa57640 1 --2- 172.21.15.55:0/1517486609 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90480c8fe0 0x7f90480c9460 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9040021060 tx=0x7f9034005840 comp rx=0 tx=0).stop 2024-03-19T01:55:44.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:44.223+0000 7f904fa57640 1 -- 172.21.15.55:0/1517486609 shutdown_connections 2024-03-19T01:55:44.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:44.223+0000 7f904fa57640 1 --2- 172.21.15.55:0/1517486609 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f902c067660 0x7f902c069b20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:44.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:44.223+0000 7f904fa57640 1 --2- 172.21.15.55:0/1517486609 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90480c8fe0 0x7f90480c9460 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:44.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:44.223+0000 7f904fa57640 1 --2- 172.21.15.55:0/1517486609 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f90481efdd0 0x7f904814f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:44.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:44.223+0000 7f904fa57640 1 --2- 172.21.15.55:0/1517486609 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f90340051d0 0x7f9048157990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:44.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:44.223+0000 7f904fa57640 1 -- 172.21.15.55:0/1517486609 >> 172.21.15.55:0/1517486609 conn(0x7f90480cc9c0 msgr2=0x7f9048158b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:44.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:44.223+0000 7f904fa57640 1 -- 172.21.15.55:0/1517486609 shutdown_connections 2024-03-19T01:55:44.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:44.223+0000 7f904fa57640 1 -- 172.21.15.55:0/1517486609 wait complete. 2024-03-19T01:55:44.841 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:55:44.841 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:45.841 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:45.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.971+0000 7f5aa28df640 1 Processor -- start 2024-03-19T01:55:45.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.971+0000 7f5aa28df640 1 -- start start 2024-03-19T01:55:45.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.975+0000 7f5aa28df640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f5a9c0c60d0 0x7f5a9c0c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:45.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.975+0000 7f5aa28df640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a9c14ee80 con 0x7f5a9c14eaa0 2024-03-19T01:55:45.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.975+0000 7f5aa28df640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a9c0c7540 con 0x7f5a9c1585f0 2024-03-19T01:55:45.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.975+0000 7f5aa28df640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f5a9c0c77c0 con 0x7f5a9c0c60d0 2024-03-19T01:55:45.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.975+0000 7f5a9bfff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f5a9c14eaa0 0x7f5a9c06b980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:55664/0 (socket says 172.21.15.55:55664) 2024-03-19T01:55:45.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.975+0000 7f5a9bfff640 1 -- 172.21.15.55:0/3028304036 learned_addr learned my addr 172.21.15.55:0/3028304036 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:55:45.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.975+0000 7f5a9b7fe640 1 --2- 172.21.15.55:0/3028304036 >> v2:172.21.15.84:3300/0 conn(0x7f5a9c0c60d0 0x7f5a9c0c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:45.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.979+0000 7f5a9affd640 1 -- 172.21.15.55:0/3028304036 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2880947004 0 0) 0x7f5a9c14ee80 con 0x7f5a9c14eaa0 2024-03-19T01:55:45.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.979+0000 7f5a9affd640 1 -- 172.21.15.55:0/3028304036 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a84003530 con 0x7f5a9c14eaa0 2024-03-19T01:55:45.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.979+0000 7f5a9b7fe640 1 -- 172.21.15.55:0/3028304036 >> v1:172.21.15.84:6789/0 conn(0x7f5a9c1585f0 legacy=0x7f5a9c162940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:45.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.979+0000 7f5a9b7fe640 1 -- 172.21.15.55:0/3028304036 >> v1:172.21.15.55:6789/0 conn(0x7f5a9c14eaa0 legacy=0x7f5a9c06b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:45.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.979+0000 7f5a9b7fe640 1 -- 172.21.15.55:0/3028304036 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a9c0c7aa0 con 0x7f5a9c0c60d0 2024-03-19T01:55:45.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.987+0000 7f5a9b7fe640 1 --2- 172.21.15.55:0/3028304036 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a9c0c60d0 0x7f5a9c0c6540 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5a900097c0 tx=0x7f5a9001c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=931a302fec3ce04f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:45.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.999+0000 7f5a9affd640 1 -- 172.21.15.55:0/3028304036 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a900ab610 con 0x7f5a9c0c60d0 2024-03-19T01:55:45.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.999+0000 7f5a9affd640 1 -- 172.21.15.55:0/3028304036 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a900b0070 con 0x7f5a9c0c60d0 2024-03-19T01:55:45.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.999+0000 7f5a9affd640 1 -- 172.21.15.55:0/3028304036 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a900ab9d0 con 0x7f5a9c0c60d0 2024-03-19T01:55:45.997 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.999+0000 7f5aa28df640 1 -- 172.21.15.55:0/3028304036 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a9c0c60d0 msgr2=0x7f5a9c0c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:45.997 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.999+0000 7f5aa28df640 1 --2- 172.21.15.55:0/3028304036 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a9c0c60d0 0x7f5a9c0c6540 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5a900097c0 tx=0x7f5a9001c410 comp rx=0 tx=0).stop 2024-03-19T01:55:45.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.999+0000 7f5aa28df640 1 -- 172.21.15.55:0/3028304036 shutdown_connections 2024-03-19T01:55:45.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.999+0000 7f5aa28df640 1 --2- 172.21.15.55:0/3028304036 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a9c0c60d0 0x7f5a9c0c6540 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:45.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.999+0000 7f5aa28df640 1 -- 172.21.15.55:0/3028304036 >> 172.21.15.55:0/3028304036 conn(0x7f5a9c0cc9c0 msgr2=0x7f5a9c0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:45.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.999+0000 7f5aa28df640 1 -- 172.21.15.55:0/3028304036 shutdown_connections 2024-03-19T01:55:45.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:45.999+0000 7f5aa28df640 1 -- 172.21.15.55:0/3028304036 wait complete. 2024-03-19T01:55:45.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.003+0000 7f5aa28df640 1 Processor -- start 2024-03-19T01:55:45.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.003+0000 7f5aa28df640 1 -- start start 2024-03-19T01:55:45.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.003+0000 7f5aa28df640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a9c0c60d0 0x7f5a9c147a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:45.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.003+0000 7f5aa28df640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a9c14eaa0 0x7f5a9c14d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:45.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.003+0000 7f5aa28df640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a9c1585f0 0x7f5a9c14da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:46.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.003+0000 7f5aa28df640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9c06c090 con 0x7f5a9c0c60d0 2024-03-19T01:55:46.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.003+0000 7f5aa28df640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9c158e10 con 0x7f5a9c1585f0 2024-03-19T01:55:46.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.003+0000 7f5aa28df640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f5a9c06c210 con 0x7f5a9c14eaa0 2024-03-19T01:55:46.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.003+0000 7f5a9bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a9c0c60d0 0x7f5a9c147a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:46.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.003+0000 7f5aa0e7c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a9c1585f0 0x7f5a9c14da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:46.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.003+0000 7f5a9bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a9c0c60d0 0x7f5a9c147a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:35750/0 (socket says 172.21.15.55:35750) 2024-03-19T01:55:46.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.003+0000 7f5a9bfff640 1 -- 172.21.15.55:0/580349475 learned_addr learned my addr 172.21.15.55:0/580349475 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:46.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.003+0000 7f5a9b7fe640 1 --2- 172.21.15.55:0/580349475 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a9c14eaa0 0x7f5a9c14d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:46.003 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.007+0000 7f5a9bfff640 1 -- 172.21.15.55:0/580349475 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a9c14eaa0 msgr2=0x7f5a9c14d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:46.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.007+0000 7f5a9bfff640 1 --2- 172.21.15.55:0/580349475 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a9c14eaa0 0x7f5a9c14d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:46.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.007+0000 7f5a9bfff640 1 -- 172.21.15.55:0/580349475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a9c1585f0 msgr2=0x7f5a9c14da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:46.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.007+0000 7f5a9bfff640 1 --2- 172.21.15.55:0/580349475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a9c1585f0 0x7f5a9c14da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:46.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.007+0000 7f5a9bfff640 1 -- 172.21.15.55:0/580349475 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a9c148710 con 0x7f5a9c0c60d0 2024-03-19T01:55:46.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.011+0000 7f5a9bfff640 1 --2- 172.21.15.55:0/580349475 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a9c0c60d0 0x7f5a9c147a00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c00c3f0 tx=0x7f5a8c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:46.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.015+0000 7f5a98ff9640 1 -- 172.21.15.55:0/580349475 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c017070 con 0x7f5a9c0c60d0 2024-03-19T01:55:46.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.015+0000 7f5a98ff9640 1 -- 172.21.15.55:0/580349475 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a8c005cf0 con 0x7f5a9c0c60d0 2024-03-19T01:55:46.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.015+0000 7f5aa28df640 1 -- 172.21.15.55:0/580349475 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a9c148980 con 0x7f5a9c0c60d0 2024-03-19T01:55:46.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.015+0000 7f5a98ff9640 1 -- 172.21.15.55:0/580349475 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c0072f0 con 0x7f5a9c0c60d0 2024-03-19T01:55:46.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.015+0000 7f5aa28df640 1 -- 172.21.15.55:0/580349475 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a9c155340 con 0x7f5a9c0c60d0 2024-03-19T01:55:46.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.015+0000 7f5aa28df640 1 -- 172.21.15.55:0/580349475 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a9c068a40 con 0x7f5a9c0c60d0 2024-03-19T01:55:46.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.047+0000 7f5a98ff9640 1 -- 172.21.15.55:0/580349475 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f5a8c01a070 con 0x7f5a9c0c60d0 2024-03-19T01:55:46.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.047+0000 7f5a98ff9640 1 --2- 172.21.15.55:0/580349475 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5a880676e0 0x7f5a88069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:46.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.047+0000 7f5a9b7fe640 1 --2- 172.21.15.55:0/580349475 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5a880676e0 0x7f5a88069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:46.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.047+0000 7f5a9b7fe640 1 --2- 172.21.15.55:0/580349475 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5a880676e0 0x7f5a88069ba0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5a900af000 tx=0x7f5a900a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:46.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.051+0000 7f5a98ff9640 1 -- 172.21.15.55:0/580349475 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f5a8c053780 con 0x7f5a9c0c60d0 2024-03-19T01:55:46.347 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.351+0000 7f5a98ff9640 1 -- 172.21.15.55:0/580349475 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f5a8c021020 con 0x7f5a9c0c60d0 2024-03-19T01:55:46.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.555+0000 7f5aa28df640 1 -- 172.21.15.55:0/580349475 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a9c1558e0 con 0x7f5a9c0c60d0 2024-03-19T01:55:46.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.659+0000 7f5a98ff9640 1 -- 172.21.15.55:0/580349475 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f5a8c0556e0 con 0x7f5a9c0c60d0 2024-03-19T01:55:46.657 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:46.657 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:55:46.657 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:55:46.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.663+0000 7f5aa28df640 1 -- 172.21.15.55:0/580349475 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5a880676e0 msgr2=0x7f5a88069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:46.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.663+0000 7f5aa28df640 1 --2- 172.21.15.55:0/580349475 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5a880676e0 0x7f5a88069ba0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5a900af000 tx=0x7f5a900a7000 comp rx=0 tx=0).stop 2024-03-19T01:55:46.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.663+0000 7f5aa28df640 1 -- 172.21.15.55:0/580349475 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a9c0c60d0 msgr2=0x7f5a9c147a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:46.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.663+0000 7f5aa28df640 1 --2- 172.21.15.55:0/580349475 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a9c0c60d0 0x7f5a9c147a00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c00c3f0 tx=0x7f5a8c00c8b0 comp rx=0 tx=0).stop 2024-03-19T01:55:46.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.663+0000 7f5aa28df640 1 -- 172.21.15.55:0/580349475 shutdown_connections 2024-03-19T01:55:46.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.663+0000 7f5aa28df640 1 --2- 172.21.15.55:0/580349475 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5a880676e0 0x7f5a88069ba0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:46.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.663+0000 7f5aa28df640 1 --2- 172.21.15.55:0/580349475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a9c1585f0 0x7f5a9c14da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:46.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.663+0000 7f5aa28df640 1 --2- 172.21.15.55:0/580349475 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a9c14eaa0 0x7f5a9c14d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:46.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.663+0000 7f5aa28df640 1 --2- 172.21.15.55:0/580349475 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a9c0c60d0 0x7f5a9c147a00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:46.661 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.663+0000 7f5aa28df640 1 -- 172.21.15.55:0/580349475 >> 172.21.15.55:0/580349475 conn(0x7f5a9c0cc9c0 msgr2=0x7f5a9c0c99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:46.661 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.663+0000 7f5aa28df640 1 -- 172.21.15.55:0/580349475 shutdown_connections 2024-03-19T01:55:46.661 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:46.663+0000 7f5aa28df640 1 -- 172.21.15.55:0/580349475 wait complete. 2024-03-19T01:55:47.285 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:55:47.285 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:48.286 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:48.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.419+0000 7f51cedad640 1 Processor -- start 2024-03-19T01:55:48.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.419+0000 7f51cedad640 1 -- start start 2024-03-19T01:55:48.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.419+0000 7f51cedad640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f51c815a920 0x7f51c80c9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:48.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.419+0000 7f51cedad640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51c8070bd0 con 0x7f51c815a540 2024-03-19T01:55:48.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.419+0000 7f51cedad640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51c8070e90 con 0x7f51c815ade0 2024-03-19T01:55:48.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.419+0000 7f51cedad640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f51c806d440 con 0x7f51c815a920 2024-03-19T01:55:48.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.423+0000 7f51cd34a640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f51c815ade0 0x7f51c81431b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:40654/0 (socket says 172.21.15.55:40654) 2024-03-19T01:55:48.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.423+0000 7f51cd34a640 1 -- 172.21.15.55:0/2954175204 learned_addr learned my addr 172.21.15.55:0/2954175204 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:55:48.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.427+0000 7f51c77fe640 1 -- 172.21.15.55:0/2954175204 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2905022716 0 0) 0x7f51c8070e90 con 0x7f51c815ade0 2024-03-19T01:55:48.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.427+0000 7f51c77fe640 1 -- 172.21.15.55:0/2954175204 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51b4003530 con 0x7f51c815ade0 2024-03-19T01:55:48.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.431+0000 7f51c77fe640 1 -- 172.21.15.55:0/2954175204 <== mon.1 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2019383563 0 0) 0x7f51b4003530 con 0x7f51c815ade0 2024-03-19T01:55:48.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.431+0000 7f51c77fe640 1 -- 172.21.15.55:0/2954175204 >> v2:172.21.15.55:3300/0 conn(0x7f51c815a920 msgr2=0x7f51c80c9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:48.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.431+0000 7f51c77fe640 1 --2- 172.21.15.55:0/2954175204 >> v2:172.21.15.55:3300/0 conn(0x7f51c815a920 0x7f51c80c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:48.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.431+0000 7f51c77fe640 1 -- 172.21.15.55:0/2954175204 >> v1:172.21.15.55:6789/0 conn(0x7f51c815a540 legacy=0x7f51c806cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:48.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.431+0000 7f51c77fe640 1 -- 172.21.15.55:0/2954175204 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51c806f830 con 0x7f51c815ade0 2024-03-19T01:55:48.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.431+0000 7f51c77fe640 1 -- 172.21.15.55:0/2954175204 <== mon.1 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f51b8001680 con 0x7f51c815ade0 2024-03-19T01:55:48.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.431+0000 7f51c77fe640 1 -- 172.21.15.55:0/2954175204 >> v1:172.21.15.84:6789/0 conn(0x7f51c815ade0 legacy=0x7f51c81431b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:48.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.431+0000 7f51c77fe640 1 --2- 172.21.15.55:0/2954175204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51b4003a10 0x7f51b4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:48.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.431+0000 7f51c77fe640 1 --2- 172.21.15.55:0/2954175204 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51b4004810 0x7f51b4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:48.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.431+0000 7f51c77fe640 1 --2- 172.21.15.55:0/2954175204 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51b40052f0 0x7f51b400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:48.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.431+0000 7f51c77fe640 1 -- 172.21.15.55:0/2954175204 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f51b400c200 con 0x7f51b4003a10 2024-03-19T01:55:48.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.431+0000 7f51c77fe640 1 -- 172.21.15.55:0/2954175204 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f51b400c3b0 con 0x7f51b40052f0 2024-03-19T01:55:48.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.431+0000 7f51c77fe640 1 -- 172.21.15.55:0/2954175204 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f51b400c5b0 con 0x7f51b4004810 2024-03-19T01:55:48.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.435+0000 7f51c7fff640 1 --2- 172.21.15.55:0/2954175204 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51b4004810 0x7f51b4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:48.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.439+0000 7f51c7fff640 1 -- 172.21.15.55:0/2954175204 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51b40052f0 msgr2=0x7f51b400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:48.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.439+0000 7f51c7fff640 1 --2- 172.21.15.55:0/2954175204 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51b40052f0 0x7f51b400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:48.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.439+0000 7f51c7fff640 1 -- 172.21.15.55:0/2954175204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51b4003a10 msgr2=0x7f51b4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:48.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.439+0000 7f51c7fff640 1 --2- 172.21.15.55:0/2954175204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51b4003a10 0x7f51b4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:48.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.439+0000 7f51c7fff640 1 -- 172.21.15.55:0/2954175204 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b4003530 con 0x7f51b4004810 2024-03-19T01:55:48.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.439+0000 7f51cd34a640 1 -- 172.21.15.55:0/2954175204 reap_dead start 2024-03-19T01:55:48.438 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.443+0000 7f51c7fff640 1 --2- 172.21.15.55:0/2954175204 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51b4004810 0x7f51b4004c90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f51b0009860 tx=0x7f51b001c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f98bc1d2a698908 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:48.445 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.447+0000 7f51c77fe640 1 -- 172.21.15.55:0/2954175204 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b00b0070 con 0x7f51b4004810 2024-03-19T01:55:48.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.447+0000 7f51c77fe640 1 -- 172.21.15.55:0/2954175204 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51b00ab620 con 0x7f51b4004810 2024-03-19T01:55:48.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.447+0000 7f51c77fe640 1 -- 172.21.15.55:0/2954175204 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0009460 con 0x7f51b4004810 2024-03-19T01:55:48.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.451+0000 7f51cedad640 1 -- 172.21.15.55:0/2954175204 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51b4004810 msgr2=0x7f51b4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:48.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.451+0000 7f51cedad640 1 --2- 172.21.15.55:0/2954175204 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51b4004810 0x7f51b4004c90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f51b0009860 tx=0x7f51b001c320 comp rx=0 tx=0).stop 2024-03-19T01:55:48.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.451+0000 7f51cedad640 1 -- 172.21.15.55:0/2954175204 shutdown_connections 2024-03-19T01:55:48.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.451+0000 7f51cedad640 1 --2- 172.21.15.55:0/2954175204 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51b4004810 0x7f51b4004c90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:48.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.451+0000 7f51cedad640 1 -- 172.21.15.55:0/2954175204 >> 172.21.15.55:0/2954175204 conn(0x7f51c8067600 msgr2=0x7f51c8071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:48.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.451+0000 7f51cedad640 1 -- 172.21.15.55:0/2954175204 shutdown_connections 2024-03-19T01:55:48.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.451+0000 7f51cedad640 1 -- 172.21.15.55:0/2954175204 wait complete. 2024-03-19T01:55:48.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.451+0000 7f51cedad640 1 Processor -- start 2024-03-19T01:55:48.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.451+0000 7f51cedad640 1 -- start start 2024-03-19T01:55:48.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.451+0000 7f51cedad640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51c813d990 0x7f51c81efc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:48.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.451+0000 7f51cedad640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51c813dd70 0x7f51c80cda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:48.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.451+0000 7f51cedad640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51c81f01a0 0x7f51c80cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:48.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.451+0000 7f51cedad640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f51c806fdf0 con 0x7f51c813dd70 2024-03-19T01:55:48.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.451+0000 7f51cedad640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f51c806f5e0 con 0x7f51c813d990 2024-03-19T01:55:48.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.451+0000 7f51cedad640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f51c8070860 con 0x7f51c81f01a0 2024-03-19T01:55:48.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.451+0000 7f51c7fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51c813dd70 0x7f51c80cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:48.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.451+0000 7f51c7fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51c813dd70 0x7f51c80cda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:35774/0 (socket says 172.21.15.55:35774) 2024-03-19T01:55:48.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.451+0000 7f51c7fff640 1 -- 172.21.15.55:0/2803252314 learned_addr learned my addr 172.21.15.55:0/2803252314 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:48.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.451+0000 7f51ccb49640 1 --2- 172.21.15.55:0/2803252314 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51c813d990 0x7f51c81efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:48.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.451+0000 7f51cd34a640 1 --2- 172.21.15.55:0/2803252314 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51c81f01a0 0x7f51c80cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:48.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.455+0000 7f51c7fff640 1 -- 172.21.15.55:0/2803252314 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51c81f01a0 msgr2=0x7f51c80cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:48.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.455+0000 7f51c7fff640 1 --2- 172.21.15.55:0/2803252314 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51c81f01a0 0x7f51c80cdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:48.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.455+0000 7f51c7fff640 1 -- 172.21.15.55:0/2803252314 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51c813d990 msgr2=0x7f51c81efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:48.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.455+0000 7f51c7fff640 1 --2- 172.21.15.55:0/2803252314 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51c813d990 0x7f51c81efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:48.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.455+0000 7f51c7fff640 1 -- 172.21.15.55:0/2803252314 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51c8157c60 con 0x7f51c813dd70 2024-03-19T01:55:48.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.459+0000 7f51c7fff640 1 --2- 172.21.15.55:0/2803252314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51c813dd70 0x7f51c80cda90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f51b00ae040 tx=0x7f51b00190c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:48.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.463+0000 7f51c57fa640 1 -- 172.21.15.55:0/2803252314 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b00b0050 con 0x7f51c813dd70 2024-03-19T01:55:48.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.463+0000 7f51c57fa640 1 -- 172.21.15.55:0/2803252314 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51b00bc070 con 0x7f51c813dd70 2024-03-19T01:55:48.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.463+0000 7f51c57fa640 1 -- 172.21.15.55:0/2803252314 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0006d70 con 0x7f51c813dd70 2024-03-19T01:55:48.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.463+0000 7f51cedad640 1 -- 172.21.15.55:0/2803252314 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51c8154f40 con 0x7f51c813dd70 2024-03-19T01:55:48.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.463+0000 7f51cedad640 1 -- 172.21.15.55:0/2803252314 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51c8156ea0 con 0x7f51c813dd70 2024-03-19T01:55:48.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.467+0000 7f51cedad640 1 -- 172.21.15.55:0/2803252314 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51c8068a40 con 0x7f51c813dd70 2024-03-19T01:55:48.499 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.503+0000 7f51c57fa640 1 -- 172.21.15.55:0/2803252314 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f51b00b7a60 con 0x7f51c813dd70 2024-03-19T01:55:48.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.503+0000 7f51c57fa640 1 --2- 172.21.15.55:0/2803252314 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f51a80676e0 0x7f51a8069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:48.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.503+0000 7f51ccb49640 1 --2- 172.21.15.55:0/2803252314 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f51a80676e0 0x7f51a8069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:48.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.503+0000 7f51ccb49640 1 --2- 172.21.15.55:0/2803252314 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f51a80676e0 0x7f51a8069ba0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f51c0001f10 tx=0x7f51c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:48.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.775+0000 7f51c57fa640 1 -- 172.21.15.55:0/2803252314 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f51b00ef500 con 0x7f51c813dd70 2024-03-19T01:55:48.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.779+0000 7f51c57fa640 1 -- 172.21.15.55:0/2803252314 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f51b0016030 con 0x7f51c813dd70 2024-03-19T01:55:48.981 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:48.983+0000 7f51cedad640 1 -- 172.21.15.55:0/2803252314 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51c8142b30 con 0x7f51c813dd70 2024-03-19T01:55:49.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:49.011+0000 7f51c57fa640 1 -- 172.21.15.55:0/2803252314 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f51b00f1460 con 0x7f51c813dd70 2024-03-19T01:55:49.010 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:49.010 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:55:49.010 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:55:49.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:49.015+0000 7f51cedad640 1 -- 172.21.15.55:0/2803252314 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f51a80676e0 msgr2=0x7f51a8069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:49.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:49.015+0000 7f51cedad640 1 --2- 172.21.15.55:0/2803252314 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f51a80676e0 0x7f51a8069ba0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f51c0001f10 tx=0x7f51c0009040 comp rx=0 tx=0).stop 2024-03-19T01:55:49.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:49.015+0000 7f51cedad640 1 -- 172.21.15.55:0/2803252314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51c813dd70 msgr2=0x7f51c80cda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:49.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:49.015+0000 7f51cedad640 1 --2- 172.21.15.55:0/2803252314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51c813dd70 0x7f51c80cda90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f51b00ae040 tx=0x7f51b00190c0 comp rx=0 tx=0).stop 2024-03-19T01:55:49.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:49.015+0000 7f51cedad640 1 -- 172.21.15.55:0/2803252314 shutdown_connections 2024-03-19T01:55:49.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:49.015+0000 7f51cedad640 1 --2- 172.21.15.55:0/2803252314 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f51a80676e0 0x7f51a8069ba0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:49.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:49.015+0000 7f51cedad640 1 --2- 172.21.15.55:0/2803252314 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51c81f01a0 0x7f51c80cdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:49.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:49.015+0000 7f51cedad640 1 --2- 172.21.15.55:0/2803252314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51c813dd70 0x7f51c80cda90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:49.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:49.015+0000 7f51cedad640 1 --2- 172.21.15.55:0/2803252314 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51c813d990 0x7f51c81efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:49.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:49.015+0000 7f51cedad640 1 -- 172.21.15.55:0/2803252314 >> 172.21.15.55:0/2803252314 conn(0x7f51c8067600 msgr2=0x7f51c80cd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:49.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:49.015+0000 7f51cedad640 1 -- 172.21.15.55:0/2803252314 shutdown_connections 2024-03-19T01:55:49.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:49.015+0000 7f51cedad640 1 -- 172.21.15.55:0/2803252314 wait complete. 2024-03-19T01:55:49.632 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:55:49.632 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:50.634 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:50.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.751+0000 7fdf660f6640 1 Processor -- start 2024-03-19T01:55:50.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.751+0000 7fdf660f6640 1 -- start start 2024-03-19T01:55:50.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.755+0000 7fdf660f6640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fdf6015acb0 0x7fdf60073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:50.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.755+0000 7fdf660f6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf600c7fa0 con 0x7fdf601544f0 2024-03-19T01:55:50.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.755+0000 7fdf660f6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf600c8240 con 0x7fdf6006b2a0 2024-03-19T01:55:50.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.755+0000 7fdf660f6640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fdf6015b210 con 0x7fdf6015acb0 2024-03-19T01:55:50.752 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.755+0000 7fdf5ffff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fdf6015acb0 0x7fdf60073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:50.752 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.755+0000 7fdf5effd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fdf6006b2a0 0x7fdf600cce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:48982/0 (socket says 172.21.15.55:48982) 2024-03-19T01:55:50.752 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.755+0000 7fdf5effd640 1 -- 172.21.15.55:0/3797467029 learned_addr learned my addr 172.21.15.55:0/3797467029 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:55:50.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.759+0000 7fdf5ffff640 1 -- 172.21.15.55:0/3797467029 >> v1:172.21.15.84:6789/0 conn(0x7fdf6006b2a0 legacy=0x7fdf600cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:50.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.759+0000 7fdf5ffff640 1 -- 172.21.15.55:0/3797467029 >> v1:172.21.15.55:6789/0 conn(0x7fdf601544f0 legacy=0x7fdf6015e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:50.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.759+0000 7fdf5ffff640 1 -- 172.21.15.55:0/3797467029 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf6015d920 con 0x7fdf6015acb0 2024-03-19T01:55:50.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.763+0000 7fdf5ffff640 1 --2- 172.21.15.55:0/3797467029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf6015acb0 0x7fdf60073540 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdf58009d90 tx=0x7fdf5801f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c33ac8b783d9e2e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:50.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.767+0000 7fdf5e7fc640 1 -- 172.21.15.55:0/3797467029 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf580b4070 con 0x7fdf6015acb0 2024-03-19T01:55:50.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.767+0000 7fdf5e7fc640 1 -- 172.21.15.55:0/3797467029 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf580af600 con 0x7fdf6015acb0 2024-03-19T01:55:50.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.767+0000 7fdf5e7fc640 1 -- 172.21.15.55:0/3797467029 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf580af9d0 con 0x7fdf6015acb0 2024-03-19T01:55:50.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.771+0000 7fdf660f6640 1 -- 172.21.15.55:0/3797467029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf6015acb0 msgr2=0x7fdf60073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:50.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.771+0000 7fdf660f6640 1 --2- 172.21.15.55:0/3797467029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf6015acb0 0x7fdf60073540 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdf58009d90 tx=0x7fdf5801f150 comp rx=0 tx=0).stop 2024-03-19T01:55:50.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.771+0000 7fdf660f6640 1 -- 172.21.15.55:0/3797467029 shutdown_connections 2024-03-19T01:55:50.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.771+0000 7fdf660f6640 1 --2- 172.21.15.55:0/3797467029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf6015acb0 0x7fdf60073540 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:50.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.771+0000 7fdf660f6640 1 -- 172.21.15.55:0/3797467029 >> 172.21.15.55:0/3797467029 conn(0x7fdf6006e670 msgr2=0x7fdf6006ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:50.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.771+0000 7fdf660f6640 1 -- 172.21.15.55:0/3797467029 shutdown_connections 2024-03-19T01:55:50.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.771+0000 7fdf660f6640 1 -- 172.21.15.55:0/3797467029 wait complete. 2024-03-19T01:55:50.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.771+0000 7fdf660f6640 1 Processor -- start 2024-03-19T01:55:50.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.771+0000 7fdf660f6640 1 -- start start 2024-03-19T01:55:50.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.771+0000 7fdf660f6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdf6006b2a0 0x7fdf60157f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:50.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.771+0000 7fdf660f6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf601544f0 0x7fdf601f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:50.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.771+0000 7fdf660f6640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdf6015acb0 0x7fdf6014fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:50.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.771+0000 7fdf660f6640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdf600cd540 con 0x7fdf601544f0 2024-03-19T01:55:50.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.771+0000 7fdf660f6640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fdf601548d0 con 0x7fdf6015acb0 2024-03-19T01:55:50.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.771+0000 7fdf660f6640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fdf600cd6c0 con 0x7fdf6006b2a0 2024-03-19T01:55:50.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.771+0000 7fdf5effd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf601544f0 0x7fdf601f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:50.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.771+0000 7fdf5effd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf601544f0 0x7fdf601f3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41330/0 (socket says 172.21.15.55:41330) 2024-03-19T01:55:50.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.771+0000 7fdf5effd640 1 -- 172.21.15.55:0/846293411 learned_addr learned my addr 172.21.15.55:0/846293411 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:50.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.771+0000 7fdf5f7fe640 1 --2- 172.21.15.55:0/846293411 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdf6006b2a0 0x7fdf60157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:50.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.779+0000 7fdf5effd640 1 -- 172.21.15.55:0/846293411 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdf6006b2a0 msgr2=0x7fdf60157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:50.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.779+0000 7fdf5effd640 1 --2- 172.21.15.55:0/846293411 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdf6006b2a0 0x7fdf60157f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:50.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.779+0000 7fdf5effd640 1 -- 172.21.15.55:0/846293411 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdf6015acb0 msgr2=0x7fdf6014fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:50.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.779+0000 7fdf5effd640 1 --2- 172.21.15.55:0/846293411 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdf6015acb0 0x7fdf6014fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:50.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.779+0000 7fdf5effd640 1 -- 172.21.15.55:0/846293411 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf60158540 con 0x7fdf601544f0 2024-03-19T01:55:50.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.783+0000 7fdf5effd640 1 --2- 172.21.15.55:0/846293411 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf601544f0 0x7fdf601f3df0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400c500 tx=0x7fdf5400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:50.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.787+0000 7fdf47fff640 1 -- 172.21.15.55:0/846293411 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf54017070 con 0x7fdf601544f0 2024-03-19T01:55:50.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.787+0000 7fdf47fff640 1 -- 172.21.15.55:0/846293411 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf54005cf0 con 0x7fdf601544f0 2024-03-19T01:55:50.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.787+0000 7fdf47fff640 1 -- 172.21.15.55:0/846293411 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf540072b0 con 0x7fdf601544f0 2024-03-19T01:55:50.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.787+0000 7fdf660f6640 1 -- 172.21.15.55:0/846293411 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf60158820 con 0x7fdf601544f0 2024-03-19T01:55:50.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.787+0000 7fdf660f6640 1 -- 172.21.15.55:0/846293411 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf601f35f0 con 0x7fdf601544f0 2024-03-19T01:55:50.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:50.787+0000 7fdf660f6640 1 -- 172.21.15.55:0/846293411 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf601f33e0 con 0x7fdf601544f0 2024-03-19T01:55:51.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:51.059+0000 7fdf47fff640 1 -- 172.21.15.55:0/846293411 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fdf5401a070 con 0x7fdf601544f0 2024-03-19T01:55:51.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:51.059+0000 7fdf47fff640 1 --2- 172.21.15.55:0/846293411 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdf4c0676b0 0x7fdf4c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:51.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:51.059+0000 7fdf5f7fe640 1 --2- 172.21.15.55:0/846293411 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdf4c0676b0 0x7fdf4c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:51.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:51.059+0000 7fdf5f7fe640 1 --2- 172.21.15.55:0/846293411 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdf4c0676b0 0x7fdf4c069b70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdf50007fd0 tx=0x7fdf50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:51.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:51.087+0000 7fdf47fff640 1 -- 172.21.15.55:0/846293411 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fdf54053680 con 0x7fdf601544f0 2024-03-19T01:55:51.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:51.087+0000 7fdf47fff640 1 -- 172.21.15.55:0/846293411 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fdf54021020 con 0x7fdf601544f0 2024-03-19T01:55:51.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:51.311+0000 7fdf660f6640 1 -- 172.21.15.55:0/846293411 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf60068a40 con 0x7fdf601544f0 2024-03-19T01:55:51.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:51.327+0000 7fdf47fff640 1 -- 172.21.15.55:0/846293411 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fdf540555e0 con 0x7fdf601544f0 2024-03-19T01:55:51.324 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:51.324 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:55:51.324 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:55:51.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:51.327+0000 7fdf660f6640 1 -- 172.21.15.55:0/846293411 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdf4c0676b0 msgr2=0x7fdf4c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:51.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:51.327+0000 7fdf660f6640 1 --2- 172.21.15.55:0/846293411 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdf4c0676b0 0x7fdf4c069b70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdf50007fd0 tx=0x7fdf50009040 comp rx=0 tx=0).stop 2024-03-19T01:55:51.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:51.327+0000 7fdf660f6640 1 -- 172.21.15.55:0/846293411 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf601544f0 msgr2=0x7fdf601f3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:51.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:51.327+0000 7fdf660f6640 1 --2- 172.21.15.55:0/846293411 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf601544f0 0x7fdf601f3df0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400c500 tx=0x7fdf5400c9c0 comp rx=0 tx=0).stop 2024-03-19T01:55:51.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:51.327+0000 7fdf660f6640 1 -- 172.21.15.55:0/846293411 shutdown_connections 2024-03-19T01:55:51.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:51.327+0000 7fdf660f6640 1 --2- 172.21.15.55:0/846293411 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdf4c0676b0 0x7fdf4c069b70 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:51.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:51.327+0000 7fdf660f6640 1 --2- 172.21.15.55:0/846293411 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdf6015acb0 0x7fdf6014fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:51.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:51.327+0000 7fdf660f6640 1 --2- 172.21.15.55:0/846293411 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdf601544f0 0x7fdf601f3df0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:51.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:51.327+0000 7fdf660f6640 1 --2- 172.21.15.55:0/846293411 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdf6006b2a0 0x7fdf60157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:51.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:51.327+0000 7fdf660f6640 1 -- 172.21.15.55:0/846293411 >> 172.21.15.55:0/846293411 conn(0x7fdf6006e670 msgr2=0x7fdf600c72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:51.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:51.327+0000 7fdf660f6640 1 -- 172.21.15.55:0/846293411 shutdown_connections 2024-03-19T01:55:51.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:51.327+0000 7fdf660f6640 1 -- 172.21.15.55:0/846293411 wait complete. 2024-03-19T01:55:51.948 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:55:51.948 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:52.949 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:53.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.071+0000 7f955669b640 1 Processor -- start 2024-03-19T01:55:53.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.071+0000 7f955669b640 1 -- start start 2024-03-19T01:55:53.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.075+0000 7f955669b640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f9550070780 0x7f955015b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:53.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.075+0000 7f955669b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f955015bb00 0x7f955015bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:53.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.075+0000 7f955669b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f955015c4d0 con 0x7f9550152120 2024-03-19T01:55:53.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.075+0000 7f955669b640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f9550152500 con 0x7f9550070780 2024-03-19T01:55:53.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.075+0000 7f955669b640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f9550070590 con 0x7f955015bb00 2024-03-19T01:55:53.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.075+0000 7f9554c38640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f955015bb00 0x7f955015bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:53.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.075+0000 7f954ffff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f9550152120 0x7f95500c90d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:48984/0 (socket says 172.21.15.55:48984) 2024-03-19T01:55:53.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.075+0000 7f954ffff640 1 -- 172.21.15.55:0/313416616 learned_addr learned my addr 172.21.15.55:0/313416616 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:55:53.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.075+0000 7f954f7fe640 1 --2- 172.21.15.55:0/313416616 >> v2:172.21.15.84:3300/0 conn(0x7f9550070780 0x7f955015b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:53.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.079+0000 7f9554c38640 1 -- 172.21.15.55:0/313416616 >> v2:172.21.15.84:3300/0 conn(0x7f9550070780 msgr2=0x7f955015b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:53.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.079+0000 7f9554c38640 1 --2- 172.21.15.55:0/313416616 >> v2:172.21.15.84:3300/0 conn(0x7f9550070780 0x7f955015b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:53.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.079+0000 7f9554c38640 1 -- 172.21.15.55:0/313416616 >> v1:172.21.15.84:6789/0 conn(0x7f9550152120 legacy=0x7f95500c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:53.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.079+0000 7f9554c38640 1 -- 172.21.15.55:0/313416616 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f955006fff0 con 0x7f955015bb00 2024-03-19T01:55:53.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.083+0000 7f9554c38640 1 --2- 172.21.15.55:0/313416616 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f955015bb00 0x7f955015bf90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9548009d90 tx=0x7f954801c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc20dbed520d7d07 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:53.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.087+0000 7f954effd640 1 -- 172.21.15.55:0/313416616 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95480b2070 con 0x7f955015bb00 2024-03-19T01:55:53.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.087+0000 7f954effd640 1 -- 172.21.15.55:0/313416616 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95480a5830 con 0x7f955015bb00 2024-03-19T01:55:53.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.087+0000 7f954effd640 1 -- 172.21.15.55:0/313416616 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95480a5c00 con 0x7f955015bb00 2024-03-19T01:55:53.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.091+0000 7f955669b640 1 -- 172.21.15.55:0/313416616 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f955015bb00 msgr2=0x7f955015bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:53.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.091+0000 7f955669b640 1 --2- 172.21.15.55:0/313416616 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f955015bb00 0x7f955015bf90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9548009d90 tx=0x7f954801c970 comp rx=0 tx=0).stop 2024-03-19T01:55:53.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.091+0000 7f955669b640 1 -- 172.21.15.55:0/313416616 shutdown_connections 2024-03-19T01:55:53.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.091+0000 7f955669b640 1 --2- 172.21.15.55:0/313416616 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f955015bb00 0x7f955015bf90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:53.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.091+0000 7f955669b640 1 --2- 172.21.15.55:0/313416616 >> v2:172.21.15.84:3300/0 conn(0x7f9550070780 0x7f955015b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:53.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.091+0000 7f955669b640 1 -- 172.21.15.55:0/313416616 >> 172.21.15.55:0/313416616 conn(0x7f9550067600 msgr2=0x7f9550071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:53.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.091+0000 7f955669b640 1 -- 172.21.15.55:0/313416616 shutdown_connections 2024-03-19T01:55:53.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.091+0000 7f955669b640 1 -- 172.21.15.55:0/313416616 wait complete. 2024-03-19T01:55:53.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.091+0000 7f955669b640 1 Processor -- start 2024-03-19T01:55:53.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.091+0000 7f955669b640 1 -- start start 2024-03-19T01:55:53.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.091+0000 7f955669b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9550070780 0x7f955015a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:53.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.091+0000 7f955669b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9550152120 0x7f95500cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:53.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.091+0000 7f955669b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f955015bb00 0x7f95500cc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:53.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.091+0000 7f955669b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f95500c31c0 con 0x7f9550152120 2024-03-19T01:55:53.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.091+0000 7f955669b640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f95500c2ee0 con 0x7f955015bb00 2024-03-19T01:55:53.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.091+0000 7f955669b640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f95500c97e0 con 0x7f9550070780 2024-03-19T01:55:53.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.095+0000 7f954f7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9550152120 0x7f95500cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:53.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.095+0000 7f9554c38640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f955015bb00 0x7f95500cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:53.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.095+0000 7f954ffff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9550070780 0x7f955015a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:53.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.095+0000 7f954f7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9550152120 0x7f95500cbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41344/0 (socket says 172.21.15.55:41344) 2024-03-19T01:55:53.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.095+0000 7f954f7fe640 1 -- 172.21.15.55:0/1869353459 learned_addr learned my addr 172.21.15.55:0/1869353459 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:53.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.099+0000 7f954f7fe640 1 -- 172.21.15.55:0/1869353459 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9550070780 msgr2=0x7f955015a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:53.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.099+0000 7f954f7fe640 1 --2- 172.21.15.55:0/1869353459 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9550070780 0x7f955015a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:53.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.099+0000 7f954f7fe640 1 -- 172.21.15.55:0/1869353459 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f955015bb00 msgr2=0x7f95500cc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:53.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.099+0000 7f954f7fe640 1 --2- 172.21.15.55:0/1869353459 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f955015bb00 0x7f95500cc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:53.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.099+0000 7f954f7fe640 1 -- 172.21.15.55:0/1869353459 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95501f5220 con 0x7f9550152120 2024-03-19T01:55:53.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.103+0000 7f954f7fe640 1 --2- 172.21.15.55:0/1869353459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9550152120 0x7f95500cbfe0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f954400c7f0 tx=0x7f954400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:53.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.107+0000 7f954cff9640 1 -- 172.21.15.55:0/1869353459 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9544015070 con 0x7f9550152120 2024-03-19T01:55:53.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.107+0000 7f954cff9640 1 -- 172.21.15.55:0/1869353459 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9544004240 con 0x7f9550152120 2024-03-19T01:55:53.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.107+0000 7f955669b640 1 -- 172.21.15.55:0/1869353459 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9550152b10 con 0x7f9550152120 2024-03-19T01:55:53.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.107+0000 7f954cff9640 1 -- 172.21.15.55:0/1869353459 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9544003620 con 0x7f9550152120 2024-03-19T01:55:53.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.107+0000 7f955669b640 1 -- 172.21.15.55:0/1869353459 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95500cbac0 con 0x7f9550152120 2024-03-19T01:55:53.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.107+0000 7f955669b640 1 -- 172.21.15.55:0/1869353459 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95500cbcc0 con 0x7f9550152120 2024-03-19T01:55:53.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.391+0000 7f954cff9640 1 -- 172.21.15.55:0/1869353459 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f95440039d0 con 0x7f9550152120 2024-03-19T01:55:53.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.391+0000 7f954cff9640 1 --2- 172.21.15.55:0/1869353459 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f953c067690 0x7f953c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:53.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.391+0000 7f954ffff640 1 --2- 172.21.15.55:0/1869353459 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f953c067690 0x7f953c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:53.388 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.391+0000 7f954ffff640 1 --2- 172.21.15.55:0/1869353459 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f953c067690 0x7f953c069b50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9540000f70 tx=0x7f9540007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:53.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.431+0000 7f954cff9640 1 -- 172.21.15.55:0/1869353459 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f95440543c0 con 0x7f9550152120 2024-03-19T01:55:53.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.431+0000 7f954cff9640 1 -- 172.21.15.55:0/1869353459 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f95500cbcc0 con 0x7f9550152120 2024-03-19T01:55:53.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.759+0000 7f955669b640 1 -- 172.21.15.55:0/1869353459 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95500c84b0 con 0x7f9550152120 2024-03-19T01:55:53.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.779+0000 7f954cff9640 1 -- 172.21.15.55:0/1869353459 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f9544056320 con 0x7f9550152120 2024-03-19T01:55:53.778 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:53.778 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:55:53.779 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:55:53.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.783+0000 7f955669b640 1 -- 172.21.15.55:0/1869353459 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f953c067690 msgr2=0x7f953c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:53.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.783+0000 7f955669b640 1 --2- 172.21.15.55:0/1869353459 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f953c067690 0x7f953c069b50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9540000f70 tx=0x7f9540007e90 comp rx=0 tx=0).stop 2024-03-19T01:55:53.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.783+0000 7f955669b640 1 -- 172.21.15.55:0/1869353459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9550152120 msgr2=0x7f95500cbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:53.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.783+0000 7f955669b640 1 --2- 172.21.15.55:0/1869353459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9550152120 0x7f95500cbfe0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f954400c7f0 tx=0x7f954400ccb0 comp rx=0 tx=0).stop 2024-03-19T01:55:53.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.783+0000 7f955669b640 1 -- 172.21.15.55:0/1869353459 shutdown_connections 2024-03-19T01:55:53.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.783+0000 7f955669b640 1 --2- 172.21.15.55:0/1869353459 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f953c067690 0x7f953c069b50 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:53.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.783+0000 7f955669b640 1 --2- 172.21.15.55:0/1869353459 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f955015bb00 0x7f95500cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:53.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.783+0000 7f955669b640 1 --2- 172.21.15.55:0/1869353459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9550152120 0x7f95500cbfe0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:53.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.783+0000 7f955669b640 1 --2- 172.21.15.55:0/1869353459 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9550070780 0x7f955015a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:53.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.783+0000 7f955669b640 1 -- 172.21.15.55:0/1869353459 >> 172.21.15.55:0/1869353459 conn(0x7f9550067600 msgr2=0x7f95500c86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:53.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.783+0000 7f955669b640 1 -- 172.21.15.55:0/1869353459 shutdown_connections 2024-03-19T01:55:53.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:53.783+0000 7f955669b640 1 -- 172.21.15.55:0/1869353459 wait complete. 2024-03-19T01:55:54.400 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:55:54.400 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:55.401 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:55.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.519+0000 7f6191135640 1 Processor -- start 2024-03-19T01:55:55.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.519+0000 7f6191135640 1 -- start start 2024-03-19T01:55:55.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.519+0000 7f6191135640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f618c14eaa0 0x7f618c06b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:55.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.519+0000 7f6191135640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f618c06bec0 0x7f618c0c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:55.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.519+0000 7f6191135640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f618c0c6610 0x7f618c1585f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:55.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.519+0000 7f6191135640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f618c14ee80 con 0x7f618c0c6610 2024-03-19T01:55:55.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.519+0000 7f6191135640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f618c0c7540 con 0x7f618c14eaa0 2024-03-19T01:55:55.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.519+0000 7f6191135640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f618c0c77c0 con 0x7f618c06bec0 2024-03-19T01:55:55.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.519+0000 7f618a59c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f618c06bec0 0x7f618c0c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:55.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.519+0000 7f618b59e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f618c0c6610 0x7f618c1585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:55.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.523+0000 7f618ad9d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f618c14eaa0 0x7f618c06b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:55.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.523+0000 7f618b59e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f618c0c6610 0x7f618c1585f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41352/0 (socket says 172.21.15.55:41352) 2024-03-19T01:55:55.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.523+0000 7f618b59e640 1 -- 172.21.15.55:0/1489873770 learned_addr learned my addr 172.21.15.55:0/1489873770 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:55.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.523+0000 7f618b59e640 1 -- 172.21.15.55:0/1489873770 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f618c06bec0 msgr2=0x7f618c0c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:55.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.523+0000 7f618b59e640 1 --2- 172.21.15.55:0/1489873770 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f618c06bec0 0x7f618c0c60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:55.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.523+0000 7f618b59e640 1 -- 172.21.15.55:0/1489873770 >> v2:172.21.15.84:3300/0 conn(0x7f618c14eaa0 msgr2=0x7f618c06b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:55.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.523+0000 7f618b59e640 1 --2- 172.21.15.55:0/1489873770 >> v2:172.21.15.84:3300/0 conn(0x7f618c14eaa0 0x7f618c06b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:55.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.523+0000 7f618b59e640 1 -- 172.21.15.55:0/1489873770 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618c0c7aa0 con 0x7f618c0c6610 2024-03-19T01:55:55.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.531+0000 7f618b59e640 1 --2- 172.21.15.55:0/1489873770 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f618c0c6610 0x7f618c1585f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f618c06d470 tx=0x7f618401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ad417fb8996676f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:55.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.535+0000 7f6189d9b640 1 -- 172.21.15.55:0/1489873770 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61840a56f0 con 0x7f618c0c6610 2024-03-19T01:55:55.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.535+0000 7f6189d9b640 1 -- 172.21.15.55:0/1489873770 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61840a5890 con 0x7f618c0c6610 2024-03-19T01:55:55.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.535+0000 7f6189d9b640 1 -- 172.21.15.55:0/1489873770 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61840a5c60 con 0x7f618c0c6610 2024-03-19T01:55:55.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.535+0000 7f6191135640 1 -- 172.21.15.55:0/1489873770 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f618c0c6610 msgr2=0x7f618c1585f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:55.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.535+0000 7f6191135640 1 --2- 172.21.15.55:0/1489873770 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f618c0c6610 0x7f618c1585f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f618c06d470 tx=0x7f618401c960 comp rx=0 tx=0).stop 2024-03-19T01:55:55.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.535+0000 7f6191135640 1 -- 172.21.15.55:0/1489873770 shutdown_connections 2024-03-19T01:55:55.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.535+0000 7f6191135640 1 --2- 172.21.15.55:0/1489873770 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f618c06bec0 0x7f618c0c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:55.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.535+0000 7f6191135640 1 --2- 172.21.15.55:0/1489873770 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f618c0c6610 0x7f618c1585f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:55.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.535+0000 7f6191135640 1 --2- 172.21.15.55:0/1489873770 >> v2:172.21.15.84:3300/0 conn(0x7f618c14eaa0 0x7f618c06b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:55.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.535+0000 7f6191135640 1 -- 172.21.15.55:0/1489873770 >> 172.21.15.55:0/1489873770 conn(0x7f618c0cc9c0 msgr2=0x7f618c0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:55.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.535+0000 7f6191135640 1 -- 172.21.15.55:0/1489873770 shutdown_connections 2024-03-19T01:55:55.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.535+0000 7f6191135640 1 -- 172.21.15.55:0/1489873770 wait complete. 2024-03-19T01:55:55.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.535+0000 7f6191135640 1 Processor -- start 2024-03-19T01:55:55.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.535+0000 7f6191135640 1 -- start start 2024-03-19T01:55:55.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.539+0000 7f6191135640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f618c06bec0 0x7f618c160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:55.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.539+0000 7f6191135640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f618c0c6610 0x7f618c147d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:55.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.539+0000 7f6191135640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f618c14eaa0 0x7f618c0cb870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:55.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.539+0000 7f6191135640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f618c0c6bf0 con 0x7f618c06bec0 2024-03-19T01:55:55.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.539+0000 7f6191135640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f618c158f40 con 0x7f618c0c6610 2024-03-19T01:55:55.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.539+0000 7f6191135640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f618c0c6d70 con 0x7f618c14eaa0 2024-03-19T01:55:55.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.539+0000 7f618ad9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f618c06bec0 0x7f618c160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:55.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.539+0000 7f618b59e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f618c14eaa0 0x7f618c0cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:55.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.539+0000 7f618ad9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f618c06bec0 0x7f618c160ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41356/0 (socket says 172.21.15.55:41356) 2024-03-19T01:55:55.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.539+0000 7f618ad9d640 1 -- 172.21.15.55:0/1072814510 learned_addr learned my addr 172.21.15.55:0/1072814510 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:55.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.539+0000 7f618a59c640 1 --2- 172.21.15.55:0/1072814510 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f618c0c6610 0x7f618c147d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:55.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.543+0000 7f618ad9d640 1 -- 172.21.15.55:0/1072814510 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f618c14eaa0 msgr2=0x7f618c0cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:55.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.543+0000 7f618ad9d640 1 --2- 172.21.15.55:0/1072814510 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f618c14eaa0 0x7f618c0cb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:55.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.543+0000 7f618ad9d640 1 -- 172.21.15.55:0/1072814510 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f618c0c6610 msgr2=0x7f618c147d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:55.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.543+0000 7f618ad9d640 1 --2- 172.21.15.55:0/1072814510 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f618c0c6610 0x7f618c147d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:55.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.543+0000 7f618ad9d640 1 -- 172.21.15.55:0/1072814510 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618c162290 con 0x7f618c06bec0 2024-03-19T01:55:55.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.543+0000 7f618ad9d640 1 --2- 172.21.15.55:0/1072814510 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f618c06bec0 0x7f618c160ad0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f617c00e8a0 tx=0x7f617c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:55.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.547+0000 7f61737fe640 1 -- 172.21.15.55:0/1072814510 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c017070 con 0x7f618c06bec0 2024-03-19T01:55:55.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.547+0000 7f61737fe640 1 -- 172.21.15.55:0/1072814510 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f617c00c940 con 0x7f618c06bec0 2024-03-19T01:55:55.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.547+0000 7f61737fe640 1 -- 172.21.15.55:0/1072814510 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c00ccf0 con 0x7f618c06bec0 2024-03-19T01:55:55.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.547+0000 7f6191135640 1 -- 172.21.15.55:0/1072814510 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f618c162570 con 0x7f618c06bec0 2024-03-19T01:55:55.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.547+0000 7f6191135640 1 -- 172.21.15.55:0/1072814510 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f618c141db0 con 0x7f618c06bec0 2024-03-19T01:55:55.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.551+0000 7f6191135640 1 -- 172.21.15.55:0/1072814510 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f618c068a40 con 0x7f618c06bec0 2024-03-19T01:55:55.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.843+0000 7f61737fe640 1 -- 172.21.15.55:0/1072814510 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f617c01a070 con 0x7f618c06bec0 2024-03-19T01:55:55.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.843+0000 7f61737fe640 1 --2- 172.21.15.55:0/1072814510 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f61780676b0 0x7f6178069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:55.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.843+0000 7f618a59c640 1 --2- 172.21.15.55:0/1072814510 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f61780676b0 0x7f6178069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:55.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.843+0000 7f618a59c640 1 --2- 172.21.15.55:0/1072814510 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f61780676b0 0x7f6178069b70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f618c140fe0 tx=0x7f6180009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:55.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.875+0000 7f61737fe640 1 -- 172.21.15.55:0/1072814510 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f617c0545b0 con 0x7f618c06bec0 2024-03-19T01:55:55.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:55.875+0000 7f61737fe640 1 -- 172.21.15.55:0/1072814510 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f617c056510 con 0x7f618c06bec0 2024-03-19T01:55:56.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:56.079+0000 7f6191135640 1 -- 172.21.15.55:0/1072814510 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f618c0c2170 con 0x7f618c06bec0 2024-03-19T01:55:56.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:56.111+0000 7f61737fe640 1 -- 172.21.15.55:0/1072814510 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f617c05a5b0 con 0x7f618c06bec0 2024-03-19T01:55:56.107 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:56.108 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:55:56.108 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:55:56.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:56.111+0000 7f6191135640 1 -- 172.21.15.55:0/1072814510 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f61780676b0 msgr2=0x7f6178069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:56.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:56.111+0000 7f6191135640 1 --2- 172.21.15.55:0/1072814510 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f61780676b0 0x7f6178069b70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f618c140fe0 tx=0x7f6180009190 comp rx=0 tx=0).stop 2024-03-19T01:55:56.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:56.111+0000 7f6191135640 1 -- 172.21.15.55:0/1072814510 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f618c06bec0 msgr2=0x7f618c160ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:56.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:56.111+0000 7f6191135640 1 --2- 172.21.15.55:0/1072814510 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f618c06bec0 0x7f618c160ad0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f617c00e8a0 tx=0x7f617c00eca0 comp rx=0 tx=0).stop 2024-03-19T01:55:56.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:56.111+0000 7f6191135640 1 -- 172.21.15.55:0/1072814510 shutdown_connections 2024-03-19T01:55:56.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:56.111+0000 7f6191135640 1 --2- 172.21.15.55:0/1072814510 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f61780676b0 0x7f6178069b70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:56.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:56.111+0000 7f6191135640 1 --2- 172.21.15.55:0/1072814510 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f618c14eaa0 0x7f618c0cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:56.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:56.111+0000 7f6191135640 1 --2- 172.21.15.55:0/1072814510 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f618c0c6610 0x7f618c147d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:56.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:56.111+0000 7f6191135640 1 --2- 172.21.15.55:0/1072814510 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f618c06bec0 0x7f618c160ad0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:56.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:56.111+0000 7f6191135640 1 -- 172.21.15.55:0/1072814510 >> 172.21.15.55:0/1072814510 conn(0x7f618c0cc9c0 msgr2=0x7f618c14a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:56.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:56.111+0000 7f6191135640 1 -- 172.21.15.55:0/1072814510 shutdown_connections 2024-03-19T01:55:56.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:56.111+0000 7f6191135640 1 -- 172.21.15.55:0/1072814510 wait complete. 2024-03-19T01:55:56.730 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:55:56.730 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:55:57.731 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:55:57.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.847+0000 7f9aca6a4640 1 Processor -- start 2024-03-19T01:55:57.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.847+0000 7f9aca6a4640 1 -- start start 2024-03-19T01:55:57.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.847+0000 7f9aca6a4640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f9ac4152120 0x7f9ac40c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:57.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.847+0000 7f9aca6a4640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9ac40c9610 0x7f9ac4157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:57.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.847+0000 7f9aca6a4640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ac406fff0 con 0x7f9ac4070780 2024-03-19T01:55:57.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.847+0000 7f9aca6a4640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f9ac4152500 con 0x7f9ac4152120 2024-03-19T01:55:57.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.847+0000 7f9aca6a4640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f9ac4070590 con 0x7f9ac40c9610 2024-03-19T01:55:57.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.847+0000 7f9ac37fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9ac40c9610 0x7f9ac4157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:57.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.851+0000 7f9ac3fff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f9ac4152120 0x7f9ac40c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:57.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.851+0000 7f9ac37fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9ac40c9610 0x7f9ac4157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:38676/0 (socket says 172.21.15.55:38676) 2024-03-19T01:55:57.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.851+0000 7f9ac37fe640 1 -- 172.21.15.55:0/2708464705 learned_addr learned my addr 172.21.15.55:0/2708464705 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:57.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.855+0000 7f9ac3fff640 1 -- 172.21.15.55:0/2708464705 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9ac40c9610 msgr2=0x7f9ac4157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:55:57.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.855+0000 7f9ac3fff640 1 --2- 172.21.15.55:0/2708464705 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9ac40c9610 0x7f9ac4157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:57.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.855+0000 7f9ac3fff640 1 -- 172.21.15.55:0/2708464705 >> v1:172.21.15.84:6789/0 conn(0x7f9ac4070780 legacy=0x7f9ac4158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:57.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.855+0000 7f9ac3fff640 1 -- 172.21.15.55:0/2708464705 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac40c39c0 con 0x7f9ac4152120 2024-03-19T01:55:57.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.863+0000 7f9ac3fff640 1 --2- 172.21.15.55:0/2708464705 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ac4152120 0x7f9ac40c90d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9ab40097c0 tx=0x7f9ab401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d0f3195ecc4d94a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:57.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.871+0000 7f9ac2ffd640 1 -- 172.21.15.55:0/2708464705 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab40b0070 con 0x7f9ac4152120 2024-03-19T01:55:57.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.871+0000 7f9ac2ffd640 1 -- 172.21.15.55:0/2708464705 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ab40ab600 con 0x7f9ac4152120 2024-03-19T01:55:57.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.871+0000 7f9ac2ffd640 1 -- 172.21.15.55:0/2708464705 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab40ab9d0 con 0x7f9ac4152120 2024-03-19T01:55:57.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.871+0000 7f9aca6a4640 1 -- 172.21.15.55:0/2708464705 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ac4152120 msgr2=0x7f9ac40c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:57.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.871+0000 7f9aca6a4640 1 --2- 172.21.15.55:0/2708464705 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ac4152120 0x7f9ac40c90d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9ab40097c0 tx=0x7f9ab401c4a0 comp rx=0 tx=0).stop 2024-03-19T01:55:57.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.875+0000 7f9aca6a4640 1 -- 172.21.15.55:0/2708464705 shutdown_connections 2024-03-19T01:55:57.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.875+0000 7f9aca6a4640 1 --2- 172.21.15.55:0/2708464705 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9ac40c9610 0x7f9ac4157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:57.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.875+0000 7f9aca6a4640 1 --2- 172.21.15.55:0/2708464705 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ac4152120 0x7f9ac40c90d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:57.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.875+0000 7f9aca6a4640 1 -- 172.21.15.55:0/2708464705 >> 172.21.15.55:0/2708464705 conn(0x7f9ac4067600 msgr2=0x7f9ac4071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:57.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.875+0000 7f9aca6a4640 1 -- 172.21.15.55:0/2708464705 shutdown_connections 2024-03-19T01:55:57.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.875+0000 7f9aca6a4640 1 -- 172.21.15.55:0/2708464705 wait complete. 2024-03-19T01:55:57.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.875+0000 7f9aca6a4640 1 Processor -- start 2024-03-19T01:55:57.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.875+0000 7f9aca6a4640 1 -- start start 2024-03-19T01:55:57.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.875+0000 7f9aca6a4640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9ac4070780 0x7f9ac40cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:57.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.875+0000 7f9aca6a4640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ac40c9610 0x7f9ac41f1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:57.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.875+0000 7f9aca6a4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9ac4152120 0x7f9ac40c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:57.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.875+0000 7f9aca6a4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac40c86d0 con 0x7f9ac4152120 2024-03-19T01:55:57.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.875+0000 7f9aca6a4640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac40c31c0 con 0x7f9ac40c9610 2024-03-19T01:55:57.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.875+0000 7f9aca6a4640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f9ac4165170 con 0x7f9ac4070780 2024-03-19T01:55:57.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.875+0000 7f9ac8c41640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9ac4152120 0x7f9ac40c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:57.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.879+0000 7f9ac37fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ac40c9610 0x7f9ac41f1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:57.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.879+0000 7f9ac8c41640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9ac4152120 0x7f9ac40c5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41362/0 (socket says 172.21.15.55:41362) 2024-03-19T01:55:57.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.879+0000 7f9ac8c41640 1 -- 172.21.15.55:0/3310923061 learned_addr learned my addr 172.21.15.55:0/3310923061 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:55:57.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.879+0000 7f9ac3fff640 1 --2- 172.21.15.55:0/3310923061 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9ac4070780 0x7f9ac40cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:57.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.883+0000 7f9ac8c41640 1 -- 172.21.15.55:0/3310923061 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9ac4070780 msgr2=0x7f9ac40cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:57.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.883+0000 7f9ac8c41640 1 --2- 172.21.15.55:0/3310923061 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9ac4070780 0x7f9ac40cdae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:57.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.883+0000 7f9ac8c41640 1 -- 172.21.15.55:0/3310923061 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ac40c9610 msgr2=0x7f9ac41f1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:57.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.883+0000 7f9ac8c41640 1 --2- 172.21.15.55:0/3310923061 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ac40c9610 0x7f9ac41f1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:57.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.883+0000 7f9ac8c41640 1 -- 172.21.15.55:0/3310923061 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac40ce310 con 0x7f9ac4152120 2024-03-19T01:55:57.881 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.883+0000 7f9ac8c41640 1 --2- 172.21.15.55:0/3310923061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9ac4152120 0x7f9ac40c5e70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00c770 tx=0x7f9abc00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:57.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.891+0000 7f9ac0ff9640 1 -- 172.21.15.55:0/3310923061 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc003b80 con 0x7f9ac4152120 2024-03-19T01:55:57.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.891+0000 7f9ac0ff9640 1 -- 172.21.15.55:0/3310923061 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9abc003d20 con 0x7f9ac4152120 2024-03-19T01:55:57.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.891+0000 7f9aca6a4640 1 -- 172.21.15.55:0/3310923061 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ac40ce5f0 con 0x7f9ac4152120 2024-03-19T01:55:57.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.891+0000 7f9ac0ff9640 1 -- 172.21.15.55:0/3310923061 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc00aa90 con 0x7f9ac4152120 2024-03-19T01:55:57.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.891+0000 7f9aca6a4640 1 -- 172.21.15.55:0/3310923061 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ac40cd770 con 0x7f9ac4152120 2024-03-19T01:55:57.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.891+0000 7f9aca6a4640 1 -- 172.21.15.55:0/3310923061 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ac4068a40 con 0x7f9ac4152120 2024-03-19T01:55:57.913 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.915+0000 7f9ac0ff9640 1 -- 172.21.15.55:0/3310923061 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f9abc00ac30 con 0x7f9ac4152120 2024-03-19T01:55:57.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.915+0000 7f9ac0ff9640 1 --2- 172.21.15.55:0/3310923061 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9ab0067690 0x7f9ab0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:55:57.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.919+0000 7f9ac3fff640 1 --2- 172.21.15.55:0/3310923061 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9ab0067690 0x7f9ab0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:55:57.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.919+0000 7f9ac3fff640 1 --2- 172.21.15.55:0/3310923061 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9ab0067690 0x7f9ab0069b50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9ab40ad000 tx=0x7f9ab40a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:55:57.917 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:57.919+0000 7f9ac0ff9640 1 -- 172.21.15.55:0/3310923061 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f9abc054830 con 0x7f9ac4152120 2024-03-19T01:55:58.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:58.191+0000 7f9ac0ff9640 1 -- 172.21.15.55:0/3310923061 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f9abc056790 con 0x7f9ac4152120 2024-03-19T01:55:58.400 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:58.403+0000 7f9aca6a4640 1 -- 172.21.15.55:0/3310923061 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ac40c84b0 con 0x7f9ac4152120 2024-03-19T01:55:58.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:58.423+0000 7f9ac0ff9640 1 -- 172.21.15.55:0/3310923061 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f9abc05a830 con 0x7f9ac4152120 2024-03-19T01:55:58.422 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:55:58.422 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:55:58.422 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:55:58.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:58.427+0000 7f9aca6a4640 1 -- 172.21.15.55:0/3310923061 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9ab0067690 msgr2=0x7f9ab0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:58.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:58.427+0000 7f9aca6a4640 1 --2- 172.21.15.55:0/3310923061 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9ab0067690 0x7f9ab0069b50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9ab40ad000 tx=0x7f9ab40a6000 comp rx=0 tx=0).stop 2024-03-19T01:55:58.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:58.427+0000 7f9aca6a4640 1 -- 172.21.15.55:0/3310923061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9ac4152120 msgr2=0x7f9ac40c5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:55:58.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:58.427+0000 7f9aca6a4640 1 --2- 172.21.15.55:0/3310923061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9ac4152120 0x7f9ac40c5e70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00c770 tx=0x7f9abc00cc30 comp rx=0 tx=0).stop 2024-03-19T01:55:58.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:58.427+0000 7f9aca6a4640 1 -- 172.21.15.55:0/3310923061 shutdown_connections 2024-03-19T01:55:58.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:58.427+0000 7f9aca6a4640 1 --2- 172.21.15.55:0/3310923061 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9ab0067690 0x7f9ab0069b50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:58.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:58.427+0000 7f9aca6a4640 1 --2- 172.21.15.55:0/3310923061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9ac4152120 0x7f9ac40c5e70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:58.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:58.427+0000 7f9aca6a4640 1 --2- 172.21.15.55:0/3310923061 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ac40c9610 0x7f9ac41f1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:58.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:58.427+0000 7f9aca6a4640 1 --2- 172.21.15.55:0/3310923061 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9ac4070780 0x7f9ac40cdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:55:58.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:58.427+0000 7f9aca6a4640 1 -- 172.21.15.55:0/3310923061 >> 172.21.15.55:0/3310923061 conn(0x7f9ac4067600 msgr2=0x7f9ac40ca660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:55:58.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:58.427+0000 7f9aca6a4640 1 -- 172.21.15.55:0/3310923061 shutdown_connections 2024-03-19T01:55:58.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:55:58.427+0000 7f9aca6a4640 1 -- 172.21.15.55:0/3310923061 wait complete. 2024-03-19T01:55:59.047 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:55:59.047 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:00.047 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:00.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.163+0000 7f107f8e5640 1 Processor -- start 2024-03-19T01:56:00.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.163+0000 7f107f8e5640 1 -- start start 2024-03-19T01:56:00.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.163+0000 7f107f8e5640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f107813de70 0x7f107806e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:00.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.163+0000 7f107f8e5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f107806eb70 0x7f10780c5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:00.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.163+0000 7f107f8e5640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f107806f1b0 0x7f10780c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:00.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.163+0000 7f107f8e5640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f107813e250 con 0x7f107806eb70 2024-03-19T01:56:00.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.163+0000 7f107f8e5640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f107815b740 con 0x7f107813de70 2024-03-19T01:56:00.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.163+0000 7f107f8e5640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f107815ba20 con 0x7f107806f1b0 2024-03-19T01:56:00.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.167+0000 7f107ce80640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f107806eb70 0x7f10780c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:00.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.167+0000 7f107d681640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f107813de70 0x7f107806e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:00.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.167+0000 7f107ce80640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f107806eb70 0x7f10780c5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51568/0 (socket says 172.21.15.55:51568) 2024-03-19T01:56:00.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.167+0000 7f107ce80640 1 -- 172.21.15.55:0/3684568678 learned_addr learned my addr 172.21.15.55:0/3684568678 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:00.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.167+0000 7f107de82640 1 --2- 172.21.15.55:0/3684568678 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f107806f1b0 0x7f10780c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:00.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.167+0000 7f107ce80640 1 -- 172.21.15.55:0/3684568678 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f107806f1b0 msgr2=0x7f10780c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:00.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.167+0000 7f107ce80640 1 --2- 172.21.15.55:0/3684568678 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f107806f1b0 0x7f10780c5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:00.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.167+0000 7f107ce80640 1 -- 172.21.15.55:0/3684568678 >> v2:172.21.15.84:3300/0 conn(0x7f107813de70 msgr2=0x7f107806e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:00.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.167+0000 7f107ce80640 1 --2- 172.21.15.55:0/3684568678 >> v2:172.21.15.84:3300/0 conn(0x7f107813de70 0x7f107806e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:00.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.167+0000 7f107ce80640 1 -- 172.21.15.55:0/3684568678 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f107815bd00 con 0x7f107806eb70 2024-03-19T01:56:00.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.171+0000 7f107ce80640 1 --2- 172.21.15.55:0/3684568678 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f107806eb70 0x7f10780c5810 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f10780c6540 tx=0x7f106801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71ce07d2d390f8f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:00.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.175+0000 7f106ffff640 1 -- 172.21.15.55:0/3684568678 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10680b5070 con 0x7f107806eb70 2024-03-19T01:56:00.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.175+0000 7f106ffff640 1 -- 172.21.15.55:0/3684568678 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10680a89d0 con 0x7f107806eb70 2024-03-19T01:56:00.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.175+0000 7f106ffff640 1 -- 172.21.15.55:0/3684568678 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10680a8da0 con 0x7f107806eb70 2024-03-19T01:56:00.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.175+0000 7f107f8e5640 1 -- 172.21.15.55:0/3684568678 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f107806eb70 msgr2=0x7f10780c5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:00.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.175+0000 7f107f8e5640 1 --2- 172.21.15.55:0/3684568678 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f107806eb70 0x7f10780c5810 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f10780c6540 tx=0x7f106801ed10 comp rx=0 tx=0).stop 2024-03-19T01:56:00.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.175+0000 7f107f8e5640 1 -- 172.21.15.55:0/3684568678 shutdown_connections 2024-03-19T01:56:00.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.175+0000 7f107f8e5640 1 --2- 172.21.15.55:0/3684568678 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f107806f1b0 0x7f10780c5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:00.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.175+0000 7f107f8e5640 1 --2- 172.21.15.55:0/3684568678 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f107806eb70 0x7f10780c5810 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:00.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.175+0000 7f107f8e5640 1 --2- 172.21.15.55:0/3684568678 >> v2:172.21.15.84:3300/0 conn(0x7f107813de70 0x7f107806e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:00.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.175+0000 7f107f8e5640 1 -- 172.21.15.55:0/3684568678 >> 172.21.15.55:0/3684568678 conn(0x7f107806fb40 msgr2=0x7f107806ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:00.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.175+0000 7f107f8e5640 1 -- 172.21.15.55:0/3684568678 shutdown_connections 2024-03-19T01:56:00.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.175+0000 7f107f8e5640 1 -- 172.21.15.55:0/3684568678 wait complete. 2024-03-19T01:56:00.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.179+0000 7f107f8e5640 1 Processor -- start 2024-03-19T01:56:00.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.179+0000 7f107f8e5640 1 -- start start 2024-03-19T01:56:00.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.179+0000 7f107f8e5640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f107806eb70 0x7f107806b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:00.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.179+0000 7f107f8e5640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f107806f1b0 0x7f10781f3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:00.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.179+0000 7f107f8e5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f107813de70 0x7f107814f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:00.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.179+0000 7f107f8e5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1078158420 con 0x7f107813de70 2024-03-19T01:56:00.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.179+0000 7f107f8e5640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f107815c2c0 con 0x7f107806eb70 2024-03-19T01:56:00.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.179+0000 7f107f8e5640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f10781585a0 con 0x7f107806f1b0 2024-03-19T01:56:00.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.179+0000 7f107de82640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f107813de70 0x7f107814f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:00.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.179+0000 7f107d681640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f107806eb70 0x7f107806b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:00.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.179+0000 7f107de82640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f107813de70 0x7f107814f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51572/0 (socket says 172.21.15.55:51572) 2024-03-19T01:56:00.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.179+0000 7f107de82640 1 -- 172.21.15.55:0/3866931682 learned_addr learned my addr 172.21.15.55:0/3866931682 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:00.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.183+0000 7f107ce80640 1 --2- 172.21.15.55:0/3866931682 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f107806f1b0 0x7f10781f3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:00.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.183+0000 7f107de82640 1 -- 172.21.15.55:0/3866931682 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f107806f1b0 msgr2=0x7f10781f3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:00.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.183+0000 7f107de82640 1 --2- 172.21.15.55:0/3866931682 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f107806f1b0 0x7f10781f3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:00.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.183+0000 7f107de82640 1 -- 172.21.15.55:0/3866931682 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f107806eb70 msgr2=0x7f107806b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:00.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.183+0000 7f107de82640 1 --2- 172.21.15.55:0/3866931682 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f107806eb70 0x7f107806b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:00.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.183+0000 7f107de82640 1 -- 172.21.15.55:0/3866931682 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10781573d0 con 0x7f107813de70 2024-03-19T01:56:00.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.187+0000 7f107de82640 1 --2- 172.21.15.55:0/3866931682 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f107813de70 0x7f107814f930 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f107000ef00 tx=0x7f107000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:00.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.191+0000 7f106dffb640 1 -- 172.21.15.55:0/3866931682 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1070018020 con 0x7f107813de70 2024-03-19T01:56:00.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.191+0000 7f106dffb640 1 -- 172.21.15.55:0/3866931682 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1070014400 con 0x7f107813de70 2024-03-19T01:56:00.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.191+0000 7f107f8e5640 1 -- 172.21.15.55:0/3866931682 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10781f0fa0 con 0x7f107813de70 2024-03-19T01:56:00.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.191+0000 7f106dffb640 1 -- 172.21.15.55:0/3866931682 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10700147f0 con 0x7f107813de70 2024-03-19T01:56:00.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.191+0000 7f107f8e5640 1 -- 172.21.15.55:0/3866931682 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1078157930 con 0x7f107813de70 2024-03-19T01:56:00.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.191+0000 7f107f8e5640 1 -- 172.21.15.55:0/3866931682 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1078068a40 con 0x7f107813de70 2024-03-19T01:56:00.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.231+0000 7f106dffb640 1 -- 172.21.15.55:0/3866931682 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f107001b020 con 0x7f107813de70 2024-03-19T01:56:00.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.231+0000 7f106dffb640 1 --2- 172.21.15.55:0/3866931682 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f105c067660 0x7f105c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:00.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.231+0000 7f107d681640 1 --2- 172.21.15.55:0/3866931682 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f105c067660 0x7f105c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:00.229 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.231+0000 7f107d681640 1 --2- 172.21.15.55:0/3866931682 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f105c067660 0x7f105c069b20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1074009ac0 tx=0x7f1074009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:00.232 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.235+0000 7f106dffb640 1 -- 172.21.15.55:0/3866931682 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f1070054d80 con 0x7f107813de70 2024-03-19T01:56:00.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.543+0000 7f106dffb640 1 -- 172.21.15.55:0/3866931682 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1070022030 con 0x7f107813de70 2024-03-19T01:56:00.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.759+0000 7f107f8e5640 1 -- 172.21.15.55:0/3866931682 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f107813c930 con 0x7f107813de70 2024-03-19T01:56:00.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.775+0000 7f106dffb640 1 -- 172.21.15.55:0/3866931682 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f10781573d0 con 0x7f107813de70 2024-03-19T01:56:00.771 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:00.771 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:00.772 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:00.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.775+0000 7f107f8e5640 1 -- 172.21.15.55:0/3866931682 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f105c067660 msgr2=0x7f105c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:00.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.775+0000 7f107f8e5640 1 --2- 172.21.15.55:0/3866931682 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f105c067660 0x7f105c069b20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1074009ac0 tx=0x7f1074009190 comp rx=0 tx=0).stop 2024-03-19T01:56:00.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.775+0000 7f107f8e5640 1 -- 172.21.15.55:0/3866931682 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f107813de70 msgr2=0x7f107814f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:00.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.775+0000 7f107f8e5640 1 --2- 172.21.15.55:0/3866931682 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f107813de70 0x7f107814f930 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f107000ef00 tx=0x7f107000b480 comp rx=0 tx=0).stop 2024-03-19T01:56:00.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.775+0000 7f107f8e5640 1 -- 172.21.15.55:0/3866931682 shutdown_connections 2024-03-19T01:56:00.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.775+0000 7f107f8e5640 1 --2- 172.21.15.55:0/3866931682 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f105c067660 0x7f105c069b20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:00.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.775+0000 7f107f8e5640 1 --2- 172.21.15.55:0/3866931682 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f107813de70 0x7f107814f930 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:00.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.775+0000 7f107f8e5640 1 --2- 172.21.15.55:0/3866931682 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f107806f1b0 0x7f10781f3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:00.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.775+0000 7f107f8e5640 1 --2- 172.21.15.55:0/3866931682 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f107806eb70 0x7f107806b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:00.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.775+0000 7f107f8e5640 1 -- 172.21.15.55:0/3866931682 >> 172.21.15.55:0/3866931682 conn(0x7f107806fb40 msgr2=0x7f107813b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:00.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.775+0000 7f107f8e5640 1 -- 172.21.15.55:0/3866931682 shutdown_connections 2024-03-19T01:56:00.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:00.775+0000 7f107f8e5640 1 -- 172.21.15.55:0/3866931682 wait complete. 2024-03-19T01:56:01.392 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:01.392 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:02.393 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:02.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.511+0000 7f39b9d4b640 1 Processor -- start 2024-03-19T01:56:02.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.511+0000 7f39b9d4b640 1 -- start start 2024-03-19T01:56:02.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.511+0000 7f39b9d4b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f39b4150f50 0x7f39b415b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:02.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.511+0000 7f39b9d4b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39b4151330 con 0x7f39b40c3320 2024-03-19T01:56:02.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.511+0000 7f39b9d4b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39b40c3840 con 0x7f39b40ca350 2024-03-19T01:56:02.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.511+0000 7f39b9d4b640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f39b40c3e00 con 0x7f39b4150f50 2024-03-19T01:56:02.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.511+0000 7f39b3fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f39b40c3320 0x7f39b40ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:45952/0 (socket says 172.21.15.55:45952) 2024-03-19T01:56:02.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.511+0000 7f39b3fff640 1 -- 172.21.15.55:0/3219835906 learned_addr learned my addr 172.21.15.55:0/3219835906 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:56:02.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.511+0000 7f39b37fe640 1 --2- 172.21.15.55:0/3219835906 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f39b4150f50 0x7f39b415b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:02.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.515+0000 7f39b27fc640 1 -- 172.21.15.55:0/3219835906 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 692661435 0 0) 0x7f39b4151330 con 0x7f39b40c3320 2024-03-19T01:56:02.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.515+0000 7f39b27fc640 1 -- 172.21.15.55:0/3219835906 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f399c005740 con 0x7f39b40c3320 2024-03-19T01:56:02.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.519+0000 7f39b27fc640 1 -- 172.21.15.55:0/3219835906 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2612001762 0 0) 0x7f39b40c3840 con 0x7f39b40ca350 2024-03-19T01:56:02.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.519+0000 7f39b27fc640 1 -- 172.21.15.55:0/3219835906 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39b4151330 con 0x7f39b40ca350 2024-03-19T01:56:02.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.519+0000 7f39b27fc640 1 -- 172.21.15.55:0/3219835906 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 324456955 0 0) 0x7f399c005740 con 0x7f39b40c3320 2024-03-19T01:56:02.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.519+0000 7f39b27fc640 1 -- 172.21.15.55:0/3219835906 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f39b4150f50 msgr2=0x7f39b415b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:02.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.519+0000 7f39b27fc640 1 --2- 172.21.15.55:0/3219835906 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f39b4150f50 0x7f39b415b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:02.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.519+0000 7f39b27fc640 1 -- 172.21.15.55:0/3219835906 >> v1:172.21.15.84:6789/0 conn(0x7f39b40ca350 legacy=0x7f39b4157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:02.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.519+0000 7f39b27fc640 1 -- 172.21.15.55:0/3219835906 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39b40c3b20 con 0x7f39b40c3320 2024-03-19T01:56:02.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.519+0000 7f39b27fc640 1 -- 172.21.15.55:0/3219835906 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f39ac001a00 con 0x7f39b40c3320 2024-03-19T01:56:02.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.519+0000 7f39b27fc640 1 -- 172.21.15.55:0/3219835906 >> v1:172.21.15.55:6789/0 conn(0x7f39b40c3320 legacy=0x7f39b40ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:02.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.519+0000 7f39b27fc640 1 --2- 172.21.15.55:0/3219835906 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f399c005d90 0x7f399c006190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:02.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.519+0000 7f39b27fc640 1 --2- 172.21.15.55:0/3219835906 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f399c006a20 0x7f399c006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:02.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.519+0000 7f39b27fc640 1 --2- 172.21.15.55:0/3219835906 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f39b4150f50 0x7f399c0073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:02.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.519+0000 7f39b27fc640 1 -- 172.21.15.55:0/3219835906 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f39b40c3e00 con 0x7f399c005d90 2024-03-19T01:56:02.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.519+0000 7f39b27fc640 1 -- 172.21.15.55:0/3219835906 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f399c00dc90 con 0x7f399c006a20 2024-03-19T01:56:02.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.519+0000 7f39b27fc640 1 -- 172.21.15.55:0/3219835906 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f399c00de90 con 0x7f39b4150f50 2024-03-19T01:56:02.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.519+0000 7f39b37fe640 1 --2- 172.21.15.55:0/3219835906 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f399c005d90 0x7f399c006190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:02.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.523+0000 7f39b2ffd640 1 --2- 172.21.15.55:0/3219835906 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f399c006a20 0x7f399c006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:02.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.523+0000 7f39b2ffd640 1 -- 172.21.15.55:0/3219835906 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f39b4150f50 msgr2=0x7f399c0073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:02.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.523+0000 7f39b2ffd640 1 --2- 172.21.15.55:0/3219835906 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f39b4150f50 0x7f399c0073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:02.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.523+0000 7f39b2ffd640 1 -- 172.21.15.55:0/3219835906 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f399c005d90 msgr2=0x7f399c006190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:02.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.523+0000 7f39b2ffd640 1 --2- 172.21.15.55:0/3219835906 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f399c005d90 0x7f399c006190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:02.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.523+0000 7f39b2ffd640 1 -- 172.21.15.55:0/3219835906 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f399c005740 con 0x7f399c006a20 2024-03-19T01:56:02.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.527+0000 7f39b2ffd640 1 --2- 172.21.15.55:0/3219835906 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f399c006a20 0x7f399c006ea0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f39b40ca790 tx=0x7f39a801c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36fddfa64ed59556 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:02.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.531+0000 7f39b27fc640 1 -- 172.21.15.55:0/3219835906 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a8001520 con 0x7f399c006a20 2024-03-19T01:56:02.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.531+0000 7f39b27fc640 1 -- 172.21.15.55:0/3219835906 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39a80052f0 con 0x7f399c006a20 2024-03-19T01:56:02.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.531+0000 7f39b27fc640 1 -- 172.21.15.55:0/3219835906 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a8003840 con 0x7f399c006a20 2024-03-19T01:56:02.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.535+0000 7f39b9d4b640 1 -- 172.21.15.55:0/3219835906 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f399c006a20 msgr2=0x7f399c006ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:02.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.535+0000 7f39b9d4b640 1 --2- 172.21.15.55:0/3219835906 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f399c006a20 0x7f399c006ea0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f39b40ca790 tx=0x7f39a801c560 comp rx=0 tx=0).stop 2024-03-19T01:56:02.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.535+0000 7f39b3fff640 1 -- 172.21.15.55:0/3219835906 reap_dead start 2024-03-19T01:56:02.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.535+0000 7f39b9d4b640 1 -- 172.21.15.55:0/3219835906 shutdown_connections 2024-03-19T01:56:02.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.535+0000 7f39b9d4b640 1 -- 172.21.15.55:0/3219835906 >> 172.21.15.55:0/3219835906 conn(0x7f39b406e670 msgr2=0x7f39b406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:02.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.535+0000 7f39b9d4b640 1 -- 172.21.15.55:0/3219835906 shutdown_connections 2024-03-19T01:56:02.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.535+0000 7f39b9d4b640 1 -- 172.21.15.55:0/3219835906 wait complete. 2024-03-19T01:56:02.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.535+0000 7f39b9d4b640 1 Processor -- start 2024-03-19T01:56:02.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.535+0000 7f39b9d4b640 1 -- start start 2024-03-19T01:56:02.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.535+0000 7f39b9d4b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f399c006a20 0x7f39b4159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:02.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.535+0000 7f39b9d4b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f39b4159970 0x7f39b414f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:02.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.535+0000 7f39b9d4b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39b414fc40 0x7f39b40b18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:02.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.535+0000 7f39b9d4b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f39b40cd7f0 con 0x7f39b414fc40 2024-03-19T01:56:02.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.535+0000 7f39b9d4b640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f39b40cd670 con 0x7f39b4159970 2024-03-19T01:56:02.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.535+0000 7f39b9d4b640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f39b406bab0 con 0x7f399c006a20 2024-03-19T01:56:02.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.535+0000 7f39b3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39b414fc40 0x7f39b40b18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:02.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.535+0000 7f39b2ffd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f39b4159970 0x7f39b414f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:02.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.535+0000 7f39b3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39b414fc40 0x7f39b40b18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51590/0 (socket says 172.21.15.55:51590) 2024-03-19T01:56:02.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.535+0000 7f39b3fff640 1 -- 172.21.15.55:0/1935850757 learned_addr learned my addr 172.21.15.55:0/1935850757 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:02.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.535+0000 7f39b37fe640 1 --2- 172.21.15.55:0/1935850757 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f399c006a20 0x7f39b4159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:02.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.539+0000 7f39b3fff640 1 -- 172.21.15.55:0/1935850757 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f399c006a20 msgr2=0x7f39b4159430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:02.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.539+0000 7f39b3fff640 1 --2- 172.21.15.55:0/1935850757 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f399c006a20 0x7f39b4159430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:02.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.539+0000 7f39b3fff640 1 -- 172.21.15.55:0/1935850757 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f39b4159970 msgr2=0x7f39b414f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:02.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.539+0000 7f39b3fff640 1 --2- 172.21.15.55:0/1935850757 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f39b4159970 0x7f39b414f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:02.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.539+0000 7f39b3fff640 1 -- 172.21.15.55:0/1935850757 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39b41500d0 con 0x7f39b414fc40 2024-03-19T01:56:02.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.543+0000 7f39b3fff640 1 --2- 172.21.15.55:0/1935850757 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39b414fc40 0x7f39b40b18f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f39b4156ea0 tx=0x7f39ac00cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:02.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.547+0000 7f399bfff640 1 -- 172.21.15.55:0/1935850757 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac007070 con 0x7f39b414fc40 2024-03-19T01:56:02.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.547+0000 7f399bfff640 1 -- 172.21.15.55:0/1935850757 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39ac007210 con 0x7f39b414fc40 2024-03-19T01:56:02.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.547+0000 7f399bfff640 1 -- 172.21.15.55:0/1935850757 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac0075e0 con 0x7f39b414fc40 2024-03-19T01:56:02.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.547+0000 7f39b9d4b640 1 -- 172.21.15.55:0/1935850757 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39b4159db0 con 0x7f39b414fc40 2024-03-19T01:56:02.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.547+0000 7f39b9d4b640 1 -- 172.21.15.55:0/1935850757 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39b41583e0 con 0x7f39b414fc40 2024-03-19T01:56:02.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.547+0000 7f39b9d4b640 1 -- 172.21.15.55:0/1935850757 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39b40cc2f0 con 0x7f39b414fc40 2024-03-19T01:56:02.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.575+0000 7f399bfff640 1 -- 172.21.15.55:0/1935850757 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f39ac007780 con 0x7f39b414fc40 2024-03-19T01:56:02.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.575+0000 7f399bfff640 1 --2- 172.21.15.55:0/1935850757 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f39a00676b0 0x7f39a0069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:02.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.575+0000 7f39b37fe640 1 --2- 172.21.15.55:0/1935850757 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f39a00676b0 0x7f39a0069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:02.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.575+0000 7f39b37fe640 1 --2- 172.21.15.55:0/1935850757 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f39a00676b0 0x7f39a0069b70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f39a4009b00 tx=0x7f39a4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:02.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.871+0000 7f399bfff640 1 -- 172.21.15.55:0/1935850757 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f39ac053a20 con 0x7f39b414fc40 2024-03-19T01:56:02.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:02.871+0000 7f399bfff640 1 -- 172.21.15.55:0/1935850757 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f39ac056080 con 0x7f39b414fc40 2024-03-19T01:56:03.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:03.119+0000 7f39b9d4b640 1 -- 172.21.15.55:0/1935850757 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39b4068a40 con 0x7f39b414fc40 2024-03-19T01:56:03.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:03.135+0000 7f399bfff640 1 -- 172.21.15.55:0/1935850757 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f39ac055980 con 0x7f39b414fc40 2024-03-19T01:56:03.134 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:03.134 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:03.134 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:03.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:03.139+0000 7f39b9d4b640 1 -- 172.21.15.55:0/1935850757 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f39a00676b0 msgr2=0x7f39a0069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:03.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:03.139+0000 7f39b9d4b640 1 --2- 172.21.15.55:0/1935850757 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f39a00676b0 0x7f39a0069b70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f39a4009b00 tx=0x7f39a4009190 comp rx=0 tx=0).stop 2024-03-19T01:56:03.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:03.139+0000 7f39b9d4b640 1 -- 172.21.15.55:0/1935850757 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39b414fc40 msgr2=0x7f39b40b18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:03.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:03.139+0000 7f39b9d4b640 1 --2- 172.21.15.55:0/1935850757 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39b414fc40 0x7f39b40b18f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f39b4156ea0 tx=0x7f39ac00cad0 comp rx=0 tx=0).stop 2024-03-19T01:56:03.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:03.139+0000 7f39b9d4b640 1 -- 172.21.15.55:0/1935850757 shutdown_connections 2024-03-19T01:56:03.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:03.139+0000 7f39b9d4b640 1 --2- 172.21.15.55:0/1935850757 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f39a00676b0 0x7f39a0069b70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:03.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:03.139+0000 7f39b9d4b640 1 --2- 172.21.15.55:0/1935850757 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39b414fc40 0x7f39b40b18f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:03.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:03.139+0000 7f39b9d4b640 1 --2- 172.21.15.55:0/1935850757 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f39b4159970 0x7f39b414f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:03.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:03.139+0000 7f39b9d4b640 1 --2- 172.21.15.55:0/1935850757 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f399c006a20 0x7f39b4159430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:03.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:03.139+0000 7f39b9d4b640 1 -- 172.21.15.55:0/1935850757 >> 172.21.15.55:0/1935850757 conn(0x7f39b406e670 msgr2=0x7f39b41437d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:03.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:03.139+0000 7f39b9d4b640 1 -- 172.21.15.55:0/1935850757 shutdown_connections 2024-03-19T01:56:03.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:03.139+0000 7f39b9d4b640 1 -- 172.21.15.55:0/1935850757 wait complete. 2024-03-19T01:56:03.759 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:03.760 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:04.760 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:04.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.875+0000 7f943c9db640 1 Processor -- start 2024-03-19T01:56:04.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.875+0000 7f943c9db640 1 -- start start 2024-03-19T01:56:04.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.875+0000 7f943c9db640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f943806c610 0x7f943815ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:04.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.875+0000 7f943c9db640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f943806ceb0 0x7f94381427f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:04.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.875+0000 7f943c9db640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9438070e80 con 0x7f943806c9f0 2024-03-19T01:56:04.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.875+0000 7f943c9db640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f943815b250 con 0x7f943806c610 2024-03-19T01:56:04.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.875+0000 7f943c9db640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f943806fb00 con 0x7f943806ceb0 2024-03-19T01:56:04.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.879+0000 7f9435d9b640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f943806c9f0 0x7f94380c92c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:45954/0 (socket says 172.21.15.55:45954) 2024-03-19T01:56:04.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.879+0000 7f9435d9b640 1 -- 172.21.15.55:0/3793744792 learned_addr learned my addr 172.21.15.55:0/3793744792 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:56:04.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.879+0000 7f9436d9d640 1 --2- 172.21.15.55:0/3793744792 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f943806ceb0 0x7f94381427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:04.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.879+0000 7f943659c640 1 --2- 172.21.15.55:0/3793744792 >> v2:172.21.15.84:3300/0 conn(0x7f943806c610 0x7f943815ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:04.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.883+0000 7f943559a640 1 -- 172.21.15.55:0/3793744792 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3179882419 0 0) 0x7f9438070e80 con 0x7f943806c9f0 2024-03-19T01:56:04.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.883+0000 7f943559a640 1 -- 172.21.15.55:0/3793744792 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9420003530 con 0x7f943806c9f0 2024-03-19T01:56:04.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.883+0000 7f9436d9d640 1 -- 172.21.15.55:0/3793744792 >> v2:172.21.15.84:3300/0 conn(0x7f943806c610 msgr2=0x7f943815ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:04.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.883+0000 7f9436d9d640 1 --2- 172.21.15.55:0/3793744792 >> v2:172.21.15.84:3300/0 conn(0x7f943806c610 0x7f943815ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:04.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.883+0000 7f9436d9d640 1 -- 172.21.15.55:0/3793744792 >> v1:172.21.15.55:6789/0 conn(0x7f943806c9f0 legacy=0x7f94380c92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:04.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.883+0000 7f9436d9d640 1 -- 172.21.15.55:0/3793744792 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f943806fde0 con 0x7f943806ceb0 2024-03-19T01:56:04.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.883+0000 7f943659c640 1 --2- 172.21.15.55:0/3793744792 >> v2:172.21.15.84:3300/0 conn(0x7f943806c610 0x7f943815ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:56:04.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.887+0000 7f9436d9d640 1 --2- 172.21.15.55:0/3793744792 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f943806ceb0 0x7f94381427f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9430009d90 tx=0x7f943001f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a7a199a02678e1a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:04.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.891+0000 7f943559a640 1 -- 172.21.15.55:0/3793744792 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94300b4070 con 0x7f943806ceb0 2024-03-19T01:56:04.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.891+0000 7f943559a640 1 -- 172.21.15.55:0/3793744792 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94300af600 con 0x7f943806ceb0 2024-03-19T01:56:04.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.891+0000 7f943559a640 1 -- 172.21.15.55:0/3793744792 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94300af9d0 con 0x7f943806ceb0 2024-03-19T01:56:04.891 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.895+0000 7f943c9db640 1 -- 172.21.15.55:0/3793744792 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f943806ceb0 msgr2=0x7f94381427f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:04.891 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.895+0000 7f943c9db640 1 --2- 172.21.15.55:0/3793744792 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f943806ceb0 0x7f94381427f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9430009d90 tx=0x7f943001f150 comp rx=0 tx=0).stop 2024-03-19T01:56:04.891 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.895+0000 7f943c9db640 1 -- 172.21.15.55:0/3793744792 shutdown_connections 2024-03-19T01:56:04.891 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.895+0000 7f943c9db640 1 --2- 172.21.15.55:0/3793744792 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f943806ceb0 0x7f94381427f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:04.891 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.895+0000 7f943c9db640 1 --2- 172.21.15.55:0/3793744792 >> v2:172.21.15.84:3300/0 conn(0x7f943806c610 0x7f943815ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:04.891 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.895+0000 7f943c9db640 1 -- 172.21.15.55:0/3793744792 >> 172.21.15.55:0/3793744792 conn(0x7f9438067600 msgr2=0x7f9438071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:04.891 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.895+0000 7f943c9db640 1 -- 172.21.15.55:0/3793744792 shutdown_connections 2024-03-19T01:56:04.891 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.895+0000 7f943c9db640 1 -- 172.21.15.55:0/3793744792 wait complete. 2024-03-19T01:56:04.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.895+0000 7f943c9db640 1 Processor -- start 2024-03-19T01:56:04.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.895+0000 7f943c9db640 1 -- start start 2024-03-19T01:56:04.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.895+0000 7f943c9db640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f943806c610 0x7f94380c5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:04.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.895+0000 7f943c9db640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f943806c9f0 0x7f943815df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:04.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.895+0000 7f943c9db640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f943806ceb0 0x7f94380ad9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:04.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.895+0000 7f943c9db640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9438070680 con 0x7f943806ceb0 2024-03-19T01:56:04.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.895+0000 7f943c9db640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f94380703a0 con 0x7f943806c9f0 2024-03-19T01:56:04.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.895+0000 7f943c9db640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f9438070960 con 0x7f943806c610 2024-03-19T01:56:04.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.895+0000 7f9436d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f943806ceb0 0x7f94380ad9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:04.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.895+0000 7f943659c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f943806c610 0x7f94380c5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:04.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.895+0000 7f9435d9b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f943806c9f0 0x7f943815df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:04.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.895+0000 7f9436d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f943806ceb0 0x7f94380ad9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51600/0 (socket says 172.21.15.55:51600) 2024-03-19T01:56:04.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.895+0000 7f9436d9d640 1 -- 172.21.15.55:0/1852135995 learned_addr learned my addr 172.21.15.55:0/1852135995 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:04.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.899+0000 7f9436d9d640 1 -- 172.21.15.55:0/1852135995 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f943806c610 msgr2=0x7f94380c5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:04.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.899+0000 7f9436d9d640 1 --2- 172.21.15.55:0/1852135995 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f943806c610 0x7f94380c5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:04.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.899+0000 7f9436d9d640 1 -- 172.21.15.55:0/1852135995 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f943806c9f0 msgr2=0x7f943815df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:04.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.899+0000 7f9436d9d640 1 --2- 172.21.15.55:0/1852135995 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f943806c9f0 0x7f943815df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:04.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.899+0000 7f9436d9d640 1 -- 172.21.15.55:0/1852135995 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94380c6110 con 0x7f943806ceb0 2024-03-19T01:56:04.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.903+0000 7f9436d9d640 1 --2- 172.21.15.55:0/1852135995 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f943806ceb0 0x7f94380ad9b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f94300b2040 tx=0x7f943001bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:04.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.907+0000 7f9426ffd640 1 -- 172.21.15.55:0/1852135995 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94300bf020 con 0x7f943806ceb0 2024-03-19T01:56:04.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.907+0000 7f9426ffd640 1 -- 172.21.15.55:0/1852135995 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94300afe50 con 0x7f943806ceb0 2024-03-19T01:56:04.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.907+0000 7f9426ffd640 1 -- 172.21.15.55:0/1852135995 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94300b4040 con 0x7f943806ceb0 2024-03-19T01:56:04.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.907+0000 7f943c9db640 1 -- 172.21.15.55:0/1852135995 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94380c8040 con 0x7f943806ceb0 2024-03-19T01:56:04.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.907+0000 7f943c9db640 1 -- 172.21.15.55:0/1852135995 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94381f3280 con 0x7f943806ceb0 2024-03-19T01:56:04.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.907+0000 7f943c9db640 1 -- 172.21.15.55:0/1852135995 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9438068a40 con 0x7f943806ceb0 2024-03-19T01:56:04.926 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.931+0000 7f9426ffd640 1 -- 172.21.15.55:0/1852135995 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f94300c2020 con 0x7f943806ceb0 2024-03-19T01:56:04.927 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.931+0000 7f9426ffd640 1 --2- 172.21.15.55:0/1852135995 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9418067660 0x7f9418069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:04.927 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.931+0000 7f943659c640 1 --2- 172.21.15.55:0/1852135995 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9418067660 0x7f9418069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:04.928 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:04.931+0000 7f943659c640 1 --2- 172.21.15.55:0/1852135995 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9418067660 0x7f9418069b20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f94381f0a30 tx=0x7f9428009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:05.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:05.223+0000 7f9426ffd640 1 -- 172.21.15.55:0/1852135995 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f94300f29c0 con 0x7f943806ceb0 2024-03-19T01:56:05.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:05.223+0000 7f9426ffd640 1 -- 172.21.15.55:0/1852135995 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f94300c9030 con 0x7f943806ceb0 2024-03-19T01:56:05.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:05.503+0000 7f943c9db640 1 -- 172.21.15.55:0/1852135995 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f943813e650 con 0x7f943806ceb0 2024-03-19T01:56:05.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:05.563+0000 7f9426ffd640 1 -- 172.21.15.55:0/1852135995 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f94380c6110 con 0x7f943806ceb0 2024-03-19T01:56:05.561 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:05.561 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:05.561 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:05.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:05.567+0000 7f943c9db640 1 -- 172.21.15.55:0/1852135995 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9418067660 msgr2=0x7f9418069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:05.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:05.567+0000 7f943c9db640 1 --2- 172.21.15.55:0/1852135995 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9418067660 0x7f9418069b20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f94381f0a30 tx=0x7f9428009190 comp rx=0 tx=0).stop 2024-03-19T01:56:05.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:05.567+0000 7f943c9db640 1 -- 172.21.15.55:0/1852135995 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f943806ceb0 msgr2=0x7f94380ad9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:05.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:05.567+0000 7f943c9db640 1 --2- 172.21.15.55:0/1852135995 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f943806ceb0 0x7f94380ad9b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f94300b2040 tx=0x7f943001bef0 comp rx=0 tx=0).stop 2024-03-19T01:56:05.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:05.567+0000 7f943c9db640 1 -- 172.21.15.55:0/1852135995 shutdown_connections 2024-03-19T01:56:05.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:05.567+0000 7f943c9db640 1 --2- 172.21.15.55:0/1852135995 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9418067660 0x7f9418069b20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:05.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:05.567+0000 7f943c9db640 1 --2- 172.21.15.55:0/1852135995 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f943806ceb0 0x7f94380ad9b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:05.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:05.567+0000 7f943c9db640 1 --2- 172.21.15.55:0/1852135995 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f943806c9f0 0x7f943815df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:05.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:05.567+0000 7f943c9db640 1 --2- 172.21.15.55:0/1852135995 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f943806c610 0x7f94380c5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:05.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:05.567+0000 7f943c9db640 1 -- 172.21.15.55:0/1852135995 >> 172.21.15.55:0/1852135995 conn(0x7f9438067600 msgr2=0x7f9438140230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:05.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:05.567+0000 7f943c9db640 1 -- 172.21.15.55:0/1852135995 shutdown_connections 2024-03-19T01:56:05.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:05.567+0000 7f943c9db640 1 -- 172.21.15.55:0/1852135995 wait complete. 2024-03-19T01:56:06.184 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:06.185 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:07.186 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:07.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.311+0000 7f00397d0640 1 Processor -- start 2024-03-19T01:56:07.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.311+0000 7f00397d0640 1 -- start start 2024-03-19T01:56:07.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.311+0000 7f00397d0640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f00341522a0 0x7f00340c8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:07.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.311+0000 7f00397d0640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00340c8830 con 0x7f00340cd150 2024-03-19T01:56:07.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.311+0000 7f00397d0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0034158240 con 0x7f00340cd530 2024-03-19T01:56:07.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.311+0000 7f00397d0640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f00341584e0 con 0x7f00341522a0 2024-03-19T01:56:07.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.311+0000 7f00337fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f00341522a0 0x7f00340c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:07.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.311+0000 7f00327fc640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f00340cd530 0x7f0034151b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:41768/0 (socket says 172.21.15.55:41768) 2024-03-19T01:56:07.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.311+0000 7f00327fc640 1 -- 172.21.15.55:0/2338823865 learned_addr learned my addr 172.21.15.55:0/2338823865 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:56:07.313 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.315+0000 7f0031ffb640 1 -- 172.21.15.55:0/2338823865 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2885871355 0 0) 0x7f0024001120 con 0x7f00340cd150 2024-03-19T01:56:07.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.315+0000 7f0031ffb640 1 -- 172.21.15.55:0/2338823865 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f001c003530 con 0x7f00340cd150 2024-03-19T01:56:07.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.319+0000 7f0031ffb640 1 -- 172.21.15.55:0/2338823865 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3471431827 0 0) 0x7f0034158240 con 0x7f00340cd530 2024-03-19T01:56:07.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.319+0000 7f0031ffb640 1 -- 172.21.15.55:0/2338823865 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0024001120 con 0x7f00340cd530 2024-03-19T01:56:07.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.319+0000 7f00337fe640 1 -- 172.21.15.55:0/2338823865 >> v1:172.21.15.84:6789/0 conn(0x7f00340cd530 legacy=0x7f0034151b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:07.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.319+0000 7f00337fe640 1 -- 172.21.15.55:0/2338823865 >> v1:172.21.15.55:6789/0 conn(0x7f00340cd150 legacy=0x7f00340732e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:07.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.319+0000 7f00337fe640 1 -- 172.21.15.55:0/2338823865 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00341587c0 con 0x7f00341522a0 2024-03-19T01:56:07.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.323+0000 7f00337fe640 1 --2- 172.21.15.55:0/2338823865 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00341522a0 0x7f00340c8010 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f002c009d90 tx=0x7f002c01f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e958dcc36cab9c81 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:07.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.327+0000 7f0031ffb640 1 -- 172.21.15.55:0/2338823865 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c0b4070 con 0x7f00341522a0 2024-03-19T01:56:07.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.327+0000 7f0031ffb640 1 -- 172.21.15.55:0/2338823865 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f002c0af600 con 0x7f00341522a0 2024-03-19T01:56:07.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.327+0000 7f0031ffb640 1 -- 172.21.15.55:0/2338823865 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c0af9d0 con 0x7f00341522a0 2024-03-19T01:56:07.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.331+0000 7f00397d0640 1 -- 172.21.15.55:0/2338823865 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00341522a0 msgr2=0x7f00340c8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:07.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.331+0000 7f00397d0640 1 --2- 172.21.15.55:0/2338823865 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00341522a0 0x7f00340c8010 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f002c009d90 tx=0x7f002c01f150 comp rx=0 tx=0).stop 2024-03-19T01:56:07.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.331+0000 7f00397d0640 1 -- 172.21.15.55:0/2338823865 shutdown_connections 2024-03-19T01:56:07.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.331+0000 7f00397d0640 1 --2- 172.21.15.55:0/2338823865 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00341522a0 0x7f00340c8010 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:07.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.331+0000 7f00397d0640 1 -- 172.21.15.55:0/2338823865 >> 172.21.15.55:0/2338823865 conn(0x7f003406e670 msgr2=0x7f003406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:07.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.331+0000 7f00397d0640 1 -- 172.21.15.55:0/2338823865 shutdown_connections 2024-03-19T01:56:07.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.331+0000 7f00397d0640 1 -- 172.21.15.55:0/2338823865 wait complete. 2024-03-19T01:56:07.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.331+0000 7f00397d0640 1 Processor -- start 2024-03-19T01:56:07.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.331+0000 7f00397d0640 1 -- start start 2024-03-19T01:56:07.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.331+0000 7f00397d0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00340cd150 0x7f00340c2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:07.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.331+0000 7f00397d0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f00340cd530 0x7f003414b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:07.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.331+0000 7f00397d0640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f00341522a0 0x7f003414b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:07.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.331+0000 7f00397d0640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f003406b680 con 0x7f00340cd150 2024-03-19T01:56:07.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.331+0000 7f00397d0640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f0034158d80 con 0x7f00341522a0 2024-03-19T01:56:07.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.331+0000 7f00397d0640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f003406b800 con 0x7f00340cd530 2024-03-19T01:56:07.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.331+0000 7f0032ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00340cd150 0x7f00340c2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:07.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.331+0000 7f0032ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00340cd150 0x7f00340c2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51630/0 (socket says 172.21.15.55:51630) 2024-03-19T01:56:07.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.331+0000 7f0032ffd640 1 -- 172.21.15.55:0/3777637099 learned_addr learned my addr 172.21.15.55:0/3777637099 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:07.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.335+0000 7f00337fe640 1 --2- 172.21.15.55:0/3777637099 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f00341522a0 0x7f003414b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:07.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.335+0000 7f00327fc640 1 --2- 172.21.15.55:0/3777637099 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f00340cd530 0x7f003414b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:07.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.335+0000 7f0032ffd640 1 -- 172.21.15.55:0/3777637099 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f00340cd530 msgr2=0x7f003414b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:07.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.335+0000 7f0032ffd640 1 --2- 172.21.15.55:0/3777637099 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f00340cd530 0x7f003414b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:07.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.335+0000 7f0032ffd640 1 -- 172.21.15.55:0/3777637099 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f00341522a0 msgr2=0x7f003414b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:07.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.335+0000 7f0032ffd640 1 --2- 172.21.15.55:0/3777637099 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f00341522a0 0x7f003414b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:07.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.335+0000 7f0032ffd640 1 -- 172.21.15.55:0/3777637099 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0034157c00 con 0x7f00340cd150 2024-03-19T01:56:07.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.339+0000 7f0032ffd640 1 --2- 172.21.15.55:0/3777637099 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00340cd150 0x7f00340c2610 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f00240010f0 tx=0x7f002400cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:07.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.343+0000 7f001b7fe640 1 -- 172.21.15.55:0/3777637099 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00240052d0 con 0x7f00340cd150 2024-03-19T01:56:07.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.343+0000 7f001b7fe640 1 -- 172.21.15.55:0/3777637099 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0024005470 con 0x7f00340cd150 2024-03-19T01:56:07.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.343+0000 7f00397d0640 1 -- 172.21.15.55:0/3777637099 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0034157ea0 con 0x7f00340cd150 2024-03-19T01:56:07.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.343+0000 7f001b7fe640 1 -- 172.21.15.55:0/3777637099 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002400a840 con 0x7f00340cd150 2024-03-19T01:56:07.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.343+0000 7f00397d0640 1 -- 172.21.15.55:0/3777637099 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00340c3240 con 0x7f00340cd150 2024-03-19T01:56:07.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.343+0000 7f00397d0640 1 -- 172.21.15.55:0/3777637099 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0034068a40 con 0x7f00340cd150 2024-03-19T01:56:07.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.363+0000 7f001b7fe640 1 -- 172.21.15.55:0/3777637099 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f002400a9e0 con 0x7f00340cd150 2024-03-19T01:56:07.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.363+0000 7f001b7fe640 1 --2- 172.21.15.55:0/3777637099 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f00200676b0 0x7f0020069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:07.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.367+0000 7f00327fc640 1 --2- 172.21.15.55:0/3777637099 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f00200676b0 0x7f0020069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:07.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.367+0000 7f00327fc640 1 --2- 172.21.15.55:0/3777637099 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f00200676b0 0x7f0020069b70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0028007fd0 tx=0x7f0028009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:07.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.655+0000 7f001b7fe640 1 -- 172.21.15.55:0/3777637099 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f0024054760 con 0x7f00340cd150 2024-03-19T01:56:07.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.655+0000 7f001b7fe640 1 -- 172.21.15.55:0/3777637099 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f00340c8830 con 0x7f00340cd150 2024-03-19T01:56:07.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.867+0000 7f00397d0640 1 -- 172.21.15.55:0/3777637099 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f003414e020 con 0x7f00340cd150 2024-03-19T01:56:07.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.899+0000 7f001b7fe640 1 -- 172.21.15.55:0/3777637099 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f00240566c0 con 0x7f00340cd150 2024-03-19T01:56:07.898 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:07.898 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:07.898 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:07.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.903+0000 7f00397d0640 1 -- 172.21.15.55:0/3777637099 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f00200676b0 msgr2=0x7f0020069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:07.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.903+0000 7f00397d0640 1 --2- 172.21.15.55:0/3777637099 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f00200676b0 0x7f0020069b70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0028007fd0 tx=0x7f0028009040 comp rx=0 tx=0).stop 2024-03-19T01:56:07.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.903+0000 7f00397d0640 1 -- 172.21.15.55:0/3777637099 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00340cd150 msgr2=0x7f00340c2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:07.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.903+0000 7f00397d0640 1 --2- 172.21.15.55:0/3777637099 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00340cd150 0x7f00340c2610 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f00240010f0 tx=0x7f002400cb20 comp rx=0 tx=0).stop 2024-03-19T01:56:07.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.903+0000 7f00397d0640 1 -- 172.21.15.55:0/3777637099 shutdown_connections 2024-03-19T01:56:07.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.903+0000 7f00397d0640 1 --2- 172.21.15.55:0/3777637099 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f00200676b0 0x7f0020069b70 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:07.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.903+0000 7f00397d0640 1 --2- 172.21.15.55:0/3777637099 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f00341522a0 0x7f003414b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:07.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.903+0000 7f00397d0640 1 --2- 172.21.15.55:0/3777637099 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f00340cd530 0x7f003414b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:07.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.903+0000 7f00397d0640 1 --2- 172.21.15.55:0/3777637099 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f00340cd150 0x7f00340c2610 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:07.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.903+0000 7f00397d0640 1 -- 172.21.15.55:0/3777637099 >> 172.21.15.55:0/3777637099 conn(0x7f003406e670 msgr2=0x7f0034150540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:07.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.903+0000 7f00397d0640 1 -- 172.21.15.55:0/3777637099 shutdown_connections 2024-03-19T01:56:07.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:07.903+0000 7f00397d0640 1 -- 172.21.15.55:0/3777637099 wait complete. 2024-03-19T01:56:08.525 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:08.525 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:09.200 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:56:09.202 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:56:09.526 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:09.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.654+0000 7fe21a49f640 1 Processor -- start 2024-03-19T01:56:09.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.654+0000 7fe21a49f640 1 -- start start 2024-03-19T01:56:09.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.654+0000 7fe21a49f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fe21414eaa0 0x7fe21406b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:09.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.654+0000 7fe21a49f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe2141585f0 0x7fe214158a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:09.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.654+0000 7fe21a49f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe21414ee80 con 0x7fe21406bec0 2024-03-19T01:56:09.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.654+0000 7fe21a49f640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fe214158f50 con 0x7fe21414eaa0 2024-03-19T01:56:09.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.654+0000 7fe21a49f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fe2140c7540 con 0x7fe2141585f0 2024-03-19T01:56:09.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.658+0000 7fe2137fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fe21406bec0 0x7fe2140c60d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:46922/0 (socket says 172.21.15.55:46922) 2024-03-19T01:56:09.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.658+0000 7fe2137fe640 1 -- 172.21.15.55:0/4015610556 learned_addr learned my addr 172.21.15.55:0/4015610556 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:56:09.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.658+0000 7fe213fff640 1 --2- 172.21.15.55:0/4015610556 >> v2:172.21.15.84:3300/0 conn(0x7fe21414eaa0 0x7fe21406b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:09.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.658+0000 7fe218a3c640 1 --2- 172.21.15.55:0/4015610556 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe2141585f0 0x7fe214158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:09.657 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.658+0000 7fe212ffd640 1 -- 172.21.15.55:0/4015610556 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2025850835 0 0) 0x7fe21414ee80 con 0x7fe21406bec0 2024-03-19T01:56:09.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.658+0000 7fe212ffd640 1 -- 172.21.15.55:0/4015610556 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1fc003530 con 0x7fe21406bec0 2024-03-19T01:56:09.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.662+0000 7fe213fff640 1 -- 172.21.15.55:0/4015610556 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe2141585f0 msgr2=0x7fe214158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:09.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.662+0000 7fe213fff640 1 --2- 172.21.15.55:0/4015610556 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe2141585f0 0x7fe214158a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:09.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.662+0000 7fe213fff640 1 -- 172.21.15.55:0/4015610556 >> v1:172.21.15.55:6789/0 conn(0x7fe21406bec0 legacy=0x7fe2140c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:09.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.662+0000 7fe213fff640 1 -- 172.21.15.55:0/4015610556 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2140c77c0 con 0x7fe21414eaa0 2024-03-19T01:56:09.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.666+0000 7fe213fff640 1 --2- 172.21.15.55:0/4015610556 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe21414eaa0 0x7fe21406b980 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe20c009d90 tx=0x7fe20c01c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7b05aa1d75ee908 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:09.671 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.674+0000 7fe212ffd640 1 -- 172.21.15.55:0/4015610556 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20c0b2070 con 0x7fe21414eaa0 2024-03-19T01:56:09.672 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.674+0000 7fe212ffd640 1 -- 172.21.15.55:0/4015610556 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe20c0a59a0 con 0x7fe21414eaa0 2024-03-19T01:56:09.672 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.674+0000 7fe212ffd640 1 -- 172.21.15.55:0/4015610556 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20c0a5d70 con 0x7fe21414eaa0 2024-03-19T01:56:09.672 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.674+0000 7fe21a49f640 1 -- 172.21.15.55:0/4015610556 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe21414eaa0 msgr2=0x7fe21406b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:09.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.674+0000 7fe21a49f640 1 --2- 172.21.15.55:0/4015610556 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe21414eaa0 0x7fe21406b980 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe20c009d90 tx=0x7fe20c01c8e0 comp rx=0 tx=0).stop 2024-03-19T01:56:09.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.674+0000 7fe21a49f640 1 -- 172.21.15.55:0/4015610556 shutdown_connections 2024-03-19T01:56:09.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.674+0000 7fe21a49f640 1 --2- 172.21.15.55:0/4015610556 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe2141585f0 0x7fe214158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:09.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.674+0000 7fe21a49f640 1 --2- 172.21.15.55:0/4015610556 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe21414eaa0 0x7fe21406b980 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:09.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.674+0000 7fe21a49f640 1 -- 172.21.15.55:0/4015610556 >> 172.21.15.55:0/4015610556 conn(0x7fe2140cc9c0 msgr2=0x7fe2140ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:09.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.674+0000 7fe21a49f640 1 -- 172.21.15.55:0/4015610556 shutdown_connections 2024-03-19T01:56:09.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.674+0000 7fe21a49f640 1 -- 172.21.15.55:0/4015610556 wait complete. 2024-03-19T01:56:09.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.674+0000 7fe21a49f640 1 Processor -- start 2024-03-19T01:56:09.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.674+0000 7fe21a49f640 1 -- start start 2024-03-19T01:56:09.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.678+0000 7fe21a49f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe21406bec0 0x7fe214142e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:09.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.678+0000 7fe21a49f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe21414eaa0 0x7fe214161f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:09.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.678+0000 7fe21a49f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2141585f0 0x7fe2141410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:09.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.678+0000 7fe21a49f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe2140c67e0 con 0x7fe2141585f0 2024-03-19T01:56:09.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.678+0000 7fe21a49f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fe2140c8060 con 0x7fe21414eaa0 2024-03-19T01:56:09.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.678+0000 7fe21a49f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fe2140c6960 con 0x7fe21406bec0 2024-03-19T01:56:09.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.678+0000 7fe218a3c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2141585f0 0x7fe2141410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:09.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.678+0000 7fe213fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe21406bec0 0x7fe214142e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:09.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.678+0000 7fe218a3c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2141585f0 0x7fe2141410d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48314/0 (socket says 172.21.15.55:48314) 2024-03-19T01:56:09.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.678+0000 7fe218a3c640 1 -- 172.21.15.55:0/3145947318 learned_addr learned my addr 172.21.15.55:0/3145947318 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:09.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.678+0000 7fe2137fe640 1 --2- 172.21.15.55:0/3145947318 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe21414eaa0 0x7fe214161f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:09.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.682+0000 7fe218a3c640 1 -- 172.21.15.55:0/3145947318 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe21406bec0 msgr2=0x7fe214142e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:09.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.682+0000 7fe218a3c640 1 --2- 172.21.15.55:0/3145947318 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe21406bec0 0x7fe214142e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:09.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.682+0000 7fe218a3c640 1 -- 172.21.15.55:0/3145947318 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe21414eaa0 msgr2=0x7fe214161f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:09.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.682+0000 7fe218a3c640 1 --2- 172.21.15.55:0/3145947318 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe21414eaa0 0x7fe214161f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:09.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.682+0000 7fe218a3c640 1 -- 172.21.15.55:0/3145947318 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe214143390 con 0x7fe2141585f0 2024-03-19T01:56:09.681 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.682+0000 7fe218a3c640 1 --2- 172.21.15.55:0/3145947318 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2141585f0 0x7fe2141410d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe20400e960 tx=0x7fe20400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:09.686 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.686+0000 7fe210ff9640 1 -- 172.21.15.55:0/3145947318 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe204017070 con 0x7fe2141585f0 2024-03-19T01:56:09.686 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.686+0000 7fe210ff9640 1 -- 172.21.15.55:0/3145947318 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe20400c9a0 con 0x7fe2141585f0 2024-03-19T01:56:09.686 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.686+0000 7fe210ff9640 1 -- 172.21.15.55:0/3145947318 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20400cd70 con 0x7fe2141585f0 2024-03-19T01:56:09.686 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.686+0000 7fe21a49f640 1 -- 172.21.15.55:0/3145947318 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2140c5430 con 0x7fe2141585f0 2024-03-19T01:56:09.686 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.686+0000 7fe21a49f640 1 -- 172.21.15.55:0/3145947318 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe21414cfd0 con 0x7fe2141585f0 2024-03-19T01:56:09.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.690+0000 7fe21a49f640 1 -- 172.21.15.55:0/3145947318 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe214068a40 con 0x7fe2141585f0 2024-03-19T01:56:09.712 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.714+0000 7fe210ff9640 1 -- 172.21.15.55:0/3145947318 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fe20401a070 con 0x7fe2141585f0 2024-03-19T01:56:09.712 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.714+0000 7fe210ff9640 1 --2- 172.21.15.55:0/3145947318 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe200067660 0x7fe200069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:09.713 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.714+0000 7fe213fff640 1 --2- 172.21.15.55:0/3145947318 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe200067660 0x7fe200069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:09.713 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:09.714+0000 7fe213fff640 1 --2- 172.21.15.55:0/3145947318 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe200067660 0x7fe200069b20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe20c0af000 tx=0x7fe20c0a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:10.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:10.014+0000 7fe210ff9640 1 -- 172.21.15.55:0/3145947318 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fe204054700 con 0x7fe2141585f0 2024-03-19T01:56:10.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:10.018+0000 7fe210ff9640 1 -- 172.21.15.55:0/3145947318 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fe204056660 con 0x7fe2141585f0 2024-03-19T01:56:10.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:10.214+0000 7fe21a49f640 1 -- 172.21.15.55:0/3145947318 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2141558e0 con 0x7fe2141585f0 2024-03-19T01:56:10.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:10.234+0000 7fe210ff9640 1 -- 172.21.15.55:0/3145947318 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fe20405a700 con 0x7fe2141585f0 2024-03-19T01:56:10.231 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:10.232 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:10.232 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:10.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:10.234+0000 7fe21a49f640 1 -- 172.21.15.55:0/3145947318 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe200067660 msgr2=0x7fe200069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:10.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:10.234+0000 7fe21a49f640 1 --2- 172.21.15.55:0/3145947318 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe200067660 0x7fe200069b20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe20c0af000 tx=0x7fe20c0a7070 comp rx=0 tx=0).stop 2024-03-19T01:56:10.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:10.234+0000 7fe21a49f640 1 -- 172.21.15.55:0/3145947318 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2141585f0 msgr2=0x7fe2141410d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:10.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:10.234+0000 7fe21a49f640 1 --2- 172.21.15.55:0/3145947318 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2141585f0 0x7fe2141410d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe20400e960 tx=0x7fe20400ee20 comp rx=0 tx=0).stop 2024-03-19T01:56:10.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:10.234+0000 7fe21a49f640 1 -- 172.21.15.55:0/3145947318 shutdown_connections 2024-03-19T01:56:10.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:10.234+0000 7fe21a49f640 1 --2- 172.21.15.55:0/3145947318 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe200067660 0x7fe200069b20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:10.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:10.234+0000 7fe21a49f640 1 --2- 172.21.15.55:0/3145947318 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2141585f0 0x7fe2141410d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:10.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:10.234+0000 7fe21a49f640 1 --2- 172.21.15.55:0/3145947318 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe21414eaa0 0x7fe214161f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:10.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:10.234+0000 7fe21a49f640 1 --2- 172.21.15.55:0/3145947318 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe21406bec0 0x7fe214142e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:10.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:10.234+0000 7fe21a49f640 1 -- 172.21.15.55:0/3145947318 >> 172.21.15.55:0/3145947318 conn(0x7fe2140cc9c0 msgr2=0x7fe2140ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:10.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:10.234+0000 7fe21a49f640 1 -- 172.21.15.55:0/3145947318 shutdown_connections 2024-03-19T01:56:10.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:10.234+0000 7fe21a49f640 1 -- 172.21.15.55:0/3145947318 wait complete. 2024-03-19T01:56:10.854 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:10.854 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:11.855 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:11.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.978+0000 7f4f2e439640 1 Processor -- start 2024-03-19T01:56:11.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.978+0000 7f4f2e439640 1 -- start start 2024-03-19T01:56:11.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.978+0000 7f4f2e439640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4f2806ceb0 0x7f4f281427f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:11.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.978+0000 7f4f2e439640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f28070e80 con 0x7f4f2806c610 2024-03-19T01:56:11.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.978+0000 7f4f2e439640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f2806fb00 con 0x7f4f2806c9f0 2024-03-19T01:56:11.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.978+0000 7f4f2e439640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4f2806fde0 con 0x7f4f2806ceb0 2024-03-19T01:56:11.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.978+0000 7f4f27fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f4f2806c610 0x7f4f2815ad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:46932/0 (socket says 172.21.15.55:46932) 2024-03-19T01:56:11.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.978+0000 7f4f27fff640 1 -- 172.21.15.55:0/2649357665 learned_addr learned my addr 172.21.15.55:0/2649357665 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:56:11.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.978+0000 7f4f2c9d6640 1 --2- 172.21.15.55:0/2649357665 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4f2806ceb0 0x7f4f281427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:11.980 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.982+0000 7f4f26ffd640 1 -- 172.21.15.55:0/2649357665 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2736343503 0 0) 0x7f4f28070e80 con 0x7f4f2806c610 2024-03-19T01:56:11.980 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.982+0000 7f4f26ffd640 1 -- 172.21.15.55:0/2649357665 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f10003530 con 0x7f4f2806c610 2024-03-19T01:56:11.981 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.982+0000 7f4f26ffd640 1 -- 172.21.15.55:0/2649357665 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1952896908 0 0) 0x7f4f2806fb00 con 0x7f4f2806c9f0 2024-03-19T01:56:11.981 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.982+0000 7f4f26ffd640 1 -- 172.21.15.55:0/2649357665 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f28070e80 con 0x7f4f2806c9f0 2024-03-19T01:56:11.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.986+0000 7f4f2c9d6640 1 -- 172.21.15.55:0/2649357665 >> v1:172.21.15.84:6789/0 conn(0x7f4f2806c9f0 legacy=0x7f4f280c92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:11.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.986+0000 7f4f2c9d6640 1 -- 172.21.15.55:0/2649357665 >> v1:172.21.15.55:6789/0 conn(0x7f4f2806c610 legacy=0x7f4f2815ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:11.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.986+0000 7f4f2c9d6640 1 -- 172.21.15.55:0/2649357665 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f280700c0 con 0x7f4f2806ceb0 2024-03-19T01:56:11.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.990+0000 7f4f2c9d6640 1 --2- 172.21.15.55:0/2649357665 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4f2806ceb0 0x7f4f281427f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4f20009dc0 tx=0x7f4f2001f170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebf0221894e028bb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:11.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.994+0000 7f4f26ffd640 1 -- 172.21.15.55:0/2649357665 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f200b4070 con 0x7f4f2806ceb0 2024-03-19T01:56:11.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.994+0000 7f4f26ffd640 1 -- 172.21.15.55:0/2649357665 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f200af600 con 0x7f4f2806ceb0 2024-03-19T01:56:11.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.994+0000 7f4f26ffd640 1 -- 172.21.15.55:0/2649357665 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f200af9d0 con 0x7f4f2806ceb0 2024-03-19T01:56:11.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.994+0000 7f4f2e439640 1 -- 172.21.15.55:0/2649357665 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4f2806ceb0 msgr2=0x7f4f281427f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:11.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.994+0000 7f4f2e439640 1 --2- 172.21.15.55:0/2649357665 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4f2806ceb0 0x7f4f281427f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4f20009dc0 tx=0x7f4f2001f170 comp rx=0 tx=0).stop 2024-03-19T01:56:11.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.994+0000 7f4f2e439640 1 -- 172.21.15.55:0/2649357665 shutdown_connections 2024-03-19T01:56:11.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.994+0000 7f4f2e439640 1 --2- 172.21.15.55:0/2649357665 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4f2806ceb0 0x7f4f281427f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:11.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.994+0000 7f4f2e439640 1 -- 172.21.15.55:0/2649357665 >> 172.21.15.55:0/2649357665 conn(0x7f4f28067600 msgr2=0x7f4f28071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:11.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.994+0000 7f4f2e439640 1 -- 172.21.15.55:0/2649357665 shutdown_connections 2024-03-19T01:56:11.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.994+0000 7f4f2e439640 1 -- 172.21.15.55:0/2649357665 wait complete. 2024-03-19T01:56:11.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.994+0000 7f4f2e439640 1 Processor -- start 2024-03-19T01:56:11.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.994+0000 7f4f2e439640 1 -- start start 2024-03-19T01:56:11.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.998+0000 7f4f2e439640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f2806c610 0x7f4f281f0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:11.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.998+0000 7f4f2e439640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4f2806c9f0 0x7f4f28136e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:11.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.998+0000 7f4f2e439640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f2806ceb0 0x7f4f2813f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:11.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.998+0000 7f4f2e439640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4f28070960 con 0x7f4f2806c610 2024-03-19T01:56:11.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.998+0000 7f4f2e439640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4f28070680 con 0x7f4f2806ceb0 2024-03-19T01:56:11.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.998+0000 7f4f2e439640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4f280c99d0 con 0x7f4f2806c9f0 2024-03-19T01:56:11.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.998+0000 7f4f27fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f2806c610 0x7f4f281f0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:11.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.998+0000 7f4f277fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4f2806c9f0 0x7f4f28136e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:11.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.998+0000 7f4f2c9d6640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f2806ceb0 0x7f4f2813f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:11.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.998+0000 7f4f27fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f2806c610 0x7f4f281f0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48320/0 (socket says 172.21.15.55:48320) 2024-03-19T01:56:11.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:11.998+0000 7f4f27fff640 1 -- 172.21.15.55:0/230417978 learned_addr learned my addr 172.21.15.55:0/230417978 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:11.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.002+0000 7f4f27fff640 1 -- 172.21.15.55:0/230417978 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4f2806c9f0 msgr2=0x7f4f28136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:12.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.002+0000 7f4f27fff640 1 --2- 172.21.15.55:0/230417978 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4f2806c9f0 0x7f4f28136e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:12.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.002+0000 7f4f27fff640 1 -- 172.21.15.55:0/230417978 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f2806ceb0 msgr2=0x7f4f2813f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:12.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.002+0000 7f4f27fff640 1 --2- 172.21.15.55:0/230417978 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f2806ceb0 0x7f4f2813f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:12.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.002+0000 7f4f27fff640 1 -- 172.21.15.55:0/230417978 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f280723b0 con 0x7f4f2806c610 2024-03-19T01:56:12.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.006+0000 7f4f27fff640 1 --2- 172.21.15.55:0/230417978 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f2806c610 0x7f4f281f0b10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4f1800c500 tx=0x7f4f1800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:12.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.010+0000 7f4f24ff9640 1 -- 172.21.15.55:0/230417978 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f18017070 con 0x7f4f2806c610 2024-03-19T01:56:12.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.010+0000 7f4f24ff9640 1 -- 172.21.15.55:0/230417978 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f18005cf0 con 0x7f4f2806c610 2024-03-19T01:56:12.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.010+0000 7f4f2e439640 1 -- 172.21.15.55:0/230417978 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f28137350 con 0x7f4f2806c610 2024-03-19T01:56:12.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.010+0000 7f4f24ff9640 1 -- 172.21.15.55:0/230417978 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f180072f0 con 0x7f4f2806c610 2024-03-19T01:56:12.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.010+0000 7f4f2e439640 1 -- 172.21.15.55:0/230417978 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f280ce760 con 0x7f4f2806c610 2024-03-19T01:56:12.008 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.010+0000 7f4f2e439640 1 -- 172.21.15.55:0/230417978 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f28068a40 con 0x7f4f2806c610 2024-03-19T01:56:12.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.030+0000 7f4f24ff9640 1 -- 172.21.15.55:0/230417978 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f4f1801a070 con 0x7f4f2806c610 2024-03-19T01:56:12.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.030+0000 7f4f24ff9640 1 --2- 172.21.15.55:0/230417978 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4f140676e0 0x7f4f14069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:12.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.030+0000 7f4f277fe640 1 --2- 172.21.15.55:0/230417978 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4f140676e0 0x7f4f14069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:12.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.034+0000 7f4f277fe640 1 --2- 172.21.15.55:0/230417978 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4f140676e0 0x7f4f14069ba0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4f280690c0 tx=0x7f4f1c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:12.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.034+0000 7f4f24ff9640 1 -- 172.21.15.55:0/230417978 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f4f180537a0 con 0x7f4f2806c610 2024-03-19T01:56:12.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.310+0000 7f4f24ff9640 1 -- 172.21.15.55:0/230417978 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4f18055700 con 0x7f4f2806c610 2024-03-19T01:56:12.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.522+0000 7f4f2e439640 1 -- 172.21.15.55:0/230417978 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f28142280 con 0x7f4f2806c610 2024-03-19T01:56:12.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.630+0000 7f4f24ff9640 1 -- 172.21.15.55:0/230417978 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f4f180597a0 con 0x7f4f2806c610 2024-03-19T01:56:12.630 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:12.631 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:12.631 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:12.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.634+0000 7f4f2e439640 1 -- 172.21.15.55:0/230417978 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4f140676e0 msgr2=0x7f4f14069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:12.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.634+0000 7f4f2e439640 1 --2- 172.21.15.55:0/230417978 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4f140676e0 0x7f4f14069ba0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4f280690c0 tx=0x7f4f1c009040 comp rx=0 tx=0).stop 2024-03-19T01:56:12.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.634+0000 7f4f2e439640 1 -- 172.21.15.55:0/230417978 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f2806c610 msgr2=0x7f4f281f0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:12.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.634+0000 7f4f2e439640 1 --2- 172.21.15.55:0/230417978 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f2806c610 0x7f4f281f0b10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4f1800c500 tx=0x7f4f1800c9c0 comp rx=0 tx=0).stop 2024-03-19T01:56:12.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.634+0000 7f4f2e439640 1 -- 172.21.15.55:0/230417978 shutdown_connections 2024-03-19T01:56:12.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.634+0000 7f4f2e439640 1 --2- 172.21.15.55:0/230417978 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4f140676e0 0x7f4f14069ba0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:12.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.634+0000 7f4f2e439640 1 --2- 172.21.15.55:0/230417978 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f2806ceb0 0x7f4f2813f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:12.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.634+0000 7f4f2e439640 1 --2- 172.21.15.55:0/230417978 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4f2806c9f0 0x7f4f28136e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:12.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.634+0000 7f4f2e439640 1 --2- 172.21.15.55:0/230417978 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4f2806c610 0x7f4f281f0b10 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:12.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.634+0000 7f4f2e439640 1 -- 172.21.15.55:0/230417978 >> 172.21.15.55:0/230417978 conn(0x7f4f28067600 msgr2=0x7f4f28074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:12.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.634+0000 7f4f2e439640 1 -- 172.21.15.55:0/230417978 shutdown_connections 2024-03-19T01:56:12.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:12.634+0000 7f4f2e439640 1 -- 172.21.15.55:0/230417978 wait complete. 2024-03-19T01:56:13.255 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:13.255 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:14.257 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:14.372 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.374+0000 7f6f6da00640 1 Processor -- start 2024-03-19T01:56:14.372 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.374+0000 7f6f6da00640 1 -- start start 2024-03-19T01:56:14.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.374+0000 7f6f6da00640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f6f681544f0 0x7f6f68073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:14.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.374+0000 7f6f6da00640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6f6806b2a0 0x7f6f6815e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:14.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.374+0000 7f6f6da00640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f680c7fa0 con 0x7f6f6815acb0 2024-03-19T01:56:14.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.374+0000 7f6f6da00640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f6f6815b210 con 0x7f6f681544f0 2024-03-19T01:56:14.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.374+0000 7f6f6da00640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f6f680c8240 con 0x7f6f6806b2a0 2024-03-19T01:56:14.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.374+0000 7f6f66ffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f6f681544f0 0x7f6f68073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:14.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.374+0000 7f6f667fc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6f6806b2a0 0x7f6f6815e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:14.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.378+0000 7f6f677fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f6f6815acb0 0x7f6f680cce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:46946/0 (socket says 172.21.15.55:46946) 2024-03-19T01:56:14.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.378+0000 7f6f677fe640 1 -- 172.21.15.55:0/655705302 learned_addr learned my addr 172.21.15.55:0/655705302 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:56:14.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.378+0000 7f6f667fc640 1 -- 172.21.15.55:0/655705302 >> v2:172.21.15.55:3300/0 conn(0x7f6f681544f0 msgr2=0x7f6f68073540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:14.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.378+0000 7f6f667fc640 1 --2- 172.21.15.55:0/655705302 >> v2:172.21.15.55:3300/0 conn(0x7f6f681544f0 0x7f6f68073540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:14.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.378+0000 7f6f667fc640 1 -- 172.21.15.55:0/655705302 >> v1:172.21.15.55:6789/0 conn(0x7f6f6815acb0 legacy=0x7f6f680cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:14.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.378+0000 7f6f667fc640 1 -- 172.21.15.55:0/655705302 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f6815d920 con 0x7f6f6806b2a0 2024-03-19T01:56:14.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.382+0000 7f6f667fc640 1 --2- 172.21.15.55:0/655705302 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6f6806b2a0 0x7f6f6815e120 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c0097c0 tx=0x7f6f5c01ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adb1ea03c0302ede server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:14.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.390+0000 7f6f65ffb640 1 -- 172.21.15.55:0/655705302 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c0b5070 con 0x7f6f6806b2a0 2024-03-19T01:56:14.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.390+0000 7f6f65ffb640 1 -- 172.21.15.55:0/655705302 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f5c0a8990 con 0x7f6f6806b2a0 2024-03-19T01:56:14.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.390+0000 7f6f65ffb640 1 -- 172.21.15.55:0/655705302 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c0a8d60 con 0x7f6f6806b2a0 2024-03-19T01:56:14.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.390+0000 7f6f6da00640 1 -- 172.21.15.55:0/655705302 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6f6806b2a0 msgr2=0x7f6f6815e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:14.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.390+0000 7f6f6da00640 1 --2- 172.21.15.55:0/655705302 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6f6806b2a0 0x7f6f6815e120 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c0097c0 tx=0x7f6f5c01ec90 comp rx=0 tx=0).stop 2024-03-19T01:56:14.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.390+0000 7f6f6da00640 1 -- 172.21.15.55:0/655705302 shutdown_connections 2024-03-19T01:56:14.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.390+0000 7f6f6da00640 1 --2- 172.21.15.55:0/655705302 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6f6806b2a0 0x7f6f6815e120 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:14.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.390+0000 7f6f6da00640 1 --2- 172.21.15.55:0/655705302 >> v2:172.21.15.55:3300/0 conn(0x7f6f681544f0 0x7f6f68073540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:14.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.390+0000 7f6f6da00640 1 -- 172.21.15.55:0/655705302 >> 172.21.15.55:0/655705302 conn(0x7f6f6806e670 msgr2=0x7f6f6806ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:14.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.390+0000 7f6f6da00640 1 -- 172.21.15.55:0/655705302 shutdown_connections 2024-03-19T01:56:14.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.390+0000 7f6f6da00640 1 -- 172.21.15.55:0/655705302 wait complete. 2024-03-19T01:56:14.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.394+0000 7f6f6da00640 1 Processor -- start 2024-03-19T01:56:14.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.394+0000 7f6f6da00640 1 -- start start 2024-03-19T01:56:14.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.394+0000 7f6f6da00640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f6806b2a0 0x7f6f6815c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:14.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.394+0000 7f6f6da00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6f681544f0 0x7f6f68151030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:14.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.394+0000 7f6f6da00640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6f6815acb0 0x7f6f68151570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:14.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.394+0000 7f6f6da00640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6f681548d0 con 0x7f6f6806b2a0 2024-03-19T01:56:14.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.394+0000 7f6f6da00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6815e660 con 0x7f6f681544f0 2024-03-19T01:56:14.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.394+0000 7f6f6da00640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f6f680cd540 con 0x7f6f6815acb0 2024-03-19T01:56:14.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.394+0000 7f6f66ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f6806b2a0 0x7f6f6815c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:14.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.394+0000 7f6f667fc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6f681544f0 0x7f6f68151030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:14.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.394+0000 7f6f677fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6f6815acb0 0x7f6f68151570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:14.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.394+0000 7f6f66ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f6806b2a0 0x7f6f6815c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48346/0 (socket says 172.21.15.55:48346) 2024-03-19T01:56:14.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.394+0000 7f6f66ffd640 1 -- 172.21.15.55:0/3891127770 learned_addr learned my addr 172.21.15.55:0/3891127770 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:14.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.398+0000 7f6f66ffd640 1 -- 172.21.15.55:0/3891127770 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6f6815acb0 msgr2=0x7f6f68151570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:14.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.398+0000 7f6f66ffd640 1 --2- 172.21.15.55:0/3891127770 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6f6815acb0 0x7f6f68151570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:14.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.398+0000 7f6f66ffd640 1 -- 172.21.15.55:0/3891127770 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6f681544f0 msgr2=0x7f6f68151030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:14.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.398+0000 7f6f66ffd640 1 --2- 172.21.15.55:0/3891127770 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6f681544f0 0x7f6f68151030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:14.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.398+0000 7f6f66ffd640 1 -- 172.21.15.55:0/3891127770 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f681f0130 con 0x7f6f6806b2a0 2024-03-19T01:56:14.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.402+0000 7f6f66ffd640 1 --2- 172.21.15.55:0/3891127770 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f6806b2a0 0x7f6f6815c980 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6f5800e930 tx=0x7f6f5800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:14.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.406+0000 7f6f4ffff640 1 -- 172.21.15.55:0/3891127770 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f58017070 con 0x7f6f6806b2a0 2024-03-19T01:56:14.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.406+0000 7f6f4ffff640 1 -- 172.21.15.55:0/3891127770 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f5800ca00 con 0x7f6f6806b2a0 2024-03-19T01:56:14.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.406+0000 7f6f6da00640 1 -- 172.21.15.55:0/3891127770 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f681f03d0 con 0x7f6f6806b2a0 2024-03-19T01:56:14.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.406+0000 7f6f4ffff640 1 -- 172.21.15.55:0/3891127770 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f5800cdb0 con 0x7f6f6806b2a0 2024-03-19T01:56:14.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.406+0000 7f6f6da00640 1 -- 172.21.15.55:0/3891127770 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f681f0990 con 0x7f6f6806b2a0 2024-03-19T01:56:14.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.406+0000 7f6f6da00640 1 -- 172.21.15.55:0/3891127770 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f68068a40 con 0x7f6f6806b2a0 2024-03-19T01:56:14.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.438+0000 7f6f4ffff640 1 -- 172.21.15.55:0/3891127770 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f6f5801a070 con 0x7f6f6806b2a0 2024-03-19T01:56:14.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.438+0000 7f6f4ffff640 1 --2- 172.21.15.55:0/3891127770 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6f540676b0 0x7f6f54069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:14.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.438+0000 7f6f667fc640 1 --2- 172.21.15.55:0/3891127770 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6f540676b0 0x7f6f54069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:14.437 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.438+0000 7f6f667fc640 1 --2- 172.21.15.55:0/3891127770 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6f540676b0 0x7f6f54069b70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c0b2000 tx=0x7f6f5c0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:14.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.770+0000 7f6f4ffff640 1 -- 172.21.15.55:0/3891127770 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f6f580547b0 con 0x7f6f6806b2a0 2024-03-19T01:56:14.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.770+0000 7f6f4ffff640 1 -- 172.21.15.55:0/3891127770 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f6f58056710 con 0x7f6f6806b2a0 2024-03-19T01:56:14.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:14.970+0000 7f6f6da00640 1 -- 172.21.15.55:0/3891127770 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f680c7be0 con 0x7f6f6806b2a0 2024-03-19T01:56:15.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:15.082+0000 7f6f4ffff640 1 -- 172.21.15.55:0/3891127770 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f6f5805a7b0 con 0x7f6f6806b2a0 2024-03-19T01:56:15.082 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:15.082 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:15.082 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:15.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:15.086+0000 7f6f6da00640 1 -- 172.21.15.55:0/3891127770 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6f540676b0 msgr2=0x7f6f54069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:15.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:15.086+0000 7f6f6da00640 1 --2- 172.21.15.55:0/3891127770 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6f540676b0 0x7f6f54069b70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c0b2000 tx=0x7f6f5c0aa070 comp rx=0 tx=0).stop 2024-03-19T01:56:15.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:15.086+0000 7f6f6da00640 1 -- 172.21.15.55:0/3891127770 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f6806b2a0 msgr2=0x7f6f6815c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:15.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:15.086+0000 7f6f6da00640 1 --2- 172.21.15.55:0/3891127770 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f6806b2a0 0x7f6f6815c980 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6f5800e930 tx=0x7f6f5800edf0 comp rx=0 tx=0).stop 2024-03-19T01:56:15.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:15.086+0000 7f6f6da00640 1 -- 172.21.15.55:0/3891127770 shutdown_connections 2024-03-19T01:56:15.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:15.086+0000 7f6f6da00640 1 --2- 172.21.15.55:0/3891127770 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6f540676b0 0x7f6f54069b70 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:15.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:15.086+0000 7f6f6da00640 1 --2- 172.21.15.55:0/3891127770 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6f6815acb0 0x7f6f68151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:15.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:15.086+0000 7f6f6da00640 1 --2- 172.21.15.55:0/3891127770 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6f681544f0 0x7f6f68151030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:15.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:15.086+0000 7f6f6da00640 1 --2- 172.21.15.55:0/3891127770 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f6806b2a0 0x7f6f6815c980 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:15.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:15.086+0000 7f6f6da00640 1 -- 172.21.15.55:0/3891127770 >> 172.21.15.55:0/3891127770 conn(0x7f6f6806e670 msgr2=0x7f6f6806ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:15.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:15.086+0000 7f6f6da00640 1 -- 172.21.15.55:0/3891127770 shutdown_connections 2024-03-19T01:56:15.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:15.086+0000 7f6f6da00640 1 -- 172.21.15.55:0/3891127770 wait complete. 2024-03-19T01:56:15.708 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:15.708 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:16.709 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:16.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.826+0000 7f5751b88640 1 Processor -- start 2024-03-19T01:56:16.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.826+0000 7f5751b88640 1 -- start start 2024-03-19T01:56:16.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.826+0000 7f5751b88640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f574c154720 0x7f574c15e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:16.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.826+0000 7f5751b88640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f574c071280 con 0x7f574c0c73c0 2024-03-19T01:56:16.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.826+0000 7f5751b88640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f574c15d900 con 0x7f574c154340 2024-03-19T01:56:16.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.826+0000 7f5751b88640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f574c073840 con 0x7f574c154720 2024-03-19T01:56:16.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.826+0000 7f574affd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f574c154720 0x7f574c15e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:16.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.826+0000 7f574affd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f574c154720 0x7f574c15e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48350/0 (socket says 172.21.15.55:48350) 2024-03-19T01:56:16.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.826+0000 7f574affd640 1 -- 172.21.15.55:0/1483145628 learned_addr learned my addr 172.21.15.55:0/1483145628 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:16.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.830+0000 7f574affd640 1 -- 172.21.15.55:0/1483145628 >> v1:172.21.15.84:6789/0 conn(0x7f574c154340 legacy=0x7f574c072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:16.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.830+0000 7f574affd640 1 -- 172.21.15.55:0/1483145628 >> v1:172.21.15.55:6789/0 conn(0x7f574c0c73c0 legacy=0x7f574c0c91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:16.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.830+0000 7f574affd640 1 -- 172.21.15.55:0/1483145628 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f574c070cc0 con 0x7f574c154720 2024-03-19T01:56:16.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.838+0000 7f574affd640 1 --2- 172.21.15.55:0/1483145628 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f574c154720 0x7f574c15e3a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5740009870 tx=0x7f574001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b09e03a9e9ac882 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:16.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.842+0000 7f574a7fc640 1 -- 172.21.15.55:0/1483145628 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57400b2070 con 0x7f574c154720 2024-03-19T01:56:16.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.842+0000 7f574a7fc640 1 -- 172.21.15.55:0/1483145628 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57400a9070 con 0x7f574c154720 2024-03-19T01:56:16.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.842+0000 7f574a7fc640 1 -- 172.21.15.55:0/1483145628 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57400ad880 con 0x7f574c154720 2024-03-19T01:56:16.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.846+0000 7f5751b88640 1 -- 172.21.15.55:0/1483145628 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f574c154720 msgr2=0x7f574c15e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:16.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.846+0000 7f5751b88640 1 --2- 172.21.15.55:0/1483145628 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f574c154720 0x7f574c15e3a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5740009870 tx=0x7f574001c560 comp rx=0 tx=0).stop 2024-03-19T01:56:16.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.846+0000 7f5751b88640 1 -- 172.21.15.55:0/1483145628 shutdown_connections 2024-03-19T01:56:16.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.846+0000 7f5751b88640 1 --2- 172.21.15.55:0/1483145628 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f574c154720 0x7f574c15e3a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:16.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.846+0000 7f5751b88640 1 -- 172.21.15.55:0/1483145628 >> 172.21.15.55:0/1483145628 conn(0x7f574c0cc9c0 msgr2=0x7f574c0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:16.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.846+0000 7f5751b88640 1 -- 172.21.15.55:0/1483145628 shutdown_connections 2024-03-19T01:56:16.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.846+0000 7f5751b88640 1 -- 172.21.15.55:0/1483145628 wait complete. 2024-03-19T01:56:16.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.846+0000 7f5751b88640 1 Processor -- start 2024-03-19T01:56:16.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.846+0000 7f5751b88640 1 -- start start 2024-03-19T01:56:16.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.846+0000 7f5751b88640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f574c0c73c0 0x7f574c154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:16.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.846+0000 7f5751b88640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f574c154340 0x7f574c14de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:16.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.846+0000 7f5751b88640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f574c154720 0x7f574c14c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:16.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.846+0000 7f5751b88640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f574c168390 con 0x7f574c154720 2024-03-19T01:56:16.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.846+0000 7f5751b88640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f574c15b0b0 con 0x7f574c0c73c0 2024-03-19T01:56:16.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.846+0000 7f5751b88640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f574c168510 con 0x7f574c154340 2024-03-19T01:56:16.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.846+0000 7f574bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f574c154720 0x7f574c14c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:16.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.846+0000 7f574affd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f574c154340 0x7f574c14de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:16.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.846+0000 7f574bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f574c154720 0x7f574c14c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48362/0 (socket says 172.21.15.55:48362) 2024-03-19T01:56:16.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.846+0000 7f574bfff640 1 -- 172.21.15.55:0/610370057 learned_addr learned my addr 172.21.15.55:0/610370057 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:16.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.846+0000 7f574b7fe640 1 --2- 172.21.15.55:0/610370057 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f574c0c73c0 0x7f574c154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:16.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.850+0000 7f574bfff640 1 -- 172.21.15.55:0/610370057 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f574c154340 msgr2=0x7f574c14de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:16.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.850+0000 7f574bfff640 1 --2- 172.21.15.55:0/610370057 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f574c154340 0x7f574c14de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:16.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.850+0000 7f574bfff640 1 -- 172.21.15.55:0/610370057 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f574c0c73c0 msgr2=0x7f574c154b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:16.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.850+0000 7f574bfff640 1 --2- 172.21.15.55:0/610370057 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f574c0c73c0 0x7f574c154b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:16.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.850+0000 7f574bfff640 1 -- 172.21.15.55:0/610370057 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f574c156550 con 0x7f574c154720 2024-03-19T01:56:16.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.854+0000 7f574bfff640 1 --2- 172.21.15.55:0/610370057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f574c154720 0x7f574c14c010 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f574400c7a0 tx=0x7f574400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:16.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.858+0000 7f5733fff640 1 -- 172.21.15.55:0/610370057 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5744003b80 con 0x7f574c154720 2024-03-19T01:56:16.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.858+0000 7f5751b88640 1 -- 172.21.15.55:0/610370057 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f574c152190 con 0x7f574c154720 2024-03-19T01:56:16.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.858+0000 7f5751b88640 1 -- 172.21.15.55:0/610370057 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f574c15b7b0 con 0x7f574c154720 2024-03-19T01:56:16.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.858+0000 7f5751b88640 1 -- 172.21.15.55:0/610370057 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f574c15cfc0 con 0x7f574c154720 2024-03-19T01:56:16.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.858+0000 7f5733fff640 1 -- 172.21.15.55:0/610370057 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5744003d20 con 0x7f574c154720 2024-03-19T01:56:16.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:16.858+0000 7f5733fff640 1 -- 172.21.15.55:0/610370057 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574400aa50 con 0x7f574c154720 2024-03-19T01:56:17.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:17.134+0000 7f5733fff640 1 -- 172.21.15.55:0/610370057 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f574400abf0 con 0x7f574c154720 2024-03-19T01:56:17.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:17.138+0000 7f5733fff640 1 --2- 172.21.15.55:0/610370057 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5738067660 0x7f5738069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:17.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:17.138+0000 7f574b7fe640 1 --2- 172.21.15.55:0/610370057 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5738067660 0x7f5738069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:17.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:17.138+0000 7f574b7fe640 1 --2- 172.21.15.55:0/610370057 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5738067660 0x7f5738069b20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f573c007fd0 tx=0x7f573c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:17.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:17.162+0000 7f5733fff640 1 -- 172.21.15.55:0/610370057 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f57440547c0 con 0x7f574c154720 2024-03-19T01:56:17.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:17.162+0000 7f5733fff640 1 -- 172.21.15.55:0/610370057 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f574c156550 con 0x7f574c154720 2024-03-19T01:56:17.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:17.406+0000 7f5751b88640 1 -- 172.21.15.55:0/610370057 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f574c0cd1b0 con 0x7f574c154720 2024-03-19T01:56:17.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:17.430+0000 7f5733fff640 1 -- 172.21.15.55:0/610370057 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f5744056720 con 0x7f574c154720 2024-03-19T01:56:17.427 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:17.427 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:17.428 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:17.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:17.430+0000 7f5751b88640 1 -- 172.21.15.55:0/610370057 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5738067660 msgr2=0x7f5738069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:17.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:17.430+0000 7f5751b88640 1 --2- 172.21.15.55:0/610370057 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5738067660 0x7f5738069b20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f573c007fd0 tx=0x7f573c009040 comp rx=0 tx=0).stop 2024-03-19T01:56:17.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:17.430+0000 7f5751b88640 1 -- 172.21.15.55:0/610370057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f574c154720 msgr2=0x7f574c14c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:17.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:17.430+0000 7f5751b88640 1 --2- 172.21.15.55:0/610370057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f574c154720 0x7f574c14c010 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f574400c7a0 tx=0x7f574400cc60 comp rx=0 tx=0).stop 2024-03-19T01:56:17.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:17.430+0000 7f5751b88640 1 -- 172.21.15.55:0/610370057 shutdown_connections 2024-03-19T01:56:17.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:17.430+0000 7f5751b88640 1 --2- 172.21.15.55:0/610370057 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5738067660 0x7f5738069b20 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:17.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:17.430+0000 7f5751b88640 1 --2- 172.21.15.55:0/610370057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f574c154720 0x7f574c14c010 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:17.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:17.430+0000 7f5751b88640 1 --2- 172.21.15.55:0/610370057 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f574c154340 0x7f574c14de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:17.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:17.430+0000 7f5751b88640 1 --2- 172.21.15.55:0/610370057 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f574c0c73c0 0x7f574c154b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:17.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:17.430+0000 7f5751b88640 1 -- 172.21.15.55:0/610370057 >> 172.21.15.55:0/610370057 conn(0x7f574c0cc9c0 msgr2=0x7f574c0cb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:17.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:17.430+0000 7f5751b88640 1 -- 172.21.15.55:0/610370057 shutdown_connections 2024-03-19T01:56:17.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:17.430+0000 7f5751b88640 1 -- 172.21.15.55:0/610370057 wait complete. 2024-03-19T01:56:18.049 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:18.049 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:19.050 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:19.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.178+0000 7f3cf750a640 1 Processor -- start 2024-03-19T01:56:19.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.178+0000 7f3cf750a640 1 -- start start 2024-03-19T01:56:19.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.182+0000 7f3cf750a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3cf0152120 0x7f3cf00c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:19.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.182+0000 7f3cf750a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f3cf0070780 0x7f3cf01586e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:19.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.182+0000 7f3cf750a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cf006fff0 con 0x7f3cf00c9610 2024-03-19T01:56:19.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.182+0000 7f3cf750a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f3cf0152500 con 0x7f3cf0152120 2024-03-19T01:56:19.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.182+0000 7f3cf750a640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f3cf0070590 con 0x7f3cf0070780 2024-03-19T01:56:19.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.182+0000 7f3cf52a6640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3cf0152120 0x7f3cf00c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:19.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.182+0000 7f3cf5aa7640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f3cf0070780 0x7f3cf01586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:19.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.182+0000 7f3cf52a6640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3cf0152120 0x7f3cf00c90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57962/0 (socket says 172.21.15.55:57962) 2024-03-19T01:56:19.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.182+0000 7f3cf52a6640 1 -- 172.21.15.55:0/3585320266 learned_addr learned my addr 172.21.15.55:0/3585320266 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:19.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.186+0000 7f3cf5aa7640 1 -- 172.21.15.55:0/3585320266 >> v2:172.21.15.55:3300/0 conn(0x7f3cf0152120 msgr2=0x7f3cf00c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:19.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.186+0000 7f3cf5aa7640 1 --2- 172.21.15.55:0/3585320266 >> v2:172.21.15.55:3300/0 conn(0x7f3cf0152120 0x7f3cf00c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:19.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.186+0000 7f3cf5aa7640 1 -- 172.21.15.55:0/3585320266 >> v1:172.21.15.55:6789/0 conn(0x7f3cf00c9610 legacy=0x7f3cf0157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:19.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.186+0000 7f3cf5aa7640 1 -- 172.21.15.55:0/3585320266 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf00c39c0 con 0x7f3cf0070780 2024-03-19T01:56:19.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.190+0000 7f3cf5aa7640 1 --2- 172.21.15.55:0/3585320266 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3cf0070780 0x7f3cf01586e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3cf00cb940 tx=0x7f3ce801c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=191134b00cefa63d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:19.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.194+0000 7f3ce7fff640 1 -- 172.21.15.55:0/3585320266 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce80b2070 con 0x7f3cf0070780 2024-03-19T01:56:19.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.194+0000 7f3ce7fff640 1 -- 172.21.15.55:0/3585320266 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ce80a57f0 con 0x7f3cf0070780 2024-03-19T01:56:19.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.194+0000 7f3ce7fff640 1 -- 172.21.15.55:0/3585320266 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce80a5bc0 con 0x7f3cf0070780 2024-03-19T01:56:19.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.198+0000 7f3cf750a640 1 -- 172.21.15.55:0/3585320266 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3cf0070780 msgr2=0x7f3cf01586e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:19.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.198+0000 7f3cf750a640 1 --2- 172.21.15.55:0/3585320266 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3cf0070780 0x7f3cf01586e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3cf00cb940 tx=0x7f3ce801c930 comp rx=0 tx=0).stop 2024-03-19T01:56:19.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.198+0000 7f3cf750a640 1 -- 172.21.15.55:0/3585320266 shutdown_connections 2024-03-19T01:56:19.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.198+0000 7f3cf750a640 1 --2- 172.21.15.55:0/3585320266 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3cf0070780 0x7f3cf01586e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:19.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.198+0000 7f3cf750a640 1 --2- 172.21.15.55:0/3585320266 >> v2:172.21.15.55:3300/0 conn(0x7f3cf0152120 0x7f3cf00c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:19.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.198+0000 7f3cf750a640 1 -- 172.21.15.55:0/3585320266 >> 172.21.15.55:0/3585320266 conn(0x7f3cf0067600 msgr2=0x7f3cf0071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:19.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.198+0000 7f3cf750a640 1 -- 172.21.15.55:0/3585320266 shutdown_connections 2024-03-19T01:56:19.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.198+0000 7f3cf750a640 1 -- 172.21.15.55:0/3585320266 wait complete. 2024-03-19T01:56:19.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.198+0000 7f3cf750a640 1 Processor -- start 2024-03-19T01:56:19.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.198+0000 7f3cf750a640 1 -- start start 2024-03-19T01:56:19.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.198+0000 7f3cf750a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cf0070780 0x7f3cf00c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:19.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.198+0000 7f3cf750a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3cf00c9610 0x7f3cf014e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:19.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.198+0000 7f3cf750a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3cf0152120 0x7f3cf01597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:19.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.198+0000 7f3cf750a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf00c86d0 con 0x7f3cf0070780 2024-03-19T01:56:19.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.198+0000 7f3cf750a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf00c31c0 con 0x7f3cf00c9610 2024-03-19T01:56:19.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.198+0000 7f3cf750a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3cf0165170 con 0x7f3cf0152120 2024-03-19T01:56:19.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.202+0000 7f3cf52a6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cf0070780 0x7f3cf00c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:19.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.202+0000 7f3cf4aa5640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3cf00c9610 0x7f3cf014e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:19.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.202+0000 7f3cf5aa7640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3cf0152120 0x7f3cf01597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:19.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.202+0000 7f3cf52a6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cf0070780 0x7f3cf00c6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57972/0 (socket says 172.21.15.55:57972) 2024-03-19T01:56:19.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.202+0000 7f3cf52a6640 1 -- 172.21.15.55:0/3690385489 learned_addr learned my addr 172.21.15.55:0/3690385489 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:19.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.206+0000 7f3cf52a6640 1 -- 172.21.15.55:0/3690385489 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3cf0152120 msgr2=0x7f3cf01597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:19.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.206+0000 7f3cf52a6640 1 --2- 172.21.15.55:0/3690385489 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3cf0152120 0x7f3cf01597e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:19.203 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.206+0000 7f3cf52a6640 1 -- 172.21.15.55:0/3690385489 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3cf00c9610 msgr2=0x7f3cf014e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:19.203 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.206+0000 7f3cf52a6640 1 --2- 172.21.15.55:0/3690385489 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3cf00c9610 0x7f3cf014e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:19.203 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.206+0000 7f3cf52a6640 1 -- 172.21.15.55:0/3690385489 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf01f2600 con 0x7f3cf0070780 2024-03-19T01:56:19.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.206+0000 7f3cf52a6640 1 --2- 172.21.15.55:0/3690385489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cf0070780 0x7f3cf00c6180 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3cec00e820 tx=0x7f3cec00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:19.209 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.210+0000 7f3ce5ffb640 1 -- 172.21.15.55:0/3690385489 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec017070 con 0x7f3cf0070780 2024-03-19T01:56:19.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.210+0000 7f3ce5ffb640 1 -- 172.21.15.55:0/3690385489 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cec00c940 con 0x7f3cf0070780 2024-03-19T01:56:19.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.210+0000 7f3cf750a640 1 -- 172.21.15.55:0/3690385489 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cf01f1a80 con 0x7f3cf0070780 2024-03-19T01:56:19.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.210+0000 7f3ce5ffb640 1 -- 172.21.15.55:0/3690385489 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec00cd30 con 0x7f3cf0070780 2024-03-19T01:56:19.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.210+0000 7f3cf750a640 1 -- 172.21.15.55:0/3690385489 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cf01f2040 con 0x7f3cf0070780 2024-03-19T01:56:19.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.214+0000 7f3cf750a640 1 -- 172.21.15.55:0/3690385489 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cf0068a40 con 0x7f3cf0070780 2024-03-19T01:56:19.229 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.230+0000 7f3ce5ffb640 1 -- 172.21.15.55:0/3690385489 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f3cec01a070 con 0x7f3cf0070780 2024-03-19T01:56:19.230 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.230+0000 7f3ce5ffb640 1 --2- 172.21.15.55:0/3690385489 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3cd40676b0 0x7f3cd4069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:19.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.230+0000 7f3cf4aa5640 1 --2- 172.21.15.55:0/3690385489 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3cd40676b0 0x7f3cd4069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:19.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.234+0000 7f3cf4aa5640 1 --2- 172.21.15.55:0/3690385489 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3cd40676b0 0x7f3cd4069b70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3cf01511b0 tx=0x7f3ce0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:19.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.238+0000 7f3ce5ffb640 1 -- 172.21.15.55:0/3690385489 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f3cec054530 con 0x7f3cf0070780 2024-03-19T01:56:19.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.530+0000 7f3ce5ffb640 1 -- 172.21.15.55:0/3690385489 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f3cec056490 con 0x7f3cf0070780 2024-03-19T01:56:19.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.750+0000 7f3cf750a640 1 -- 172.21.15.55:0/3690385489 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3cf014c1b0 con 0x7f3cf0070780 2024-03-19T01:56:19.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.770+0000 7f3ce5ffb640 1 -- 172.21.15.55:0/3690385489 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f3cec05a530 con 0x7f3cf0070780 2024-03-19T01:56:19.770 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:19.771 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:19.771 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:19.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.774+0000 7f3cf750a640 1 -- 172.21.15.55:0/3690385489 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3cd40676b0 msgr2=0x7f3cd4069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:19.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.774+0000 7f3cf750a640 1 --2- 172.21.15.55:0/3690385489 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3cd40676b0 0x7f3cd4069b70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3cf01511b0 tx=0x7f3ce0009040 comp rx=0 tx=0).stop 2024-03-19T01:56:19.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.774+0000 7f3cf750a640 1 -- 172.21.15.55:0/3690385489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cf0070780 msgr2=0x7f3cf00c6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:19.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.774+0000 7f3cf750a640 1 --2- 172.21.15.55:0/3690385489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cf0070780 0x7f3cf00c6180 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3cec00e820 tx=0x7f3cec00ece0 comp rx=0 tx=0).stop 2024-03-19T01:56:19.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.774+0000 7f3cf750a640 1 -- 172.21.15.55:0/3690385489 shutdown_connections 2024-03-19T01:56:19.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.774+0000 7f3cf750a640 1 --2- 172.21.15.55:0/3690385489 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3cd40676b0 0x7f3cd4069b70 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:19.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.774+0000 7f3cf750a640 1 --2- 172.21.15.55:0/3690385489 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3cf0152120 0x7f3cf01597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:19.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.774+0000 7f3cf750a640 1 --2- 172.21.15.55:0/3690385489 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3cf00c9610 0x7f3cf014e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:19.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.774+0000 7f3cf750a640 1 --2- 172.21.15.55:0/3690385489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3cf0070780 0x7f3cf00c6180 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:19.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.774+0000 7f3cf750a640 1 -- 172.21.15.55:0/3690385489 >> 172.21.15.55:0/3690385489 conn(0x7f3cf0067600 msgr2=0x7f3cf00c8970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:19.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.774+0000 7f3cf750a640 1 -- 172.21.15.55:0/3690385489 shutdown_connections 2024-03-19T01:56:19.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:19.774+0000 7f3cf750a640 1 -- 172.21.15.55:0/3690385489 wait complete. 2024-03-19T01:56:20.393 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:20.393 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:21.394 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:21.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.510+0000 7f0691efd640 1 Processor -- start 2024-03-19T01:56:21.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.510+0000 7f0691efd640 1 -- start start 2024-03-19T01:56:21.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.510+0000 7f0691efd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f068c0c96b0 0x7f068c070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:21.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.510+0000 7f0691efd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f068c0c9a90 0x7f068c0c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:21.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.510+0000 7f0691efd640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f068c15e3a0 con 0x7f068c1540c0 2024-03-19T01:56:21.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.510+0000 7f0691efd640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f068c15e680 con 0x7f068c0c96b0 2024-03-19T01:56:21.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.510+0000 7f0691efd640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f068c15a330 con 0x7f068c0c9a90 2024-03-19T01:56:21.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.510+0000 7f068b7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f068c0c96b0 0x7f068c070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:21.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.510+0000 7f068b7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f068c0c96b0 0x7f068c070f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57976/0 (socket says 172.21.15.55:57976) 2024-03-19T01:56:21.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.510+0000 7f068b7fe640 1 -- 172.21.15.55:0/2650983423 learned_addr learned my addr 172.21.15.55:0/2650983423 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:21.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.510+0000 7f068affd640 1 --2- 172.21.15.55:0/2650983423 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f068c0c9a90 0x7f068c0c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:21.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.514+0000 7f068b7fe640 1 -- 172.21.15.55:0/2650983423 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f068c0c9a90 msgr2=0x7f068c0c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:21.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.514+0000 7f068b7fe640 1 --2- 172.21.15.55:0/2650983423 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f068c0c9a90 0x7f068c0c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:21.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.514+0000 7f068b7fe640 1 -- 172.21.15.55:0/2650983423 >> v1:172.21.15.84:6789/0 conn(0x7f068c1540c0 legacy=0x7f068c15d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:21.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.514+0000 7f068b7fe640 1 -- 172.21.15.55:0/2650983423 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068c15a610 con 0x7f068c0c96b0 2024-03-19T01:56:21.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.518+0000 7f068b7fe640 1 --2- 172.21.15.55:0/2650983423 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f068c0c96b0 0x7f068c070f20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f067c009870 tx=0x7f067c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1064d0905413616 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:21.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.522+0000 7f068a7fc640 1 -- 172.21.15.55:0/2650983423 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067c0b1070 con 0x7f068c0c96b0 2024-03-19T01:56:21.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.522+0000 7f068a7fc640 1 -- 172.21.15.55:0/2650983423 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f067c0ac620 con 0x7f068c0c96b0 2024-03-19T01:56:21.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.522+0000 7f068a7fc640 1 -- 172.21.15.55:0/2650983423 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067c0ac9f0 con 0x7f068c0c96b0 2024-03-19T01:56:21.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.522+0000 7f0691efd640 1 -- 172.21.15.55:0/2650983423 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f068c0c96b0 msgr2=0x7f068c070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:21.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.522+0000 7f0691efd640 1 --2- 172.21.15.55:0/2650983423 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f068c0c96b0 0x7f068c070f20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f067c009870 tx=0x7f067c01c4d0 comp rx=0 tx=0).stop 2024-03-19T01:56:21.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.522+0000 7f0691efd640 1 -- 172.21.15.55:0/2650983423 shutdown_connections 2024-03-19T01:56:21.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.522+0000 7f0691efd640 1 --2- 172.21.15.55:0/2650983423 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f068c0c9a90 0x7f068c0c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:21.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.522+0000 7f0691efd640 1 --2- 172.21.15.55:0/2650983423 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f068c0c96b0 0x7f068c070f20 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:21.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.522+0000 7f0691efd640 1 -- 172.21.15.55:0/2650983423 >> 172.21.15.55:0/2650983423 conn(0x7f068c0cc9c0 msgr2=0x7f068c0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:21.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.522+0000 7f0691efd640 1 -- 172.21.15.55:0/2650983423 shutdown_connections 2024-03-19T01:56:21.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.522+0000 7f0691efd640 1 -- 172.21.15.55:0/2650983423 wait complete. 2024-03-19T01:56:21.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.526+0000 7f0691efd640 1 Processor -- start 2024-03-19T01:56:21.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.526+0000 7f0691efd640 1 -- start start 2024-03-19T01:56:21.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.526+0000 7f0691efd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f068c0c96b0 0x7f068c155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:21.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.526+0000 7f0691efd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f068c0c9a90 0x7f068c1f2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:21.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.526+0000 7f0691efd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f068c1540c0 0x7f068c151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:21.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.526+0000 7f0691efd640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f068c0c7900 con 0x7f068c0c9a90 2024-03-19T01:56:21.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.526+0000 7f0691efd640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f068c15b0f0 con 0x7f068c1540c0 2024-03-19T01:56:21.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.526+0000 7f0691efd640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f068c0c7a80 con 0x7f068c0c96b0 2024-03-19T01:56:21.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.526+0000 7f068affd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f068c0c9a90 0x7f068c1f2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:21.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.526+0000 7f068affd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f068c0c9a90 0x7f068c1f2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57984/0 (socket says 172.21.15.55:57984) 2024-03-19T01:56:21.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.526+0000 7f068affd640 1 -- 172.21.15.55:0/2534824856 learned_addr learned my addr 172.21.15.55:0/2534824856 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:21.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.526+0000 7f068bfff640 1 --2- 172.21.15.55:0/2534824856 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f068c1540c0 0x7f068c151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:21.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.526+0000 7f068b7fe640 1 --2- 172.21.15.55:0/2534824856 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f068c0c96b0 0x7f068c155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:21.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.530+0000 7f068affd640 1 -- 172.21.15.55:0/2534824856 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f068c0c96b0 msgr2=0x7f068c155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:21.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.530+0000 7f068affd640 1 --2- 172.21.15.55:0/2534824856 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f068c0c96b0 0x7f068c155860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:21.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.530+0000 7f068affd640 1 -- 172.21.15.55:0/2534824856 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f068c1540c0 msgr2=0x7f068c151940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:21.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.530+0000 7f068affd640 1 --2- 172.21.15.55:0/2534824856 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f068c1540c0 0x7f068c151940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:21.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.530+0000 7f068affd640 1 -- 172.21.15.55:0/2534824856 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068c1f76b0 con 0x7f068c0c9a90 2024-03-19T01:56:21.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.534+0000 7f068affd640 1 --2- 172.21.15.55:0/2534824856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f068c0c9a90 0x7f068c1f2860 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f068000d800 tx=0x7f068000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:21.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.534+0000 7f0673fff640 1 -- 172.21.15.55:0/2534824856 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f068000b6c0 con 0x7f068c0c9a90 2024-03-19T01:56:21.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.534+0000 7f0691efd640 1 -- 172.21.15.55:0/2534824856 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f068c14b720 con 0x7f068c0c9a90 2024-03-19T01:56:21.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.534+0000 7f0691efd640 1 -- 172.21.15.55:0/2534824856 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f068c14b160 con 0x7f068c0c9a90 2024-03-19T01:56:21.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.538+0000 7f0691efd640 1 -- 172.21.15.55:0/2534824856 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f068c068a40 con 0x7f068c0c9a90 2024-03-19T01:56:21.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.538+0000 7f0673fff640 1 -- 172.21.15.55:0/2534824856 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0680019070 con 0x7f068c0c9a90 2024-03-19T01:56:21.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.538+0000 7f0673fff640 1 -- 172.21.15.55:0/2534824856 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0680007390 con 0x7f068c0c9a90 2024-03-19T01:56:21.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.806+0000 7f0673fff640 1 -- 172.21.15.55:0/2534824856 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f068000ba60 con 0x7f068c0c9a90 2024-03-19T01:56:21.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.806+0000 7f0673fff640 1 --2- 172.21.15.55:0/2534824856 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0678067660 0x7f0678069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:21.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.806+0000 7f068b7fe640 1 --2- 172.21.15.55:0/2534824856 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0678067660 0x7f0678069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:21.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.806+0000 7f068b7fe640 1 --2- 172.21.15.55:0/2534824856 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0678067660 0x7f0678069b20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f067c0ae000 tx=0x7f067c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:21.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.834+0000 7f0673fff640 1 -- 172.21.15.55:0/2534824856 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f0680054d00 con 0x7f068c0c9a90 2024-03-19T01:56:21.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:21.834+0000 7f0673fff640 1 -- 172.21.15.55:0/2534824856 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f0680056c60 con 0x7f068c0c9a90 2024-03-19T01:56:22.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:22.030+0000 7f0691efd640 1 -- 172.21.15.55:0/2534824856 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f068c0c8dc0 con 0x7f068c0c9a90 2024-03-19T01:56:22.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:22.054+0000 7f0673fff640 1 -- 172.21.15.55:0/2534824856 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f068005ad00 con 0x7f068c0c9a90 2024-03-19T01:56:22.052 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:22.052 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:22.053 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:22.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:22.058+0000 7f0691efd640 1 -- 172.21.15.55:0/2534824856 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0678067660 msgr2=0x7f0678069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:22.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:22.058+0000 7f0691efd640 1 --2- 172.21.15.55:0/2534824856 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0678067660 0x7f0678069b20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f067c0ae000 tx=0x7f067c0a7000 comp rx=0 tx=0).stop 2024-03-19T01:56:22.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:22.058+0000 7f0691efd640 1 -- 172.21.15.55:0/2534824856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f068c0c9a90 msgr2=0x7f068c1f2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:22.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:22.058+0000 7f0691efd640 1 --2- 172.21.15.55:0/2534824856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f068c0c9a90 0x7f068c1f2860 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f068000d800 tx=0x7f068000dcc0 comp rx=0 tx=0).stop 2024-03-19T01:56:22.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:22.058+0000 7f0691efd640 1 -- 172.21.15.55:0/2534824856 shutdown_connections 2024-03-19T01:56:22.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:22.058+0000 7f0691efd640 1 --2- 172.21.15.55:0/2534824856 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0678067660 0x7f0678069b20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:22.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:22.058+0000 7f0691efd640 1 --2- 172.21.15.55:0/2534824856 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f068c1540c0 0x7f068c151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:22.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:22.058+0000 7f0691efd640 1 --2- 172.21.15.55:0/2534824856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f068c0c9a90 0x7f068c1f2860 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:22.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:22.058+0000 7f0691efd640 1 --2- 172.21.15.55:0/2534824856 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f068c0c96b0 0x7f068c155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:22.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:22.058+0000 7f0691efd640 1 -- 172.21.15.55:0/2534824856 >> 172.21.15.55:0/2534824856 conn(0x7f068c0cc9c0 msgr2=0x7f068c15d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:22.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:22.058+0000 7f0691efd640 1 -- 172.21.15.55:0/2534824856 shutdown_connections 2024-03-19T01:56:22.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:22.058+0000 7f0691efd640 1 -- 172.21.15.55:0/2534824856 wait complete. 2024-03-19T01:56:22.680 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:22.680 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:23.681 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:23.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.806+0000 7ff72dd69640 1 Processor -- start 2024-03-19T01:56:23.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.806+0000 7ff72dd69640 1 -- start start 2024-03-19T01:56:23.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.806+0000 7ff72dd69640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff728154340 0x7ff72815e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:23.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.806+0000 7ff72dd69640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff7280c73c0 0x7ff72815aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:23.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.806+0000 7ff72dd69640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff728071280 con 0x7ff728154720 2024-03-19T01:56:23.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.806+0000 7ff72dd69640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7ff728073840 con 0x7ff728154340 2024-03-19T01:56:23.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.806+0000 7ff72dd69640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff7280c7850 con 0x7ff7280c73c0 2024-03-19T01:56:23.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.806+0000 7ff7277fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff728154340 0x7ff72815e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:23.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.806+0000 7ff727fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff7280c73c0 0x7ff72815aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:23.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.810+0000 7ff7277fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff728154340 0x7ff72815e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:47096/0 (socket says 172.21.15.55:47096) 2024-03-19T01:56:23.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.810+0000 7ff7277fe640 1 -- 172.21.15.55:0/2679484866 learned_addr learned my addr 172.21.15.55:0/2679484866 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:23.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.814+0000 7ff7277fe640 1 -- 172.21.15.55:0/2679484866 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff7280c73c0 msgr2=0x7ff72815aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:23.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.814+0000 7ff7277fe640 1 --2- 172.21.15.55:0/2679484866 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff7280c73c0 0x7ff72815aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:23.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.814+0000 7ff7277fe640 1 -- 172.21.15.55:0/2679484866 >> v1:172.21.15.84:6789/0 conn(0x7ff728154720 legacy=0x7ff728072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:23.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.814+0000 7ff7277fe640 1 -- 172.21.15.55:0/2679484866 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff72815d900 con 0x7ff728154340 2024-03-19T01:56:23.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.818+0000 7ff7277fe640 1 --2- 172.21.15.55:0/2679484866 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff728154340 0x7ff72815e3a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff7180098a0 tx=0x7ff71801c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3683fce94cbd1ff server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:23.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.822+0000 7ff7267fc640 1 -- 172.21.15.55:0/2679484866 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7180b0070 con 0x7ff728154340 2024-03-19T01:56:23.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.826+0000 7ff7267fc640 1 -- 172.21.15.55:0/2679484866 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7180ab600 con 0x7ff728154340 2024-03-19T01:56:23.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.826+0000 7ff7267fc640 1 -- 172.21.15.55:0/2679484866 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7180ab9d0 con 0x7ff728154340 2024-03-19T01:56:23.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.826+0000 7ff72dd69640 1 -- 172.21.15.55:0/2679484866 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff728154340 msgr2=0x7ff72815e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:23.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.826+0000 7ff72dd69640 1 --2- 172.21.15.55:0/2679484866 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff728154340 0x7ff72815e3a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff7180098a0 tx=0x7ff71801c540 comp rx=0 tx=0).stop 2024-03-19T01:56:23.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.826+0000 7ff72dd69640 1 -- 172.21.15.55:0/2679484866 shutdown_connections 2024-03-19T01:56:23.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.826+0000 7ff72dd69640 1 --2- 172.21.15.55:0/2679484866 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff7280c73c0 0x7ff72815aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:23.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.826+0000 7ff72dd69640 1 --2- 172.21.15.55:0/2679484866 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff728154340 0x7ff72815e3a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:23.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.826+0000 7ff72dd69640 1 -- 172.21.15.55:0/2679484866 >> 172.21.15.55:0/2679484866 conn(0x7ff7280cc9c0 msgr2=0x7ff7280ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:23.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.826+0000 7ff72dd69640 1 -- 172.21.15.55:0/2679484866 shutdown_connections 2024-03-19T01:56:23.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.826+0000 7ff72dd69640 1 -- 172.21.15.55:0/2679484866 wait complete. 2024-03-19T01:56:23.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.826+0000 7ff72dd69640 1 Processor -- start 2024-03-19T01:56:23.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.830+0000 7ff72dd69640 1 -- start start 2024-03-19T01:56:23.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.830+0000 7ff72dd69640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7280c73c0 0x7ff72814c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:23.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.830+0000 7ff72dd69640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff728154340 0x7ff728152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:23.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.830+0000 7ff72dd69640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff728154720 0x7ff728152fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:23.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.830+0000 7ff72dd69640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff728166b70 con 0x7ff728154340 2024-03-19T01:56:23.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.830+0000 7ff72dd69640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff72815b1e0 con 0x7ff7280c73c0 2024-03-19T01:56:23.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.830+0000 7ff72dd69640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff728166cf0 con 0x7ff728154720 2024-03-19T01:56:23.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.830+0000 7ff726ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff728154340 0x7ff728152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:23.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.830+0000 7ff7277fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7280c73c0 0x7ff72814c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:23.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.830+0000 7ff727fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff728154720 0x7ff728152fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:23.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.830+0000 7ff726ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff728154340 0x7ff728152a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57996/0 (socket says 172.21.15.55:57996) 2024-03-19T01:56:23.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.830+0000 7ff726ffd640 1 -- 172.21.15.55:0/2628153768 learned_addr learned my addr 172.21.15.55:0/2628153768 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:23.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.834+0000 7ff726ffd640 1 -- 172.21.15.55:0/2628153768 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff728154720 msgr2=0x7ff728152fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:23.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.834+0000 7ff726ffd640 1 --2- 172.21.15.55:0/2628153768 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff728154720 0x7ff728152fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:23.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.834+0000 7ff726ffd640 1 -- 172.21.15.55:0/2628153768 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7280c73c0 msgr2=0x7ff72814c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:23.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.834+0000 7ff726ffd640 1 --2- 172.21.15.55:0/2628153768 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7280c73c0 0x7ff72814c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:23.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.834+0000 7ff726ffd640 1 -- 172.21.15.55:0/2628153768 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff728151a30 con 0x7ff728154340 2024-03-19T01:56:23.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.838+0000 7ff726ffd640 1 --2- 172.21.15.55:0/2628153768 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff728154340 0x7ff728152a80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff71c00c450 tx=0x7ff71c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:23.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.842+0000 7ff70ffff640 1 -- 172.21.15.55:0/2628153768 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71c016020 con 0x7ff728154340 2024-03-19T01:56:23.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.842+0000 7ff70ffff640 1 -- 172.21.15.55:0/2628153768 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff71c0052d0 con 0x7ff728154340 2024-03-19T01:56:23.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.842+0000 7ff70ffff640 1 -- 172.21.15.55:0/2628153768 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71c007260 con 0x7ff728154340 2024-03-19T01:56:23.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.842+0000 7ff72dd69640 1 -- 172.21.15.55:0/2628153768 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff728151d10 con 0x7ff728154340 2024-03-19T01:56:23.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.842+0000 7ff72dd69640 1 -- 172.21.15.55:0/2628153768 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff728152560 con 0x7ff728154340 2024-03-19T01:56:23.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:23.842+0000 7ff72dd69640 1 -- 172.21.15.55:0/2628153768 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff728152230 con 0x7ff728154340 2024-03-19T01:56:24.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:24.110+0000 7ff70ffff640 1 -- 172.21.15.55:0/2628153768 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ff71c007400 con 0x7ff728154340 2024-03-19T01:56:24.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:24.114+0000 7ff70ffff640 1 --2- 172.21.15.55:0/2628153768 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff7140676e0 0x7ff714069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:24.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:24.114+0000 7ff7277fe640 1 --2- 172.21.15.55:0/2628153768 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff7140676e0 0x7ff714069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:24.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:24.114+0000 7ff7277fe640 1 --2- 172.21.15.55:0/2628153768 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff7140676e0 0x7ff714069ba0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff7280709e0 tx=0x7ff7180a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:24.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:24.126+0000 7ff70ffff640 1 -- 172.21.15.55:0/2628153768 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7ff71c054690 con 0x7ff728154340 2024-03-19T01:56:24.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:24.126+0000 7ff70ffff640 1 -- 172.21.15.55:0/2628153768 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ff728152230 con 0x7ff728154340 2024-03-19T01:56:24.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:24.374+0000 7ff72dd69640 1 -- 172.21.15.55:0/2628153768 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff728068a40 con 0x7ff728154340 2024-03-19T01:56:24.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:24.390+0000 7ff70ffff640 1 -- 172.21.15.55:0/2628153768 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7ff71c020020 con 0x7ff728154340 2024-03-19T01:56:24.387 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:24.387 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:24.388 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:24.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:24.394+0000 7ff72dd69640 1 -- 172.21.15.55:0/2628153768 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff7140676e0 msgr2=0x7ff714069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:24.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:24.394+0000 7ff72dd69640 1 --2- 172.21.15.55:0/2628153768 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff7140676e0 0x7ff714069ba0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff7280709e0 tx=0x7ff7180a6000 comp rx=0 tx=0).stop 2024-03-19T01:56:24.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:24.394+0000 7ff72dd69640 1 -- 172.21.15.55:0/2628153768 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff728154340 msgr2=0x7ff728152a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:24.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:24.394+0000 7ff72dd69640 1 --2- 172.21.15.55:0/2628153768 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff728154340 0x7ff728152a80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff71c00c450 tx=0x7ff71c00c910 comp rx=0 tx=0).stop 2024-03-19T01:56:24.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:24.394+0000 7ff72dd69640 1 -- 172.21.15.55:0/2628153768 shutdown_connections 2024-03-19T01:56:24.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:24.394+0000 7ff72dd69640 1 --2- 172.21.15.55:0/2628153768 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff7140676e0 0x7ff714069ba0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:24.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:24.394+0000 7ff72dd69640 1 --2- 172.21.15.55:0/2628153768 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff728154720 0x7ff728152fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:24.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:24.394+0000 7ff72dd69640 1 --2- 172.21.15.55:0/2628153768 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff728154340 0x7ff728152a80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:24.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:24.394+0000 7ff72dd69640 1 --2- 172.21.15.55:0/2628153768 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7280c73c0 0x7ff72814c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:24.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:24.394+0000 7ff72dd69640 1 -- 172.21.15.55:0/2628153768 >> 172.21.15.55:0/2628153768 conn(0x7ff7280cc9c0 msgr2=0x7ff7280c83f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:24.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:24.394+0000 7ff72dd69640 1 -- 172.21.15.55:0/2628153768 shutdown_connections 2024-03-19T01:56:24.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:24.394+0000 7ff72dd69640 1 -- 172.21.15.55:0/2628153768 wait complete. 2024-03-19T01:56:25.010 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:25.011 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:26.012 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:26.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.130+0000 7fc821788640 1 Processor -- start 2024-03-19T01:56:26.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.130+0000 7fc821788640 1 -- start start 2024-03-19T01:56:26.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.130+0000 7fc821788640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fc81c070780 0x7fc81c15b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:26.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.130+0000 7fc821788640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc81c06fff0 con 0x7fc81c152120 2024-03-19T01:56:26.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.130+0000 7fc821788640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc81c0c39c0 con 0x7fc81c15bb00 2024-03-19T01:56:26.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.130+0000 7fc821788640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fc81c152500 con 0x7fc81c070780 2024-03-19T01:56:26.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.130+0000 7fc81affd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fc81c152120 0x7fc81c0c90d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:46656/0 (socket says 172.21.15.55:46656) 2024-03-19T01:56:26.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.130+0000 7fc81affd640 1 -- 172.21.15.55:0/1320492687 learned_addr learned my addr 172.21.15.55:0/1320492687 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:56:26.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.130+0000 7fc81a7fc640 1 --2- 172.21.15.55:0/1320492687 >> v2:172.21.15.84:3300/0 conn(0x7fc81c070780 0x7fc81c15b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:26.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.134+0000 7fc819ffb640 1 -- 172.21.15.55:0/1320492687 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1454091934 0 0) 0x7fc81c06fff0 con 0x7fc81c152120 2024-03-19T01:56:26.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.134+0000 7fc819ffb640 1 -- 172.21.15.55:0/1320492687 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc804003530 con 0x7fc81c152120 2024-03-19T01:56:26.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.138+0000 7fc81a7fc640 1 -- 172.21.15.55:0/1320492687 >> v1:172.21.15.84:6789/0 conn(0x7fc81c15bb00 legacy=0x7fc81c15bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:26.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.138+0000 7fc81a7fc640 1 -- 172.21.15.55:0/1320492687 >> v1:172.21.15.55:6789/0 conn(0x7fc81c152120 legacy=0x7fc81c0c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:26.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.138+0000 7fc81a7fc640 1 -- 172.21.15.55:0/1320492687 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc81c0702b0 con 0x7fc81c070780 2024-03-19T01:56:26.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.142+0000 7fc81a7fc640 1 --2- 172.21.15.55:0/1320492687 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc81c070780 0x7fc81c15b5c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc8100097c0 tx=0x7fc81001c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5642e28ca94502d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:26.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.150+0000 7fc819ffb640 1 -- 172.21.15.55:0/1320492687 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8100b0070 con 0x7fc81c070780 2024-03-19T01:56:26.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.150+0000 7fc819ffb640 1 -- 172.21.15.55:0/1320492687 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8100ab600 con 0x7fc81c070780 2024-03-19T01:56:26.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.150+0000 7fc819ffb640 1 -- 172.21.15.55:0/1320492687 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8100ab9d0 con 0x7fc81c070780 2024-03-19T01:56:26.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.150+0000 7fc821788640 1 -- 172.21.15.55:0/1320492687 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc81c070780 msgr2=0x7fc81c15b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:26.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.150+0000 7fc821788640 1 --2- 172.21.15.55:0/1320492687 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc81c070780 0x7fc81c15b5c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc8100097c0 tx=0x7fc81001c460 comp rx=0 tx=0).stop 2024-03-19T01:56:26.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.150+0000 7fc821788640 1 -- 172.21.15.55:0/1320492687 shutdown_connections 2024-03-19T01:56:26.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.150+0000 7fc821788640 1 --2- 172.21.15.55:0/1320492687 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc81c070780 0x7fc81c15b5c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:26.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.150+0000 7fc821788640 1 -- 172.21.15.55:0/1320492687 >> 172.21.15.55:0/1320492687 conn(0x7fc81c067600 msgr2=0x7fc81c071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:26.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.150+0000 7fc821788640 1 -- 172.21.15.55:0/1320492687 shutdown_connections 2024-03-19T01:56:26.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.150+0000 7fc821788640 1 -- 172.21.15.55:0/1320492687 wait complete. 2024-03-19T01:56:26.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.150+0000 7fc821788640 1 Processor -- start 2024-03-19T01:56:26.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.150+0000 7fc821788640 1 -- start start 2024-03-19T01:56:26.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.154+0000 7fc821788640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc81c070780 0x7fc81c15ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:26.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.154+0000 7fc821788640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc81c152120 0x7fc81c1f1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:26.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.154+0000 7fc821788640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc81c15bb00 0x7fc81c1f1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:26.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.154+0000 7fc821788640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc81c0c97e0 con 0x7fc81c152120 2024-03-19T01:56:26.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.154+0000 7fc821788640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc81c0c3530 con 0x7fc81c15bb00 2024-03-19T01:56:26.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.154+0000 7fc821788640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc81c0c86d0 con 0x7fc81c070780 2024-03-19T01:56:26.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.154+0000 7fc81a7fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc81c152120 0x7fc81c1f1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:26.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.154+0000 7fc81a7fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc81c152120 0x7fc81c1f1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58002/0 (socket says 172.21.15.55:58002) 2024-03-19T01:56:26.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.154+0000 7fc81a7fc640 1 -- 172.21.15.55:0/1177443567 learned_addr learned my addr 172.21.15.55:0/1177443567 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:26.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.154+0000 7fc81affd640 1 --2- 172.21.15.55:0/1177443567 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc81c070780 0x7fc81c15ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:26.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.154+0000 7fc81b7fe640 1 --2- 172.21.15.55:0/1177443567 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc81c15bb00 0x7fc81c1f1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:26.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.158+0000 7fc81a7fc640 1 -- 172.21.15.55:0/1177443567 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc81c070780 msgr2=0x7fc81c15ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:26.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.158+0000 7fc81a7fc640 1 --2- 172.21.15.55:0/1177443567 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc81c070780 0x7fc81c15ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:26.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.158+0000 7fc81a7fc640 1 -- 172.21.15.55:0/1177443567 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc81c15bb00 msgr2=0x7fc81c1f1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:26.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.158+0000 7fc81a7fc640 1 --2- 172.21.15.55:0/1177443567 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc81c15bb00 0x7fc81c1f1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:26.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.158+0000 7fc81a7fc640 1 -- 172.21.15.55:0/1177443567 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc81c1421b0 con 0x7fc81c152120 2024-03-19T01:56:26.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.158+0000 7fc81a7fc640 1 --2- 172.21.15.55:0/1177443567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc81c152120 0x7fc81c1f1300 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc810019070 tx=0x7fc810019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:26.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.162+0000 7fc8037fe640 1 -- 172.21.15.55:0/1177443567 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8100ba070 con 0x7fc81c152120 2024-03-19T01:56:26.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.162+0000 7fc8037fe640 1 -- 172.21.15.55:0/1177443567 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8100b43f0 con 0x7fc81c152120 2024-03-19T01:56:26.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.162+0000 7fc8037fe640 1 -- 172.21.15.55:0/1177443567 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8100b0040 con 0x7fc81c152120 2024-03-19T01:56:26.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.166+0000 7fc821788640 1 -- 172.21.15.55:0/1177443567 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc81c14b7e0 con 0x7fc81c152120 2024-03-19T01:56:26.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.166+0000 7fc821788640 1 -- 172.21.15.55:0/1177443567 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc81c142450 con 0x7fc81c152120 2024-03-19T01:56:26.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.166+0000 7fc821788640 1 -- 172.21.15.55:0/1177443567 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc81c068a40 con 0x7fc81c152120 2024-03-19T01:56:26.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.198+0000 7fc8037fe640 1 -- 172.21.15.55:0/1177443567 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc8100b47c0 con 0x7fc81c152120 2024-03-19T01:56:26.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.198+0000 7fc8037fe640 1 --2- 172.21.15.55:0/1177443567 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc808067690 0x7fc808069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:26.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.198+0000 7fc81affd640 1 --2- 172.21.15.55:0/1177443567 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc808067690 0x7fc808069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:26.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.198+0000 7fc81affd640 1 --2- 172.21.15.55:0/1177443567 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc808067690 0x7fc808069b50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc80c007fd0 tx=0x7fc80c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:26.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.202+0000 7fc8037fe640 1 -- 172.21.15.55:0/1177443567 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fc8100ed3c0 con 0x7fc81c152120 2024-03-19T01:56:26.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.486+0000 7fc8037fe640 1 -- 172.21.15.55:0/1177443567 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc8100c3020 con 0x7fc81c152120 2024-03-19T01:56:26.701 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.702+0000 7fc821788640 1 -- 172.21.15.55:0/1177443567 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc81c0c84b0 con 0x7fc81c152120 2024-03-19T01:56:26.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.722+0000 7fc8037fe640 1 -- 172.21.15.55:0/1177443567 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fc810016050 con 0x7fc81c152120 2024-03-19T01:56:26.721 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:26.721 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:26.721 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:26.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.726+0000 7fc821788640 1 -- 172.21.15.55:0/1177443567 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc808067690 msgr2=0x7fc808069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:26.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.726+0000 7fc821788640 1 --2- 172.21.15.55:0/1177443567 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc808067690 0x7fc808069b50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc80c007fd0 tx=0x7fc80c009040 comp rx=0 tx=0).stop 2024-03-19T01:56:26.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.726+0000 7fc821788640 1 -- 172.21.15.55:0/1177443567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc81c152120 msgr2=0x7fc81c1f1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:26.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.726+0000 7fc821788640 1 --2- 172.21.15.55:0/1177443567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc81c152120 0x7fc81c1f1300 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc810019070 tx=0x7fc810019200 comp rx=0 tx=0).stop 2024-03-19T01:56:26.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.726+0000 7fc821788640 1 -- 172.21.15.55:0/1177443567 shutdown_connections 2024-03-19T01:56:26.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.726+0000 7fc821788640 1 --2- 172.21.15.55:0/1177443567 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc808067690 0x7fc808069b50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:26.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.726+0000 7fc821788640 1 --2- 172.21.15.55:0/1177443567 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc81c15bb00 0x7fc81c1f1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:26.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.726+0000 7fc821788640 1 --2- 172.21.15.55:0/1177443567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc81c152120 0x7fc81c1f1300 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:26.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.726+0000 7fc821788640 1 --2- 172.21.15.55:0/1177443567 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc81c070780 0x7fc81c15ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:26.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.726+0000 7fc821788640 1 -- 172.21.15.55:0/1177443567 >> 172.21.15.55:0/1177443567 conn(0x7fc81c067600 msgr2=0x7fc81c150c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:26.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.726+0000 7fc821788640 1 -- 172.21.15.55:0/1177443567 shutdown_connections 2024-03-19T01:56:26.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:26.726+0000 7fc821788640 1 -- 172.21.15.55:0/1177443567 wait complete. 2024-03-19T01:56:27.343 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:27.343 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:28.344 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:28.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.454+0000 7fcc89cd0640 1 Processor -- start 2024-03-19T01:56:28.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.454+0000 7fcc89cd0640 1 -- start start 2024-03-19T01:56:28.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.458+0000 7fcc89cd0640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fcc84158450 0x7fcc8406b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:28.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.458+0000 7fcc89cd0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc84162130 0x7fcc841625c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:28.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.458+0000 7fcc89cd0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc84162b00 con 0x7fcc84158830 2024-03-19T01:56:28.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.458+0000 7fcc89cd0640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fcc8415e620 con 0x7fcc84158450 2024-03-19T01:56:28.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.458+0000 7fcc89cd0640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fcc8406b7e0 con 0x7fcc84162130 2024-03-19T01:56:28.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.458+0000 7fcc83fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc84162130 0x7fcc841625c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:28.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.458+0000 7fcc837fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fcc84158450 0x7fcc8406b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:28.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.458+0000 7fcc82ffd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fcc84158830 0x7fcc8406bf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:54144/0 (socket says 172.21.15.55:54144) 2024-03-19T01:56:28.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.458+0000 7fcc82ffd640 1 -- 172.21.15.55:0/2086301450 learned_addr learned my addr 172.21.15.55:0/2086301450 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:56:28.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.466+0000 7fcc83fff640 1 -- 172.21.15.55:0/2086301450 >> v2:172.21.15.84:3300/0 conn(0x7fcc84158450 msgr2=0x7fcc8406b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:28.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.466+0000 7fcc83fff640 1 --2- 172.21.15.55:0/2086301450 >> v2:172.21.15.84:3300/0 conn(0x7fcc84158450 0x7fcc8406b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:28.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.466+0000 7fcc83fff640 1 -- 172.21.15.55:0/2086301450 >> v1:172.21.15.84:6789/0 conn(0x7fcc84158830 legacy=0x7fcc8406bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:28.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.466+0000 7fcc83fff640 1 -- 172.21.15.55:0/2086301450 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc8415e8c0 con 0x7fcc84162130 2024-03-19T01:56:28.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.470+0000 7fcc83fff640 1 --2- 172.21.15.55:0/2086301450 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc84162130 0x7fcc841625c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c009dc0 tx=0x7fcc7c01c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cfd18f084ecf877 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:28.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.478+0000 7fcc827fc640 1 -- 172.21.15.55:0/2086301450 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c0b2070 con 0x7fcc84162130 2024-03-19T01:56:28.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.478+0000 7fcc827fc640 1 -- 172.21.15.55:0/2086301450 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc7c0a5900 con 0x7fcc84162130 2024-03-19T01:56:28.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.478+0000 7fcc827fc640 1 -- 172.21.15.55:0/2086301450 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c0a5cd0 con 0x7fcc84162130 2024-03-19T01:56:28.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.478+0000 7fcc89cd0640 1 -- 172.21.15.55:0/2086301450 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc84162130 msgr2=0x7fcc841625c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:28.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.478+0000 7fcc89cd0640 1 --2- 172.21.15.55:0/2086301450 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc84162130 0x7fcc841625c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c009dc0 tx=0x7fcc7c01c950 comp rx=0 tx=0).stop 2024-03-19T01:56:28.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.478+0000 7fcc89cd0640 1 -- 172.21.15.55:0/2086301450 shutdown_connections 2024-03-19T01:56:28.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.478+0000 7fcc89cd0640 1 --2- 172.21.15.55:0/2086301450 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc84162130 0x7fcc841625c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:28.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.478+0000 7fcc89cd0640 1 --2- 172.21.15.55:0/2086301450 >> v2:172.21.15.84:3300/0 conn(0x7fcc84158450 0x7fcc8406b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:28.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.478+0000 7fcc89cd0640 1 -- 172.21.15.55:0/2086301450 >> 172.21.15.55:0/2086301450 conn(0x7fcc8406fb40 msgr2=0x7fcc8406ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:28.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.478+0000 7fcc89cd0640 1 -- 172.21.15.55:0/2086301450 shutdown_connections 2024-03-19T01:56:28.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.478+0000 7fcc89cd0640 1 -- 172.21.15.55:0/2086301450 wait complete. 2024-03-19T01:56:28.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.478+0000 7fcc89cd0640 1 Processor -- start 2024-03-19T01:56:28.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.478+0000 7fcc89cd0640 1 -- start start 2024-03-19T01:56:28.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.478+0000 7fcc89cd0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc84158450 0x7fcc84160e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:28.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.478+0000 7fcc89cd0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc84158830 0x7fcc8414d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:28.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.478+0000 7fcc89cd0640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc84162130 0x7fcc84145e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:28.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.478+0000 7fcc89cd0640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcc840c6b70 con 0x7fcc84158450 2024-03-19T01:56:28.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.478+0000 7fcc89cd0640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8415f440 con 0x7fcc84162130 2024-03-19T01:56:28.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.478+0000 7fcc89cd0640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fcc840c9340 con 0x7fcc84158830 2024-03-19T01:56:28.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.482+0000 7fcc837fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc84158450 0x7fcc84160e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:28.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.482+0000 7fcc83fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc84162130 0x7fcc84145e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:28.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.482+0000 7fcc837fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc84158450 0x7fcc84160e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58010/0 (socket says 172.21.15.55:58010) 2024-03-19T01:56:28.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.482+0000 7fcc837fe640 1 -- 172.21.15.55:0/1599318034 learned_addr learned my addr 172.21.15.55:0/1599318034 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:28.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.482+0000 7fcc82ffd640 1 --2- 172.21.15.55:0/1599318034 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc84158830 0x7fcc8414d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:28.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.486+0000 7fcc837fe640 1 -- 172.21.15.55:0/1599318034 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc84158830 msgr2=0x7fcc8414d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:28.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.486+0000 7fcc837fe640 1 --2- 172.21.15.55:0/1599318034 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc84158830 0x7fcc8414d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:28.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.486+0000 7fcc837fe640 1 -- 172.21.15.55:0/1599318034 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc84162130 msgr2=0x7fcc84145e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:28.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.486+0000 7fcc837fe640 1 --2- 172.21.15.55:0/1599318034 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc84162130 0x7fcc84145e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:28.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.486+0000 7fcc837fe640 1 -- 172.21.15.55:0/1599318034 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc841613d0 con 0x7fcc84158450 2024-03-19T01:56:28.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.486+0000 7fcc837fe640 1 --2- 172.21.15.55:0/1599318034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc84158450 0x7fcc84160e90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcc7400e8a0 tx=0x7fcc7400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:28.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.494+0000 7fcc7bfff640 1 -- 172.21.15.55:0/1599318034 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc74017070 con 0x7fcc84158450 2024-03-19T01:56:28.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.494+0000 7fcc7bfff640 1 -- 172.21.15.55:0/1599318034 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc7400c940 con 0x7fcc84158450 2024-03-19T01:56:28.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.494+0000 7fcc89cd0640 1 -- 172.21.15.55:0/1599318034 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc84161a00 con 0x7fcc84158450 2024-03-19T01:56:28.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.494+0000 7fcc7bfff640 1 -- 172.21.15.55:0/1599318034 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc7400ccf0 con 0x7fcc84158450 2024-03-19T01:56:28.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.494+0000 7fcc89cd0640 1 -- 172.21.15.55:0/1599318034 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc8415fb60 con 0x7fcc84158450 2024-03-19T01:56:28.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.494+0000 7fcc89cd0640 1 -- 172.21.15.55:0/1599318034 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc84068a40 con 0x7fcc84158450 2024-03-19T01:56:28.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.514+0000 7fcc7bfff640 1 -- 172.21.15.55:0/1599318034 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fcc7401a070 con 0x7fcc84158450 2024-03-19T01:56:28.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.514+0000 7fcc7bfff640 1 --2- 172.21.15.55:0/1599318034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcc640676b0 0x7fcc64069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:28.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.514+0000 7fcc82ffd640 1 --2- 172.21.15.55:0/1599318034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcc640676b0 0x7fcc64069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:28.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.514+0000 7fcc82ffd640 1 --2- 172.21.15.55:0/1599318034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcc640676b0 0x7fcc64069b70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcc84146b10 tx=0x7fcc6c007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:28.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.778+0000 7fcc7bfff640 1 -- 172.21.15.55:0/1599318034 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fcc740545c0 con 0x7fcc84158450 2024-03-19T01:56:28.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:28.778+0000 7fcc7bfff640 1 -- 172.21.15.55:0/1599318034 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fcc74056520 con 0x7fcc84158450 2024-03-19T01:56:29.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:29.062+0000 7fcc89cd0640 1 -- 172.21.15.55:0/1599318034 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc840c31d0 con 0x7fcc84158450 2024-03-19T01:56:29.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:29.078+0000 7fcc7bfff640 1 -- 172.21.15.55:0/1599318034 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fcc7405a5c0 con 0x7fcc84158450 2024-03-19T01:56:29.078 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:29.079 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:29.079 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:29.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:29.082+0000 7fcc89cd0640 1 -- 172.21.15.55:0/1599318034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcc640676b0 msgr2=0x7fcc64069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:29.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:29.082+0000 7fcc89cd0640 1 --2- 172.21.15.55:0/1599318034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcc640676b0 0x7fcc64069b70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcc84146b10 tx=0x7fcc6c007e90 comp rx=0 tx=0).stop 2024-03-19T01:56:29.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:29.082+0000 7fcc89cd0640 1 -- 172.21.15.55:0/1599318034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc84158450 msgr2=0x7fcc84160e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:29.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:29.082+0000 7fcc89cd0640 1 --2- 172.21.15.55:0/1599318034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc84158450 0x7fcc84160e90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcc7400e8a0 tx=0x7fcc7400eca0 comp rx=0 tx=0).stop 2024-03-19T01:56:29.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:29.082+0000 7fcc89cd0640 1 -- 172.21.15.55:0/1599318034 shutdown_connections 2024-03-19T01:56:29.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:29.082+0000 7fcc89cd0640 1 --2- 172.21.15.55:0/1599318034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcc640676b0 0x7fcc64069b70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:29.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:29.082+0000 7fcc89cd0640 1 --2- 172.21.15.55:0/1599318034 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc84162130 0x7fcc84145e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:29.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:29.082+0000 7fcc89cd0640 1 --2- 172.21.15.55:0/1599318034 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc84158830 0x7fcc8414d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:29.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:29.082+0000 7fcc89cd0640 1 --2- 172.21.15.55:0/1599318034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc84158450 0x7fcc84160e90 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:29.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:29.082+0000 7fcc89cd0640 1 -- 172.21.15.55:0/1599318034 >> 172.21.15.55:0/1599318034 conn(0x7fcc8406fb40 msgr2=0x7fcc84156580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:29.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:29.082+0000 7fcc89cd0640 1 -- 172.21.15.55:0/1599318034 shutdown_connections 2024-03-19T01:56:29.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:29.082+0000 7fcc89cd0640 1 -- 172.21.15.55:0/1599318034 wait complete. 2024-03-19T01:56:29.700 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:29.700 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:30.701 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:30.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.826+0000 7f1c368f7640 1 Processor -- start 2024-03-19T01:56:30.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.826+0000 7f1c368f7640 1 -- start start 2024-03-19T01:56:30.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.826+0000 7f1c368f7640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1c300cbf30 0x7f1c300cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:30.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.826+0000 7f1c368f7640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1c3006c6f0 0x7f1c300cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:30.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.826+0000 7f1c368f7640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f1c3015a3b0 0x7f1c3015a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:30.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.826+0000 7f1c368f7640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f1c300ce640 con 0x7f1c300cbf30 2024-03-19T01:56:30.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.826+0000 7f1c368f7640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f1c3006cb90 con 0x7f1c3015a3b0 2024-03-19T01:56:30.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.826+0000 7f1c368f7640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1c300cda30 con 0x7f1c3006c6f0 2024-03-19T01:56:30.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.826+0000 7f1c2ffff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1c300cbf30 0x7f1c300cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:30.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.826+0000 7f1c2f7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1c3006c6f0 0x7f1c300cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:30.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.830+0000 7f1c2ffff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1c300cbf30 0x7f1c300cead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:36764/0 (socket says 172.21.15.55:36764) 2024-03-19T01:56:30.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.830+0000 7f1c2ffff640 1 -- 172.21.15.55:0/3684064534 learned_addr learned my addr 172.21.15.55:0/3684064534 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:30.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.830+0000 7f1c34e94640 1 --2- 172.21.15.55:0/3684064534 >> v2:172.21.15.84:3300/0 conn(0x7f1c3015a3b0 0x7f1c3015a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:30.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.830+0000 7f1c2ffff640 1 -- 172.21.15.55:0/3684064534 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1c3006c6f0 msgr2=0x7f1c300cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:30.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.830+0000 7f1c2ffff640 1 --2- 172.21.15.55:0/3684064534 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1c3006c6f0 0x7f1c300cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:30.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.830+0000 7f1c2ffff640 1 -- 172.21.15.55:0/3684064534 >> v2:172.21.15.84:3300/0 conn(0x7f1c3015a3b0 msgr2=0x7f1c3015a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:30.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.830+0000 7f1c2ffff640 1 --2- 172.21.15.55:0/3684064534 >> v2:172.21.15.84:3300/0 conn(0x7f1c3015a3b0 0x7f1c3015a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:30.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.830+0000 7f1c2ffff640 1 -- 172.21.15.55:0/3684064534 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c300cdd10 con 0x7f1c300cbf30 2024-03-19T01:56:30.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.830+0000 7f1c2f7fe640 1 --2- 172.21.15.55:0/3684064534 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1c3006c6f0 0x7f1c300cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:56:30.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.834+0000 7f1c2ffff640 1 --2- 172.21.15.55:0/3684064534 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c300cbf30 0x7f1c300cead0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1c300c3350 tx=0x7f1c2001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4448e14b085bcd52 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:30.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.842+0000 7f1c2effd640 1 -- 172.21.15.55:0/3684064534 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c200ab650 con 0x7f1c300cbf30 2024-03-19T01:56:30.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.842+0000 7f1c2effd640 1 -- 172.21.15.55:0/3684064534 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c200b0070 con 0x7f1c300cbf30 2024-03-19T01:56:30.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.842+0000 7f1c2effd640 1 -- 172.21.15.55:0/3684064534 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c200aba10 con 0x7f1c300cbf30 2024-03-19T01:56:30.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.842+0000 7f1c368f7640 1 -- 172.21.15.55:0/3684064534 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c300cbf30 msgr2=0x7f1c300cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:30.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.842+0000 7f1c368f7640 1 --2- 172.21.15.55:0/3684064534 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c300cbf30 0x7f1c300cead0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1c300c3350 tx=0x7f1c2001c490 comp rx=0 tx=0).stop 2024-03-19T01:56:30.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.842+0000 7f1c368f7640 1 -- 172.21.15.55:0/3684064534 shutdown_connections 2024-03-19T01:56:30.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.842+0000 7f1c368f7640 1 --2- 172.21.15.55:0/3684064534 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1c3006c6f0 0x7f1c300cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:30.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.842+0000 7f1c368f7640 1 --2- 172.21.15.55:0/3684064534 >> v2:172.21.15.84:3300/0 conn(0x7f1c3015a3b0 0x7f1c3015a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:30.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.842+0000 7f1c368f7640 1 --2- 172.21.15.55:0/3684064534 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c300cbf30 0x7f1c300cead0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:30.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.842+0000 7f1c368f7640 1 -- 172.21.15.55:0/3684064534 >> 172.21.15.55:0/3684064534 conn(0x7f1c300b16a0 msgr2=0x7f1c300b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:30.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.842+0000 7f1c368f7640 1 -- 172.21.15.55:0/3684064534 shutdown_connections 2024-03-19T01:56:30.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.842+0000 7f1c368f7640 1 -- 172.21.15.55:0/3684064534 wait complete. 2024-03-19T01:56:30.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.842+0000 7f1c368f7640 1 Processor -- start 2024-03-19T01:56:30.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.842+0000 7f1c368f7640 1 -- start start 2024-03-19T01:56:30.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.846+0000 7f1c368f7640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1c3006c6f0 0x7f1c301f3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:30.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.846+0000 7f1c368f7640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1c300cbf30 0x7f1c301f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:30.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.846+0000 7f1c368f7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c3015a3b0 0x7f1c300b2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:30.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.846+0000 7f1c368f7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1c300b23b0 con 0x7f1c3015a3b0 2024-03-19T01:56:30.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.846+0000 7f1c368f7640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3006ce50 con 0x7f1c3006c6f0 2024-03-19T01:56:30.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.846+0000 7f1c368f7640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1c300cc310 con 0x7f1c300cbf30 2024-03-19T01:56:30.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.846+0000 7f1c34e94640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c3015a3b0 0x7f1c300b2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:30.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.846+0000 7f1c34e94640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c3015a3b0 0x7f1c300b2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:36768/0 (socket says 172.21.15.55:36768) 2024-03-19T01:56:30.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.846+0000 7f1c34e94640 1 -- 172.21.15.55:0/4152264812 learned_addr learned my addr 172.21.15.55:0/4152264812 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:30.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.846+0000 7f1c2ffff640 1 --2- 172.21.15.55:0/4152264812 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1c3006c6f0 0x7f1c301f3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:30.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.846+0000 7f1c2f7fe640 1 --2- 172.21.15.55:0/4152264812 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1c300cbf30 0x7f1c301f3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:30.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.850+0000 7f1c34e94640 1 -- 172.21.15.55:0/4152264812 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1c300cbf30 msgr2=0x7f1c301f3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:30.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.850+0000 7f1c34e94640 1 --2- 172.21.15.55:0/4152264812 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1c300cbf30 0x7f1c301f3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:30.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.850+0000 7f1c34e94640 1 -- 172.21.15.55:0/4152264812 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1c3006c6f0 msgr2=0x7f1c301f3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:30.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.850+0000 7f1c34e94640 1 --2- 172.21.15.55:0/4152264812 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1c3006c6f0 0x7f1c301f3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:30.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.850+0000 7f1c34e94640 1 -- 172.21.15.55:0/4152264812 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c300b4d50 con 0x7f1c3015a3b0 2024-03-19T01:56:30.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.850+0000 7f1c34e94640 1 --2- 172.21.15.55:0/4152264812 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c3015a3b0 0x7f1c300b2a30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1c28009cb0 tx=0x7f1c2800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:30.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.854+0000 7f1c2cff9640 1 -- 172.21.15.55:0/4152264812 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c28018020 con 0x7f1c3015a3b0 2024-03-19T01:56:30.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.854+0000 7f1c2cff9640 1 -- 172.21.15.55:0/4152264812 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c28014480 con 0x7f1c3015a3b0 2024-03-19T01:56:30.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.854+0000 7f1c368f7640 1 -- 172.21.15.55:0/4152264812 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c300b5030 con 0x7f1c3015a3b0 2024-03-19T01:56:30.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.854+0000 7f1c2cff9640 1 -- 172.21.15.55:0/4152264812 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c28014870 con 0x7f1c3015a3b0 2024-03-19T01:56:30.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.854+0000 7f1c368f7640 1 -- 172.21.15.55:0/4152264812 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c300b55f0 con 0x7f1c3015a3b0 2024-03-19T01:56:30.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:30.858+0000 7f1c368f7640 1 -- 172.21.15.55:0/4152264812 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c30068330 con 0x7f1c3015a3b0 2024-03-19T01:56:31.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:31.130+0000 7f1c2cff9640 1 -- 172.21.15.55:0/4152264812 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1c2801b020 con 0x7f1c3015a3b0 2024-03-19T01:56:31.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:31.130+0000 7f1c2cff9640 1 --2- 172.21.15.55:0/4152264812 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1c1c067690 0x7f1c1c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:31.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:31.130+0000 7f1c2ffff640 1 --2- 172.21.15.55:0/4152264812 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1c1c067690 0x7f1c1c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:31.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:31.130+0000 7f1c2ffff640 1 --2- 172.21.15.55:0/4152264812 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1c1c067690 0x7f1c1c069b50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1c200af000 tx=0x7f1c200a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:31.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:31.146+0000 7f1c2cff9640 1 -- 172.21.15.55:0/4152264812 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f1c280550a0 con 0x7f1c3015a3b0 2024-03-19T01:56:31.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:31.146+0000 7f1c2cff9640 1 -- 172.21.15.55:0/4152264812 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1c28022030 con 0x7f1c3015a3b0 2024-03-19T01:56:31.354 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:31.354+0000 7f1c368f7640 1 -- 172.21.15.55:0/4152264812 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c300b2190 con 0x7f1c3015a3b0 2024-03-19T01:56:31.377 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:31.378+0000 7f1c2cff9640 1 -- 172.21.15.55:0/4152264812 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f1c28010040 con 0x7f1c3015a3b0 2024-03-19T01:56:31.377 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:31.377 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:31.378 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:31.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:31.382+0000 7f1c368f7640 1 -- 172.21.15.55:0/4152264812 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1c1c067690 msgr2=0x7f1c1c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:31.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:31.382+0000 7f1c368f7640 1 --2- 172.21.15.55:0/4152264812 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1c1c067690 0x7f1c1c069b50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1c200af000 tx=0x7f1c200a7000 comp rx=0 tx=0).stop 2024-03-19T01:56:31.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:31.382+0000 7f1c368f7640 1 -- 172.21.15.55:0/4152264812 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c3015a3b0 msgr2=0x7f1c300b2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:31.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:31.382+0000 7f1c368f7640 1 --2- 172.21.15.55:0/4152264812 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c3015a3b0 0x7f1c300b2a30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1c28009cb0 tx=0x7f1c2800c490 comp rx=0 tx=0).stop 2024-03-19T01:56:31.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:31.382+0000 7f1c368f7640 1 -- 172.21.15.55:0/4152264812 shutdown_connections 2024-03-19T01:56:31.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:31.382+0000 7f1c368f7640 1 --2- 172.21.15.55:0/4152264812 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1c1c067690 0x7f1c1c069b50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:31.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:31.382+0000 7f1c368f7640 1 --2- 172.21.15.55:0/4152264812 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c3015a3b0 0x7f1c300b2a30 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:31.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:31.382+0000 7f1c368f7640 1 --2- 172.21.15.55:0/4152264812 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1c300cbf30 0x7f1c301f3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:31.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:31.382+0000 7f1c368f7640 1 --2- 172.21.15.55:0/4152264812 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1c3006c6f0 0x7f1c301f3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:31.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:31.382+0000 7f1c368f7640 1 -- 172.21.15.55:0/4152264812 >> 172.21.15.55:0/4152264812 conn(0x7f1c300b16a0 msgr2=0x7f1c3006bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:31.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:31.382+0000 7f1c368f7640 1 -- 172.21.15.55:0/4152264812 shutdown_connections 2024-03-19T01:56:31.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:31.382+0000 7f1c368f7640 1 -- 172.21.15.55:0/4152264812 wait complete. 2024-03-19T01:56:32.000 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:32.000 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:33.002 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:33.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.134+0000 7f6ddf5dc640 1 Processor -- start 2024-03-19T01:56:33.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.134+0000 7f6ddf5dc640 1 -- start start 2024-03-19T01:56:33.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.134+0000 7f6ddf5dc640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f6dd8154290 0x7f6dd815e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:33.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.134+0000 7f6ddf5dc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6dd80ca310 0x7f6dd8154670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:33.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.134+0000 7f6ddf5dc640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f6dd815ad40 0x7f6dd8072900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:33.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.134+0000 7f6ddf5dc640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f6dd815b2c0 con 0x7f6dd815ad40 2024-03-19T01:56:33.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.134+0000 7f6ddf5dc640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f6dd806c1d0 con 0x7f6dd8154290 2024-03-19T01:56:33.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.134+0000 7f6ddf5dc640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f6dd815e030 con 0x7f6dd80ca310 2024-03-19T01:56:33.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.134+0000 7f6dddb79640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f6dd815ad40 0x7f6dd8072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:33.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.134+0000 7f6ddcb77640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6dd80ca310 0x7f6dd8154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:33.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.134+0000 7f6ddd378640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f6dd8154290 0x7f6dd815e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:33.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.134+0000 7f6dddb79640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f6dd815ad40 0x7f6dd8072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:36778/0 (socket says 172.21.15.55:36778) 2024-03-19T01:56:33.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.134+0000 7f6dddb79640 1 -- 172.21.15.55:0/2823878761 learned_addr learned my addr 172.21.15.55:0/2823878761 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:33.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.138+0000 7f6dddb79640 1 -- 172.21.15.55:0/2823878761 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6dd80ca310 msgr2=0x7f6dd8154670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:33.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.138+0000 7f6dddb79640 1 --2- 172.21.15.55:0/2823878761 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6dd80ca310 0x7f6dd8154670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:33.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.138+0000 7f6dddb79640 1 -- 172.21.15.55:0/2823878761 >> v2:172.21.15.84:3300/0 conn(0x7f6dd8154290 msgr2=0x7f6dd815e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:33.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.138+0000 7f6dddb79640 1 --2- 172.21.15.55:0/2823878761 >> v2:172.21.15.84:3300/0 conn(0x7f6dd8154290 0x7f6dd815e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:33.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.138+0000 7f6dddb79640 1 -- 172.21.15.55:0/2823878761 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd806bc90 con 0x7f6dd815ad40 2024-03-19T01:56:33.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.142+0000 7f6dddb79640 1 --2- 172.21.15.55:0/2823878761 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6dd815ad40 0x7f6dd8072900 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8071360 tx=0x7f6dd001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96bfb4d12cf3df3c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:33.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.146+0000 7f6dcffff640 1 -- 172.21.15.55:0/2823878761 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd00a5890 con 0x7f6dd815ad40 2024-03-19T01:56:33.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.146+0000 7f6dcffff640 1 -- 172.21.15.55:0/2823878761 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dd00a5a30 con 0x7f6dd815ad40 2024-03-19T01:56:33.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.146+0000 7f6dcffff640 1 -- 172.21.15.55:0/2823878761 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd00a5e00 con 0x7f6dd815ad40 2024-03-19T01:56:33.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.150+0000 7f6ddf5dc640 1 -- 172.21.15.55:0/2823878761 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6dd815ad40 msgr2=0x7f6dd8072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:33.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.150+0000 7f6ddf5dc640 1 --2- 172.21.15.55:0/2823878761 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6dd815ad40 0x7f6dd8072900 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8071360 tx=0x7f6dd001c9f0 comp rx=0 tx=0).stop 2024-03-19T01:56:33.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.150+0000 7f6ddf5dc640 1 -- 172.21.15.55:0/2823878761 shutdown_connections 2024-03-19T01:56:33.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.150+0000 7f6ddf5dc640 1 --2- 172.21.15.55:0/2823878761 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6dd80ca310 0x7f6dd8154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:33.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.150+0000 7f6ddf5dc640 1 --2- 172.21.15.55:0/2823878761 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6dd815ad40 0x7f6dd8072900 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:33.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.150+0000 7f6ddf5dc640 1 --2- 172.21.15.55:0/2823878761 >> v2:172.21.15.84:3300/0 conn(0x7f6dd8154290 0x7f6dd815e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:33.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.150+0000 7f6ddf5dc640 1 -- 172.21.15.55:0/2823878761 >> 172.21.15.55:0/2823878761 conn(0x7f6dd806c580 msgr2=0x7f6dd806c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:33.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.150+0000 7f6ddf5dc640 1 -- 172.21.15.55:0/2823878761 shutdown_connections 2024-03-19T01:56:33.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.150+0000 7f6ddf5dc640 1 -- 172.21.15.55:0/2823878761 wait complete. 2024-03-19T01:56:33.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.150+0000 7f6ddf5dc640 1 Processor -- start 2024-03-19T01:56:33.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.150+0000 7f6ddf5dc640 1 -- start start 2024-03-19T01:56:33.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.150+0000 7f6ddf5dc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6dd80ca310 0x7f6dd81f38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:33.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.150+0000 7f6ddf5dc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6dd8154290 0x7f6dd80c2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:33.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.150+0000 7f6ddf5dc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6dd815ad40 0x7f6dd81587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:33.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.150+0000 7f6ddf5dc640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd80ccb50 con 0x7f6dd8154290 2024-03-19T01:56:33.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.150+0000 7f6ddf5dc640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd806bf30 con 0x7f6dd80ca310 2024-03-19T01:56:33.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.150+0000 7f6ddf5dc640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f6dd80cccd0 con 0x7f6dd815ad40 2024-03-19T01:56:33.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.154+0000 7f6ddcb77640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6dd8154290 0x7f6dd80c2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:33.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.154+0000 7f6dddb79640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6dd815ad40 0x7f6dd81587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:33.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.154+0000 7f6ddd378640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6dd80ca310 0x7f6dd81f38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:33.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.154+0000 7f6ddcb77640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6dd8154290 0x7f6dd80c2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:36782/0 (socket says 172.21.15.55:36782) 2024-03-19T01:56:33.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.154+0000 7f6ddcb77640 1 -- 172.21.15.55:0/1050254825 learned_addr learned my addr 172.21.15.55:0/1050254825 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:33.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.158+0000 7f6ddcb77640 1 -- 172.21.15.55:0/1050254825 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6dd815ad40 msgr2=0x7f6dd81587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:33.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.158+0000 7f6ddcb77640 1 --2- 172.21.15.55:0/1050254825 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6dd815ad40 0x7f6dd81587d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:33.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.158+0000 7f6ddcb77640 1 -- 172.21.15.55:0/1050254825 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6dd80ca310 msgr2=0x7f6dd81f38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:33.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.158+0000 7f6ddcb77640 1 --2- 172.21.15.55:0/1050254825 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6dd80ca310 0x7f6dd81f38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:33.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.158+0000 7f6ddcb77640 1 -- 172.21.15.55:0/1050254825 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd81f3e30 con 0x7f6dd8154290 2024-03-19T01:56:33.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.158+0000 7f6ddcb77640 1 --2- 172.21.15.55:0/1050254825 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6dd8154290 0x7f6dd80c2ba0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800e8a0 tx=0x7f6dc800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:33.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.162+0000 7f6dcdffb640 1 -- 172.21.15.55:0/1050254825 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc8017070 con 0x7f6dd8154290 2024-03-19T01:56:33.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.162+0000 7f6dcdffb640 1 -- 172.21.15.55:0/1050254825 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dc800ee90 con 0x7f6dd8154290 2024-03-19T01:56:33.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.162+0000 7f6dcdffb640 1 -- 172.21.15.55:0/1050254825 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc800cb80 con 0x7f6dd8154290 2024-03-19T01:56:33.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.166+0000 7f6ddf5dc640 1 -- 172.21.15.55:0/1050254825 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dd815c450 con 0x7f6dd8154290 2024-03-19T01:56:33.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.166+0000 7f6ddf5dc640 1 -- 172.21.15.55:0/1050254825 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dd815b690 con 0x7f6dd8154290 2024-03-19T01:56:33.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.166+0000 7f6ddf5dc640 1 -- 172.21.15.55:0/1050254825 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dd8068a40 con 0x7f6dd8154290 2024-03-19T01:56:33.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.186+0000 7f6dcdffb640 1 -- 172.21.15.55:0/1050254825 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f6dc801a070 con 0x7f6dd8154290 2024-03-19T01:56:33.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.186+0000 7f6dcdffb640 1 --2- 172.21.15.55:0/1050254825 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6dbc067660 0x7f6dbc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:33.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.186+0000 7f6ddd378640 1 --2- 172.21.15.55:0/1050254825 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6dbc067660 0x7f6dbc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:33.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.186+0000 7f6ddd378640 1 --2- 172.21.15.55:0/1050254825 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6dbc067660 0x7f6dbc069b20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4009b60 tx=0x7f6dd4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:33.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.186+0000 7f6dcdffb640 1 -- 172.21.15.55:0/1050254825 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f6dc80544f0 con 0x7f6dd8154290 2024-03-19T01:56:33.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.478+0000 7f6dcdffb640 1 -- 172.21.15.55:0/1050254825 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f6dc8056450 con 0x7f6dd8154290 2024-03-19T01:56:33.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.730+0000 7f6ddf5dc640 1 -- 172.21.15.55:0/1050254825 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6dd80b18f0 con 0x7f6dd8154290 2024-03-19T01:56:33.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.754+0000 7f6dcdffb640 1 -- 172.21.15.55:0/1050254825 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f6dc805a4f0 con 0x7f6dd8154290 2024-03-19T01:56:33.751 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:33.751 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:33.751 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:33.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.758+0000 7f6ddf5dc640 1 -- 172.21.15.55:0/1050254825 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6dbc067660 msgr2=0x7f6dbc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:33.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.758+0000 7f6ddf5dc640 1 --2- 172.21.15.55:0/1050254825 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6dbc067660 0x7f6dbc069b20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4009b60 tx=0x7f6dd4009190 comp rx=0 tx=0).stop 2024-03-19T01:56:33.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.758+0000 7f6ddf5dc640 1 -- 172.21.15.55:0/1050254825 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6dd8154290 msgr2=0x7f6dd80c2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:33.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.758+0000 7f6ddf5dc640 1 --2- 172.21.15.55:0/1050254825 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6dd8154290 0x7f6dd80c2ba0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800e8a0 tx=0x7f6dc800ed60 comp rx=0 tx=0).stop 2024-03-19T01:56:33.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.758+0000 7f6ddf5dc640 1 -- 172.21.15.55:0/1050254825 shutdown_connections 2024-03-19T01:56:33.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.758+0000 7f6ddf5dc640 1 --2- 172.21.15.55:0/1050254825 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6dbc067660 0x7f6dbc069b20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:33.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.758+0000 7f6ddf5dc640 1 --2- 172.21.15.55:0/1050254825 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6dd815ad40 0x7f6dd81587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:33.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.758+0000 7f6ddf5dc640 1 --2- 172.21.15.55:0/1050254825 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6dd8154290 0x7f6dd80c2ba0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:33.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.758+0000 7f6ddf5dc640 1 --2- 172.21.15.55:0/1050254825 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6dd80ca310 0x7f6dd81f38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:33.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.758+0000 7f6ddf5dc640 1 -- 172.21.15.55:0/1050254825 >> 172.21.15.55:0/1050254825 conn(0x7f6dd806c580 msgr2=0x7f6dd80c21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:33.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.758+0000 7f6ddf5dc640 1 -- 172.21.15.55:0/1050254825 shutdown_connections 2024-03-19T01:56:33.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:33.758+0000 7f6ddf5dc640 1 -- 172.21.15.55:0/1050254825 wait complete. 2024-03-19T01:56:34.375 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:34.376 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:35.377 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:35.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.498+0000 7f071211f640 1 Processor -- start 2024-03-19T01:56:35.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.498+0000 7f071211f640 1 -- start start 2024-03-19T01:56:35.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.498+0000 7f071211f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f070c070a50 0x7f070c0c6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:35.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.498+0000 7f071211f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f070c13d010 con 0x7f070c13dbf0 2024-03-19T01:56:35.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.498+0000 7f071211f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f070c0ccbf0 con 0x7f070c070670 2024-03-19T01:56:35.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.498+0000 7f071211f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f070c0cce90 con 0x7f070c070a50 2024-03-19T01:56:35.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.498+0000 7f070affd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f070c070a50 0x7f070c0c6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:35.499 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.502+0000 7f070affd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f070c070a50 0x7f070c0c6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:36788/0 (socket says 172.21.15.55:36788) 2024-03-19T01:56:35.499 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.502+0000 7f070b7fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f070c070670 0x7f070c13d4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:57678/0 (socket says 172.21.15.55:57678) 2024-03-19T01:56:35.499 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.502+0000 7f070affd640 1 -- 172.21.15.55:0/2989174911 learned_addr learned my addr 172.21.15.55:0/2989174911 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:35.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.506+0000 7f070affd640 1 -- 172.21.15.55:0/2989174911 >> v1:172.21.15.84:6789/0 conn(0x7f070c070670 legacy=0x7f070c13d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:35.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.506+0000 7f070affd640 1 -- 172.21.15.55:0/2989174911 >> v1:172.21.15.55:6789/0 conn(0x7f070c13dbf0 legacy=0x7f070c06b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:35.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.506+0000 7f070affd640 1 -- 172.21.15.55:0/2989174911 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f070c0cd170 con 0x7f070c070a50 2024-03-19T01:56:35.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.506+0000 7f070affd640 1 --2- 172.21.15.55:0/2989174911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f070c070a50 0x7f070c0c6750 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0700009870 tx=0x7f070001ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=480ec80be4a547fb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:35.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.510+0000 7f070a7fc640 1 -- 172.21.15.55:0/2989174911 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07000b5070 con 0x7f070c070a50 2024-03-19T01:56:35.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.514+0000 7f070a7fc640 1 -- 172.21.15.55:0/2989174911 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07000a8a50 con 0x7f070c070a50 2024-03-19T01:56:35.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.514+0000 7f070a7fc640 1 -- 172.21.15.55:0/2989174911 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07000a8e20 con 0x7f070c070a50 2024-03-19T01:56:35.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.514+0000 7f071211f640 1 -- 172.21.15.55:0/2989174911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f070c070a50 msgr2=0x7f070c0c6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:35.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.514+0000 7f071211f640 1 --2- 172.21.15.55:0/2989174911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f070c070a50 0x7f070c0c6750 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0700009870 tx=0x7f070001ed50 comp rx=0 tx=0).stop 2024-03-19T01:56:35.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.514+0000 7f071211f640 1 -- 172.21.15.55:0/2989174911 shutdown_connections 2024-03-19T01:56:35.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.514+0000 7f071211f640 1 --2- 172.21.15.55:0/2989174911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f070c070a50 0x7f070c0c6750 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:35.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.514+0000 7f071211f640 1 -- 172.21.15.55:0/2989174911 >> 172.21.15.55:0/2989174911 conn(0x7f070c06c3b0 msgr2=0x7f070c06c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:35.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.514+0000 7f071211f640 1 -- 172.21.15.55:0/2989174911 shutdown_connections 2024-03-19T01:56:35.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.514+0000 7f071211f640 1 -- 172.21.15.55:0/2989174911 wait complete. 2024-03-19T01:56:35.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.514+0000 7f071211f640 1 Processor -- start 2024-03-19T01:56:35.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.514+0000 7f071211f640 1 -- start start 2024-03-19T01:56:35.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.514+0000 7f071211f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f070c070670 0x7f070c1374d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:35.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.514+0000 7f071211f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f070c070a50 0x7f070c13f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:35.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.514+0000 7f071211f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f070c13dbf0 0x7f070c13fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:35.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.514+0000 7f071211f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f070c0c6c90 con 0x7f070c070670 2024-03-19T01:56:35.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.514+0000 7f071211f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f070c0cd730 con 0x7f070c070a50 2024-03-19T01:56:35.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.514+0000 7f071211f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f070c0c6e10 con 0x7f070c13dbf0 2024-03-19T01:56:35.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.514+0000 7f070b7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f070c070670 0x7f070c1374d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:35.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.518+0000 7f070b7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f070c070670 0x7f070c1374d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:36796/0 (socket says 172.21.15.55:36796) 2024-03-19T01:56:35.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.518+0000 7f070b7fe640 1 -- 172.21.15.55:0/3652963066 learned_addr learned my addr 172.21.15.55:0/3652963066 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:35.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.518+0000 7f070affd640 1 --2- 172.21.15.55:0/3652963066 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f070c070a50 0x7f070c13f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:35.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.518+0000 7f070bfff640 1 --2- 172.21.15.55:0/3652963066 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f070c13dbf0 0x7f070c13fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:35.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.518+0000 7f070b7fe640 1 -- 172.21.15.55:0/3652963066 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f070c13dbf0 msgr2=0x7f070c13fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:35.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.518+0000 7f070b7fe640 1 --2- 172.21.15.55:0/3652963066 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f070c13dbf0 0x7f070c13fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:35.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.518+0000 7f070b7fe640 1 -- 172.21.15.55:0/3652963066 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f070c070a50 msgr2=0x7f070c13f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:35.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.518+0000 7f070b7fe640 1 --2- 172.21.15.55:0/3652963066 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f070c070a50 0x7f070c13f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:35.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.518+0000 7f070b7fe640 1 -- 172.21.15.55:0/3652963066 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f070c13ef90 con 0x7f070c070670 2024-03-19T01:56:35.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.522+0000 7f070b7fe640 1 --2- 172.21.15.55:0/3652963066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f070c070670 0x7f070c1374d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f06fc00c4c0 tx=0x7f06fc00c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:35.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.526+0000 7f06f3fff640 1 -- 172.21.15.55:0/3652963066 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06fc018070 con 0x7f070c070670 2024-03-19T01:56:35.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.526+0000 7f06f3fff640 1 -- 172.21.15.55:0/3652963066 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06fc007480 con 0x7f070c070670 2024-03-19T01:56:35.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.526+0000 7f06f3fff640 1 -- 172.21.15.55:0/3652963066 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06fc0047f0 con 0x7f070c070670 2024-03-19T01:56:35.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.526+0000 7f071211f640 1 -- 172.21.15.55:0/3652963066 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f070c13e410 con 0x7f070c070670 2024-03-19T01:56:35.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.526+0000 7f071211f640 1 -- 172.21.15.55:0/3652963066 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f070c13e9d0 con 0x7f070c070670 2024-03-19T01:56:35.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.530+0000 7f071211f640 1 -- 172.21.15.55:0/3652963066 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f070c068a40 con 0x7f070c070670 2024-03-19T01:56:35.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.794+0000 7f06f3fff640 1 -- 172.21.15.55:0/3652963066 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f06fc004990 con 0x7f070c070670 2024-03-19T01:56:35.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.794+0000 7f06f3fff640 1 --2- 172.21.15.55:0/3652963066 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f06f8067660 0x7f06f8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:35.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.794+0000 7f070affd640 1 --2- 172.21.15.55:0/3652963066 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f06f8067660 0x7f06f8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:35.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.794+0000 7f070affd640 1 --2- 172.21.15.55:0/3652963066 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f06f8067660 0x7f06f8069b20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f07000b2000 tx=0x7f0700009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:35.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.810+0000 7f06f3fff640 1 -- 172.21.15.55:0/3652963066 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f06fc053620 con 0x7f070c070670 2024-03-19T01:56:35.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:35.810+0000 7f06f3fff640 1 -- 172.21.15.55:0/3652963066 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f06fc055580 con 0x7f070c070670 2024-03-19T01:56:36.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:36.022+0000 7f071211f640 1 -- 172.21.15.55:0/3652963066 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f070c06cab0 con 0x7f070c070670 2024-03-19T01:56:36.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:36.042+0000 7f06f3fff640 1 -- 172.21.15.55:0/3652963066 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f06fc059620 con 0x7f070c070670 2024-03-19T01:56:36.042 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:36.042 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:36.042 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:36.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:36.046+0000 7f071211f640 1 -- 172.21.15.55:0/3652963066 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f06f8067660 msgr2=0x7f06f8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:36.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:36.046+0000 7f071211f640 1 --2- 172.21.15.55:0/3652963066 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f06f8067660 0x7f06f8069b20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f07000b2000 tx=0x7f0700009040 comp rx=0 tx=0).stop 2024-03-19T01:56:36.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:36.046+0000 7f071211f640 1 -- 172.21.15.55:0/3652963066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f070c070670 msgr2=0x7f070c1374d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:36.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:36.046+0000 7f071211f640 1 --2- 172.21.15.55:0/3652963066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f070c070670 0x7f070c1374d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f06fc00c4c0 tx=0x7f06fc00c980 comp rx=0 tx=0).stop 2024-03-19T01:56:36.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:36.046+0000 7f071211f640 1 -- 172.21.15.55:0/3652963066 shutdown_connections 2024-03-19T01:56:36.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:36.046+0000 7f071211f640 1 --2- 172.21.15.55:0/3652963066 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f06f8067660 0x7f06f8069b20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:36.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:36.046+0000 7f071211f640 1 --2- 172.21.15.55:0/3652963066 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f070c13dbf0 0x7f070c13fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:36.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:36.046+0000 7f071211f640 1 --2- 172.21.15.55:0/3652963066 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f070c070a50 0x7f070c13f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:36.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:36.046+0000 7f071211f640 1 --2- 172.21.15.55:0/3652963066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f070c070670 0x7f070c1374d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:36.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:36.046+0000 7f071211f640 1 -- 172.21.15.55:0/3652963066 >> 172.21.15.55:0/3652963066 conn(0x7f070c06c3b0 msgr2=0x7f070c155b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:36.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:36.046+0000 7f071211f640 1 -- 172.21.15.55:0/3652963066 shutdown_connections 2024-03-19T01:56:36.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:36.046+0000 7f071211f640 1 -- 172.21.15.55:0/3652963066 wait complete. 2024-03-19T01:56:36.664 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:36.664 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:37.666 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:37.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.798+0000 7f464402f640 1 Processor -- start 2024-03-19T01:56:37.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.798+0000 7f464402f640 1 -- start start 2024-03-19T01:56:37.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.798+0000 7f464402f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f463c13dec0 0x7f463c0c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:37.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.798+0000 7f464402f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f463c067440 0x7f463c06be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:37.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.798+0000 7f464402f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f463c158220 con 0x7f463c067000 2024-03-19T01:56:37.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.798+0000 7f464402f640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f463c158500 con 0x7f463c067440 2024-03-19T01:56:37.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.798+0000 7f464402f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f463c1587e0 con 0x7f463c13dec0 2024-03-19T01:56:37.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.798+0000 7f4641dcb640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f463c13dec0 0x7f463c0c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:37.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.798+0000 7f46425cc640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f463c067440 0x7f463c06be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:37.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.798+0000 7f46415ca640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f463c067000 0x7f463c0c5980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:58518/0 (socket says 172.21.15.55:58518) 2024-03-19T01:56:37.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.798+0000 7f46415ca640 1 -- 172.21.15.55:0/4014610566 learned_addr learned my addr 172.21.15.55:0/4014610566 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:56:37.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.802+0000 7f4640dc9640 1 -- 172.21.15.55:0/4014610566 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3536564444 0 0) 0x7f463c158220 con 0x7f463c067000 2024-03-19T01:56:37.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.802+0000 7f4640dc9640 1 -- 172.21.15.55:0/4014610566 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4628003530 con 0x7f463c067000 2024-03-19T01:56:37.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.802+0000 7f4641dcb640 1 -- 172.21.15.55:0/4014610566 >> v2:172.21.15.84:3300/0 conn(0x7f463c067440 msgr2=0x7f463c06be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:37.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.802+0000 7f4641dcb640 1 --2- 172.21.15.55:0/4014610566 >> v2:172.21.15.84:3300/0 conn(0x7f463c067440 0x7f463c06be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:37.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.802+0000 7f4641dcb640 1 -- 172.21.15.55:0/4014610566 >> v1:172.21.15.55:6789/0 conn(0x7f463c067000 legacy=0x7f463c0c5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:37.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.802+0000 7f4641dcb640 1 -- 172.21.15.55:0/4014610566 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f463c158ac0 con 0x7f463c13dec0 2024-03-19T01:56:37.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.810+0000 7f4641dcb640 1 --2- 172.21.15.55:0/4014610566 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f463c13dec0 0x7f463c0c76f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f46380097c0 tx=0x7f463801ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=165337a389871850 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:37.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.814+0000 7f4640dc9640 1 -- 172.21.15.55:0/4014610566 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46380b5070 con 0x7f463c13dec0 2024-03-19T01:56:37.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.814+0000 7f4640dc9640 1 -- 172.21.15.55:0/4014610566 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46380a8990 con 0x7f463c13dec0 2024-03-19T01:56:37.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.814+0000 7f4640dc9640 1 -- 172.21.15.55:0/4014610566 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46380a8d60 con 0x7f463c13dec0 2024-03-19T01:56:37.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.814+0000 7f464402f640 1 -- 172.21.15.55:0/4014610566 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f463c13dec0 msgr2=0x7f463c0c76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:37.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.814+0000 7f464402f640 1 --2- 172.21.15.55:0/4014610566 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f463c13dec0 0x7f463c0c76f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f46380097c0 tx=0x7f463801ec90 comp rx=0 tx=0).stop 2024-03-19T01:56:37.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.814+0000 7f464402f640 1 -- 172.21.15.55:0/4014610566 shutdown_connections 2024-03-19T01:56:37.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.814+0000 7f464402f640 1 --2- 172.21.15.55:0/4014610566 >> v2:172.21.15.84:3300/0 conn(0x7f463c067440 0x7f463c06be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:37.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.814+0000 7f464402f640 1 --2- 172.21.15.55:0/4014610566 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f463c13dec0 0x7f463c0c76f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:37.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.814+0000 7f464402f640 1 -- 172.21.15.55:0/4014610566 >> 172.21.15.55:0/4014610566 conn(0x7f463c0cc270 msgr2=0x7f463c0cc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:37.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.814+0000 7f464402f640 1 -- 172.21.15.55:0/4014610566 shutdown_connections 2024-03-19T01:56:37.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.814+0000 7f464402f640 1 -- 172.21.15.55:0/4014610566 wait complete. 2024-03-19T01:56:37.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.818+0000 7f464402f640 1 Processor -- start 2024-03-19T01:56:37.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.818+0000 7f464402f640 1 -- start start 2024-03-19T01:56:37.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.818+0000 7f464402f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f463c067000 0x7f463c1368e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:37.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.818+0000 7f464402f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f463c067440 0x7f463c1ed960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:37.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.818+0000 7f464402f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f463c13dec0 0x7f463c13c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:37.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.818+0000 7f464402f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f463c0c6090 con 0x7f463c13dec0 2024-03-19T01:56:37.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.818+0000 7f464402f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f463c13e2a0 con 0x7f463c067000 2024-03-19T01:56:37.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.818+0000 7f464402f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f463c0c6210 con 0x7f463c067440 2024-03-19T01:56:37.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.818+0000 7f46415ca640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f463c067440 0x7f463c1ed960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:37.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.818+0000 7f4641dcb640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f463c067000 0x7f463c1368e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:37.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.818+0000 7f46425cc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f463c13dec0 0x7f463c13c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:37.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.818+0000 7f46425cc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f463c13dec0 0x7f463c13c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:36812/0 (socket says 172.21.15.55:36812) 2024-03-19T01:56:37.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.818+0000 7f46425cc640 1 -- 172.21.15.55:0/401487270 learned_addr learned my addr 172.21.15.55:0/401487270 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:37.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.822+0000 7f46425cc640 1 -- 172.21.15.55:0/401487270 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f463c067440 msgr2=0x7f463c1ed960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:37.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.822+0000 7f46425cc640 1 --2- 172.21.15.55:0/401487270 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f463c067440 0x7f463c1ed960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:37.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.822+0000 7f46425cc640 1 -- 172.21.15.55:0/401487270 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f463c067000 msgr2=0x7f463c1368e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:37.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.822+0000 7f46425cc640 1 --2- 172.21.15.55:0/401487270 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f463c067000 0x7f463c1368e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:37.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.822+0000 7f46425cc640 1 -- 172.21.15.55:0/401487270 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f463c159450 con 0x7f463c13dec0 2024-03-19T01:56:37.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.826+0000 7f46425cc640 1 --2- 172.21.15.55:0/401487270 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f463c13dec0 0x7f463c13c340 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f463400efc0 tx=0x7f463400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:37.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.830+0000 7f46327fc640 1 -- 172.21.15.55:0/401487270 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4634018020 con 0x7f463c13dec0 2024-03-19T01:56:37.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.830+0000 7f46327fc640 1 -- 172.21.15.55:0/401487270 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4634014480 con 0x7f463c13dec0 2024-03-19T01:56:37.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.830+0000 7f46327fc640 1 -- 172.21.15.55:0/401487270 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4634014870 con 0x7f463c13dec0 2024-03-19T01:56:37.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.830+0000 7f464402f640 1 -- 172.21.15.55:0/401487270 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f463c136ed0 con 0x7f463c13dec0 2024-03-19T01:56:37.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.830+0000 7f464402f640 1 -- 172.21.15.55:0/401487270 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f463c1f1050 con 0x7f463c13dec0 2024-03-19T01:56:37.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.830+0000 7f464402f640 1 -- 172.21.15.55:0/401487270 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f463c068a40 con 0x7f463c13dec0 2024-03-19T01:56:37.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.850+0000 7f46327fc640 1 -- 172.21.15.55:0/401487270 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f463401b020 con 0x7f463c13dec0 2024-03-19T01:56:37.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.850+0000 7f46327fc640 1 --2- 172.21.15.55:0/401487270 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4620067660 0x7f4620069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:37.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.850+0000 7f4641dcb640 1 --2- 172.21.15.55:0/401487270 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4620067660 0x7f4620069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:37.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:37.854+0000 7f4641dcb640 1 --2- 172.21.15.55:0/401487270 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4620067660 0x7f4620069b20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f46380b2000 tx=0x7f4638008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:38.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:38.150+0000 7f46327fc640 1 -- 172.21.15.55:0/401487270 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f4634055160 con 0x7f463c13dec0 2024-03-19T01:56:38.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:38.150+0000 7f46327fc640 1 -- 172.21.15.55:0/401487270 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4634022030 con 0x7f463c13dec0 2024-03-19T01:56:38.372 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:38.374+0000 7f464402f640 1 -- 172.21.15.55:0/401487270 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f463c13d180 con 0x7f463c13dec0 2024-03-19T01:56:38.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:38.390+0000 7f46327fc640 1 -- 172.21.15.55:0/401487270 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f4634010040 con 0x7f463c13dec0 2024-03-19T01:56:38.390 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:38.391 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:38.391 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:38.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:38.394+0000 7f464402f640 1 -- 172.21.15.55:0/401487270 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4620067660 msgr2=0x7f4620069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:38.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:38.394+0000 7f464402f640 1 --2- 172.21.15.55:0/401487270 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4620067660 0x7f4620069b20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f46380b2000 tx=0x7f4638008040 comp rx=0 tx=0).stop 2024-03-19T01:56:38.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:38.394+0000 7f464402f640 1 -- 172.21.15.55:0/401487270 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f463c13dec0 msgr2=0x7f463c13c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:38.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:38.394+0000 7f464402f640 1 --2- 172.21.15.55:0/401487270 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f463c13dec0 0x7f463c13c340 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f463400efc0 tx=0x7f463400c490 comp rx=0 tx=0).stop 2024-03-19T01:56:38.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:38.394+0000 7f464402f640 1 -- 172.21.15.55:0/401487270 shutdown_connections 2024-03-19T01:56:38.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:38.394+0000 7f464402f640 1 --2- 172.21.15.55:0/401487270 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4620067660 0x7f4620069b20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:38.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:38.394+0000 7f464402f640 1 --2- 172.21.15.55:0/401487270 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f463c13dec0 0x7f463c13c340 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:38.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:38.394+0000 7f464402f640 1 --2- 172.21.15.55:0/401487270 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f463c067440 0x7f463c1ed960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:38.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:38.394+0000 7f464402f640 1 --2- 172.21.15.55:0/401487270 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f463c067000 0x7f463c1368e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:38.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:38.394+0000 7f464402f640 1 -- 172.21.15.55:0/401487270 >> 172.21.15.55:0/401487270 conn(0x7f463c0cc270 msgr2=0x7f463c13b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:38.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:38.394+0000 7f464402f640 1 -- 172.21.15.55:0/401487270 shutdown_connections 2024-03-19T01:56:38.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:38.394+0000 7f464402f640 1 -- 172.21.15.55:0/401487270 wait complete. 2024-03-19T01:56:39.013 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:39.013 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:39.319 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:56:39.322 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:56:40.014 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:40.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.130+0000 7fa9d9fde640 1 Processor -- start 2024-03-19T01:56:40.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.130+0000 7fa9d9fde640 1 -- start start 2024-03-19T01:56:40.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.134+0000 7fa9d9fde640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa9d40cbf30 0x7fa9d40cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:40.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.134+0000 7fa9d9fde640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa9d406c6f0 0x7fa9d40cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:40.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.134+0000 7fa9d9fde640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9d406cb90 con 0x7fa9d415a3b0 2024-03-19T01:56:40.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.134+0000 7fa9d9fde640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fa9d40ce640 con 0x7fa9d40cbf30 2024-03-19T01:56:40.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.134+0000 7fa9d9fde640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa9d40cda30 con 0x7fa9d406c6f0 2024-03-19T01:56:40.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.134+0000 7fa9cbfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa9d406c6f0 0x7fa9d40cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:40.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.134+0000 7fa9d37fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa9d40cbf30 0x7fa9d40cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:40.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.134+0000 7fa9cbfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa9d406c6f0 0x7fa9d40cf010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:56620/0 (socket says 172.21.15.55:56620) 2024-03-19T01:56:40.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.134+0000 7fa9cbfff640 1 -- 172.21.15.55:0/3348691808 learned_addr learned my addr 172.21.15.55:0/3348691808 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:40.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.138+0000 7fa9cbfff640 1 -- 172.21.15.55:0/3348691808 >> v2:172.21.15.84:3300/0 conn(0x7fa9d40cbf30 msgr2=0x7fa9d40cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:40.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.138+0000 7fa9cbfff640 1 --2- 172.21.15.55:0/3348691808 >> v2:172.21.15.84:3300/0 conn(0x7fa9d40cbf30 0x7fa9d40cead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:40.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.138+0000 7fa9cbfff640 1 -- 172.21.15.55:0/3348691808 >> v1:172.21.15.84:6789/0 conn(0x7fa9d415a3b0 legacy=0x7fa9d415a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:40.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.138+0000 7fa9cbfff640 1 -- 172.21.15.55:0/3348691808 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d40cdd10 con 0x7fa9d406c6f0 2024-03-19T01:56:40.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.146+0000 7fa9cbfff640 1 --2- 172.21.15.55:0/3348691808 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa9d406c6f0 0x7fa9d40cf010 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40ce7c0 tx=0x7fa9bc01c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0de5e856aa865b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:40.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.150+0000 7fa9d2ffd640 1 -- 172.21.15.55:0/3348691808 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc0b1070 con 0x7fa9d406c6f0 2024-03-19T01:56:40.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.150+0000 7fa9d2ffd640 1 -- 172.21.15.55:0/3348691808 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9bc0ac620 con 0x7fa9d406c6f0 2024-03-19T01:56:40.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.150+0000 7fa9d2ffd640 1 -- 172.21.15.55:0/3348691808 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc0ac9f0 con 0x7fa9d406c6f0 2024-03-19T01:56:40.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.154+0000 7fa9d9fde640 1 -- 172.21.15.55:0/3348691808 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa9d406c6f0 msgr2=0x7fa9d40cf010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:40.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.154+0000 7fa9d9fde640 1 --2- 172.21.15.55:0/3348691808 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa9d406c6f0 0x7fa9d40cf010 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40ce7c0 tx=0x7fa9bc01c4b0 comp rx=0 tx=0).stop 2024-03-19T01:56:40.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.154+0000 7fa9d9fde640 1 -- 172.21.15.55:0/3348691808 shutdown_connections 2024-03-19T01:56:40.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.154+0000 7fa9d9fde640 1 --2- 172.21.15.55:0/3348691808 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa9d406c6f0 0x7fa9d40cf010 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:40.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.154+0000 7fa9d9fde640 1 --2- 172.21.15.55:0/3348691808 >> v2:172.21.15.84:3300/0 conn(0x7fa9d40cbf30 0x7fa9d40cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:40.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.154+0000 7fa9d9fde640 1 -- 172.21.15.55:0/3348691808 >> 172.21.15.55:0/3348691808 conn(0x7fa9d40b16a0 msgr2=0x7fa9d40b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:40.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.154+0000 7fa9d9fde640 1 -- 172.21.15.55:0/3348691808 shutdown_connections 2024-03-19T01:56:40.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.154+0000 7fa9d9fde640 1 -- 172.21.15.55:0/3348691808 wait complete. 2024-03-19T01:56:40.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.154+0000 7fa9d9fde640 1 Processor -- start 2024-03-19T01:56:40.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.154+0000 7fa9d9fde640 1 -- start start 2024-03-19T01:56:40.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.154+0000 7fa9d9fde640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa9d406c6f0 0x7fa9d414e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:40.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.154+0000 7fa9d9fde640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa9d40cbf30 0x7fa9d41f3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:40.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.154+0000 7fa9d9fde640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9d415a3b0 0x7fa9d415ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:40.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.154+0000 7fa9d9fde640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d40cc310 con 0x7fa9d415a3b0 2024-03-19T01:56:40.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.154+0000 7fa9d9fde640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d40b23b0 con 0x7fa9d40cbf30 2024-03-19T01:56:40.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.154+0000 7fa9d9fde640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa9d40cc490 con 0x7fa9d406c6f0 2024-03-19T01:56:40.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.154+0000 7fa9d3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9d415a3b0 0x7fa9d415ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:40.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.154+0000 7fa9d3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9d415a3b0 0x7fa9d415ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55620/0 (socket says 172.21.15.55:55620) 2024-03-19T01:56:40.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.154+0000 7fa9d3fff640 1 -- 172.21.15.55:0/2633826614 learned_addr learned my addr 172.21.15.55:0/2633826614 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:40.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.154+0000 7fa9cbfff640 1 --2- 172.21.15.55:0/2633826614 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa9d40cbf30 0x7fa9d41f3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:40.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.154+0000 7fa9d37fe640 1 --2- 172.21.15.55:0/2633826614 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa9d406c6f0 0x7fa9d414e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:40.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.158+0000 7fa9d3fff640 1 -- 172.21.15.55:0/2633826614 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa9d406c6f0 msgr2=0x7fa9d414e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:40.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.158+0000 7fa9d3fff640 1 --2- 172.21.15.55:0/2633826614 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa9d406c6f0 0x7fa9d414e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:40.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.158+0000 7fa9d3fff640 1 -- 172.21.15.55:0/2633826614 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa9d40cbf30 msgr2=0x7fa9d41f3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:40.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.158+0000 7fa9d3fff640 1 --2- 172.21.15.55:0/2633826614 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa9d40cbf30 0x7fa9d41f3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:40.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.158+0000 7fa9d3fff640 1 -- 172.21.15.55:0/2633826614 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d415e300 con 0x7fa9d415a3b0 2024-03-19T01:56:40.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.162+0000 7fa9d3fff640 1 --2- 172.21.15.55:0/2633826614 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9d415a3b0 0x7fa9d415ddc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00c7a0 tx=0x7fa9cc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:40.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.166+0000 7fa9d0ff9640 1 -- 172.21.15.55:0/2633826614 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc003b80 con 0x7fa9d415a3b0 2024-03-19T01:56:40.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.166+0000 7fa9d0ff9640 1 -- 172.21.15.55:0/2633826614 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9cc003d20 con 0x7fa9d415a3b0 2024-03-19T01:56:40.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.166+0000 7fa9d9fde640 1 -- 172.21.15.55:0/2633826614 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9d414eb80 con 0x7fa9d415a3b0 2024-03-19T01:56:40.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.166+0000 7fa9d0ff9640 1 -- 172.21.15.55:0/2633826614 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc00aa30 con 0x7fa9d415a3b0 2024-03-19T01:56:40.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.166+0000 7fa9d9fde640 1 -- 172.21.15.55:0/2633826614 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9d414d380 con 0x7fa9d415a3b0 2024-03-19T01:56:40.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.166+0000 7fa9d9fde640 1 -- 172.21.15.55:0/2633826614 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9d4068330 con 0x7fa9d415a3b0 2024-03-19T01:56:40.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.194+0000 7fa9d0ff9640 1 -- 172.21.15.55:0/2633826614 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa9cc00abd0 con 0x7fa9d415a3b0 2024-03-19T01:56:40.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.194+0000 7fa9d0ff9640 1 --2- 172.21.15.55:0/2633826614 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa9b40676b0 0x7fa9b4069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:40.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.194+0000 7fa9d37fe640 1 --2- 172.21.15.55:0/2633826614 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa9b40676b0 0x7fa9b4069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:40.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.194+0000 7fa9d37fe640 1 --2- 172.21.15.55:0/2633826614 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa9b40676b0 0x7fa9b4069b70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4009ac0 tx=0x7fa9c4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:40.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.562+0000 7fa9d0ff9640 1 -- 172.21.15.55:0/2633826614 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fa9cc0547d0 con 0x7fa9d415a3b0 2024-03-19T01:56:40.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.562+0000 7fa9d0ff9640 1 -- 172.21.15.55:0/2633826614 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa9cc056730 con 0x7fa9d415a3b0 2024-03-19T01:56:40.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.830+0000 7fa9d9fde640 1 -- 172.21.15.55:0/2633826614 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9d40b2190 con 0x7fa9d415a3b0 2024-03-19T01:56:40.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.894+0000 7fa9d0ff9640 1 -- 172.21.15.55:0/2633826614 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fa9cc05a7d0 con 0x7fa9d415a3b0 2024-03-19T01:56:40.894 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:40.894 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:40.895 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:40.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.898+0000 7fa9d9fde640 1 -- 172.21.15.55:0/2633826614 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa9b40676b0 msgr2=0x7fa9b4069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:40.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.898+0000 7fa9d9fde640 1 --2- 172.21.15.55:0/2633826614 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa9b40676b0 0x7fa9b4069b70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4009ac0 tx=0x7fa9c4009190 comp rx=0 tx=0).stop 2024-03-19T01:56:40.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.898+0000 7fa9d9fde640 1 -- 172.21.15.55:0/2633826614 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9d415a3b0 msgr2=0x7fa9d415ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:40.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.898+0000 7fa9d9fde640 1 --2- 172.21.15.55:0/2633826614 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9d415a3b0 0x7fa9d415ddc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00c7a0 tx=0x7fa9cc00cc60 comp rx=0 tx=0).stop 2024-03-19T01:56:40.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.898+0000 7fa9d9fde640 1 -- 172.21.15.55:0/2633826614 shutdown_connections 2024-03-19T01:56:40.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.898+0000 7fa9d9fde640 1 --2- 172.21.15.55:0/2633826614 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa9b40676b0 0x7fa9b4069b70 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:40.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.898+0000 7fa9d9fde640 1 --2- 172.21.15.55:0/2633826614 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9d415a3b0 0x7fa9d415ddc0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:40.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.898+0000 7fa9d9fde640 1 --2- 172.21.15.55:0/2633826614 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa9d40cbf30 0x7fa9d41f3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:40.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.898+0000 7fa9d9fde640 1 --2- 172.21.15.55:0/2633826614 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa9d406c6f0 0x7fa9d414e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:40.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.898+0000 7fa9d9fde640 1 -- 172.21.15.55:0/2633826614 >> 172.21.15.55:0/2633826614 conn(0x7fa9d40b16a0 msgr2=0x7fa9d40b0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:40.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.898+0000 7fa9d9fde640 1 -- 172.21.15.55:0/2633826614 shutdown_connections 2024-03-19T01:56:40.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:40.898+0000 7fa9d9fde640 1 -- 172.21.15.55:0/2633826614 wait complete. 2024-03-19T01:56:41.520 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:41.521 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:42.522 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:42.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.650+0000 7f7491c13640 1 Processor -- start 2024-03-19T01:56:42.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.650+0000 7f7491c13640 1 -- start start 2024-03-19T01:56:42.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.650+0000 7f7491c13640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f748c152120 0x7f748c0c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:42.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.650+0000 7f7491c13640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f748c06fff0 con 0x7f748c070780 2024-03-19T01:56:42.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.650+0000 7f7491c13640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f748c0c39c0 con 0x7f748c0c9610 2024-03-19T01:56:42.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.650+0000 7f7491c13640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f748c152500 con 0x7f748c152120 2024-03-19T01:56:42.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.654+0000 7f748bfff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f748c070780 0x7f748c1586e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:42030/0 (socket says 172.21.15.55:42030) 2024-03-19T01:56:42.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.654+0000 7f748bfff640 1 -- 172.21.15.55:0/3440134382 learned_addr learned my addr 172.21.15.55:0/3440134382 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:56:42.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.654+0000 7f748b7fe640 1 --2- 172.21.15.55:0/3440134382 >> v2:172.21.15.84:3300/0 conn(0x7f748c152120 0x7f748c0c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:42.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.654+0000 7f748a7fc640 1 -- 172.21.15.55:0/3440134382 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2201586050 0 0) 0x7f748c06fff0 con 0x7f748c070780 2024-03-19T01:56:42.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.654+0000 7f748a7fc640 1 -- 172.21.15.55:0/3440134382 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7474003530 con 0x7f748c070780 2024-03-19T01:56:42.657 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.658+0000 7f748b7fe640 1 -- 172.21.15.55:0/3440134382 >> v1:172.21.15.84:6789/0 conn(0x7f748c0c9610 legacy=0x7f748c157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:42.657 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.658+0000 7f748b7fe640 1 -- 172.21.15.55:0/3440134382 >> v1:172.21.15.55:6789/0 conn(0x7f748c070780 legacy=0x7f748c1586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:42.657 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.658+0000 7f748b7fe640 1 -- 172.21.15.55:0/3440134382 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f748c0702b0 con 0x7f748c152120 2024-03-19T01:56:42.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.662+0000 7f748b7fe640 1 --2- 172.21.15.55:0/3440134382 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f748c152120 0x7f748c0c90d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f747c0097c0 tx=0x7f747c01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95bc3ad51a8665b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:42.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.670+0000 7f748a7fc640 1 -- 172.21.15.55:0/3440134382 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747c0b0070 con 0x7f748c152120 2024-03-19T01:56:42.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.670+0000 7f748a7fc640 1 -- 172.21.15.55:0/3440134382 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f747c0ab600 con 0x7f748c152120 2024-03-19T01:56:42.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.670+0000 7f748a7fc640 1 -- 172.21.15.55:0/3440134382 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747c0ab9d0 con 0x7f748c152120 2024-03-19T01:56:42.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.670+0000 7f7491c13640 1 -- 172.21.15.55:0/3440134382 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f748c152120 msgr2=0x7f748c0c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:42.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.670+0000 7f7491c13640 1 --2- 172.21.15.55:0/3440134382 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f748c152120 0x7f748c0c90d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f747c0097c0 tx=0x7f747c01c4a0 comp rx=0 tx=0).stop 2024-03-19T01:56:42.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.670+0000 7f7491c13640 1 -- 172.21.15.55:0/3440134382 shutdown_connections 2024-03-19T01:56:42.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.670+0000 7f7491c13640 1 --2- 172.21.15.55:0/3440134382 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f748c152120 0x7f748c0c90d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:42.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.670+0000 7f7491c13640 1 -- 172.21.15.55:0/3440134382 >> 172.21.15.55:0/3440134382 conn(0x7f748c067600 msgr2=0x7f748c071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:42.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.670+0000 7f7491c13640 1 -- 172.21.15.55:0/3440134382 shutdown_connections 2024-03-19T01:56:42.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.670+0000 7f7491c13640 1 -- 172.21.15.55:0/3440134382 wait complete. 2024-03-19T01:56:42.671 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.674+0000 7f7491c13640 1 Processor -- start 2024-03-19T01:56:42.671 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.674+0000 7f7491c13640 1 -- start start 2024-03-19T01:56:42.672 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.674+0000 7f7491c13640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f748c070780 0x7f748c141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:42.672 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.674+0000 7f7491c13640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f748c0c9610 0x7f748c1f5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:42.672 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.674+0000 7f7491c13640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f748c152120 0x7f748c0b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:42.672 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.674+0000 7f7491c13640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f748c158df0 con 0x7f748c070780 2024-03-19T01:56:42.672 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.674+0000 7f7491c13640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f748c0c3530 con 0x7f748c152120 2024-03-19T01:56:42.672 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.674+0000 7f7491c13640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f748c0c86d0 con 0x7f748c0c9610 2024-03-19T01:56:42.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.674+0000 7f748b7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f748c070780 0x7f748c141aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:42.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.674+0000 7f748b7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f748c070780 0x7f748c141aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55632/0 (socket says 172.21.15.55:55632) 2024-03-19T01:56:42.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.674+0000 7f748b7fe640 1 -- 172.21.15.55:0/3851861913 learned_addr learned my addr 172.21.15.55:0/3851861913 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:42.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.674+0000 7f748bfff640 1 --2- 172.21.15.55:0/3851861913 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f748c152120 0x7f748c0b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:42.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.674+0000 7f748affd640 1 --2- 172.21.15.55:0/3851861913 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f748c0c9610 0x7f748c1f5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:42.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.678+0000 7f748b7fe640 1 -- 172.21.15.55:0/3851861913 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f748c0c9610 msgr2=0x7f748c1f5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:42.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.678+0000 7f748b7fe640 1 --2- 172.21.15.55:0/3851861913 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f748c0c9610 0x7f748c1f5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:42.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.678+0000 7f748b7fe640 1 -- 172.21.15.55:0/3851861913 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f748c152120 msgr2=0x7f748c0b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:42.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.678+0000 7f748b7fe640 1 --2- 172.21.15.55:0/3851861913 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f748c152120 0x7f748c0b20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:42.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.678+0000 7f748b7fe640 1 -- 172.21.15.55:0/3851861913 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f748c1417a0 con 0x7f748c070780 2024-03-19T01:56:42.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.682+0000 7f748b7fe640 1 --2- 172.21.15.55:0/3851861913 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f748c070780 0x7f748c141aa0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f747c0190b0 tx=0x7f747c01ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:42.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.686+0000 7f7473fff640 1 -- 172.21.15.55:0/3851861913 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747c0b0050 con 0x7f748c070780 2024-03-19T01:56:42.684 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.686+0000 7f7473fff640 1 -- 172.21.15.55:0/3851861913 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f747c0bc070 con 0x7f748c070780 2024-03-19T01:56:42.684 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.686+0000 7f7491c13640 1 -- 172.21.15.55:0/3851861913 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f748c1f5750 con 0x7f748c070780 2024-03-19T01:56:42.684 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.686+0000 7f7473fff640 1 -- 172.21.15.55:0/3851861913 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747c0b5830 con 0x7f748c070780 2024-03-19T01:56:42.684 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.686+0000 7f7491c13640 1 -- 172.21.15.55:0/3851861913 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f748c0cbd10 con 0x7f748c070780 2024-03-19T01:56:42.684 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.686+0000 7f7491c13640 1 -- 172.21.15.55:0/3851861913 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f748c074bc0 con 0x7f748c070780 2024-03-19T01:56:42.703 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.706+0000 7f7473fff640 1 -- 172.21.15.55:0/3851861913 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f747c0b59d0 con 0x7f748c070780 2024-03-19T01:56:42.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.706+0000 7f7473fff640 1 --2- 172.21.15.55:0/3851861913 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f74780676e0 0x7f7478069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:42.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.706+0000 7f748affd640 1 --2- 172.21.15.55:0/3851861913 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f74780676e0 0x7f7478069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:42.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.706+0000 7f748affd640 1 --2- 172.21.15.55:0/3851861913 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f74780676e0 0x7f7478069ba0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f748c0b2e50 tx=0x7f7480009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:42.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.710+0000 7f7473fff640 1 -- 172.21.15.55:0/3851861913 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f747c0ef040 con 0x7f748c070780 2024-03-19T01:56:42.991 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:42.994+0000 7f7473fff640 1 -- 172.21.15.55:0/3851861913 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f747c016090 con 0x7f748c070780 2024-03-19T01:56:43.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:43.194+0000 7f7491c13640 1 -- 172.21.15.55:0/3851861913 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f748c068a40 con 0x7f748c070780 2024-03-19T01:56:43.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:43.230+0000 7f7473fff640 1 -- 172.21.15.55:0/3851861913 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f747c0f0fa0 con 0x7f748c070780 2024-03-19T01:56:43.229 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:43.229 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:43.229 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:43.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:43.234+0000 7f7491c13640 1 -- 172.21.15.55:0/3851861913 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f74780676e0 msgr2=0x7f7478069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:43.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:43.234+0000 7f7491c13640 1 --2- 172.21.15.55:0/3851861913 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f74780676e0 0x7f7478069ba0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f748c0b2e50 tx=0x7f7480009040 comp rx=0 tx=0).stop 2024-03-19T01:56:43.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:43.234+0000 7f7491c13640 1 -- 172.21.15.55:0/3851861913 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f748c070780 msgr2=0x7f748c141aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:43.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:43.234+0000 7f7491c13640 1 --2- 172.21.15.55:0/3851861913 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f748c070780 0x7f748c141aa0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f747c0190b0 tx=0x7f747c01ace0 comp rx=0 tx=0).stop 2024-03-19T01:56:43.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:43.234+0000 7f7491c13640 1 -- 172.21.15.55:0/3851861913 shutdown_connections 2024-03-19T01:56:43.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:43.234+0000 7f7491c13640 1 --2- 172.21.15.55:0/3851861913 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f74780676e0 0x7f7478069ba0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:43.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:43.234+0000 7f7491c13640 1 --2- 172.21.15.55:0/3851861913 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f748c152120 0x7f748c0b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:43.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:43.234+0000 7f7491c13640 1 --2- 172.21.15.55:0/3851861913 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f748c0c9610 0x7f748c1f5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:43.232 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:43.234+0000 7f7491c13640 1 --2- 172.21.15.55:0/3851861913 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f748c070780 0x7f748c141aa0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:43.232 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:43.234+0000 7f7491c13640 1 -- 172.21.15.55:0/3851861913 >> 172.21.15.55:0/3851861913 conn(0x7f748c067600 msgr2=0x7f748c14d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:43.232 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:43.234+0000 7f7491c13640 1 -- 172.21.15.55:0/3851861913 shutdown_connections 2024-03-19T01:56:43.232 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:43.234+0000 7f7491c13640 1 -- 172.21.15.55:0/3851861913 wait complete. 2024-03-19T01:56:43.856 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:43.856 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:44.857 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:44.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.970+0000 7f0a6c502640 1 Processor -- start 2024-03-19T01:56:44.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.970+0000 7f0a6c502640 1 -- start start 2024-03-19T01:56:44.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.974+0000 7f0a6c502640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0a640c9780 0x7f0a640c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:44.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.974+0000 7f0a6c502640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a640c8f70 con 0x7f0a6413dd00 2024-03-19T01:56:44.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.974+0000 7f0a6c502640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a640c9250 con 0x7f0a6415b550 2024-03-19T01:56:44.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.974+0000 7f0a6c502640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f0a641582b0 con 0x7f0a640c9780 2024-03-19T01:56:44.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.974+0000 7f0a69a9d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0a640c9780 0x7f0a640c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:44.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.974+0000 7f0a6aa9f640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f0a6415b550 0x7f0a640c5730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:50114/0 (socket says 172.21.15.55:50114) 2024-03-19T01:56:44.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.974+0000 7f0a6aa9f640 1 -- 172.21.15.55:0/3497255492 learned_addr learned my addr 172.21.15.55:0/3497255492 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:56:44.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.978+0000 7f0a69a9d640 1 -- 172.21.15.55:0/3497255492 >> v1:172.21.15.84:6789/0 conn(0x7f0a6415b550 legacy=0x7f0a640c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:44.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.978+0000 7f0a69a9d640 1 -- 172.21.15.55:0/3497255492 >> v1:172.21.15.55:6789/0 conn(0x7f0a6413dd00 legacy=0x7f0a640c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:44.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.978+0000 7f0a69a9d640 1 -- 172.21.15.55:0/3497255492 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a64158590 con 0x7f0a640c9780 2024-03-19T01:56:44.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.986+0000 7f0a69a9d640 1 --2- 172.21.15.55:0/3497255492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a640c9780 0x7f0a640c7430 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0a540097c0 tx=0x7f0a5401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4faf31752ef48327 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:44.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.990+0000 7f0a6929c640 1 -- 172.21.15.55:0/3497255492 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a540b1070 con 0x7f0a640c9780 2024-03-19T01:56:44.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.990+0000 7f0a6929c640 1 -- 172.21.15.55:0/3497255492 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a540ac410 con 0x7f0a640c9780 2024-03-19T01:56:44.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.990+0000 7f0a6929c640 1 -- 172.21.15.55:0/3497255492 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a540ac7e0 con 0x7f0a640c9780 2024-03-19T01:56:44.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.990+0000 7f0a6c502640 1 -- 172.21.15.55:0/3497255492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a640c9780 msgr2=0x7f0a640c7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:44.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.990+0000 7f0a6c502640 1 --2- 172.21.15.55:0/3497255492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a640c9780 0x7f0a640c7430 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0a540097c0 tx=0x7f0a5401c4a0 comp rx=0 tx=0).stop 2024-03-19T01:56:44.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.990+0000 7f0a6c502640 1 -- 172.21.15.55:0/3497255492 shutdown_connections 2024-03-19T01:56:44.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.990+0000 7f0a6c502640 1 --2- 172.21.15.55:0/3497255492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a640c9780 0x7f0a640c7430 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:44.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.990+0000 7f0a6c502640 1 -- 172.21.15.55:0/3497255492 >> 172.21.15.55:0/3497255492 conn(0x7f0a6406e5a0 msgr2=0x7f0a64072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:44.991 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.994+0000 7f0a6c502640 1 -- 172.21.15.55:0/3497255492 shutdown_connections 2024-03-19T01:56:44.991 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.994+0000 7f0a6c502640 1 -- 172.21.15.55:0/3497255492 wait complete. 2024-03-19T01:56:44.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.994+0000 7f0a6c502640 1 Processor -- start 2024-03-19T01:56:44.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.994+0000 7f0a6c502640 1 -- start start 2024-03-19T01:56:44.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.994+0000 7f0a6c502640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a640c9780 0x7f0a6413a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:44.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.994+0000 7f0a6c502640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0a6413dd00 0x7f0a6413e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:44.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.994+0000 7f0a6c502640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a6415b550 0x7f0a6413e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:44.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.994+0000 7f0a6c502640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0a640c5fc0 con 0x7f0a640c9780 2024-03-19T01:56:44.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.994+0000 7f0a6c502640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f0a640c5e40 con 0x7f0a6415b550 2024-03-19T01:56:44.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.994+0000 7f0a6c502640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f0a64166520 con 0x7f0a6413dd00 2024-03-19T01:56:44.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.994+0000 7f0a6a29e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a640c9780 0x7f0a6413a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:44.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.998+0000 7f0a6aa9f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a6415b550 0x7f0a6413e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:44.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.998+0000 7f0a69a9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0a6413dd00 0x7f0a6413e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:44.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.998+0000 7f0a6a29e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a640c9780 0x7f0a6413a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55640/0 (socket says 172.21.15.55:55640) 2024-03-19T01:56:44.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:44.998+0000 7f0a6a29e640 1 -- 172.21.15.55:0/1640131164 learned_addr learned my addr 172.21.15.55:0/1640131164 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:45.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.002+0000 7f0a6a29e640 1 -- 172.21.15.55:0/1640131164 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0a6413dd00 msgr2=0x7f0a6413e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:45.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.002+0000 7f0a6a29e640 1 --2- 172.21.15.55:0/1640131164 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0a6413dd00 0x7f0a6413e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:45.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.002+0000 7f0a6a29e640 1 -- 172.21.15.55:0/1640131164 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a6415b550 msgr2=0x7f0a6413e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:45.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.002+0000 7f0a6a29e640 1 --2- 172.21.15.55:0/1640131164 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a6415b550 0x7f0a6413e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:45.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.002+0000 7f0a6a29e640 1 -- 172.21.15.55:0/1640131164 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a6413fa50 con 0x7f0a640c9780 2024-03-19T01:56:45.003 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.006+0000 7f0a6a29e640 1 --2- 172.21.15.55:0/1640131164 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a640c9780 0x7f0a6413a830 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0a6000c370 tx=0x7f0a6000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:45.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.010+0000 7f0a5affd640 1 -- 172.21.15.55:0/1640131164 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a600052d0 con 0x7f0a640c9780 2024-03-19T01:56:45.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.010+0000 7f0a5affd640 1 -- 172.21.15.55:0/1640131164 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a60005470 con 0x7f0a640c9780 2024-03-19T01:56:45.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.010+0000 7f0a5affd640 1 -- 172.21.15.55:0/1640131164 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a60007260 con 0x7f0a640c9780 2024-03-19T01:56:45.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.010+0000 7f0a6c502640 1 -- 172.21.15.55:0/1640131164 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a64155b40 con 0x7f0a640c9780 2024-03-19T01:56:45.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.010+0000 7f0a6c502640 1 -- 172.21.15.55:0/1640131164 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a64156100 con 0x7f0a640c9780 2024-03-19T01:56:45.010 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.010+0000 7f0a6c502640 1 -- 172.21.15.55:0/1640131164 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a64068a40 con 0x7f0a640c9780 2024-03-19T01:56:45.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.290+0000 7f0a5affd640 1 -- 172.21.15.55:0/1640131164 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f0a60007400 con 0x7f0a640c9780 2024-03-19T01:56:45.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.290+0000 7f0a5affd640 1 --2- 172.21.15.55:0/1640131164 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0a480676e0 0x7f0a48069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:45.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.290+0000 7f0a69a9d640 1 --2- 172.21.15.55:0/1640131164 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0a480676e0 0x7f0a48069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:45.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.294+0000 7f0a69a9d640 1 --2- 172.21.15.55:0/1640131164 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0a480676e0 0x7f0a48069ba0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0a540ae000 tx=0x7f0a54008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:45.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.306+0000 7f0a5affd640 1 -- 172.21.15.55:0/1640131164 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f0a60053840 con 0x7f0a640c9780 2024-03-19T01:56:45.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.306+0000 7f0a5affd640 1 -- 172.21.15.55:0/1640131164 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f0a600557a0 con 0x7f0a640c9780 2024-03-19T01:56:45.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.574+0000 7f0a6c502640 1 -- 172.21.15.55:0/1640131164 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a641414a0 con 0x7f0a640c9780 2024-03-19T01:56:45.599 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.602+0000 7f0a5affd640 1 -- 172.21.15.55:0/1640131164 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f0a60059840 con 0x7f0a640c9780 2024-03-19T01:56:45.600 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:45.600 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:45.600 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:45.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.602+0000 7f0a6c502640 1 -- 172.21.15.55:0/1640131164 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0a480676e0 msgr2=0x7f0a48069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:45.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.602+0000 7f0a6c502640 1 --2- 172.21.15.55:0/1640131164 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0a480676e0 0x7f0a48069ba0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0a540ae000 tx=0x7f0a54008040 comp rx=0 tx=0).stop 2024-03-19T01:56:45.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.602+0000 7f0a6c502640 1 -- 172.21.15.55:0/1640131164 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a640c9780 msgr2=0x7f0a6413a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:45.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.602+0000 7f0a6c502640 1 --2- 172.21.15.55:0/1640131164 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a640c9780 0x7f0a6413a830 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0a6000c370 tx=0x7f0a6000c830 comp rx=0 tx=0).stop 2024-03-19T01:56:45.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.602+0000 7f0a6c502640 1 -- 172.21.15.55:0/1640131164 shutdown_connections 2024-03-19T01:56:45.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.602+0000 7f0a6c502640 1 --2- 172.21.15.55:0/1640131164 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0a480676e0 0x7f0a48069ba0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:45.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.602+0000 7f0a6c502640 1 --2- 172.21.15.55:0/1640131164 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a6415b550 0x7f0a6413e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:45.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.602+0000 7f0a6c502640 1 --2- 172.21.15.55:0/1640131164 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0a6413dd00 0x7f0a6413e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:45.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.602+0000 7f0a6c502640 1 --2- 172.21.15.55:0/1640131164 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a640c9780 0x7f0a6413a830 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:45.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.602+0000 7f0a6c502640 1 -- 172.21.15.55:0/1640131164 >> 172.21.15.55:0/1640131164 conn(0x7f0a6406e5a0 msgr2=0x7f0a64067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:45.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.602+0000 7f0a6c502640 1 -- 172.21.15.55:0/1640131164 shutdown_connections 2024-03-19T01:56:45.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:45.602+0000 7f0a6c502640 1 -- 172.21.15.55:0/1640131164 wait complete. 2024-03-19T01:56:46.223 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:46.223 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:47.224 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:47.354 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.354+0000 7f20028d1640 1 Processor -- start 2024-03-19T01:56:47.354 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.354+0000 7f20028d1640 1 -- start start 2024-03-19T01:56:47.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.358+0000 7f20028d1640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ffc14eaa0 0x7f1ffc06b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:47.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.358+0000 7f20028d1640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f1ffc1585f0 0x7f1ffc158a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:47.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.358+0000 7f20028d1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ffc14ee80 con 0x7f1ffc06bec0 2024-03-19T01:56:47.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.358+0000 7f20028d1640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f1ffc0c7540 con 0x7f1ffc1585f0 2024-03-19T01:56:47.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.358+0000 7f20028d1640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1ffc0c77c0 con 0x7f1ffc14eaa0 2024-03-19T01:56:47.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.358+0000 7f1ffbfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ffc14eaa0 0x7f1ffc06b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:47.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.358+0000 7f1ffbfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ffc14eaa0 0x7f1ffc06b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:56658/0 (socket says 172.21.15.55:56658) 2024-03-19T01:56:47.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.358+0000 7f1ffbfff640 1 -- 172.21.15.55:0/2276990938 learned_addr learned my addr 172.21.15.55:0/2276990938 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:47.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.362+0000 7f1ffbfff640 1 -- 172.21.15.55:0/2276990938 >> v2:172.21.15.84:3300/0 conn(0x7f1ffc1585f0 msgr2=0x7f1ffc158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:47.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.362+0000 7f1ffbfff640 1 --2- 172.21.15.55:0/2276990938 >> v2:172.21.15.84:3300/0 conn(0x7f1ffc1585f0 0x7f1ffc158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:47.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.362+0000 7f1ffbfff640 1 -- 172.21.15.55:0/2276990938 >> v1:172.21.15.84:6789/0 conn(0x7f1ffc06bec0 legacy=0x7f1ffc0c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:47.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.362+0000 7f1ffbfff640 1 -- 172.21.15.55:0/2276990938 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ffc0c7aa0 con 0x7f1ffc14eaa0 2024-03-19T01:56:47.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.370+0000 7f1ffbfff640 1 --2- 172.21.15.55:0/2276990938 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ffc14eaa0 0x7f1ffc06b980 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1fec009870 tx=0x7f1fec01c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75de7fe8edc6f580 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:47.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.378+0000 7f1ffaffd640 1 -- 172.21.15.55:0/2276990938 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec0b2070 con 0x7f1ffc14eaa0 2024-03-19T01:56:47.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.378+0000 7f1ffaffd640 1 -- 172.21.15.55:0/2276990938 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fec0a9070 con 0x7f1ffc14eaa0 2024-03-19T01:56:47.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.378+0000 7f1ffaffd640 1 -- 172.21.15.55:0/2276990938 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec0ad880 con 0x7f1ffc14eaa0 2024-03-19T01:56:47.376 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.378+0000 7f20028d1640 1 -- 172.21.15.55:0/2276990938 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ffc14eaa0 msgr2=0x7f1ffc06b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:47.376 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.378+0000 7f20028d1640 1 --2- 172.21.15.55:0/2276990938 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ffc14eaa0 0x7f1ffc06b980 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1fec009870 tx=0x7f1fec01c490 comp rx=0 tx=0).stop 2024-03-19T01:56:47.376 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.378+0000 7f20028d1640 1 -- 172.21.15.55:0/2276990938 shutdown_connections 2024-03-19T01:56:47.377 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.378+0000 7f20028d1640 1 --2- 172.21.15.55:0/2276990938 >> v2:172.21.15.84:3300/0 conn(0x7f1ffc1585f0 0x7f1ffc158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:47.377 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.378+0000 7f20028d1640 1 --2- 172.21.15.55:0/2276990938 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ffc14eaa0 0x7f1ffc06b980 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:47.377 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.378+0000 7f20028d1640 1 -- 172.21.15.55:0/2276990938 >> 172.21.15.55:0/2276990938 conn(0x7f1ffc0cc9c0 msgr2=0x7f1ffc0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:47.377 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.378+0000 7f20028d1640 1 -- 172.21.15.55:0/2276990938 shutdown_connections 2024-03-19T01:56:47.377 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.378+0000 7f20028d1640 1 -- 172.21.15.55:0/2276990938 wait complete. 2024-03-19T01:56:47.377 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.378+0000 7f20028d1640 1 Processor -- start 2024-03-19T01:56:47.377 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.378+0000 7f20028d1640 1 -- start start 2024-03-19T01:56:47.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.382+0000 7f20028d1640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1ffc06bec0 0x7f1ffc144720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:47.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.382+0000 7f20028d1640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ffc14eaa0 0x7f1ffc157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:47.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.382+0000 7f20028d1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ffc1585f0 0x7f1ffc1f14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:47.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.382+0000 7f20028d1640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc0c6960 con 0x7f1ffc1585f0 2024-03-19T01:56:47.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.382+0000 7f20028d1640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc0c67e0 con 0x7f1ffc06bec0 2024-03-19T01:56:47.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.382+0000 7f20028d1640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1ffc0c6bf0 con 0x7f1ffc14eaa0 2024-03-19T01:56:47.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.382+0000 7f2000e6e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ffc1585f0 0x7f1ffc1f14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:47.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.382+0000 7f2000e6e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ffc1585f0 0x7f1ffc1f14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55644/0 (socket says 172.21.15.55:55644) 2024-03-19T01:56:47.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.382+0000 7f2000e6e640 1 -- 172.21.15.55:0/2214060764 learned_addr learned my addr 172.21.15.55:0/2214060764 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:47.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.382+0000 7f1ffb7fe640 1 --2- 172.21.15.55:0/2214060764 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ffc14eaa0 0x7f1ffc157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:47.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.386+0000 7f2000e6e640 1 -- 172.21.15.55:0/2214060764 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ffc14eaa0 msgr2=0x7f1ffc157680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:47.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.386+0000 7f2000e6e640 1 --2- 172.21.15.55:0/2214060764 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ffc14eaa0 0x7f1ffc157680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:47.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.386+0000 7f2000e6e640 1 -- 172.21.15.55:0/2214060764 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1ffc06bec0 msgr2=0x7f1ffc144720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:47.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.386+0000 7f2000e6e640 1 --2- 172.21.15.55:0/2214060764 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1ffc06bec0 0x7f1ffc144720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:47.384 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.386+0000 7f2000e6e640 1 -- 172.21.15.55:0/2214060764 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ffc14dba0 con 0x7f1ffc1585f0 2024-03-19T01:56:47.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.386+0000 7f2000e6e640 1 --2- 172.21.15.55:0/2214060764 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ffc1585f0 0x7f1ffc1f14b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1ff400b970 tx=0x7f1ff400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:47.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.414+0000 7f1ff8ff9640 1 -- 172.21.15.55:0/2214060764 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4019070 con 0x7f1ffc1585f0 2024-03-19T01:56:47.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.414+0000 7f1ff8ff9640 1 -- 172.21.15.55:0/2214060764 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ff400cb00 con 0x7f1ffc1585f0 2024-03-19T01:56:47.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.414+0000 7f20028d1640 1 -- 172.21.15.55:0/2214060764 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ffc14de40 con 0x7f1ffc1585f0 2024-03-19T01:56:47.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.414+0000 7f1ff8ff9640 1 -- 172.21.15.55:0/2214060764 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4014440 con 0x7f1ffc1585f0 2024-03-19T01:56:47.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.414+0000 7f20028d1640 1 -- 172.21.15.55:0/2214060764 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ffc142c20 con 0x7f1ffc1585f0 2024-03-19T01:56:47.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.418+0000 7f20028d1640 1 -- 172.21.15.55:0/2214060764 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ffc068a40 con 0x7f1ffc1585f0 2024-03-19T01:56:47.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.442+0000 7f1ff8ff9640 1 -- 172.21.15.55:0/2214060764 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1ff401c070 con 0x7f1ffc1585f0 2024-03-19T01:56:47.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.442+0000 7f1ff8ff9640 1 --2- 172.21.15.55:0/2214060764 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1fe8067660 0x7f1fe8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:47.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.446+0000 7f1ffbfff640 1 --2- 172.21.15.55:0/2214060764 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1fe8067660 0x7f1fe8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:47.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.446+0000 7f1ffbfff640 1 --2- 172.21.15.55:0/2214060764 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1fe8067660 0x7f1fe8069b20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0af000 tx=0x7f1fec004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:47.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.734+0000 7f1ff8ff9640 1 -- 172.21.15.55:0/2214060764 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f1ff4054060 con 0x7f1ffc1585f0 2024-03-19T01:56:47.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.734+0000 7f1ff8ff9640 1 -- 172.21.15.55:0/2214060764 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1ff4023020 con 0x7f1ffc1585f0 2024-03-19T01:56:47.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.954+0000 7f20028d1640 1 -- 172.21.15.55:0/2214060764 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ffc074b20 con 0x7f1ffc1585f0 2024-03-19T01:56:47.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.974+0000 7f1ff8ff9640 1 -- 172.21.15.55:0/2214060764 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f1ff4055fc0 con 0x7f1ffc1585f0 2024-03-19T01:56:47.972 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:47.973 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:47.973 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:47.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.978+0000 7f20028d1640 1 -- 172.21.15.55:0/2214060764 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1fe8067660 msgr2=0x7f1fe8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:47.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.978+0000 7f20028d1640 1 --2- 172.21.15.55:0/2214060764 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1fe8067660 0x7f1fe8069b20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0af000 tx=0x7f1fec004740 comp rx=0 tx=0).stop 2024-03-19T01:56:47.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.978+0000 7f20028d1640 1 -- 172.21.15.55:0/2214060764 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ffc1585f0 msgr2=0x7f1ffc1f14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:47.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.978+0000 7f20028d1640 1 --2- 172.21.15.55:0/2214060764 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ffc1585f0 0x7f1ffc1f14b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1ff400b970 tx=0x7f1ff400be30 comp rx=0 tx=0).stop 2024-03-19T01:56:47.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.978+0000 7f20028d1640 1 -- 172.21.15.55:0/2214060764 shutdown_connections 2024-03-19T01:56:47.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.978+0000 7f20028d1640 1 --2- 172.21.15.55:0/2214060764 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1fe8067660 0x7f1fe8069b20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:47.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.978+0000 7f20028d1640 1 --2- 172.21.15.55:0/2214060764 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ffc1585f0 0x7f1ffc1f14b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:47.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.978+0000 7f20028d1640 1 --2- 172.21.15.55:0/2214060764 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ffc14eaa0 0x7f1ffc157680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:47.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.978+0000 7f20028d1640 1 --2- 172.21.15.55:0/2214060764 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1ffc06bec0 0x7f1ffc144720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:47.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.978+0000 7f20028d1640 1 -- 172.21.15.55:0/2214060764 >> 172.21.15.55:0/2214060764 conn(0x7f1ffc0cc9c0 msgr2=0x7f1ffc0c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:47.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.978+0000 7f20028d1640 1 -- 172.21.15.55:0/2214060764 shutdown_connections 2024-03-19T01:56:47.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:47.978+0000 7f20028d1640 1 -- 172.21.15.55:0/2214060764 wait complete. 2024-03-19T01:56:48.597 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:48.597 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:49.598 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:49.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.725+0000 7fb23e731640 1 Processor -- start 2024-03-19T01:56:49.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.725+0000 7fb23e731640 1 -- start start 2024-03-19T01:56:49.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.729+0000 7fb23e731640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fb2381510f0 0x7fb2380c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:49.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.729+0000 7fb23e731640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb2380c73c0 0x7fb2380c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:49.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.729+0000 7fb23e731640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2380c3f60 con 0x7fb23815a410 2024-03-19T01:56:49.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.729+0000 7fb23e731640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fb238070fb0 con 0x7fb2380c73c0 2024-03-19T01:56:49.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.729+0000 7fb23e731640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fb2381514d0 con 0x7fb2381510f0 2024-03-19T01:56:49.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.729+0000 7fb23ccce640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb2380c73c0 0x7fb2380c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:49.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.729+0000 7fb237fff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fb2381510f0 0x7fb2380c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:49.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.729+0000 7fb23ccce640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb2380c73c0 0x7fb2380c77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51030/0 (socket says 172.21.15.55:51030) 2024-03-19T01:56:49.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.729+0000 7fb23ccce640 1 -- 172.21.15.55:0/1404407519 learned_addr learned my addr 172.21.15.55:0/1404407519 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:49.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.733+0000 7fb23ccce640 1 -- 172.21.15.55:0/1404407519 >> v2:172.21.15.84:3300/0 conn(0x7fb2381510f0 msgr2=0x7fb2380c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:49.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.733+0000 7fb23ccce640 1 --2- 172.21.15.55:0/1404407519 >> v2:172.21.15.84:3300/0 conn(0x7fb2381510f0 0x7fb2380c61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:49.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.733+0000 7fb23ccce640 1 -- 172.21.15.55:0/1404407519 >> v1:172.21.15.84:6789/0 conn(0x7fb23815a410 legacy=0x7fb2380708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:49.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.733+0000 7fb23ccce640 1 -- 172.21.15.55:0/1404407519 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2380c6870 con 0x7fb2380c73c0 2024-03-19T01:56:49.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.733+0000 7fb23ccce640 1 --2- 172.21.15.55:0/1404407519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2380c73c0 0x7fb2380c77f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb23815b510 tx=0x7fb23001f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf9bb0650e47bf93 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:49.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.741+0000 7fb236ffd640 1 -- 172.21.15.55:0/1404407519 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2300b4070 con 0x7fb2380c73c0 2024-03-19T01:56:49.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.741+0000 7fb236ffd640 1 -- 172.21.15.55:0/1404407519 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2300af400 con 0x7fb2380c73c0 2024-03-19T01:56:49.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.741+0000 7fb236ffd640 1 -- 172.21.15.55:0/1404407519 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2300af7d0 con 0x7fb2380c73c0 2024-03-19T01:56:49.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.741+0000 7fb23e731640 1 -- 172.21.15.55:0/1404407519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2380c73c0 msgr2=0x7fb2380c77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:49.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.741+0000 7fb23e731640 1 --2- 172.21.15.55:0/1404407519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2380c73c0 0x7fb2380c77f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb23815b510 tx=0x7fb23001f180 comp rx=0 tx=0).stop 2024-03-19T01:56:49.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.741+0000 7fb23e731640 1 -- 172.21.15.55:0/1404407519 shutdown_connections 2024-03-19T01:56:49.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.741+0000 7fb23e731640 1 --2- 172.21.15.55:0/1404407519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb2380c73c0 0x7fb2380c77f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:49.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.741+0000 7fb23e731640 1 --2- 172.21.15.55:0/1404407519 >> v2:172.21.15.84:3300/0 conn(0x7fb2381510f0 0x7fb2380c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:49.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.741+0000 7fb23e731640 1 -- 172.21.15.55:0/1404407519 >> 172.21.15.55:0/1404407519 conn(0x7fb2380cc9c0 msgr2=0x7fb2380ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:49.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.741+0000 7fb23e731640 1 -- 172.21.15.55:0/1404407519 shutdown_connections 2024-03-19T01:56:49.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.741+0000 7fb23e731640 1 -- 172.21.15.55:0/1404407519 wait complete. 2024-03-19T01:56:49.742 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.741+0000 7fb23e731640 1 Processor -- start 2024-03-19T01:56:49.742 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.741+0000 7fb23e731640 1 -- start start 2024-03-19T01:56:49.743 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.745+0000 7fb23e731640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb2380c73c0 0x7fb2381e4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:49.743 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.745+0000 7fb23e731640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb2381510f0 0x7fb2381f0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:49.743 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.745+0000 7fb23e731640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23815a410 0x7fb2380cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:49.743 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.745+0000 7fb23e731640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb2380c44e0 con 0x7fb23815a410 2024-03-19T01:56:49.743 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.745+0000 7fb23e731640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb2380c4200 con 0x7fb2381510f0 2024-03-19T01:56:49.743 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.745+0000 7fb23e731640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fb2380c6e30 con 0x7fb2380c73c0 2024-03-19T01:56:49.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.745+0000 7fb2377fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb2381510f0 0x7fb2381f0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:49.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.745+0000 7fb23ccce640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23815a410 0x7fb2380cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:49.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.745+0000 7fb2377fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb2381510f0 0x7fb2381f0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:60382/0 (socket says 172.21.15.55:60382) 2024-03-19T01:56:49.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.745+0000 7fb2377fe640 1 -- 172.21.15.55:0/2381764760 learned_addr learned my addr 172.21.15.55:0/2381764760 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:49.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.745+0000 7fb237fff640 1 --2- 172.21.15.55:0/2381764760 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb2380c73c0 0x7fb2381e4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:49.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.749+0000 7fb23ccce640 1 -- 172.21.15.55:0/2381764760 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb2380c73c0 msgr2=0x7fb2381e4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:49.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.749+0000 7fb23ccce640 1 --2- 172.21.15.55:0/2381764760 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb2380c73c0 0x7fb2381e4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:49.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.749+0000 7fb23ccce640 1 -- 172.21.15.55:0/2381764760 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb2381510f0 msgr2=0x7fb2381f0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:49.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.749+0000 7fb23ccce640 1 --2- 172.21.15.55:0/2381764760 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb2381510f0 0x7fb2381f0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:49.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.749+0000 7fb23ccce640 1 -- 172.21.15.55:0/2381764760 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb23814c490 con 0x7fb23815a410 2024-03-19T01:56:49.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.749+0000 7fb2377fe640 1 --2- 172.21.15.55:0/2381764760 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb2381510f0 0x7fb2381f0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:56:49.752 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.753+0000 7fb23ccce640 1 --2- 172.21.15.55:0/2381764760 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23815a410 0x7fb2380cbba0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb2300ae000 tx=0x7fb23001bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:49.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.757+0000 7fb234ff9640 1 -- 172.21.15.55:0/2381764760 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2300b4040 con 0x7fb23815a410 2024-03-19T01:56:49.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.757+0000 7fb234ff9640 1 -- 172.21.15.55:0/2381764760 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2300c0070 con 0x7fb23815a410 2024-03-19T01:56:49.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.757+0000 7fb234ff9640 1 -- 172.21.15.55:0/2381764760 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb230009510 con 0x7fb23815a410 2024-03-19T01:56:49.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.757+0000 7fb23e731640 1 -- 172.21.15.55:0/2381764760 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb23814d4f0 con 0x7fb23815a410 2024-03-19T01:56:49.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.757+0000 7fb23e731640 1 -- 172.21.15.55:0/2381764760 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb23814ca10 con 0x7fb23815a410 2024-03-19T01:56:49.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.757+0000 7fb23e731640 1 -- 172.21.15.55:0/2381764760 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb238068a40 con 0x7fb23815a410 2024-03-19T01:56:49.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.793+0000 7fb234ff9640 1 -- 172.21.15.55:0/2381764760 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb2300c5460 con 0x7fb23815a410 2024-03-19T01:56:49.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.793+0000 7fb234ff9640 1 --2- 172.21.15.55:0/2381764760 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb2240676b0 0x7fb224069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:49.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.793+0000 7fb237fff640 1 --2- 172.21.15.55:0/2381764760 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb2240676b0 0x7fb224069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:49.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:49.793+0000 7fb237fff640 1 --2- 172.21.15.55:0/2381764760 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb2240676b0 0x7fb224069b70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb228009b60 tx=0x7fb228009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:50.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:50.093+0000 7fb234ff9640 1 -- 172.21.15.55:0/2381764760 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fb2300f2a80 con 0x7fb23815a410 2024-03-19T01:56:50.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:50.093+0000 7fb234ff9640 1 -- 172.21.15.55:0/2381764760 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fb230018ce0 con 0x7fb23815a410 2024-03-19T01:56:50.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:50.309+0000 7fb23e731640 1 -- 172.21.15.55:0/2381764760 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb238150920 con 0x7fb23815a410 2024-03-19T01:56:50.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:50.325+0000 7fb234ff9640 1 -- 172.21.15.55:0/2381764760 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fb2300f49e0 con 0x7fb23815a410 2024-03-19T01:56:50.323 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:50.323 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:50.323 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:50.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:50.325+0000 7fb23e731640 1 -- 172.21.15.55:0/2381764760 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb2240676b0 msgr2=0x7fb224069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:50.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:50.325+0000 7fb23e731640 1 --2- 172.21.15.55:0/2381764760 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb2240676b0 0x7fb224069b70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb228009b60 tx=0x7fb228009190 comp rx=0 tx=0).stop 2024-03-19T01:56:50.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:50.325+0000 7fb23e731640 1 -- 172.21.15.55:0/2381764760 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23815a410 msgr2=0x7fb2380cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:50.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:50.325+0000 7fb23e731640 1 --2- 172.21.15.55:0/2381764760 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23815a410 0x7fb2380cbba0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb2300ae000 tx=0x7fb23001bf20 comp rx=0 tx=0).stop 2024-03-19T01:56:50.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:50.325+0000 7fb23e731640 1 -- 172.21.15.55:0/2381764760 shutdown_connections 2024-03-19T01:56:50.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:50.325+0000 7fb23e731640 1 --2- 172.21.15.55:0/2381764760 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb2240676b0 0x7fb224069b70 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:50.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:50.325+0000 7fb23e731640 1 --2- 172.21.15.55:0/2381764760 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23815a410 0x7fb2380cbba0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:50.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:50.325+0000 7fb23e731640 1 --2- 172.21.15.55:0/2381764760 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb2381510f0 0x7fb2381f0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:50.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:50.325+0000 7fb23e731640 1 --2- 172.21.15.55:0/2381764760 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb2380c73c0 0x7fb2381e4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:50.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:50.325+0000 7fb23e731640 1 -- 172.21.15.55:0/2381764760 >> 172.21.15.55:0/2381764760 conn(0x7fb2380cc9c0 msgr2=0x7fb2380c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:50.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:50.325+0000 7fb23e731640 1 -- 172.21.15.55:0/2381764760 shutdown_connections 2024-03-19T01:56:50.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:50.325+0000 7fb23e731640 1 -- 172.21.15.55:0/2381764760 wait complete. 2024-03-19T01:56:50.945 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:50.945 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:51.946 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:52.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.061+0000 7f1977cf2640 1 Processor -- start 2024-03-19T01:56:52.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.061+0000 7f1977cf2640 1 -- start start 2024-03-19T01:56:52.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.061+0000 7f1977cf2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1970154340 0x7f197015e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:52.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.061+0000 7f1977cf2640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1970154720 0x7f19700c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:52.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.061+0000 7f1977cf2640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1970071280 con 0x7f197015aca0 2024-03-19T01:56:52.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.061+0000 7f1977cf2640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f1970073840 con 0x7f1970154720 2024-03-19T01:56:52.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.061+0000 7f1977cf2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f197015df10 con 0x7f1970154340 2024-03-19T01:56:52.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.061+0000 7f197528d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1970154720 0x7f19700c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:52.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.065+0000 7f197528d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1970154720 0x7f19700c73c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51042/0 (socket says 172.21.15.55:51042) 2024-03-19T01:56:52.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.065+0000 7f197528d640 1 -- 172.21.15.55:0/2466929287 learned_addr learned my addr 172.21.15.55:0/2466929287 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:52.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.065+0000 7f1975a8e640 1 --2- 172.21.15.55:0/2466929287 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1970154340 0x7f197015e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:52.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.065+0000 7f197528d640 1 -- 172.21.15.55:0/2466929287 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1970154340 msgr2=0x7f197015e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:52.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.065+0000 7f197528d640 1 --2- 172.21.15.55:0/2466929287 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1970154340 0x7f197015e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:52.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.065+0000 7f197528d640 1 -- 172.21.15.55:0/2466929287 >> v1:172.21.15.84:6789/0 conn(0x7f197015aca0 legacy=0x7f1970072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:52.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.065+0000 7f197528d640 1 -- 172.21.15.55:0/2466929287 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f197015d900 con 0x7f1970154720 2024-03-19T01:56:52.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.073+0000 7f197528d640 1 --2- 172.21.15.55:0/2466929287 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1970154720 0x7f19700c73c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1960009870 tx=0x7f196001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68dd678b9bfe04a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:52.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.077+0000 7f1974a8c640 1 -- 172.21.15.55:0/2466929287 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19600ab650 con 0x7f1970154720 2024-03-19T01:56:52.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.077+0000 7f1974a8c640 1 -- 172.21.15.55:0/2466929287 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19600b0070 con 0x7f1970154720 2024-03-19T01:56:52.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.077+0000 7f1974a8c640 1 -- 172.21.15.55:0/2466929287 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19600aba10 con 0x7f1970154720 2024-03-19T01:56:52.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.077+0000 7f1977cf2640 1 -- 172.21.15.55:0/2466929287 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1970154720 msgr2=0x7f19700c73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:52.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.077+0000 7f1977cf2640 1 --2- 172.21.15.55:0/2466929287 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1970154720 0x7f19700c73c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1960009870 tx=0x7f196001c560 comp rx=0 tx=0).stop 2024-03-19T01:56:52.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.077+0000 7f1977cf2640 1 -- 172.21.15.55:0/2466929287 shutdown_connections 2024-03-19T01:56:52.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.077+0000 7f1977cf2640 1 --2- 172.21.15.55:0/2466929287 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1970154720 0x7f19700c73c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:52.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.077+0000 7f1977cf2640 1 --2- 172.21.15.55:0/2466929287 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1970154340 0x7f197015e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:52.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.077+0000 7f1977cf2640 1 -- 172.21.15.55:0/2466929287 >> 172.21.15.55:0/2466929287 conn(0x7f19700cc9c0 msgr2=0x7f19700ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:52.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.077+0000 7f1977cf2640 1 -- 172.21.15.55:0/2466929287 shutdown_connections 2024-03-19T01:56:52.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.077+0000 7f1977cf2640 1 -- 172.21.15.55:0/2466929287 wait complete. 2024-03-19T01:56:52.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.081+0000 7f1977cf2640 1 Processor -- start 2024-03-19T01:56:52.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.081+0000 7f1977cf2640 1 -- start start 2024-03-19T01:56:52.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.081+0000 7f1977cf2640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1970154340 0x7f1970150020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:52.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.081+0000 7f1977cf2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1970154720 0x7f197014ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:52.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.081+0000 7f1977cf2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f197015aca0 0x7f197014b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:52.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.081+0000 7f1977cf2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f197015b200 con 0x7f197015aca0 2024-03-19T01:56:52.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.081+0000 7f1977cf2640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f197015b080 con 0x7f1970154340 2024-03-19T01:56:52.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.081+0000 7f1977cf2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1970166b70 con 0x7f1970154720 2024-03-19T01:56:52.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.081+0000 7f197628f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f197015aca0 0x7f197014b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:52.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.081+0000 7f197528d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1970154720 0x7f197014ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:52.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.081+0000 7f1975a8e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1970154340 0x7f1970150020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:52.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.081+0000 7f197628f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f197015aca0 0x7f197014b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51054/0 (socket says 172.21.15.55:51054) 2024-03-19T01:56:52.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.081+0000 7f197628f640 1 -- 172.21.15.55:0/3058746969 learned_addr learned my addr 172.21.15.55:0/3058746969 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:52.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.085+0000 7f1975a8e640 1 -- 172.21.15.55:0/3058746969 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1970154720 msgr2=0x7f197014ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:52.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.085+0000 7f1975a8e640 1 --2- 172.21.15.55:0/3058746969 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1970154720 0x7f197014ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:52.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.085+0000 7f1975a8e640 1 -- 172.21.15.55:0/3058746969 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f197015aca0 msgr2=0x7f197014b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:52.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.085+0000 7f1975a8e640 1 --2- 172.21.15.55:0/3058746969 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f197015aca0 0x7f197014b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:52.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.085+0000 7f1975a8e640 1 -- 172.21.15.55:0/3058746969 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19701533a0 con 0x7f1970154340 2024-03-19T01:56:52.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.089+0000 7f1975a8e640 1 --2- 172.21.15.55:0/3058746969 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1970154340 0x7f1970150020 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f196c00e870 tx=0x7f196c00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:52.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.093+0000 7f19667fc640 1 -- 172.21.15.55:0/3058746969 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c017070 con 0x7f1970154340 2024-03-19T01:56:52.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.097+0000 7f19667fc640 1 -- 172.21.15.55:0/3058746969 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f196c00c940 con 0x7f1970154340 2024-03-19T01:56:52.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.097+0000 7f1977cf2640 1 -- 172.21.15.55:0/3058746969 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1970153680 con 0x7f1970154340 2024-03-19T01:56:52.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.097+0000 7f19667fc640 1 -- 172.21.15.55:0/3058746969 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c00ccf0 con 0x7f1970154340 2024-03-19T01:56:52.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.097+0000 7f1977cf2640 1 -- 172.21.15.55:0/3058746969 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f197014a640 con 0x7f1970154340 2024-03-19T01:56:52.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.097+0000 7f1977cf2640 1 -- 172.21.15.55:0/3058746969 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1970068a40 con 0x7f1970154340 2024-03-19T01:56:52.372 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.373+0000 7f19667fc640 1 -- 172.21.15.55:0/3058746969 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f196c01a070 con 0x7f1970154340 2024-03-19T01:56:52.372 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.373+0000 7f19667fc640 1 --2- 172.21.15.55:0/3058746969 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f19540676b0 0x7f1954069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:52.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.373+0000 7f197528d640 1 --2- 172.21.15.55:0/3058746969 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f19540676b0 0x7f1954069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:52.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.373+0000 7f197528d640 1 --2- 172.21.15.55:0/3058746969 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f19540676b0 0x7f1954069b70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f19600af000 tx=0x7f19600a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:52.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.377+0000 7f19667fc640 1 -- 172.21.15.55:0/3058746969 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f196c054610 con 0x7f1970154340 2024-03-19T01:56:52.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.385+0000 7f19667fc640 1 -- 172.21.15.55:0/3058746969 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f196c056570 con 0x7f1970154340 2024-03-19T01:56:52.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.589+0000 7f1977cf2640 1 -- 172.21.15.55:0/3058746969 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19700c7e50 con 0x7f1970154340 2024-03-19T01:56:52.614 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.613+0000 7f19667fc640 1 -- 172.21.15.55:0/3058746969 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f196c05a610 con 0x7f1970154340 2024-03-19T01:56:52.614 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:52.615 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:52.615 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:52.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.617+0000 7f1977cf2640 1 -- 172.21.15.55:0/3058746969 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f19540676b0 msgr2=0x7f1954069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:52.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.617+0000 7f1977cf2640 1 --2- 172.21.15.55:0/3058746969 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f19540676b0 0x7f1954069b70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f19600af000 tx=0x7f19600a7000 comp rx=0 tx=0).stop 2024-03-19T01:56:52.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.617+0000 7f1977cf2640 1 -- 172.21.15.55:0/3058746969 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1970154340 msgr2=0x7f1970150020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:52.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.617+0000 7f1977cf2640 1 --2- 172.21.15.55:0/3058746969 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1970154340 0x7f1970150020 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f196c00e870 tx=0x7f196c00ed30 comp rx=0 tx=0).stop 2024-03-19T01:56:52.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.617+0000 7f1977cf2640 1 -- 172.21.15.55:0/3058746969 shutdown_connections 2024-03-19T01:56:52.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.617+0000 7f1977cf2640 1 --2- 172.21.15.55:0/3058746969 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f19540676b0 0x7f1954069b70 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:52.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.617+0000 7f1977cf2640 1 --2- 172.21.15.55:0/3058746969 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f197015aca0 0x7f197014b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:52.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.617+0000 7f1977cf2640 1 --2- 172.21.15.55:0/3058746969 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1970154720 0x7f197014ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:52.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.617+0000 7f1977cf2640 1 --2- 172.21.15.55:0/3058746969 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1970154340 0x7f1970150020 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:52.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.617+0000 7f1977cf2640 1 -- 172.21.15.55:0/3058746969 >> 172.21.15.55:0/3058746969 conn(0x7f19700cc9c0 msgr2=0x7f197014bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:52.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.617+0000 7f1977cf2640 1 -- 172.21.15.55:0/3058746969 shutdown_connections 2024-03-19T01:56:52.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:52.617+0000 7f1977cf2640 1 -- 172.21.15.55:0/3058746969 wait complete. 2024-03-19T01:56:53.240 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:53.240 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:54.241 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:54.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.361+0000 7f07ad646640 1 Processor -- start 2024-03-19T01:56:54.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.361+0000 7f07ad646640 1 -- start start 2024-03-19T01:56:54.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.365+0000 7f07ad646640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f07a81523e0 0x7f07a806b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:54.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.365+0000 7f07ad646640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07a806be80 0x7f07a815bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:54.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.365+0000 7f07ad646640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07a8158330 con 0x7f07a80ca1a0 2024-03-19T01:56:54.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.365+0000 7f07ad646640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f07a8158610 con 0x7f07a81523e0 2024-03-19T01:56:54.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.365+0000 7f07ad646640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f07a81588f0 con 0x7f07a806be80 2024-03-19T01:56:54.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.365+0000 7f07a6ffd640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f07a81523e0 0x7f07a806b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:54.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.365+0000 7f07a77fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07a806be80 0x7f07a815bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:54.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.365+0000 7f07a77fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07a806be80 0x7f07a815bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:53190/0 (socket says 172.21.15.55:53190) 2024-03-19T01:56:54.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.365+0000 7f07a77fe640 1 -- 172.21.15.55:0/1444300375 learned_addr learned my addr 172.21.15.55:0/1444300375 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:54.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.369+0000 7f07a77fe640 1 -- 172.21.15.55:0/1444300375 >> v2:172.21.15.84:3300/0 conn(0x7f07a81523e0 msgr2=0x7f07a806b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:54.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.369+0000 7f07a77fe640 1 --2- 172.21.15.55:0/1444300375 >> v2:172.21.15.84:3300/0 conn(0x7f07a81523e0 0x7f07a806b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:54.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.369+0000 7f07a77fe640 1 -- 172.21.15.55:0/1444300375 >> v1:172.21.15.84:6789/0 conn(0x7f07a80ca1a0 legacy=0x7f07a8070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:54.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.369+0000 7f07a77fe640 1 -- 172.21.15.55:0/1444300375 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a8158bd0 con 0x7f07a806be80 2024-03-19T01:56:54.372 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.373+0000 7f07a77fe640 1 --2- 172.21.15.55:0/1444300375 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07a806be80 0x7f07a815bb30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f07a0009e50 tx=0x7f07a001f1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6ac61aac3d2291f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:54.377 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.377+0000 7f07a5ffb640 1 -- 172.21.15.55:0/1444300375 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a00b4070 con 0x7f07a806be80 2024-03-19T01:56:54.377 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.377+0000 7f07a5ffb640 1 -- 172.21.15.55:0/1444300375 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07a00af400 con 0x7f07a806be80 2024-03-19T01:56:54.377 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.377+0000 7f07a5ffb640 1 -- 172.21.15.55:0/1444300375 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a00af7d0 con 0x7f07a806be80 2024-03-19T01:56:54.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.377+0000 7f07ad646640 1 -- 172.21.15.55:0/1444300375 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07a806be80 msgr2=0x7f07a815bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:54.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.377+0000 7f07ad646640 1 --2- 172.21.15.55:0/1444300375 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07a806be80 0x7f07a815bb30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f07a0009e50 tx=0x7f07a001f1e0 comp rx=0 tx=0).stop 2024-03-19T01:56:54.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.377+0000 7f07ad646640 1 -- 172.21.15.55:0/1444300375 shutdown_connections 2024-03-19T01:56:54.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.377+0000 7f07ad646640 1 --2- 172.21.15.55:0/1444300375 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07a806be80 0x7f07a815bb30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:54.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.377+0000 7f07ad646640 1 --2- 172.21.15.55:0/1444300375 >> v2:172.21.15.84:3300/0 conn(0x7f07a81523e0 0x7f07a806b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:54.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.377+0000 7f07ad646640 1 -- 172.21.15.55:0/1444300375 >> 172.21.15.55:0/1444300375 conn(0x7f07a806c3b0 msgr2=0x7f07a806c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:54.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.377+0000 7f07ad646640 1 -- 172.21.15.55:0/1444300375 shutdown_connections 2024-03-19T01:56:54.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.377+0000 7f07ad646640 1 -- 172.21.15.55:0/1444300375 wait complete. 2024-03-19T01:56:54.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.381+0000 7f07ad646640 1 Processor -- start 2024-03-19T01:56:54.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.381+0000 7f07ad646640 1 -- start start 2024-03-19T01:56:54.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.381+0000 7f07ad646640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07a806be80 0x7f07a80c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:54.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.381+0000 7f07ad646640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07a80ca1a0 0x7f07a80c2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:54.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.381+0000 7f07ad646640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07a81523e0 0x7f07a81f3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:54.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.381+0000 7f07ad646640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f07a8157f80 con 0x7f07a81523e0 2024-03-19T01:56:54.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.381+0000 7f07ad646640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f07a815b610 con 0x7f07a80ca1a0 2024-03-19T01:56:54.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.381+0000 7f07ad646640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f07a81e2b30 con 0x7f07a806be80 2024-03-19T01:56:54.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.381+0000 7f07a6ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07a806be80 0x7f07a80c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:54.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.381+0000 7f07a77fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07a81523e0 0x7f07a81f3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:54.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.381+0000 7f07a67fc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07a80ca1a0 0x7f07a80c2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:54.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.381+0000 7f07a6ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07a806be80 0x7f07a80c1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:53206/0 (socket says 172.21.15.55:53206) 2024-03-19T01:56:54.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.381+0000 7f07a6ffd640 1 -- 172.21.15.55:0/4051493708 learned_addr learned my addr 172.21.15.55:0/4051493708 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:54.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.385+0000 7f07a77fe640 1 -- 172.21.15.55:0/4051493708 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07a806be80 msgr2=0x7f07a80c1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:54.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.385+0000 7f07a77fe640 1 --2- 172.21.15.55:0/4051493708 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07a806be80 0x7f07a80c1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:54.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.385+0000 7f07a77fe640 1 -- 172.21.15.55:0/4051493708 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07a80ca1a0 msgr2=0x7f07a80c2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:54.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.385+0000 7f07a77fe640 1 --2- 172.21.15.55:0/4051493708 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07a80ca1a0 0x7f07a80c2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:54.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.385+0000 7f07a77fe640 1 -- 172.21.15.55:0/4051493708 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a81572e0 con 0x7f07a81523e0 2024-03-19T01:56:54.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.385+0000 7f07a67fc640 1 --2- 172.21.15.55:0/4051493708 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07a80ca1a0 0x7f07a80c2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:56:54.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.389+0000 7f07a77fe640 1 --2- 172.21.15.55:0/4051493708 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07a81523e0 0x7f07a81f3500 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f07a00ae000 tx=0x7f07a001bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:54.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.393+0000 7f078f7fe640 1 -- 172.21.15.55:0/4051493708 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a00b4040 con 0x7f07a81523e0 2024-03-19T01:56:54.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.393+0000 7f078f7fe640 1 -- 172.21.15.55:0/4051493708 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07a00bf070 con 0x7f07a81523e0 2024-03-19T01:56:54.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.393+0000 7f078f7fe640 1 -- 172.21.15.55:0/4051493708 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a00b9410 con 0x7f07a81523e0 2024-03-19T01:56:54.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.397+0000 7f07ad646640 1 -- 172.21.15.55:0/4051493708 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07a81f41e0 con 0x7f07a81523e0 2024-03-19T01:56:54.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.397+0000 7f07ad646640 1 -- 172.21.15.55:0/4051493708 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07a8157000 con 0x7f07a81523e0 2024-03-19T01:56:54.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.397+0000 7f07ad646640 1 -- 172.21.15.55:0/4051493708 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07a8068a40 con 0x7f07a81523e0 2024-03-19T01:56:54.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.429+0000 7f078f7fe640 1 -- 172.21.15.55:0/4051493708 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f07a00b95b0 con 0x7f07a81523e0 2024-03-19T01:56:54.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.429+0000 7f078f7fe640 1 --2- 172.21.15.55:0/4051493708 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0794067660 0x7f0794069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:54.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.429+0000 7f07a6ffd640 1 --2- 172.21.15.55:0/4051493708 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0794067660 0x7f0794069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:54.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.433+0000 7f07a6ffd640 1 --2- 172.21.15.55:0/4051493708 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0794067660 0x7f0794069b20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0798009b80 tx=0x7f0798009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:54.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.717+0000 7f078f7fe640 1 -- 172.21.15.55:0/4051493708 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f07a0002020 con 0x7f07a81523e0 2024-03-19T01:56:54.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.717+0000 7f078f7fe640 1 -- 172.21.15.55:0/4051493708 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f07a0018d40 con 0x7f07a81523e0 2024-03-19T01:56:54.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.937+0000 7f07ad646640 1 -- 172.21.15.55:0/4051493708 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07a806b2a0 con 0x7f07a81523e0 2024-03-19T01:56:54.960 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.961+0000 7f078f7fe640 1 -- 172.21.15.55:0/4051493708 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f07a00f4800 con 0x7f07a81523e0 2024-03-19T01:56:54.960 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:54.960 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:54.960 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:54.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.961+0000 7f07ad646640 1 -- 172.21.15.55:0/4051493708 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0794067660 msgr2=0x7f0794069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:54.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.961+0000 7f07ad646640 1 --2- 172.21.15.55:0/4051493708 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0794067660 0x7f0794069b20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0798009b80 tx=0x7f0798009190 comp rx=0 tx=0).stop 2024-03-19T01:56:54.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.961+0000 7f07ad646640 1 -- 172.21.15.55:0/4051493708 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07a81523e0 msgr2=0x7f07a81f3500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:54.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.961+0000 7f07ad646640 1 --2- 172.21.15.55:0/4051493708 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07a81523e0 0x7f07a81f3500 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f07a00ae000 tx=0x7f07a001bf80 comp rx=0 tx=0).stop 2024-03-19T01:56:54.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.961+0000 7f07ad646640 1 -- 172.21.15.55:0/4051493708 shutdown_connections 2024-03-19T01:56:54.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.961+0000 7f07ad646640 1 --2- 172.21.15.55:0/4051493708 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0794067660 0x7f0794069b20 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:54.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.961+0000 7f07ad646640 1 --2- 172.21.15.55:0/4051493708 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07a81523e0 0x7f07a81f3500 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:54.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.961+0000 7f07ad646640 1 --2- 172.21.15.55:0/4051493708 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07a80ca1a0 0x7f07a80c2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:54.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.961+0000 7f07ad646640 1 --2- 172.21.15.55:0/4051493708 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07a806be80 0x7f07a80c1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:54.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.961+0000 7f07ad646640 1 -- 172.21.15.55:0/4051493708 >> 172.21.15.55:0/4051493708 conn(0x7f07a806c3b0 msgr2=0x7f07a80c7500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:54.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.961+0000 7f07ad646640 1 -- 172.21.15.55:0/4051493708 shutdown_connections 2024-03-19T01:56:54.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:54.961+0000 7f07ad646640 1 -- 172.21.15.55:0/4051493708 wait complete. 2024-03-19T01:56:55.585 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:55.585 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:56.586 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:56.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.701+0000 7f36ee04f640 1 Processor -- start 2024-03-19T01:56:56.701 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.701+0000 7f36ee04f640 1 -- start start 2024-03-19T01:56:56.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.701+0000 7f36ee04f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f36e80c60d0 0x7f36e80c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:56.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.701+0000 7f36ee04f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f36e81585f0 0x7f36e8158a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:56.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.701+0000 7f36ee04f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36e814ee80 con 0x7f36e814eaa0 2024-03-19T01:56:56.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.701+0000 7f36ee04f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f36e8158f50 con 0x7f36e80c60d0 2024-03-19T01:56:56.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.701+0000 7f36ee04f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f36e80c7540 con 0x7f36e81585f0 2024-03-19T01:56:56.703 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.705+0000 7f36e77fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f36e814eaa0 0x7f36e806b980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:37740/0 (socket says 172.21.15.55:37740) 2024-03-19T01:56:56.703 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.705+0000 7f36e77fe640 1 -- 172.21.15.55:0/868636082 learned_addr learned my addr 172.21.15.55:0/868636082 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:56:56.703 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.705+0000 7f36e7fff640 1 --2- 172.21.15.55:0/868636082 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f36e81585f0 0x7f36e8158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:56.703 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.705+0000 7f36e6ffd640 1 --2- 172.21.15.55:0/868636082 >> v2:172.21.15.55:3300/0 conn(0x7f36e80c60d0 0x7f36e80c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:56.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.709+0000 7f36e67fc640 1 -- 172.21.15.55:0/868636082 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2410434925 0 0) 0x7f36e814ee80 con 0x7f36e814eaa0 2024-03-19T01:56:56.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.709+0000 7f36e67fc640 1 -- 172.21.15.55:0/868636082 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36d0003530 con 0x7f36e814eaa0 2024-03-19T01:56:56.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.709+0000 7f36e7fff640 1 -- 172.21.15.55:0/868636082 >> v2:172.21.15.55:3300/0 conn(0x7f36e80c60d0 msgr2=0x7f36e80c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:56.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.709+0000 7f36e7fff640 1 --2- 172.21.15.55:0/868636082 >> v2:172.21.15.55:3300/0 conn(0x7f36e80c60d0 0x7f36e80c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:56.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.709+0000 7f36e7fff640 1 -- 172.21.15.55:0/868636082 >> v1:172.21.15.55:6789/0 conn(0x7f36e814eaa0 legacy=0x7f36e806b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:56.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.709+0000 7f36e7fff640 1 -- 172.21.15.55:0/868636082 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36e80c77c0 con 0x7f36e81585f0 2024-03-19T01:56:56.712 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.713+0000 7f36e7fff640 1 --2- 172.21.15.55:0/868636082 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f36e81585f0 0x7f36e8158a10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f36e0009d90 tx=0x7f36e001c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4ebc378f131bcb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:56.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.717+0000 7f36e67fc640 1 -- 172.21.15.55:0/868636082 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e00b2070 con 0x7f36e81585f0 2024-03-19T01:56:56.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.721+0000 7f36e67fc640 1 -- 172.21.15.55:0/868636082 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36e00a59a0 con 0x7f36e81585f0 2024-03-19T01:56:56.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.721+0000 7f36e67fc640 1 -- 172.21.15.55:0/868636082 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e00a5d50 con 0x7f36e81585f0 2024-03-19T01:56:56.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.721+0000 7f36ee04f640 1 -- 172.21.15.55:0/868636082 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f36e81585f0 msgr2=0x7f36e8158a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:56.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.721+0000 7f36ee04f640 1 --2- 172.21.15.55:0/868636082 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f36e81585f0 0x7f36e8158a10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f36e0009d90 tx=0x7f36e001c8e0 comp rx=0 tx=0).stop 2024-03-19T01:56:56.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.721+0000 7f36ee04f640 1 -- 172.21.15.55:0/868636082 shutdown_connections 2024-03-19T01:56:56.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.721+0000 7f36ee04f640 1 --2- 172.21.15.55:0/868636082 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f36e81585f0 0x7f36e8158a10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:56.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.721+0000 7f36ee04f640 1 --2- 172.21.15.55:0/868636082 >> v2:172.21.15.55:3300/0 conn(0x7f36e80c60d0 0x7f36e80c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:56.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.721+0000 7f36ee04f640 1 -- 172.21.15.55:0/868636082 >> 172.21.15.55:0/868636082 conn(0x7f36e80cc9c0 msgr2=0x7f36e80ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:56.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.721+0000 7f36ee04f640 1 -- 172.21.15.55:0/868636082 shutdown_connections 2024-03-19T01:56:56.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.721+0000 7f36ee04f640 1 -- 172.21.15.55:0/868636082 wait complete. 2024-03-19T01:56:56.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.721+0000 7f36ee04f640 1 Processor -- start 2024-03-19T01:56:56.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.721+0000 7f36ee04f640 1 -- start start 2024-03-19T01:56:56.722 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.721+0000 7f36ee04f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f36e80c60d0 0x7f36e80c4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:56.722 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.721+0000 7f36ee04f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f36e814eaa0 0x7f36e8143af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:56.722 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.721+0000 7f36ee04f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36e81585f0 0x7f36e80caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:56.722 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.721+0000 7f36ee04f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f36e806c090 con 0x7f36e814eaa0 2024-03-19T01:56:56.722 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.721+0000 7f36ee04f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f36e80c8060 con 0x7f36e81585f0 2024-03-19T01:56:56.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.721+0000 7f36ee04f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f36e806c210 con 0x7f36e80c60d0 2024-03-19T01:56:56.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.725+0000 7f36e6ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f36e814eaa0 0x7f36e8143af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:56.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.725+0000 7f36e6ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f36e814eaa0 0x7f36e8143af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51078/0 (socket says 172.21.15.55:51078) 2024-03-19T01:56:56.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.725+0000 7f36e6ffd640 1 -- 172.21.15.55:0/2787826259 learned_addr learned my addr 172.21.15.55:0/2787826259 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:56.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.725+0000 7f36e7fff640 1 --2- 172.21.15.55:0/2787826259 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36e81585f0 0x7f36e80caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:56.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.729+0000 7f36e6ffd640 1 -- 172.21.15.55:0/2787826259 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f36e80c60d0 msgr2=0x7f36e80c4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:56.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.729+0000 7f36e6ffd640 1 --2- 172.21.15.55:0/2787826259 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f36e80c60d0 0x7f36e80c4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:56.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.729+0000 7f36e6ffd640 1 -- 172.21.15.55:0/2787826259 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36e81585f0 msgr2=0x7f36e80caa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:56.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.729+0000 7f36e6ffd640 1 --2- 172.21.15.55:0/2787826259 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36e81585f0 0x7f36e80caa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:56.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.729+0000 7f36e6ffd640 1 -- 172.21.15.55:0/2787826259 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36e80c5ca0 con 0x7f36e814eaa0 2024-03-19T01:56:56.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.733+0000 7f36e6ffd640 1 --2- 172.21.15.55:0/2787826259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f36e814eaa0 0x7f36e8143af0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f36dc00e8a0 tx=0x7f36dc00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:56.735 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.737+0000 7f36cffff640 1 -- 172.21.15.55:0/2787826259 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc017070 con 0x7f36e814eaa0 2024-03-19T01:56:56.735 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.737+0000 7f36cffff640 1 -- 172.21.15.55:0/2787826259 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36dc00c940 con 0x7f36e814eaa0 2024-03-19T01:56:56.735 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.737+0000 7f36ee04f640 1 -- 172.21.15.55:0/2787826259 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36e80c53f0 con 0x7f36e814eaa0 2024-03-19T01:56:56.735 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.737+0000 7f36cffff640 1 -- 172.21.15.55:0/2787826259 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc00ccf0 con 0x7f36e814eaa0 2024-03-19T01:56:56.735 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.737+0000 7f36ee04f640 1 -- 172.21.15.55:0/2787826259 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36e8162400 con 0x7f36e814eaa0 2024-03-19T01:56:56.736 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.737+0000 7f36ee04f640 1 -- 172.21.15.55:0/2787826259 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36e8068a40 con 0x7f36e814eaa0 2024-03-19T01:56:56.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.761+0000 7f36cffff640 1 -- 172.21.15.55:0/2787826259 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f36dc01a070 con 0x7f36e814eaa0 2024-03-19T01:56:56.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.761+0000 7f36cffff640 1 --2- 172.21.15.55:0/2787826259 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f36d40676e0 0x7f36d4069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:56.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.761+0000 7f36e77fe640 1 --2- 172.21.15.55:0/2787826259 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f36d40676e0 0x7f36d4069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:56.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.761+0000 7f36e77fe640 1 --2- 172.21.15.55:0/2787826259 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f36d40676e0 0x7f36d4069ba0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f36d8007fd0 tx=0x7f36d8007900 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:56.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:56.765+0000 7f36cffff640 1 -- 172.21.15.55:0/2787826259 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f36dc054590 con 0x7f36e814eaa0 2024-03-19T01:56:57.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:57.049+0000 7f36cffff640 1 -- 172.21.15.55:0/2787826259 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f36dc0564f0 con 0x7f36e814eaa0 2024-03-19T01:56:57.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:57.253+0000 7f36ee04f640 1 -- 172.21.15.55:0/2787826259 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36e8162720 con 0x7f36e814eaa0 2024-03-19T01:56:57.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:57.329+0000 7f36cffff640 1 -- 172.21.15.55:0/2787826259 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f36dc05a590 con 0x7f36e814eaa0 2024-03-19T01:56:57.328 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:57.328 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:57.328 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:57.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:57.329+0000 7f36ee04f640 1 -- 172.21.15.55:0/2787826259 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f36d40676e0 msgr2=0x7f36d4069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:57.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:57.329+0000 7f36ee04f640 1 --2- 172.21.15.55:0/2787826259 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f36d40676e0 0x7f36d4069ba0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f36d8007fd0 tx=0x7f36d8007900 comp rx=0 tx=0).stop 2024-03-19T01:56:57.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:57.333+0000 7f36ee04f640 1 -- 172.21.15.55:0/2787826259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f36e814eaa0 msgr2=0x7f36e8143af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:57.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:57.333+0000 7f36ee04f640 1 --2- 172.21.15.55:0/2787826259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f36e814eaa0 0x7f36e8143af0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f36dc00e8a0 tx=0x7f36dc00eca0 comp rx=0 tx=0).stop 2024-03-19T01:56:57.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:57.333+0000 7f36ee04f640 1 -- 172.21.15.55:0/2787826259 shutdown_connections 2024-03-19T01:56:57.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:57.333+0000 7f36ee04f640 1 --2- 172.21.15.55:0/2787826259 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f36d40676e0 0x7f36d4069ba0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:57.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:57.333+0000 7f36ee04f640 1 --2- 172.21.15.55:0/2787826259 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36e81585f0 0x7f36e80caa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:57.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:57.333+0000 7f36ee04f640 1 --2- 172.21.15.55:0/2787826259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f36e814eaa0 0x7f36e8143af0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:57.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:57.333+0000 7f36ee04f640 1 --2- 172.21.15.55:0/2787826259 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f36e80c60d0 0x7f36e80c4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:57.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:57.333+0000 7f36ee04f640 1 -- 172.21.15.55:0/2787826259 >> 172.21.15.55:0/2787826259 conn(0x7f36e80cc9c0 msgr2=0x7f36e814b630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:57.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:57.333+0000 7f36ee04f640 1 -- 172.21.15.55:0/2787826259 shutdown_connections 2024-03-19T01:56:57.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:57.333+0000 7f36ee04f640 1 -- 172.21.15.55:0/2787826259 wait complete. 2024-03-19T01:56:57.955 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:56:57.955 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:56:58.955 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:56:59.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.085+0000 7ff2e9c70640 1 Processor -- start 2024-03-19T01:56:59.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.085+0000 7ff2e9c70640 1 -- start start 2024-03-19T01:56:59.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.089+0000 7ff2e9c70640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff2e40cbf30 0x7ff2e40cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:59.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.089+0000 7ff2e9c70640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff2e406c6f0 0x7ff2e40cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:59.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.089+0000 7ff2e9c70640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2e406cb90 con 0x7ff2e415a3b0 2024-03-19T01:56:59.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.089+0000 7ff2e9c70640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7ff2e40ce640 con 0x7ff2e40cbf30 2024-03-19T01:56:59.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.089+0000 7ff2e9c70640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff2e40cda30 con 0x7ff2e406c6f0 2024-03-19T01:56:59.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.089+0000 7ff2e37fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff2e40cbf30 0x7ff2e40cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:59.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.089+0000 7ff2e3fff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7ff2e415a3b0 0x7ff2e415a830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:45514/0 (socket says 172.21.15.55:45514) 2024-03-19T01:56:59.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.089+0000 7ff2e3fff640 1 -- 172.21.15.55:0/1547194333 learned_addr learned my addr 172.21.15.55:0/1547194333 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:56:59.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.089+0000 7ff2e2ffd640 1 --2- 172.21.15.55:0/1547194333 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff2e406c6f0 0x7ff2e40cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:59.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.093+0000 7ff2e27fc640 1 -- 172.21.15.55:0/1547194333 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2742608863 0 0) 0x7ff2e406cb90 con 0x7ff2e415a3b0 2024-03-19T01:56:59.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.093+0000 7ff2e27fc640 1 -- 172.21.15.55:0/1547194333 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2cc003530 con 0x7ff2e415a3b0 2024-03-19T01:56:59.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.093+0000 7ff2e37fe640 1 -- 172.21.15.55:0/1547194333 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff2e406c6f0 msgr2=0x7ff2e40cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:56:59.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.093+0000 7ff2e37fe640 1 --2- 172.21.15.55:0/1547194333 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff2e406c6f0 0x7ff2e40cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:59.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.093+0000 7ff2e37fe640 1 -- 172.21.15.55:0/1547194333 >> v1:172.21.15.84:6789/0 conn(0x7ff2e415a3b0 legacy=0x7ff2e415a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:59.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.093+0000 7ff2e37fe640 1 -- 172.21.15.55:0/1547194333 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2e40cdd10 con 0x7ff2e40cbf30 2024-03-19T01:56:59.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.097+0000 7ff2e37fe640 1 --2- 172.21.15.55:0/1547194333 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2e40cbf30 0x7ff2e40cead0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff2e40ce7c0 tx=0x7ff2d401c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e83ae9d547d353 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:59.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.101+0000 7ff2e27fc640 1 -- 172.21.15.55:0/1547194333 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d40b2070 con 0x7ff2e40cbf30 2024-03-19T01:56:59.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.101+0000 7ff2e27fc640 1 -- 172.21.15.55:0/1547194333 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2d40a5890 con 0x7ff2e40cbf30 2024-03-19T01:56:59.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.101+0000 7ff2e27fc640 1 -- 172.21.15.55:0/1547194333 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d40a5c60 con 0x7ff2e40cbf30 2024-03-19T01:56:59.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.101+0000 7ff2e9c70640 1 -- 172.21.15.55:0/1547194333 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2e40cbf30 msgr2=0x7ff2e40cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:59.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.101+0000 7ff2e9c70640 1 --2- 172.21.15.55:0/1547194333 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2e40cbf30 0x7ff2e40cead0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff2e40ce7c0 tx=0x7ff2d401c8c0 comp rx=0 tx=0).stop 2024-03-19T01:56:59.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.101+0000 7ff2e9c70640 1 -- 172.21.15.55:0/1547194333 shutdown_connections 2024-03-19T01:56:59.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.101+0000 7ff2e9c70640 1 --2- 172.21.15.55:0/1547194333 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff2e406c6f0 0x7ff2e40cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:59.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.101+0000 7ff2e9c70640 1 --2- 172.21.15.55:0/1547194333 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2e40cbf30 0x7ff2e40cead0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:59.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.101+0000 7ff2e9c70640 1 -- 172.21.15.55:0/1547194333 >> 172.21.15.55:0/1547194333 conn(0x7ff2e40b16a0 msgr2=0x7ff2e40b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:59.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.101+0000 7ff2e9c70640 1 -- 172.21.15.55:0/1547194333 shutdown_connections 2024-03-19T01:56:59.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.101+0000 7ff2e9c70640 1 -- 172.21.15.55:0/1547194333 wait complete. 2024-03-19T01:56:59.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.101+0000 7ff2e9c70640 1 Processor -- start 2024-03-19T01:56:59.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.101+0000 7ff2e9c70640 1 -- start start 2024-03-19T01:56:59.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.101+0000 7ff2e9c70640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2e406c6f0 0x7ff2e415b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:59.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.101+0000 7ff2e9c70640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2e40cbf30 0x7ff2e414d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:59.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.105+0000 7ff2e9c70640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff2e415a3b0 0x7ff2e414d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:59.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.105+0000 7ff2e9c70640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e40cc310 con 0x7ff2e40cbf30 2024-03-19T01:56:59.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.105+0000 7ff2e9c70640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e40b23b0 con 0x7ff2e406c6f0 2024-03-19T01:56:59.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.105+0000 7ff2e9c70640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff2e40cc490 con 0x7ff2e415a3b0 2024-03-19T01:56:59.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.105+0000 7ff2e2ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2e40cbf30 0x7ff2e414d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:59.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.105+0000 7ff2e37fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2e406c6f0 0x7ff2e415b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:59.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.105+0000 7ff2e37fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2e406c6f0 0x7ff2e415b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:53878/0 (socket says 172.21.15.55:53878) 2024-03-19T01:56:59.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.105+0000 7ff2e37fe640 1 -- 172.21.15.55:0/2519290355 learned_addr learned my addr 172.21.15.55:0/2519290355 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:56:59.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.105+0000 7ff2e2ffd640 1 --2- 172.21.15.55:0/2519290355 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2e40cbf30 0x7ff2e414d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52692/0 (socket says 172.21.15.55:52692) 2024-03-19T01:56:59.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.105+0000 7ff2e3fff640 1 --2- 172.21.15.55:0/2519290355 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff2e415a3b0 0x7ff2e414d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:59.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.109+0000 7ff2e2ffd640 1 -- 172.21.15.55:0/2519290355 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff2e415a3b0 msgr2=0x7ff2e414d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:59.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.109+0000 7ff2e2ffd640 1 --2- 172.21.15.55:0/2519290355 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff2e415a3b0 0x7ff2e414d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:59.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.109+0000 7ff2e2ffd640 1 -- 172.21.15.55:0/2519290355 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2e406c6f0 msgr2=0x7ff2e415b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:59.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.109+0000 7ff2e2ffd640 1 --2- 172.21.15.55:0/2519290355 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2e406c6f0 0x7ff2e415b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:59.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.109+0000 7ff2e2ffd640 1 -- 172.21.15.55:0/2519290355 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2e415ba50 con 0x7ff2e40cbf30 2024-03-19T01:56:59.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.109+0000 7ff2e2ffd640 1 --2- 172.21.15.55:0/2519290355 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2e40cbf30 0x7ff2e414d390 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff2d800e870 tx=0x7ff2d800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:59.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.113+0000 7ff2cbfff640 1 -- 172.21.15.55:0/2519290355 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d8017070 con 0x7ff2e40cbf30 2024-03-19T01:56:59.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.113+0000 7ff2cbfff640 1 -- 172.21.15.55:0/2519290355 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2d800ee60 con 0x7ff2e40cbf30 2024-03-19T01:56:59.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.113+0000 7ff2e9c70640 1 -- 172.21.15.55:0/2519290355 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2e414e030 con 0x7ff2e40cbf30 2024-03-19T01:56:59.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.113+0000 7ff2cbfff640 1 -- 172.21.15.55:0/2519290355 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d800cb60 con 0x7ff2e40cbf30 2024-03-19T01:56:59.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.113+0000 7ff2e9c70640 1 -- 172.21.15.55:0/2519290355 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2e415d920 con 0x7ff2e40cbf30 2024-03-19T01:56:59.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.117+0000 7ff2e9c70640 1 -- 172.21.15.55:0/2519290355 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2e4068330 con 0x7ff2e40cbf30 2024-03-19T01:56:59.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.133+0000 7ff2cbfff640 1 -- 172.21.15.55:0/2519290355 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ff2d801a070 con 0x7ff2e40cbf30 2024-03-19T01:56:59.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.133+0000 7ff2cbfff640 1 --2- 172.21.15.55:0/2519290355 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff2d00676b0 0x7ff2d0069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:56:59.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.137+0000 7ff2e37fe640 1 --2- 172.21.15.55:0/2519290355 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff2d00676b0 0x7ff2d0069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:56:59.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.137+0000 7ff2e37fe640 1 --2- 172.21.15.55:0/2519290355 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff2d00676b0 0x7ff2d0069b70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff2d4000f90 tx=0x7ff2d40a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:56:59.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.137+0000 7ff2cbfff640 1 -- 172.21.15.55:0/2519290355 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7ff2d8054440 con 0x7ff2e40cbf30 2024-03-19T01:56:59.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.425+0000 7ff2cbfff640 1 -- 172.21.15.55:0/2519290355 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ff2d80563a0 con 0x7ff2e40cbf30 2024-03-19T01:56:59.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.705+0000 7ff2e9c70640 1 -- 172.21.15.55:0/2519290355 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2e40b2190 con 0x7ff2e40cbf30 2024-03-19T01:56:59.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.725+0000 7ff2cbfff640 1 -- 172.21.15.55:0/2519290355 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7ff2d805a440 con 0x7ff2e40cbf30 2024-03-19T01:56:59.725 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:56:59.725 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:56:59.726 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:56:59.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.729+0000 7ff2e9c70640 1 -- 172.21.15.55:0/2519290355 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff2d00676b0 msgr2=0x7ff2d0069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:59.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.729+0000 7ff2e9c70640 1 --2- 172.21.15.55:0/2519290355 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff2d00676b0 0x7ff2d0069b70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff2d4000f90 tx=0x7ff2d40a7070 comp rx=0 tx=0).stop 2024-03-19T01:56:59.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.729+0000 7ff2e9c70640 1 -- 172.21.15.55:0/2519290355 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2e40cbf30 msgr2=0x7ff2e414d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:56:59.729 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.729+0000 7ff2e9c70640 1 --2- 172.21.15.55:0/2519290355 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2e40cbf30 0x7ff2e414d390 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff2d800e870 tx=0x7ff2d800ed30 comp rx=0 tx=0).stop 2024-03-19T01:56:59.729 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.729+0000 7ff2e9c70640 1 -- 172.21.15.55:0/2519290355 shutdown_connections 2024-03-19T01:56:59.729 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.729+0000 7ff2e9c70640 1 --2- 172.21.15.55:0/2519290355 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff2d00676b0 0x7ff2d0069b70 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:59.729 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.729+0000 7ff2e9c70640 1 --2- 172.21.15.55:0/2519290355 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff2e415a3b0 0x7ff2e414d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:59.729 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.729+0000 7ff2e9c70640 1 --2- 172.21.15.55:0/2519290355 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2e40cbf30 0x7ff2e414d390 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:59.729 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.729+0000 7ff2e9c70640 1 --2- 172.21.15.55:0/2519290355 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2e406c6f0 0x7ff2e415b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:56:59.729 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.729+0000 7ff2e9c70640 1 -- 172.21.15.55:0/2519290355 >> 172.21.15.55:0/2519290355 conn(0x7ff2e40b16a0 msgr2=0x7ff2e40affc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:56:59.729 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.729+0000 7ff2e9c70640 1 -- 172.21.15.55:0/2519290355 shutdown_connections 2024-03-19T01:56:59.729 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:56:59.729+0000 7ff2e9c70640 1 -- 172.21.15.55:0/2519290355 wait complete. 2024-03-19T01:57:00.348 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:00.348 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:01.349 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:01.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.469+0000 7fc407865640 1 Processor -- start 2024-03-19T01:57:01.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.469+0000 7fc407865640 1 -- start start 2024-03-19T01:57:01.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.469+0000 7fc407865640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc4000cd090 0x7fc4000732b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:01.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.469+0000 7fc407865640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc400158330 con 0x7fc40014ec60 2024-03-19T01:57:01.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.469+0000 7fc407865640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4001585f0 con 0x7fc4000cd470 2024-03-19T01:57:01.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.469+0000 7fc407865640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc400158e90 con 0x7fc4000cd090 2024-03-19T01:57:01.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.473+0000 7fc405e02640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fc40014ec60 0x7fc40006b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:60732/0 (socket says 172.21.15.55:60732) 2024-03-19T01:57:01.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.473+0000 7fc405e02640 1 -- 172.21.15.55:0/1735653725 learned_addr learned my addr 172.21.15.55:0/1735653725 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:57:01.471 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.473+0000 7fc405601640 1 --2- 172.21.15.55:0/1735653725 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc4000cd090 0x7fc4000732b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:01.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.473+0000 7fc3f7fff640 1 -- 172.21.15.55:0/1735653725 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1380578934 0 0) 0x7fc400158330 con 0x7fc40014ec60 2024-03-19T01:57:01.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.473+0000 7fc3f7fff640 1 -- 172.21.15.55:0/1735653725 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3ec003530 con 0x7fc40014ec60 2024-03-19T01:57:01.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.477+0000 7fc405601640 1 -- 172.21.15.55:0/1735653725 >> v1:172.21.15.84:6789/0 conn(0x7fc4000cd470 legacy=0x7fc40014e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:01.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.477+0000 7fc405601640 1 -- 172.21.15.55:0/1735653725 >> v1:172.21.15.55:6789/0 conn(0x7fc40014ec60 legacy=0x7fc40006b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:01.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.477+0000 7fc405601640 1 -- 172.21.15.55:0/1735653725 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4001588d0 con 0x7fc4000cd090 2024-03-19T01:57:01.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.481+0000 7fc405601640 1 --2- 172.21.15.55:0/1735653725 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc4000cd090 0x7fc4000732b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc400159010 tx=0x7fc3fc01c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9c7901ff27368b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:01.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.485+0000 7fc3f7fff640 1 -- 172.21.15.55:0/1735653725 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc0b0070 con 0x7fc4000cd090 2024-03-19T01:57:01.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.485+0000 7fc3f7fff640 1 -- 172.21.15.55:0/1735653725 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3fc0ab600 con 0x7fc4000cd090 2024-03-19T01:57:01.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.485+0000 7fc3f7fff640 1 -- 172.21.15.55:0/1735653725 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc0ab9b0 con 0x7fc4000cd090 2024-03-19T01:57:01.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.489+0000 7fc407865640 1 -- 172.21.15.55:0/1735653725 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc4000cd090 msgr2=0x7fc4000732b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:01.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.489+0000 7fc407865640 1 --2- 172.21.15.55:0/1735653725 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc4000cd090 0x7fc4000732b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc400159010 tx=0x7fc3fc01c480 comp rx=0 tx=0).stop 2024-03-19T01:57:01.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.489+0000 7fc407865640 1 -- 172.21.15.55:0/1735653725 shutdown_connections 2024-03-19T01:57:01.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.489+0000 7fc407865640 1 --2- 172.21.15.55:0/1735653725 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc4000cd090 0x7fc4000732b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:01.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.489+0000 7fc407865640 1 -- 172.21.15.55:0/1735653725 >> 172.21.15.55:0/1735653725 conn(0x7fc40006e670 msgr2=0x7fc40006ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:01.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.489+0000 7fc407865640 1 -- 172.21.15.55:0/1735653725 shutdown_connections 2024-03-19T01:57:01.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.489+0000 7fc407865640 1 -- 172.21.15.55:0/1735653725 wait complete. 2024-03-19T01:57:01.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.489+0000 7fc407865640 1 Processor -- start 2024-03-19T01:57:01.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.489+0000 7fc407865640 1 -- start start 2024-03-19T01:57:01.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.489+0000 7fc407865640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc4000cd090 0x7fc400159f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:01.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.489+0000 7fc407865640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc4000cd470 0x7fc400145f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:01.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.489+0000 7fc407865640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc40014ec60 0x7fc4000c5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:01.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.489+0000 7fc407865640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc4000737f0 con 0x7fc40014ec60 2024-03-19T01:57:01.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.489+0000 7fc407865640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc400166130 con 0x7fc4000cd090 2024-03-19T01:57:01.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.489+0000 7fc407865640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc40015a6b0 con 0x7fc4000cd470 2024-03-19T01:57:01.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.489+0000 7fc405e02640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc40014ec60 0x7fc4000c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:01.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.489+0000 7fc404e00640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc4000cd470 0x7fc400145f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:01.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.489+0000 7fc405e02640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc40014ec60 0x7fc4000c5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52704/0 (socket says 172.21.15.55:52704) 2024-03-19T01:57:01.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.489+0000 7fc405e02640 1 -- 172.21.15.55:0/3787205025 learned_addr learned my addr 172.21.15.55:0/3787205025 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:01.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.493+0000 7fc405e02640 1 -- 172.21.15.55:0/3787205025 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc4000cd470 msgr2=0x7fc400145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:01.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.493+0000 7fc405e02640 1 --2- 172.21.15.55:0/3787205025 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc4000cd470 0x7fc400145f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:01.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.493+0000 7fc405e02640 1 -- 172.21.15.55:0/3787205025 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc4000cd090 msgr2=0x7fc400159f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:01.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.493+0000 7fc405e02640 1 --2- 172.21.15.55:0/3787205025 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc4000cd090 0x7fc400159f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:01.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.493+0000 7fc405e02640 1 -- 172.21.15.55:0/3787205025 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4000cc9f0 con 0x7fc40014ec60 2024-03-19T01:57:01.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.497+0000 7fc405e02640 1 --2- 172.21.15.55:0/3787205025 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc40014ec60 0x7fc4000c5510 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc3f800ca30 tx=0x7fc3f800cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:01.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.501+0000 7fc3f5ffb640 1 -- 172.21.15.55:0/3787205025 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8017070 con 0x7fc40014ec60 2024-03-19T01:57:01.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.501+0000 7fc3f5ffb640 1 -- 172.21.15.55:0/3787205025 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3f800abd0 con 0x7fc40014ec60 2024-03-19T01:57:01.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.501+0000 7fc407865640 1 -- 172.21.15.55:0/3787205025 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc400162bc0 con 0x7fc40014ec60 2024-03-19T01:57:01.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.501+0000 7fc3f5ffb640 1 -- 172.21.15.55:0/3787205025 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8012400 con 0x7fc40014ec60 2024-03-19T01:57:01.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.501+0000 7fc407865640 1 -- 172.21.15.55:0/3787205025 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4000cc0c0 con 0x7fc40014ec60 2024-03-19T01:57:01.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.501+0000 7fc407865640 1 -- 172.21.15.55:0/3787205025 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc400068a40 con 0x7fc40014ec60 2024-03-19T01:57:01.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.809+0000 7fc3f5ffb640 1 -- 172.21.15.55:0/3787205025 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc3f801a070 con 0x7fc40014ec60 2024-03-19T01:57:01.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.809+0000 7fc3f5ffb640 1 --2- 172.21.15.55:0/3787205025 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3e40676b0 0x7fc3e4069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:01.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.809+0000 7fc405601640 1 --2- 172.21.15.55:0/3787205025 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3e40676b0 0x7fc3e4069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:01.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.809+0000 7fc405601640 1 --2- 172.21.15.55:0/3787205025 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3e40676b0 0x7fc3e4069b70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc0ad000 tx=0x7fc3fc004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:01.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.853+0000 7fc3f5ffb640 1 -- 172.21.15.55:0/3787205025 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fc3f8054070 con 0x7fc40014ec60 2024-03-19T01:57:01.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:01.873+0000 7fc3f5ffb640 1 -- 172.21.15.55:0/3787205025 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc3f8021020 con 0x7fc40014ec60 2024-03-19T01:57:02.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:02.085+0000 7fc407865640 1 -- 172.21.15.55:0/3787205025 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc40006ee60 con 0x7fc40014ec60 2024-03-19T01:57:02.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:02.101+0000 7fc3f5ffb640 1 -- 172.21.15.55:0/3787205025 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fc4000cc9f0 con 0x7fc40014ec60 2024-03-19T01:57:02.101 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:02.101 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:02.101 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:02.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:02.105+0000 7fc407865640 1 -- 172.21.15.55:0/3787205025 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3e40676b0 msgr2=0x7fc3e4069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:02.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:02.105+0000 7fc407865640 1 --2- 172.21.15.55:0/3787205025 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3e40676b0 0x7fc3e4069b70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc0ad000 tx=0x7fc3fc004740 comp rx=0 tx=0).stop 2024-03-19T01:57:02.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:02.105+0000 7fc407865640 1 -- 172.21.15.55:0/3787205025 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc40014ec60 msgr2=0x7fc4000c5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:02.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:02.105+0000 7fc407865640 1 --2- 172.21.15.55:0/3787205025 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc40014ec60 0x7fc4000c5510 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc3f800ca30 tx=0x7fc3f800cef0 comp rx=0 tx=0).stop 2024-03-19T01:57:02.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:02.105+0000 7fc407865640 1 -- 172.21.15.55:0/3787205025 shutdown_connections 2024-03-19T01:57:02.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:02.105+0000 7fc407865640 1 --2- 172.21.15.55:0/3787205025 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3e40676b0 0x7fc3e4069b70 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:02.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:02.105+0000 7fc407865640 1 --2- 172.21.15.55:0/3787205025 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc40014ec60 0x7fc4000c5510 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:02.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:02.105+0000 7fc407865640 1 --2- 172.21.15.55:0/3787205025 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc4000cd470 0x7fc400145f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:02.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:02.105+0000 7fc407865640 1 --2- 172.21.15.55:0/3787205025 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc4000cd090 0x7fc400159f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:02.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:02.105+0000 7fc407865640 1 -- 172.21.15.55:0/3787205025 >> 172.21.15.55:0/3787205025 conn(0x7fc40006e670 msgr2=0x7fc40014b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:02.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:02.105+0000 7fc407865640 1 -- 172.21.15.55:0/3787205025 shutdown_connections 2024-03-19T01:57:02.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:02.105+0000 7fc407865640 1 -- 172.21.15.55:0/3787205025 wait complete. 2024-03-19T01:57:02.723 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:02.724 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:03.724 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:03.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.845+0000 7f700d024640 1 Processor -- start 2024-03-19T01:57:03.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.845+0000 7f700d024640 1 -- start start 2024-03-19T01:57:03.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.845+0000 7f700d024640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7008152120 0x7f70080c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:03.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.845+0000 7f700d024640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7008070780 0x7f70081586e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:03.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.845+0000 7f700d024640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f700806fff0 con 0x7f70080c9610 2024-03-19T01:57:03.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.845+0000 7f700d024640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f7008152500 con 0x7f7008152120 2024-03-19T01:57:03.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.845+0000 7f700d024640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f7008070590 con 0x7f7008070780 2024-03-19T01:57:03.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.849+0000 7f7006d9d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7008152120 0x7f70080c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:03.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.849+0000 7f700759e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7008070780 0x7f70081586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:03.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.849+0000 7f700659c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f70080c9610 0x7f7008157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:60740/0 (socket says 172.21.15.55:60740) 2024-03-19T01:57:03.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.849+0000 7f700659c640 1 -- 172.21.15.55:0/61676055 learned_addr learned my addr 172.21.15.55:0/61676055 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:57:03.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.853+0000 7f7006d9d640 1 -- 172.21.15.55:0/61676055 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7008070780 msgr2=0x7f70081586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:03.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.853+0000 7f7006d9d640 1 --2- 172.21.15.55:0/61676055 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7008070780 0x7f70081586e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:03.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.853+0000 7f7006d9d640 1 -- 172.21.15.55:0/61676055 >> v1:172.21.15.55:6789/0 conn(0x7f70080c9610 legacy=0x7f7008157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:03.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.853+0000 7f7006d9d640 1 -- 172.21.15.55:0/61676055 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70080c39c0 con 0x7f7008152120 2024-03-19T01:57:03.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.857+0000 7f7006d9d640 1 --2- 172.21.15.55:0/61676055 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7008152120 0x7f70080c90d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6ff80097c0 tx=0x7f6ff801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26704b1812e3a7aa server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:03.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.861+0000 7f7005d9b640 1 -- 172.21.15.55:0/61676055 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff80b0070 con 0x7f7008152120 2024-03-19T01:57:03.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.861+0000 7f7005d9b640 1 -- 172.21.15.55:0/61676055 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ff80ab600 con 0x7f7008152120 2024-03-19T01:57:03.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.861+0000 7f7005d9b640 1 -- 172.21.15.55:0/61676055 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff80ab9d0 con 0x7f7008152120 2024-03-19T01:57:03.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.865+0000 7f700d024640 1 -- 172.21.15.55:0/61676055 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7008152120 msgr2=0x7f70080c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:03.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.865+0000 7f700d024640 1 --2- 172.21.15.55:0/61676055 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7008152120 0x7f70080c90d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6ff80097c0 tx=0x7f6ff801c4a0 comp rx=0 tx=0).stop 2024-03-19T01:57:03.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.865+0000 7f700d024640 1 -- 172.21.15.55:0/61676055 shutdown_connections 2024-03-19T01:57:03.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.865+0000 7f700d024640 1 --2- 172.21.15.55:0/61676055 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7008070780 0x7f70081586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:03.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.865+0000 7f700d024640 1 --2- 172.21.15.55:0/61676055 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7008152120 0x7f70080c90d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:03.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.865+0000 7f700d024640 1 -- 172.21.15.55:0/61676055 >> 172.21.15.55:0/61676055 conn(0x7f7008067600 msgr2=0x7f7008071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:03.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.865+0000 7f700d024640 1 -- 172.21.15.55:0/61676055 shutdown_connections 2024-03-19T01:57:03.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.865+0000 7f700d024640 1 -- 172.21.15.55:0/61676055 wait complete. 2024-03-19T01:57:03.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.865+0000 7f700d024640 1 Processor -- start 2024-03-19T01:57:03.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.865+0000 7f700d024640 1 -- start start 2024-03-19T01:57:03.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.865+0000 7f700d024640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7008070780 0x7f7008150730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:03.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.865+0000 7f700d024640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f70080c9610 0x7f7008141660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:03.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.865+0000 7f700d024640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7008152120 0x7f7008141ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:03.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.865+0000 7f700d024640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f70080c86d0 con 0x7f7008070780 2024-03-19T01:57:03.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.865+0000 7f700d024640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f70080c31c0 con 0x7f7008152120 2024-03-19T01:57:03.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.865+0000 7f700d024640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f7008165170 con 0x7f70080c9610 2024-03-19T01:57:03.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.865+0000 7f7006d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7008070780 0x7f7008150730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:03.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.869+0000 7f7006d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7008070780 0x7f7008150730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52716/0 (socket says 172.21.15.55:52716) 2024-03-19T01:57:03.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.869+0000 7f7006d9d640 1 -- 172.21.15.55:0/3456473056 learned_addr learned my addr 172.21.15.55:0/3456473056 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:03.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.869+0000 7f700759e640 1 --2- 172.21.15.55:0/3456473056 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7008152120 0x7f7008141ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:03.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.869+0000 7f700659c640 1 --2- 172.21.15.55:0/3456473056 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f70080c9610 0x7f7008141660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:03.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.873+0000 7f7006d9d640 1 -- 172.21.15.55:0/3456473056 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f70080c9610 msgr2=0x7f7008141660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:03.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.873+0000 7f7006d9d640 1 --2- 172.21.15.55:0/3456473056 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f70080c9610 0x7f7008141660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:03.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.873+0000 7f7006d9d640 1 -- 172.21.15.55:0/3456473056 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7008152120 msgr2=0x7f7008141ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:03.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.873+0000 7f7006d9d640 1 --2- 172.21.15.55:0/3456473056 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7008152120 0x7f7008141ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:03.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.873+0000 7f7006d9d640 1 -- 172.21.15.55:0/3456473056 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7008141180 con 0x7f7008070780 2024-03-19T01:57:03.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.873+0000 7f7006d9d640 1 --2- 172.21.15.55:0/3456473056 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7008070780 0x7f7008150730 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6ff801a0d0 tx=0x7f6ff8019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:03.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.877+0000 7f6fef7fe640 1 -- 172.21.15.55:0/3456473056 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff80b9020 con 0x7f7008070780 2024-03-19T01:57:03.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.877+0000 7f6fef7fe640 1 -- 172.21.15.55:0/3456473056 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ff80b5480 con 0x7f7008070780 2024-03-19T01:57:03.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.877+0000 7f700d024640 1 -- 172.21.15.55:0/3456473056 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7008142700 con 0x7f7008070780 2024-03-19T01:57:03.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.877+0000 7f6fef7fe640 1 -- 172.21.15.55:0/3456473056 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff80b0040 con 0x7f7008070780 2024-03-19T01:57:03.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.877+0000 7f700d024640 1 -- 172.21.15.55:0/3456473056 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7008142cc0 con 0x7f7008070780 2024-03-19T01:57:03.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:03.877+0000 7f700d024640 1 -- 172.21.15.55:0/3456473056 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7008068a40 con 0x7f7008070780 2024-03-19T01:57:04.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:04.177+0000 7f6fef7fe640 1 -- 172.21.15.55:0/3456473056 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f6ff80bc020 con 0x7f7008070780 2024-03-19T01:57:04.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:04.177+0000 7f6fef7fe640 1 --2- 172.21.15.55:0/3456473056 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6ff40676e0 0x7f6ff4069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:04.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:04.177+0000 7f700659c640 1 --2- 172.21.15.55:0/3456473056 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6ff40676e0 0x7f6ff4069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:04.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:04.177+0000 7f700659c640 1 --2- 172.21.15.55:0/3456473056 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6ff40676e0 0x7f6ff4069ba0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7008142410 tx=0x7f6ffc007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:04.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:04.217+0000 7f6fef7fe640 1 -- 172.21.15.55:0/3456473056 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f6ff80ee3e0 con 0x7f7008070780 2024-03-19T01:57:04.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:04.217+0000 7f6fef7fe640 1 -- 172.21.15.55:0/3456473056 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f6ff8016090 con 0x7f7008070780 2024-03-19T01:57:04.415 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:04.417+0000 7f700d024640 1 -- 172.21.15.55:0/3456473056 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f700814d020 con 0x7f7008070780 2024-03-19T01:57:04.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:04.433+0000 7f6fef7fe640 1 -- 172.21.15.55:0/3456473056 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f6ff80f0340 con 0x7f7008070780 2024-03-19T01:57:04.432 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:04.432 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:04.433 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:04.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:04.437+0000 7f700d024640 1 -- 172.21.15.55:0/3456473056 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6ff40676e0 msgr2=0x7f6ff4069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:04.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:04.437+0000 7f700d024640 1 --2- 172.21.15.55:0/3456473056 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6ff40676e0 0x7f6ff4069ba0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7008142410 tx=0x7f6ffc007e90 comp rx=0 tx=0).stop 2024-03-19T01:57:04.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:04.437+0000 7f700d024640 1 -- 172.21.15.55:0/3456473056 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7008070780 msgr2=0x7f7008150730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:04.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:04.437+0000 7f700d024640 1 --2- 172.21.15.55:0/3456473056 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7008070780 0x7f7008150730 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6ff801a0d0 tx=0x7f6ff8019240 comp rx=0 tx=0).stop 2024-03-19T01:57:04.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:04.437+0000 7f700d024640 1 -- 172.21.15.55:0/3456473056 shutdown_connections 2024-03-19T01:57:04.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:04.437+0000 7f700d024640 1 --2- 172.21.15.55:0/3456473056 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6ff40676e0 0x7f6ff4069ba0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:04.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:04.437+0000 7f700d024640 1 --2- 172.21.15.55:0/3456473056 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7008152120 0x7f7008141ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:04.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:04.437+0000 7f700d024640 1 --2- 172.21.15.55:0/3456473056 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f70080c9610 0x7f7008141660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:04.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:04.437+0000 7f700d024640 1 --2- 172.21.15.55:0/3456473056 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7008070780 0x7f7008150730 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:04.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:04.437+0000 7f700d024640 1 -- 172.21.15.55:0/3456473056 >> 172.21.15.55:0/3456473056 conn(0x7f7008067600 msgr2=0x7f70080c9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:04.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:04.437+0000 7f700d024640 1 -- 172.21.15.55:0/3456473056 shutdown_connections 2024-03-19T01:57:04.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:04.437+0000 7f700d024640 1 -- 172.21.15.55:0/3456473056 wait complete. 2024-03-19T01:57:05.055 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:05.056 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:06.056 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:06.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.173+0000 7ffb03128640 1 Processor -- start 2024-03-19T01:57:06.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.173+0000 7ffb03128640 1 -- start start 2024-03-19T01:57:06.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.173+0000 7ffb03128640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ffafc0c39c0 0x7ffafc158a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:06.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.173+0000 7ffb03128640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ffafc072c60 0x7ffafc15b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:06.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.173+0000 7ffb03128640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffafc152660 con 0x7ffafc06b9f0 2024-03-19T01:57:06.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.173+0000 7ffb03128640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7ffafc073130 con 0x7ffafc072c60 2024-03-19T01:57:06.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.173+0000 7ffb03128640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7ffafc0733f0 con 0x7ffafc0c39c0 2024-03-19T01:57:06.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.177+0000 7ffafbfff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ffafc072c60 0x7ffafc15b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:06.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.177+0000 7ffb00ec4640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ffafc0c39c0 0x7ffafc158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:06.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.177+0000 7ffafbfff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ffafc072c60 0x7ffafc15b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52730/0 (socket says 172.21.15.55:52730) 2024-03-19T01:57:06.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.177+0000 7ffafbfff640 1 -- 172.21.15.55:0/2853466110 learned_addr learned my addr 172.21.15.55:0/2853466110 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:06.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.181+0000 7ffafbfff640 1 -- 172.21.15.55:0/2853466110 >> v2:172.21.15.84:3300/0 conn(0x7ffafc0c39c0 msgr2=0x7ffafc158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:06.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.181+0000 7ffafbfff640 1 --2- 172.21.15.55:0/2853466110 >> v2:172.21.15.84:3300/0 conn(0x7ffafc0c39c0 0x7ffafc158a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:06.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.181+0000 7ffafbfff640 1 -- 172.21.15.55:0/2853466110 >> v1:172.21.15.84:6789/0 conn(0x7ffafc06b9f0 legacy=0x7ffafc0c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:06.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.181+0000 7ffafbfff640 1 -- 172.21.15.55:0/2853466110 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffafc0736d0 con 0x7ffafc072c60 2024-03-19T01:57:06.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.185+0000 7ffafbfff640 1 --2- 172.21.15.55:0/2853466110 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffafc072c60 0x7ffafc15b540 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ffafc0cd9b0 tx=0x7ffae801eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d080b236206b1c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:06.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.189+0000 7ffafb7fe640 1 -- 172.21.15.55:0/2853466110 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae80b5070 con 0x7ffafc072c60 2024-03-19T01:57:06.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.189+0000 7ffafb7fe640 1 -- 172.21.15.55:0/2853466110 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffae80a87f0 con 0x7ffafc072c60 2024-03-19T01:57:06.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.189+0000 7ffafb7fe640 1 -- 172.21.15.55:0/2853466110 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae80a8bc0 con 0x7ffafc072c60 2024-03-19T01:57:06.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.189+0000 7ffb03128640 1 -- 172.21.15.55:0/2853466110 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffafc072c60 msgr2=0x7ffafc15b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:06.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.189+0000 7ffb03128640 1 --2- 172.21.15.55:0/2853466110 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffafc072c60 0x7ffafc15b540 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ffafc0cd9b0 tx=0x7ffae801eaf0 comp rx=0 tx=0).stop 2024-03-19T01:57:06.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.189+0000 7ffb03128640 1 -- 172.21.15.55:0/2853466110 shutdown_connections 2024-03-19T01:57:06.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.189+0000 7ffb03128640 1 --2- 172.21.15.55:0/2853466110 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffafc072c60 0x7ffafc15b540 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:06.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.189+0000 7ffb03128640 1 --2- 172.21.15.55:0/2853466110 >> v2:172.21.15.84:3300/0 conn(0x7ffafc0c39c0 0x7ffafc158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:06.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.189+0000 7ffb03128640 1 -- 172.21.15.55:0/2853466110 >> 172.21.15.55:0/2853466110 conn(0x7ffafc0cc9c0 msgr2=0x7ffafc0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:06.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.189+0000 7ffb03128640 1 -- 172.21.15.55:0/2853466110 shutdown_connections 2024-03-19T01:57:06.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.189+0000 7ffb03128640 1 -- 172.21.15.55:0/2853466110 wait complete. 2024-03-19T01:57:06.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.193+0000 7ffb03128640 1 Processor -- start 2024-03-19T01:57:06.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.193+0000 7ffb03128640 1 -- start start 2024-03-19T01:57:06.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.193+0000 7ffb03128640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffafc06b9f0 0x7ffafc159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:06.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.193+0000 7ffb03128640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffafc072c60 0x7ffafc15b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:06.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.193+0000 7ffb03128640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffafc0c39c0 0x7ffafc15bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:06.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.193+0000 7ffb03128640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc157f50 con 0x7ffafc06b9f0 2024-03-19T01:57:06.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.193+0000 7ffb03128640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc158560 con 0x7ffafc0c39c0 2024-03-19T01:57:06.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.193+0000 7ffb03128640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ffafc0c67e0 con 0x7ffafc072c60 2024-03-19T01:57:06.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.193+0000 7ffb00ec4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffafc06b9f0 0x7ffafc159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:06.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.193+0000 7ffb016c5640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffafc0c39c0 0x7ffafc15bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:06.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.193+0000 7ffafbfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffafc072c60 0x7ffafc15b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:06.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.193+0000 7ffb00ec4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffafc06b9f0 0x7ffafc159370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52732/0 (socket says 172.21.15.55:52732) 2024-03-19T01:57:06.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.193+0000 7ffb00ec4640 1 -- 172.21.15.55:0/2378337245 learned_addr learned my addr 172.21.15.55:0/2378337245 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:06.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.197+0000 7ffb00ec4640 1 -- 172.21.15.55:0/2378337245 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffafc072c60 msgr2=0x7ffafc15b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:06.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.197+0000 7ffb00ec4640 1 --2- 172.21.15.55:0/2378337245 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffafc072c60 0x7ffafc15b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:06.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.197+0000 7ffb00ec4640 1 -- 172.21.15.55:0/2378337245 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffafc0c39c0 msgr2=0x7ffafc15bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:06.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.197+0000 7ffb00ec4640 1 --2- 172.21.15.55:0/2378337245 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffafc0c39c0 0x7ffafc15bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:06.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.197+0000 7ffb00ec4640 1 -- 172.21.15.55:0/2378337245 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffafc150c00 con 0x7ffafc06b9f0 2024-03-19T01:57:06.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.201+0000 7ffb00ec4640 1 --2- 172.21.15.55:0/2378337245 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffafc06b9f0 0x7ffafc159370 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ffaf000e870 tx=0x7ffaf000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:06.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.205+0000 7ffaf97fa640 1 -- 172.21.15.55:0/2378337245 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0017070 con 0x7ffafc06b9f0 2024-03-19T01:57:06.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.205+0000 7ffaf97fa640 1 -- 172.21.15.55:0/2378337245 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaf000c940 con 0x7ffafc06b9f0 2024-03-19T01:57:06.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.205+0000 7ffaf97fa640 1 -- 172.21.15.55:0/2378337245 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf000ccf0 con 0x7ffafc06b9f0 2024-03-19T01:57:06.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.205+0000 7ffb03128640 1 -- 172.21.15.55:0/2378337245 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffafc1513b0 con 0x7ffafc06b9f0 2024-03-19T01:57:06.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.205+0000 7ffb03128640 1 -- 172.21.15.55:0/2378337245 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffafc150920 con 0x7ffafc06b9f0 2024-03-19T01:57:06.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.205+0000 7ffb03128640 1 -- 172.21.15.55:0/2378337245 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffafc068a40 con 0x7ffafc06b9f0 2024-03-19T01:57:06.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.485+0000 7ffaf97fa640 1 -- 172.21.15.55:0/2378337245 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ffaf001a070 con 0x7ffafc06b9f0 2024-03-19T01:57:06.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.485+0000 7ffaf97fa640 1 --2- 172.21.15.55:0/2378337245 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffae00676e0 0x7ffae0069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:06.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.485+0000 7ffafbfff640 1 --2- 172.21.15.55:0/2378337245 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffae00676e0 0x7ffae0069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:06.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.485+0000 7ffafbfff640 1 --2- 172.21.15.55:0/2378337245 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffae00676e0 0x7ffae0069ba0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ffae80b2000 tx=0x7ffae80aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:06.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.501+0000 7ffaf97fa640 1 -- 172.21.15.55:0/2378337245 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7ffaf00544a0 con 0x7ffafc06b9f0 2024-03-19T01:57:06.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.501+0000 7ffaf97fa640 1 -- 172.21.15.55:0/2378337245 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ffaf0056400 con 0x7ffafc06b9f0 2024-03-19T01:57:06.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.745+0000 7ffb03128640 1 -- 172.21.15.55:0/2378337245 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffafc0ca2b0 con 0x7ffafc06b9f0 2024-03-19T01:57:06.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.765+0000 7ffaf97fa640 1 -- 172.21.15.55:0/2378337245 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7ffaf005a4a0 con 0x7ffafc06b9f0 2024-03-19T01:57:06.763 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:06.764 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:06.764 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:06.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.769+0000 7ffb03128640 1 -- 172.21.15.55:0/2378337245 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffae00676e0 msgr2=0x7ffae0069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:06.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.769+0000 7ffb03128640 1 --2- 172.21.15.55:0/2378337245 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffae00676e0 0x7ffae0069ba0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ffae80b2000 tx=0x7ffae80aa070 comp rx=0 tx=0).stop 2024-03-19T01:57:06.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.769+0000 7ffb03128640 1 -- 172.21.15.55:0/2378337245 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffafc06b9f0 msgr2=0x7ffafc159370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:06.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.769+0000 7ffb03128640 1 --2- 172.21.15.55:0/2378337245 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffafc06b9f0 0x7ffafc159370 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ffaf000e870 tx=0x7ffaf000ec70 comp rx=0 tx=0).stop 2024-03-19T01:57:06.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.769+0000 7ffb03128640 1 -- 172.21.15.55:0/2378337245 shutdown_connections 2024-03-19T01:57:06.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.769+0000 7ffb03128640 1 --2- 172.21.15.55:0/2378337245 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffae00676e0 0x7ffae0069ba0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:06.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.769+0000 7ffb03128640 1 --2- 172.21.15.55:0/2378337245 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffafc0c39c0 0x7ffafc15bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:06.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.769+0000 7ffb03128640 1 --2- 172.21.15.55:0/2378337245 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffafc072c60 0x7ffafc15b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:06.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.769+0000 7ffb03128640 1 --2- 172.21.15.55:0/2378337245 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffafc06b9f0 0x7ffafc159370 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:06.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.769+0000 7ffb03128640 1 -- 172.21.15.55:0/2378337245 >> 172.21.15.55:0/2378337245 conn(0x7ffafc0cc9c0 msgr2=0x7ffafc155dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:06.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.769+0000 7ffb03128640 1 -- 172.21.15.55:0/2378337245 shutdown_connections 2024-03-19T01:57:06.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:06.769+0000 7ffb03128640 1 -- 172.21.15.55:0/2378337245 wait complete. 2024-03-19T01:57:07.388 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:07.388 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:08.389 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:08.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.509+0000 7f6490ce7640 1 Processor -- start 2024-03-19T01:57:08.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.509+0000 7f6490ce7640 1 -- start start 2024-03-19T01:57:08.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.509+0000 7f6490ce7640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f648c1520b0 0x7f648c0709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:08.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.509+0000 7f6490ce7640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f648c0c9000 0x7f648c0c9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:08.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.509+0000 7f6490ce7640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f648c158d50 con 0x7f648c0c2ad0 2024-03-19T01:57:08.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.509+0000 7f6490ce7640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f648c15c4e0 con 0x7f648c1520b0 2024-03-19T01:57:08.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.509+0000 7f6490ce7640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f648c15c050 con 0x7f648c0c9000 2024-03-19T01:57:08.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.509+0000 7f648ad9d640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f648c0c2ad0 0x7f648c0c2f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:60750/0 (socket says 172.21.15.55:60750) 2024-03-19T01:57:08.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.509+0000 7f648ad9d640 1 -- 172.21.15.55:0/1824662259 learned_addr learned my addr 172.21.15.55:0/1824662259 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:57:08.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.509+0000 7f6489d9b640 1 --2- 172.21.15.55:0/1824662259 >> v2:172.21.15.84:3300/0 conn(0x7f648c0c9000 0x7f648c0c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:08.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.513+0000 7f648a59c640 1 --2- 172.21.15.55:0/1824662259 >> v2:172.21.15.55:3300/0 conn(0x7f648c1520b0 0x7f648c0709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:08.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.513+0000 7f648959a640 1 -- 172.21.15.55:0/1824662259 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1464975644 0 0) 0x7f648c158d50 con 0x7f648c0c2ad0 2024-03-19T01:57:08.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.513+0000 7f648959a640 1 -- 172.21.15.55:0/1824662259 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6474003530 con 0x7f648c0c2ad0 2024-03-19T01:57:08.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.517+0000 7f648a59c640 1 -- 172.21.15.55:0/1824662259 >> v2:172.21.15.84:3300/0 conn(0x7f648c0c9000 msgr2=0x7f648c0c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:08.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.517+0000 7f648a59c640 1 --2- 172.21.15.55:0/1824662259 >> v2:172.21.15.84:3300/0 conn(0x7f648c0c9000 0x7f648c0c9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:08.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.517+0000 7f648a59c640 1 -- 172.21.15.55:0/1824662259 >> v1:172.21.15.55:6789/0 conn(0x7f648c0c2ad0 legacy=0x7f648c0c2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:08.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.517+0000 7f648a59c640 1 -- 172.21.15.55:0/1824662259 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648c15b720 con 0x7f648c1520b0 2024-03-19T01:57:08.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.525+0000 7f648a59c640 1 --2- 172.21.15.55:0/1824662259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f648c1520b0 0x7f648c0709a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f648c072580 tx=0x7f647c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fe89e8a425f15eb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:08.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.533+0000 7f648959a640 1 -- 172.21.15.55:0/1824662259 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c0b0070 con 0x7f648c1520b0 2024-03-19T01:57:08.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.533+0000 7f648959a640 1 -- 172.21.15.55:0/1824662259 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f647c0ab600 con 0x7f648c1520b0 2024-03-19T01:57:08.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.533+0000 7f648959a640 1 -- 172.21.15.55:0/1824662259 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c0ab9d0 con 0x7f648c1520b0 2024-03-19T01:57:08.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.533+0000 7f6490ce7640 1 -- 172.21.15.55:0/1824662259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f648c1520b0 msgr2=0x7f648c0709a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:08.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.533+0000 7f6490ce7640 1 --2- 172.21.15.55:0/1824662259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f648c1520b0 0x7f648c0709a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f648c072580 tx=0x7f647c01c480 comp rx=0 tx=0).stop 2024-03-19T01:57:08.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.533+0000 7f6490ce7640 1 -- 172.21.15.55:0/1824662259 shutdown_connections 2024-03-19T01:57:08.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.533+0000 7f6490ce7640 1 --2- 172.21.15.55:0/1824662259 >> v2:172.21.15.84:3300/0 conn(0x7f648c0c9000 0x7f648c0c9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:08.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.533+0000 7f6490ce7640 1 --2- 172.21.15.55:0/1824662259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f648c1520b0 0x7f648c0709a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:08.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.533+0000 7f6490ce7640 1 -- 172.21.15.55:0/1824662259 >> 172.21.15.55:0/1824662259 conn(0x7f648c067600 msgr2=0x7f648c071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:08.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.533+0000 7f6490ce7640 1 -- 172.21.15.55:0/1824662259 shutdown_connections 2024-03-19T01:57:08.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.533+0000 7f6490ce7640 1 -- 172.21.15.55:0/1824662259 wait complete. 2024-03-19T01:57:08.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.533+0000 7f6490ce7640 1 Processor -- start 2024-03-19T01:57:08.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.533+0000 7f6490ce7640 1 -- start start 2024-03-19T01:57:08.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.533+0000 7f6490ce7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f648c0c2ad0 0x7f648c14e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:08.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.533+0000 7f6490ce7640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f648c0c9000 0x7f648c157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:08.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.533+0000 7f6490ce7640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f648c1520b0 0x7f648c157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:08.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.533+0000 7f6490ce7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f648c152610 con 0x7f648c0c2ad0 2024-03-19T01:57:08.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.533+0000 7f6490ce7640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f648c152490 con 0x7f648c1520b0 2024-03-19T01:57:08.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.533+0000 7f6490ce7640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f648c070ee0 con 0x7f648c0c9000 2024-03-19T01:57:08.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.537+0000 7f648a59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f648c0c2ad0 0x7f648c14e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:08.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.537+0000 7f648a59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f648c0c2ad0 0x7f648c14e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52750/0 (socket says 172.21.15.55:52750) 2024-03-19T01:57:08.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.537+0000 7f648a59c640 1 -- 172.21.15.55:0/1808173081 learned_addr learned my addr 172.21.15.55:0/1808173081 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:08.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.537+0000 7f648ad9d640 1 --2- 172.21.15.55:0/1808173081 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f648c1520b0 0x7f648c157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:08.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.537+0000 7f6489d9b640 1 --2- 172.21.15.55:0/1808173081 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f648c0c9000 0x7f648c157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:08.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.541+0000 7f648a59c640 1 -- 172.21.15.55:0/1808173081 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f648c0c9000 msgr2=0x7f648c157200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:08.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.541+0000 7f648a59c640 1 --2- 172.21.15.55:0/1808173081 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f648c0c9000 0x7f648c157200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:08.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.541+0000 7f648a59c640 1 -- 172.21.15.55:0/1808173081 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f648c1520b0 msgr2=0x7f648c157740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:08.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.541+0000 7f648a59c640 1 --2- 172.21.15.55:0/1808173081 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f648c1520b0 0x7f648c157740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:08.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.541+0000 7f648a59c640 1 -- 172.21.15.55:0/1808173081 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648c1527f0 con 0x7f648c0c2ad0 2024-03-19T01:57:08.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.541+0000 7f648a59c640 1 --2- 172.21.15.55:0/1808173081 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f648c0c2ad0 0x7f648c14e190 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f647c019090 tx=0x7f647c019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:08.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.545+0000 7f647affd640 1 -- 172.21.15.55:0/1808173081 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c0b0040 con 0x7f648c0c2ad0 2024-03-19T01:57:08.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.545+0000 7f647affd640 1 -- 172.21.15.55:0/1808173081 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f647c0ba070 con 0x7f648c0c2ad0 2024-03-19T01:57:08.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.545+0000 7f6490ce7640 1 -- 172.21.15.55:0/1808173081 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f648c14c660 con 0x7f648c0c2ad0 2024-03-19T01:57:08.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.545+0000 7f647affd640 1 -- 172.21.15.55:0/1808173081 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c0b4800 con 0x7f648c0c2ad0 2024-03-19T01:57:08.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.549+0000 7f6490ce7640 1 -- 172.21.15.55:0/1808173081 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f648c152a90 con 0x7f648c0c2ad0 2024-03-19T01:57:08.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.549+0000 7f6490ce7640 1 -- 172.21.15.55:0/1808173081 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f648c068a40 con 0x7f648c0c2ad0 2024-03-19T01:57:08.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.569+0000 7f647affd640 1 -- 172.21.15.55:0/1808173081 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f647c0b3070 con 0x7f648c0c2ad0 2024-03-19T01:57:08.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.569+0000 7f647affd640 1 --2- 172.21.15.55:0/1808173081 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f646c0676b0 0x7f646c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:08.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.573+0000 7f6489d9b640 1 --2- 172.21.15.55:0/1808173081 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f646c0676b0 0x7f646c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:08.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.573+0000 7f6489d9b640 1 --2- 172.21.15.55:0/1808173081 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f646c0676b0 0x7f646c069b70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f648c15a250 tx=0x7f6480009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:08.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.865+0000 7f647affd640 1 -- 172.21.15.55:0/1808173081 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f647c0ee5a0 con 0x7f648c0c2ad0 2024-03-19T01:57:08.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:08.865+0000 7f647affd640 1 -- 172.21.15.55:0/1808173081 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f647c0c4020 con 0x7f648c0c2ad0 2024-03-19T01:57:09.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:09.069+0000 7f6490ce7640 1 -- 172.21.15.55:0/1808173081 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f648c0c9410 con 0x7f648c0c2ad0 2024-03-19T01:57:09.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:09.093+0000 7f647affd640 1 -- 172.21.15.55:0/1808173081 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f647c016070 con 0x7f648c0c2ad0 2024-03-19T01:57:09.095 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:09.095 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:09.095 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:09.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:09.097+0000 7f6490ce7640 1 -- 172.21.15.55:0/1808173081 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f646c0676b0 msgr2=0x7f646c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:09.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:09.097+0000 7f6490ce7640 1 --2- 172.21.15.55:0/1808173081 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f646c0676b0 0x7f646c069b70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f648c15a250 tx=0x7f6480009190 comp rx=0 tx=0).stop 2024-03-19T01:57:09.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:09.097+0000 7f6490ce7640 1 -- 172.21.15.55:0/1808173081 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f648c0c2ad0 msgr2=0x7f648c14e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:09.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:09.097+0000 7f6490ce7640 1 --2- 172.21.15.55:0/1808173081 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f648c0c2ad0 0x7f648c14e190 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f647c019090 tx=0x7f647c019220 comp rx=0 tx=0).stop 2024-03-19T01:57:09.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:09.097+0000 7f6490ce7640 1 -- 172.21.15.55:0/1808173081 shutdown_connections 2024-03-19T01:57:09.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:09.097+0000 7f6490ce7640 1 --2- 172.21.15.55:0/1808173081 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f646c0676b0 0x7f646c069b70 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:09.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:09.097+0000 7f6490ce7640 1 --2- 172.21.15.55:0/1808173081 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f648c1520b0 0x7f648c157740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:09.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:09.097+0000 7f6490ce7640 1 --2- 172.21.15.55:0/1808173081 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f648c0c9000 0x7f648c157200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:09.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:09.097+0000 7f6490ce7640 1 --2- 172.21.15.55:0/1808173081 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f648c0c2ad0 0x7f648c14e190 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:09.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:09.097+0000 7f6490ce7640 1 -- 172.21.15.55:0/1808173081 >> 172.21.15.55:0/1808173081 conn(0x7f648c067600 msgr2=0x7f648c0ca2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:09.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:09.097+0000 7f6490ce7640 1 -- 172.21.15.55:0/1808173081 shutdown_connections 2024-03-19T01:57:09.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:09.097+0000 7f6490ce7640 1 -- 172.21.15.55:0/1808173081 wait complete. 2024-03-19T01:57:09.373 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:57:09.375 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:57:09.719 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:09.719 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:10.721 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:10.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.833+0000 7fc372f18640 1 Processor -- start 2024-03-19T01:57:10.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.833+0000 7fc372f18640 1 -- start start 2024-03-19T01:57:10.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.833+0000 7fc372f18640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc36c154460 0x7fc36c06b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:10.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.833+0000 7fc372f18640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc36c0c2760 0x7fc36c0c2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:10.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.833+0000 7fc372f18640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc36c0c22b0 con 0x7fc36c073540 2024-03-19T01:57:10.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.833+0000 7fc372f18640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fc36c15a230 con 0x7fc36c0c2760 2024-03-19T01:57:10.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.833+0000 7fc372f18640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc36c0c3100 con 0x7fc36c154460 2024-03-19T01:57:10.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.837+0000 7fc3714b5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc36c0c2760 0x7fc36c0c2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:10.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.837+0000 7fc36bfff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fc36c073540 0x7fc36c0cced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:51076/0 (socket says 172.21.15.55:51076) 2024-03-19T01:57:10.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.837+0000 7fc36bfff640 1 -- 172.21.15.55:0/888911786 learned_addr learned my addr 172.21.15.55:0/888911786 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:57:10.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.837+0000 7fc370cb4640 1 --2- 172.21.15.55:0/888911786 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc36c154460 0x7fc36c06b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:10.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.841+0000 7fc3714b5640 1 -- 172.21.15.55:0/888911786 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc36c154460 msgr2=0x7fc36c06b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:10.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.841+0000 7fc3714b5640 1 --2- 172.21.15.55:0/888911786 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc36c154460 0x7fc36c06b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:10.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.841+0000 7fc3714b5640 1 -- 172.21.15.55:0/888911786 >> v1:172.21.15.84:6789/0 conn(0x7fc36c073540 legacy=0x7fc36c0cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:10.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.841+0000 7fc3714b5640 1 -- 172.21.15.55:0/888911786 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc36c15a490 con 0x7fc36c0c2760 2024-03-19T01:57:10.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.845+0000 7fc3714b5640 1 --2- 172.21.15.55:0/888911786 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc36c0c2760 0x7fc36c0c2bc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc364009d90 tx=0x7fc36401f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37be880e9cb3bde server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:10.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.849+0000 7fc36b7fe640 1 -- 172.21.15.55:0/888911786 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3640b4070 con 0x7fc36c0c2760 2024-03-19T01:57:10.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.849+0000 7fc36b7fe640 1 -- 172.21.15.55:0/888911786 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3640af600 con 0x7fc36c0c2760 2024-03-19T01:57:10.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.849+0000 7fc36b7fe640 1 -- 172.21.15.55:0/888911786 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3640af9d0 con 0x7fc36c0c2760 2024-03-19T01:57:10.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.849+0000 7fc372f18640 1 -- 172.21.15.55:0/888911786 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc36c0c2760 msgr2=0x7fc36c0c2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:10.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.849+0000 7fc372f18640 1 --2- 172.21.15.55:0/888911786 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc36c0c2760 0x7fc36c0c2bc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc364009d90 tx=0x7fc36401f150 comp rx=0 tx=0).stop 2024-03-19T01:57:10.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.849+0000 7fc372f18640 1 -- 172.21.15.55:0/888911786 shutdown_connections 2024-03-19T01:57:10.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.849+0000 7fc372f18640 1 --2- 172.21.15.55:0/888911786 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc36c0c2760 0x7fc36c0c2bc0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:10.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.849+0000 7fc372f18640 1 --2- 172.21.15.55:0/888911786 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc36c154460 0x7fc36c06b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:10.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.849+0000 7fc372f18640 1 -- 172.21.15.55:0/888911786 >> 172.21.15.55:0/888911786 conn(0x7fc36c06e670 msgr2=0x7fc36c06ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:10.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.849+0000 7fc372f18640 1 -- 172.21.15.55:0/888911786 shutdown_connections 2024-03-19T01:57:10.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.849+0000 7fc372f18640 1 -- 172.21.15.55:0/888911786 wait complete. 2024-03-19T01:57:10.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.853+0000 7fc372f18640 1 Processor -- start 2024-03-19T01:57:10.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.853+0000 7fc372f18640 1 -- start start 2024-03-19T01:57:10.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.853+0000 7fc372f18640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc36c073540 0x7fc36c149fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:10.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.853+0000 7fc372f18640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc36c0c2760 0x7fc36c15cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:10.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.853+0000 7fc372f18640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc36c154460 0x7fc36c1f0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:10.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.853+0000 7fc372f18640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c15ad30 con 0x7fc36c073540 2024-03-19T01:57:10.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.853+0000 7fc372f18640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c15aa50 con 0x7fc36c154460 2024-03-19T01:57:10.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.853+0000 7fc372f18640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc36c15b010 con 0x7fc36c0c2760 2024-03-19T01:57:10.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.853+0000 7fc370cb4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc36c073540 0x7fc36c149fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:10.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.853+0000 7fc370cb4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc36c073540 0x7fc36c149fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:45100/0 (socket says 172.21.15.55:45100) 2024-03-19T01:57:10.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.853+0000 7fc370cb4640 1 -- 172.21.15.55:0/1948710862 learned_addr learned my addr 172.21.15.55:0/1948710862 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:10.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.853+0000 7fc36bfff640 1 --2- 172.21.15.55:0/1948710862 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc36c0c2760 0x7fc36c15cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:10.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.853+0000 7fc3714b5640 1 --2- 172.21.15.55:0/1948710862 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc36c154460 0x7fc36c1f0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:10.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.857+0000 7fc370cb4640 1 -- 172.21.15.55:0/1948710862 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc36c0c2760 msgr2=0x7fc36c15cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:10.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.857+0000 7fc370cb4640 1 --2- 172.21.15.55:0/1948710862 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc36c0c2760 0x7fc36c15cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:10.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.857+0000 7fc370cb4640 1 -- 172.21.15.55:0/1948710862 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc36c154460 msgr2=0x7fc36c1f0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:10.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.857+0000 7fc370cb4640 1 --2- 172.21.15.55:0/1948710862 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc36c154460 0x7fc36c1f0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:10.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.857+0000 7fc370cb4640 1 -- 172.21.15.55:0/1948710862 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc36c15c5a0 con 0x7fc36c073540 2024-03-19T01:57:10.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.861+0000 7fc370cb4640 1 --2- 172.21.15.55:0/1948710862 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc36c073540 0x7fc36c149fc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc36000e8a0 tx=0x7fc36000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:10.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.865+0000 7fc3697fa640 1 -- 172.21.15.55:0/1948710862 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc360017070 con 0x7fc36c073540 2024-03-19T01:57:10.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.865+0000 7fc3697fa640 1 -- 172.21.15.55:0/1948710862 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc36000c940 con 0x7fc36c073540 2024-03-19T01:57:10.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.865+0000 7fc3697fa640 1 -- 172.21.15.55:0/1948710862 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc36000ccf0 con 0x7fc36c073540 2024-03-19T01:57:10.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.865+0000 7fc372f18640 1 -- 172.21.15.55:0/1948710862 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc36c14f160 con 0x7fc36c073540 2024-03-19T01:57:10.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.865+0000 7fc372f18640 1 -- 172.21.15.55:0/1948710862 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc36c15bfa0 con 0x7fc36c073540 2024-03-19T01:57:10.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.865+0000 7fc372f18640 1 -- 172.21.15.55:0/1948710862 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc36c068a40 con 0x7fc36c073540 2024-03-19T01:57:10.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.901+0000 7fc3697fa640 1 -- 172.21.15.55:0/1948710862 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc36001a070 con 0x7fc36c073540 2024-03-19T01:57:10.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.901+0000 7fc3697fa640 1 --2- 172.21.15.55:0/1948710862 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3500676b0 0x7fc350069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:10.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.901+0000 7fc36bfff640 1 --2- 172.21.15.55:0/1948710862 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3500676b0 0x7fc350069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:10.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.901+0000 7fc36bfff640 1 --2- 172.21.15.55:0/1948710862 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3500676b0 0x7fc350069b70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc36c1edf70 tx=0x7fc358007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:10.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:10.909+0000 7fc3697fa640 1 -- 172.21.15.55:0/1948710862 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fc360054590 con 0x7fc36c073540 2024-03-19T01:57:11.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:11.177+0000 7fc3697fa640 1 -- 172.21.15.55:0/1948710862 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc3600564f0 con 0x7fc36c073540 2024-03-19T01:57:11.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:11.373+0000 7fc372f18640 1 -- 172.21.15.55:0/1948710862 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc36c0c9e90 con 0x7fc36c073540 2024-03-19T01:57:11.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:11.389+0000 7fc3697fa640 1 -- 172.21.15.55:0/1948710862 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fc36005a590 con 0x7fc36c073540 2024-03-19T01:57:11.389 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:11.390 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:11.390 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:11.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:11.393+0000 7fc372f18640 1 -- 172.21.15.55:0/1948710862 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3500676b0 msgr2=0x7fc350069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:11.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:11.393+0000 7fc372f18640 1 --2- 172.21.15.55:0/1948710862 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3500676b0 0x7fc350069b70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc36c1edf70 tx=0x7fc358007e90 comp rx=0 tx=0).stop 2024-03-19T01:57:11.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:11.393+0000 7fc372f18640 1 -- 172.21.15.55:0/1948710862 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc36c073540 msgr2=0x7fc36c149fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:11.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:11.393+0000 7fc372f18640 1 --2- 172.21.15.55:0/1948710862 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc36c073540 0x7fc36c149fc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc36000e8a0 tx=0x7fc36000eca0 comp rx=0 tx=0).stop 2024-03-19T01:57:11.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:11.393+0000 7fc372f18640 1 -- 172.21.15.55:0/1948710862 shutdown_connections 2024-03-19T01:57:11.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:11.393+0000 7fc372f18640 1 --2- 172.21.15.55:0/1948710862 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3500676b0 0x7fc350069b70 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:11.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:11.393+0000 7fc372f18640 1 --2- 172.21.15.55:0/1948710862 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc36c154460 0x7fc36c1f0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:11.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:11.393+0000 7fc372f18640 1 --2- 172.21.15.55:0/1948710862 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc36c0c2760 0x7fc36c15cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:11.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:11.393+0000 7fc372f18640 1 --2- 172.21.15.55:0/1948710862 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc36c073540 0x7fc36c149fc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:11.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:11.393+0000 7fc372f18640 1 -- 172.21.15.55:0/1948710862 >> 172.21.15.55:0/1948710862 conn(0x7fc36c06e670 msgr2=0x7fc36c0cceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:11.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:11.393+0000 7fc372f18640 1 -- 172.21.15.55:0/1948710862 shutdown_connections 2024-03-19T01:57:11.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:11.393+0000 7fc372f18640 1 -- 172.21.15.55:0/1948710862 wait complete. 2024-03-19T01:57:12.014 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:12.014 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:13.015 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:13.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.145+0000 7f3d2a9b9640 1 Processor -- start 2024-03-19T01:57:13.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.145+0000 7f3d2a9b9640 1 -- start start 2024-03-19T01:57:13.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.145+0000 7f3d2a9b9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3d2415acb0 0x7f3d24073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:13.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.145+0000 7f3d2a9b9640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d240c7fa0 con 0x7f3d241544f0 2024-03-19T01:57:13.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.145+0000 7f3d2a9b9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d240c8240 con 0x7f3d2406b2a0 2024-03-19T01:57:13.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.145+0000 7f3d2a9b9640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3d2415b210 con 0x7f3d2415acb0 2024-03-19T01:57:13.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.145+0000 7f3d23fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f3d241544f0 0x7f3d2415e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:46114/0 (socket says 172.21.15.55:46114) 2024-03-19T01:57:13.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.145+0000 7f3d23fff640 1 -- 172.21.15.55:0/1656593008 learned_addr learned my addr 172.21.15.55:0/1656593008 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:57:13.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.145+0000 7f3d28f56640 1 --2- 172.21.15.55:0/1656593008 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3d2415acb0 0x7f3d24073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:13.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.149+0000 7f3d22ffd640 1 -- 172.21.15.55:0/1656593008 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2158929648 0 0) 0x7f3d240c7fa0 con 0x7f3d241544f0 2024-03-19T01:57:13.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.149+0000 7f3d22ffd640 1 -- 172.21.15.55:0/1656593008 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d0c003530 con 0x7f3d241544f0 2024-03-19T01:57:13.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.149+0000 7f3d22ffd640 1 -- 172.21.15.55:0/1656593008 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2647592830 0 0) 0x7f3d240c8240 con 0x7f3d2406b2a0 2024-03-19T01:57:13.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.149+0000 7f3d22ffd640 1 -- 172.21.15.55:0/1656593008 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d240c7fa0 con 0x7f3d2406b2a0 2024-03-19T01:57:13.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.153+0000 7f3d28f56640 1 -- 172.21.15.55:0/1656593008 >> v1:172.21.15.84:6789/0 conn(0x7f3d2406b2a0 legacy=0x7f3d240cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:13.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.153+0000 7f3d28f56640 1 -- 172.21.15.55:0/1656593008 >> v1:172.21.15.55:6789/0 conn(0x7f3d241544f0 legacy=0x7f3d2415e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:13.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.153+0000 7f3d28f56640 1 -- 172.21.15.55:0/1656593008 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2415d920 con 0x7f3d2415acb0 2024-03-19T01:57:13.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.157+0000 7f3d28f56640 1 --2- 172.21.15.55:0/1656593008 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3d2415acb0 0x7f3d24073540 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009dc0 tx=0x7f3d1c01f170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bb55ff372880705 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:13.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.165+0000 7f3d22ffd640 1 -- 172.21.15.55:0/1656593008 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c0b4070 con 0x7f3d2415acb0 2024-03-19T01:57:13.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.165+0000 7f3d22ffd640 1 -- 172.21.15.55:0/1656593008 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d1c0af600 con 0x7f3d2415acb0 2024-03-19T01:57:13.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.165+0000 7f3d22ffd640 1 -- 172.21.15.55:0/1656593008 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c0af9d0 con 0x7f3d2415acb0 2024-03-19T01:57:13.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.169+0000 7f3d2a9b9640 1 -- 172.21.15.55:0/1656593008 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3d2415acb0 msgr2=0x7f3d24073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:13.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.169+0000 7f3d2a9b9640 1 --2- 172.21.15.55:0/1656593008 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3d2415acb0 0x7f3d24073540 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009dc0 tx=0x7f3d1c01f170 comp rx=0 tx=0).stop 2024-03-19T01:57:13.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.169+0000 7f3d2a9b9640 1 -- 172.21.15.55:0/1656593008 shutdown_connections 2024-03-19T01:57:13.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.169+0000 7f3d2a9b9640 1 --2- 172.21.15.55:0/1656593008 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3d2415acb0 0x7f3d24073540 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:13.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.169+0000 7f3d2a9b9640 1 -- 172.21.15.55:0/1656593008 >> 172.21.15.55:0/1656593008 conn(0x7f3d2406e670 msgr2=0x7f3d2406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:13.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.169+0000 7f3d2a9b9640 1 -- 172.21.15.55:0/1656593008 shutdown_connections 2024-03-19T01:57:13.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.169+0000 7f3d2a9b9640 1 -- 172.21.15.55:0/1656593008 wait complete. 2024-03-19T01:57:13.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.169+0000 7f3d2a9b9640 1 Processor -- start 2024-03-19T01:57:13.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.169+0000 7f3d2a9b9640 1 -- start start 2024-03-19T01:57:13.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.169+0000 7f3d2a9b9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3d2406b2a0 0x7f3d24070660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:13.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.169+0000 7f3d2a9b9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3d241544f0 0x7f3d24151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:13.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.169+0000 7f3d2a9b9640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d2415acb0 0x7f3d241522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:13.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.169+0000 7f3d2a9b9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3d240cd540 con 0x7f3d241544f0 2024-03-19T01:57:13.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.169+0000 7f3d2a9b9640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3d241548d0 con 0x7f3d2415acb0 2024-03-19T01:57:13.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.169+0000 7f3d2a9b9640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3d240cd6c0 con 0x7f3d2406b2a0 2024-03-19T01:57:13.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.169+0000 7f3d28f56640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d2415acb0 0x7f3d241522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:13.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.169+0000 7f3d237fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3d241544f0 0x7f3d24151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:13.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.169+0000 7f3d237fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3d241544f0 0x7f3d24151d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:45112/0 (socket says 172.21.15.55:45112) 2024-03-19T01:57:13.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.169+0000 7f3d237fe640 1 -- 172.21.15.55:0/2671943649 learned_addr learned my addr 172.21.15.55:0/2671943649 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:13.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.169+0000 7f3d23fff640 1 --2- 172.21.15.55:0/2671943649 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3d2406b2a0 0x7f3d24070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:13.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.173+0000 7f3d237fe640 1 -- 172.21.15.55:0/2671943649 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3d2406b2a0 msgr2=0x7f3d24070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:13.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.173+0000 7f3d237fe640 1 --2- 172.21.15.55:0/2671943649 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3d2406b2a0 0x7f3d24070660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:13.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.173+0000 7f3d237fe640 1 -- 172.21.15.55:0/2671943649 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d2415acb0 msgr2=0x7f3d241522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:13.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.173+0000 7f3d237fe640 1 --2- 172.21.15.55:0/2671943649 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d2415acb0 0x7f3d241522b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:13.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.173+0000 7f3d237fe640 1 -- 172.21.15.55:0/2671943649 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2415b7d0 con 0x7f3d241544f0 2024-03-19T01:57:13.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.177+0000 7f3d237fe640 1 --2- 172.21.15.55:0/2671943649 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3d241544f0 0x7f3d24151d70 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800c430 tx=0x7f3d1800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:13.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.181+0000 7f3d20ff9640 1 -- 172.21.15.55:0/2671943649 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d18017070 con 0x7f3d241544f0 2024-03-19T01:57:13.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.181+0000 7f3d20ff9640 1 -- 172.21.15.55:0/2671943649 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d180052f0 con 0x7f3d241544f0 2024-03-19T01:57:13.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.181+0000 7f3d20ff9640 1 -- 172.21.15.55:0/2671943649 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d18003840 con 0x7f3d241544f0 2024-03-19T01:57:13.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.181+0000 7f3d2a9b9640 1 -- 172.21.15.55:0/2671943649 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d241f0160 con 0x7f3d241544f0 2024-03-19T01:57:13.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.181+0000 7f3d2a9b9640 1 -- 172.21.15.55:0/2671943649 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d241f0720 con 0x7f3d241544f0 2024-03-19T01:57:13.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.181+0000 7f3d2a9b9640 1 -- 172.21.15.55:0/2671943649 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d24068a40 con 0x7f3d241544f0 2024-03-19T01:57:13.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.485+0000 7f3d20ff9640 1 -- 172.21.15.55:0/2671943649 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f3d1801a070 con 0x7f3d241544f0 2024-03-19T01:57:13.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.485+0000 7f3d20ff9640 1 --2- 172.21.15.55:0/2671943649 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3d100676e0 0x7f3d10069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:13.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.485+0000 7f3d23fff640 1 --2- 172.21.15.55:0/2671943649 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3d100676e0 0x7f3d10069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:13.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.485+0000 7f3d23fff640 1 --2- 172.21.15.55:0/2671943649 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3d100676e0 0x7f3d10069ba0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3d240710f0 tx=0x7f3d14000bd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:13.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.521+0000 7f3d20ff9640 1 -- 172.21.15.55:0/2671943649 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f3d18053770 con 0x7f3d241544f0 2024-03-19T01:57:13.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.521+0000 7f3d20ff9640 1 -- 172.21.15.55:0/2671943649 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f3d18021020 con 0x7f3d241544f0 2024-03-19T01:57:13.739 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.741+0000 7f3d2a9b9640 1 -- 172.21.15.55:0/2671943649 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d240c7be0 con 0x7f3d241544f0 2024-03-19T01:57:13.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.765+0000 7f3d20ff9640 1 -- 172.21.15.55:0/2671943649 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f3d180556d0 con 0x7f3d241544f0 2024-03-19T01:57:13.763 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:13.763 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:13.764 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:13.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.765+0000 7f3d2a9b9640 1 -- 172.21.15.55:0/2671943649 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3d100676e0 msgr2=0x7f3d10069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:13.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.765+0000 7f3d2a9b9640 1 --2- 172.21.15.55:0/2671943649 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3d100676e0 0x7f3d10069ba0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3d240710f0 tx=0x7f3d14000bd0 comp rx=0 tx=0).stop 2024-03-19T01:57:13.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.765+0000 7f3d2a9b9640 1 -- 172.21.15.55:0/2671943649 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3d241544f0 msgr2=0x7f3d24151d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:13.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.765+0000 7f3d2a9b9640 1 --2- 172.21.15.55:0/2671943649 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3d241544f0 0x7f3d24151d70 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800c430 tx=0x7f3d1800c8f0 comp rx=0 tx=0).stop 2024-03-19T01:57:13.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.765+0000 7f3d2a9b9640 1 -- 172.21.15.55:0/2671943649 shutdown_connections 2024-03-19T01:57:13.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.765+0000 7f3d2a9b9640 1 --2- 172.21.15.55:0/2671943649 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3d100676e0 0x7f3d10069ba0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:13.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.765+0000 7f3d2a9b9640 1 --2- 172.21.15.55:0/2671943649 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d2415acb0 0x7f3d241522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:13.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.765+0000 7f3d2a9b9640 1 --2- 172.21.15.55:0/2671943649 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3d241544f0 0x7f3d24151d70 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:13.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.765+0000 7f3d2a9b9640 1 --2- 172.21.15.55:0/2671943649 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3d2406b2a0 0x7f3d24070660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:13.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.765+0000 7f3d2a9b9640 1 -- 172.21.15.55:0/2671943649 >> 172.21.15.55:0/2671943649 conn(0x7f3d2406e670 msgr2=0x7f3d240cce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:13.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.765+0000 7f3d2a9b9640 1 -- 172.21.15.55:0/2671943649 shutdown_connections 2024-03-19T01:57:13.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:13.765+0000 7f3d2a9b9640 1 -- 172.21.15.55:0/2671943649 wait complete. 2024-03-19T01:57:14.388 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:14.388 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:15.389 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:15.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.501+0000 7fbc04051640 1 Processor -- start 2024-03-19T01:57:15.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.501+0000 7fbc04051640 1 -- start start 2024-03-19T01:57:15.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.505+0000 7fbc04051640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fbbfc0c9000 0x7fbbfc0c9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:15.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.505+0000 7fbc04051640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbfc0c2ad0 0x7fbbfc0c2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:15.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.505+0000 7fbc04051640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbfc158d50 con 0x7fbbfc1520b0 2024-03-19T01:57:15.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.505+0000 7fbc04051640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fbbfc15c4e0 con 0x7fbbfc0c9000 2024-03-19T01:57:15.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.505+0000 7fbc04051640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbbfc15c050 con 0x7fbbfc0c2ad0 2024-03-19T01:57:15.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.505+0000 7fbc025ee640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbfc0c2ad0 0x7fbbfc0c2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:15.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.505+0000 7fbc025ee640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbfc0c2ad0 0x7fbbfc0c2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:53658/0 (socket says 172.21.15.55:53658) 2024-03-19T01:57:15.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.505+0000 7fbc025ee640 1 -- 172.21.15.55:0/1715901900 learned_addr learned my addr 172.21.15.55:0/1715901900 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:15.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.509+0000 7fbc025ee640 1 -- 172.21.15.55:0/1715901900 >> v2:172.21.15.55:3300/0 conn(0x7fbbfc0c9000 msgr2=0x7fbbfc0c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:15.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.509+0000 7fbc025ee640 1 --2- 172.21.15.55:0/1715901900 >> v2:172.21.15.55:3300/0 conn(0x7fbbfc0c9000 0x7fbbfc0c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:15.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.509+0000 7fbc025ee640 1 -- 172.21.15.55:0/1715901900 >> v1:172.21.15.55:6789/0 conn(0x7fbbfc1520b0 legacy=0x7fbbfc0709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:15.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.509+0000 7fbc025ee640 1 -- 172.21.15.55:0/1715901900 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbfc15b720 con 0x7fbbfc0c2ad0 2024-03-19T01:57:15.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.513+0000 7fbc025ee640 1 --2- 172.21.15.55:0/1715901900 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbfc0c2ad0 0x7fbbfc0c2f00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4009e50 tx=0x7fbbf401c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d177f87bf0d8af32 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:15.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.517+0000 7fbc00deb640 1 -- 172.21.15.55:0/1715901900 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf40a5670 con 0x7fbbfc0c2ad0 2024-03-19T01:57:15.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.517+0000 7fbc00deb640 1 -- 172.21.15.55:0/1715901900 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbf40a5810 con 0x7fbbfc0c2ad0 2024-03-19T01:57:15.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.517+0000 7fbc00deb640 1 -- 172.21.15.55:0/1715901900 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf40a5be0 con 0x7fbbfc0c2ad0 2024-03-19T01:57:15.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.517+0000 7fbc04051640 1 -- 172.21.15.55:0/1715901900 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbfc0c2ad0 msgr2=0x7fbbfc0c2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:15.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.517+0000 7fbc04051640 1 --2- 172.21.15.55:0/1715901900 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbfc0c2ad0 0x7fbbfc0c2f00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4009e50 tx=0x7fbbf401c9f0 comp rx=0 tx=0).stop 2024-03-19T01:57:15.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.517+0000 7fbc04051640 1 -- 172.21.15.55:0/1715901900 shutdown_connections 2024-03-19T01:57:15.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.517+0000 7fbc04051640 1 --2- 172.21.15.55:0/1715901900 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbfc0c2ad0 0x7fbbfc0c2f00 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:15.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.517+0000 7fbc04051640 1 --2- 172.21.15.55:0/1715901900 >> v2:172.21.15.55:3300/0 conn(0x7fbbfc0c9000 0x7fbbfc0c9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:15.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.517+0000 7fbc04051640 1 -- 172.21.15.55:0/1715901900 >> 172.21.15.55:0/1715901900 conn(0x7fbbfc067600 msgr2=0x7fbbfc071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:15.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.517+0000 7fbc04051640 1 -- 172.21.15.55:0/1715901900 shutdown_connections 2024-03-19T01:57:15.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.521+0000 7fbc04051640 1 -- 172.21.15.55:0/1715901900 wait complete. 2024-03-19T01:57:15.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.521+0000 7fbc04051640 1 Processor -- start 2024-03-19T01:57:15.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.521+0000 7fbc04051640 1 -- start start 2024-03-19T01:57:15.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.521+0000 7fbc04051640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbfc0c2ad0 0x7fbbfc0b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:15.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.521+0000 7fbc04051640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbbfc0c9000 0x7fbbfc14da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:15.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.521+0000 7fbc04051640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbfc1520b0 0x7fbbfc14b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:15.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.521+0000 7fbc04051640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbbfc0c3850 con 0x7fbbfc0c2ad0 2024-03-19T01:57:15.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.521+0000 7fbc04051640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fbbfc0710b0 con 0x7fbbfc0c9000 2024-03-19T01:57:15.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.521+0000 7fbc04051640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbbfc152490 con 0x7fbbfc1520b0 2024-03-19T01:57:15.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.521+0000 7fbc015ec640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbbfc0c9000 0x7fbbfc14da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:15.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.521+0000 7fbc015ec640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbbfc0c9000 0x7fbbfc14da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:47440/0 (socket says 172.21.15.55:47440) 2024-03-19T01:57:15.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.521+0000 7fbc015ec640 1 -- 172.21.15.55:0/543567460 learned_addr learned my addr 172.21.15.55:0/543567460 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:15.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.521+0000 7fbc01ded640 1 --2- 172.21.15.55:0/543567460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbfc0c2ad0 0x7fbbfc0b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:15.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.525+0000 7fbc025ee640 1 --2- 172.21.15.55:0/543567460 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbfc1520b0 0x7fbbfc14b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:15.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.529+0000 7fbc015ec640 1 -- 172.21.15.55:0/543567460 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbfc1520b0 msgr2=0x7fbbfc14b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:15.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.529+0000 7fbc015ec640 1 --2- 172.21.15.55:0/543567460 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbfc1520b0 0x7fbbfc14b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:15.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.529+0000 7fbc015ec640 1 -- 172.21.15.55:0/543567460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbfc0c2ad0 msgr2=0x7fbbfc0b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:15.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.529+0000 7fbc015ec640 1 --2- 172.21.15.55:0/543567460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbfc0c2ad0 0x7fbbfc0b11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:15.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.529+0000 7fbc015ec640 1 -- 172.21.15.55:0/543567460 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbfc0b1710 con 0x7fbbfc0c9000 2024-03-19T01:57:15.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.533+0000 7fbc015ec640 1 --2- 172.21.15.55:0/543567460 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbbfc0c9000 0x7fbbfc14da30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbbec00b550 tx=0x7fbbec00ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:15.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.537+0000 7fbbf27fc640 1 -- 172.21.15.55:0/543567460 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbec014070 con 0x7fbbfc0c9000 2024-03-19T01:57:15.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.537+0000 7fbbf27fc640 1 -- 172.21.15.55:0/543567460 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbec010890 con 0x7fbbfc0c9000 2024-03-19T01:57:15.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.537+0000 7fbbf27fc640 1 -- 172.21.15.55:0/543567460 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbec010c80 con 0x7fbbfc0c9000 2024-03-19T01:57:15.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.537+0000 7fbc04051640 1 -- 172.21.15.55:0/543567460 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbfc14fc30 con 0x7fbbfc0c9000 2024-03-19T01:57:15.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.537+0000 7fbc04051640 1 -- 172.21.15.55:0/543567460 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbfc1501f0 con 0x7fbbfc0c9000 2024-03-19T01:57:15.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.541+0000 7fbc04051640 1 -- 172.21.15.55:0/543567460 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbfc068a40 con 0x7fbbfc0c9000 2024-03-19T01:57:15.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.577+0000 7fbbf27fc640 1 -- 172.21.15.55:0/543567460 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fbbec017070 con 0x7fbbfc0c9000 2024-03-19T01:57:15.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.577+0000 7fbbf27fc640 1 --2- 172.21.15.55:0/543567460 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbbe00676b0 0x7fbbe0069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:15.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.577+0000 7fbc01ded640 1 --2- 172.21.15.55:0/543567460 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbbe00676b0 0x7fbbe0069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:15.576 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.577+0000 7fbc01ded640 1 --2- 172.21.15.55:0/543567460 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbbe00676b0 0x7fbbe0069b70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8007fd0 tx=0x7fbbf8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:15.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.845+0000 7fbbf27fc640 1 -- 172.21.15.55:0/543567460 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fbbec0540e0 con 0x7fbbfc0c9000 2024-03-19T01:57:15.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:15.885+0000 7fbbf27fc640 1 -- 172.21.15.55:0/543567460 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fbbec056040 con 0x7fbbfc0c9000 2024-03-19T01:57:16.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:16.097+0000 7fbc04051640 1 -- 172.21.15.55:0/543567460 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbbfc0cc010 con 0x7fbbfc0c9000 2024-03-19T01:57:16.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:16.121+0000 7fbbf27fc640 1 -- 172.21.15.55:0/543567460 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fbbec010e20 con 0x7fbbfc0c9000 2024-03-19T01:57:16.120 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:16.120 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:16.120 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:16.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:16.121+0000 7fbc04051640 1 -- 172.21.15.55:0/543567460 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbbe00676b0 msgr2=0x7fbbe0069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:16.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:16.121+0000 7fbc04051640 1 --2- 172.21.15.55:0/543567460 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbbe00676b0 0x7fbbe0069b70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8007fd0 tx=0x7fbbf8009040 comp rx=0 tx=0).stop 2024-03-19T01:57:16.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:16.121+0000 7fbc04051640 1 -- 172.21.15.55:0/543567460 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbbfc0c9000 msgr2=0x7fbbfc14da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:16.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:16.121+0000 7fbc04051640 1 --2- 172.21.15.55:0/543567460 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbbfc0c9000 0x7fbbfc14da30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbbec00b550 tx=0x7fbbec00ba10 comp rx=0 tx=0).stop 2024-03-19T01:57:16.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:16.121+0000 7fbc04051640 1 -- 172.21.15.55:0/543567460 shutdown_connections 2024-03-19T01:57:16.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:16.121+0000 7fbc04051640 1 --2- 172.21.15.55:0/543567460 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbbe00676b0 0x7fbbe0069b70 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:16.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:16.121+0000 7fbc04051640 1 --2- 172.21.15.55:0/543567460 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbfc1520b0 0x7fbbfc14b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:16.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:16.121+0000 7fbc04051640 1 --2- 172.21.15.55:0/543567460 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbbfc0c9000 0x7fbbfc14da30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:16.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:16.121+0000 7fbc04051640 1 --2- 172.21.15.55:0/543567460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbfc0c2ad0 0x7fbbfc0b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:16.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:16.121+0000 7fbc04051640 1 -- 172.21.15.55:0/543567460 >> 172.21.15.55:0/543567460 conn(0x7fbbfc067600 msgr2=0x7fbbfc14ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:16.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:16.121+0000 7fbc04051640 1 -- 172.21.15.55:0/543567460 shutdown_connections 2024-03-19T01:57:16.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:16.121+0000 7fbc04051640 1 -- 172.21.15.55:0/543567460 wait complete. 2024-03-19T01:57:16.743 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:16.743 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:17.744 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:17.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.869+0000 7ff05c778640 1 Processor -- start 2024-03-19T01:57:17.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.869+0000 7ff05c778640 1 -- start start 2024-03-19T01:57:17.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.869+0000 7ff05c778640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0541520b0 0x7ff0540709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:17.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.869+0000 7ff05c778640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff0540c9000 0x7ff0540c9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:17.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.869+0000 7ff05c778640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff054158d50 con 0x7ff0540c2ad0 2024-03-19T01:57:17.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.869+0000 7ff05c778640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7ff05415c4e0 con 0x7ff0540c9000 2024-03-19T01:57:17.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.869+0000 7ff05c778640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff05415c050 con 0x7ff0541520b0 2024-03-19T01:57:17.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.873+0000 7ff05a514640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0541520b0 0x7ff0540709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:17.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.873+0000 7ff059d13640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff0540c9000 0x7ff0540c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:17.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.873+0000 7ff05a514640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0541520b0 0x7ff0540709a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:53678/0 (socket says 172.21.15.55:53678) 2024-03-19T01:57:17.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.873+0000 7ff05a514640 1 -- 172.21.15.55:0/1432665439 learned_addr learned my addr 172.21.15.55:0/1432665439 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:17.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.877+0000 7ff05a514640 1 -- 172.21.15.55:0/1432665439 >> v2:172.21.15.84:3300/0 conn(0x7ff0540c9000 msgr2=0x7ff0540c9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:17.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.877+0000 7ff05a514640 1 --2- 172.21.15.55:0/1432665439 >> v2:172.21.15.84:3300/0 conn(0x7ff0540c9000 0x7ff0540c9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:17.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.877+0000 7ff05a514640 1 -- 172.21.15.55:0/1432665439 >> v1:172.21.15.84:6789/0 conn(0x7ff0540c2ad0 legacy=0x7ff0540c2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:17.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.877+0000 7ff05a514640 1 -- 172.21.15.55:0/1432665439 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff05415b720 con 0x7ff0541520b0 2024-03-19T01:57:17.880 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.881+0000 7ff05a514640 1 --2- 172.21.15.55:0/1432665439 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0541520b0 0x7ff0540709a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff050009870 tx=0x7ff05001ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f030e63074181ea server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:17.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.885+0000 7ff059512640 1 -- 172.21.15.55:0/1432665439 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0500b5070 con 0x7ff0541520b0 2024-03-19T01:57:17.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.885+0000 7ff059512640 1 -- 172.21.15.55:0/1432665439 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0500a89d0 con 0x7ff0541520b0 2024-03-19T01:57:17.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.885+0000 7ff059512640 1 -- 172.21.15.55:0/1432665439 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0500a8da0 con 0x7ff0541520b0 2024-03-19T01:57:17.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.889+0000 7ff05c778640 1 -- 172.21.15.55:0/1432665439 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0541520b0 msgr2=0x7ff0540709a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:17.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.889+0000 7ff05c778640 1 --2- 172.21.15.55:0/1432665439 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0541520b0 0x7ff0540709a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff050009870 tx=0x7ff05001ed10 comp rx=0 tx=0).stop 2024-03-19T01:57:17.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.889+0000 7ff05c778640 1 -- 172.21.15.55:0/1432665439 shutdown_connections 2024-03-19T01:57:17.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.889+0000 7ff05c778640 1 --2- 172.21.15.55:0/1432665439 >> v2:172.21.15.84:3300/0 conn(0x7ff0540c9000 0x7ff0540c9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:17.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.889+0000 7ff05c778640 1 --2- 172.21.15.55:0/1432665439 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0541520b0 0x7ff0540709a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:17.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.889+0000 7ff05c778640 1 -- 172.21.15.55:0/1432665439 >> 172.21.15.55:0/1432665439 conn(0x7ff054067600 msgr2=0x7ff054071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:17.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.889+0000 7ff05c778640 1 -- 172.21.15.55:0/1432665439 shutdown_connections 2024-03-19T01:57:17.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.889+0000 7ff05c778640 1 -- 172.21.15.55:0/1432665439 wait complete. 2024-03-19T01:57:17.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.889+0000 7ff05c778640 1 Processor -- start 2024-03-19T01:57:17.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.889+0000 7ff05c778640 1 -- start start 2024-03-19T01:57:17.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.889+0000 7ff05c778640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0540c2ad0 0x7ff0541ee9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:17.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.889+0000 7ff05c778640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0540c9000 0x7ff0540c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:17.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.889+0000 7ff05c778640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0541520b0 0x7ff054155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:17.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.889+0000 7ff05c778640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff0541589a0 con 0x7ff0541520b0 2024-03-19T01:57:17.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.889+0000 7ff05c778640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff05415bce0 con 0x7ff0540c9000 2024-03-19T01:57:17.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.889+0000 7ff05c778640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff0540c6ee0 con 0x7ff0540c2ad0 2024-03-19T01:57:17.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.889+0000 7ff05ad15640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0541520b0 0x7ff054155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:17.891 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.893+0000 7ff05a514640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0540c2ad0 0x7ff0541ee9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:17.891 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.893+0000 7ff05ad15640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0541520b0 0x7ff054155cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:45138/0 (socket says 172.21.15.55:45138) 2024-03-19T01:57:17.891 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.893+0000 7ff05ad15640 1 -- 172.21.15.55:0/1879733704 learned_addr learned my addr 172.21.15.55:0/1879733704 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:17.891 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.893+0000 7ff059d13640 1 --2- 172.21.15.55:0/1879733704 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0540c9000 0x7ff0540c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:17.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.893+0000 7ff05ad15640 1 -- 172.21.15.55:0/1879733704 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0540c2ad0 msgr2=0x7ff0541ee9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:17.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.893+0000 7ff05ad15640 1 --2- 172.21.15.55:0/1879733704 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0540c2ad0 0x7ff0541ee9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:17.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.893+0000 7ff05ad15640 1 -- 172.21.15.55:0/1879733704 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0540c9000 msgr2=0x7ff0540c68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:17.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.893+0000 7ff05ad15640 1 --2- 172.21.15.55:0/1879733704 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0540c9000 0x7ff0540c68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:17.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.893+0000 7ff05ad15640 1 -- 172.21.15.55:0/1879733704 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0540c71c0 con 0x7ff0541520b0 2024-03-19T01:57:17.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.897+0000 7ff05ad15640 1 --2- 172.21.15.55:0/1879733704 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0541520b0 0x7ff054155cf0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff04c00c7b0 tx=0x7ff04c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:17.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.901+0000 7ff04affd640 1 -- 172.21.15.55:0/1879733704 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c004b70 con 0x7ff0541520b0 2024-03-19T01:57:17.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.901+0000 7ff04affd640 1 -- 172.21.15.55:0/1879733704 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff04c004d10 con 0x7ff0541520b0 2024-03-19T01:57:17.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.901+0000 7ff04affd640 1 -- 172.21.15.55:0/1879733704 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c00aa90 con 0x7ff0541520b0 2024-03-19T01:57:17.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.901+0000 7ff05c778640 1 -- 172.21.15.55:0/1879733704 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0540c74a0 con 0x7ff0541520b0 2024-03-19T01:57:17.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.901+0000 7ff05c778640 1 -- 172.21.15.55:0/1879733704 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0541ee1c0 con 0x7ff0541520b0 2024-03-19T01:57:17.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.905+0000 7ff05c778640 1 -- 172.21.15.55:0/1879733704 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff054068a40 con 0x7ff0541520b0 2024-03-19T01:57:17.922 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.925+0000 7ff04affd640 1 -- 172.21.15.55:0/1879733704 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ff04c00ac30 con 0x7ff0541520b0 2024-03-19T01:57:17.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.925+0000 7ff04affd640 1 --2- 172.21.15.55:0/1879733704 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff0380676e0 0x7ff038069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:17.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.925+0000 7ff05a514640 1 --2- 172.21.15.55:0/1879733704 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff0380676e0 0x7ff038069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:17.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.925+0000 7ff05a514640 1 --2- 172.21.15.55:0/1879733704 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff0380676e0 0x7ff038069ba0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff0500b2000 tx=0x7ff0500aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:17.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:17.929+0000 7ff04affd640 1 -- 172.21.15.55:0/1879733704 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7ff04c054820 con 0x7ff0541520b0 2024-03-19T01:57:18.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:18.225+0000 7ff04affd640 1 -- 172.21.15.55:0/1879733704 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ff04c020070 con 0x7ff0541520b0 2024-03-19T01:57:18.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:18.445+0000 7ff05c778640 1 -- 172.21.15.55:0/1879733704 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff05414c0f0 con 0x7ff0541520b0 2024-03-19T01:57:18.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:18.473+0000 7ff04affd640 1 -- 172.21.15.55:0/1879733704 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7ff04c056780 con 0x7ff0541520b0 2024-03-19T01:57:18.474 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:18.475 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:18.475 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:18.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:18.477+0000 7ff05c778640 1 -- 172.21.15.55:0/1879733704 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff0380676e0 msgr2=0x7ff038069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:18.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:18.477+0000 7ff05c778640 1 --2- 172.21.15.55:0/1879733704 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff0380676e0 0x7ff038069ba0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff0500b2000 tx=0x7ff0500aa070 comp rx=0 tx=0).stop 2024-03-19T01:57:18.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:18.477+0000 7ff05c778640 1 -- 172.21.15.55:0/1879733704 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0541520b0 msgr2=0x7ff054155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:18.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:18.477+0000 7ff05c778640 1 --2- 172.21.15.55:0/1879733704 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0541520b0 0x7ff054155cf0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff04c00c7b0 tx=0x7ff04c00cc70 comp rx=0 tx=0).stop 2024-03-19T01:57:18.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:18.477+0000 7ff05c778640 1 -- 172.21.15.55:0/1879733704 shutdown_connections 2024-03-19T01:57:18.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:18.477+0000 7ff05c778640 1 --2- 172.21.15.55:0/1879733704 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff0380676e0 0x7ff038069ba0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:18.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:18.477+0000 7ff05c778640 1 --2- 172.21.15.55:0/1879733704 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0541520b0 0x7ff054155cf0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:18.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:18.477+0000 7ff05c778640 1 --2- 172.21.15.55:0/1879733704 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0540c9000 0x7ff0540c68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:18.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:18.477+0000 7ff05c778640 1 --2- 172.21.15.55:0/1879733704 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0540c2ad0 0x7ff0541ee9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:18.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:18.477+0000 7ff05c778640 1 -- 172.21.15.55:0/1879733704 >> 172.21.15.55:0/1879733704 conn(0x7ff054067600 msgr2=0x7ff0540b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:18.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:18.477+0000 7ff05c778640 1 -- 172.21.15.55:0/1879733704 shutdown_connections 2024-03-19T01:57:18.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:18.477+0000 7ff05c778640 1 -- 172.21.15.55:0/1879733704 wait complete. 2024-03-19T01:57:19.098 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:19.098 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:20.099 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:20.229 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.229+0000 7fa07a726640 1 Processor -- start 2024-03-19T01:57:20.229 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.229+0000 7fa07a726640 1 -- start start 2024-03-19T01:57:20.230 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.229+0000 7fa07a726640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa0741544b0 0x7fa07406b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:20.230 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.229+0000 7fa07a726640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa0740c9050 0x7fa0740c94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:20.230 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.229+0000 7fa07a726640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa07415a6b0 con 0x7fa07415a0e0 2024-03-19T01:57:20.230 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.229+0000 7fa07a726640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fa07415a990 con 0x7fa0741544b0 2024-03-19T01:57:20.230 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.229+0000 7fa07a726640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa07415ac70 con 0x7fa0740c9050 2024-03-19T01:57:20.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.233+0000 7fa073fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa0741544b0 0x7fa07406b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:20.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.233+0000 7fa073fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa0741544b0 0x7fa07406b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53456/0 (socket says 172.21.15.55:53456) 2024-03-19T01:57:20.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.233+0000 7fa073fff640 1 -- 172.21.15.55:0/3117995988 learned_addr learned my addr 172.21.15.55:0/3117995988 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:20.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.233+0000 7fa078cc3640 1 --2- 172.21.15.55:0/3117995988 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa0740c9050 0x7fa0740c94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:20.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.233+0000 7fa073fff640 1 -- 172.21.15.55:0/3117995988 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa0740c9050 msgr2=0x7fa0740c94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:20.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.233+0000 7fa073fff640 1 --2- 172.21.15.55:0/3117995988 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa0740c9050 0x7fa0740c94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:20.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.233+0000 7fa073fff640 1 -- 172.21.15.55:0/3117995988 >> v1:172.21.15.55:6789/0 conn(0x7fa07415a0e0 legacy=0x7fa0740cbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:20.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.233+0000 7fa073fff640 1 -- 172.21.15.55:0/3117995988 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa07415af50 con 0x7fa0741544b0 2024-03-19T01:57:20.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.241+0000 7fa073fff640 1 --2- 172.21.15.55:0/3117995988 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0741544b0 0x7fa07406b2a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa064009870 tx=0x7fa06401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92cca03ca527a91c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:20.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.245+0000 7fa072ffd640 1 -- 172.21.15.55:0/3117995988 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0640b5070 con 0x7fa0741544b0 2024-03-19T01:57:20.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.245+0000 7fa072ffd640 1 -- 172.21.15.55:0/3117995988 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0640a89d0 con 0x7fa0741544b0 2024-03-19T01:57:20.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.245+0000 7fa072ffd640 1 -- 172.21.15.55:0/3117995988 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0640a8da0 con 0x7fa0741544b0 2024-03-19T01:57:20.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.245+0000 7fa07a726640 1 -- 172.21.15.55:0/3117995988 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0741544b0 msgr2=0x7fa07406b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:20.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.245+0000 7fa07a726640 1 --2- 172.21.15.55:0/3117995988 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0741544b0 0x7fa07406b2a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa064009870 tx=0x7fa06401ed10 comp rx=0 tx=0).stop 2024-03-19T01:57:20.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.245+0000 7fa07a726640 1 -- 172.21.15.55:0/3117995988 shutdown_connections 2024-03-19T01:57:20.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.245+0000 7fa07a726640 1 --2- 172.21.15.55:0/3117995988 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa0740c9050 0x7fa0740c94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:20.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.245+0000 7fa07a726640 1 --2- 172.21.15.55:0/3117995988 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0741544b0 0x7fa07406b2a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:20.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.245+0000 7fa07a726640 1 -- 172.21.15.55:0/3117995988 >> 172.21.15.55:0/3117995988 conn(0x7fa07406e5a0 msgr2=0x7fa074072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:20.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.245+0000 7fa07a726640 1 -- 172.21.15.55:0/3117995988 shutdown_connections 2024-03-19T01:57:20.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.245+0000 7fa07a726640 1 -- 172.21.15.55:0/3117995988 wait complete. 2024-03-19T01:57:20.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.245+0000 7fa07a726640 1 Processor -- start 2024-03-19T01:57:20.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.245+0000 7fa07a726640 1 -- start start 2024-03-19T01:57:20.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.249+0000 7fa07a726640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0740c9050 0x7fa0741483d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:20.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.249+0000 7fa07a726640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa0741544b0 0x7fa0741494c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:20.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.249+0000 7fa07a726640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa07415a0e0 0x7fa074149a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:20.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.249+0000 7fa07a726640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa07406b7e0 con 0x7fa0740c9050 2024-03-19T01:57:20.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.249+0000 7fa07a726640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa0740c6be0 con 0x7fa0741544b0 2024-03-19T01:57:20.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.249+0000 7fa07a726640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa0740c8760 con 0x7fa07415a0e0 2024-03-19T01:57:20.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.249+0000 7fa073fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0740c9050 0x7fa0741483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:20.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.249+0000 7fa0737fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa0741544b0 0x7fa0741494c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:20.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.249+0000 7fa073fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0740c9050 0x7fa0741483d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53466/0 (socket says 172.21.15.55:53466) 2024-03-19T01:57:20.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.249+0000 7fa073fff640 1 -- 172.21.15.55:0/3825701593 learned_addr learned my addr 172.21.15.55:0/3825701593 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:20.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.249+0000 7fa078cc3640 1 --2- 172.21.15.55:0/3825701593 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa07415a0e0 0x7fa074149a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:20.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.253+0000 7fa073fff640 1 -- 172.21.15.55:0/3825701593 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa07415a0e0 msgr2=0x7fa074149a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:20.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.253+0000 7fa073fff640 1 --2- 172.21.15.55:0/3825701593 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa07415a0e0 0x7fa074149a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:20.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.253+0000 7fa073fff640 1 -- 172.21.15.55:0/3825701593 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa0741544b0 msgr2=0x7fa0741494c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:20.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.253+0000 7fa073fff640 1 --2- 172.21.15.55:0/3825701593 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa0741544b0 0x7fa0741494c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:20.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.253+0000 7fa073fff640 1 -- 172.21.15.55:0/3825701593 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0741f3540 con 0x7fa0740c9050 2024-03-19T01:57:20.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.253+0000 7fa073fff640 1 --2- 172.21.15.55:0/3825701593 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0740c9050 0x7fa0741483d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa06401f5b0 tx=0x7fa06401bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:20.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.257+0000 7fa070ff9640 1 -- 172.21.15.55:0/3825701593 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0640b5040 con 0x7fa0740c9050 2024-03-19T01:57:20.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.257+0000 7fa070ff9640 1 -- 172.21.15.55:0/3825701593 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0640be070 con 0x7fa0740c9050 2024-03-19T01:57:20.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.257+0000 7fa070ff9640 1 -- 172.21.15.55:0/3825701593 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0640b19b0 con 0x7fa0740c9050 2024-03-19T01:57:20.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.261+0000 7fa07a726640 1 -- 172.21.15.55:0/3825701593 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa07415bcb0 con 0x7fa0740c9050 2024-03-19T01:57:20.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.261+0000 7fa07a726640 1 -- 172.21.15.55:0/3825701593 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0741f3ae0 con 0x7fa0740c9050 2024-03-19T01:57:20.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.261+0000 7fa07a726640 1 -- 172.21.15.55:0/3825701593 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa074068a40 con 0x7fa0740c9050 2024-03-19T01:57:20.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.537+0000 7fa070ff9640 1 -- 172.21.15.55:0/3825701593 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa0640b1b50 con 0x7fa0740c9050 2024-03-19T01:57:20.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.537+0000 7fa070ff9640 1 --2- 172.21.15.55:0/3825701593 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa0600676e0 0x7fa060069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:20.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.537+0000 7fa0737fe640 1 --2- 172.21.15.55:0/3825701593 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa0600676e0 0x7fa060069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:20.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.541+0000 7fa0737fe640 1 --2- 172.21.15.55:0/3825701593 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa0600676e0 0x7fa060069ba0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa074153ab0 tx=0x7fa068009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:20.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.557+0000 7fa070ff9640 1 -- 172.21.15.55:0/3825701593 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fa0640f1ee0 con 0x7fa0740c9050 2024-03-19T01:57:20.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.557+0000 7fa070ff9640 1 -- 172.21.15.55:0/3825701593 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa064018870 con 0x7fa0740c9050 2024-03-19T01:57:20.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.797+0000 7fa07a726640 1 -- 172.21.15.55:0/3825701593 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa07406bb10 con 0x7fa0740c9050 2024-03-19T01:57:20.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.865+0000 7fa070ff9640 1 -- 172.21.15.55:0/3825701593 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fa0640f3e40 con 0x7fa0740c9050 2024-03-19T01:57:20.865 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:20.865 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:20.865 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:20.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.869+0000 7fa07a726640 1 -- 172.21.15.55:0/3825701593 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa0600676e0 msgr2=0x7fa060069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:20.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.869+0000 7fa07a726640 1 --2- 172.21.15.55:0/3825701593 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa0600676e0 0x7fa060069ba0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa074153ab0 tx=0x7fa068009060 comp rx=0 tx=0).stop 2024-03-19T01:57:20.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.869+0000 7fa07a726640 1 -- 172.21.15.55:0/3825701593 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0740c9050 msgr2=0x7fa0741483d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:20.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.869+0000 7fa07a726640 1 --2- 172.21.15.55:0/3825701593 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0740c9050 0x7fa0741483d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa06401f5b0 tx=0x7fa06401bab0 comp rx=0 tx=0).stop 2024-03-19T01:57:20.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.869+0000 7fa07a726640 1 -- 172.21.15.55:0/3825701593 shutdown_connections 2024-03-19T01:57:20.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.869+0000 7fa07a726640 1 --2- 172.21.15.55:0/3825701593 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa0600676e0 0x7fa060069ba0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:20.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.869+0000 7fa07a726640 1 --2- 172.21.15.55:0/3825701593 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa07415a0e0 0x7fa074149a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:20.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.869+0000 7fa07a726640 1 --2- 172.21.15.55:0/3825701593 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa0741544b0 0x7fa0741494c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:20.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.869+0000 7fa07a726640 1 --2- 172.21.15.55:0/3825701593 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0740c9050 0x7fa0741483d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:20.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.869+0000 7fa07a726640 1 -- 172.21.15.55:0/3825701593 >> 172.21.15.55:0/3825701593 conn(0x7fa07406e5a0 msgr2=0x7fa0740c7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:20.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.869+0000 7fa07a726640 1 -- 172.21.15.55:0/3825701593 shutdown_connections 2024-03-19T01:57:20.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:20.869+0000 7fa07a726640 1 -- 172.21.15.55:0/3825701593 wait complete. 2024-03-19T01:57:21.494 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:21.494 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:22.495 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:22.609 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.609+0000 7f079eb54640 1 Processor -- start 2024-03-19T01:57:22.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.609+0000 7f079eb54640 1 -- start start 2024-03-19T01:57:22.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.609+0000 7f079eb54640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07980c8750 0x7f07980cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:22.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.613+0000 7f079eb54640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0798154a80 con 0x7f07980ce020 2024-03-19T01:57:22.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.613+0000 7f079eb54640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f079815dc70 con 0x7f07981546a0 2024-03-19T01:57:22.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.613+0000 7f079eb54640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f079815e7f0 con 0x7f07980c8750 2024-03-19T01:57:22.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.613+0000 7f079d0f1640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f07980ce020 0x7f079806cea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:59894/0 (socket says 172.21.15.55:59894) 2024-03-19T01:57:22.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.613+0000 7f079d0f1640 1 -- 172.21.15.55:0/1512611761 learned_addr learned my addr 172.21.15.55:0/1512611761 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:57:22.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.613+0000 7f0797fff640 1 --2- 172.21.15.55:0/1512611761 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07980c8750 0x7f07980cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:22.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.617+0000 7f07977fe640 1 -- 172.21.15.55:0/1512611761 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 615464320 0 0) 0x7f0798154a80 con 0x7f07980ce020 2024-03-19T01:57:22.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.617+0000 7f07977fe640 1 -- 172.21.15.55:0/1512611761 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0788003530 con 0x7f07980ce020 2024-03-19T01:57:22.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.617+0000 7f0797fff640 1 -- 172.21.15.55:0/1512611761 >> v1:172.21.15.84:6789/0 conn(0x7f07981546a0 legacy=0x7f079815ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:22.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.617+0000 7f0797fff640 1 -- 172.21.15.55:0/1512611761 >> v1:172.21.15.55:6789/0 conn(0x7f07980ce020 legacy=0x7f079806cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:22.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.617+0000 7f0797fff640 1 -- 172.21.15.55:0/1512611761 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f079815df50 con 0x7f07980c8750 2024-03-19T01:57:22.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.621+0000 7f0797fff640 1 --2- 172.21.15.55:0/1512611761 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07980c8750 0x7f07980cdae0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f07840097f0 tx=0x7f078401c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9aebeaff0174d52a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:22.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.625+0000 7f07977fe640 1 -- 172.21.15.55:0/1512611761 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07840b0070 con 0x7f07980c8750 2024-03-19T01:57:22.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.625+0000 7f07977fe640 1 -- 172.21.15.55:0/1512611761 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07840ab600 con 0x7f07980c8750 2024-03-19T01:57:22.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.625+0000 7f07977fe640 1 -- 172.21.15.55:0/1512611761 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07840ab9d0 con 0x7f07980c8750 2024-03-19T01:57:22.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.625+0000 7f079eb54640 1 -- 172.21.15.55:0/1512611761 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07980c8750 msgr2=0x7f07980cdae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:22.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.625+0000 7f079eb54640 1 --2- 172.21.15.55:0/1512611761 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07980c8750 0x7f07980cdae0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f07840097f0 tx=0x7f078401c4e0 comp rx=0 tx=0).stop 2024-03-19T01:57:22.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.625+0000 7f079eb54640 1 -- 172.21.15.55:0/1512611761 shutdown_connections 2024-03-19T01:57:22.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.625+0000 7f079eb54640 1 --2- 172.21.15.55:0/1512611761 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07980c8750 0x7f07980cdae0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:22.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.625+0000 7f079eb54640 1 -- 172.21.15.55:0/1512611761 >> 172.21.15.55:0/1512611761 conn(0x7f07980714d0 msgr2=0x7f07980718e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:22.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.625+0000 7f079eb54640 1 -- 172.21.15.55:0/1512611761 shutdown_connections 2024-03-19T01:57:22.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.625+0000 7f079eb54640 1 -- 172.21.15.55:0/1512611761 wait complete. 2024-03-19T01:57:22.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.625+0000 7f079eb54640 1 Processor -- start 2024-03-19T01:57:22.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.629+0000 7f079eb54640 1 -- start start 2024-03-19T01:57:22.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.629+0000 7f079eb54640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07980c8750 0x7f07981406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:22.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.629+0000 7f079eb54640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07980ce020 0x7f079815b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:22.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.629+0000 7f079eb54640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07981546a0 0x7f07980ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:22.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.629+0000 7f079eb54640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f079815da20 con 0x7f07980c8750 2024-03-19T01:57:22.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.629+0000 7f079eb54640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f079815d8a0 con 0x7f07980ce020 2024-03-19T01:57:22.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.629+0000 7f079eb54640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f0798168420 con 0x7f07981546a0 2024-03-19T01:57:22.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.629+0000 7f079c8f0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07980c8750 0x7f07981406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:22.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.629+0000 7f079c8f0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07980c8750 0x7f07981406b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53472/0 (socket says 172.21.15.55:53472) 2024-03-19T01:57:22.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.629+0000 7f079c8f0640 1 -- 172.21.15.55:0/3682382669 learned_addr learned my addr 172.21.15.55:0/3682382669 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:22.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.629+0000 7f0797fff640 1 --2- 172.21.15.55:0/3682382669 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07980ce020 0x7f079815b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:22.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.629+0000 7f079d0f1640 1 --2- 172.21.15.55:0/3682382669 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07981546a0 0x7f07980ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:22.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.633+0000 7f079c8f0640 1 -- 172.21.15.55:0/3682382669 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07981546a0 msgr2=0x7f07980ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:22.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.633+0000 7f079c8f0640 1 --2- 172.21.15.55:0/3682382669 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07981546a0 0x7f07980ca680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:22.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.633+0000 7f079c8f0640 1 -- 172.21.15.55:0/3682382669 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07980ce020 msgr2=0x7f079815b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:22.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.633+0000 7f079c8f0640 1 --2- 172.21.15.55:0/3682382669 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07980ce020 0x7f079815b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:22.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.633+0000 7f079c8f0640 1 -- 172.21.15.55:0/3682382669 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07981f72e0 con 0x7f07980c8750 2024-03-19T01:57:22.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.637+0000 7f079c8f0640 1 --2- 172.21.15.55:0/3682382669 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07980c8750 0x7f07981406b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f078c00c480 tx=0x7f078c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:22.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.641+0000 7f07957fa640 1 -- 172.21.15.55:0/3682382669 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c016020 con 0x7f07980c8750 2024-03-19T01:57:22.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.641+0000 7f07957fa640 1 -- 172.21.15.55:0/3682382669 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f078c0052d0 con 0x7f07980c8750 2024-03-19T01:57:22.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.641+0000 7f07957fa640 1 -- 172.21.15.55:0/3682382669 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c007260 con 0x7f07980c8750 2024-03-19T01:57:22.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.641+0000 7f079eb54640 1 -- 172.21.15.55:0/3682382669 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0798158030 con 0x7f07980c8750 2024-03-19T01:57:22.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.641+0000 7f079eb54640 1 -- 172.21.15.55:0/3682382669 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07981585f0 con 0x7f07980c8750 2024-03-19T01:57:22.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.641+0000 7f079eb54640 1 -- 172.21.15.55:0/3682382669 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0798068330 con 0x7f07980c8750 2024-03-19T01:57:22.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.945+0000 7f07957fa640 1 -- 172.21.15.55:0/3682382669 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f078c007400 con 0x7f07980c8750 2024-03-19T01:57:22.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.945+0000 7f07957fa640 1 --2- 172.21.15.55:0/3682382669 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f077c0676b0 0x7f077c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:22.945 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.945+0000 7f0797fff640 1 --2- 172.21.15.55:0/3682382669 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f077c0676b0 0x7f077c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:22.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.945+0000 7f0797fff640 1 --2- 172.21.15.55:0/3682382669 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f077c0676b0 0x7f077c069b70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f07840ad000 tx=0x7f07840a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:22.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.973+0000 7f07957fa640 1 -- 172.21.15.55:0/3682382669 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f078c054670 con 0x7f07980c8750 2024-03-19T01:57:22.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:22.973+0000 7f07957fa640 1 -- 172.21.15.55:0/3682382669 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f078c020020 con 0x7f07980c8750 2024-03-19T01:57:23.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:23.189+0000 7f079eb54640 1 -- 172.21.15.55:0/3682382669 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07980c2c10 con 0x7f07980c8750 2024-03-19T01:57:23.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:23.201+0000 7f07957fa640 1 -- 172.21.15.55:0/3682382669 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f078c005480 con 0x7f07980c8750 2024-03-19T01:57:23.202 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:23.202 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:23.202 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:23.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:23.205+0000 7f079eb54640 1 -- 172.21.15.55:0/3682382669 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f077c0676b0 msgr2=0x7f077c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:23.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:23.205+0000 7f079eb54640 1 --2- 172.21.15.55:0/3682382669 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f077c0676b0 0x7f077c069b70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f07840ad000 tx=0x7f07840a6000 comp rx=0 tx=0).stop 2024-03-19T01:57:23.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:23.205+0000 7f079eb54640 1 -- 172.21.15.55:0/3682382669 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07980c8750 msgr2=0x7f07981406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:23.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:23.205+0000 7f079eb54640 1 --2- 172.21.15.55:0/3682382669 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07980c8750 0x7f07981406b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f078c00c480 tx=0x7f078c00c940 comp rx=0 tx=0).stop 2024-03-19T01:57:23.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:23.205+0000 7f079eb54640 1 -- 172.21.15.55:0/3682382669 shutdown_connections 2024-03-19T01:57:23.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:23.205+0000 7f079eb54640 1 --2- 172.21.15.55:0/3682382669 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f077c0676b0 0x7f077c069b70 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:23.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:23.205+0000 7f079eb54640 1 --2- 172.21.15.55:0/3682382669 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07981546a0 0x7f07980ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:23.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:23.205+0000 7f079eb54640 1 --2- 172.21.15.55:0/3682382669 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07980ce020 0x7f079815b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:23.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:23.205+0000 7f079eb54640 1 --2- 172.21.15.55:0/3682382669 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07980c8750 0x7f07981406b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:23.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:23.205+0000 7f079eb54640 1 -- 172.21.15.55:0/3682382669 >> 172.21.15.55:0/3682382669 conn(0x7f07980714d0 msgr2=0x7f0798143c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:23.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:23.205+0000 7f079eb54640 1 -- 172.21.15.55:0/3682382669 shutdown_connections 2024-03-19T01:57:23.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:23.205+0000 7f079eb54640 1 -- 172.21.15.55:0/3682382669 wait complete. 2024-03-19T01:57:23.824 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:23.824 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:24.825 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:24.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.937+0000 7fd732535640 1 Processor -- start 2024-03-19T01:57:24.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.937+0000 7fd732535640 1 -- start start 2024-03-19T01:57:24.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.941+0000 7fd732535640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fd72c1544f0 0x7fd72c073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:24.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.941+0000 7fd732535640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd72c06b2a0 0x7fd72c15e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:24.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.941+0000 7fd732535640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd72c0c7fa0 con 0x7fd72c15acb0 2024-03-19T01:57:24.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.941+0000 7fd732535640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fd72c15b210 con 0x7fd72c1544f0 2024-03-19T01:57:24.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.941+0000 7fd732535640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd72c0c8240 con 0x7fd72c06b2a0 2024-03-19T01:57:24.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.941+0000 7fd730ad2640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fd72c15acb0 0x7fd72c0cce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:59900/0 (socket says 172.21.15.55:59900) 2024-03-19T01:57:24.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.941+0000 7fd730ad2640 1 -- 172.21.15.55:0/169884103 learned_addr learned my addr 172.21.15.55:0/169884103 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:57:24.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.941+0000 7fd72bfff640 1 --2- 172.21.15.55:0/169884103 >> v2:172.21.15.84:3300/0 conn(0x7fd72c1544f0 0x7fd72c073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:24.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.941+0000 7fd7237fe640 1 --2- 172.21.15.55:0/169884103 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd72c06b2a0 0x7fd72c15e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:24.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.941+0000 7fd72b7fe640 1 -- 172.21.15.55:0/169884103 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4184139690 0 0) 0x7fd72c0c7fa0 con 0x7fd72c15acb0 2024-03-19T01:57:24.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.941+0000 7fd72b7fe640 1 -- 172.21.15.55:0/169884103 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd718003530 con 0x7fd72c15acb0 2024-03-19T01:57:24.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.945+0000 7fd72bfff640 1 -- 172.21.15.55:0/169884103 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd72c06b2a0 msgr2=0x7fd72c15e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:24.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.945+0000 7fd72bfff640 1 --2- 172.21.15.55:0/169884103 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd72c06b2a0 0x7fd72c15e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:24.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.945+0000 7fd72bfff640 1 -- 172.21.15.55:0/169884103 >> v1:172.21.15.55:6789/0 conn(0x7fd72c15acb0 legacy=0x7fd72c0cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:24.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.945+0000 7fd72bfff640 1 -- 172.21.15.55:0/169884103 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72c15d920 con 0x7fd72c1544f0 2024-03-19T01:57:24.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.949+0000 7fd72bfff640 1 --2- 172.21.15.55:0/169884103 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd72c1544f0 0x7fd72c073540 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd71c0097c0 tx=0x7fd71c01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63ecb1eb01785dac server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:24.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.953+0000 7fd72b7fe640 1 -- 172.21.15.55:0/169884103 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c0b5070 con 0x7fd72c1544f0 2024-03-19T01:57:24.956 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.957+0000 7fd72b7fe640 1 -- 172.21.15.55:0/169884103 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd71c0a8990 con 0x7fd72c1544f0 2024-03-19T01:57:24.956 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.957+0000 7fd72b7fe640 1 -- 172.21.15.55:0/169884103 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c0a8d40 con 0x7fd72c1544f0 2024-03-19T01:57:24.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.957+0000 7fd732535640 1 -- 172.21.15.55:0/169884103 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd72c1544f0 msgr2=0x7fd72c073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:24.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.957+0000 7fd732535640 1 --2- 172.21.15.55:0/169884103 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd72c1544f0 0x7fd72c073540 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd71c0097c0 tx=0x7fd71c01ec90 comp rx=0 tx=0).stop 2024-03-19T01:57:24.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.957+0000 7fd732535640 1 -- 172.21.15.55:0/169884103 shutdown_connections 2024-03-19T01:57:24.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.957+0000 7fd732535640 1 --2- 172.21.15.55:0/169884103 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd72c06b2a0 0x7fd72c15e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:24.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.957+0000 7fd732535640 1 --2- 172.21.15.55:0/169884103 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd72c1544f0 0x7fd72c073540 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:24.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.957+0000 7fd732535640 1 -- 172.21.15.55:0/169884103 >> 172.21.15.55:0/169884103 conn(0x7fd72c06e670 msgr2=0x7fd72c06ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:24.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.957+0000 7fd732535640 1 -- 172.21.15.55:0/169884103 shutdown_connections 2024-03-19T01:57:24.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.957+0000 7fd732535640 1 -- 172.21.15.55:0/169884103 wait complete. 2024-03-19T01:57:24.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.957+0000 7fd732535640 1 Processor -- start 2024-03-19T01:57:24.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.957+0000 7fd732535640 1 -- start start 2024-03-19T01:57:24.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.961+0000 7fd732535640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd72c06b2a0 0x7fd72c15b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:24.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.961+0000 7fd732535640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd72c1544f0 0x7fd72c0c9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:24.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.961+0000 7fd732535640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd72c15acb0 0x7fd72c15b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:24.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.961+0000 7fd732535640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c1548d0 con 0x7fd72c06b2a0 2024-03-19T01:57:24.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.961+0000 7fd732535640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c15e660 con 0x7fd72c1544f0 2024-03-19T01:57:24.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.961+0000 7fd732535640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd72c0cd540 con 0x7fd72c15acb0 2024-03-19T01:57:24.960 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.961+0000 7fd72bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd72c06b2a0 0x7fd72c15b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:24.960 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.961+0000 7fd72bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd72c06b2a0 0x7fd72c15b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53488/0 (socket says 172.21.15.55:53488) 2024-03-19T01:57:24.960 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.961+0000 7fd72bfff640 1 -- 172.21.15.55:0/3495080946 learned_addr learned my addr 172.21.15.55:0/3495080946 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:24.960 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.961+0000 7fd730ad2640 1 --2- 172.21.15.55:0/3495080946 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd72c15acb0 0x7fd72c15b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:24.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.961+0000 7fd7237fe640 1 --2- 172.21.15.55:0/3495080946 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd72c1544f0 0x7fd72c0c9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:24.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.965+0000 7fd72bfff640 1 -- 172.21.15.55:0/3495080946 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd72c15acb0 msgr2=0x7fd72c15b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:24.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.965+0000 7fd72bfff640 1 --2- 172.21.15.55:0/3495080946 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd72c15acb0 0x7fd72c15b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:24.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.965+0000 7fd72bfff640 1 -- 172.21.15.55:0/3495080946 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd72c1544f0 msgr2=0x7fd72c0c9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:24.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.965+0000 7fd72bfff640 1 --2- 172.21.15.55:0/3495080946 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd72c1544f0 0x7fd72c0c9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:24.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.965+0000 7fd72bfff640 1 -- 172.21.15.55:0/3495080946 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72c1f4090 con 0x7fd72c06b2a0 2024-03-19T01:57:24.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.965+0000 7fd72bfff640 1 --2- 172.21.15.55:0/3495080946 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd72c06b2a0 0x7fd72c15b390 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd71c0b3040 tx=0x7fd71c01ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:24.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.973+0000 7fd7297fa640 1 -- 172.21.15.55:0/3495080946 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c0be070 con 0x7fd72c06b2a0 2024-03-19T01:57:24.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.973+0000 7fd7297fa640 1 -- 172.21.15.55:0/3495080946 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd71c004230 con 0x7fd72c06b2a0 2024-03-19T01:57:24.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.973+0000 7fd7297fa640 1 -- 172.21.15.55:0/3495080946 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c0b5040 con 0x7fd72c06b2a0 2024-03-19T01:57:24.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.973+0000 7fd732535640 1 -- 172.21.15.55:0/3495080946 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd72c1f10c0 con 0x7fd72c06b2a0 2024-03-19T01:57:24.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.973+0000 7fd732535640 1 -- 172.21.15.55:0/3495080946 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd72c1f0300 con 0x7fd72c06b2a0 2024-03-19T01:57:24.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.973+0000 7fd732535640 1 -- 172.21.15.55:0/3495080946 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd72c068a40 con 0x7fd72c06b2a0 2024-03-19T01:57:24.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.997+0000 7fd7297fa640 1 -- 172.21.15.55:0/3495080946 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fd71c0c1070 con 0x7fd72c06b2a0 2024-03-19T01:57:24.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.997+0000 7fd7297fa640 1 --2- 172.21.15.55:0/3495080946 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd70c0676b0 0x7fd70c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:24.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.997+0000 7fd7237fe640 1 --2- 172.21.15.55:0/3495080946 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd70c0676b0 0x7fd70c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:24.997 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:24.997+0000 7fd7237fe640 1 --2- 172.21.15.55:0/3495080946 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd70c0676b0 0x7fd70c069b70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd72c0c70b0 tx=0x7fd714009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:25.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:25.001+0000 7fd7297fa640 1 -- 172.21.15.55:0/3495080946 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fd71c0f0800 con 0x7fd72c06b2a0 2024-03-19T01:57:25.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:25.309+0000 7fd7297fa640 1 -- 172.21.15.55:0/3495080946 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fd71c0187f0 con 0x7fd72c06b2a0 2024-03-19T01:57:25.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:25.557+0000 7fd732535640 1 -- 172.21.15.55:0/3495080946 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd72c0c7be0 con 0x7fd72c06b2a0 2024-03-19T01:57:25.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:25.673+0000 7fd7297fa640 1 -- 172.21.15.55:0/3495080946 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fd71c0f2760 con 0x7fd72c06b2a0 2024-03-19T01:57:25.675 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:25.675 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:25.675 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:25.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:25.677+0000 7fd732535640 1 -- 172.21.15.55:0/3495080946 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd70c0676b0 msgr2=0x7fd70c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:25.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:25.677+0000 7fd732535640 1 --2- 172.21.15.55:0/3495080946 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd70c0676b0 0x7fd70c069b70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd72c0c70b0 tx=0x7fd714009190 comp rx=0 tx=0).stop 2024-03-19T01:57:25.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:25.677+0000 7fd732535640 1 -- 172.21.15.55:0/3495080946 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd72c06b2a0 msgr2=0x7fd72c15b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:25.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:25.677+0000 7fd732535640 1 --2- 172.21.15.55:0/3495080946 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd72c06b2a0 0x7fd72c15b390 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd71c0b3040 tx=0x7fd71c01ba30 comp rx=0 tx=0).stop 2024-03-19T01:57:25.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:25.677+0000 7fd732535640 1 -- 172.21.15.55:0/3495080946 shutdown_connections 2024-03-19T01:57:25.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:25.677+0000 7fd732535640 1 --2- 172.21.15.55:0/3495080946 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd70c0676b0 0x7fd70c069b70 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:25.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:25.677+0000 7fd732535640 1 --2- 172.21.15.55:0/3495080946 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd72c15acb0 0x7fd72c15b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:25.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:25.677+0000 7fd732535640 1 --2- 172.21.15.55:0/3495080946 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd72c1544f0 0x7fd72c0c9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:25.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:25.677+0000 7fd732535640 1 --2- 172.21.15.55:0/3495080946 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd72c06b2a0 0x7fd72c15b390 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:25.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:25.677+0000 7fd732535640 1 -- 172.21.15.55:0/3495080946 >> 172.21.15.55:0/3495080946 conn(0x7fd72c06e670 msgr2=0x7fd72c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:25.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:25.677+0000 7fd732535640 1 -- 172.21.15.55:0/3495080946 shutdown_connections 2024-03-19T01:57:25.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:25.677+0000 7fd732535640 1 -- 172.21.15.55:0/3495080946 wait complete. 2024-03-19T01:57:26.301 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:26.301 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:27.302 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:27.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.425+0000 7f95f8c63640 1 Processor -- start 2024-03-19T01:57:27.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.425+0000 7f95f8c63640 1 -- start start 2024-03-19T01:57:27.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.425+0000 7f95f8c63640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95f415acb0 0x7f95f4073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:27.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.425+0000 7f95f8c63640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95f40c7fa0 con 0x7f95f406b2a0 2024-03-19T01:57:27.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.425+0000 7f95f8c63640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95f40c8240 con 0x7f95f41544f0 2024-03-19T01:57:27.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.425+0000 7f95f8c63640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f95f415b210 con 0x7f95f415acb0 2024-03-19T01:57:27.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.425+0000 7f95f1d9b640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f95f406b2a0 0x7f95f40cce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:59910/0 (socket says 172.21.15.55:59910) 2024-03-19T01:57:27.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.425+0000 7f95f1d9b640 1 -- 172.21.15.55:0/1108862019 learned_addr learned my addr 172.21.15.55:0/1108862019 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:57:27.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.429+0000 7f95f2d9d640 1 --2- 172.21.15.55:0/1108862019 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95f415acb0 0x7f95f4073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:27.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.429+0000 7f95f159a640 1 -- 172.21.15.55:0/1108862019 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1247117883 0 0) 0x7f95f40c7fa0 con 0x7f95f406b2a0 2024-03-19T01:57:27.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.429+0000 7f95f159a640 1 -- 172.21.15.55:0/1108862019 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95dc003530 con 0x7f95f406b2a0 2024-03-19T01:57:27.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.433+0000 7f95f159a640 1 -- 172.21.15.55:0/1108862019 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 840299432 0 0) 0x7f95f40c8240 con 0x7f95f41544f0 2024-03-19T01:57:27.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.433+0000 7f95f159a640 1 -- 172.21.15.55:0/1108862019 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95f40c7fa0 con 0x7f95f41544f0 2024-03-19T01:57:27.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.433+0000 7f95f159a640 1 -- 172.21.15.55:0/1108862019 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 395064877 0 0) 0x7f95dc003530 con 0x7f95f406b2a0 2024-03-19T01:57:27.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.433+0000 7f95f159a640 1 -- 172.21.15.55:0/1108862019 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95f415acb0 msgr2=0x7f95f4073540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:27.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.433+0000 7f95f159a640 1 --2- 172.21.15.55:0/1108862019 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95f415acb0 0x7f95f4073540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:27.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.433+0000 7f95f159a640 1 -- 172.21.15.55:0/1108862019 >> v1:172.21.15.84:6789/0 conn(0x7f95f41544f0 legacy=0x7f95f415e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:27.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.433+0000 7f95f159a640 1 -- 172.21.15.55:0/1108862019 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95f415d920 con 0x7f95f406b2a0 2024-03-19T01:57:27.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.433+0000 7f95f159a640 1 -- 172.21.15.55:0/1108862019 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f95e80016a0 con 0x7f95f406b2a0 2024-03-19T01:57:27.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.433+0000 7f95f159a640 1 -- 172.21.15.55:0/1108862019 >> v1:172.21.15.55:6789/0 conn(0x7f95f406b2a0 legacy=0x7f95f40cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:27.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.433+0000 7f95f159a640 1 --2- 172.21.15.55:0/1108862019 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95dc003b80 0x7f95dc003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:27.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.433+0000 7f95f159a640 1 --2- 172.21.15.55:0/1108862019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f95dc004810 0x7f95dc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:27.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.433+0000 7f95f159a640 1 --2- 172.21.15.55:0/1108862019 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95f415acb0 0x7f95dc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:27.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.433+0000 7f95f159a640 1 -- 172.21.15.55:0/1108862019 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f95f415b210 con 0x7f95dc003b80 2024-03-19T01:57:27.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.433+0000 7f95f159a640 1 -- 172.21.15.55:0/1108862019 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f95dc00bab0 con 0x7f95dc004810 2024-03-19T01:57:27.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.433+0000 7f95f159a640 1 -- 172.21.15.55:0/1108862019 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f95dc00bcb0 con 0x7f95f415acb0 2024-03-19T01:57:27.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.437+0000 7f95f259c640 1 --2- 172.21.15.55:0/1108862019 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95dc003b80 0x7f95dc003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:27.439 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.441+0000 7f95f2d9d640 1 --2- 172.21.15.55:0/1108862019 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95f415acb0 0x7f95dc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:27.440 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.441+0000 7f95f1d9b640 1 --2- 172.21.15.55:0/1108862019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f95dc004810 0x7f95dc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:27.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.445+0000 7f95f259c640 1 -- 172.21.15.55:0/1108862019 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95f415acb0 msgr2=0x7f95dc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:27.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.445+0000 7f95f259c640 1 --2- 172.21.15.55:0/1108862019 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95f415acb0 0x7f95dc0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:27.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.445+0000 7f95f259c640 1 -- 172.21.15.55:0/1108862019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f95dc004810 msgr2=0x7f95dc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:27.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.445+0000 7f95f259c640 1 --2- 172.21.15.55:0/1108862019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f95dc004810 0x7f95dc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:27.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.445+0000 7f95f259c640 1 -- 172.21.15.55:0/1108862019 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95dc003530 con 0x7f95dc003b80 2024-03-19T01:57:27.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.445+0000 7f95f259c640 1 --2- 172.21.15.55:0/1108862019 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95dc003b80 0x7f95dc003f80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f95e40018d0 tx=0x7f95e401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf590dc1e1fd54b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:27.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.453+0000 7f95f159a640 1 -- 172.21.15.55:0/1108862019 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e4001520 con 0x7f95dc003b80 2024-03-19T01:57:27.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.453+0000 7f95f159a640 1 -- 172.21.15.55:0/1108862019 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95e40052f0 con 0x7f95dc003b80 2024-03-19T01:57:27.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.453+0000 7f95f159a640 1 -- 172.21.15.55:0/1108862019 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e40a8950 con 0x7f95dc003b80 2024-03-19T01:57:27.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.453+0000 7f95f8c63640 1 -- 172.21.15.55:0/1108862019 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95dc003b80 msgr2=0x7f95dc003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:27.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.453+0000 7f95f8c63640 1 --2- 172.21.15.55:0/1108862019 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95dc003b80 0x7f95dc003f80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f95e40018d0 tx=0x7f95e401ecf0 comp rx=0 tx=0).stop 2024-03-19T01:57:27.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.453+0000 7f95f2d9d640 1 -- 172.21.15.55:0/1108862019 reap_dead start 2024-03-19T01:57:27.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.453+0000 7f95f8c63640 1 -- 172.21.15.55:0/1108862019 shutdown_connections 2024-03-19T01:57:27.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.453+0000 7f95f8c63640 1 -- 172.21.15.55:0/1108862019 >> 172.21.15.55:0/1108862019 conn(0x7f95f406e670 msgr2=0x7f95f406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:27.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.453+0000 7f95f8c63640 1 -- 172.21.15.55:0/1108862019 shutdown_connections 2024-03-19T01:57:27.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.453+0000 7f95f8c63640 1 -- 172.21.15.55:0/1108862019 wait complete. 2024-03-19T01:57:27.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.453+0000 7f95f8c63640 1 Processor -- start 2024-03-19T01:57:27.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.453+0000 7f95f8c63640 1 -- start start 2024-03-19T01:57:27.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.453+0000 7f95f8c63640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95dc003b80 0x7f95f415cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:27.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.453+0000 7f95f8c63640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95f40cbc00 0x7f95f40706a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:27.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.453+0000 7f95f8c63640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f95f414d920 0x7f95f414dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:27.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.453+0000 7f95f8c63640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f95f40cd540 con 0x7f95dc003b80 2024-03-19T01:57:27.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.453+0000 7f95f8c63640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f95f41548d0 con 0x7f95f414d920 2024-03-19T01:57:27.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.453+0000 7f95f8c63640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f95f40cd6c0 con 0x7f95f40cbc00 2024-03-19T01:57:27.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.457+0000 7f95f259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95dc003b80 0x7f95f415cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:27.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.457+0000 7f95f259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95dc003b80 0x7f95f415cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53498/0 (socket says 172.21.15.55:53498) 2024-03-19T01:57:27.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.457+0000 7f95f259c640 1 -- 172.21.15.55:0/3345469316 learned_addr learned my addr 172.21.15.55:0/3345469316 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:27.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.457+0000 7f95f1d9b640 1 --2- 172.21.15.55:0/3345469316 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95f40cbc00 0x7f95f40706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:27.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.457+0000 7f95f2d9d640 1 --2- 172.21.15.55:0/3345469316 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f95f414d920 0x7f95f414dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:27.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.461+0000 7f95f259c640 1 -- 172.21.15.55:0/3345469316 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95f40cbc00 msgr2=0x7f95f40706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:27.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.461+0000 7f95f259c640 1 --2- 172.21.15.55:0/3345469316 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95f40cbc00 0x7f95f40706a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:27.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.461+0000 7f95f259c640 1 -- 172.21.15.55:0/3345469316 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f95f414d920 msgr2=0x7f95f414dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:27.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.461+0000 7f95f259c640 1 --2- 172.21.15.55:0/3345469316 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f95f414d920 0x7f95f414dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:27.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.461+0000 7f95f259c640 1 -- 172.21.15.55:0/3345469316 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95f415b760 con 0x7f95dc003b80 2024-03-19T01:57:27.461 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.461+0000 7f95f259c640 1 --2- 172.21.15.55:0/3345469316 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95dc003b80 0x7f95f415cfd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f95e4005040 tx=0x7f95dc005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:27.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.465+0000 7f95e2ffd640 1 -- 172.21.15.55:0/3345469316 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e40c0070 con 0x7f95dc003b80 2024-03-19T01:57:27.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.465+0000 7f95e2ffd640 1 -- 172.21.15.55:0/3345469316 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95e4001500 con 0x7f95dc003b80 2024-03-19T01:57:27.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.465+0000 7f95e2ffd640 1 -- 172.21.15.55:0/3345469316 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e40054a0 con 0x7f95dc003b80 2024-03-19T01:57:27.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.465+0000 7f95f8c63640 1 -- 172.21.15.55:0/3345469316 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95f415bd40 con 0x7f95dc003b80 2024-03-19T01:57:27.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.465+0000 7f95f8c63640 1 -- 172.21.15.55:0/3345469316 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95f415c300 con 0x7f95dc003b80 2024-03-19T01:57:27.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.465+0000 7f95f8c63640 1 -- 172.21.15.55:0/3345469316 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95f4068a40 con 0x7f95dc003b80 2024-03-19T01:57:27.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.757+0000 7f95e2ffd640 1 -- 172.21.15.55:0/3345469316 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f95e40c3070 con 0x7f95dc003b80 2024-03-19T01:57:27.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.757+0000 7f95e2ffd640 1 --2- 172.21.15.55:0/3345469316 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f95d4067690 0x7f95d4069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:27.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.757+0000 7f95f1d9b640 1 --2- 172.21.15.55:0/3345469316 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f95d4067690 0x7f95d4069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:27.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.757+0000 7f95f1d9b640 1 --2- 172.21.15.55:0/3345469316 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f95d4067690 0x7f95d4069b50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f95e8000c10 tx=0x7f95e8007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:27.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.797+0000 7f95e2ffd640 1 -- 172.21.15.55:0/3345469316 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f95e40f0ea0 con 0x7f95dc003b80 2024-03-19T01:57:27.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:27.825+0000 7f95e2ffd640 1 -- 172.21.15.55:0/3345469316 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f95e40ca020 con 0x7f95dc003b80 2024-03-19T01:57:28.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:28.041+0000 7f95f8c63640 1 -- 172.21.15.55:0/3345469316 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95f4071310 con 0x7f95dc003b80 2024-03-19T01:57:28.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:28.061+0000 7f95e2ffd640 1 -- 172.21.15.55:0/3345469316 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f95e4018a00 con 0x7f95dc003b80 2024-03-19T01:57:28.062 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:28.062 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:28.062 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:28.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:28.065+0000 7f95f8c63640 1 -- 172.21.15.55:0/3345469316 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f95d4067690 msgr2=0x7f95d4069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:28.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:28.065+0000 7f95f8c63640 1 --2- 172.21.15.55:0/3345469316 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f95d4067690 0x7f95d4069b50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f95e8000c10 tx=0x7f95e8007040 comp rx=0 tx=0).stop 2024-03-19T01:57:28.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:28.065+0000 7f95f8c63640 1 -- 172.21.15.55:0/3345469316 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95dc003b80 msgr2=0x7f95f415cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:28.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:28.065+0000 7f95f8c63640 1 --2- 172.21.15.55:0/3345469316 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95dc003b80 0x7f95f415cfd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f95e4005040 tx=0x7f95dc005840 comp rx=0 tx=0).stop 2024-03-19T01:57:28.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:28.065+0000 7f95f8c63640 1 -- 172.21.15.55:0/3345469316 shutdown_connections 2024-03-19T01:57:28.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:28.065+0000 7f95f8c63640 1 --2- 172.21.15.55:0/3345469316 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f95d4067690 0x7f95d4069b50 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:28.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:28.065+0000 7f95f8c63640 1 --2- 172.21.15.55:0/3345469316 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f95f414d920 0x7f95f414dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:28.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:28.065+0000 7f95f8c63640 1 --2- 172.21.15.55:0/3345469316 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95f40cbc00 0x7f95f40706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:28.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:28.065+0000 7f95f8c63640 1 --2- 172.21.15.55:0/3345469316 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95dc003b80 0x7f95f415cfd0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:28.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:28.065+0000 7f95f8c63640 1 -- 172.21.15.55:0/3345469316 >> 172.21.15.55:0/3345469316 conn(0x7f95f406e670 msgr2=0x7f95f406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:28.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:28.065+0000 7f95f8c63640 1 -- 172.21.15.55:0/3345469316 shutdown_connections 2024-03-19T01:57:28.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:28.065+0000 7f95f8c63640 1 -- 172.21.15.55:0/3345469316 wait complete. 2024-03-19T01:57:28.702 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:28.703 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:29.703 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:29.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.829+0000 7f33bdbcc640 1 Processor -- start 2024-03-19T01:57:29.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.829+0000 7f33bdbcc640 1 -- start start 2024-03-19T01:57:29.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.829+0000 7f33bdbcc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f33b80c5b70 0x7f33b815b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:29.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.829+0000 7f33bdbcc640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33b806b3b0 con 0x7f33b8067000 2024-03-19T01:57:29.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.829+0000 7f33bdbcc640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33b806b690 con 0x7f33b80c5f50 2024-03-19T01:57:29.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.829+0000 7f33bdbcc640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f33b8158370 con 0x7f33b80c5b70 2024-03-19T01:57:29.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.833+0000 7f33b7fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f33b8067000 0x7f33b81629c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:56068/0 (socket says 172.21.15.55:56068) 2024-03-19T01:57:29.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.833+0000 7f33b7fff640 1 -- 172.21.15.55:0/2148517596 learned_addr learned my addr 172.21.15.55:0/2148517596 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:57:29.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.833+0000 7f33b77fe640 1 --2- 172.21.15.55:0/2148517596 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f33b80c5b70 0x7f33b815b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:29.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.837+0000 7f33b77fe640 1 -- 172.21.15.55:0/2148517596 >> v1:172.21.15.84:6789/0 conn(0x7f33b80c5f50 legacy=0x7f33b80c2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:29.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.837+0000 7f33b77fe640 1 -- 172.21.15.55:0/2148517596 >> v1:172.21.15.55:6789/0 conn(0x7f33b8067000 legacy=0x7f33b81629c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:29.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.837+0000 7f33b77fe640 1 -- 172.21.15.55:0/2148517596 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33b8158650 con 0x7f33b80c5b70 2024-03-19T01:57:29.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.841+0000 7f33b77fe640 1 --2- 172.21.15.55:0/2148517596 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f33b80c5b70 0x7f33b815b630 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f33a80097f0 tx=0x7f33a801ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5d3f6f6a6a8e22a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:29.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.845+0000 7f33b67fc640 1 -- 172.21.15.55:0/2148517596 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a80b5070 con 0x7f33b80c5b70 2024-03-19T01:57:29.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.845+0000 7f33b67fc640 1 -- 172.21.15.55:0/2148517596 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33a80a89d0 con 0x7f33b80c5b70 2024-03-19T01:57:29.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.845+0000 7f33b67fc640 1 -- 172.21.15.55:0/2148517596 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a80a8da0 con 0x7f33b80c5b70 2024-03-19T01:57:29.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.849+0000 7f33bdbcc640 1 -- 172.21.15.55:0/2148517596 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f33b80c5b70 msgr2=0x7f33b815b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:29.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.849+0000 7f33bdbcc640 1 --2- 172.21.15.55:0/2148517596 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f33b80c5b70 0x7f33b815b630 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f33a80097f0 tx=0x7f33a801ecd0 comp rx=0 tx=0).stop 2024-03-19T01:57:29.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.849+0000 7f33bdbcc640 1 -- 172.21.15.55:0/2148517596 shutdown_connections 2024-03-19T01:57:29.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.849+0000 7f33bdbcc640 1 --2- 172.21.15.55:0/2148517596 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f33b80c5b70 0x7f33b815b630 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:29.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.849+0000 7f33bdbcc640 1 -- 172.21.15.55:0/2148517596 >> 172.21.15.55:0/2148517596 conn(0x7f33b80cc200 msgr2=0x7f33b80cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:29.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.849+0000 7f33bdbcc640 1 -- 172.21.15.55:0/2148517596 shutdown_connections 2024-03-19T01:57:29.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.849+0000 7f33bdbcc640 1 -- 172.21.15.55:0/2148517596 wait complete. 2024-03-19T01:57:29.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.849+0000 7f33bdbcc640 1 Processor -- start 2024-03-19T01:57:29.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.849+0000 7f33bdbcc640 1 -- start start 2024-03-19T01:57:29.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.849+0000 7f33bdbcc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f33b8067000 0x7f33b80cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:29.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.849+0000 7f33bdbcc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33b80c5b70 0x7f33b814fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:29.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.849+0000 7f33bdbcc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33b80c5f50 0x7f33b814ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:29.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.849+0000 7f33bdbcc640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f33b8158e50 con 0x7f33b80c5b70 2024-03-19T01:57:29.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.849+0000 7f33bdbcc640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f33b8158040 con 0x7f33b80c5f50 2024-03-19T01:57:29.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.849+0000 7f33bdbcc640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f33b81f20c0 con 0x7f33b8067000 2024-03-19T01:57:29.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.849+0000 7f33b6ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33b80c5b70 0x7f33b814fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:29.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.849+0000 7f33b77fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f33b8067000 0x7f33b80cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:29.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.849+0000 7f33b6ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33b80c5b70 0x7f33b814fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:37638/0 (socket says 172.21.15.55:37638) 2024-03-19T01:57:29.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.849+0000 7f33b6ffd640 1 -- 172.21.15.55:0/3841426747 learned_addr learned my addr 172.21.15.55:0/3841426747 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:29.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.853+0000 7f33b7fff640 1 --2- 172.21.15.55:0/3841426747 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33b80c5f50 0x7f33b814ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:29.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.853+0000 7f33b6ffd640 1 -- 172.21.15.55:0/3841426747 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f33b8067000 msgr2=0x7f33b80cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:29.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.853+0000 7f33b6ffd640 1 --2- 172.21.15.55:0/3841426747 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f33b8067000 0x7f33b80cb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:29.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.853+0000 7f33b6ffd640 1 -- 172.21.15.55:0/3841426747 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33b80c5f50 msgr2=0x7f33b814ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:29.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.853+0000 7f33b6ffd640 1 --2- 172.21.15.55:0/3841426747 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33b80c5f50 0x7f33b814ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:29.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.853+0000 7f33b6ffd640 1 -- 172.21.15.55:0/3841426747 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33b8152b10 con 0x7f33b80c5b70 2024-03-19T01:57:29.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.857+0000 7f33b6ffd640 1 --2- 172.21.15.55:0/3841426747 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33b80c5b70 0x7f33b814fa40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f33ac00c4d0 tx=0x7f33ac00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:29.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.861+0000 7f339ffff640 1 -- 172.21.15.55:0/3841426747 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ac017070 con 0x7f33b80c5b70 2024-03-19T01:57:29.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.861+0000 7f339ffff640 1 -- 172.21.15.55:0/3841426747 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33ac0052d0 con 0x7f33b80c5b70 2024-03-19T01:57:29.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.861+0000 7f339ffff640 1 -- 172.21.15.55:0/3841426747 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ac003800 con 0x7f33b80c5b70 2024-03-19T01:57:29.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.861+0000 7f33bdbcc640 1 -- 172.21.15.55:0/3841426747 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33b8152db0 con 0x7f33b80c5b70 2024-03-19T01:57:29.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.861+0000 7f33bdbcc640 1 -- 172.21.15.55:0/3841426747 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33b814c3d0 con 0x7f33b80c5b70 2024-03-19T01:57:29.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.861+0000 7f33bdbcc640 1 -- 172.21.15.55:0/3841426747 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33b8068a40 con 0x7f33b80c5b70 2024-03-19T01:57:29.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.893+0000 7f339ffff640 1 -- 172.21.15.55:0/3841426747 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f33ac01a070 con 0x7f33b80c5b70 2024-03-19T01:57:29.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.893+0000 7f339ffff640 1 --2- 172.21.15.55:0/3841426747 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f33a40676b0 0x7f33a4069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:29.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.897+0000 7f33b77fe640 1 --2- 172.21.15.55:0/3841426747 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f33a40676b0 0x7f33a4069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:29.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:29.897+0000 7f33b77fe640 1 --2- 172.21.15.55:0/3841426747 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f33a40676b0 0x7f33a4069b70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f33a80b2000 tx=0x7f33a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:30.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:30.169+0000 7f339ffff640 1 -- 172.21.15.55:0/3841426747 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f33ac053720 con 0x7f33b80c5b70 2024-03-19T01:57:30.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:30.169+0000 7f339ffff640 1 -- 172.21.15.55:0/3841426747 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f33ac005480 con 0x7f33b80c5b70 2024-03-19T01:57:30.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:30.401+0000 7f33bdbcc640 1 -- 172.21.15.55:0/3841426747 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33b8158fd0 con 0x7f33b80c5b70 2024-03-19T01:57:30.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:30.417+0000 7f339ffff640 1 -- 172.21.15.55:0/3841426747 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f33ac055680 con 0x7f33b80c5b70 2024-03-19T01:57:30.416 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:30.416 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:30.417 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:30.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:30.421+0000 7f33bdbcc640 1 -- 172.21.15.55:0/3841426747 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f33a40676b0 msgr2=0x7f33a4069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:30.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:30.421+0000 7f33bdbcc640 1 --2- 172.21.15.55:0/3841426747 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f33a40676b0 0x7f33a4069b70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f33a80b2000 tx=0x7f33a8008040 comp rx=0 tx=0).stop 2024-03-19T01:57:30.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:30.421+0000 7f33bdbcc640 1 -- 172.21.15.55:0/3841426747 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33b80c5b70 msgr2=0x7f33b814fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:30.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:30.421+0000 7f33bdbcc640 1 --2- 172.21.15.55:0/3841426747 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33b80c5b70 0x7f33b814fa40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f33ac00c4d0 tx=0x7f33ac00c990 comp rx=0 tx=0).stop 2024-03-19T01:57:30.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:30.421+0000 7f33bdbcc640 1 -- 172.21.15.55:0/3841426747 shutdown_connections 2024-03-19T01:57:30.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:30.421+0000 7f33bdbcc640 1 --2- 172.21.15.55:0/3841426747 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f33a40676b0 0x7f33a4069b70 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:30.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:30.421+0000 7f33bdbcc640 1 --2- 172.21.15.55:0/3841426747 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33b80c5f50 0x7f33b814ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:30.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:30.421+0000 7f33bdbcc640 1 --2- 172.21.15.55:0/3841426747 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f33b80c5b70 0x7f33b814fa40 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:30.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:30.421+0000 7f33bdbcc640 1 --2- 172.21.15.55:0/3841426747 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f33b8067000 0x7f33b80cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:30.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:30.421+0000 7f33bdbcc640 1 -- 172.21.15.55:0/3841426747 >> 172.21.15.55:0/3841426747 conn(0x7f33b80cc200 msgr2=0x7f33b81571a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:30.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:30.421+0000 7f33bdbcc640 1 -- 172.21.15.55:0/3841426747 shutdown_connections 2024-03-19T01:57:30.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:30.421+0000 7f33bdbcc640 1 -- 172.21.15.55:0/3841426747 wait complete. 2024-03-19T01:57:31.040 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:31.040 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:32.041 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:32.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.161+0000 7fa457a7a640 1 Processor -- start 2024-03-19T01:57:32.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.161+0000 7fa457a7a640 1 -- start start 2024-03-19T01:57:32.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.161+0000 7fa457a7a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa450150f50 0x7fa45015b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:32.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.161+0000 7fa457a7a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa4500ca350 0x7fa4500728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:32.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.161+0000 7fa457a7a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa450151330 con 0x7fa4500c3320 2024-03-19T01:57:32.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.161+0000 7fa457a7a640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fa4500c3e00 con 0x7fa450150f50 2024-03-19T01:57:32.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.161+0000 7fa457a7a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa45015ac10 con 0x7fa4500ca350 2024-03-19T01:57:32.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.161+0000 7fa455015640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa4500ca350 0x7fa4500728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:32.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.165+0000 7fa455816640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa450150f50 0x7fa45015b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:32.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.165+0000 7fa455015640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa4500ca350 0x7fa4500728b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:60338/0 (socket says 172.21.15.55:60338) 2024-03-19T01:57:32.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.165+0000 7fa455015640 1 -- 172.21.15.55:0/2070426198 learned_addr learned my addr 172.21.15.55:0/2070426198 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:32.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.165+0000 7fa455015640 1 -- 172.21.15.55:0/2070426198 >> v2:172.21.15.84:3300/0 conn(0x7fa450150f50 msgr2=0x7fa45015b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:32.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.165+0000 7fa455015640 1 --2- 172.21.15.55:0/2070426198 >> v2:172.21.15.84:3300/0 conn(0x7fa450150f50 0x7fa45015b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:32.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.165+0000 7fa455015640 1 -- 172.21.15.55:0/2070426198 >> v1:172.21.15.84:6789/0 conn(0x7fa4500c3320 legacy=0x7fa450157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:32.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.165+0000 7fa455015640 1 -- 172.21.15.55:0/2070426198 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4500c3840 con 0x7fa4500ca350 2024-03-19T01:57:32.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.173+0000 7fa455015640 1 --2- 172.21.15.55:0/2070426198 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa4500ca350 0x7fa4500728b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa440009870 tx=0x7fa44001c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90686f2e2f170cf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:32.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.177+0000 7fa454814640 1 -- 172.21.15.55:0/2070426198 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4400a9620 con 0x7fa4500ca350 2024-03-19T01:57:32.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.177+0000 7fa454814640 1 -- 172.21.15.55:0/2070426198 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4400a97c0 con 0x7fa4500ca350 2024-03-19T01:57:32.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.177+0000 7fa454814640 1 -- 172.21.15.55:0/2070426198 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4400a9b90 con 0x7fa4500ca350 2024-03-19T01:57:32.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.177+0000 7fa457a7a640 1 -- 172.21.15.55:0/2070426198 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa4500ca350 msgr2=0x7fa4500728b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:32.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.177+0000 7fa457a7a640 1 --2- 172.21.15.55:0/2070426198 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa4500ca350 0x7fa4500728b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa440009870 tx=0x7fa44001c560 comp rx=0 tx=0).stop 2024-03-19T01:57:32.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.177+0000 7fa457a7a640 1 -- 172.21.15.55:0/2070426198 shutdown_connections 2024-03-19T01:57:32.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.177+0000 7fa457a7a640 1 --2- 172.21.15.55:0/2070426198 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa4500ca350 0x7fa4500728b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:32.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.177+0000 7fa457a7a640 1 --2- 172.21.15.55:0/2070426198 >> v2:172.21.15.84:3300/0 conn(0x7fa450150f50 0x7fa45015b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:32.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.177+0000 7fa457a7a640 1 -- 172.21.15.55:0/2070426198 >> 172.21.15.55:0/2070426198 conn(0x7fa45006e670 msgr2=0x7fa45006ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:32.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.177+0000 7fa457a7a640 1 -- 172.21.15.55:0/2070426198 shutdown_connections 2024-03-19T01:57:32.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.177+0000 7fa457a7a640 1 -- 172.21.15.55:0/2070426198 wait complete. 2024-03-19T01:57:32.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.181+0000 7fa457a7a640 1 Processor -- start 2024-03-19T01:57:32.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.181+0000 7fa457a7a640 1 -- start start 2024-03-19T01:57:32.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.181+0000 7fa457a7a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4500c3320 0x7fa450144650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:32.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.181+0000 7fa457a7a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4500ca350 0x7fa450144b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:32.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.181+0000 7fa457a7a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa450150f50 0x7fa450146600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:32.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.181+0000 7fa457a7a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa4501639b0 con 0x7fa4500ca350 2024-03-19T01:57:32.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.181+0000 7fa457a7a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa450163830 con 0x7fa4500c3320 2024-03-19T01:57:32.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.181+0000 7fa457a7a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa450163b30 con 0x7fa450150f50 2024-03-19T01:57:32.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.181+0000 7fa455015640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4500ca350 0x7fa450144b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:32.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.181+0000 7fa456017640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa450150f50 0x7fa450146600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:32.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.181+0000 7fa455816640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4500c3320 0x7fa450144650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:32.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.181+0000 7fa455015640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4500ca350 0x7fa450144b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:37652/0 (socket says 172.21.15.55:37652) 2024-03-19T01:57:32.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.181+0000 7fa455015640 1 -- 172.21.15.55:0/4100871322 learned_addr learned my addr 172.21.15.55:0/4100871322 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:32.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.189+0000 7fa455015640 1 -- 172.21.15.55:0/4100871322 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa450150f50 msgr2=0x7fa450146600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:32.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.189+0000 7fa455015640 1 --2- 172.21.15.55:0/4100871322 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa450150f50 0x7fa450146600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:32.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.189+0000 7fa455015640 1 -- 172.21.15.55:0/4100871322 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4500c3320 msgr2=0x7fa450144650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:32.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.189+0000 7fa455015640 1 --2- 172.21.15.55:0/4100871322 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4500c3320 0x7fa450144650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:32.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.189+0000 7fa455015640 1 -- 172.21.15.55:0/4100871322 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa450149880 con 0x7fa4500ca350 2024-03-19T01:57:32.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.193+0000 7fa455015640 1 --2- 172.21.15.55:0/4100871322 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4500ca350 0x7fa450144b90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa4400ab040 tx=0x7fa44001ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:32.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.197+0000 7fa4467fc640 1 -- 172.21.15.55:0/4100871322 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4400bc450 con 0x7fa4500ca350 2024-03-19T01:57:32.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.197+0000 7fa4467fc640 1 -- 172.21.15.55:0/4100871322 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4400bc610 con 0x7fa4500ca350 2024-03-19T01:57:32.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.197+0000 7fa4467fc640 1 -- 172.21.15.55:0/4100871322 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4400bca00 con 0x7fa4500ca350 2024-03-19T01:57:32.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.197+0000 7fa457a7a640 1 -- 172.21.15.55:0/4100871322 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa450148950 con 0x7fa4500ca350 2024-03-19T01:57:32.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.197+0000 7fa457a7a640 1 -- 172.21.15.55:0/4100871322 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa45014a540 con 0x7fa4500ca350 2024-03-19T01:57:32.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.197+0000 7fa457a7a640 1 -- 172.21.15.55:0/4100871322 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa450068a40 con 0x7fa4500ca350 2024-03-19T01:57:32.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.229+0000 7fa4467fc640 1 -- 172.21.15.55:0/4100871322 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa4400c53f0 con 0x7fa4500ca350 2024-03-19T01:57:32.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.229+0000 7fa4467fc640 1 --2- 172.21.15.55:0/4100871322 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa4340676b0 0x7fa434069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:32.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.229+0000 7fa455816640 1 --2- 172.21.15.55:0/4100871322 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa4340676b0 0x7fa434069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:32.229 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.229+0000 7fa455816640 1 --2- 172.21.15.55:0/4100871322 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa4340676b0 0x7fa434069b70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa44c0079c0 tx=0x7fa44c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:32.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.545+0000 7fa4467fc640 1 -- 172.21.15.55:0/4100871322 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fa4400a9760 con 0x7fa4500ca350 2024-03-19T01:57:32.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.545+0000 7fa4467fc640 1 -- 172.21.15.55:0/4100871322 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa4400ad020 con 0x7fa4500ca350 2024-03-19T01:57:32.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.757+0000 7fa457a7a640 1 -- 172.21.15.55:0/4100871322 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4500c7600 con 0x7fa4500ca350 2024-03-19T01:57:32.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.861+0000 7fa4467fc640 1 -- 172.21.15.55:0/4100871322 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fa440016150 con 0x7fa4500ca350 2024-03-19T01:57:32.859 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:32.859 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:32.859 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:32.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.865+0000 7fa457a7a640 1 -- 172.21.15.55:0/4100871322 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa4340676b0 msgr2=0x7fa434069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:32.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.865+0000 7fa457a7a640 1 --2- 172.21.15.55:0/4100871322 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa4340676b0 0x7fa434069b70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa44c0079c0 tx=0x7fa44c008040 comp rx=0 tx=0).stop 2024-03-19T01:57:32.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.865+0000 7fa457a7a640 1 -- 172.21.15.55:0/4100871322 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4500ca350 msgr2=0x7fa450144b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:32.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.865+0000 7fa457a7a640 1 --2- 172.21.15.55:0/4100871322 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4500ca350 0x7fa450144b90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa4400ab040 tx=0x7fa44001ada0 comp rx=0 tx=0).stop 2024-03-19T01:57:32.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.865+0000 7fa457a7a640 1 -- 172.21.15.55:0/4100871322 shutdown_connections 2024-03-19T01:57:32.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.865+0000 7fa457a7a640 1 --2- 172.21.15.55:0/4100871322 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa4340676b0 0x7fa434069b70 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:32.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.865+0000 7fa457a7a640 1 --2- 172.21.15.55:0/4100871322 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa450150f50 0x7fa450146600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:32.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.865+0000 7fa457a7a640 1 --2- 172.21.15.55:0/4100871322 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4500ca350 0x7fa450144b90 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:32.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.865+0000 7fa457a7a640 1 --2- 172.21.15.55:0/4100871322 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4500c3320 0x7fa450144650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:32.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.865+0000 7fa457a7a640 1 -- 172.21.15.55:0/4100871322 >> 172.21.15.55:0/4100871322 conn(0x7fa45006e670 msgr2=0x7fa450074290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:32.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.865+0000 7fa457a7a640 1 -- 172.21.15.55:0/4100871322 shutdown_connections 2024-03-19T01:57:32.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:32.865+0000 7fa457a7a640 1 -- 172.21.15.55:0/4100871322 wait complete. 2024-03-19T01:57:33.487 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:33.487 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:34.488 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:34.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.600+0000 7f4ecc735640 1 Processor -- start 2024-03-19T01:57:34.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.600+0000 7f4ecc735640 1 -- start start 2024-03-19T01:57:34.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.604+0000 7f4ecc735640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f4ec415a540 0x7f4ec40c9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:34.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.604+0000 7f4ecc735640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ec415a920 0x7f4ec406cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:34.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.604+0000 7f4ecc735640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ec4070bd0 con 0x7f4ec415ade0 2024-03-19T01:57:34.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.604+0000 7f4ecc735640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f4ec406f5e0 con 0x7f4ec415a540 2024-03-19T01:57:34.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.604+0000 7f4ecc735640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4ec4070e90 con 0x7f4ec415a920 2024-03-19T01:57:34.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.604+0000 7f4eca4d1640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f4ec415a540 0x7f4ec40c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:34.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.604+0000 7f4ec9cd0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ec415a920 0x7f4ec406cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:34.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.604+0000 7f4eca4d1640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f4ec415a540 0x7f4ec40c9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:42276/0 (socket says 172.21.15.55:42276) 2024-03-19T01:57:34.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.604+0000 7f4eca4d1640 1 -- 172.21.15.55:0/3037966845 learned_addr learned my addr 172.21.15.55:0/3037966845 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:34.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.608+0000 7f4eca4d1640 1 -- 172.21.15.55:0/3037966845 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ec415a920 msgr2=0x7f4ec406cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:34.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.608+0000 7f4eca4d1640 1 --2- 172.21.15.55:0/3037966845 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ec415a920 0x7f4ec406cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:34.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.608+0000 7f4eca4d1640 1 -- 172.21.15.55:0/3037966845 >> v1:172.21.15.84:6789/0 conn(0x7f4ec415ade0 legacy=0x7f4ec41431b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:34.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.608+0000 7f4eca4d1640 1 -- 172.21.15.55:0/3037966845 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec406f830 con 0x7f4ec415a540 2024-03-19T01:57:34.614 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.612+0000 7f4eca4d1640 1 --2- 172.21.15.55:0/3037966845 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ec415a540 0x7f4ec40c9480 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0009870 tx=0x7f4ec001ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a81127645c20487d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:34.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.620+0000 7f4ec94cf640 1 -- 172.21.15.55:0/3037966845 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec00b5070 con 0x7f4ec415a540 2024-03-19T01:57:34.622 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.624+0000 7f4ec94cf640 1 -- 172.21.15.55:0/3037966845 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ec00a89d0 con 0x7f4ec415a540 2024-03-19T01:57:34.622 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.624+0000 7f4ec94cf640 1 -- 172.21.15.55:0/3037966845 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec00a8da0 con 0x7f4ec415a540 2024-03-19T01:57:34.623 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.624+0000 7f4ecc735640 1 -- 172.21.15.55:0/3037966845 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ec415a540 msgr2=0x7f4ec40c9480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:34.624 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.624+0000 7f4ecc735640 1 --2- 172.21.15.55:0/3037966845 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ec415a540 0x7f4ec40c9480 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0009870 tx=0x7f4ec001ed10 comp rx=0 tx=0).stop 2024-03-19T01:57:34.624 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.624+0000 7f4ecc735640 1 -- 172.21.15.55:0/3037966845 shutdown_connections 2024-03-19T01:57:34.624 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.624+0000 7f4ecc735640 1 --2- 172.21.15.55:0/3037966845 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ec415a920 0x7f4ec406cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:34.624 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.624+0000 7f4ecc735640 1 --2- 172.21.15.55:0/3037966845 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ec415a540 0x7f4ec40c9480 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:34.624 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.624+0000 7f4ecc735640 1 -- 172.21.15.55:0/3037966845 >> 172.21.15.55:0/3037966845 conn(0x7f4ec4067600 msgr2=0x7f4ec4071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:34.624 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.624+0000 7f4ecc735640 1 -- 172.21.15.55:0/3037966845 shutdown_connections 2024-03-19T01:57:34.624 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.624+0000 7f4ecc735640 1 -- 172.21.15.55:0/3037966845 wait complete. 2024-03-19T01:57:34.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.624+0000 7f4ecc735640 1 Processor -- start 2024-03-19T01:57:34.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.624+0000 7f4ecc735640 1 -- start start 2024-03-19T01:57:34.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.624+0000 7f4ecc735640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ec415a540 0x7f4ec4154be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:34.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.624+0000 7f4ecc735640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ec415a920 0x7f4ec4156c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:34.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.624+0000 7f4ecc735640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ec415ade0 0x7f4ec41384c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:34.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.624+0000 7f4ecc735640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec4070860 con 0x7f4ec415ade0 2024-03-19T01:57:34.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.624+0000 7f4ecc735640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec406fdf0 con 0x7f4ec415a540 2024-03-19T01:57:34.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.624+0000 7f4ecc735640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4ec406d270 con 0x7f4ec415a920 2024-03-19T01:57:34.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.624+0000 7f4ecacd2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ec415ade0 0x7f4ec41384c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:34.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.624+0000 7f4ecacd2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ec415ade0 0x7f4ec41384c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:37654/0 (socket says 172.21.15.55:37654) 2024-03-19T01:57:34.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.624+0000 7f4ecacd2640 1 -- 172.21.15.55:0/1073686171 learned_addr learned my addr 172.21.15.55:0/1073686171 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:34.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.628+0000 7f4eca4d1640 1 --2- 172.21.15.55:0/1073686171 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ec415a540 0x7f4ec4154be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:34.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.628+0000 7f4ec9cd0640 1 --2- 172.21.15.55:0/1073686171 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ec415a920 0x7f4ec4156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:34.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.628+0000 7f4ecacd2640 1 -- 172.21.15.55:0/1073686171 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ec415a920 msgr2=0x7f4ec4156c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:34.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.628+0000 7f4ecacd2640 1 --2- 172.21.15.55:0/1073686171 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ec415a920 0x7f4ec4156c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:34.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.628+0000 7f4ecacd2640 1 -- 172.21.15.55:0/1073686171 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ec415a540 msgr2=0x7f4ec4154be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:34.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.628+0000 7f4ecacd2640 1 --2- 172.21.15.55:0/1073686171 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ec415a540 0x7f4ec4154be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:34.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.628+0000 7f4ecacd2640 1 -- 172.21.15.55:0/1073686171 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec4155120 con 0x7f4ec415ade0 2024-03-19T01:57:34.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.632+0000 7f4ecacd2640 1 --2- 172.21.15.55:0/1073686171 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ec415ade0 0x7f4ec41384c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc00c7b0 tx=0x7f4ebc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:34.637 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.636+0000 7f4ebaffd640 1 -- 172.21.15.55:0/1073686171 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc017070 con 0x7f4ec415ade0 2024-03-19T01:57:34.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.636+0000 7f4ebaffd640 1 -- 172.21.15.55:0/1073686171 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ebc003db0 con 0x7f4ec415ade0 2024-03-19T01:57:34.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.636+0000 7f4ecc735640 1 -- 172.21.15.55:0/1073686171 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ec41f0180 con 0x7f4ec415ade0 2024-03-19T01:57:34.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.636+0000 7f4ebaffd640 1 -- 172.21.15.55:0/1073686171 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc0052b0 con 0x7f4ec415ade0 2024-03-19T01:57:34.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.636+0000 7f4ecc735640 1 -- 172.21.15.55:0/1073686171 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ec41f0740 con 0x7f4ec415ade0 2024-03-19T01:57:34.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.640+0000 7f4ecc735640 1 -- 172.21.15.55:0/1073686171 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ec4068a40 con 0x7f4ec415ade0 2024-03-19T01:57:34.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.928+0000 7f4ebaffd640 1 -- 172.21.15.55:0/1073686171 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f4ebc01a070 con 0x7f4ec415ade0 2024-03-19T01:57:34.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.932+0000 7f4ebaffd640 1 --2- 172.21.15.55:0/1073686171 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4ea80676e0 0x7f4ea8069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:34.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.932+0000 7f4eca4d1640 1 --2- 172.21.15.55:0/1073686171 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4ea80676e0 0x7f4ea8069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:34.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.932+0000 7f4eca4d1640 1 --2- 172.21.15.55:0/1073686171 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4ea80676e0 0x7f4ea8069ba0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00b2000 tx=0x7f4ec0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:34.960 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.960+0000 7f4ebaffd640 1 -- 172.21.15.55:0/1073686171 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f4ebc053970 con 0x7f4ec415ade0 2024-03-19T01:57:34.961 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:34.960+0000 7f4ebaffd640 1 -- 172.21.15.55:0/1073686171 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4ebc021020 con 0x7f4ec415ade0 2024-03-19T01:57:35.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:35.164+0000 7f4ecc735640 1 -- 172.21.15.55:0/1073686171 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ec4141ed0 con 0x7f4ec415ade0 2024-03-19T01:57:35.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:35.192+0000 7f4ebaffd640 1 -- 172.21.15.55:0/1073686171 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f4ebc0558d0 con 0x7f4ec415ade0 2024-03-19T01:57:35.195 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:35.195 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:35.195 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:35.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:35.196+0000 7f4ecc735640 1 -- 172.21.15.55:0/1073686171 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4ea80676e0 msgr2=0x7f4ea8069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:35.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:35.196+0000 7f4ecc735640 1 --2- 172.21.15.55:0/1073686171 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4ea80676e0 0x7f4ea8069ba0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00b2000 tx=0x7f4ec0009040 comp rx=0 tx=0).stop 2024-03-19T01:57:35.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:35.196+0000 7f4ecc735640 1 -- 172.21.15.55:0/1073686171 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ec415ade0 msgr2=0x7f4ec41384c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:35.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:35.196+0000 7f4ecc735640 1 --2- 172.21.15.55:0/1073686171 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ec415ade0 0x7f4ec41384c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc00c7b0 tx=0x7f4ebc00cc70 comp rx=0 tx=0).stop 2024-03-19T01:57:35.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:35.196+0000 7f4ecc735640 1 -- 172.21.15.55:0/1073686171 shutdown_connections 2024-03-19T01:57:35.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:35.196+0000 7f4ecc735640 1 --2- 172.21.15.55:0/1073686171 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4ea80676e0 0x7f4ea8069ba0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:35.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:35.196+0000 7f4ecc735640 1 --2- 172.21.15.55:0/1073686171 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ec415ade0 0x7f4ec41384c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:35.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:35.196+0000 7f4ecc735640 1 --2- 172.21.15.55:0/1073686171 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ec415a920 0x7f4ec4156c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:35.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:35.196+0000 7f4ecc735640 1 --2- 172.21.15.55:0/1073686171 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ec415a540 0x7f4ec4154be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:35.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:35.196+0000 7f4ecc735640 1 -- 172.21.15.55:0/1073686171 >> 172.21.15.55:0/1073686171 conn(0x7f4ec4067600 msgr2=0x7f4ec4141390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:35.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:35.196+0000 7f4ecc735640 1 -- 172.21.15.55:0/1073686171 shutdown_connections 2024-03-19T01:57:35.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:35.196+0000 7f4ecc735640 1 -- 172.21.15.55:0/1073686171 wait complete. 2024-03-19T01:57:35.823 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:35.823 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:36.824 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:36.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.952+0000 7f4ed4a4d640 1 Processor -- start 2024-03-19T01:57:36.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.952+0000 7f4ed4a4d640 1 -- start start 2024-03-19T01:57:36.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.952+0000 7f4ed4a4d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f4ed006c610 0x7f4ed015ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:36.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.952+0000 7f4ed4a4d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4ed006c9f0 0x7f4ed00c92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:36.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.952+0000 7f4ed4a4d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ed00c9800 0x7f4ed01427f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:36.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.952+0000 7f4ed4a4d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f4ed0070e80 con 0x7f4ed006c9f0 2024-03-19T01:57:36.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.952+0000 7f4ed4a4d640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f4ed015b250 con 0x7f4ed006c610 2024-03-19T01:57:36.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.952+0000 7f4ed4a4d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4ed006fb00 con 0x7f4ed00c9800 2024-03-19T01:57:36.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.952+0000 7f4ecdd9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4ed006c9f0 0x7f4ed00c92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:36.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.952+0000 7f4ecdd9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4ed006c9f0 0x7f4ed00c92c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:37666/0 (socket says 172.21.15.55:37666) 2024-03-19T01:57:36.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.952+0000 7f4ecdd9b640 1 -- 172.21.15.55:0/2964231788 learned_addr learned my addr 172.21.15.55:0/2964231788 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:36.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.956+0000 7f4ece59c640 1 --2- 172.21.15.55:0/2964231788 >> v2:172.21.15.84:3300/0 conn(0x7f4ed006c610 0x7f4ed015ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:36.956 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.956+0000 7f4eced9d640 1 --2- 172.21.15.55:0/2964231788 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ed00c9800 0x7f4ed01427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:36.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.956+0000 7f4ecdd9b640 1 -- 172.21.15.55:0/2964231788 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ed00c9800 msgr2=0x7f4ed01427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:36.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.956+0000 7f4ecdd9b640 1 --2- 172.21.15.55:0/2964231788 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ed00c9800 0x7f4ed01427f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:36.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.956+0000 7f4ecdd9b640 1 -- 172.21.15.55:0/2964231788 >> v2:172.21.15.84:3300/0 conn(0x7f4ed006c610 msgr2=0x7f4ed015ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:36.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.956+0000 7f4ecdd9b640 1 --2- 172.21.15.55:0/2964231788 >> v2:172.21.15.84:3300/0 conn(0x7f4ed006c610 0x7f4ed015ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:36.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.956+0000 7f4ecdd9b640 1 -- 172.21.15.55:0/2964231788 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed006fde0 con 0x7f4ed006c9f0 2024-03-19T01:57:36.960 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.960+0000 7f4ecdd9b640 1 --2- 172.21.15.55:0/2964231788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ed006c9f0 0x7f4ed00c92c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4ed00cb100 tx=0x7f4ec401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c061ef0db9a6583a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:36.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.964+0000 7f4ecd59a640 1 -- 172.21.15.55:0/2964231788 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec40b5070 con 0x7f4ed006c9f0 2024-03-19T01:57:36.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.964+0000 7f4ecd59a640 1 -- 172.21.15.55:0/2964231788 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ec40a89d0 con 0x7f4ed006c9f0 2024-03-19T01:57:36.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.964+0000 7f4ecd59a640 1 -- 172.21.15.55:0/2964231788 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec40a8da0 con 0x7f4ed006c9f0 2024-03-19T01:57:36.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.964+0000 7f4ed4a4d640 1 -- 172.21.15.55:0/2964231788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ed006c9f0 msgr2=0x7f4ed00c92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:36.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.964+0000 7f4ed4a4d640 1 --2- 172.21.15.55:0/2964231788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ed006c9f0 0x7f4ed00c92c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4ed00cb100 tx=0x7f4ec401ed10 comp rx=0 tx=0).stop 2024-03-19T01:57:36.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.964+0000 7f4ed4a4d640 1 -- 172.21.15.55:0/2964231788 shutdown_connections 2024-03-19T01:57:36.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.964+0000 7f4ed4a4d640 1 --2- 172.21.15.55:0/2964231788 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ed00c9800 0x7f4ed01427f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:36.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.964+0000 7f4ed4a4d640 1 --2- 172.21.15.55:0/2964231788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ed006c9f0 0x7f4ed00c92c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:36.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.964+0000 7f4ed4a4d640 1 --2- 172.21.15.55:0/2964231788 >> v2:172.21.15.84:3300/0 conn(0x7f4ed006c610 0x7f4ed015ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:36.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.964+0000 7f4ed4a4d640 1 -- 172.21.15.55:0/2964231788 >> 172.21.15.55:0/2964231788 conn(0x7f4ed0067600 msgr2=0x7f4ed0071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:36.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.964+0000 7f4ed4a4d640 1 -- 172.21.15.55:0/2964231788 shutdown_connections 2024-03-19T01:57:36.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.964+0000 7f4ed4a4d640 1 -- 172.21.15.55:0/2964231788 wait complete. 2024-03-19T01:57:36.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.968+0000 7f4ed4a4d640 1 Processor -- start 2024-03-19T01:57:36.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.968+0000 7f4ed4a4d640 1 -- start start 2024-03-19T01:57:36.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.968+0000 7f4ed4a4d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ed006c610 0x7f4ed0153300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:36.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.968+0000 7f4ed4a4d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ed006c9f0 0x7f4ed01eff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:36.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.968+0000 7f4ed4a4d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ed00c9800 0x7f4ed01f0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:36.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.968+0000 7f4ed4a4d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed0070680 con 0x7f4ed00c9800 2024-03-19T01:57:36.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.968+0000 7f4ed4a4d640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed00703a0 con 0x7f4ed006c9f0 2024-03-19T01:57:36.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.968+0000 7f4ed4a4d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4ed0070960 con 0x7f4ed006c610 2024-03-19T01:57:36.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.968+0000 7f4eced9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ed00c9800 0x7f4ed01f0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:36.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.968+0000 7f4ece59c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ed006c610 0x7f4ed0153300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:36.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.968+0000 7f4eced9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ed00c9800 0x7f4ed01f0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:37676/0 (socket says 172.21.15.55:37676) 2024-03-19T01:57:36.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.968+0000 7f4eced9d640 1 -- 172.21.15.55:0/3642360237 learned_addr learned my addr 172.21.15.55:0/3642360237 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:36.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.968+0000 7f4ecdd9b640 1 --2- 172.21.15.55:0/3642360237 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ed006c9f0 0x7f4ed01eff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:36.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.972+0000 7f4eced9d640 1 -- 172.21.15.55:0/3642360237 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ed006c610 msgr2=0x7f4ed0153300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:36.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.972+0000 7f4eced9d640 1 --2- 172.21.15.55:0/3642360237 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ed006c610 0x7f4ed0153300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:36.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.972+0000 7f4eced9d640 1 -- 172.21.15.55:0/3642360237 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ed006c9f0 msgr2=0x7f4ed01eff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:36.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.972+0000 7f4eced9d640 1 --2- 172.21.15.55:0/3642360237 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ed006c9f0 0x7f4ed01eff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:36.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.972+0000 7f4eced9d640 1 -- 172.21.15.55:0/3642360237 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed01efa00 con 0x7f4ed00c9800 2024-03-19T01:57:36.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.976+0000 7f4eced9d640 1 --2- 172.21.15.55:0/3642360237 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ed00c9800 0x7f4ed01f0440 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8007e60 tx=0x7f4ec800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:36.979 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.980+0000 7f4ebeffd640 1 -- 172.21.15.55:0/3642360237 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8014420 con 0x7f4ed00c9800 2024-03-19T01:57:36.979 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.980+0000 7f4ebeffd640 1 -- 172.21.15.55:0/3642360237 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ec8019070 con 0x7f4ed00c9800 2024-03-19T01:57:36.979 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.980+0000 7f4ebeffd640 1 -- 172.21.15.55:0/3642360237 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8014820 con 0x7f4ed00c9800 2024-03-19T01:57:36.979 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.980+0000 7f4ed4a4d640 1 -- 172.21.15.55:0/3642360237 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ed015b5e0 con 0x7f4ed00c9800 2024-03-19T01:57:36.979 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.980+0000 7f4ed4a4d640 1 -- 172.21.15.55:0/3642360237 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ed01404d0 con 0x7f4ed00c9800 2024-03-19T01:57:36.979 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:36.980+0000 7f4ed4a4d640 1 -- 172.21.15.55:0/3642360237 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ed0068a40 con 0x7f4ed00c9800 2024-03-19T01:57:36.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:37.000+0000 7f4ebeffd640 1 -- 172.21.15.55:0/3642360237 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f4ec80149c0 con 0x7f4ed00c9800 2024-03-19T01:57:36.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:37.000+0000 7f4ebeffd640 1 --2- 172.21.15.55:0/3642360237 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4eb00676e0 0x7f4eb0069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:37.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:37.000+0000 7f4ece59c640 1 --2- 172.21.15.55:0/3642360237 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4eb00676e0 0x7f4eb0069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:37.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:37.000+0000 7f4ece59c640 1 --2- 172.21.15.55:0/3642360237 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4eb00676e0 0x7f4eb0069ba0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0007a60 tx=0x7f4ec0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:37.003 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:37.004+0000 7f4ebeffd640 1 -- 172.21.15.55:0/3642360237 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f4ec80554a0 con 0x7f4ed00c9800 2024-03-19T01:57:37.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:37.288+0000 7f4ebeffd640 1 -- 172.21.15.55:0/3642360237 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4ec8057400 con 0x7f4ed00c9800 2024-03-19T01:57:37.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:37.524+0000 7f4ed4a4d640 1 -- 172.21.15.55:0/3642360237 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ed013c910 con 0x7f4ed00c9800 2024-03-19T01:57:37.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:37.644+0000 7f4ebeffd640 1 -- 172.21.15.55:0/3642360237 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f4ec805b4a0 con 0x7f4ed00c9800 2024-03-19T01:57:37.645 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:37.645 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:37.646 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:37.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:37.648+0000 7f4ed4a4d640 1 -- 172.21.15.55:0/3642360237 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4eb00676e0 msgr2=0x7f4eb0069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:37.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:37.648+0000 7f4ed4a4d640 1 --2- 172.21.15.55:0/3642360237 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4eb00676e0 0x7f4eb0069ba0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0007a60 tx=0x7f4ec0008040 comp rx=0 tx=0).stop 2024-03-19T01:57:37.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:37.648+0000 7f4ed4a4d640 1 -- 172.21.15.55:0/3642360237 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ed00c9800 msgr2=0x7f4ed01f0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:37.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:37.648+0000 7f4ed4a4d640 1 --2- 172.21.15.55:0/3642360237 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ed00c9800 0x7f4ed01f0440 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8007e60 tx=0x7f4ec800e4b0 comp rx=0 tx=0).stop 2024-03-19T01:57:37.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:37.648+0000 7f4ed4a4d640 1 -- 172.21.15.55:0/3642360237 shutdown_connections 2024-03-19T01:57:37.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:37.648+0000 7f4ed4a4d640 1 --2- 172.21.15.55:0/3642360237 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4eb00676e0 0x7f4eb0069ba0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:37.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:37.648+0000 7f4ed4a4d640 1 --2- 172.21.15.55:0/3642360237 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ed00c9800 0x7f4ed01f0440 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:37.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:37.648+0000 7f4ed4a4d640 1 --2- 172.21.15.55:0/3642360237 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ed006c9f0 0x7f4ed01eff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:37.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:37.648+0000 7f4ed4a4d640 1 --2- 172.21.15.55:0/3642360237 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4ed006c610 0x7f4ed0153300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:37.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:37.648+0000 7f4ed4a4d640 1 -- 172.21.15.55:0/3642360237 >> 172.21.15.55:0/3642360237 conn(0x7f4ed0067600 msgr2=0x7f4ed0071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:37.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:37.648+0000 7f4ed4a4d640 1 -- 172.21.15.55:0/3642360237 shutdown_connections 2024-03-19T01:57:37.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:37.648+0000 7f4ed4a4d640 1 -- 172.21.15.55:0/3642360237 wait complete. 2024-03-19T01:57:38.272 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:38.272 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:39.273 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:39.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.392+0000 7f21149aa640 1 Processor -- start 2024-03-19T01:57:39.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.392+0000 7f21149aa640 1 -- start start 2024-03-19T01:57:39.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.392+0000 7f21149aa640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f21101437f0 0x7f211006cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:39.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.392+0000 7f21149aa640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f21100c2e20 0x7f21100c3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:39.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.392+0000 7f21149aa640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f211006d300 con 0x7f21100c84b0 2024-03-19T01:57:39.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.392+0000 7f21149aa640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f211015e600 con 0x7f21100c2e20 2024-03-19T01:57:39.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.392+0000 7f21149aa640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f21100707c0 con 0x7f21101437f0 2024-03-19T01:57:39.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.396+0000 7f210e59c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f21101437f0 0x7f211006cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:39.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.396+0000 7f210ed9d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f21100c2e20 0x7f21100c3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:39.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.396+0000 7f210e59c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f21101437f0 0x7f211006cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:50594/0 (socket says 172.21.15.55:50594) 2024-03-19T01:57:39.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.396+0000 7f210e59c640 1 -- 172.21.15.55:0/784015343 learned_addr learned my addr 172.21.15.55:0/784015343 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:39.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.400+0000 7f210e59c640 1 -- 172.21.15.55:0/784015343 >> v2:172.21.15.84:3300/0 conn(0x7f21100c2e20 msgr2=0x7f21100c3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:39.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.400+0000 7f210e59c640 1 --2- 172.21.15.55:0/784015343 >> v2:172.21.15.84:3300/0 conn(0x7f21100c2e20 0x7f21100c3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:39.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.400+0000 7f210e59c640 1 -- 172.21.15.55:0/784015343 >> v1:172.21.15.84:6789/0 conn(0x7f21100c84b0 legacy=0x7f211015d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:39.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.400+0000 7f210e59c640 1 -- 172.21.15.55:0/784015343 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f211006f8d0 con 0x7f21101437f0 2024-03-19T01:57:39.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.404+0000 7f210e59c640 1 --2- 172.21.15.55:0/784015343 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f21101437f0 0x7f211006cdc0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2100009870 tx=0x7f210001ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d08700535c39186 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:39.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.412+0000 7f210d59a640 1 -- 172.21.15.55:0/784015343 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21000b5070 con 0x7f21101437f0 2024-03-19T01:57:39.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.412+0000 7f210d59a640 1 -- 172.21.15.55:0/784015343 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21000a8a50 con 0x7f21101437f0 2024-03-19T01:57:39.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.412+0000 7f210d59a640 1 -- 172.21.15.55:0/784015343 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21000a8e20 con 0x7f21101437f0 2024-03-19T01:57:39.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.412+0000 7f21149aa640 1 -- 172.21.15.55:0/784015343 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f21101437f0 msgr2=0x7f211006cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:39.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.412+0000 7f21149aa640 1 --2- 172.21.15.55:0/784015343 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f21101437f0 0x7f211006cdc0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2100009870 tx=0x7f210001ed50 comp rx=0 tx=0).stop 2024-03-19T01:57:39.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.412+0000 7f21149aa640 1 -- 172.21.15.55:0/784015343 shutdown_connections 2024-03-19T01:57:39.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.412+0000 7f21149aa640 1 --2- 172.21.15.55:0/784015343 >> v2:172.21.15.84:3300/0 conn(0x7f21100c2e20 0x7f21100c3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:39.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.412+0000 7f21149aa640 1 --2- 172.21.15.55:0/784015343 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f21101437f0 0x7f211006cdc0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:39.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.412+0000 7f21149aa640 1 -- 172.21.15.55:0/784015343 >> 172.21.15.55:0/784015343 conn(0x7f21100714f0 msgr2=0x7f2110071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:39.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.412+0000 7f21149aa640 1 -- 172.21.15.55:0/784015343 shutdown_connections 2024-03-19T01:57:39.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.412+0000 7f21149aa640 1 -- 172.21.15.55:0/784015343 wait complete. 2024-03-19T01:57:39.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.412+0000 7f21149aa640 1 Processor -- start 2024-03-19T01:57:39.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.412+0000 7f21149aa640 1 -- start start 2024-03-19T01:57:39.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.412+0000 7f21149aa640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21100c2e20 0x7f211015cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:39.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.412+0000 7f21149aa640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f21100c84b0 0x7f21101f09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:39.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.412+0000 7f21149aa640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21101437f0 0x7f21101f1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:39.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.412+0000 7f21149aa640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f211006fe90 con 0x7f21101437f0 2024-03-19T01:57:39.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.412+0000 7f21149aa640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2110143bd0 con 0x7f21100c2e20 2024-03-19T01:57:39.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.412+0000 7f21149aa640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2110070170 con 0x7f21100c84b0 2024-03-19T01:57:39.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.412+0000 7f210dd9b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f21100c84b0 0x7f21101f09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:39.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.412+0000 7f210ed9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21101437f0 0x7f21101f1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:39.415 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.416+0000 7f210e59c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21100c2e20 0x7f211015cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:39.415 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.416+0000 7f210dd9b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f21100c84b0 0x7f21101f09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:50600/0 (socket says 172.21.15.55:50600) 2024-03-19T01:57:39.415 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.416+0000 7f210dd9b640 1 -- 172.21.15.55:0/1768698611 learned_addr learned my addr 172.21.15.55:0/1768698611 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:39.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.416+0000 7f210ed9d640 1 -- 172.21.15.55:0/1768698611 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f21100c84b0 msgr2=0x7f21101f09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:39.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.416+0000 7f210ed9d640 1 --2- 172.21.15.55:0/1768698611 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f21100c84b0 0x7f21101f09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:39.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.416+0000 7f210ed9d640 1 -- 172.21.15.55:0/1768698611 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21100c2e20 msgr2=0x7f211015cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:39.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.416+0000 7f210ed9d640 1 --2- 172.21.15.55:0/1768698611 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21100c2e20 0x7f211015cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:39.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.416+0000 7f210ed9d640 1 -- 172.21.15.55:0/1768698611 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f211015d600 con 0x7f21101437f0 2024-03-19T01:57:39.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.420+0000 7f210ed9d640 1 --2- 172.21.15.55:0/1768698611 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21101437f0 0x7f21101f1390 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2108009d90 tx=0x7f210800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:39.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.424+0000 7f20feffd640 1 -- 172.21.15.55:0/1768698611 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2108018020 con 0x7f21101437f0 2024-03-19T01:57:39.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.424+0000 7f20feffd640 1 -- 172.21.15.55:0/1768698611 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2108014480 con 0x7f21101437f0 2024-03-19T01:57:39.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.424+0000 7f20feffd640 1 -- 172.21.15.55:0/1768698611 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2108014870 con 0x7f21101437f0 2024-03-19T01:57:39.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.424+0000 7f21149aa640 1 -- 172.21.15.55:0/1768698611 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f211015b3e0 con 0x7f21101437f0 2024-03-19T01:57:39.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.424+0000 7f21149aa640 1 -- 172.21.15.55:0/1768698611 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f211015b9a0 con 0x7f21101437f0 2024-03-19T01:57:39.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.428+0000 7f21149aa640 1 -- 172.21.15.55:0/1768698611 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2110068a40 con 0x7f21101437f0 2024-03-19T01:57:39.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.452+0000 7f20feffd640 1 -- 172.21.15.55:0/1768698611 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f210801b020 con 0x7f21101437f0 2024-03-19T01:57:39.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.452+0000 7f20feffd640 1 --2- 172.21.15.55:0/1768698611 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f20f00676b0 0x7f20f0069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:39.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.452+0000 7f210e59c640 1 --2- 172.21.15.55:0/1768698611 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f20f00676b0 0x7f20f0069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:39.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.452+0000 7f210e59c640 1 --2- 172.21.15.55:0/1768698611 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f20f00676b0 0x7f20f0069b70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f21000b2000 tx=0x7f2100009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:39.477 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:57:39.480 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:57:39.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.732+0000 7f20feffd640 1 -- 172.21.15.55:0/1768698611 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f21080551f0 con 0x7f21101437f0 2024-03-19T01:57:39.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.732+0000 7f20feffd640 1 -- 172.21.15.55:0/1768698611 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2108022030 con 0x7f21101437f0 2024-03-19T01:57:39.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.944+0000 7f21149aa640 1 -- 172.21.15.55:0/1768698611 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f211015e3e0 con 0x7f21101437f0 2024-03-19T01:57:39.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.964+0000 7f20feffd640 1 -- 172.21.15.55:0/1768698611 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f2108010040 con 0x7f21101437f0 2024-03-19T01:57:39.966 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:39.966 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:39.967 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:39.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.968+0000 7f21149aa640 1 -- 172.21.15.55:0/1768698611 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f20f00676b0 msgr2=0x7f20f0069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:39.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.968+0000 7f21149aa640 1 --2- 172.21.15.55:0/1768698611 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f20f00676b0 0x7f20f0069b70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f21000b2000 tx=0x7f2100009040 comp rx=0 tx=0).stop 2024-03-19T01:57:39.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.968+0000 7f21149aa640 1 -- 172.21.15.55:0/1768698611 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21101437f0 msgr2=0x7f21101f1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:39.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.968+0000 7f21149aa640 1 --2- 172.21.15.55:0/1768698611 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21101437f0 0x7f21101f1390 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2108009d90 tx=0x7f210800c490 comp rx=0 tx=0).stop 2024-03-19T01:57:39.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.968+0000 7f21149aa640 1 -- 172.21.15.55:0/1768698611 shutdown_connections 2024-03-19T01:57:39.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.968+0000 7f21149aa640 1 --2- 172.21.15.55:0/1768698611 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f20f00676b0 0x7f20f0069b70 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:39.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.968+0000 7f21149aa640 1 --2- 172.21.15.55:0/1768698611 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f21101437f0 0x7f21101f1390 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:39.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.968+0000 7f21149aa640 1 --2- 172.21.15.55:0/1768698611 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f21100c84b0 0x7f21101f09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:39.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.968+0000 7f21149aa640 1 --2- 172.21.15.55:0/1768698611 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21100c2e20 0x7f211015cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:39.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.968+0000 7f21149aa640 1 -- 172.21.15.55:0/1768698611 >> 172.21.15.55:0/1768698611 conn(0x7f21100714f0 msgr2=0x7f21101423e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:39.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.968+0000 7f21149aa640 1 -- 172.21.15.55:0/1768698611 shutdown_connections 2024-03-19T01:57:39.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:39.968+0000 7f21149aa640 1 -- 172.21.15.55:0/1768698611 wait complete. 2024-03-19T01:57:40.591 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:40.591 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:41.592 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:41.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.708+0000 7f3fab3d2640 1 Processor -- start 2024-03-19T01:57:41.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.708+0000 7f3fab3d2640 1 -- start start 2024-03-19T01:57:41.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.708+0000 7f3fab3d2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fa4070780 0x7f3fa415b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:41.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.708+0000 7f3fab3d2640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fa406fff0 con 0x7f3fa4152120 2024-03-19T01:57:41.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.708+0000 7f3fab3d2640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fa40c39c0 con 0x7f3fa415bb00 2024-03-19T01:57:41.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.708+0000 7f3fab3d2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3fa4152500 con 0x7f3fa4070780 2024-03-19T01:57:41.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.708+0000 7f3fa916e640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f3fa4152120 0x7f3fa40c90d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:44324/0 (socket says 172.21.15.55:44324) 2024-03-19T01:57:41.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.708+0000 7f3fa916e640 1 -- 172.21.15.55:0/445792486 learned_addr learned my addr 172.21.15.55:0/445792486 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:57:41.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.708+0000 7f3fa896d640 1 --2- 172.21.15.55:0/445792486 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fa4070780 0x7f3fa415b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:41.714 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.712+0000 7f3f9bfff640 1 -- 172.21.15.55:0/445792486 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4041581842 0 0) 0x7f3fa406fff0 con 0x7f3fa4152120 2024-03-19T01:57:41.714 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.712+0000 7f3fa896d640 1 -- 172.21.15.55:0/445792486 >> v1:172.21.15.84:6789/0 conn(0x7f3fa415bb00 legacy=0x7f3fa415bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:41.714 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.712+0000 7f3fa896d640 1 -- 172.21.15.55:0/445792486 >> v1:172.21.15.55:6789/0 conn(0x7f3fa4152120 legacy=0x7f3fa40c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:41.714 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.712+0000 7f3fa896d640 1 -- 172.21.15.55:0/445792486 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa40702b0 con 0x7f3fa4070780 2024-03-19T01:57:41.743 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.744+0000 7f3fa896d640 1 --2- 172.21.15.55:0/445792486 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fa4070780 0x7f3fa415b5c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3f940097f0 tx=0x7f3f9401c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b195e6338a23a8bc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:41.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.756+0000 7f3f9bfff640 1 -- 172.21.15.55:0/445792486 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f940b0070 con 0x7f3fa4070780 2024-03-19T01:57:41.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.756+0000 7f3f9bfff640 1 -- 172.21.15.55:0/445792486 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f940ab600 con 0x7f3fa4070780 2024-03-19T01:57:41.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.756+0000 7f3f9bfff640 1 -- 172.21.15.55:0/445792486 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f940ab9d0 con 0x7f3fa4070780 2024-03-19T01:57:41.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.760+0000 7f3fab3d2640 1 -- 172.21.15.55:0/445792486 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fa4070780 msgr2=0x7f3fa415b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:41.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.760+0000 7f3fab3d2640 1 --2- 172.21.15.55:0/445792486 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fa4070780 0x7f3fa415b5c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3f940097f0 tx=0x7f3f9401c4e0 comp rx=0 tx=0).stop 2024-03-19T01:57:41.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.760+0000 7f3fab3d2640 1 -- 172.21.15.55:0/445792486 shutdown_connections 2024-03-19T01:57:41.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.760+0000 7f3fab3d2640 1 --2- 172.21.15.55:0/445792486 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fa4070780 0x7f3fa415b5c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:41.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.760+0000 7f3fab3d2640 1 -- 172.21.15.55:0/445792486 >> 172.21.15.55:0/445792486 conn(0x7f3fa4067600 msgr2=0x7f3fa4071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:41.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.760+0000 7f3fab3d2640 1 -- 172.21.15.55:0/445792486 shutdown_connections 2024-03-19T01:57:41.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.760+0000 7f3fab3d2640 1 -- 172.21.15.55:0/445792486 wait complete. 2024-03-19T01:57:41.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.760+0000 7f3fab3d2640 1 Processor -- start 2024-03-19T01:57:41.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.760+0000 7f3fab3d2640 1 -- start start 2024-03-19T01:57:41.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.764+0000 7f3fab3d2640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3fa4070780 0x7f3fa40ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:41.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.764+0000 7f3fab3d2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fa4152120 0x7f3fa40ce010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:41.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.764+0000 7f3fab3d2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fa415bb00 0x7f3fa40ce550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:41.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.764+0000 7f3fab3d2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa40c97e0 con 0x7f3fa415bb00 2024-03-19T01:57:41.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.764+0000 7f3fab3d2640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa40c3530 con 0x7f3fa4070780 2024-03-19T01:57:41.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.764+0000 7f3fab3d2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3fa40c86d0 con 0x7f3fa4152120 2024-03-19T01:57:41.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.764+0000 7f3fa896d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fa4152120 0x7f3fa40ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:41.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.764+0000 7f3fa916e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3fa4070780 0x7f3fa40ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:41.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.764+0000 7f3fa896d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fa4152120 0x7f3fa40ce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:50624/0 (socket says 172.21.15.55:50624) 2024-03-19T01:57:41.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.764+0000 7f3fa896d640 1 -- 172.21.15.55:0/3231843055 learned_addr learned my addr 172.21.15.55:0/3231843055 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:41.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.772+0000 7f3fa896d640 1 -- 172.21.15.55:0/3231843055 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3fa4070780 msgr2=0x7f3fa40ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:41.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.772+0000 7f3fa896d640 1 --2- 172.21.15.55:0/3231843055 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3fa4070780 0x7f3fa40ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:41.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.772+0000 7f3fa896d640 1 -- 172.21.15.55:0/3231843055 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fa415bb00 msgr2=0x7f3fa40ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:41.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.772+0000 7f3fa896d640 1 --2- 172.21.15.55:0/3231843055 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fa415bb00 0x7f3fa40ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:41.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.772+0000 7f3fa896d640 1 -- 172.21.15.55:0/3231843055 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa41452f0 con 0x7f3fa4152120 2024-03-19T01:57:41.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.776+0000 7f3fa896d640 1 --2- 172.21.15.55:0/3231843055 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fa4152120 0x7f3fa40ce010 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3f940190f0 tx=0x7f3f9401ad20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:41.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.784+0000 7f3f99ffb640 1 -- 172.21.15.55:0/3231843055 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f940b0050 con 0x7f3fa4152120 2024-03-19T01:57:41.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.784+0000 7f3f99ffb640 1 -- 172.21.15.55:0/3231843055 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f940bc070 con 0x7f3fa4152120 2024-03-19T01:57:41.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.784+0000 7f3fab3d2640 1 -- 172.21.15.55:0/3231843055 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fa4145590 con 0x7f3fa4152120 2024-03-19T01:57:41.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.784+0000 7f3f99ffb640 1 -- 172.21.15.55:0/3231843055 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f940b5850 con 0x7f3fa4152120 2024-03-19T01:57:41.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.784+0000 7f3fab3d2640 1 -- 172.21.15.55:0/3231843055 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fa4145020 con 0x7f3fa4152120 2024-03-19T01:57:41.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.784+0000 7f3fab3d2640 1 -- 172.21.15.55:0/3231843055 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fa40c84b0 con 0x7f3fa4152120 2024-03-19T01:57:41.820 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.820+0000 7f3f99ffb640 1 -- 172.21.15.55:0/3231843055 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f3f940b59f0 con 0x7f3fa4152120 2024-03-19T01:57:41.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.820+0000 7f3f99ffb640 1 --2- 172.21.15.55:0/3231843055 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3f880676e0 0x7f3f88069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:41.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.820+0000 7f3fa916e640 1 --2- 172.21.15.55:0/3231843055 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3f880676e0 0x7f3f88069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:41.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:41.820+0000 7f3fa916e640 1 --2- 172.21.15.55:0/3231843055 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3f880676e0 0x7f3f88069ba0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0000f70 tx=0x7f3fa0007240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:42.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:42.080+0000 7f3f99ffb640 1 -- 172.21.15.55:0/3231843055 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f3f940ef1b0 con 0x7f3fa4152120 2024-03-19T01:57:42.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:42.100+0000 7f3f99ffb640 1 -- 172.21.15.55:0/3231843055 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f3f940160d0 con 0x7f3fa4152120 2024-03-19T01:57:42.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:42.320+0000 7f3fab3d2640 1 -- 172.21.15.55:0/3231843055 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3fa41507c0 con 0x7f3fa4152120 2024-03-19T01:57:42.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:42.348+0000 7f3f99ffb640 1 -- 172.21.15.55:0/3231843055 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f3f940f1810 con 0x7f3fa4152120 2024-03-19T01:57:42.348 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:42.348 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:42.348 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:42.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:42.352+0000 7f3fab3d2640 1 -- 172.21.15.55:0/3231843055 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3f880676e0 msgr2=0x7f3f88069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:42.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:42.352+0000 7f3fab3d2640 1 --2- 172.21.15.55:0/3231843055 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3f880676e0 0x7f3f88069ba0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0000f70 tx=0x7f3fa0007240 comp rx=0 tx=0).stop 2024-03-19T01:57:42.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:42.352+0000 7f3fab3d2640 1 -- 172.21.15.55:0/3231843055 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fa4152120 msgr2=0x7f3fa40ce010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:42.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:42.352+0000 7f3fab3d2640 1 --2- 172.21.15.55:0/3231843055 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fa4152120 0x7f3fa40ce010 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3f940190f0 tx=0x7f3f9401ad20 comp rx=0 tx=0).stop 2024-03-19T01:57:42.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:42.352+0000 7f3fab3d2640 1 -- 172.21.15.55:0/3231843055 shutdown_connections 2024-03-19T01:57:42.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:42.352+0000 7f3fab3d2640 1 --2- 172.21.15.55:0/3231843055 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3f880676e0 0x7f3f88069ba0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:42.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:42.352+0000 7f3fab3d2640 1 --2- 172.21.15.55:0/3231843055 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fa415bb00 0x7f3fa40ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:42.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:42.352+0000 7f3fab3d2640 1 --2- 172.21.15.55:0/3231843055 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fa4152120 0x7f3fa40ce010 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:42.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:42.352+0000 7f3fab3d2640 1 --2- 172.21.15.55:0/3231843055 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3fa4070780 0x7f3fa40ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:42.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:42.352+0000 7f3fab3d2640 1 -- 172.21.15.55:0/3231843055 >> 172.21.15.55:0/3231843055 conn(0x7f3fa4067600 msgr2=0x7f3fa4150fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:42.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:42.352+0000 7f3fab3d2640 1 -- 172.21.15.55:0/3231843055 shutdown_connections 2024-03-19T01:57:42.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:42.352+0000 7f3fab3d2640 1 -- 172.21.15.55:0/3231843055 wait complete. 2024-03-19T01:57:42.972 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:42.972 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:43.973 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:44.092 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.092+0000 7fd2d1a98640 1 Processor -- start 2024-03-19T01:57:44.092 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.092+0000 7fd2d1a98640 1 -- start start 2024-03-19T01:57:44.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.092+0000 7fd2d1a98640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd2cc0c96b0 0x7fd2cc070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:44.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.092+0000 7fd2d1a98640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2cc15e3a0 con 0x7fd2cc0c9a90 2024-03-19T01:57:44.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.092+0000 7fd2d1a98640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2cc15e680 con 0x7fd2cc1540c0 2024-03-19T01:57:44.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.092+0000 7fd2d1a98640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fd2cc0c7ad0 con 0x7fd2cc0c96b0 2024-03-19T01:57:44.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.092+0000 7fd2cb7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd2cc0c96b0 0x7fd2cc070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:44.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.092+0000 7fd2cb7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd2cc0c96b0 0x7fd2cc070f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:44874/0 (socket says 172.21.15.55:44874) 2024-03-19T01:57:44.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.092+0000 7fd2cb7fe640 1 -- 172.21.15.55:0/2828998044 learned_addr learned my addr 172.21.15.55:0/2828998044 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:44.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.096+0000 7fd2cb7fe640 1 -- 172.21.15.55:0/2828998044 >> v1:172.21.15.84:6789/0 conn(0x7fd2cc1540c0 legacy=0x7fd2cc15d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:44.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.096+0000 7fd2cb7fe640 1 -- 172.21.15.55:0/2828998044 >> v1:172.21.15.55:6789/0 conn(0x7fd2cc0c9a90 legacy=0x7fd2cc0c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:44.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.096+0000 7fd2cb7fe640 1 -- 172.21.15.55:0/2828998044 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2cc15a330 con 0x7fd2cc0c96b0 2024-03-19T01:57:44.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.100+0000 7fd2cb7fe640 1 --2- 172.21.15.55:0/2828998044 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2cc0c96b0 0x7fd2cc070f20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4009870 tx=0x7fd2c401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=952294b84e1bcb5a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:44.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.108+0000 7fd2ca7fc640 1 -- 172.21.15.55:0/2828998044 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c40b1070 con 0x7fd2cc0c96b0 2024-03-19T01:57:44.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.108+0000 7fd2ca7fc640 1 -- 172.21.15.55:0/2828998044 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2c40ac620 con 0x7fd2cc0c96b0 2024-03-19T01:57:44.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.108+0000 7fd2ca7fc640 1 -- 172.21.15.55:0/2828998044 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c40ac9f0 con 0x7fd2cc0c96b0 2024-03-19T01:57:44.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.108+0000 7fd2d1a98640 1 -- 172.21.15.55:0/2828998044 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2cc0c96b0 msgr2=0x7fd2cc070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:44.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.108+0000 7fd2d1a98640 1 --2- 172.21.15.55:0/2828998044 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2cc0c96b0 0x7fd2cc070f20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4009870 tx=0x7fd2c401c4d0 comp rx=0 tx=0).stop 2024-03-19T01:57:44.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.108+0000 7fd2d1a98640 1 -- 172.21.15.55:0/2828998044 shutdown_connections 2024-03-19T01:57:44.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.108+0000 7fd2d1a98640 1 --2- 172.21.15.55:0/2828998044 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2cc0c96b0 0x7fd2cc070f20 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:44.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.108+0000 7fd2d1a98640 1 -- 172.21.15.55:0/2828998044 >> 172.21.15.55:0/2828998044 conn(0x7fd2cc0cc9c0 msgr2=0x7fd2cc0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:44.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.108+0000 7fd2d1a98640 1 -- 172.21.15.55:0/2828998044 shutdown_connections 2024-03-19T01:57:44.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.108+0000 7fd2d1a98640 1 -- 172.21.15.55:0/2828998044 wait complete. 2024-03-19T01:57:44.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.112+0000 7fd2d1a98640 1 Processor -- start 2024-03-19T01:57:44.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.112+0000 7fd2d1a98640 1 -- start start 2024-03-19T01:57:44.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.112+0000 7fd2d1a98640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2cc0c96b0 0x7fd2cc155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:44.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.112+0000 7fd2d1a98640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd2cc0c9a90 0x7fd2cc14d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:44.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.112+0000 7fd2d1a98640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd2cc1540c0 0x7fd2cc14c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:44.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.112+0000 7fd2d1a98640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc1548e0 con 0x7fd2cc0c96b0 2024-03-19T01:57:44.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.112+0000 7fd2d1a98640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc15b0f0 con 0x7fd2cc0c9a90 2024-03-19T01:57:44.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.112+0000 7fd2d1a98640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd2cc15df20 con 0x7fd2cc1540c0 2024-03-19T01:57:44.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.112+0000 7fd2cb7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2cc0c96b0 0x7fd2cc155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:44.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.112+0000 7fd2caffd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd2cc0c9a90 0x7fd2cc14d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:44.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.112+0000 7fd2cb7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2cc0c96b0 0x7fd2cc155800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:44880/0 (socket says 172.21.15.55:44880) 2024-03-19T01:57:44.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.112+0000 7fd2cb7fe640 1 -- 172.21.15.55:0/2178682604 learned_addr learned my addr 172.21.15.55:0/2178682604 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:44.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.112+0000 7fd2cbfff640 1 --2- 172.21.15.55:0/2178682604 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd2cc1540c0 0x7fd2cc14c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:44.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.116+0000 7fd2cb7fe640 1 -- 172.21.15.55:0/2178682604 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd2cc1540c0 msgr2=0x7fd2cc14c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:44.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.116+0000 7fd2cb7fe640 1 --2- 172.21.15.55:0/2178682604 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd2cc1540c0 0x7fd2cc14c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:44.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.116+0000 7fd2cb7fe640 1 -- 172.21.15.55:0/2178682604 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd2cc0c9a90 msgr2=0x7fd2cc14d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:44.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.116+0000 7fd2cb7fe640 1 --2- 172.21.15.55:0/2178682604 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd2cc0c9a90 0x7fd2cc14d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:44.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.116+0000 7fd2cb7fe640 1 -- 172.21.15.55:0/2178682604 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2cc14b1d0 con 0x7fd2cc0c96b0 2024-03-19T01:57:44.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.116+0000 7fd2cb7fe640 1 --2- 172.21.15.55:0/2178682604 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2cc0c96b0 0x7fd2cc155800 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd2c40af040 tx=0x7fd2c401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:44.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.124+0000 7fd2c3fff640 1 -- 172.21.15.55:0/2178682604 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c40bb020 con 0x7fd2cc0c96b0 2024-03-19T01:57:44.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.124+0000 7fd2c3fff640 1 -- 172.21.15.55:0/2178682604 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2c40b7640 con 0x7fd2cc0c96b0 2024-03-19T01:57:44.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.124+0000 7fd2d1a98640 1 -- 172.21.15.55:0/2178682604 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2cc14d010 con 0x7fd2cc0c96b0 2024-03-19T01:57:44.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.124+0000 7fd2c3fff640 1 -- 172.21.15.55:0/2178682604 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c40b1040 con 0x7fd2cc0c96b0 2024-03-19T01:57:44.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.124+0000 7fd2d1a98640 1 -- 172.21.15.55:0/2178682604 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2cc14b730 con 0x7fd2cc0c96b0 2024-03-19T01:57:44.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.124+0000 7fd2d1a98640 1 -- 172.21.15.55:0/2178682604 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2cc068a40 con 0x7fd2cc0c96b0 2024-03-19T01:57:44.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.144+0000 7fd2c3fff640 1 -- 172.21.15.55:0/2178682604 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fd2c40be020 con 0x7fd2cc0c96b0 2024-03-19T01:57:44.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.144+0000 7fd2c3fff640 1 --2- 172.21.15.55:0/2178682604 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd2ac067660 0x7fd2ac069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:44.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.144+0000 7fd2caffd640 1 --2- 172.21.15.55:0/2178682604 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd2ac067660 0x7fd2ac069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:44.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.144+0000 7fd2caffd640 1 --2- 172.21.15.55:0/2178682604 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd2ac067660 0x7fd2ac069b20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc0c37e0 tx=0x7fd2b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:44.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.148+0000 7fd2c3fff640 1 -- 172.21.15.55:0/2178682604 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fd2c40ac5c0 con 0x7fd2cc0c96b0 2024-03-19T01:57:44.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.428+0000 7fd2c3fff640 1 -- 172.21.15.55:0/2178682604 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fd2c40c5030 con 0x7fd2cc0c96b0 2024-03-19T01:57:44.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.644+0000 7fd2d1a98640 1 -- 172.21.15.55:0/2178682604 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2cc0c88a0 con 0x7fd2cc0c96b0 2024-03-19T01:57:44.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.740+0000 7fd2c3fff640 1 -- 172.21.15.55:0/2178682604 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fd2cc14b1d0 con 0x7fd2cc0c96b0 2024-03-19T01:57:44.742 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:44.742 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:44.742 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:44.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.744+0000 7fd2d1a98640 1 -- 172.21.15.55:0/2178682604 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd2ac067660 msgr2=0x7fd2ac069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:44.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.744+0000 7fd2d1a98640 1 --2- 172.21.15.55:0/2178682604 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd2ac067660 0x7fd2ac069b20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc0c37e0 tx=0x7fd2b4009040 comp rx=0 tx=0).stop 2024-03-19T01:57:44.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.744+0000 7fd2d1a98640 1 -- 172.21.15.55:0/2178682604 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2cc0c96b0 msgr2=0x7fd2cc155800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:44.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.744+0000 7fd2d1a98640 1 --2- 172.21.15.55:0/2178682604 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2cc0c96b0 0x7fd2cc155800 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd2c40af040 tx=0x7fd2c401ad10 comp rx=0 tx=0).stop 2024-03-19T01:57:44.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.744+0000 7fd2d1a98640 1 -- 172.21.15.55:0/2178682604 shutdown_connections 2024-03-19T01:57:44.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.744+0000 7fd2d1a98640 1 --2- 172.21.15.55:0/2178682604 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd2ac067660 0x7fd2ac069b20 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:44.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.744+0000 7fd2d1a98640 1 --2- 172.21.15.55:0/2178682604 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd2cc1540c0 0x7fd2cc14c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:44.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.744+0000 7fd2d1a98640 1 --2- 172.21.15.55:0/2178682604 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd2cc0c9a90 0x7fd2cc14d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:44.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.744+0000 7fd2d1a98640 1 --2- 172.21.15.55:0/2178682604 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2cc0c96b0 0x7fd2cc155800 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:44.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.744+0000 7fd2d1a98640 1 -- 172.21.15.55:0/2178682604 >> 172.21.15.55:0/2178682604 conn(0x7fd2cc0cc9c0 msgr2=0x7fd2cc0c5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:44.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.744+0000 7fd2d1a98640 1 -- 172.21.15.55:0/2178682604 shutdown_connections 2024-03-19T01:57:44.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:44.744+0000 7fd2d1a98640 1 -- 172.21.15.55:0/2178682604 wait complete. 2024-03-19T01:57:45.365 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:45.366 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:46.367 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:46.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.480+0000 7fca7b244640 1 Processor -- start 2024-03-19T01:57:46.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.480+0000 7fca7b244640 1 -- start start 2024-03-19T01:57:46.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.480+0000 7fca7b244640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fca741520b0 0x7fca740709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:46.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.480+0000 7fca7b244640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fca740c9420 0x7fca740c31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:46.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.480+0000 7fca7b244640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca740c9850 con 0x7fca740c9000 2024-03-19T01:57:46.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.480+0000 7fca7b244640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fca7415c4e0 con 0x7fca740c9420 2024-03-19T01:57:46.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.480+0000 7fca7b244640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fca7415c050 con 0x7fca741520b0 2024-03-19T01:57:46.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.480+0000 7fca78fe0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fca741520b0 0x7fca740709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:46.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.480+0000 7fca73fff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fca740c9000 0x7fca740c2ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:51012/0 (socket says 172.21.15.55:51012) 2024-03-19T01:57:46.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.480+0000 7fca73fff640 1 -- 172.21.15.55:0/1960366997 learned_addr learned my addr 172.21.15.55:0/1960366997 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:57:46.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.484+0000 7fca78fe0640 1 -- 172.21.15.55:0/1960366997 >> v2:172.21.15.55:3300/0 conn(0x7fca740c9420 msgr2=0x7fca740c31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:46.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.484+0000 7fca78fe0640 1 --2- 172.21.15.55:0/1960366997 >> v2:172.21.15.55:3300/0 conn(0x7fca740c9420 0x7fca740c31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:46.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.484+0000 7fca78fe0640 1 -- 172.21.15.55:0/1960366997 >> v1:172.21.15.84:6789/0 conn(0x7fca740c9000 legacy=0x7fca740c2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:46.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.484+0000 7fca78fe0640 1 -- 172.21.15.55:0/1960366997 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca74158d50 con 0x7fca741520b0 2024-03-19T01:57:46.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.492+0000 7fca78fe0640 1 --2- 172.21.15.55:0/1960366997 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fca741520b0 0x7fca740709a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fca640097c0 tx=0x7fca6401ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfb40cfc8cb409f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:46.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.500+0000 7fca737fe640 1 -- 172.21.15.55:0/1960366997 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca640b5070 con 0x7fca741520b0 2024-03-19T01:57:46.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.500+0000 7fca737fe640 1 -- 172.21.15.55:0/1960366997 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca640a8990 con 0x7fca741520b0 2024-03-19T01:57:46.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.500+0000 7fca737fe640 1 -- 172.21.15.55:0/1960366997 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca640a8d60 con 0x7fca741520b0 2024-03-19T01:57:46.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.500+0000 7fca7b244640 1 -- 172.21.15.55:0/1960366997 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fca741520b0 msgr2=0x7fca740709a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:46.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.500+0000 7fca7b244640 1 --2- 172.21.15.55:0/1960366997 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fca741520b0 0x7fca740709a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fca640097c0 tx=0x7fca6401ec90 comp rx=0 tx=0).stop 2024-03-19T01:57:46.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.500+0000 7fca7b244640 1 -- 172.21.15.55:0/1960366997 shutdown_connections 2024-03-19T01:57:46.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.500+0000 7fca7b244640 1 --2- 172.21.15.55:0/1960366997 >> v2:172.21.15.55:3300/0 conn(0x7fca740c9420 0x7fca740c31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:46.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.500+0000 7fca7b244640 1 --2- 172.21.15.55:0/1960366997 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fca741520b0 0x7fca740709a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:46.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.500+0000 7fca7b244640 1 -- 172.21.15.55:0/1960366997 >> 172.21.15.55:0/1960366997 conn(0x7fca74067600 msgr2=0x7fca74071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:46.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.500+0000 7fca7b244640 1 -- 172.21.15.55:0/1960366997 shutdown_connections 2024-03-19T01:57:46.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.500+0000 7fca7b244640 1 -- 172.21.15.55:0/1960366997 wait complete. 2024-03-19T01:57:46.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.504+0000 7fca7b244640 1 Processor -- start 2024-03-19T01:57:46.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.504+0000 7fca7b244640 1 -- start start 2024-03-19T01:57:46.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.504+0000 7fca7b244640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fca740c9000 0x7fca741f3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:46.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.504+0000 7fca7b244640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca740c9420 0x7fca7406db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:46.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.504+0000 7fca7b244640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca741520b0 0x7fca741590e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:46.504 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.504+0000 7fca7b244640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fca7415bce0 con 0x7fca740c9420 2024-03-19T01:57:46.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.504+0000 7fca7b244640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fca7415ba00 con 0x7fca741520b0 2024-03-19T01:57:46.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.504+0000 7fca7b244640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fca741589a0 con 0x7fca740c9000 2024-03-19T01:57:46.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.504+0000 7fca73fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca740c9420 0x7fca7406db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:46.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.504+0000 7fca73fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca740c9420 0x7fca7406db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:44910/0 (socket says 172.21.15.55:44910) 2024-03-19T01:57:46.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.504+0000 7fca73fff640 1 -- 172.21.15.55:0/165433976 learned_addr learned my addr 172.21.15.55:0/165433976 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:46.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.504+0000 7fca797e1640 1 --2- 172.21.15.55:0/165433976 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca741520b0 0x7fca741590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:46.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.504+0000 7fca78fe0640 1 --2- 172.21.15.55:0/165433976 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fca740c9000 0x7fca741f3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:46.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.508+0000 7fca73fff640 1 -- 172.21.15.55:0/165433976 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fca740c9000 msgr2=0x7fca741f3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:46.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.508+0000 7fca73fff640 1 --2- 172.21.15.55:0/165433976 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fca740c9000 0x7fca741f3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:46.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.508+0000 7fca73fff640 1 -- 172.21.15.55:0/165433976 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca741520b0 msgr2=0x7fca741590e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:46.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.508+0000 7fca73fff640 1 --2- 172.21.15.55:0/165433976 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca741520b0 0x7fca741590e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:46.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.508+0000 7fca73fff640 1 -- 172.21.15.55:0/165433976 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca741efd90 con 0x7fca740c9420 2024-03-19T01:57:46.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.512+0000 7fca73fff640 1 --2- 172.21.15.55:0/165433976 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca740c9420 0x7fca7406db90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fca6800c370 tx=0x7fca6800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:46.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.516+0000 7fca717fa640 1 -- 172.21.15.55:0/165433976 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca68017070 con 0x7fca740c9420 2024-03-19T01:57:46.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.516+0000 7fca717fa640 1 -- 172.21.15.55:0/165433976 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca68005cf0 con 0x7fca740c9420 2024-03-19T01:57:46.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.516+0000 7fca7b244640 1 -- 172.21.15.55:0/165433976 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca7415b1b0 con 0x7fca740c9420 2024-03-19T01:57:46.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.516+0000 7fca717fa640 1 -- 172.21.15.55:0/165433976 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca680072f0 con 0x7fca740c9420 2024-03-19T01:57:46.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.516+0000 7fca7b244640 1 -- 172.21.15.55:0/165433976 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca741f0310 con 0x7fca740c9420 2024-03-19T01:57:46.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.516+0000 7fca7b244640 1 -- 172.21.15.55:0/165433976 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca74068a40 con 0x7fca740c9420 2024-03-19T01:57:46.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.792+0000 7fca717fa640 1 -- 172.21.15.55:0/165433976 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fca6801a070 con 0x7fca740c9420 2024-03-19T01:57:46.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.792+0000 7fca717fa640 1 --2- 172.21.15.55:0/165433976 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fca60067690 0x7fca60069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:46.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.792+0000 7fca78fe0640 1 --2- 172.21.15.55:0/165433976 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fca60067690 0x7fca60069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:46.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.792+0000 7fca78fe0640 1 --2- 172.21.15.55:0/165433976 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fca60067690 0x7fca60069b50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fca640b2000 tx=0x7fca640aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:46.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.820+0000 7fca717fa640 1 -- 172.21.15.55:0/165433976 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fca680536a0 con 0x7fca740c9420 2024-03-19T01:57:46.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:46.820+0000 7fca717fa640 1 -- 172.21.15.55:0/165433976 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fca68021020 con 0x7fca740c9420 2024-03-19T01:57:47.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:47.056+0000 7fca7b244640 1 -- 172.21.15.55:0/165433976 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca740cba50 con 0x7fca740c9420 2024-03-19T01:57:47.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:47.084+0000 7fca717fa640 1 -- 172.21.15.55:0/165433976 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fca68055600 con 0x7fca740c9420 2024-03-19T01:57:47.085 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:47.085 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:47.085 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:47.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:47.088+0000 7fca7b244640 1 -- 172.21.15.55:0/165433976 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fca60067690 msgr2=0x7fca60069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:47.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:47.088+0000 7fca7b244640 1 --2- 172.21.15.55:0/165433976 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fca60067690 0x7fca60069b50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fca640b2000 tx=0x7fca640aa070 comp rx=0 tx=0).stop 2024-03-19T01:57:47.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:47.088+0000 7fca7b244640 1 -- 172.21.15.55:0/165433976 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca740c9420 msgr2=0x7fca7406db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:47.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:47.088+0000 7fca7b244640 1 --2- 172.21.15.55:0/165433976 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca740c9420 0x7fca7406db90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fca6800c370 tx=0x7fca6800c830 comp rx=0 tx=0).stop 2024-03-19T01:57:47.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:47.088+0000 7fca7b244640 1 -- 172.21.15.55:0/165433976 shutdown_connections 2024-03-19T01:57:47.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:47.088+0000 7fca7b244640 1 --2- 172.21.15.55:0/165433976 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fca60067690 0x7fca60069b50 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:47.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:47.088+0000 7fca7b244640 1 --2- 172.21.15.55:0/165433976 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca741520b0 0x7fca741590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:47.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:47.088+0000 7fca7b244640 1 --2- 172.21.15.55:0/165433976 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca740c9420 0x7fca7406db90 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:47.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:47.088+0000 7fca7b244640 1 --2- 172.21.15.55:0/165433976 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fca740c9000 0x7fca741f3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:47.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:47.088+0000 7fca7b244640 1 -- 172.21.15.55:0/165433976 >> 172.21.15.55:0/165433976 conn(0x7fca74067600 msgr2=0x7fca740cda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:47.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:47.088+0000 7fca7b244640 1 -- 172.21.15.55:0/165433976 shutdown_connections 2024-03-19T01:57:47.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:47.088+0000 7fca7b244640 1 -- 172.21.15.55:0/165433976 wait complete. 2024-03-19T01:57:47.708 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:47.708 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:48.709 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:48.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.820+0000 7f92aa425640 1 Processor -- start 2024-03-19T01:57:48.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.820+0000 7f92aa425640 1 -- start start 2024-03-19T01:57:48.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.820+0000 7f92aa425640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f92a41544f0 0x7f92a4073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:48.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.820+0000 7f92aa425640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f92a415acb0 0x7f92a40cce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:48.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.820+0000 7f92aa425640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92a40c7fa0 con 0x7f92a406b2a0 2024-03-19T01:57:48.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.820+0000 7f92aa425640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f92a415b210 con 0x7f92a415acb0 2024-03-19T01:57:48.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.820+0000 7f92aa425640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f92a40c8240 con 0x7f92a41544f0 2024-03-19T01:57:48.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.824+0000 7f92a89c2640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f92a415acb0 0x7f92a40cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:48.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.824+0000 7f92a3fff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f92a41544f0 0x7f92a4073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:48.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.824+0000 7f92a89c2640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f92a415acb0 0x7f92a40cce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58976/0 (socket says 172.21.15.55:58976) 2024-03-19T01:57:48.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.824+0000 7f92a89c2640 1 -- 172.21.15.55:0/3161750499 learned_addr learned my addr 172.21.15.55:0/3161750499 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:48.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.824+0000 7f92a89c2640 1 -- 172.21.15.55:0/3161750499 >> v2:172.21.15.84:3300/0 conn(0x7f92a41544f0 msgr2=0x7f92a4073540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:48.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.824+0000 7f92a89c2640 1 --2- 172.21.15.55:0/3161750499 >> v2:172.21.15.84:3300/0 conn(0x7f92a41544f0 0x7f92a4073540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:48.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.824+0000 7f92a89c2640 1 -- 172.21.15.55:0/3161750499 >> v1:172.21.15.84:6789/0 conn(0x7f92a406b2a0 legacy=0x7f92a415e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:48.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.824+0000 7f92a89c2640 1 -- 172.21.15.55:0/3161750499 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a415d920 con 0x7f92a415acb0 2024-03-19T01:57:48.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.828+0000 7f92a89c2640 1 --2- 172.21.15.55:0/3161750499 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92a415acb0 0x7f92a40cce30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f92a40ca6b0 tx=0x7f929c01f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e86f221210d8ad server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:48.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.832+0000 7f92a2ffd640 1 -- 172.21.15.55:0/3161750499 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929c0b4070 con 0x7f92a415acb0 2024-03-19T01:57:48.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.836+0000 7f92a2ffd640 1 -- 172.21.15.55:0/3161750499 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f929c0af400 con 0x7f92a415acb0 2024-03-19T01:57:48.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.836+0000 7f92a2ffd640 1 -- 172.21.15.55:0/3161750499 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929c0af7d0 con 0x7f92a415acb0 2024-03-19T01:57:48.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.836+0000 7f92aa425640 1 -- 172.21.15.55:0/3161750499 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92a415acb0 msgr2=0x7f92a40cce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:48.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.836+0000 7f92aa425640 1 --2- 172.21.15.55:0/3161750499 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92a415acb0 0x7f92a40cce30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f92a40ca6b0 tx=0x7f929c01f180 comp rx=0 tx=0).stop 2024-03-19T01:57:48.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.836+0000 7f92aa425640 1 -- 172.21.15.55:0/3161750499 shutdown_connections 2024-03-19T01:57:48.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.836+0000 7f92aa425640 1 --2- 172.21.15.55:0/3161750499 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92a415acb0 0x7f92a40cce30 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:48.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.836+0000 7f92aa425640 1 --2- 172.21.15.55:0/3161750499 >> v2:172.21.15.84:3300/0 conn(0x7f92a41544f0 0x7f92a4073540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:48.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.836+0000 7f92aa425640 1 -- 172.21.15.55:0/3161750499 >> 172.21.15.55:0/3161750499 conn(0x7f92a406e670 msgr2=0x7f92a406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:48.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.836+0000 7f92aa425640 1 -- 172.21.15.55:0/3161750499 shutdown_connections 2024-03-19T01:57:48.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.836+0000 7f92aa425640 1 -- 172.21.15.55:0/3161750499 wait complete. 2024-03-19T01:57:48.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.840+0000 7f92aa425640 1 Processor -- start 2024-03-19T01:57:48.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.840+0000 7f92aa425640 1 -- start start 2024-03-19T01:57:48.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.840+0000 7f92aa425640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92a406b2a0 0x7f92a41e5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:48.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.840+0000 7f92aa425640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f92a41544f0 0x7f92a414fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:48.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.840+0000 7f92aa425640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f92a415acb0 0x7f92a41f3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:48.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.840+0000 7f92aa425640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f92a40ca120 con 0x7f92a406b2a0 2024-03-19T01:57:48.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.840+0000 7f92aa425640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f92a41548d0 con 0x7f92a415acb0 2024-03-19T01:57:48.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.840+0000 7f92aa425640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f92a40ca2a0 con 0x7f92a41544f0 2024-03-19T01:57:48.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.840+0000 7f92a89c2640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f92a415acb0 0x7f92a41f3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:48.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.840+0000 7f92a37fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f92a41544f0 0x7f92a414fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:48.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.840+0000 7f92a89c2640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f92a415acb0 0x7f92a41f3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:59712/0 (socket says 172.21.15.55:59712) 2024-03-19T01:57:48.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.840+0000 7f92a89c2640 1 -- 172.21.15.55:0/1019584001 learned_addr learned my addr 172.21.15.55:0/1019584001 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:48.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.844+0000 7f92a3fff640 1 --2- 172.21.15.55:0/1019584001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92a406b2a0 0x7f92a41e5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:48.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.848+0000 7f92a37fe640 1 -- 172.21.15.55:0/1019584001 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f92a415acb0 msgr2=0x7f92a41f3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:48.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.848+0000 7f92a37fe640 1 --2- 172.21.15.55:0/1019584001 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f92a415acb0 0x7f92a41f3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:48.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.848+0000 7f92a37fe640 1 -- 172.21.15.55:0/1019584001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92a406b2a0 msgr2=0x7f92a41e5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:48.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.848+0000 7f92a37fe640 1 --2- 172.21.15.55:0/1019584001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92a406b2a0 0x7f92a41e5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:48.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.848+0000 7f92a37fe640 1 -- 172.21.15.55:0/1019584001 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a41f44e0 con 0x7f92a41544f0 2024-03-19T01:57:48.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.852+0000 7f92a37fe640 1 --2- 172.21.15.55:0/1019584001 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f92a41544f0 0x7f92a414fbf0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f929800c410 tx=0x7f929800c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:48.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.856+0000 7f92a0ff9640 1 -- 172.21.15.55:0/1019584001 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9298017070 con 0x7f92a41544f0 2024-03-19T01:57:48.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.856+0000 7f92a0ff9640 1 -- 172.21.15.55:0/1019584001 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92980069e0 con 0x7f92a41544f0 2024-03-19T01:57:48.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.856+0000 7f92a0ff9640 1 -- 172.21.15.55:0/1019584001 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9298006e10 con 0x7f92a41544f0 2024-03-19T01:57:48.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.856+0000 7f92aa425640 1 -- 172.21.15.55:0/1019584001 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92a40cb6e0 con 0x7f92a41544f0 2024-03-19T01:57:48.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.856+0000 7f92aa425640 1 -- 172.21.15.55:0/1019584001 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f92a41e4890 con 0x7f92a41544f0 2024-03-19T01:57:48.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.860+0000 7f92aa425640 1 -- 172.21.15.55:0/1019584001 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92a4068a40 con 0x7f92a41544f0 2024-03-19T01:57:48.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.896+0000 7f92a0ff9640 1 -- 172.21.15.55:0/1019584001 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f929801a070 con 0x7f92a41544f0 2024-03-19T01:57:48.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.896+0000 7f92a0ff9640 1 --2- 172.21.15.55:0/1019584001 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92900676b0 0x7f9290069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:48.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.896+0000 7f92a3fff640 1 --2- 172.21.15.55:0/1019584001 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92900676b0 0x7f9290069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:48.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.896+0000 7f92a3fff640 1 --2- 172.21.15.55:0/1019584001 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92900676b0 0x7f9290069b70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9294009b60 tx=0x7f9294009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:48.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:48.900+0000 7f92a0ff9640 1 -- 172.21.15.55:0/1019584001 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f92980536e0 con 0x7f92a41544f0 2024-03-19T01:57:49.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:49.204+0000 7f92a0ff9640 1 -- 172.21.15.55:0/1019584001 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f9298021020 con 0x7f92a41544f0 2024-03-19T01:57:49.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:49.512+0000 7f92aa425640 1 -- 172.21.15.55:0/1019584001 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92a41e6520 con 0x7f92a41544f0 2024-03-19T01:57:49.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:49.568+0000 7f92a0ff9640 1 -- 172.21.15.55:0/1019584001 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f9298055640 con 0x7f92a41544f0 2024-03-19T01:57:49.571 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:49.571 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:49.571 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:49.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:49.572+0000 7f92aa425640 1 -- 172.21.15.55:0/1019584001 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92900676b0 msgr2=0x7f9290069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:49.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:49.572+0000 7f92aa425640 1 --2- 172.21.15.55:0/1019584001 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92900676b0 0x7f9290069b70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9294009b60 tx=0x7f9294009190 comp rx=0 tx=0).stop 2024-03-19T01:57:49.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:49.572+0000 7f92aa425640 1 -- 172.21.15.55:0/1019584001 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f92a41544f0 msgr2=0x7f92a414fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:49.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:49.572+0000 7f92aa425640 1 --2- 172.21.15.55:0/1019584001 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f92a41544f0 0x7f92a414fbf0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f929800c410 tx=0x7f929800c8d0 comp rx=0 tx=0).stop 2024-03-19T01:57:49.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:49.572+0000 7f92aa425640 1 -- 172.21.15.55:0/1019584001 shutdown_connections 2024-03-19T01:57:49.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:49.572+0000 7f92aa425640 1 --2- 172.21.15.55:0/1019584001 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92900676b0 0x7f9290069b70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:49.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:49.572+0000 7f92aa425640 1 --2- 172.21.15.55:0/1019584001 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f92a415acb0 0x7f92a41f3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:49.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:49.572+0000 7f92aa425640 1 --2- 172.21.15.55:0/1019584001 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f92a41544f0 0x7f92a414fbf0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:49.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:49.572+0000 7f92aa425640 1 --2- 172.21.15.55:0/1019584001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92a406b2a0 0x7f92a41e5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:49.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:49.572+0000 7f92aa425640 1 -- 172.21.15.55:0/1019584001 >> 172.21.15.55:0/1019584001 conn(0x7f92a406e670 msgr2=0x7f92a4070690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:49.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:49.572+0000 7f92aa425640 1 -- 172.21.15.55:0/1019584001 shutdown_connections 2024-03-19T01:57:49.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:49.572+0000 7f92aa425640 1 -- 172.21.15.55:0/1019584001 wait complete. 2024-03-19T01:57:50.195 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:50.196 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:51.196 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:51.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.316+0000 7fb34cf0d640 1 Processor -- start 2024-03-19T01:57:51.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.316+0000 7fb34cf0d640 1 -- start start 2024-03-19T01:57:51.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.316+0000 7fb34cf0d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb348152120 0x7fb3480c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:51.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.316+0000 7fb34cf0d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb348070780 0x7fb3481586e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:51.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.316+0000 7fb34cf0d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb34806fff0 con 0x7fb3480c9610 2024-03-19T01:57:51.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.316+0000 7fb34cf0d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fb348152500 con 0x7fb348070780 2024-03-19T01:57:51.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.316+0000 7fb34cf0d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fb348070590 con 0x7fb348152120 2024-03-19T01:57:51.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.320+0000 7fb346d9d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb348070780 0x7fb3481586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:51.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.320+0000 7fb34659c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb348152120 0x7fb3480c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:51.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.320+0000 7fb345d9b640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fb3480c9610 0x7fb348157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:42616/0 (socket says 172.21.15.55:42616) 2024-03-19T01:57:51.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.320+0000 7fb345d9b640 1 -- 172.21.15.55:0/2147396230 learned_addr learned my addr 172.21.15.55:0/2147396230 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:57:51.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.320+0000 7fb346d9d640 1 -- 172.21.15.55:0/2147396230 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb348152120 msgr2=0x7fb3480c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:51.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.320+0000 7fb346d9d640 1 --2- 172.21.15.55:0/2147396230 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb348152120 0x7fb3480c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:51.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.320+0000 7fb346d9d640 1 -- 172.21.15.55:0/2147396230 >> v1:172.21.15.84:6789/0 conn(0x7fb3480c9610 legacy=0x7fb348157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:51.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.320+0000 7fb346d9d640 1 -- 172.21.15.55:0/2147396230 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3480c39c0 con 0x7fb348070780 2024-03-19T01:57:51.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.324+0000 7fb346d9d640 1 --2- 172.21.15.55:0/2147396230 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb348070780 0x7fb3481586e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb340009d90 tx=0x7fb34001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6bd02b0cb07cd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:51.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.332+0000 7fb34559a640 1 -- 172.21.15.55:0/2147396230 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3400b2070 con 0x7fb348070780 2024-03-19T01:57:51.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.332+0000 7fb34559a640 1 -- 172.21.15.55:0/2147396230 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3400a5830 con 0x7fb348070780 2024-03-19T01:57:51.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.332+0000 7fb34559a640 1 -- 172.21.15.55:0/2147396230 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3400a5c00 con 0x7fb348070780 2024-03-19T01:57:51.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.332+0000 7fb34cf0d640 1 -- 172.21.15.55:0/2147396230 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb348070780 msgr2=0x7fb3481586e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:51.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.332+0000 7fb34cf0d640 1 --2- 172.21.15.55:0/2147396230 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb348070780 0x7fb3481586e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb340009d90 tx=0x7fb34001c970 comp rx=0 tx=0).stop 2024-03-19T01:57:51.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.332+0000 7fb34cf0d640 1 -- 172.21.15.55:0/2147396230 shutdown_connections 2024-03-19T01:57:51.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.332+0000 7fb34cf0d640 1 --2- 172.21.15.55:0/2147396230 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb348070780 0x7fb3481586e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:51.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.332+0000 7fb34cf0d640 1 --2- 172.21.15.55:0/2147396230 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb348152120 0x7fb3480c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:51.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.332+0000 7fb34cf0d640 1 -- 172.21.15.55:0/2147396230 >> 172.21.15.55:0/2147396230 conn(0x7fb348067600 msgr2=0x7fb348071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:51.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.332+0000 7fb34cf0d640 1 -- 172.21.15.55:0/2147396230 shutdown_connections 2024-03-19T01:57:51.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.332+0000 7fb34cf0d640 1 -- 172.21.15.55:0/2147396230 wait complete. 2024-03-19T01:57:51.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.332+0000 7fb34cf0d640 1 Processor -- start 2024-03-19T01:57:51.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.332+0000 7fb34cf0d640 1 -- start start 2024-03-19T01:57:51.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.336+0000 7fb34cf0d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb348070780 0x7fb3480b1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:51.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.336+0000 7fb34cf0d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb3480c9610 0x7fb3481f3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:51.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.336+0000 7fb34cf0d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb348152120 0x7fb348160b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:51.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.336+0000 7fb34cf0d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb3480c86d0 con 0x7fb348070780 2024-03-19T01:57:51.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.336+0000 7fb34cf0d640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb3480c31c0 con 0x7fb348152120 2024-03-19T01:57:51.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.336+0000 7fb34cf0d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fb348165170 con 0x7fb3480c9610 2024-03-19T01:57:51.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.336+0000 7fb34659c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb348070780 0x7fb3480b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:51.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.336+0000 7fb345d9b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb3480c9610 0x7fb3481f3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:51.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.336+0000 7fb346d9d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb348152120 0x7fb348160b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:51.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.336+0000 7fb34659c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb348070780 0x7fb3480b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58994/0 (socket says 172.21.15.55:58994) 2024-03-19T01:57:51.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.336+0000 7fb34659c640 1 -- 172.21.15.55:0/3588606606 learned_addr learned my addr 172.21.15.55:0/3588606606 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:51.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.340+0000 7fb34659c640 1 -- 172.21.15.55:0/3588606606 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb3480c9610 msgr2=0x7fb3481f3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:51.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.340+0000 7fb34659c640 1 --2- 172.21.15.55:0/3588606606 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb3480c9610 0x7fb3481f3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:51.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.340+0000 7fb34659c640 1 -- 172.21.15.55:0/3588606606 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb348152120 msgr2=0x7fb348160b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:51.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.340+0000 7fb34659c640 1 --2- 172.21.15.55:0/3588606606 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb348152120 0x7fb348160b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:51.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.340+0000 7fb34659c640 1 -- 172.21.15.55:0/3588606606 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb348161cf0 con 0x7fb348070780 2024-03-19T01:57:51.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.340+0000 7fb34659c640 1 --2- 172.21.15.55:0/3588606606 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb348070780 0x7fb3480b1070 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb33800e8a0 tx=0x7fb33800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:51.346 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.344+0000 7fb336ffd640 1 -- 172.21.15.55:0/3588606606 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb338017070 con 0x7fb348070780 2024-03-19T01:57:51.346 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.344+0000 7fb336ffd640 1 -- 172.21.15.55:0/3588606606 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb33800c940 con 0x7fb348070780 2024-03-19T01:57:51.346 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.344+0000 7fb34cf0d640 1 -- 172.21.15.55:0/3588606606 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb348161f90 con 0x7fb348070780 2024-03-19T01:57:51.346 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.344+0000 7fb336ffd640 1 -- 172.21.15.55:0/3588606606 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb33800ccf0 con 0x7fb348070780 2024-03-19T01:57:51.346 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.344+0000 7fb34cf0d640 1 -- 172.21.15.55:0/3588606606 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3481f6db0 con 0x7fb348070780 2024-03-19T01:57:51.347 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.348+0000 7fb34cf0d640 1 -- 172.21.15.55:0/3588606606 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3481f68e0 con 0x7fb348070780 2024-03-19T01:57:51.622 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.620+0000 7fb336ffd640 1 -- 172.21.15.55:0/3588606606 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb33801a070 con 0x7fb348070780 2024-03-19T01:57:51.622 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.624+0000 7fb336ffd640 1 --2- 172.21.15.55:0/3588606606 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb3280676b0 0x7fb328069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:51.623 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.624+0000 7fb345d9b640 1 --2- 172.21.15.55:0/3588606606 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb3280676b0 0x7fb328069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:51.624 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.624+0000 7fb345d9b640 1 --2- 172.21.15.55:0/3588606606 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb3280676b0 0x7fb328069b70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb3480ca4e0 tx=0x7fb33c007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:51.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.652+0000 7fb336ffd640 1 -- 172.21.15.55:0/3588606606 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fb338054570 con 0x7fb348070780 2024-03-19T01:57:51.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.652+0000 7fb336ffd640 1 -- 172.21.15.55:0/3588606606 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fb3380564d0 con 0x7fb348070780 2024-03-19T01:57:51.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:51.884+0000 7fb34cf0d640 1 -- 172.21.15.55:0/3588606606 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb348068a40 con 0x7fb348070780 2024-03-19T01:57:51.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:52.000+0000 7fb336ffd640 1 -- 172.21.15.55:0/3588606606 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fb33805a570 con 0x7fb348070780 2024-03-19T01:57:51.999 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:51.999 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:52.000 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:52.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:52.000+0000 7fb34cf0d640 1 -- 172.21.15.55:0/3588606606 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb3280676b0 msgr2=0x7fb328069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:52.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:52.000+0000 7fb34cf0d640 1 --2- 172.21.15.55:0/3588606606 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb3280676b0 0x7fb328069b70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb3480ca4e0 tx=0x7fb33c007e90 comp rx=0 tx=0).stop 2024-03-19T01:57:52.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:52.000+0000 7fb34cf0d640 1 -- 172.21.15.55:0/3588606606 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb348070780 msgr2=0x7fb3480b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:52.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:52.000+0000 7fb34cf0d640 1 --2- 172.21.15.55:0/3588606606 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb348070780 0x7fb3480b1070 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb33800e8a0 tx=0x7fb33800eca0 comp rx=0 tx=0).stop 2024-03-19T01:57:52.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:52.000+0000 7fb34cf0d640 1 -- 172.21.15.55:0/3588606606 shutdown_connections 2024-03-19T01:57:52.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:52.000+0000 7fb34cf0d640 1 --2- 172.21.15.55:0/3588606606 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb3280676b0 0x7fb328069b70 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:52.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:52.000+0000 7fb34cf0d640 1 --2- 172.21.15.55:0/3588606606 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb348152120 0x7fb348160b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:52.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:52.000+0000 7fb34cf0d640 1 --2- 172.21.15.55:0/3588606606 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb3480c9610 0x7fb3481f3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:52.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:52.000+0000 7fb34cf0d640 1 --2- 172.21.15.55:0/3588606606 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb348070780 0x7fb3480b1070 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:52.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:52.000+0000 7fb34cf0d640 1 -- 172.21.15.55:0/3588606606 >> 172.21.15.55:0/3588606606 conn(0x7fb348067600 msgr2=0x7fb3480c9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:52.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:52.000+0000 7fb34cf0d640 1 -- 172.21.15.55:0/3588606606 shutdown_connections 2024-03-19T01:57:52.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:52.000+0000 7fb34cf0d640 1 -- 172.21.15.55:0/3588606606 wait complete. 2024-03-19T01:57:52.632 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:52.632 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:53.633 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:53.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.748+0000 7fbc9fef6640 1 Processor -- start 2024-03-19T01:57:53.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.748+0000 7fbc9fef6640 1 -- start start 2024-03-19T01:57:53.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.748+0000 7fbc9fef6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbc9815a490 0x7fbc98072900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:53.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.748+0000 7fbc9fef6640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fbc980ca350 0x7fbc98070500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:53.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.748+0000 7fbc9fef6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc980738b0 con 0x7fbc98150e50 2024-03-19T01:57:53.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.748+0000 7fbc9fef6640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fbc98072f40 con 0x7fbc980ca350 2024-03-19T01:57:53.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.748+0000 7fbc9fef6640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbc980732f0 con 0x7fbc9815a490 2024-03-19T01:57:53.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.752+0000 7fbc9d491640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbc9815a490 0x7fbc98072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:53.752 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.752+0000 7fbc9e493640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fbc980ca350 0x7fbc98070500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:53.752 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.752+0000 7fbc9d491640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbc9815a490 0x7fbc98072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:58266/0 (socket says 172.21.15.55:58266) 2024-03-19T01:57:53.752 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.752+0000 7fbc9d491640 1 -- 172.21.15.55:0/2776043445 learned_addr learned my addr 172.21.15.55:0/2776043445 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:53.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.756+0000 7fbc9d491640 1 -- 172.21.15.55:0/2776043445 >> v2:172.21.15.84:3300/0 conn(0x7fbc980ca350 msgr2=0x7fbc98070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:53.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.756+0000 7fbc9d491640 1 --2- 172.21.15.55:0/2776043445 >> v2:172.21.15.84:3300/0 conn(0x7fbc980ca350 0x7fbc98070500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:53.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.756+0000 7fbc9d491640 1 -- 172.21.15.55:0/2776043445 >> v1:172.21.15.55:6789/0 conn(0x7fbc98150e50 legacy=0x7fbc981578b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:53.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.756+0000 7fbc9d491640 1 -- 172.21.15.55:0/2776043445 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc980735d0 con 0x7fbc9815a490 2024-03-19T01:57:53.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.760+0000 7fbc9d491640 1 --2- 172.21.15.55:0/2776043445 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbc9815a490 0x7fbc98072900 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbc88009870 tx=0x7fbc8801c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e559d9bd4682ced server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:53.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.764+0000 7fbc9cc90640 1 -- 172.21.15.55:0/2776043445 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc880b2070 con 0x7fbc9815a490 2024-03-19T01:57:53.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.764+0000 7fbc9cc90640 1 -- 172.21.15.55:0/2776043445 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc880a9070 con 0x7fbc9815a490 2024-03-19T01:57:53.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.764+0000 7fbc9cc90640 1 -- 172.21.15.55:0/2776043445 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc880ad880 con 0x7fbc9815a490 2024-03-19T01:57:53.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.764+0000 7fbc9fef6640 1 -- 172.21.15.55:0/2776043445 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbc9815a490 msgr2=0x7fbc98072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:53.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.764+0000 7fbc9fef6640 1 --2- 172.21.15.55:0/2776043445 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbc9815a490 0x7fbc98072900 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbc88009870 tx=0x7fbc8801c560 comp rx=0 tx=0).stop 2024-03-19T01:57:53.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.764+0000 7fbc9fef6640 1 -- 172.21.15.55:0/2776043445 shutdown_connections 2024-03-19T01:57:53.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.764+0000 7fbc9fef6640 1 --2- 172.21.15.55:0/2776043445 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbc9815a490 0x7fbc98072900 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:53.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.764+0000 7fbc9fef6640 1 --2- 172.21.15.55:0/2776043445 >> v2:172.21.15.84:3300/0 conn(0x7fbc980ca350 0x7fbc98070500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:53.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.764+0000 7fbc9fef6640 1 -- 172.21.15.55:0/2776043445 >> 172.21.15.55:0/2776043445 conn(0x7fbc9806c3b0 msgr2=0x7fbc9806c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:53.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.764+0000 7fbc9fef6640 1 -- 172.21.15.55:0/2776043445 shutdown_connections 2024-03-19T01:57:53.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.764+0000 7fbc9fef6640 1 -- 172.21.15.55:0/2776043445 wait complete. 2024-03-19T01:57:53.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.764+0000 7fbc9fef6640 1 Processor -- start 2024-03-19T01:57:53.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.768+0000 7fbc9fef6640 1 -- start start 2024-03-19T01:57:53.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.768+0000 7fbc9fef6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbc980ca350 0x7fbc9814d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:53.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.768+0000 7fbc9fef6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc98150e50 0x7fbc981f3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:53.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.768+0000 7fbc9fef6640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc9815a490 0x7fbc98145f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:53.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.768+0000 7fbc9fef6640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbc981513b0 con 0x7fbc98150e50 2024-03-19T01:57:53.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.768+0000 7fbc9fef6640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fbc98151230 con 0x7fbc9815a490 2024-03-19T01:57:53.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.768+0000 7fbc9fef6640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbc980c1b60 con 0x7fbc980ca350 2024-03-19T01:57:53.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.768+0000 7fbc9d491640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc98150e50 0x7fbc981f3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:53.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.768+0000 7fbc9d491640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc98150e50 0x7fbc981f3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59008/0 (socket says 172.21.15.55:59008) 2024-03-19T01:57:53.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.768+0000 7fbc9d491640 1 -- 172.21.15.55:0/2904590114 learned_addr learned my addr 172.21.15.55:0/2904590114 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:53.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.768+0000 7fbc9e493640 1 --2- 172.21.15.55:0/2904590114 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc9815a490 0x7fbc98145f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:53.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.768+0000 7fbc9dc92640 1 --2- 172.21.15.55:0/2904590114 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbc980ca350 0x7fbc9814d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:53.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.772+0000 7fbc9d491640 1 -- 172.21.15.55:0/2904590114 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbc980ca350 msgr2=0x7fbc9814d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:53.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.772+0000 7fbc9d491640 1 --2- 172.21.15.55:0/2904590114 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbc980ca350 0x7fbc9814d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:53.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.772+0000 7fbc9d491640 1 -- 172.21.15.55:0/2904590114 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc9815a490 msgr2=0x7fbc98145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:53.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.772+0000 7fbc9d491640 1 --2- 172.21.15.55:0/2904590114 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc9815a490 0x7fbc98145f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:53.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.772+0000 7fbc9d491640 1 -- 172.21.15.55:0/2904590114 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc980c8540 con 0x7fbc98150e50 2024-03-19T01:57:53.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.772+0000 7fbc9d491640 1 --2- 172.21.15.55:0/2904590114 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc98150e50 0x7fbc981f3980 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbc88019170 tx=0x7fbc88019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:53.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.776+0000 7fbc8e7fc640 1 -- 172.21.15.55:0/2904590114 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc880bc070 con 0x7fbc98150e50 2024-03-19T01:57:53.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.776+0000 7fbc8e7fc640 1 -- 172.21.15.55:0/2904590114 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc880b3040 con 0x7fbc98150e50 2024-03-19T01:57:53.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.780+0000 7fbc9fef6640 1 -- 172.21.15.55:0/2904590114 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc9814a2b0 con 0x7fbc98150e50 2024-03-19T01:57:53.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.780+0000 7fbc8e7fc640 1 -- 172.21.15.55:0/2904590114 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc880b2070 con 0x7fbc98150e50 2024-03-19T01:57:53.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.780+0000 7fbc9fef6640 1 -- 172.21.15.55:0/2904590114 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc981f7490 con 0x7fbc98150e50 2024-03-19T01:57:53.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:53.780+0000 7fbc9fef6640 1 -- 172.21.15.55:0/2904590114 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc98068a40 con 0x7fbc98150e50 2024-03-19T01:57:54.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:54.064+0000 7fbc8e7fc640 1 -- 172.21.15.55:0/2904590114 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fbc880bf020 con 0x7fbc98150e50 2024-03-19T01:57:54.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:54.064+0000 7fbc8e7fc640 1 --2- 172.21.15.55:0/2904590114 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbc7c0676e0 0x7fbc7c069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:54.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:54.064+0000 7fbc9dc92640 1 --2- 172.21.15.55:0/2904590114 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbc7c0676e0 0x7fbc7c069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:54.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:54.068+0000 7fbc9dc92640 1 --2- 172.21.15.55:0/2904590114 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbc7c0676e0 0x7fbc7c069ba0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbc94001770 tx=0x7fbc94009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:54.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:54.104+0000 7fbc8e7fc640 1 -- 172.21.15.55:0/2904590114 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fbc880ed160 con 0x7fbc98150e50 2024-03-19T01:57:54.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:54.104+0000 7fbc8e7fc640 1 -- 172.21.15.55:0/2904590114 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fbc88016150 con 0x7fbc98150e50 2024-03-19T01:57:54.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:54.304+0000 7fbc9fef6640 1 -- 172.21.15.55:0/2904590114 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc9814e590 con 0x7fbc98150e50 2024-03-19T01:57:54.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:54.332+0000 7fbc8e7fc640 1 -- 172.21.15.55:0/2904590114 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fbc880ef0c0 con 0x7fbc98150e50 2024-03-19T01:57:54.331 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:54.331 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:54.332 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:54.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:54.332+0000 7fbc9fef6640 1 -- 172.21.15.55:0/2904590114 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbc7c0676e0 msgr2=0x7fbc7c069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:54.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:54.332+0000 7fbc9fef6640 1 --2- 172.21.15.55:0/2904590114 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbc7c0676e0 0x7fbc7c069ba0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbc94001770 tx=0x7fbc94009040 comp rx=0 tx=0).stop 2024-03-19T01:57:54.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:54.332+0000 7fbc9fef6640 1 -- 172.21.15.55:0/2904590114 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc98150e50 msgr2=0x7fbc981f3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:54.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:54.332+0000 7fbc9fef6640 1 --2- 172.21.15.55:0/2904590114 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc98150e50 0x7fbc981f3980 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbc88019170 tx=0x7fbc88019300 comp rx=0 tx=0).stop 2024-03-19T01:57:54.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:54.332+0000 7fbc9fef6640 1 -- 172.21.15.55:0/2904590114 shutdown_connections 2024-03-19T01:57:54.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:54.332+0000 7fbc9fef6640 1 --2- 172.21.15.55:0/2904590114 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbc7c0676e0 0x7fbc7c069ba0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:54.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:54.332+0000 7fbc9fef6640 1 --2- 172.21.15.55:0/2904590114 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc9815a490 0x7fbc98145f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:54.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:54.332+0000 7fbc9fef6640 1 --2- 172.21.15.55:0/2904590114 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbc98150e50 0x7fbc981f3980 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:54.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:54.332+0000 7fbc9fef6640 1 --2- 172.21.15.55:0/2904590114 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbc980ca350 0x7fbc9814d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:54.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:54.332+0000 7fbc9fef6640 1 -- 172.21.15.55:0/2904590114 >> 172.21.15.55:0/2904590114 conn(0x7fbc9806c3b0 msgr2=0x7fbc98154ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:54.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:54.332+0000 7fbc9fef6640 1 -- 172.21.15.55:0/2904590114 shutdown_connections 2024-03-19T01:57:54.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:54.332+0000 7fbc9fef6640 1 -- 172.21.15.55:0/2904590114 wait complete. 2024-03-19T01:57:54.953 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:54.953 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:55.955 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:56.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.068+0000 7fcf10d0b640 1 Processor -- start 2024-03-19T01:57:56.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.068+0000 7fcf10d0b640 1 -- start start 2024-03-19T01:57:56.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.072+0000 7fcf10d0b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf0c154290 0x7fcf0c15e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:56.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.072+0000 7fcf10d0b640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fcf0c15ad40 0x7fcf0c154670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:56.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.072+0000 7fcf10d0b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf0c06bc90 con 0x7fcf0c0ca310 2024-03-19T01:57:56.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.072+0000 7fcf10d0b640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fcf0c15b2c0 con 0x7fcf0c15ad40 2024-03-19T01:57:56.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.072+0000 7fcf10d0b640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fcf0c06c1d0 con 0x7fcf0c154290 2024-03-19T01:57:56.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.072+0000 7fcf0a59c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf0c154290 0x7fcf0c15e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:56.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.072+0000 7fcf0a59c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf0c154290 0x7fcf0c15e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:58274/0 (socket says 172.21.15.55:58274) 2024-03-19T01:57:56.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.072+0000 7fcf0a59c640 1 -- 172.21.15.55:0/1742156128 learned_addr learned my addr 172.21.15.55:0/1742156128 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:56.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.072+0000 7fcf0ad9d640 1 --2- 172.21.15.55:0/1742156128 >> v2:172.21.15.84:3300/0 conn(0x7fcf0c15ad40 0x7fcf0c154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:56.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.076+0000 7fcf0a59c640 1 -- 172.21.15.55:0/1742156128 >> v2:172.21.15.84:3300/0 conn(0x7fcf0c15ad40 msgr2=0x7fcf0c154670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:56.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.076+0000 7fcf0a59c640 1 --2- 172.21.15.55:0/1742156128 >> v2:172.21.15.84:3300/0 conn(0x7fcf0c15ad40 0x7fcf0c154670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:56.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.076+0000 7fcf0a59c640 1 -- 172.21.15.55:0/1742156128 >> v1:172.21.15.84:6789/0 conn(0x7fcf0c0ca310 legacy=0x7fcf0c070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:56.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.076+0000 7fcf0a59c640 1 -- 172.21.15.55:0/1742156128 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf0c06bf30 con 0x7fcf0c154290 2024-03-19T01:57:56.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.080+0000 7fcf0a59c640 1 --2- 172.21.15.55:0/1742156128 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf0c154290 0x7fcf0c15e4c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcefc009870 tx=0x7fcefc01c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b54c8aac932e6b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:56.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.084+0000 7fcf09d9b640 1 -- 172.21.15.55:0/1742156128 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc0a7070 con 0x7fcf0c154290 2024-03-19T01:57:56.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.084+0000 7fcf09d9b640 1 -- 172.21.15.55:0/1742156128 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcefc0b0070 con 0x7fcf0c154290 2024-03-19T01:57:56.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.084+0000 7fcf09d9b640 1 -- 172.21.15.55:0/1742156128 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc0ab890 con 0x7fcf0c154290 2024-03-19T01:57:56.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.084+0000 7fcf10d0b640 1 -- 172.21.15.55:0/1742156128 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf0c154290 msgr2=0x7fcf0c15e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:56.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.084+0000 7fcf10d0b640 1 --2- 172.21.15.55:0/1742156128 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf0c154290 0x7fcf0c15e4c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcefc009870 tx=0x7fcefc01c560 comp rx=0 tx=0).stop 2024-03-19T01:57:56.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.084+0000 7fcf10d0b640 1 -- 172.21.15.55:0/1742156128 shutdown_connections 2024-03-19T01:57:56.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.084+0000 7fcf10d0b640 1 --2- 172.21.15.55:0/1742156128 >> v2:172.21.15.84:3300/0 conn(0x7fcf0c15ad40 0x7fcf0c154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:56.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.084+0000 7fcf10d0b640 1 --2- 172.21.15.55:0/1742156128 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf0c154290 0x7fcf0c15e4c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:56.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.084+0000 7fcf10d0b640 1 -- 172.21.15.55:0/1742156128 >> 172.21.15.55:0/1742156128 conn(0x7fcf0c06c580 msgr2=0x7fcf0c06c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:56.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.084+0000 7fcf10d0b640 1 -- 172.21.15.55:0/1742156128 shutdown_connections 2024-03-19T01:57:56.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.084+0000 7fcf10d0b640 1 -- 172.21.15.55:0/1742156128 wait complete. 2024-03-19T01:57:56.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.088+0000 7fcf10d0b640 1 Processor -- start 2024-03-19T01:57:56.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.088+0000 7fcf10d0b640 1 -- start start 2024-03-19T01:57:56.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.088+0000 7fcf10d0b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcf0c0ca310 0x7fcf0c0cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:56.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.088+0000 7fcf10d0b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf0c154290 0x7fcf0c0cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:56.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.088+0000 7fcf10d0b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf0c15ad40 0x7fcf0c14d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:56.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.088+0000 7fcf10d0b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0c0b2460 con 0x7fcf0c154290 2024-03-19T01:57:56.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.088+0000 7fcf10d0b640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0c0b22e0 con 0x7fcf0c0ca310 2024-03-19T01:57:56.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.088+0000 7fcf10d0b640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fcf0c070d50 con 0x7fcf0c15ad40 2024-03-19T01:57:56.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.088+0000 7fcf01d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf0c154290 0x7fcf0c0cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:56.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.088+0000 7fcf01d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf0c154290 0x7fcf0c0cd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59016/0 (socket says 172.21.15.55:59016) 2024-03-19T01:57:56.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.088+0000 7fcf01d9b640 1 -- 172.21.15.55:0/3459689905 learned_addr learned my addr 172.21.15.55:0/3459689905 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:56.092 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.092+0000 7fcf0a59c640 1 --2- 172.21.15.55:0/3459689905 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcf0c0ca310 0x7fcf0c0cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:56.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.092+0000 7fcf0ad9d640 1 --2- 172.21.15.55:0/3459689905 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf0c15ad40 0x7fcf0c14d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:56.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.096+0000 7fcf01d9b640 1 -- 172.21.15.55:0/3459689905 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf0c15ad40 msgr2=0x7fcf0c14d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:56.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.096+0000 7fcf01d9b640 1 --2- 172.21.15.55:0/3459689905 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf0c15ad40 0x7fcf0c14d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:56.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.096+0000 7fcf01d9b640 1 -- 172.21.15.55:0/3459689905 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcf0c0ca310 msgr2=0x7fcf0c0cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:56.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.096+0000 7fcf01d9b640 1 --2- 172.21.15.55:0/3459689905 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcf0c0ca310 0x7fcf0c0cceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:56.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.096+0000 7fcf01d9b640 1 -- 172.21.15.55:0/3459689905 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf0c157a50 con 0x7fcf0c154290 2024-03-19T01:57:56.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.100+0000 7fcf01d9b640 1 --2- 172.21.15.55:0/3459689905 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf0c154290 0x7fcf0c0cd3f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fcef400c3c0 tx=0x7fcef400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:56.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.104+0000 7fcf037fe640 1 -- 172.21.15.55:0/3459689905 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef4017070 con 0x7fcf0c154290 2024-03-19T01:57:56.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.108+0000 7fcf037fe640 1 -- 172.21.15.55:0/3459689905 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcef40052f0 con 0x7fcf0c154290 2024-03-19T01:57:56.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.108+0000 7fcf10d0b640 1 -- 172.21.15.55:0/3459689905 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf0c155170 con 0x7fcf0c154290 2024-03-19T01:57:56.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.108+0000 7fcf037fe640 1 -- 172.21.15.55:0/3459689905 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef4003840 con 0x7fcf0c154290 2024-03-19T01:57:56.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.108+0000 7fcf10d0b640 1 -- 172.21.15.55:0/3459689905 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf0c156e40 con 0x7fcf0c154290 2024-03-19T01:57:56.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.108+0000 7fcf10d0b640 1 -- 172.21.15.55:0/3459689905 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf0c0ccba0 con 0x7fcf0c154290 2024-03-19T01:57:56.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.140+0000 7fcf037fe640 1 -- 172.21.15.55:0/3459689905 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fcef401a070 con 0x7fcf0c154290 2024-03-19T01:57:56.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.140+0000 7fcf037fe640 1 --2- 172.21.15.55:0/3459689905 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fceec067660 0x7fceec069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:56.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.140+0000 7fcf0a59c640 1 --2- 172.21.15.55:0/3459689905 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fceec067660 0x7fceec069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:56.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.140+0000 7fcf0a59c640 1 --2- 172.21.15.55:0/3459689905 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fceec067660 0x7fceec069b20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0af000 tx=0x7fcefc0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:56.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.144+0000 7fcf037fe640 1 -- 172.21.15.55:0/3459689905 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fcef40533e0 con 0x7fcf0c154290 2024-03-19T01:57:56.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.428+0000 7fcf037fe640 1 -- 172.21.15.55:0/3459689905 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fcef4021080 con 0x7fcf0c154290 2024-03-19T01:57:56.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.664+0000 7fcf10d0b640 1 -- 172.21.15.55:0/3459689905 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf0c06b960 con 0x7fcf0c154290 2024-03-19T01:57:56.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.728+0000 7fcf037fe640 1 -- 172.21.15.55:0/3459689905 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fcef4055340 con 0x7fcf0c154290 2024-03-19T01:57:56.731 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:56.731 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:56.731 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:56.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.732+0000 7fcf10d0b640 1 -- 172.21.15.55:0/3459689905 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fceec067660 msgr2=0x7fceec069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:56.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.732+0000 7fcf10d0b640 1 --2- 172.21.15.55:0/3459689905 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fceec067660 0x7fceec069b20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0af000 tx=0x7fcefc0a7000 comp rx=0 tx=0).stop 2024-03-19T01:57:56.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.732+0000 7fcf10d0b640 1 -- 172.21.15.55:0/3459689905 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf0c154290 msgr2=0x7fcf0c0cd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:56.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.732+0000 7fcf10d0b640 1 --2- 172.21.15.55:0/3459689905 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf0c154290 0x7fcf0c0cd3f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fcef400c3c0 tx=0x7fcef400c880 comp rx=0 tx=0).stop 2024-03-19T01:57:56.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.732+0000 7fcf10d0b640 1 -- 172.21.15.55:0/3459689905 shutdown_connections 2024-03-19T01:57:56.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.732+0000 7fcf10d0b640 1 --2- 172.21.15.55:0/3459689905 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fceec067660 0x7fceec069b20 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:56.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.732+0000 7fcf10d0b640 1 --2- 172.21.15.55:0/3459689905 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf0c15ad40 0x7fcf0c14d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:56.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.732+0000 7fcf10d0b640 1 --2- 172.21.15.55:0/3459689905 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf0c154290 0x7fcf0c0cd3f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:56.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.732+0000 7fcf10d0b640 1 --2- 172.21.15.55:0/3459689905 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcf0c0ca310 0x7fcf0c0cceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:56.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.732+0000 7fcf10d0b640 1 -- 172.21.15.55:0/3459689905 >> 172.21.15.55:0/3459689905 conn(0x7fcf0c06c580 msgr2=0x7fcf0c1504f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:56.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.732+0000 7fcf10d0b640 1 -- 172.21.15.55:0/3459689905 shutdown_connections 2024-03-19T01:57:56.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:56.732+0000 7fcf10d0b640 1 -- 172.21.15.55:0/3459689905 wait complete. 2024-03-19T01:57:57.354 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:57.354 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:57:58.355 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:57:58.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.464+0000 7f0309bb4640 1 Processor -- start 2024-03-19T01:57:58.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.464+0000 7f0309bb4640 1 -- start start 2024-03-19T01:57:58.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.464+0000 7f0309bb4640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f030406b2a0 0x7f030415ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:58.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.464+0000 7f0309bb4640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f030406c110 0x7f030406c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:58.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.464+0000 7f0309bb4640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03040c8840 con 0x7f0304154300 2024-03-19T01:57:58.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.464+0000 7f0309bb4640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f030415b260 con 0x7f030406b2a0 2024-03-19T01:57:58.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.464+0000 7f0309bb4640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f03040c8ab0 con 0x7f030406c110 2024-03-19T01:57:58.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.464+0000 7f0303fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f030406c110 0x7f030406c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:58.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.468+0000 7f0303fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f030406c110 0x7f030406c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:58288/0 (socket says 172.21.15.55:58288) 2024-03-19T01:57:58.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.468+0000 7f0303fff640 1 -- 172.21.15.55:0/3032630421 learned_addr learned my addr 172.21.15.55:0/3032630421 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:58.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.468+0000 7f0303fff640 1 -- 172.21.15.55:0/3032630421 >> v2:172.21.15.55:3300/0 conn(0x7f030406b2a0 msgr2=0x7f030415ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:58.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.468+0000 7f0303fff640 1 --2- 172.21.15.55:0/3032630421 >> v2:172.21.15.55:3300/0 conn(0x7f030406b2a0 0x7f030415ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:58.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.468+0000 7f0303fff640 1 -- 172.21.15.55:0/3032630421 >> v1:172.21.15.55:6789/0 conn(0x7f0304154300 legacy=0x7f030415e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:57:58.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.468+0000 7f0303fff640 1 -- 172.21.15.55:0/3032630421 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03040c8d90 con 0x7f030406c110 2024-03-19T01:57:58.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.472+0000 7f0303fff640 1 --2- 172.21.15.55:0/3032630421 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f030406c110 0x7f030406c560 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f02fc009e50 tx=0x7f02fc01c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efac2de1e343b14f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:58.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.480+0000 7f03027fc640 1 -- 172.21.15.55:0/3032630421 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02fc0a5840 con 0x7f030406c110 2024-03-19T01:57:58.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.480+0000 7f03027fc640 1 -- 172.21.15.55:0/3032630421 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02fc0a59e0 con 0x7f030406c110 2024-03-19T01:57:58.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.480+0000 7f03027fc640 1 -- 172.21.15.55:0/3032630421 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02fc0a5db0 con 0x7f030406c110 2024-03-19T01:57:58.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.480+0000 7f0309bb4640 1 -- 172.21.15.55:0/3032630421 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f030406c110 msgr2=0x7f030406c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:58.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.480+0000 7f0309bb4640 1 --2- 172.21.15.55:0/3032630421 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f030406c110 0x7f030406c560 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f02fc009e50 tx=0x7f02fc01c9c0 comp rx=0 tx=0).stop 2024-03-19T01:57:58.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.480+0000 7f0309bb4640 1 -- 172.21.15.55:0/3032630421 shutdown_connections 2024-03-19T01:57:58.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.480+0000 7f0309bb4640 1 --2- 172.21.15.55:0/3032630421 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f030406c110 0x7f030406c560 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:58.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.480+0000 7f0309bb4640 1 --2- 172.21.15.55:0/3032630421 >> v2:172.21.15.55:3300/0 conn(0x7f030406b2a0 0x7f030415ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:58.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.480+0000 7f0309bb4640 1 -- 172.21.15.55:0/3032630421 >> 172.21.15.55:0/3032630421 conn(0x7f030406fb40 msgr2=0x7f030406ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:58.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.480+0000 7f0309bb4640 1 -- 172.21.15.55:0/3032630421 shutdown_connections 2024-03-19T01:57:58.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.480+0000 7f0309bb4640 1 -- 172.21.15.55:0/3032630421 wait complete. 2024-03-19T01:57:58.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.484+0000 7f0309bb4640 1 Processor -- start 2024-03-19T01:57:58.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.484+0000 7f0309bb4640 1 -- start start 2024-03-19T01:57:58.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.484+0000 7f0309bb4640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f030406b2a0 0x7f0304146ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:58.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.484+0000 7f0309bb4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f030406c110 0x7f0304141990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:58.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.484+0000 7f0309bb4640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0304154300 0x7f0304141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:58.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.484+0000 7f0309bb4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f030415e7d0 con 0x7f030406c110 2024-03-19T01:57:58.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.484+0000 7f0309bb4640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f030415dba0 con 0x7f0304154300 2024-03-19T01:57:58.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.484+0000 7f0309bb4640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f03040c74c0 con 0x7f030406b2a0 2024-03-19T01:57:58.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.484+0000 7f0303fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0304154300 0x7f0304141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:58.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.484+0000 7f0303fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0304154300 0x7f0304141ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:59778/0 (socket says 172.21.15.55:59778) 2024-03-19T01:57:58.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.484+0000 7f0303fff640 1 -- 172.21.15.55:0/99870266 learned_addr learned my addr 172.21.15.55:0/99870266 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:57:58.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.488+0000 7f0302ffd640 1 --2- 172.21.15.55:0/99870266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f030406c110 0x7f0304141990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:58.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.488+0000 7f03037fe640 1 --2- 172.21.15.55:0/99870266 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f030406b2a0 0x7f0304146ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:58.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.492+0000 7f0303fff640 1 -- 172.21.15.55:0/99870266 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f030406b2a0 msgr2=0x7f0304146ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:58.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.492+0000 7f0303fff640 1 --2- 172.21.15.55:0/99870266 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f030406b2a0 0x7f0304146ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:58.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.492+0000 7f0303fff640 1 -- 172.21.15.55:0/99870266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f030406c110 msgr2=0x7f0304141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:58.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.492+0000 7f0303fff640 1 --2- 172.21.15.55:0/99870266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f030406c110 0x7f0304141990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:58.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.492+0000 7f0303fff640 1 -- 172.21.15.55:0/99870266 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f030414e5b0 con 0x7f0304154300 2024-03-19T01:57:58.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.496+0000 7f0303fff640 1 --2- 172.21.15.55:0/99870266 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0304154300 0x7f0304141ed0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f02fc0af040 tx=0x7f02fc019760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:58.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.500+0000 7f02ebfff640 1 -- 172.21.15.55:0/99870266 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02fc0b8880 con 0x7f0304154300 2024-03-19T01:57:58.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.500+0000 7f02ebfff640 1 -- 172.21.15.55:0/99870266 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02fc0be070 con 0x7f0304154300 2024-03-19T01:57:58.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.500+0000 7f0309bb4640 1 -- 172.21.15.55:0/99870266 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0304150390 con 0x7f0304154300 2024-03-19T01:57:58.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.500+0000 7f02ebfff640 1 -- 172.21.15.55:0/99870266 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02fc0b8ca0 con 0x7f0304154300 2024-03-19T01:57:58.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.500+0000 7f0309bb4640 1 -- 172.21.15.55:0/99870266 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f030414e850 con 0x7f0304154300 2024-03-19T01:57:58.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.500+0000 7f0309bb4640 1 -- 172.21.15.55:0/99870266 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0304068a40 con 0x7f0304154300 2024-03-19T01:57:58.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.536+0000 7f02ebfff640 1 -- 172.21.15.55:0/99870266 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f02fc0c3460 con 0x7f0304154300 2024-03-19T01:57:58.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.540+0000 7f02ebfff640 1 --2- 172.21.15.55:0/99870266 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f02f0067660 0x7f02f0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:57:58.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.540+0000 7f03037fe640 1 --2- 172.21.15.55:0/99870266 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f02f0067660 0x7f02f0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:57:58.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.540+0000 7f03037fe640 1 --2- 172.21.15.55:0/99870266 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f02f0067660 0x7f02f0069b20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f02f80098e0 tx=0x7f02f8009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:57:58.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.800+0000 7f02ebfff640 1 -- 172.21.15.55:0/99870266 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f02fc0efd20 con 0x7f0304154300 2024-03-19T01:57:58.820 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:58.820+0000 7f02ebfff640 1 -- 172.21.15.55:0/99870266 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f02fc0165e0 con 0x7f0304154300 2024-03-19T01:57:59.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:59.044+0000 7f0309bb4640 1 -- 172.21.15.55:0/99870266 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0304068f00 con 0x7f0304154300 2024-03-19T01:57:59.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:59.116+0000 7f02ebfff640 1 -- 172.21.15.55:0/99870266 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f02fc0f1c80 con 0x7f0304154300 2024-03-19T01:57:59.118 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:57:59.119 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:57:59.119 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:57:59.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:59.120+0000 7f0309bb4640 1 -- 172.21.15.55:0/99870266 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f02f0067660 msgr2=0x7f02f0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:59.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:59.120+0000 7f0309bb4640 1 --2- 172.21.15.55:0/99870266 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f02f0067660 0x7f02f0069b20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f02f80098e0 tx=0x7f02f8009060 comp rx=0 tx=0).stop 2024-03-19T01:57:59.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:59.120+0000 7f0309bb4640 1 -- 172.21.15.55:0/99870266 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0304154300 msgr2=0x7f0304141ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:57:59.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:59.120+0000 7f0309bb4640 1 --2- 172.21.15.55:0/99870266 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0304154300 0x7f0304141ed0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f02fc0af040 tx=0x7f02fc019760 comp rx=0 tx=0).stop 2024-03-19T01:57:59.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:59.120+0000 7f0309bb4640 1 -- 172.21.15.55:0/99870266 shutdown_connections 2024-03-19T01:57:59.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:59.120+0000 7f0309bb4640 1 --2- 172.21.15.55:0/99870266 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f02f0067660 0x7f02f0069b20 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:59.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:59.120+0000 7f0309bb4640 1 --2- 172.21.15.55:0/99870266 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0304154300 0x7f0304141ed0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:59.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:59.120+0000 7f0309bb4640 1 --2- 172.21.15.55:0/99870266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f030406c110 0x7f0304141990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:59.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:59.120+0000 7f0309bb4640 1 --2- 172.21.15.55:0/99870266 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f030406b2a0 0x7f0304146ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:57:59.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:59.120+0000 7f0309bb4640 1 -- 172.21.15.55:0/99870266 >> 172.21.15.55:0/99870266 conn(0x7f030406fb40 msgr2=0x7f03040c8080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:57:59.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:59.120+0000 7f0309bb4640 1 -- 172.21.15.55:0/99870266 shutdown_connections 2024-03-19T01:57:59.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:57:59.120+0000 7f0309bb4640 1 -- 172.21.15.55:0/99870266 wait complete. 2024-03-19T01:57:59.746 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:57:59.746 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:00.747 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:00.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.876+0000 7f666c9e4640 1 Processor -- start 2024-03-19T01:58:00.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.876+0000 7f666c9e4640 1 -- start start 2024-03-19T01:58:00.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.876+0000 7f666c9e4640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f666806c6f0 0x7f666815d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:00.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.876+0000 7f666c9e4640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f66680cf1e0 0x7f666815de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:00.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.876+0000 7f666c9e4640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f666806cb90 con 0x7f66680cbf30 2024-03-19T01:58:00.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.876+0000 7f666c9e4640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f66680ce640 con 0x7f666806c6f0 2024-03-19T01:58:00.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.876+0000 7f666c9e4640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f66680cda30 con 0x7f66680cf1e0 2024-03-19T01:58:00.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.880+0000 7f6665d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f666806c6f0 0x7f666815d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:00.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.880+0000 7f6666d9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f66680cf1e0 0x7f666815de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:00.880 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.880+0000 7f6665d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f666806c6f0 0x7f666815d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34840/0 (socket says 172.21.15.55:34840) 2024-03-19T01:58:00.880 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.880+0000 7f6665d9b640 1 -- 172.21.15.55:0/2687302024 learned_addr learned my addr 172.21.15.55:0/2687302024 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:00.882 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.884+0000 7f6665d9b640 1 -- 172.21.15.55:0/2687302024 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f66680cf1e0 msgr2=0x7f666815de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:00.882 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.884+0000 7f6665d9b640 1 --2- 172.21.15.55:0/2687302024 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f66680cf1e0 0x7f666815de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:00.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.884+0000 7f6665d9b640 1 -- 172.21.15.55:0/2687302024 >> v1:172.21.15.55:6789/0 conn(0x7f66680cbf30 legacy=0x7f66680cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:00.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.884+0000 7f6665d9b640 1 -- 172.21.15.55:0/2687302024 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66680cdd10 con 0x7f666806c6f0 2024-03-19T01:58:00.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.888+0000 7f6665d9b640 1 --2- 172.21.15.55:0/2687302024 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f666806c6f0 0x7f666815d8d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f665c009870 tx=0x7f665c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea79361b4dbd3773 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:00.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.892+0000 7f666559a640 1 -- 172.21.15.55:0/2687302024 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665c0aa600 con 0x7f666806c6f0 2024-03-19T01:58:00.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.892+0000 7f666559a640 1 -- 172.21.15.55:0/2687302024 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f665c0af070 con 0x7f666806c6f0 2024-03-19T01:58:00.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.892+0000 7f666559a640 1 -- 172.21.15.55:0/2687302024 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665c0aa9c0 con 0x7f666806c6f0 2024-03-19T01:58:00.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.892+0000 7f666c9e4640 1 -- 172.21.15.55:0/2687302024 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f666806c6f0 msgr2=0x7f666815d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:00.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.892+0000 7f666c9e4640 1 --2- 172.21.15.55:0/2687302024 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f666806c6f0 0x7f666815d8d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f665c009870 tx=0x7f665c01c4a0 comp rx=0 tx=0).stop 2024-03-19T01:58:00.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.892+0000 7f666c9e4640 1 -- 172.21.15.55:0/2687302024 shutdown_connections 2024-03-19T01:58:00.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.892+0000 7f666c9e4640 1 --2- 172.21.15.55:0/2687302024 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f66680cf1e0 0x7f666815de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:00.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.892+0000 7f666c9e4640 1 --2- 172.21.15.55:0/2687302024 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f666806c6f0 0x7f666815d8d0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:00.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.892+0000 7f666c9e4640 1 -- 172.21.15.55:0/2687302024 >> 172.21.15.55:0/2687302024 conn(0x7f66680b16a0 msgr2=0x7f66680b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:00.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.892+0000 7f666c9e4640 1 -- 172.21.15.55:0/2687302024 shutdown_connections 2024-03-19T01:58:00.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.892+0000 7f666c9e4640 1 -- 172.21.15.55:0/2687302024 wait complete. 2024-03-19T01:58:00.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.896+0000 7f666c9e4640 1 Processor -- start 2024-03-19T01:58:00.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.896+0000 7f666c9e4640 1 -- start start 2024-03-19T01:58:00.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.896+0000 7f666c9e4640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f666806c6f0 0x7f666815d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:00.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.896+0000 7f666c9e4640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f66680cbf30 0x7f666814cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:00.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.896+0000 7f666c9e4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66680cf1e0 0x7f6668146dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:00.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.896+0000 7f666c9e4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f66680b23b0 con 0x7f66680cf1e0 2024-03-19T01:58:00.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.896+0000 7f666c9e4640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f666806ce50 con 0x7f666806c6f0 2024-03-19T01:58:00.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.896+0000 7f666c9e4640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f66680cc310 con 0x7f66680cbf30 2024-03-19T01:58:00.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.896+0000 7f6666d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66680cf1e0 0x7f6668146dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:00.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.896+0000 7f6665d9b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f66680cbf30 0x7f666814cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:00.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.896+0000 7f666659c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f666806c6f0 0x7f666815d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:00.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.896+0000 7f6666d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66680cf1e0 0x7f6668146dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34856/0 (socket says 172.21.15.55:34856) 2024-03-19T01:58:00.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.896+0000 7f6666d9d640 1 -- 172.21.15.55:0/4284519477 learned_addr learned my addr 172.21.15.55:0/4284519477 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:00.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.904+0000 7f6666d9d640 1 -- 172.21.15.55:0/4284519477 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f66680cbf30 msgr2=0x7f666814cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:00.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.904+0000 7f6666d9d640 1 --2- 172.21.15.55:0/4284519477 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f66680cbf30 0x7f666814cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:00.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.904+0000 7f6666d9d640 1 -- 172.21.15.55:0/4284519477 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f666806c6f0 msgr2=0x7f666815d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:00.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.904+0000 7f6666d9d640 1 --2- 172.21.15.55:0/4284519477 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f666806c6f0 0x7f666815d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:00.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.904+0000 7f6666d9d640 1 -- 172.21.15.55:0/4284519477 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66681f3680 con 0x7f66680cf1e0 2024-03-19T01:58:00.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.904+0000 7f6666d9d640 1 --2- 172.21.15.55:0/4284519477 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66680cf1e0 0x7f6668146dc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f666000efc0 tx=0x7f666000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:00.910 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.908+0000 7f6656ffd640 1 -- 172.21.15.55:0/4284519477 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6660018020 con 0x7f66680cf1e0 2024-03-19T01:58:00.911 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.908+0000 7f6656ffd640 1 -- 172.21.15.55:0/4284519477 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6660014480 con 0x7f66680cf1e0 2024-03-19T01:58:00.911 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.908+0000 7f666c9e4640 1 -- 172.21.15.55:0/4284519477 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f666814c080 con 0x7f66680cf1e0 2024-03-19T01:58:00.911 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.908+0000 7f6656ffd640 1 -- 172.21.15.55:0/4284519477 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6660014870 con 0x7f66680cf1e0 2024-03-19T01:58:00.911 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.908+0000 7f666c9e4640 1 -- 172.21.15.55:0/4284519477 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66681f3c00 con 0x7f66680cf1e0 2024-03-19T01:58:00.911 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:00.912+0000 7f666c9e4640 1 -- 172.21.15.55:0/4284519477 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6668068330 con 0x7f66680cf1e0 2024-03-19T01:58:01.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:01.188+0000 7f6656ffd640 1 -- 172.21.15.55:0/4284519477 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f666001b020 con 0x7f66680cf1e0 2024-03-19T01:58:01.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:01.188+0000 7f6656ffd640 1 --2- 172.21.15.55:0/4284519477 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f66480676b0 0x7f6648069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:01.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:01.188+0000 7f666659c640 1 --2- 172.21.15.55:0/4284519477 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f66480676b0 0x7f6648069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:01.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:01.188+0000 7f666659c640 1 --2- 172.21.15.55:0/4284519477 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f66480676b0 0x7f6648069b70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6658001770 tx=0x7f6658009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:01.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:01.204+0000 7f6656ffd640 1 -- 172.21.15.55:0/4284519477 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f66600552a0 con 0x7f66680cf1e0 2024-03-19T01:58:01.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:01.204+0000 7f6656ffd640 1 -- 172.21.15.55:0/4284519477 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f6660022030 con 0x7f66680cf1e0 2024-03-19T01:58:01.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:01.444+0000 7f666c9e4640 1 -- 172.21.15.55:0/4284519477 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66680b2190 con 0x7f66680cf1e0 2024-03-19T01:58:01.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:01.460+0000 7f6656ffd640 1 -- 172.21.15.55:0/4284519477 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f6660010040 con 0x7f66680cf1e0 2024-03-19T01:58:01.459 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:01.459 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:01.459 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:01.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:01.460+0000 7f666c9e4640 1 -- 172.21.15.55:0/4284519477 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f66480676b0 msgr2=0x7f6648069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:01.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:01.460+0000 7f666c9e4640 1 --2- 172.21.15.55:0/4284519477 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f66480676b0 0x7f6648069b70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6658001770 tx=0x7f6658009040 comp rx=0 tx=0).stop 2024-03-19T01:58:01.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:01.460+0000 7f666c9e4640 1 -- 172.21.15.55:0/4284519477 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66680cf1e0 msgr2=0x7f6668146dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:01.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:01.460+0000 7f666c9e4640 1 --2- 172.21.15.55:0/4284519477 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66680cf1e0 0x7f6668146dc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f666000efc0 tx=0x7f666000c490 comp rx=0 tx=0).stop 2024-03-19T01:58:01.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:01.460+0000 7f666c9e4640 1 -- 172.21.15.55:0/4284519477 shutdown_connections 2024-03-19T01:58:01.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:01.460+0000 7f666c9e4640 1 --2- 172.21.15.55:0/4284519477 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f66480676b0 0x7f6648069b70 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:01.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:01.460+0000 7f666c9e4640 1 --2- 172.21.15.55:0/4284519477 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66680cf1e0 0x7f6668146dc0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:01.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:01.460+0000 7f666c9e4640 1 --2- 172.21.15.55:0/4284519477 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f66680cbf30 0x7f666814cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:01.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:01.460+0000 7f666c9e4640 1 --2- 172.21.15.55:0/4284519477 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f666806c6f0 0x7f666815d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:01.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:01.460+0000 7f666c9e4640 1 -- 172.21.15.55:0/4284519477 >> 172.21.15.55:0/4284519477 conn(0x7f66680b16a0 msgr2=0x7f66680cead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:01.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:01.460+0000 7f666c9e4640 1 -- 172.21.15.55:0/4284519477 shutdown_connections 2024-03-19T01:58:01.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:01.460+0000 7f666c9e4640 1 -- 172.21.15.55:0/4284519477 wait complete. 2024-03-19T01:58:02.081 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:02.081 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:03.082 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:03.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.208+0000 7fc51a66c640 1 Processor -- start 2024-03-19T01:58:03.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.208+0000 7fc51a66c640 1 -- start start 2024-03-19T01:58:03.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.208+0000 7fc51a66c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc514152120 0x7fc5140c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:03.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.208+0000 7fc51a66c640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fc514070780 0x7fc5141586e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:03.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.208+0000 7fc51a66c640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc51406fff0 con 0x7fc5140c9610 2024-03-19T01:58:03.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.208+0000 7fc51a66c640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fc514152500 con 0x7fc514070780 2024-03-19T01:58:03.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.208+0000 7fc51a66c640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc514070590 con 0x7fc514152120 2024-03-19T01:58:03.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.212+0000 7fc513fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc514152120 0x7fc5140c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:03.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.212+0000 7fc518c09640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fc514070780 0x7fc5141586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:03.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.212+0000 7fc513fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc514152120 0x7fc5140c90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:59692/0 (socket says 172.21.15.55:59692) 2024-03-19T01:58:03.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.212+0000 7fc513fff640 1 -- 172.21.15.55:0/3304597242 learned_addr learned my addr 172.21.15.55:0/3304597242 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:03.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.216+0000 7fc513fff640 1 -- 172.21.15.55:0/3304597242 >> v2:172.21.15.84:3300/0 conn(0x7fc514070780 msgr2=0x7fc5141586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:03.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.216+0000 7fc513fff640 1 --2- 172.21.15.55:0/3304597242 >> v2:172.21.15.84:3300/0 conn(0x7fc514070780 0x7fc5141586e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:03.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.216+0000 7fc513fff640 1 -- 172.21.15.55:0/3304597242 >> v1:172.21.15.84:6789/0 conn(0x7fc5140c9610 legacy=0x7fc514157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:03.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.216+0000 7fc513fff640 1 -- 172.21.15.55:0/3304597242 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5140c39c0 con 0x7fc514152120 2024-03-19T01:58:03.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.220+0000 7fc513fff640 1 --2- 172.21.15.55:0/3304597242 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc514152120 0x7fc5140c90d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc504009870 tx=0x7fc50401c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=283a8c8216d3d024 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:03.232 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.232+0000 7fc512ffd640 1 -- 172.21.15.55:0/3304597242 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5040b2070 con 0x7fc514152120 2024-03-19T01:58:03.232 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.232+0000 7fc512ffd640 1 -- 172.21.15.55:0/3304597242 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5040a9070 con 0x7fc514152120 2024-03-19T01:58:03.232 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.232+0000 7fc512ffd640 1 -- 172.21.15.55:0/3304597242 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5040ad880 con 0x7fc514152120 2024-03-19T01:58:03.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.232+0000 7fc51a66c640 1 -- 172.21.15.55:0/3304597242 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc514152120 msgr2=0x7fc5140c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:03.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.232+0000 7fc51a66c640 1 --2- 172.21.15.55:0/3304597242 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc514152120 0x7fc5140c90d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc504009870 tx=0x7fc50401c560 comp rx=0 tx=0).stop 2024-03-19T01:58:03.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.232+0000 7fc51a66c640 1 -- 172.21.15.55:0/3304597242 shutdown_connections 2024-03-19T01:58:03.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.232+0000 7fc51a66c640 1 --2- 172.21.15.55:0/3304597242 >> v2:172.21.15.84:3300/0 conn(0x7fc514070780 0x7fc5141586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:03.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.232+0000 7fc51a66c640 1 --2- 172.21.15.55:0/3304597242 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc514152120 0x7fc5140c90d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:03.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.232+0000 7fc51a66c640 1 -- 172.21.15.55:0/3304597242 >> 172.21.15.55:0/3304597242 conn(0x7fc514067600 msgr2=0x7fc514071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:03.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.232+0000 7fc51a66c640 1 -- 172.21.15.55:0/3304597242 shutdown_connections 2024-03-19T01:58:03.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.232+0000 7fc51a66c640 1 -- 172.21.15.55:0/3304597242 wait complete. 2024-03-19T01:58:03.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.236+0000 7fc51a66c640 1 Processor -- start 2024-03-19T01:58:03.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.236+0000 7fc51a66c640 1 -- start start 2024-03-19T01:58:03.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.236+0000 7fc51a66c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc514070780 0x7fc5140cbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:03.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.236+0000 7fc51a66c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5140c9610 0x7fc5140c5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:03.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.236+0000 7fc51a66c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc514152120 0x7fc5140c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:03.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.236+0000 7fc51a66c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc5140c86d0 con 0x7fc514152120 2024-03-19T01:58:03.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.236+0000 7fc51a66c640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc5140c31c0 con 0x7fc5140c9610 2024-03-19T01:58:03.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.236+0000 7fc51a66c640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc514165170 con 0x7fc514070780 2024-03-19T01:58:03.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.236+0000 7fc518c09640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc514152120 0x7fc5140c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:03.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.236+0000 7fc5137fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5140c9610 0x7fc5140c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:03.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.240+0000 7fc518c09640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc514152120 0x7fc5140c5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34868/0 (socket says 172.21.15.55:34868) 2024-03-19T01:58:03.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.240+0000 7fc518c09640 1 -- 172.21.15.55:0/3171813891 learned_addr learned my addr 172.21.15.55:0/3171813891 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:03.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.244+0000 7fc518c09640 1 -- 172.21.15.55:0/3171813891 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc514070780 msgr2=0x7fc5140cbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:03.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.244+0000 7fc518c09640 1 --2- 172.21.15.55:0/3171813891 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc514070780 0x7fc5140cbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:03.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.244+0000 7fc518c09640 1 -- 172.21.15.55:0/3171813891 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5140c9610 msgr2=0x7fc5140c5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:03.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.244+0000 7fc518c09640 1 --2- 172.21.15.55:0/3171813891 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5140c9610 0x7fc5140c5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:03.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.244+0000 7fc518c09640 1 -- 172.21.15.55:0/3171813891 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5141f1500 con 0x7fc514152120 2024-03-19T01:58:03.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.244+0000 7fc518c09640 1 --2- 172.21.15.55:0/3171813891 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc514152120 0x7fc5140c5ba0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc50c009dd0 tx=0x7fc50c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:03.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.248+0000 7fc510ff9640 1 -- 172.21.15.55:0/3171813891 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50c018020 con 0x7fc514152120 2024-03-19T01:58:03.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.248+0000 7fc510ff9640 1 -- 172.21.15.55:0/3171813891 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc50c014480 con 0x7fc514152120 2024-03-19T01:58:03.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.248+0000 7fc51a66c640 1 -- 172.21.15.55:0/3171813891 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5141f17e0 con 0x7fc514152120 2024-03-19T01:58:03.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.248+0000 7fc510ff9640 1 -- 172.21.15.55:0/3171813891 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50c014870 con 0x7fc514152120 2024-03-19T01:58:03.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.248+0000 7fc51a66c640 1 -- 172.21.15.55:0/3171813891 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5141f1da0 con 0x7fc514152120 2024-03-19T01:58:03.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.256+0000 7fc51a66c640 1 -- 172.21.15.55:0/3171813891 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc514068a40 con 0x7fc514152120 2024-03-19T01:58:03.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.272+0000 7fc510ff9640 1 -- 172.21.15.55:0/3171813891 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc50c01b020 con 0x7fc514152120 2024-03-19T01:58:03.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.272+0000 7fc510ff9640 1 --2- 172.21.15.55:0/3171813891 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc5000676b0 0x7fc500069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:03.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.272+0000 7fc513fff640 1 --2- 172.21.15.55:0/3171813891 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc5000676b0 0x7fc500069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:03.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.272+0000 7fc513fff640 1 --2- 172.21.15.55:0/3171813891 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc5000676b0 0x7fc500069b70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc5040af000 tx=0x7fc504004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:03.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.532+0000 7fc510ff9640 1 -- 172.21.15.55:0/3171813891 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fc50c0554f0 con 0x7fc514152120 2024-03-19T01:58:03.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.532+0000 7fc510ff9640 1 -- 172.21.15.55:0/3171813891 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc50c022030 con 0x7fc514152120 2024-03-19T01:58:03.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.748+0000 7fc51a66c640 1 -- 172.21.15.55:0/3171813891 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5140c84b0 con 0x7fc514152120 2024-03-19T01:58:03.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.904+0000 7fc510ff9640 1 -- 172.21.15.55:0/3171813891 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fc50c010040 con 0x7fc514152120 2024-03-19T01:58:03.906 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:03.906 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:03.906 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:03.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.908+0000 7fc51a66c640 1 -- 172.21.15.55:0/3171813891 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc5000676b0 msgr2=0x7fc500069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:03.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.908+0000 7fc51a66c640 1 --2- 172.21.15.55:0/3171813891 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc5000676b0 0x7fc500069b70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc5040af000 tx=0x7fc504004740 comp rx=0 tx=0).stop 2024-03-19T01:58:03.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.908+0000 7fc51a66c640 1 -- 172.21.15.55:0/3171813891 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc514152120 msgr2=0x7fc5140c5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:03.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.908+0000 7fc51a66c640 1 --2- 172.21.15.55:0/3171813891 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc514152120 0x7fc5140c5ba0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc50c009dd0 tx=0x7fc50c00c490 comp rx=0 tx=0).stop 2024-03-19T01:58:03.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.908+0000 7fc51a66c640 1 -- 172.21.15.55:0/3171813891 shutdown_connections 2024-03-19T01:58:03.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.908+0000 7fc51a66c640 1 --2- 172.21.15.55:0/3171813891 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc5000676b0 0x7fc500069b70 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:03.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.908+0000 7fc51a66c640 1 --2- 172.21.15.55:0/3171813891 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc514152120 0x7fc5140c5ba0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:03.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.908+0000 7fc51a66c640 1 --2- 172.21.15.55:0/3171813891 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5140c9610 0x7fc5140c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:03.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.908+0000 7fc51a66c640 1 --2- 172.21.15.55:0/3171813891 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc514070780 0x7fc5140cbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:03.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.908+0000 7fc51a66c640 1 -- 172.21.15.55:0/3171813891 >> 172.21.15.55:0/3171813891 conn(0x7fc514067600 msgr2=0x7fc5140c8df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:03.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.908+0000 7fc51a66c640 1 -- 172.21.15.55:0/3171813891 shutdown_connections 2024-03-19T01:58:03.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:03.908+0000 7fc51a66c640 1 -- 172.21.15.55:0/3171813891 wait complete. 2024-03-19T01:58:04.529 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:04.529 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:05.530 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:05.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.660+0000 7f147e023640 1 Processor -- start 2024-03-19T01:58:05.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.660+0000 7f147e023640 1 -- start start 2024-03-19T01:58:05.661 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.660+0000 7f147e023640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1478152120 0x7f14780c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:05.661 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.660+0000 7f147e023640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f14780c9610 0x7f1478157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:05.661 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.660+0000 7f147e023640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f147806fff0 con 0x7f1478070780 2024-03-19T01:58:05.661 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.660+0000 7f147e023640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f1478152500 con 0x7f14780c9610 2024-03-19T01:58:05.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.660+0000 7f147e023640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1478070590 con 0x7f1478152120 2024-03-19T01:58:05.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.660+0000 7f14777fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1478152120 0x7f14780c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:05.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.660+0000 7f1476ffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f14780c9610 0x7f1478157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:05.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.660+0000 7f1477fff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f1478070780 0x7f1478158510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:56660/0 (socket says 172.21.15.55:56660) 2024-03-19T01:58:05.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.660+0000 7f1477fff640 1 -- 172.21.15.55:0/1668466387 learned_addr learned my addr 172.21.15.55:0/1668466387 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:58:05.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.664+0000 7f1476ffd640 1 -- 172.21.15.55:0/1668466387 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1478152120 msgr2=0x7f14780c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:05.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.664+0000 7f1476ffd640 1 --2- 172.21.15.55:0/1668466387 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1478152120 0x7f14780c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:05.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.664+0000 7f1476ffd640 1 -- 172.21.15.55:0/1668466387 >> v1:172.21.15.84:6789/0 conn(0x7f1478070780 legacy=0x7f1478158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:05.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.664+0000 7f1476ffd640 1 -- 172.21.15.55:0/1668466387 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14780c39c0 con 0x7f14780c9610 2024-03-19T01:58:05.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.668+0000 7f1476ffd640 1 --2- 172.21.15.55:0/1668466387 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14780c9610 0x7f1478157fd0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f146c0097c0 tx=0x7f146c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2298f0f243bc175 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:05.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.672+0000 7f14767fc640 1 -- 172.21.15.55:0/1668466387 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146c0b0070 con 0x7f14780c9610 2024-03-19T01:58:05.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.672+0000 7f14767fc640 1 -- 172.21.15.55:0/1668466387 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f146c0ab600 con 0x7f14780c9610 2024-03-19T01:58:05.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.672+0000 7f14767fc640 1 -- 172.21.15.55:0/1668466387 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146c0ab9d0 con 0x7f14780c9610 2024-03-19T01:58:05.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.676+0000 7f147e023640 1 -- 172.21.15.55:0/1668466387 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14780c9610 msgr2=0x7f1478157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:05.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.676+0000 7f147e023640 1 --2- 172.21.15.55:0/1668466387 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14780c9610 0x7f1478157fd0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f146c0097c0 tx=0x7f146c01c4a0 comp rx=0 tx=0).stop 2024-03-19T01:58:05.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.676+0000 7f147e023640 1 -- 172.21.15.55:0/1668466387 shutdown_connections 2024-03-19T01:58:05.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.676+0000 7f147e023640 1 --2- 172.21.15.55:0/1668466387 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f14780c9610 0x7f1478157fd0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:05.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.676+0000 7f147e023640 1 --2- 172.21.15.55:0/1668466387 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1478152120 0x7f14780c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:05.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.676+0000 7f147e023640 1 -- 172.21.15.55:0/1668466387 >> 172.21.15.55:0/1668466387 conn(0x7f1478067600 msgr2=0x7f1478071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:05.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.676+0000 7f147e023640 1 -- 172.21.15.55:0/1668466387 shutdown_connections 2024-03-19T01:58:05.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.676+0000 7f147e023640 1 -- 172.21.15.55:0/1668466387 wait complete. 2024-03-19T01:58:05.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.676+0000 7f147e023640 1 Processor -- start 2024-03-19T01:58:05.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.676+0000 7f147e023640 1 -- start start 2024-03-19T01:58:05.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.676+0000 7f147e023640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1478070780 0x7f14781402e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:05.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.676+0000 7f147e023640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f14780c9610 0x7f1478155a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:05.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.676+0000 7f147e023640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1478152120 0x7f1478141190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:05.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.676+0000 7f147e023640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f14780c86d0 con 0x7f1478152120 2024-03-19T01:58:05.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.676+0000 7f147e023640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f14780c31c0 con 0x7f1478070780 2024-03-19T01:58:05.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.676+0000 7f147e023640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1478165170 con 0x7f14780c9610 2024-03-19T01:58:05.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.680+0000 7f1477fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1478152120 0x7f1478141190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:05.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.680+0000 7f1476ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f14780c9610 0x7f1478155a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:05.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.680+0000 7f14777fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1478070780 0x7f14781402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:05.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.680+0000 7f1477fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1478152120 0x7f1478141190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34878/0 (socket says 172.21.15.55:34878) 2024-03-19T01:58:05.680 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.680+0000 7f1477fff640 1 -- 172.21.15.55:0/2882445949 learned_addr learned my addr 172.21.15.55:0/2882445949 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:05.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.684+0000 7f1477fff640 1 -- 172.21.15.55:0/2882445949 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f14780c9610 msgr2=0x7f1478155a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:05.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.684+0000 7f1477fff640 1 --2- 172.21.15.55:0/2882445949 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f14780c9610 0x7f1478155a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:05.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.684+0000 7f1477fff640 1 -- 172.21.15.55:0/2882445949 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1478070780 msgr2=0x7f14781402e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:05.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.684+0000 7f1477fff640 1 --2- 172.21.15.55:0/2882445949 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1478070780 0x7f14781402e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:05.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.684+0000 7f1477fff640 1 -- 172.21.15.55:0/2882445949 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14781f4530 con 0x7f1478152120 2024-03-19T01:58:05.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.684+0000 7f1477fff640 1 --2- 172.21.15.55:0/2882445949 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1478152120 0x7f1478141190 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f147000cbc0 tx=0x7f147000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:05.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.688+0000 7f145ffff640 1 -- 172.21.15.55:0/2882445949 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1470017070 con 0x7f1478152120 2024-03-19T01:58:05.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.688+0000 7f145ffff640 1 -- 172.21.15.55:0/2882445949 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f147000ace0 con 0x7f1478152120 2024-03-19T01:58:05.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.688+0000 7f145ffff640 1 -- 172.21.15.55:0/2882445949 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1470012600 con 0x7f1478152120 2024-03-19T01:58:05.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.688+0000 7f147e023640 1 -- 172.21.15.55:0/2882445949 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1478156380 con 0x7f1478152120 2024-03-19T01:58:05.691 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.688+0000 7f147e023640 1 -- 172.21.15.55:0/2882445949 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14781f3a50 con 0x7f1478152120 2024-03-19T01:58:05.691 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.692+0000 7f147e023640 1 -- 172.21.15.55:0/2882445949 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1478068a40 con 0x7f1478152120 2024-03-19T01:58:05.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.720+0000 7f145ffff640 1 -- 172.21.15.55:0/2882445949 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f147001a070 con 0x7f1478152120 2024-03-19T01:58:05.722 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.720+0000 7f145ffff640 1 --2- 172.21.15.55:0/2882445949 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1464067690 0x7f1464069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:05.722 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.720+0000 7f14777fe640 1 --2- 172.21.15.55:0/2882445949 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1464067690 0x7f1464069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:05.722 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.724+0000 7f14777fe640 1 --2- 172.21.15.55:0/2882445949 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1464067690 0x7f1464069b50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1468009ac0 tx=0x7f1468009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:05.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.728+0000 7f145ffff640 1 -- 172.21.15.55:0/2882445949 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f14700540a0 con 0x7f1478152120 2024-03-19T01:58:05.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:05.996+0000 7f145ffff640 1 -- 172.21.15.55:0/2882445949 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1470056000 con 0x7f1478152120 2024-03-19T01:58:06.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:06.216+0000 7f147e023640 1 -- 172.21.15.55:0/2882445949 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14780b0af0 con 0x7f1478152120 2024-03-19T01:58:06.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:06.240+0000 7f145ffff640 1 -- 172.21.15.55:0/2882445949 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f147005a0a0 con 0x7f1478152120 2024-03-19T01:58:06.242 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:06.242 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:06.243 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:06.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:06.244+0000 7f147e023640 1 -- 172.21.15.55:0/2882445949 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1464067690 msgr2=0x7f1464069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:06.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:06.244+0000 7f147e023640 1 --2- 172.21.15.55:0/2882445949 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1464067690 0x7f1464069b50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1468009ac0 tx=0x7f1468009190 comp rx=0 tx=0).stop 2024-03-19T01:58:06.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:06.244+0000 7f147e023640 1 -- 172.21.15.55:0/2882445949 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1478152120 msgr2=0x7f1478141190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:06.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:06.244+0000 7f147e023640 1 --2- 172.21.15.55:0/2882445949 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1478152120 0x7f1478141190 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f147000cbc0 tx=0x7f147000a430 comp rx=0 tx=0).stop 2024-03-19T01:58:06.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:06.244+0000 7f147e023640 1 -- 172.21.15.55:0/2882445949 shutdown_connections 2024-03-19T01:58:06.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:06.244+0000 7f147e023640 1 --2- 172.21.15.55:0/2882445949 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1464067690 0x7f1464069b50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:06.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:06.244+0000 7f147e023640 1 --2- 172.21.15.55:0/2882445949 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1478152120 0x7f1478141190 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:06.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:06.244+0000 7f147e023640 1 --2- 172.21.15.55:0/2882445949 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f14780c9610 0x7f1478155a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:06.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:06.244+0000 7f147e023640 1 --2- 172.21.15.55:0/2882445949 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1478070780 0x7f14781402e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:06.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:06.244+0000 7f147e023640 1 -- 172.21.15.55:0/2882445949 >> 172.21.15.55:0/2882445949 conn(0x7f1478067600 msgr2=0x7f14780b0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:06.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:06.244+0000 7f147e023640 1 -- 172.21.15.55:0/2882445949 shutdown_connections 2024-03-19T01:58:06.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:06.244+0000 7f147e023640 1 -- 172.21.15.55:0/2882445949 wait complete. 2024-03-19T01:58:06.870 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:06.870 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:07.870 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:07.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:07.992+0000 7f62b9304640 1 Processor -- start 2024-03-19T01:58:07.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:07.992+0000 7f62b9304640 1 -- start start 2024-03-19T01:58:07.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:07.992+0000 7f62b9304640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f62b40c96b0 0x7f62b4070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:07.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:07.992+0000 7f62b9304640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f62b41540c0 0x7f62b4154570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:07.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:07.992+0000 7f62b9304640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62b415e3a0 con 0x7f62b40c9a90 2024-03-19T01:58:07.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:07.992+0000 7f62b9304640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f62b40c7ad0 con 0x7f62b40c96b0 2024-03-19T01:58:07.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:07.992+0000 7f62b9304640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f62b415e680 con 0x7f62b41540c0 2024-03-19T01:58:07.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:07.992+0000 7f62b27fc640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f62b40c9a90 0x7f62b40c73c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:59874/0 (socket says 172.21.15.55:59874) 2024-03-19T01:58:07.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:07.992+0000 7f62b27fc640 1 -- 172.21.15.55:0/3546723015 learned_addr learned my addr 172.21.15.55:0/3546723015 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:58:07.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:07.992+0000 7f62b2ffd640 1 --2- 172.21.15.55:0/3546723015 >> v2:172.21.15.84:3300/0 conn(0x7f62b40c96b0 0x7f62b4070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:07.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:07.996+0000 7f62b37fe640 1 --2- 172.21.15.55:0/3546723015 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f62b41540c0 0x7f62b4154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:07.997 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:07.996+0000 7f62b1ffb640 1 -- 172.21.15.55:0/3546723015 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1835428685 0 0) 0x7f62b415e3a0 con 0x7f62b40c9a90 2024-03-19T01:58:07.997 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:07.996+0000 7f62b1ffb640 1 -- 172.21.15.55:0/3546723015 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f629c003530 con 0x7f62b40c9a90 2024-03-19T01:58:07.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:07.996+0000 7f62b2ffd640 1 -- 172.21.15.55:0/3546723015 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f62b41540c0 msgr2=0x7f62b4154570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:07.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:07.996+0000 7f62b2ffd640 1 --2- 172.21.15.55:0/3546723015 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f62b41540c0 0x7f62b4154570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:07.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:07.996+0000 7f62b2ffd640 1 -- 172.21.15.55:0/3546723015 >> v1:172.21.15.55:6789/0 conn(0x7f62b40c9a90 legacy=0x7f62b40c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:07.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:07.996+0000 7f62b2ffd640 1 -- 172.21.15.55:0/3546723015 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62b415a330 con 0x7f62b40c96b0 2024-03-19T01:58:08.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.004+0000 7f62b2ffd640 1 --2- 172.21.15.55:0/3546723015 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62b40c96b0 0x7f62b4070f20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f62a40097c0 tx=0x7f62a401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=878b7b8a8e8bec1d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:08.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.008+0000 7f62b1ffb640 1 -- 172.21.15.55:0/3546723015 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a40b0070 con 0x7f62b40c96b0 2024-03-19T01:58:08.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.008+0000 7f62b1ffb640 1 -- 172.21.15.55:0/3546723015 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62a40ab600 con 0x7f62b40c96b0 2024-03-19T01:58:08.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.008+0000 7f62b1ffb640 1 -- 172.21.15.55:0/3546723015 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a40ab9d0 con 0x7f62b40c96b0 2024-03-19T01:58:08.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.008+0000 7f62b9304640 1 -- 172.21.15.55:0/3546723015 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62b40c96b0 msgr2=0x7f62b4070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:08.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.008+0000 7f62b9304640 1 --2- 172.21.15.55:0/3546723015 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62b40c96b0 0x7f62b4070f20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f62a40097c0 tx=0x7f62a401c410 comp rx=0 tx=0).stop 2024-03-19T01:58:08.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.008+0000 7f62b9304640 1 -- 172.21.15.55:0/3546723015 shutdown_connections 2024-03-19T01:58:08.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.008+0000 7f62b9304640 1 --2- 172.21.15.55:0/3546723015 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f62b41540c0 0x7f62b4154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:08.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.008+0000 7f62b9304640 1 --2- 172.21.15.55:0/3546723015 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62b40c96b0 0x7f62b4070f20 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:08.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.008+0000 7f62b9304640 1 -- 172.21.15.55:0/3546723015 >> 172.21.15.55:0/3546723015 conn(0x7f62b40cc9c0 msgr2=0x7f62b40ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:08.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.008+0000 7f62b9304640 1 -- 172.21.15.55:0/3546723015 shutdown_connections 2024-03-19T01:58:08.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.008+0000 7f62b9304640 1 -- 172.21.15.55:0/3546723015 wait complete. 2024-03-19T01:58:08.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.012+0000 7f62b9304640 1 Processor -- start 2024-03-19T01:58:08.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.012+0000 7f62b9304640 1 -- start start 2024-03-19T01:58:08.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.012+0000 7f62b9304640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f62b40c96b0 0x7f62b40cc010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:08.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.012+0000 7f62b9304640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62b40c9a90 0x7f62b4152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:08.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.012+0000 7f62b9304640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62b41540c0 0x7f62b41530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:08.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.012+0000 7f62b9304640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f62b4166b90 con 0x7f62b41540c0 2024-03-19T01:58:08.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.012+0000 7f62b9304640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f62b415b0f0 con 0x7f62b40c9a90 2024-03-19T01:58:08.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.012+0000 7f62b9304640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f62b4166d10 con 0x7f62b40c96b0 2024-03-19T01:58:08.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.012+0000 7f62b27fc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62b40c9a90 0x7f62b4152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:08.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.012+0000 7f62b2ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f62b40c96b0 0x7f62b40cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:08.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.012+0000 7f62b27fc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62b40c9a90 0x7f62b4152b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:49764/0 (socket says 172.21.15.55:49764) 2024-03-19T01:58:08.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.012+0000 7f62b27fc640 1 -- 172.21.15.55:0/2092516711 learned_addr learned my addr 172.21.15.55:0/2092516711 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:08.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.016+0000 7f62b27fc640 1 -- 172.21.15.55:0/2092516711 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f62b40c96b0 msgr2=0x7f62b40cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:08.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.016+0000 7f62b27fc640 1 --2- 172.21.15.55:0/2092516711 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f62b40c96b0 0x7f62b40cc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:08.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.016+0000 7f62b27fc640 1 -- 172.21.15.55:0/2092516711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62b41540c0 msgr2=0x7f62b41530d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:08.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.016+0000 7f62b27fc640 1 --2- 172.21.15.55:0/2092516711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62b41540c0 0x7f62b41530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:08.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.016+0000 7f62b27fc640 1 -- 172.21.15.55:0/2092516711 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62b41f3420 con 0x7f62b40c9a90 2024-03-19T01:58:08.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.020+0000 7f62b27fc640 1 --2- 172.21.15.55:0/2092516711 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62b40c9a90 0x7f62b4152b90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f62a800c390 tx=0x7f62a800c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:08.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.028+0000 7f629b7fe640 1 -- 172.21.15.55:0/2092516711 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a8017070 con 0x7f62b40c9a90 2024-03-19T01:58:08.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.028+0000 7f629b7fe640 1 -- 172.21.15.55:0/2092516711 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62a8005d10 con 0x7f62b40c9a90 2024-03-19T01:58:08.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.028+0000 7f629b7fe640 1 -- 172.21.15.55:0/2092516711 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a8007310 con 0x7f62b40c9a90 2024-03-19T01:58:08.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.028+0000 7f62b9304640 1 -- 172.21.15.55:0/2092516711 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62b41f3700 con 0x7f62b40c9a90 2024-03-19T01:58:08.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.028+0000 7f62b9304640 1 -- 172.21.15.55:0/2092516711 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62b41f3cc0 con 0x7f62b40c9a90 2024-03-19T01:58:08.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.028+0000 7f62b9304640 1 -- 172.21.15.55:0/2092516711 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62b4152480 con 0x7f62b40c9a90 2024-03-19T01:58:08.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.064+0000 7f629b7fe640 1 -- 172.21.15.55:0/2092516711 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f62a801a070 con 0x7f62b40c9a90 2024-03-19T01:58:08.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.064+0000 7f629b7fe640 1 --2- 172.21.15.55:0/2092516711 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f62a0067660 0x7f62a0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:08.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.064+0000 7f62b2ffd640 1 --2- 172.21.15.55:0/2092516711 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f62a0067660 0x7f62a0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:08.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.068+0000 7f62b2ffd640 1 --2- 172.21.15.55:0/2092516711 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f62a0067660 0x7f62a0069b20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f62a40ad000 tx=0x7f62a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:08.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.324+0000 7f629b7fe640 1 -- 172.21.15.55:0/2092516711 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f62a80537b0 con 0x7f62b40c9a90 2024-03-19T01:58:08.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.368+0000 7f629b7fe640 1 -- 172.21.15.55:0/2092516711 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f62a8021020 con 0x7f62b40c9a90 2024-03-19T01:58:08.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.580+0000 7f62b9304640 1 -- 172.21.15.55:0/2092516711 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62b41f3e90 con 0x7f62b40c9a90 2024-03-19T01:58:08.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.756+0000 7f629b7fe640 1 -- 172.21.15.55:0/2092516711 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f62a8055710 con 0x7f62b40c9a90 2024-03-19T01:58:08.759 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:08.759 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:08.759 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:08.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.760+0000 7f62b9304640 1 -- 172.21.15.55:0/2092516711 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f62a0067660 msgr2=0x7f62a0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:08.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.760+0000 7f62b9304640 1 --2- 172.21.15.55:0/2092516711 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f62a0067660 0x7f62a0069b20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f62a40ad000 tx=0x7f62a4008040 comp rx=0 tx=0).stop 2024-03-19T01:58:08.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.760+0000 7f62b9304640 1 -- 172.21.15.55:0/2092516711 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62b40c9a90 msgr2=0x7f62b4152b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:08.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.760+0000 7f62b9304640 1 --2- 172.21.15.55:0/2092516711 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62b40c9a90 0x7f62b4152b90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f62a800c390 tx=0x7f62a800c850 comp rx=0 tx=0).stop 2024-03-19T01:58:08.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.760+0000 7f62b9304640 1 -- 172.21.15.55:0/2092516711 shutdown_connections 2024-03-19T01:58:08.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.760+0000 7f62b9304640 1 --2- 172.21.15.55:0/2092516711 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f62a0067660 0x7f62a0069b20 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:08.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.760+0000 7f62b9304640 1 --2- 172.21.15.55:0/2092516711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62b41540c0 0x7f62b41530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:08.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.760+0000 7f62b9304640 1 --2- 172.21.15.55:0/2092516711 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62b40c9a90 0x7f62b4152b90 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:08.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.760+0000 7f62b9304640 1 --2- 172.21.15.55:0/2092516711 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f62b40c96b0 0x7f62b40cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:08.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.760+0000 7f62b9304640 1 -- 172.21.15.55:0/2092516711 >> 172.21.15.55:0/2092516711 conn(0x7f62b40cc9c0 msgr2=0x7f62b4070a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:08.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.760+0000 7f62b9304640 1 -- 172.21.15.55:0/2092516711 shutdown_connections 2024-03-19T01:58:08.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:08.760+0000 7f62b9304640 1 -- 172.21.15.55:0/2092516711 wait complete. 2024-03-19T01:58:09.385 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:09.386 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:09.580 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:58:09.583 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:58:10.386 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:10.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.512+0000 7f737c0b0640 1 Processor -- start 2024-03-19T01:58:10.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.512+0000 7f737c0b0640 1 -- start start 2024-03-19T01:58:10.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.512+0000 7f737c0b0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f73741544f0 0x7f7374072c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:10.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.512+0000 7f737c0b0640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f73740c3640 0x7f73740c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:10.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.512+0000 7f737c0b0640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f737415a490 con 0x7f73741548d0 2024-03-19T01:58:10.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.512+0000 7f737c0b0640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f73740c9450 con 0x7f73740c3640 2024-03-19T01:58:10.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.512+0000 7f737c0b0640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f737415b360 con 0x7f73741544f0 2024-03-19T01:58:10.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.512+0000 7f737a64d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f73740c3640 0x7f73740c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:10.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.512+0000 7f737964b640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f73741548d0 0x7f73740ccb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:56562/0 (socket says 172.21.15.55:56562) 2024-03-19T01:58:10.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.512+0000 7f737964b640 1 -- 172.21.15.55:0/2379391524 learned_addr learned my addr 172.21.15.55:0/2379391524 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:58:10.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.512+0000 7f7379e4c640 1 --2- 172.21.15.55:0/2379391524 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f73741544f0 0x7f7374072c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:10.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.516+0000 7f7378e4a640 1 -- 172.21.15.55:0/2379391524 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3510972032 0 0) 0x7f737415a490 con 0x7f73741548d0 2024-03-19T01:58:10.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.516+0000 7f7378e4a640 1 -- 172.21.15.55:0/2379391524 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7360003530 con 0x7f73741548d0 2024-03-19T01:58:10.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.516+0000 7f7379e4c640 1 -- 172.21.15.55:0/2379391524 >> v2:172.21.15.84:3300/0 conn(0x7f73740c3640 msgr2=0x7f73740c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:10.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.516+0000 7f7379e4c640 1 --2- 172.21.15.55:0/2379391524 >> v2:172.21.15.84:3300/0 conn(0x7f73740c3640 0x7f73740c98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:10.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.516+0000 7f7379e4c640 1 -- 172.21.15.55:0/2379391524 >> v1:172.21.15.55:6789/0 conn(0x7f73741548d0 legacy=0x7f73740ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:10.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.516+0000 7f7379e4c640 1 -- 172.21.15.55:0/2379391524 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f737415a750 con 0x7f73741544f0 2024-03-19T01:58:10.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.524+0000 7f7379e4c640 1 --2- 172.21.15.55:0/2379391524 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f73741544f0 0x7f7374072c10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f73700097c0 tx=0x7f737001ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93b45de07e2f6772 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:10.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.528+0000 7f7378e4a640 1 -- 172.21.15.55:0/2379391524 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73700b5070 con 0x7f73741544f0 2024-03-19T01:58:10.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.532+0000 7f7378e4a640 1 -- 172.21.15.55:0/2379391524 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73700a8990 con 0x7f73741544f0 2024-03-19T01:58:10.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.532+0000 7f7378e4a640 1 -- 172.21.15.55:0/2379391524 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73700a8d40 con 0x7f73741544f0 2024-03-19T01:58:10.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.532+0000 7f737c0b0640 1 -- 172.21.15.55:0/2379391524 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f73741544f0 msgr2=0x7f7374072c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:10.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.532+0000 7f737c0b0640 1 --2- 172.21.15.55:0/2379391524 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f73741544f0 0x7f7374072c10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f73700097c0 tx=0x7f737001ec90 comp rx=0 tx=0).stop 2024-03-19T01:58:10.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.532+0000 7f737c0b0640 1 -- 172.21.15.55:0/2379391524 shutdown_connections 2024-03-19T01:58:10.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.532+0000 7f737c0b0640 1 --2- 172.21.15.55:0/2379391524 >> v2:172.21.15.84:3300/0 conn(0x7f73740c3640 0x7f73740c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:10.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.532+0000 7f737c0b0640 1 --2- 172.21.15.55:0/2379391524 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f73741544f0 0x7f7374072c10 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:10.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.532+0000 7f737c0b0640 1 -- 172.21.15.55:0/2379391524 >> 172.21.15.55:0/2379391524 conn(0x7f73740703c0 msgr2=0x7f73740707d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:10.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.532+0000 7f737c0b0640 1 -- 172.21.15.55:0/2379391524 shutdown_connections 2024-03-19T01:58:10.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.532+0000 7f737c0b0640 1 -- 172.21.15.55:0/2379391524 wait complete. 2024-03-19T01:58:10.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.532+0000 7f737c0b0640 1 Processor -- start 2024-03-19T01:58:10.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.536+0000 7f737c0b0640 1 -- start start 2024-03-19T01:58:10.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.536+0000 7f737c0b0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f73740c3640 0x7f73741f07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:10.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.536+0000 7f737c0b0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73741544f0 0x7f73741e0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:10.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.536+0000 7f737c0b0640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f73741548d0 0x7f73740c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:10.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.536+0000 7f737c0b0640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f737415aff0 con 0x7f73741544f0 2024-03-19T01:58:10.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.536+0000 7f737c0b0640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f737415ad10 con 0x7f73741548d0 2024-03-19T01:58:10.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.536+0000 7f737c0b0640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f7374073150 con 0x7f73740c3640 2024-03-19T01:58:10.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.536+0000 7f737964b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73741544f0 0x7f73741e0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:10.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.536+0000 7f7379e4c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f73740c3640 0x7f73741f07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:10.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.536+0000 7f737964b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73741544f0 0x7f73741e0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50418/0 (socket says 172.21.15.55:50418) 2024-03-19T01:58:10.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.536+0000 7f737964b640 1 -- 172.21.15.55:0/3231253822 learned_addr learned my addr 172.21.15.55:0/3231253822 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:10.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.536+0000 7f737a64d640 1 --2- 172.21.15.55:0/3231253822 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f73741548d0 0x7f73740c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:10.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.540+0000 7f737964b640 1 -- 172.21.15.55:0/3231253822 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f73740c3640 msgr2=0x7f73741f07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:10.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.540+0000 7f737964b640 1 --2- 172.21.15.55:0/3231253822 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f73740c3640 0x7f73741f07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:10.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.540+0000 7f737964b640 1 -- 172.21.15.55:0/3231253822 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f73741548d0 msgr2=0x7f73740c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:10.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.540+0000 7f737964b640 1 --2- 172.21.15.55:0/3231253822 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f73741548d0 0x7f73740c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:10.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.540+0000 7f737964b640 1 -- 172.21.15.55:0/3231253822 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73740ca510 con 0x7f73741544f0 2024-03-19T01:58:10.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.544+0000 7f737964b640 1 --2- 172.21.15.55:0/3231253822 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73741544f0 0x7f73741e0400 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f736400c450 tx=0x7f736400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:10.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.548+0000 7f736a7fc640 1 -- 172.21.15.55:0/3231253822 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7364017070 con 0x7f73741544f0 2024-03-19T01:58:10.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.548+0000 7f736a7fc640 1 -- 172.21.15.55:0/3231253822 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7364005cf0 con 0x7f73741544f0 2024-03-19T01:58:10.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.548+0000 7f736a7fc640 1 -- 172.21.15.55:0/3231253822 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7364007290 con 0x7f73741544f0 2024-03-19T01:58:10.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.548+0000 7f737c0b0640 1 -- 172.21.15.55:0/3231253822 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73740c61f0 con 0x7f73741544f0 2024-03-19T01:58:10.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.552+0000 7f737c0b0640 1 -- 172.21.15.55:0/3231253822 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73741e0bd0 con 0x7f73741544f0 2024-03-19T01:58:10.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.552+0000 7f737c0b0640 1 -- 172.21.15.55:0/3231253822 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7374068a40 con 0x7f73741544f0 2024-03-19T01:58:10.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.584+0000 7f736a7fc640 1 -- 172.21.15.55:0/3231253822 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f736401a070 con 0x7f73741544f0 2024-03-19T01:58:10.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.584+0000 7f736a7fc640 1 --2- 172.21.15.55:0/3231253822 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f73580676b0 0x7f7358069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:10.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.584+0000 7f7379e4c640 1 --2- 172.21.15.55:0/3231253822 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f73580676b0 0x7f7358069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:10.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.584+0000 7f7379e4c640 1 --2- 172.21.15.55:0/3231253822 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f73580676b0 0x7f7358069b70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f73700b2000 tx=0x7f73700aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:10.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.892+0000 7f736a7fc640 1 -- 172.21.15.55:0/3231253822 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f7364053570 con 0x7f73741544f0 2024-03-19T01:58:10.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:10.892+0000 7f736a7fc640 1 -- 172.21.15.55:0/3231253822 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f7364021080 con 0x7f73741544f0 2024-03-19T01:58:11.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:11.104+0000 7f737c0b0640 1 -- 172.21.15.55:0/3231253822 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73740729f0 con 0x7f73741544f0 2024-03-19T01:58:11.230 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:11.232+0000 7f736a7fc640 1 -- 172.21.15.55:0/3231253822 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f73640554d0 con 0x7f73741544f0 2024-03-19T01:58:11.231 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:11.231 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:11.231 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:11.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:11.236+0000 7f737c0b0640 1 -- 172.21.15.55:0/3231253822 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f73580676b0 msgr2=0x7f7358069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:11.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:11.236+0000 7f737c0b0640 1 --2- 172.21.15.55:0/3231253822 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f73580676b0 0x7f7358069b70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f73700b2000 tx=0x7f73700aa070 comp rx=0 tx=0).stop 2024-03-19T01:58:11.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:11.236+0000 7f737c0b0640 1 -- 172.21.15.55:0/3231253822 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73741544f0 msgr2=0x7f73741e0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:11.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:11.236+0000 7f737c0b0640 1 --2- 172.21.15.55:0/3231253822 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73741544f0 0x7f73741e0400 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f736400c450 tx=0x7f736400c910 comp rx=0 tx=0).stop 2024-03-19T01:58:11.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:11.236+0000 7f737c0b0640 1 -- 172.21.15.55:0/3231253822 shutdown_connections 2024-03-19T01:58:11.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:11.236+0000 7f737c0b0640 1 --2- 172.21.15.55:0/3231253822 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f73580676b0 0x7f7358069b70 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:11.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:11.236+0000 7f737c0b0640 1 --2- 172.21.15.55:0/3231253822 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f73741548d0 0x7f73740c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:11.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:11.236+0000 7f737c0b0640 1 --2- 172.21.15.55:0/3231253822 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73741544f0 0x7f73741e0400 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:11.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:11.236+0000 7f737c0b0640 1 --2- 172.21.15.55:0/3231253822 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f73740c3640 0x7f73741f07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:11.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:11.236+0000 7f737c0b0640 1 -- 172.21.15.55:0/3231253822 >> 172.21.15.55:0/3231253822 conn(0x7f73740703c0 msgr2=0x7f73740707a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:11.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:11.236+0000 7f737c0b0640 1 -- 172.21.15.55:0/3231253822 shutdown_connections 2024-03-19T01:58:11.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:11.236+0000 7f737c0b0640 1 -- 172.21.15.55:0/3231253822 wait complete. 2024-03-19T01:58:11.854 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:11.854 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:12.855 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:12.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:12.988+0000 7fc3e181e640 1 Processor -- start 2024-03-19T01:58:12.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:12.988+0000 7fc3e181e640 1 -- start start 2024-03-19T01:58:12.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:12.992+0000 7fc3e181e640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fc3dc1438c0 0x7fc3dc0c9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:12.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:12.992+0000 7fc3e181e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3dc070bd0 con 0x7fc3dc15a540 2024-03-19T01:58:12.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:12.992+0000 7fc3e181e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3dc070e90 con 0x7fc3dc15a920 2024-03-19T01:58:12.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:12.992+0000 7fc3e181e640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fc3dc06d440 con 0x7fc3dc1438c0 2024-03-19T01:58:12.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:12.992+0000 7fc3daffd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fc3dc15a540 0x7fc3dc06cd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:56566/0 (socket says 172.21.15.55:56566) 2024-03-19T01:58:12.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:12.992+0000 7fc3daffd640 1 -- 172.21.15.55:0/3679054902 learned_addr learned my addr 172.21.15.55:0/3679054902 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:58:12.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:12.992+0000 7fc3db7fe640 1 --2- 172.21.15.55:0/3679054902 >> v2:172.21.15.84:3300/0 conn(0x7fc3dc1438c0 0x7fc3dc0c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:12.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:12.992+0000 7fc3d9ffb640 1 -- 172.21.15.55:0/3679054902 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 680403585 0 0) 0x7fc3dc070bd0 con 0x7fc3dc15a540 2024-03-19T01:58:12.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:12.992+0000 7fc3d9ffb640 1 -- 172.21.15.55:0/3679054902 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3c4003530 con 0x7fc3dc15a540 2024-03-19T01:58:12.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:12.996+0000 7fc3db7fe640 1 -- 172.21.15.55:0/3679054902 >> v1:172.21.15.84:6789/0 conn(0x7fc3dc15a920 legacy=0x7fc3dc1431b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:12.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:12.996+0000 7fc3db7fe640 1 -- 172.21.15.55:0/3679054902 >> v1:172.21.15.55:6789/0 conn(0x7fc3dc15a540 legacy=0x7fc3dc06cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:12.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:12.996+0000 7fc3db7fe640 1 -- 172.21.15.55:0/3679054902 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3dc06f830 con 0x7fc3dc1438c0 2024-03-19T01:58:12.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.004+0000 7fc3db7fe640 1 --2- 172.21.15.55:0/3679054902 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3dc1438c0 0x7fc3dc0c9480 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4009d90 tx=0x7fc3d401f120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4966e5fcdf70c76 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:13.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.012+0000 7fc3d9ffb640 1 -- 172.21.15.55:0/3679054902 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d40b4070 con 0x7fc3dc1438c0 2024-03-19T01:58:13.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.012+0000 7fc3d9ffb640 1 -- 172.21.15.55:0/3679054902 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3d40af600 con 0x7fc3dc1438c0 2024-03-19T01:58:13.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.012+0000 7fc3d9ffb640 1 -- 172.21.15.55:0/3679054902 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d40af9d0 con 0x7fc3dc1438c0 2024-03-19T01:58:13.008 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.012+0000 7fc3e181e640 1 -- 172.21.15.55:0/3679054902 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3dc1438c0 msgr2=0x7fc3dc0c9480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:13.008 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.012+0000 7fc3e181e640 1 --2- 172.21.15.55:0/3679054902 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3dc1438c0 0x7fc3dc0c9480 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4009d90 tx=0x7fc3d401f120 comp rx=0 tx=0).stop 2024-03-19T01:58:13.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.012+0000 7fc3e181e640 1 -- 172.21.15.55:0/3679054902 shutdown_connections 2024-03-19T01:58:13.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.012+0000 7fc3e181e640 1 --2- 172.21.15.55:0/3679054902 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3dc1438c0 0x7fc3dc0c9480 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:13.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.012+0000 7fc3e181e640 1 -- 172.21.15.55:0/3679054902 >> 172.21.15.55:0/3679054902 conn(0x7fc3dc067600 msgr2=0x7fc3dc071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:13.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.012+0000 7fc3e181e640 1 -- 172.21.15.55:0/3679054902 shutdown_connections 2024-03-19T01:58:13.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.012+0000 7fc3e181e640 1 -- 172.21.15.55:0/3679054902 wait complete. 2024-03-19T01:58:13.010 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.012+0000 7fc3e181e640 1 Processor -- start 2024-03-19T01:58:13.010 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.012+0000 7fc3e181e640 1 -- start start 2024-03-19T01:58:13.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.016+0000 7fc3e181e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3dc1438c0 0x7fc3dc1f0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:13.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.016+0000 7fc3e181e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3dc15a540 0x7fc3dc1f3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:13.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.016+0000 7fc3e181e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3dc15a920 0x7fc3dc158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:13.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.016+0000 7fc3e181e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc3dc06fdf0 con 0x7fc3dc15a540 2024-03-19T01:58:13.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.016+0000 7fc3e181e640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc3dc06f5e0 con 0x7fc3dc1438c0 2024-03-19T01:58:13.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.016+0000 7fc3e181e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc3dc070860 con 0x7fc3dc15a920 2024-03-19T01:58:13.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.016+0000 7fc3da7fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3dc15a540 0x7fc3dc1f3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:13.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.016+0000 7fc3daffd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3dc1438c0 0x7fc3dc1f0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:13.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.016+0000 7fc3db7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3dc15a920 0x7fc3dc158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:13.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.016+0000 7fc3da7fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3dc15a540 0x7fc3dc1f3280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50422/0 (socket says 172.21.15.55:50422) 2024-03-19T01:58:13.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.016+0000 7fc3da7fc640 1 -- 172.21.15.55:0/2239553091 learned_addr learned my addr 172.21.15.55:0/2239553091 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:13.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.020+0000 7fc3da7fc640 1 -- 172.21.15.55:0/2239553091 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3dc15a920 msgr2=0x7fc3dc158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:13.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.020+0000 7fc3da7fc640 1 --2- 172.21.15.55:0/2239553091 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3dc15a920 0x7fc3dc158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:13.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.020+0000 7fc3da7fc640 1 -- 172.21.15.55:0/2239553091 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3dc1438c0 msgr2=0x7fc3dc1f0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:13.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.020+0000 7fc3da7fc640 1 --2- 172.21.15.55:0/2239553091 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3dc1438c0 0x7fc3dc1f0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:13.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.020+0000 7fc3da7fc640 1 -- 172.21.15.55:0/2239553091 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3dc156360 con 0x7fc3dc15a540 2024-03-19T01:58:13.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.024+0000 7fc3da7fc640 1 --2- 172.21.15.55:0/2239553091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3dc15a540 0x7fc3dc1f3280 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc3d000c480 tx=0x7fc3d000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:13.025 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.028+0000 7fc3c3fff640 1 -- 172.21.15.55:0/2239553091 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0012830 con 0x7fc3dc15a540 2024-03-19T01:58:13.025 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.028+0000 7fc3c3fff640 1 -- 172.21.15.55:0/2239553091 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3d0017070 con 0x7fc3dc15a540 2024-03-19T01:58:13.025 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.028+0000 7fc3c3fff640 1 -- 172.21.15.55:0/2239553091 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0012c30 con 0x7fc3dc15a540 2024-03-19T01:58:13.025 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.028+0000 7fc3e181e640 1 -- 172.21.15.55:0/2239553091 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3dc156600 con 0x7fc3dc15a540 2024-03-19T01:58:13.025 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.028+0000 7fc3e181e640 1 -- 172.21.15.55:0/2239553091 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3dc156ba0 con 0x7fc3dc15a540 2024-03-19T01:58:13.026 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.028+0000 7fc3e181e640 1 -- 172.21.15.55:0/2239553091 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3dc068a40 con 0x7fc3dc15a540 2024-03-19T01:58:13.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.060+0000 7fc3c3fff640 1 -- 172.21.15.55:0/2239553091 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc3d001c460 con 0x7fc3dc15a540 2024-03-19T01:58:13.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.064+0000 7fc3c3fff640 1 --2- 172.21.15.55:0/2239553091 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3c80676b0 0x7fc3c8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:13.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.064+0000 7fc3daffd640 1 --2- 172.21.15.55:0/2239553091 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3c80676b0 0x7fc3c8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:13.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.064+0000 7fc3daffd640 1 --2- 172.21.15.55:0/2239553091 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3c80676b0 0x7fc3c8069b70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc007fd0 tx=0x7fc3cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:13.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.340+0000 7fc3c3fff640 1 -- 172.21.15.55:0/2239553091 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fc3d0054760 con 0x7fc3dc15a540 2024-03-19T01:58:13.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.340+0000 7fc3c3fff640 1 -- 172.21.15.55:0/2239553091 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc3d00566c0 con 0x7fc3dc15a540 2024-03-19T01:58:13.598 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.600+0000 7fc3e181e640 1 -- 172.21.15.55:0/2239553091 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3dc154360 con 0x7fc3dc15a540 2024-03-19T01:58:13.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.648+0000 7fc3c3fff640 1 -- 172.21.15.55:0/2239553091 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fc3d005a760 con 0x7fc3dc15a540 2024-03-19T01:58:13.645 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:13.645 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:13.645 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:13.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.648+0000 7fc3e181e640 1 -- 172.21.15.55:0/2239553091 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3c80676b0 msgr2=0x7fc3c8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:13.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.648+0000 7fc3e181e640 1 --2- 172.21.15.55:0/2239553091 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3c80676b0 0x7fc3c8069b70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc007fd0 tx=0x7fc3cc009040 comp rx=0 tx=0).stop 2024-03-19T01:58:13.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.648+0000 7fc3e181e640 1 -- 172.21.15.55:0/2239553091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3dc15a540 msgr2=0x7fc3dc1f3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:13.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.648+0000 7fc3e181e640 1 --2- 172.21.15.55:0/2239553091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3dc15a540 0x7fc3dc1f3280 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc3d000c480 tx=0x7fc3d000c940 comp rx=0 tx=0).stop 2024-03-19T01:58:13.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.652+0000 7fc3e181e640 1 -- 172.21.15.55:0/2239553091 shutdown_connections 2024-03-19T01:58:13.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.652+0000 7fc3e181e640 1 --2- 172.21.15.55:0/2239553091 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3c80676b0 0x7fc3c8069b70 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:13.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.652+0000 7fc3e181e640 1 --2- 172.21.15.55:0/2239553091 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3dc15a920 0x7fc3dc158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:13.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.652+0000 7fc3e181e640 1 --2- 172.21.15.55:0/2239553091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3dc15a540 0x7fc3dc1f3280 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:13.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.652+0000 7fc3e181e640 1 --2- 172.21.15.55:0/2239553091 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3dc1438c0 0x7fc3dc1f0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:13.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.652+0000 7fc3e181e640 1 -- 172.21.15.55:0/2239553091 >> 172.21.15.55:0/2239553091 conn(0x7fc3dc067600 msgr2=0x7fc3dc0c8be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:13.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.652+0000 7fc3e181e640 1 -- 172.21.15.55:0/2239553091 shutdown_connections 2024-03-19T01:58:13.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:13.652+0000 7fc3e181e640 1 -- 172.21.15.55:0/2239553091 wait complete. 2024-03-19T01:58:14.268 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:14.268 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:15.270 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:15.401 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.404+0000 7fcbfd08a640 1 Processor -- start 2024-03-19T01:58:15.401 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.404+0000 7fcbfd08a640 1 -- start start 2024-03-19T01:58:15.402 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.404+0000 7fcbfd08a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fcbf815a540 0x7fcbf80c9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:15.402 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.404+0000 7fcbfd08a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fcbf815a920 0x7fcbf806cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:15.402 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.404+0000 7fcbfd08a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbf8070bd0 con 0x7fcbf815ade0 2024-03-19T01:58:15.402 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.404+0000 7fcbfd08a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fcbf806f5e0 con 0x7fcbf815a920 2024-03-19T01:58:15.402 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.404+0000 7fcbfd08a640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fcbf8070e90 con 0x7fcbf815a540 2024-03-19T01:58:15.402 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.404+0000 7fcbf659c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fcbf815a920 0x7fcbf806cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:15.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.408+0000 7fcbf659c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fcbf815a920 0x7fcbf806cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50428/0 (socket says 172.21.15.55:50428) 2024-03-19T01:58:15.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.408+0000 7fcbf659c640 1 -- 172.21.15.55:0/1026167646 learned_addr learned my addr 172.21.15.55:0/1026167646 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:15.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.408+0000 7fcbf6d9d640 1 --2- 172.21.15.55:0/1026167646 >> v2:172.21.15.84:3300/0 conn(0x7fcbf815a540 0x7fcbf80c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:15.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.408+0000 7fcbf659c640 1 -- 172.21.15.55:0/1026167646 >> v2:172.21.15.84:3300/0 conn(0x7fcbf815a540 msgr2=0x7fcbf80c9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:15.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.408+0000 7fcbf659c640 1 --2- 172.21.15.55:0/1026167646 >> v2:172.21.15.84:3300/0 conn(0x7fcbf815a540 0x7fcbf80c9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:15.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.408+0000 7fcbf659c640 1 -- 172.21.15.55:0/1026167646 >> v1:172.21.15.55:6789/0 conn(0x7fcbf815ade0 legacy=0x7fcbf81431b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:15.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.408+0000 7fcbf659c640 1 -- 172.21.15.55:0/1026167646 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf806f830 con 0x7fcbf815a920 2024-03-19T01:58:15.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.412+0000 7fcbf659c640 1 --2- 172.21.15.55:0/1026167646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcbf815a920 0x7fcbf806cd30 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcbf80725a0 tx=0x7fcbec01ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ae6dc77e9b8534 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:15.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.416+0000 7fcbf5d9b640 1 -- 172.21.15.55:0/1026167646 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec0b5070 con 0x7fcbf815a920 2024-03-19T01:58:15.415 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.420+0000 7fcbf5d9b640 1 -- 172.21.15.55:0/1026167646 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbec0a89d0 con 0x7fcbf815a920 2024-03-19T01:58:15.415 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.420+0000 7fcbf5d9b640 1 -- 172.21.15.55:0/1026167646 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec0a8da0 con 0x7fcbf815a920 2024-03-19T01:58:15.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.420+0000 7fcbfd08a640 1 -- 172.21.15.55:0/1026167646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcbf815a920 msgr2=0x7fcbf806cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:15.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.420+0000 7fcbfd08a640 1 --2- 172.21.15.55:0/1026167646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcbf815a920 0x7fcbf806cd30 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcbf80725a0 tx=0x7fcbec01ecd0 comp rx=0 tx=0).stop 2024-03-19T01:58:15.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.420+0000 7fcbfd08a640 1 -- 172.21.15.55:0/1026167646 shutdown_connections 2024-03-19T01:58:15.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.420+0000 7fcbfd08a640 1 --2- 172.21.15.55:0/1026167646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcbf815a920 0x7fcbf806cd30 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:15.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.420+0000 7fcbfd08a640 1 --2- 172.21.15.55:0/1026167646 >> v2:172.21.15.84:3300/0 conn(0x7fcbf815a540 0x7fcbf80c9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:15.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.420+0000 7fcbfd08a640 1 -- 172.21.15.55:0/1026167646 >> 172.21.15.55:0/1026167646 conn(0x7fcbf8067600 msgr2=0x7fcbf8071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:15.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.420+0000 7fcbfd08a640 1 -- 172.21.15.55:0/1026167646 shutdown_connections 2024-03-19T01:58:15.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.420+0000 7fcbfd08a640 1 -- 172.21.15.55:0/1026167646 wait complete. 2024-03-19T01:58:15.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.420+0000 7fcbfd08a640 1 Processor -- start 2024-03-19T01:58:15.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.420+0000 7fcbfd08a640 1 -- start start 2024-03-19T01:58:15.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.424+0000 7fcbfd08a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcbf815a540 0x7fcbf813d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:15.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.424+0000 7fcbfd08a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcbf815a920 0x7fcbf815c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:15.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.424+0000 7fcbfd08a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcbf815ade0 0x7fcbf815cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:15.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.424+0000 7fcbfd08a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf8070860 con 0x7fcbf815a920 2024-03-19T01:58:15.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.424+0000 7fcbfd08a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf806fdf0 con 0x7fcbf815a540 2024-03-19T01:58:15.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.424+0000 7fcbfd08a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fcbf806d270 con 0x7fcbf815ade0 2024-03-19T01:58:15.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.424+0000 7fcbf659c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcbf815a920 0x7fcbf815c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:15.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.424+0000 7fcbf759e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcbf815ade0 0x7fcbf815cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:15.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.424+0000 7fcbf659c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcbf815a920 0x7fcbf815c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50436/0 (socket says 172.21.15.55:50436) 2024-03-19T01:58:15.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.424+0000 7fcbf659c640 1 -- 172.21.15.55:0/4255240790 learned_addr learned my addr 172.21.15.55:0/4255240790 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:15.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.424+0000 7fcbf6d9d640 1 --2- 172.21.15.55:0/4255240790 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcbf815a540 0x7fcbf813d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:15.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.428+0000 7fcbf659c640 1 -- 172.21.15.55:0/4255240790 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcbf815ade0 msgr2=0x7fcbf815cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:15.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.428+0000 7fcbf659c640 1 --2- 172.21.15.55:0/4255240790 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcbf815ade0 0x7fcbf815cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:15.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.428+0000 7fcbf659c640 1 -- 172.21.15.55:0/4255240790 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcbf815a540 msgr2=0x7fcbf813d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:15.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.428+0000 7fcbf659c640 1 --2- 172.21.15.55:0/4255240790 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcbf815a540 0x7fcbf813d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:15.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.428+0000 7fcbf659c640 1 -- 172.21.15.55:0/4255240790 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf80704e0 con 0x7fcbf815a920 2024-03-19T01:58:15.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.428+0000 7fcbf659c640 1 --2- 172.21.15.55:0/4255240790 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcbf815a920 0x7fcbf815c860 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fcbec0b3040 tx=0x7fcbec01ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:15.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.436+0000 7fcbdf7fe640 1 -- 172.21.15.55:0/4255240790 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec0c0070 con 0x7fcbf815a920 2024-03-19T01:58:15.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.436+0000 7fcbdf7fe640 1 -- 172.21.15.55:0/4255240790 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbec0b0740 con 0x7fcbf815a920 2024-03-19T01:58:15.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.436+0000 7fcbfd08a640 1 -- 172.21.15.55:0/4255240790 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbf81f02b0 con 0x7fcbf815a920 2024-03-19T01:58:15.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.436+0000 7fcbdf7fe640 1 -- 172.21.15.55:0/4255240790 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec0b5040 con 0x7fcbf815a920 2024-03-19T01:58:15.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.436+0000 7fcbfd08a640 1 -- 172.21.15.55:0/4255240790 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbf81f0870 con 0x7fcbf815a920 2024-03-19T01:58:15.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.436+0000 7fcbfd08a640 1 -- 172.21.15.55:0/4255240790 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbf8068a40 con 0x7fcbf815a920 2024-03-19T01:58:15.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.460+0000 7fcbdf7fe640 1 -- 172.21.15.55:0/4255240790 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fcbec0c3070 con 0x7fcbf815a920 2024-03-19T01:58:15.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.460+0000 7fcbdf7fe640 1 --2- 172.21.15.55:0/4255240790 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcbe40676b0 0x7fcbe4069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:15.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.464+0000 7fcbf6d9d640 1 --2- 172.21.15.55:0/4255240790 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcbe40676b0 0x7fcbe4069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:15.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.464+0000 7fcbf6d9d640 1 --2- 172.21.15.55:0/4255240790 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcbe40676b0 0x7fcbe4069b70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009ac0 tx=0x7fcbe8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:15.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.760+0000 7fcbdf7fe640 1 -- 172.21.15.55:0/4255240790 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fcbec0a8980 con 0x7fcbf815a920 2024-03-19T01:58:15.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.760+0000 7fcbdf7fe640 1 -- 172.21.15.55:0/4255240790 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fcbec0ca080 con 0x7fcbf815a920 2024-03-19T01:58:15.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:15.980+0000 7fcbfd08a640 1 -- 172.21.15.55:0/4255240790 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcbf8155fa0 con 0x7fcbf815a920 2024-03-19T01:58:16.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:16.004+0000 7fcbdf7fe640 1 -- 172.21.15.55:0/4255240790 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fcbec018830 con 0x7fcbf815a920 2024-03-19T01:58:16.000 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:16.000 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:16.000 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:16.003 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:16.008+0000 7fcbfd08a640 1 -- 172.21.15.55:0/4255240790 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcbe40676b0 msgr2=0x7fcbe4069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:16.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:16.008+0000 7fcbfd08a640 1 --2- 172.21.15.55:0/4255240790 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcbe40676b0 0x7fcbe4069b70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009ac0 tx=0x7fcbe8009190 comp rx=0 tx=0).stop 2024-03-19T01:58:16.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:16.008+0000 7fcbfd08a640 1 -- 172.21.15.55:0/4255240790 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcbf815a920 msgr2=0x7fcbf815c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:16.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:16.008+0000 7fcbfd08a640 1 --2- 172.21.15.55:0/4255240790 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcbf815a920 0x7fcbf815c860 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fcbec0b3040 tx=0x7fcbec01ba70 comp rx=0 tx=0).stop 2024-03-19T01:58:16.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:16.008+0000 7fcbfd08a640 1 -- 172.21.15.55:0/4255240790 shutdown_connections 2024-03-19T01:58:16.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:16.008+0000 7fcbfd08a640 1 --2- 172.21.15.55:0/4255240790 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcbe40676b0 0x7fcbe4069b70 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:16.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:16.008+0000 7fcbfd08a640 1 --2- 172.21.15.55:0/4255240790 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcbf815ade0 0x7fcbf815cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:16.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:16.008+0000 7fcbfd08a640 1 --2- 172.21.15.55:0/4255240790 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcbf815a920 0x7fcbf815c860 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:16.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:16.008+0000 7fcbfd08a640 1 --2- 172.21.15.55:0/4255240790 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcbf815a540 0x7fcbf813d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:16.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:16.008+0000 7fcbfd08a640 1 -- 172.21.15.55:0/4255240790 >> 172.21.15.55:0/4255240790 conn(0x7fcbf8067600 msgr2=0x7fcbf80cbe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:16.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:16.008+0000 7fcbfd08a640 1 -- 172.21.15.55:0/4255240790 shutdown_connections 2024-03-19T01:58:16.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:16.008+0000 7fcbfd08a640 1 -- 172.21.15.55:0/4255240790 wait complete. 2024-03-19T01:58:16.625 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:16.625 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:17.626 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:17.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.760+0000 7f57d0d0c640 1 Processor -- start 2024-03-19T01:58:17.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.760+0000 7f57d0d0c640 1 -- start start 2024-03-19T01:58:17.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.760+0000 7f57d0d0c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57cc0720b0 0x7f57cc157e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:17.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.760+0000 7f57d0d0c640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57cc06f1f0 con 0x7f57cc152230 2024-03-19T01:58:17.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.760+0000 7f57d0d0c640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57cc06f4b0 con 0x7f57cc0c8de0 2024-03-19T01:58:17.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.760+0000 7f57d0d0c640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f57cc0700c0 con 0x7f57cc0720b0 2024-03-19T01:58:17.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.760+0000 7f57ca59c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f57cc152230 0x7f57cc159350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:56588/0 (socket says 172.21.15.55:56588) 2024-03-19T01:58:17.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.760+0000 7f57ca59c640 1 -- 172.21.15.55:0/3435883103 learned_addr learned my addr 172.21.15.55:0/3435883103 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:58:17.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.760+0000 7f57c9d9b640 1 --2- 172.21.15.55:0/3435883103 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57cc0720b0 0x7f57cc157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:17.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.764+0000 7f57c959a640 1 -- 172.21.15.55:0/3435883103 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 867137104 0 0) 0x7f57cc06f1f0 con 0x7f57cc152230 2024-03-19T01:58:17.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.764+0000 7f57c959a640 1 -- 172.21.15.55:0/3435883103 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57b4003530 con 0x7f57cc152230 2024-03-19T01:58:17.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.764+0000 7f57c959a640 1 -- 172.21.15.55:0/3435883103 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1825834128 0 0) 0x7f57cc06f4b0 con 0x7f57cc0c8de0 2024-03-19T01:58:17.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.764+0000 7f57c959a640 1 -- 172.21.15.55:0/3435883103 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57cc06f1f0 con 0x7f57cc0c8de0 2024-03-19T01:58:17.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.768+0000 7f57c959a640 1 -- 172.21.15.55:0/3435883103 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 313557174 0 0) 0x7f57b4003530 con 0x7f57cc152230 2024-03-19T01:58:17.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.768+0000 7f57c959a640 1 -- 172.21.15.55:0/3435883103 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57cc0720b0 msgr2=0x7f57cc157e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:17.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.768+0000 7f57c959a640 1 --2- 172.21.15.55:0/3435883103 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57cc0720b0 0x7f57cc157e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:17.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.768+0000 7f57c959a640 1 -- 172.21.15.55:0/3435883103 >> v1:172.21.15.84:6789/0 conn(0x7f57cc0c8de0 legacy=0x7f57cc0c2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:17.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.768+0000 7f57c959a640 1 -- 172.21.15.55:0/3435883103 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57cc06f790 con 0x7f57cc152230 2024-03-19T01:58:17.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.768+0000 7f57c959a640 1 -- 172.21.15.55:0/3435883103 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f57bc001680 con 0x7f57cc152230 2024-03-19T01:58:17.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.768+0000 7f57c959a640 1 -- 172.21.15.55:0/3435883103 >> v1:172.21.15.55:6789/0 conn(0x7f57cc152230 legacy=0x7f57cc159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:17.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.768+0000 7f57c959a640 1 --2- 172.21.15.55:0/3435883103 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57b4003b80 0x7f57b4003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:17.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.768+0000 7f57c959a640 1 --2- 172.21.15.55:0/3435883103 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57cc0720b0 0x7f57b4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:17.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.768+0000 7f57c959a640 1 --2- 172.21.15.55:0/3435883103 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57b4004d50 0x7f57b40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:17.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.768+0000 7f57c959a640 1 -- 172.21.15.55:0/3435883103 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f57cc0700c0 con 0x7f57b4003b80 2024-03-19T01:58:17.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.768+0000 7f57c959a640 1 -- 172.21.15.55:0/3435883103 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f57b400ba80 con 0x7f57b4004d50 2024-03-19T01:58:17.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.768+0000 7f57c959a640 1 -- 172.21.15.55:0/3435883103 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f57b400bc80 con 0x7f57cc0720b0 2024-03-19T01:58:17.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.772+0000 7f57ca59c640 1 --2- 172.21.15.55:0/3435883103 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57b4003b80 0x7f57b4003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:17.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.772+0000 7f57c9d9b640 1 --2- 172.21.15.55:0/3435883103 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57cc0720b0 0x7f57b4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:17.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.776+0000 7f57cad9d640 1 --2- 172.21.15.55:0/3435883103 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57b4004d50 0x7f57b40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:17.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.776+0000 7f57ca59c640 1 -- 172.21.15.55:0/3435883103 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57cc0720b0 msgr2=0x7f57b4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:17.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.776+0000 7f57ca59c640 1 --2- 172.21.15.55:0/3435883103 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57cc0720b0 0x7f57b4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:17.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.776+0000 7f57ca59c640 1 -- 172.21.15.55:0/3435883103 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57b4004d50 msgr2=0x7f57b40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:17.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.776+0000 7f57ca59c640 1 --2- 172.21.15.55:0/3435883103 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57b4004d50 0x7f57b40051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:17.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.776+0000 7f57ca59c640 1 -- 172.21.15.55:0/3435883103 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57b4003530 con 0x7f57b4003b80 2024-03-19T01:58:17.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.780+0000 7f57ca59c640 1 --2- 172.21.15.55:0/3435883103 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57b4003b80 0x7f57b4003f80 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00c0e0 tx=0x7f57bc01cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fce9cc9bbf4ccea8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:17.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.784+0000 7f57c959a640 1 -- 172.21.15.55:0/3435883103 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc007c50 con 0x7f57b4003b80 2024-03-19T01:58:17.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.784+0000 7f57c959a640 1 -- 172.21.15.55:0/3435883103 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57bc0a9070 con 0x7f57b4003b80 2024-03-19T01:58:17.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.784+0000 7f57c959a640 1 -- 172.21.15.55:0/3435883103 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc005d60 con 0x7f57b4003b80 2024-03-19T01:58:17.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.784+0000 7f57d0d0c640 1 -- 172.21.15.55:0/3435883103 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57b4003b80 msgr2=0x7f57b4003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:17.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.784+0000 7f57d0d0c640 1 --2- 172.21.15.55:0/3435883103 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57b4003b80 0x7f57b4003f80 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00c0e0 tx=0x7f57bc01cb80 comp rx=0 tx=0).stop 2024-03-19T01:58:17.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.784+0000 7f57cad9d640 1 -- 172.21.15.55:0/3435883103 reap_dead start 2024-03-19T01:58:17.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.784+0000 7f57d0d0c640 1 -- 172.21.15.55:0/3435883103 shutdown_connections 2024-03-19T01:58:17.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.784+0000 7f57d0d0c640 1 -- 172.21.15.55:0/3435883103 >> 172.21.15.55:0/3435883103 conn(0x7f57cc071590 msgr2=0x7f57cc0719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:17.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.784+0000 7f57d0d0c640 1 -- 172.21.15.55:0/3435883103 shutdown_connections 2024-03-19T01:58:17.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.784+0000 7f57d0d0c640 1 -- 172.21.15.55:0/3435883103 wait complete. 2024-03-19T01:58:17.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.784+0000 7f57d0d0c640 1 Processor -- start 2024-03-19T01:58:17.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.784+0000 7f57d0d0c640 1 -- start start 2024-03-19T01:58:17.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.788+0000 7f57d0d0c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57b4003b80 0x7f57cc143130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:17.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.788+0000 7f57d0d0c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57cc143d00 0x7f57cc1f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:17.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.788+0000 7f57d0d0c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57cc142980 0x7f57cc1f37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:17.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.788+0000 7f57d0d0c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f57cc0706b0 con 0x7f57b4003b80 2024-03-19T01:58:17.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.788+0000 7f57d0d0c640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f57cc070530 con 0x7f57cc143d00 2024-03-19T01:58:17.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.788+0000 7f57d0d0c640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f57cc164000 con 0x7f57cc142980 2024-03-19T01:58:17.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.788+0000 7f57ca59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57b4003b80 0x7f57cc143130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:17.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.788+0000 7f57c9d9b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57cc143d00 0x7f57cc1f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:17.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.788+0000 7f57ca59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57b4003b80 0x7f57cc143130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50446/0 (socket says 172.21.15.55:50446) 2024-03-19T01:58:17.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.788+0000 7f57ca59c640 1 -- 172.21.15.55:0/1717921578 learned_addr learned my addr 172.21.15.55:0/1717921578 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:17.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.788+0000 7f57cad9d640 1 --2- 172.21.15.55:0/1717921578 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57cc142980 0x7f57cc1f37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:17.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.792+0000 7f57c9d9b640 1 -- 172.21.15.55:0/1717921578 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57cc142980 msgr2=0x7f57cc1f37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:17.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.792+0000 7f57c9d9b640 1 --2- 172.21.15.55:0/1717921578 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57cc142980 0x7f57cc1f37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:17.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.792+0000 7f57c9d9b640 1 -- 172.21.15.55:0/1717921578 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57b4003b80 msgr2=0x7f57cc143130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:17.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.792+0000 7f57c9d9b640 1 --2- 172.21.15.55:0/1717921578 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57b4003b80 0x7f57cc143130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:17.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.792+0000 7f57c9d9b640 1 -- 172.21.15.55:0/1717921578 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57cc144260 con 0x7f57cc143d00 2024-03-19T01:58:17.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.800+0000 7f57c9d9b640 1 --2- 172.21.15.55:0/1717921578 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57cc143d00 0x7f57cc1f32b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f57c0011960 tx=0x7f57c0011e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:17.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.804+0000 7f57baffd640 1 -- 172.21.15.55:0/1717921578 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c0019070 con 0x7f57cc143d00 2024-03-19T01:58:17.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.804+0000 7f57baffd640 1 -- 172.21.15.55:0/1717921578 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57c0004300 con 0x7f57cc143d00 2024-03-19T01:58:17.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.804+0000 7f57baffd640 1 -- 172.21.15.55:0/1717921578 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c000f9e0 con 0x7f57cc143d00 2024-03-19T01:58:17.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.804+0000 7f57d0d0c640 1 -- 172.21.15.55:0/1717921578 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57cc1f6a70 con 0x7f57cc143d00 2024-03-19T01:58:17.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.804+0000 7f57d0d0c640 1 -- 172.21.15.55:0/1717921578 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57cc1f7030 con 0x7f57cc143d00 2024-03-19T01:58:17.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:17.808+0000 7f57d0d0c640 1 -- 172.21.15.55:0/1717921578 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57cc068a40 con 0x7f57cc143d00 2024-03-19T01:58:18.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:18.072+0000 7f57baffd640 1 -- 172.21.15.55:0/1717921578 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f57c001c070 con 0x7f57cc143d00 2024-03-19T01:58:18.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:18.072+0000 7f57baffd640 1 --2- 172.21.15.55:0/1717921578 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f57ac067690 0x7f57ac069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:18.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:18.072+0000 7f57ca59c640 1 --2- 172.21.15.55:0/1717921578 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f57ac067690 0x7f57ac069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:18.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:18.076+0000 7f57ca59c640 1 --2- 172.21.15.55:0/1717921578 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f57ac067690 0x7f57ac069b50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f57bc001410 tx=0x7f57bc0b1040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:18.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:18.076+0000 7f57baffd640 1 -- 172.21.15.55:0/1717921578 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f57c0054820 con 0x7f57cc143d00 2024-03-19T01:58:18.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:18.088+0000 7f57baffd640 1 -- 172.21.15.55:0/1717921578 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f57c0056780 con 0x7f57cc143d00 2024-03-19T01:58:18.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:18.316+0000 7f57d0d0c640 1 -- 172.21.15.55:0/1717921578 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f57cc155880 con 0x7f57cc143d00 2024-03-19T01:58:18.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:18.344+0000 7f57baffd640 1 -- 172.21.15.55:0/1717921578 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f57c005a820 con 0x7f57cc143d00 2024-03-19T01:58:18.339 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:18.339 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:18.340 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:18.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:18.348+0000 7f57d0d0c640 1 -- 172.21.15.55:0/1717921578 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f57ac067690 msgr2=0x7f57ac069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:18.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:18.348+0000 7f57d0d0c640 1 --2- 172.21.15.55:0/1717921578 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f57ac067690 0x7f57ac069b50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f57bc001410 tx=0x7f57bc0b1040 comp rx=0 tx=0).stop 2024-03-19T01:58:18.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:18.348+0000 7f57d0d0c640 1 -- 172.21.15.55:0/1717921578 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57cc143d00 msgr2=0x7f57cc1f32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:18.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:18.348+0000 7f57d0d0c640 1 --2- 172.21.15.55:0/1717921578 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57cc143d00 0x7f57cc1f32b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f57c0011960 tx=0x7f57c0011e20 comp rx=0 tx=0).stop 2024-03-19T01:58:18.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:18.348+0000 7f57d0d0c640 1 -- 172.21.15.55:0/1717921578 shutdown_connections 2024-03-19T01:58:18.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:18.348+0000 7f57d0d0c640 1 --2- 172.21.15.55:0/1717921578 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f57ac067690 0x7f57ac069b50 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:18.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:18.348+0000 7f57d0d0c640 1 --2- 172.21.15.55:0/1717921578 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f57cc142980 0x7f57cc1f37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:18.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:18.348+0000 7f57d0d0c640 1 --2- 172.21.15.55:0/1717921578 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57cc143d00 0x7f57cc1f32b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:18.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:18.348+0000 7f57d0d0c640 1 --2- 172.21.15.55:0/1717921578 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57b4003b80 0x7f57cc143130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:18.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:18.348+0000 7f57d0d0c640 1 -- 172.21.15.55:0/1717921578 >> 172.21.15.55:0/1717921578 conn(0x7f57cc071590 msgr2=0x7f57cc1583e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:18.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:18.348+0000 7f57d0d0c640 1 -- 172.21.15.55:0/1717921578 shutdown_connections 2024-03-19T01:58:18.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:18.348+0000 7f57d0d0c640 1 -- 172.21.15.55:0/1717921578 wait complete. 2024-03-19T01:58:18.964 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:18.965 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:19.966 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:20.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.100+0000 7fd2647ea640 1 Processor -- start 2024-03-19T01:58:20.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.100+0000 7fd2647ea640 1 -- start start 2024-03-19T01:58:20.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.100+0000 7fd2647ea640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd25c154340 0x7fd25c15e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:20.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.100+0000 7fd2647ea640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd25c154720 0x7fd25c0c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:20.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.100+0000 7fd2647ea640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd25c071280 con 0x7fd25c15aca0 2024-03-19T01:58:20.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.100+0000 7fd2647ea640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fd25c073840 con 0x7fd25c154720 2024-03-19T01:58:20.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.100+0000 7fd2647ea640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd25c15df10 con 0x7fd25c154340 2024-03-19T01:58:20.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.100+0000 7fd261d85640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd25c154720 0x7fd25c0c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:20.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.104+0000 7fd261d85640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd25c154720 0x7fd25c0c73c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34266/0 (socket says 172.21.15.55:34266) 2024-03-19T01:58:20.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.104+0000 7fd261d85640 1 -- 172.21.15.55:0/252549512 learned_addr learned my addr 172.21.15.55:0/252549512 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:20.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.104+0000 7fd261d85640 1 -- 172.21.15.55:0/252549512 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd25c154340 msgr2=0x7fd25c15e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:20.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.104+0000 7fd261d85640 1 --2- 172.21.15.55:0/252549512 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd25c154340 0x7fd25c15e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:20.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.104+0000 7fd261d85640 1 -- 172.21.15.55:0/252549512 >> v1:172.21.15.55:6789/0 conn(0x7fd25c15aca0 legacy=0x7fd25c072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:20.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.104+0000 7fd261d85640 1 -- 172.21.15.55:0/252549512 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd25c15d900 con 0x7fd25c154720 2024-03-19T01:58:20.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.108+0000 7fd261d85640 1 --2- 172.21.15.55:0/252549512 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd25c154720 0x7fd25c0c73c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd24c009870 tx=0x7fd24c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d71065518ea578a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:20.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.112+0000 7fd261584640 1 -- 172.21.15.55:0/252549512 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24c0b1070 con 0x7fd25c154720 2024-03-19T01:58:20.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.112+0000 7fd261584640 1 -- 172.21.15.55:0/252549512 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd24c0ac630 con 0x7fd25c154720 2024-03-19T01:58:20.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.112+0000 7fd261584640 1 -- 172.21.15.55:0/252549512 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24c0a8050 con 0x7fd25c154720 2024-03-19T01:58:20.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.116+0000 7fd2647ea640 1 -- 172.21.15.55:0/252549512 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd25c154720 msgr2=0x7fd25c0c73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:20.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.116+0000 7fd2647ea640 1 --2- 172.21.15.55:0/252549512 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd25c154720 0x7fd25c0c73c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd24c009870 tx=0x7fd24c01c520 comp rx=0 tx=0).stop 2024-03-19T01:58:20.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.116+0000 7fd2647ea640 1 -- 172.21.15.55:0/252549512 shutdown_connections 2024-03-19T01:58:20.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.116+0000 7fd2647ea640 1 --2- 172.21.15.55:0/252549512 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd25c154720 0x7fd25c0c73c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:20.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.116+0000 7fd2647ea640 1 --2- 172.21.15.55:0/252549512 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd25c154340 0x7fd25c15e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:20.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.116+0000 7fd2647ea640 1 -- 172.21.15.55:0/252549512 >> 172.21.15.55:0/252549512 conn(0x7fd25c0cc9c0 msgr2=0x7fd25c0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:20.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.116+0000 7fd2647ea640 1 -- 172.21.15.55:0/252549512 shutdown_connections 2024-03-19T01:58:20.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.116+0000 7fd2647ea640 1 -- 172.21.15.55:0/252549512 wait complete. 2024-03-19T01:58:20.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.116+0000 7fd2647ea640 1 Processor -- start 2024-03-19T01:58:20.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.116+0000 7fd2647ea640 1 -- start start 2024-03-19T01:58:20.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.116+0000 7fd2647ea640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd25c154340 0x7fd25c151800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:20.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.116+0000 7fd2647ea640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd25c154720 0x7fd25c14fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:20.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.116+0000 7fd2647ea640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd25c15aca0 0x7fd25c149130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:20.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.116+0000 7fd2647ea640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c166b70 con 0x7fd25c15aca0 2024-03-19T01:58:20.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.116+0000 7fd2647ea640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c15b100 con 0x7fd25c154720 2024-03-19T01:58:20.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.116+0000 7fd2647ea640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd25c166cf0 con 0x7fd25c154340 2024-03-19T01:58:20.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.116+0000 7fd262d87640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd25c15aca0 0x7fd25c149130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:20.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.120+0000 7fd262d87640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd25c15aca0 0x7fd25c149130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34274/0 (socket says 172.21.15.55:34274) 2024-03-19T01:58:20.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.120+0000 7fd262d87640 1 -- 172.21.15.55:0/3450859765 learned_addr learned my addr 172.21.15.55:0/3450859765 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:20.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.120+0000 7fd261d85640 1 --2- 172.21.15.55:0/3450859765 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd25c154720 0x7fd25c14fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:20.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.124+0000 7fd262d87640 1 -- 172.21.15.55:0/3450859765 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd25c154340 msgr2=0x7fd25c151800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:20.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.124+0000 7fd262d87640 1 --2- 172.21.15.55:0/3450859765 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd25c154340 0x7fd25c151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:20.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.124+0000 7fd262d87640 1 -- 172.21.15.55:0/3450859765 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd25c154720 msgr2=0x7fd25c14fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:20.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.124+0000 7fd262d87640 1 --2- 172.21.15.55:0/3450859765 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd25c154720 0x7fd25c14fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:20.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.124+0000 7fd262d87640 1 -- 172.21.15.55:0/3450859765 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd25c14e500 con 0x7fd25c15aca0 2024-03-19T01:58:20.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.128+0000 7fd262d87640 1 --2- 172.21.15.55:0/3450859765 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd25c15aca0 0x7fd25c149130 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd25400b9c0 tx=0x7fd25400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:20.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.132+0000 7fd252ffd640 1 -- 172.21.15.55:0/3450859765 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254019070 con 0x7fd25c15aca0 2024-03-19T01:58:20.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.132+0000 7fd252ffd640 1 -- 172.21.15.55:0/3450859765 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd25400cba0 con 0x7fd25c15aca0 2024-03-19T01:58:20.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.132+0000 7fd2647ea640 1 -- 172.21.15.55:0/3450859765 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd25c1538c0 con 0x7fd25c15aca0 2024-03-19T01:58:20.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.132+0000 7fd252ffd640 1 -- 172.21.15.55:0/3450859765 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254014440 con 0x7fd25c15aca0 2024-03-19T01:58:20.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.132+0000 7fd2647ea640 1 -- 172.21.15.55:0/3450859765 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd25c1f3510 con 0x7fd25c15aca0 2024-03-19T01:58:20.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.132+0000 7fd250ff9640 1 -- 172.21.15.55:0/3450859765 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2380052a0 con 0x7fd25c15aca0 2024-03-19T01:58:20.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.152+0000 7fd252ffd640 1 -- 172.21.15.55:0/3450859765 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fd25401c070 con 0x7fd25c15aca0 2024-03-19T01:58:20.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.152+0000 7fd252ffd640 1 --2- 172.21.15.55:0/3450859765 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd240067660 0x7fd240069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:20.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.152+0000 7fd262586640 1 --2- 172.21.15.55:0/3450859765 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd240067660 0x7fd240069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:20.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.152+0000 7fd262586640 1 --2- 172.21.15.55:0/3450859765 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd240067660 0x7fd240069b20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd258009730 tx=0x7fd258006ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:20.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.484+0000 7fd252ffd640 1 -- 172.21.15.55:0/3450859765 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fd2540543e0 con 0x7fd25c15aca0 2024-03-19T01:58:20.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.488+0000 7fd252ffd640 1 -- 172.21.15.55:0/3450859765 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fd254023070 con 0x7fd25c15aca0 2024-03-19T01:58:20.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.712+0000 7fd250ff9640 1 -- 172.21.15.55:0/3450859765 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd238005590 con 0x7fd25c15aca0 2024-03-19T01:58:20.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.728+0000 7fd252ffd640 1 -- 172.21.15.55:0/3450859765 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fd25c14e500 con 0x7fd25c15aca0 2024-03-19T01:58:20.727 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:20.727 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:20.727 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:20.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.732+0000 7fd250ff9640 1 -- 172.21.15.55:0/3450859765 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd240067660 msgr2=0x7fd240069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:20.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.732+0000 7fd250ff9640 1 --2- 172.21.15.55:0/3450859765 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd240067660 0x7fd240069b20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd258009730 tx=0x7fd258006ca0 comp rx=0 tx=0).stop 2024-03-19T01:58:20.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.732+0000 7fd250ff9640 1 -- 172.21.15.55:0/3450859765 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd25c15aca0 msgr2=0x7fd25c149130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:20.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.732+0000 7fd250ff9640 1 --2- 172.21.15.55:0/3450859765 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd25c15aca0 0x7fd25c149130 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd25400b9c0 tx=0x7fd25400be80 comp rx=0 tx=0).stop 2024-03-19T01:58:20.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.732+0000 7fd250ff9640 1 -- 172.21.15.55:0/3450859765 shutdown_connections 2024-03-19T01:58:20.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.732+0000 7fd250ff9640 1 --2- 172.21.15.55:0/3450859765 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd240067660 0x7fd240069b20 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:20.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.732+0000 7fd250ff9640 1 --2- 172.21.15.55:0/3450859765 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd25c15aca0 0x7fd25c149130 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:20.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.732+0000 7fd250ff9640 1 --2- 172.21.15.55:0/3450859765 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd25c154720 0x7fd25c14fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:20.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.732+0000 7fd250ff9640 1 --2- 172.21.15.55:0/3450859765 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd25c154340 0x7fd25c151800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:20.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.732+0000 7fd250ff9640 1 -- 172.21.15.55:0/3450859765 >> 172.21.15.55:0/3450859765 conn(0x7fd25c0cc9c0 msgr2=0x7fd25c14cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:20.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.732+0000 7fd250ff9640 1 -- 172.21.15.55:0/3450859765 shutdown_connections 2024-03-19T01:58:20.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:20.732+0000 7fd250ff9640 1 -- 172.21.15.55:0/3450859765 wait complete. 2024-03-19T01:58:21.355 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:21.355 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:22.357 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:22.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.467+0000 7f203245e640 1 Processor -- start 2024-03-19T01:58:22.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.467+0000 7f203245e640 1 -- start start 2024-03-19T01:58:22.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.471+0000 7f203245e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f202c0c73c0 0x7f202c158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:22.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.471+0000 7f203245e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f202c15b550 0x7f202c0c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:22.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.471+0000 7f203245e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f202c0c7900 con 0x7f202c0c38c0 2024-03-19T01:58:22.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.471+0000 7f203245e640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f202c0c7be0 con 0x7f202c0c73c0 2024-03-19T01:58:22.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.471+0000 7f203245e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f202c0c7ec0 con 0x7f202c15b550 2024-03-19T01:58:22.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.471+0000 7f20309fb640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f202c15b550 0x7f202c0c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:22.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.471+0000 7f20309fb640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f202c15b550 0x7f202c0c60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:46890/0 (socket says 172.21.15.55:46890) 2024-03-19T01:58:22.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.471+0000 7f20309fb640 1 -- 172.21.15.55:0/782855119 learned_addr learned my addr 172.21.15.55:0/782855119 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:22.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.475+0000 7f20309fb640 1 -- 172.21.15.55:0/782855119 >> v2:172.21.15.55:3300/0 conn(0x7f202c0c73c0 msgr2=0x7f202c158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:22.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.475+0000 7f20309fb640 1 --2- 172.21.15.55:0/782855119 >> v2:172.21.15.55:3300/0 conn(0x7f202c0c73c0 0x7f202c158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:22.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.475+0000 7f20309fb640 1 -- 172.21.15.55:0/782855119 >> v1:172.21.15.55:6789/0 conn(0x7f202c0c38c0 legacy=0x7f202c06b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:22.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.475+0000 7f20309fb640 1 -- 172.21.15.55:0/782855119 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f202c1582b0 con 0x7f202c15b550 2024-03-19T01:58:22.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.479+0000 7f20309fb640 1 --2- 172.21.15.55:0/782855119 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f202c15b550 0x7f202c0c60d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2024009e50 tx=0x7f202401f1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da185a9381671956 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:22.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.487+0000 7f202affd640 1 -- 172.21.15.55:0/782855119 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20240b4070 con 0x7f202c15b550 2024-03-19T01:58:22.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.487+0000 7f202affd640 1 -- 172.21.15.55:0/782855119 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20240af400 con 0x7f202c15b550 2024-03-19T01:58:22.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.487+0000 7f202affd640 1 -- 172.21.15.55:0/782855119 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20240af7d0 con 0x7f202c15b550 2024-03-19T01:58:22.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.487+0000 7f203245e640 1 -- 172.21.15.55:0/782855119 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f202c15b550 msgr2=0x7f202c0c60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:22.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.487+0000 7f203245e640 1 --2- 172.21.15.55:0/782855119 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f202c15b550 0x7f202c0c60d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2024009e50 tx=0x7f202401f1e0 comp rx=0 tx=0).stop 2024-03-19T01:58:22.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.487+0000 7f203245e640 1 -- 172.21.15.55:0/782855119 shutdown_connections 2024-03-19T01:58:22.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.487+0000 7f203245e640 1 --2- 172.21.15.55:0/782855119 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f202c15b550 0x7f202c0c60d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:22.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.487+0000 7f203245e640 1 --2- 172.21.15.55:0/782855119 >> v2:172.21.15.55:3300/0 conn(0x7f202c0c73c0 0x7f202c158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:22.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.487+0000 7f203245e640 1 -- 172.21.15.55:0/782855119 >> 172.21.15.55:0/782855119 conn(0x7f202c0cc9c0 msgr2=0x7f202c0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:22.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.487+0000 7f203245e640 1 -- 172.21.15.55:0/782855119 shutdown_connections 2024-03-19T01:58:22.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.487+0000 7f203245e640 1 -- 172.21.15.55:0/782855119 wait complete. 2024-03-19T01:58:22.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.487+0000 7f203245e640 1 Processor -- start 2024-03-19T01:58:22.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.487+0000 7f203245e640 1 -- start start 2024-03-19T01:58:22.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.491+0000 7f203245e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f202c0c38c0 0x7f202c1f0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:22.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.491+0000 7f203245e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f202c0c73c0 0x7f202c133e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:22.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.491+0000 7f203245e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f202c15b550 0x7f202c134380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:22.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.491+0000 7f203245e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f202c06c090 con 0x7f202c15b550 2024-03-19T01:58:22.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.491+0000 7f203245e640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f202c157f60 con 0x7f202c0c38c0 2024-03-19T01:58:22.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.491+0000 7f203245e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f202c06c210 con 0x7f202c0c73c0 2024-03-19T01:58:22.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.491+0000 7f202b7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f202c0c73c0 0x7f202c133e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:22.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.491+0000 7f202b7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f202c0c73c0 0x7f202c133e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:46894/0 (socket says 172.21.15.55:46894) 2024-03-19T01:58:22.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.491+0000 7f202b7fe640 1 -- 172.21.15.55:0/2388791092 learned_addr learned my addr 172.21.15.55:0/2388791092 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:22.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.491+0000 7f202bfff640 1 --2- 172.21.15.55:0/2388791092 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f202c0c38c0 0x7f202c1f0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:22.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.495+0000 7f202bfff640 1 -- 172.21.15.55:0/2388791092 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f202c0c73c0 msgr2=0x7f202c133e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:22.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.495+0000 7f202bfff640 1 --2- 172.21.15.55:0/2388791092 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f202c0c73c0 0x7f202c133e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:22.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.495+0000 7f202bfff640 1 -- 172.21.15.55:0/2388791092 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f202c15b550 msgr2=0x7f202c134380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:22.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.495+0000 7f202bfff640 1 --2- 172.21.15.55:0/2388791092 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f202c15b550 0x7f202c134380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:22.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.495+0000 7f202bfff640 1 -- 172.21.15.55:0/2388791092 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f202c1f1280 con 0x7f202c0c38c0 2024-03-19T01:58:22.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.503+0000 7f202bfff640 1 --2- 172.21.15.55:0/2388791092 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f202c0c38c0 0x7f202c1f0d40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f201c00c390 tx=0x7f201c00c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:22.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.507+0000 7f2028ff9640 1 -- 172.21.15.55:0/2388791092 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c017070 con 0x7f202c0c38c0 2024-03-19T01:58:22.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.507+0000 7f203245e640 1 -- 172.21.15.55:0/2388791092 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f202c156090 con 0x7f202c0c38c0 2024-03-19T01:58:22.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.507+0000 7f203245e640 1 -- 172.21.15.55:0/2388791092 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f202c1352e0 con 0x7f202c0c38c0 2024-03-19T01:58:22.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.511+0000 7f203245e640 1 -- 172.21.15.55:0/2388791092 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f202c068a40 con 0x7f202c0c38c0 2024-03-19T01:58:22.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.515+0000 7f2028ff9640 1 -- 172.21.15.55:0/2388791092 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f201c007030 con 0x7f202c0c38c0 2024-03-19T01:58:22.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.515+0000 7f2028ff9640 1 -- 172.21.15.55:0/2388791092 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c021060 con 0x7f202c0c38c0 2024-03-19T01:58:22.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.555+0000 7f2028ff9640 1 -- 172.21.15.55:0/2388791092 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f201c01a070 con 0x7f202c0c38c0 2024-03-19T01:58:22.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.559+0000 7f2028ff9640 1 --2- 172.21.15.55:0/2388791092 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2018067690 0x7f2018069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:22.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.559+0000 7f202b7fe640 1 --2- 172.21.15.55:0/2388791092 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2018067690 0x7f2018069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:22.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.559+0000 7f202b7fe640 1 --2- 172.21.15.55:0/2388791092 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2018067690 0x7f2018069b50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f202c137230 tx=0x7f2020009210 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:22.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.563+0000 7f2028ff9640 1 -- 172.21.15.55:0/2388791092 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f201c0536d0 con 0x7f202c0c38c0 2024-03-19T01:58:22.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:22.875+0000 7f2028ff9640 1 -- 172.21.15.55:0/2388791092 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f201c055630 con 0x7f202c0c38c0 2024-03-19T01:58:23.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:23.107+0000 7f203245e640 1 -- 172.21.15.55:0/2388791092 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f202c072a40 con 0x7f202c0c38c0 2024-03-19T01:58:23.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:23.127+0000 7f2028ff9640 1 -- 172.21.15.55:0/2388791092 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f201c0596d0 con 0x7f202c0c38c0 2024-03-19T01:58:23.127 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:23.127 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:23.127 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:23.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:23.131+0000 7f203245e640 1 -- 172.21.15.55:0/2388791092 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2018067690 msgr2=0x7f2018069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:23.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:23.131+0000 7f203245e640 1 --2- 172.21.15.55:0/2388791092 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2018067690 0x7f2018069b50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f202c137230 tx=0x7f2020009210 comp rx=0 tx=0).stop 2024-03-19T01:58:23.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:23.131+0000 7f203245e640 1 -- 172.21.15.55:0/2388791092 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f202c0c38c0 msgr2=0x7f202c1f0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:23.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:23.131+0000 7f203245e640 1 --2- 172.21.15.55:0/2388791092 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f202c0c38c0 0x7f202c1f0d40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f201c00c390 tx=0x7f201c00c850 comp rx=0 tx=0).stop 2024-03-19T01:58:23.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:23.131+0000 7f203245e640 1 -- 172.21.15.55:0/2388791092 shutdown_connections 2024-03-19T01:58:23.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:23.131+0000 7f203245e640 1 --2- 172.21.15.55:0/2388791092 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2018067690 0x7f2018069b50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:23.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:23.131+0000 7f203245e640 1 --2- 172.21.15.55:0/2388791092 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f202c15b550 0x7f202c134380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:23.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:23.131+0000 7f203245e640 1 --2- 172.21.15.55:0/2388791092 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f202c0c73c0 0x7f202c133e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:23.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:23.131+0000 7f203245e640 1 --2- 172.21.15.55:0/2388791092 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f202c0c38c0 0x7f202c1f0d40 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:23.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:23.131+0000 7f203245e640 1 -- 172.21.15.55:0/2388791092 >> 172.21.15.55:0/2388791092 conn(0x7f202c0cc9c0 msgr2=0x7f202c0cbee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:23.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:23.131+0000 7f203245e640 1 -- 172.21.15.55:0/2388791092 shutdown_connections 2024-03-19T01:58:23.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:23.131+0000 7f203245e640 1 -- 172.21.15.55:0/2388791092 wait complete. 2024-03-19T01:58:23.749 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:23.749 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:24.750 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:24.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.867+0000 7fd1c9d07640 1 Processor -- start 2024-03-19T01:58:24.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.867+0000 7fd1c9d07640 1 -- start start 2024-03-19T01:58:24.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.867+0000 7fd1c9d07640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd1c40c96b0 0x7fd1c4070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:24.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.867+0000 7fd1c9d07640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fd1c40c9a90 0x7fd1c40c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:24.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.867+0000 7fd1c9d07640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1c415e3a0 con 0x7fd1c41540c0 2024-03-19T01:58:24.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.867+0000 7fd1c9d07640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fd1c415e680 con 0x7fd1c40c9a90 2024-03-19T01:58:24.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.867+0000 7fd1c9d07640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c415a330 con 0x7fd1c40c96b0 2024-03-19T01:58:24.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.871+0000 7fd1c3fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fd1c41540c0 0x7fd1c415d810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:44798/0 (socket says 172.21.15.55:44798) 2024-03-19T01:58:24.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.871+0000 7fd1c3fff640 1 -- 172.21.15.55:0/1222281610 learned_addr learned my addr 172.21.15.55:0/1222281610 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:58:24.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.871+0000 7fd1c2ffd640 1 --2- 172.21.15.55:0/1222281610 >> v2:172.21.15.84:3300/0 conn(0x7fd1c40c9a90 0x7fd1c40c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:24.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.871+0000 7fd1c37fe640 1 --2- 172.21.15.55:0/1222281610 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd1c40c96b0 0x7fd1c4070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:24.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.871+0000 7fd1c27fc640 1 -- 172.21.15.55:0/1222281610 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3633814624 0 0) 0x7fd1c415e3a0 con 0x7fd1c41540c0 2024-03-19T01:58:24.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.871+0000 7fd1c27fc640 1 -- 172.21.15.55:0/1222281610 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1ac003530 con 0x7fd1c41540c0 2024-03-19T01:58:24.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.875+0000 7fd1c2ffd640 1 -- 172.21.15.55:0/1222281610 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd1c40c96b0 msgr2=0x7fd1c4070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:24.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.875+0000 7fd1c2ffd640 1 --2- 172.21.15.55:0/1222281610 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd1c40c96b0 0x7fd1c4070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:24.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.875+0000 7fd1c2ffd640 1 -- 172.21.15.55:0/1222281610 >> v1:172.21.15.55:6789/0 conn(0x7fd1c41540c0 legacy=0x7fd1c415d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:24.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.875+0000 7fd1c2ffd640 1 -- 172.21.15.55:0/1222281610 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c415a610 con 0x7fd1c40c9a90 2024-03-19T01:58:24.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.879+0000 7fd1c2ffd640 1 --2- 172.21.15.55:0/1222281610 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd1c40c9a90 0x7fd1c40c73c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd1b80097c0 tx=0x7fd1b801c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd5cc28d34627b6d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:24.880 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.883+0000 7fd1c27fc640 1 -- 172.21.15.55:0/1222281610 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b80b0070 con 0x7fd1c40c9a90 2024-03-19T01:58:24.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.887+0000 7fd1c27fc640 1 -- 172.21.15.55:0/1222281610 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1b80ab600 con 0x7fd1c40c9a90 2024-03-19T01:58:24.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.887+0000 7fd1c27fc640 1 -- 172.21.15.55:0/1222281610 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b80ab9b0 con 0x7fd1c40c9a90 2024-03-19T01:58:24.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.887+0000 7fd1c9d07640 1 -- 172.21.15.55:0/1222281610 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd1c40c9a90 msgr2=0x7fd1c40c73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:24.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.887+0000 7fd1c9d07640 1 --2- 172.21.15.55:0/1222281610 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd1c40c9a90 0x7fd1c40c73c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd1b80097c0 tx=0x7fd1b801c410 comp rx=0 tx=0).stop 2024-03-19T01:58:24.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.887+0000 7fd1c9d07640 1 -- 172.21.15.55:0/1222281610 shutdown_connections 2024-03-19T01:58:24.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.887+0000 7fd1c9d07640 1 --2- 172.21.15.55:0/1222281610 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd1c40c9a90 0x7fd1c40c73c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:24.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.887+0000 7fd1c9d07640 1 --2- 172.21.15.55:0/1222281610 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd1c40c96b0 0x7fd1c4070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:24.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.887+0000 7fd1c9d07640 1 -- 172.21.15.55:0/1222281610 >> 172.21.15.55:0/1222281610 conn(0x7fd1c40cc9c0 msgr2=0x7fd1c40ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:24.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.887+0000 7fd1c9d07640 1 -- 172.21.15.55:0/1222281610 shutdown_connections 2024-03-19T01:58:24.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.887+0000 7fd1c9d07640 1 -- 172.21.15.55:0/1222281610 wait complete. 2024-03-19T01:58:24.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.887+0000 7fd1c9d07640 1 Processor -- start 2024-03-19T01:58:24.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.887+0000 7fd1c9d07640 1 -- start start 2024-03-19T01:58:24.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.891+0000 7fd1c9d07640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd1c40c96b0 0x7fd1c40cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:24.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.891+0000 7fd1c9d07640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd1c40c9a90 0x7fd1c41f4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:24.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.891+0000 7fd1c9d07640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1c41540c0 0x7fd1c40cc030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:24.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.891+0000 7fd1c9d07640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c40c7900 con 0x7fd1c41540c0 2024-03-19T01:58:24.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.891+0000 7fd1c9d07640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c415b0f0 con 0x7fd1c40c96b0 2024-03-19T01:58:24.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.891+0000 7fd1c9d07640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c40c7a80 con 0x7fd1c40c9a90 2024-03-19T01:58:24.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.891+0000 7fd1c3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1c41540c0 0x7fd1c40cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:24.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.891+0000 7fd1c2ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd1c40c9a90 0x7fd1c41f4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:24.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.891+0000 7fd1c3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1c41540c0 0x7fd1c40cc030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34308/0 (socket says 172.21.15.55:34308) 2024-03-19T01:58:24.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.891+0000 7fd1c3fff640 1 -- 172.21.15.55:0/2632607298 learned_addr learned my addr 172.21.15.55:0/2632607298 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:24.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.891+0000 7fd1c37fe640 1 --2- 172.21.15.55:0/2632607298 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd1c40c96b0 0x7fd1c40cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:24.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.895+0000 7fd1c3fff640 1 -- 172.21.15.55:0/2632607298 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd1c40c9a90 msgr2=0x7fd1c41f4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:24.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.895+0000 7fd1c3fff640 1 --2- 172.21.15.55:0/2632607298 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd1c40c9a90 0x7fd1c41f4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:24.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.895+0000 7fd1c3fff640 1 -- 172.21.15.55:0/2632607298 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd1c40c96b0 msgr2=0x7fd1c40cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:24.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.895+0000 7fd1c3fff640 1 --2- 172.21.15.55:0/2632607298 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd1c40c96b0 0x7fd1c40cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:24.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.895+0000 7fd1c3fff640 1 -- 172.21.15.55:0/2632607298 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c415c7d0 con 0x7fd1c41540c0 2024-03-19T01:58:24.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.895+0000 7fd1c3fff640 1 --2- 172.21.15.55:0/2632607298 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1c41540c0 0x7fd1c40cc030 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00cb70 tx=0x7fd1bc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:24.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.903+0000 7fd1abfff640 1 -- 172.21.15.55:0/2632607298 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc017070 con 0x7fd1c41540c0 2024-03-19T01:58:24.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.903+0000 7fd1abfff640 1 -- 172.21.15.55:0/2632607298 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1bc00ad40 con 0x7fd1c41540c0 2024-03-19T01:58:24.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.903+0000 7fd1abfff640 1 -- 172.21.15.55:0/2632607298 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc012620 con 0x7fd1c41540c0 2024-03-19T01:58:24.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.903+0000 7fd1c9d07640 1 -- 172.21.15.55:0/2632607298 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1c415c210 con 0x7fd1c41540c0 2024-03-19T01:58:24.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.903+0000 7fd1c9d07640 1 -- 172.21.15.55:0/2632607298 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1c40ca9b0 con 0x7fd1c41540c0 2024-03-19T01:58:24.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.903+0000 7fd1c9d07640 1 -- 172.21.15.55:0/2632607298 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1c4068a40 con 0x7fd1c41540c0 2024-03-19T01:58:24.925 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.927+0000 7fd1abfff640 1 -- 172.21.15.55:0/2632607298 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fd1bc01a070 con 0x7fd1c41540c0 2024-03-19T01:58:24.925 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.927+0000 7fd1abfff640 1 --2- 172.21.15.55:0/2632607298 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd1b0067690 0x7fd1b0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:24.925 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.927+0000 7fd1c37fe640 1 --2- 172.21.15.55:0/2632607298 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd1b0067690 0x7fd1b0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:24.926 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:24.927+0000 7fd1c37fe640 1 --2- 172.21.15.55:0/2632607298 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd1b0067690 0x7fd1b0069b50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd1c40690a0 tx=0x7fd1b4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:25.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:25.235+0000 7fd1abfff640 1 -- 172.21.15.55:0/2632607298 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fd1bc054180 con 0x7fd1c41540c0 2024-03-19T01:58:25.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:25.235+0000 7fd1abfff640 1 -- 172.21.15.55:0/2632607298 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fd1bc021020 con 0x7fd1c41540c0 2024-03-19T01:58:25.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:25.455+0000 7fd1c9d07640 1 -- 172.21.15.55:0/2632607298 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1c406fb90 con 0x7fd1c41540c0 2024-03-19T01:58:25.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:25.543+0000 7fd1abfff640 1 -- 172.21.15.55:0/2632607298 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fd1bc0560e0 con 0x7fd1c41540c0 2024-03-19T01:58:25.542 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:25.542 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:25.542 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:25.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:25.547+0000 7fd1c9d07640 1 -- 172.21.15.55:0/2632607298 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd1b0067690 msgr2=0x7fd1b0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:25.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:25.547+0000 7fd1c9d07640 1 --2- 172.21.15.55:0/2632607298 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd1b0067690 0x7fd1b0069b50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd1c40690a0 tx=0x7fd1b4009190 comp rx=0 tx=0).stop 2024-03-19T01:58:25.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:25.547+0000 7fd1c9d07640 1 -- 172.21.15.55:0/2632607298 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1c41540c0 msgr2=0x7fd1c40cc030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:25.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:25.547+0000 7fd1c9d07640 1 --2- 172.21.15.55:0/2632607298 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1c41540c0 0x7fd1c40cc030 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00cb70 tx=0x7fd1bc00a430 comp rx=0 tx=0).stop 2024-03-19T01:58:25.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:25.547+0000 7fd1c9d07640 1 -- 172.21.15.55:0/2632607298 shutdown_connections 2024-03-19T01:58:25.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:25.547+0000 7fd1c9d07640 1 --2- 172.21.15.55:0/2632607298 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd1b0067690 0x7fd1b0069b50 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:25.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:25.547+0000 7fd1c9d07640 1 --2- 172.21.15.55:0/2632607298 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1c41540c0 0x7fd1c40cc030 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:25.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:25.547+0000 7fd1c9d07640 1 --2- 172.21.15.55:0/2632607298 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd1c40c9a90 0x7fd1c41f4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:25.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:25.547+0000 7fd1c9d07640 1 --2- 172.21.15.55:0/2632607298 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd1c40c96b0 0x7fd1c40cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:25.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:25.547+0000 7fd1c9d07640 1 -- 172.21.15.55:0/2632607298 >> 172.21.15.55:0/2632607298 conn(0x7fd1c40cc9c0 msgr2=0x7fd1c40c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:25.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:25.547+0000 7fd1c9d07640 1 -- 172.21.15.55:0/2632607298 shutdown_connections 2024-03-19T01:58:25.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:25.547+0000 7fd1c9d07640 1 -- 172.21.15.55:0/2632607298 wait complete. 2024-03-19T01:58:26.171 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:26.171 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:27.172 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:27.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.295+0000 7f0a754c3640 1 Processor -- start 2024-03-19T01:58:27.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.295+0000 7f0a754c3640 1 -- start start 2024-03-19T01:58:27.293 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.295+0000 7f0a754c3640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0a700ca350 0x7f0a7015b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:27.293 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.295+0000 7f0a754c3640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0a700c3320 0x7f0a700728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:27.293 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.295+0000 7f0a754c3640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a70151330 con 0x7f0a70150f50 2024-03-19T01:58:27.293 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.295+0000 7f0a754c3640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f0a700c3e00 con 0x7f0a700c3320 2024-03-19T01:58:27.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.295+0000 7f0a754c3640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f0a7015ac10 con 0x7f0a700ca350 2024-03-19T01:58:27.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.295+0000 7f0a6f7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0a700c3320 0x7f0a700728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:27.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.295+0000 7f0a6e7fc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0a700ca350 0x7f0a7015b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:27.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.295+0000 7f0a6f7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0a700c3320 0x7f0a700728b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34320/0 (socket says 172.21.15.55:34320) 2024-03-19T01:58:27.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.295+0000 7f0a6f7fe640 1 -- 172.21.15.55:0/908795728 learned_addr learned my addr 172.21.15.55:0/908795728 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:27.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.299+0000 7f0a6f7fe640 1 -- 172.21.15.55:0/908795728 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0a700ca350 msgr2=0x7f0a7015b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:27.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.299+0000 7f0a6f7fe640 1 --2- 172.21.15.55:0/908795728 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0a700ca350 0x7f0a7015b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:27.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.299+0000 7f0a6f7fe640 1 -- 172.21.15.55:0/908795728 >> v1:172.21.15.84:6789/0 conn(0x7f0a70150f50 legacy=0x7f0a70157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:27.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.299+0000 7f0a6f7fe640 1 -- 172.21.15.55:0/908795728 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a700c3840 con 0x7f0a700c3320 2024-03-19T01:58:27.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.303+0000 7f0a6f7fe640 1 --2- 172.21.15.55:0/908795728 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a700c3320 0x7f0a700728b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0a6800d150 tx=0x7f0a6801ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed79f8408f4e89df server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:27.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.307+0000 7f0a6dffb640 1 -- 172.21.15.55:0/908795728 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a680a7d00 con 0x7f0a700c3320 2024-03-19T01:58:27.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.307+0000 7f0a6dffb640 1 -- 172.21.15.55:0/908795728 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a6800b0d0 con 0x7f0a700c3320 2024-03-19T01:58:27.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.307+0000 7f0a6dffb640 1 -- 172.21.15.55:0/908795728 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a68009600 con 0x7f0a700c3320 2024-03-19T01:58:27.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.307+0000 7f0a754c3640 1 -- 172.21.15.55:0/908795728 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a700c3320 msgr2=0x7f0a700728b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:27.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.307+0000 7f0a754c3640 1 --2- 172.21.15.55:0/908795728 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a700c3320 0x7f0a700728b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0a6800d150 tx=0x7f0a6801ec30 comp rx=0 tx=0).stop 2024-03-19T01:58:27.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.307+0000 7f0a754c3640 1 -- 172.21.15.55:0/908795728 shutdown_connections 2024-03-19T01:58:27.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.307+0000 7f0a754c3640 1 --2- 172.21.15.55:0/908795728 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a700c3320 0x7f0a700728b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:27.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.307+0000 7f0a754c3640 1 --2- 172.21.15.55:0/908795728 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0a700ca350 0x7f0a7015b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:27.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.307+0000 7f0a754c3640 1 -- 172.21.15.55:0/908795728 >> 172.21.15.55:0/908795728 conn(0x7f0a7006e670 msgr2=0x7f0a7006ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:27.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.307+0000 7f0a754c3640 1 -- 172.21.15.55:0/908795728 shutdown_connections 2024-03-19T01:58:27.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.307+0000 7f0a754c3640 1 -- 172.21.15.55:0/908795728 wait complete. 2024-03-19T01:58:27.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.311+0000 7f0a754c3640 1 Processor -- start 2024-03-19T01:58:27.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.311+0000 7f0a754c3640 1 -- start start 2024-03-19T01:58:27.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.311+0000 7f0a754c3640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a700c3320 0x7f0a700cba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:27.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.311+0000 7f0a754c3640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0a700ca350 0x7f0a70070410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:27.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.311+0000 7f0a754c3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a70150f50 0x7f0a701f1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:27.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.311+0000 7f0a754c3640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7006b420 con 0x7f0a70150f50 2024-03-19T01:58:27.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.311+0000 7f0a754c3640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7006b2a0 con 0x7f0a700c3320 2024-03-19T01:58:27.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.311+0000 7f0a754c3640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f0a7006b5a0 con 0x7f0a700ca350 2024-03-19T01:58:27.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.311+0000 7f0a6f7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a70150f50 0x7f0a701f1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:27.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.311+0000 7f0a6e7fc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0a700ca350 0x7f0a70070410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:27.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.311+0000 7f0a6f7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a70150f50 0x7f0a701f1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34324/0 (socket says 172.21.15.55:34324) 2024-03-19T01:58:27.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.311+0000 7f0a6f7fe640 1 -- 172.21.15.55:0/2907046772 learned_addr learned my addr 172.21.15.55:0/2907046772 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:27.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.311+0000 7f0a6effd640 1 --2- 172.21.15.55:0/2907046772 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a700c3320 0x7f0a700cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:27.313 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.315+0000 7f0a6f7fe640 1 -- 172.21.15.55:0/2907046772 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0a700ca350 msgr2=0x7f0a70070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:27.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.315+0000 7f0a6f7fe640 1 --2- 172.21.15.55:0/2907046772 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0a700ca350 0x7f0a70070410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:27.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.315+0000 7f0a6f7fe640 1 -- 172.21.15.55:0/2907046772 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a700c3320 msgr2=0x7f0a700cba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:27.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.315+0000 7f0a6f7fe640 1 --2- 172.21.15.55:0/2907046772 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a700c3320 0x7f0a700cba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:27.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.315+0000 7f0a6f7fe640 1 -- 172.21.15.55:0/2907046772 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a70150740 con 0x7f0a70150f50 2024-03-19T01:58:27.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.319+0000 7f0a6f7fe640 1 --2- 172.21.15.55:0/2907046772 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a70150f50 0x7f0a701f1f20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0a680b1040 tx=0x7f0a6801b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:27.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.323+0000 7f0a577fe640 1 -- 172.21.15.55:0/2907046772 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a680be020 con 0x7f0a70150f50 2024-03-19T01:58:27.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.323+0000 7f0a577fe640 1 -- 172.21.15.55:0/2907046772 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a68009800 con 0x7f0a70150f50 2024-03-19T01:58:27.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.323+0000 7f0a577fe640 1 -- 172.21.15.55:0/2907046772 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a68009bf0 con 0x7f0a70150f50 2024-03-19T01:58:27.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.323+0000 7f0a754c3640 1 -- 172.21.15.55:0/2907046772 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a700cc2c0 con 0x7f0a70150f50 2024-03-19T01:58:27.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.323+0000 7f0a754c3640 1 -- 172.21.15.55:0/2907046772 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a7014fc60 con 0x7f0a70150f50 2024-03-19T01:58:27.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.323+0000 7f0a754c3640 1 -- 172.21.15.55:0/2907046772 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a70068a40 con 0x7f0a70150f50 2024-03-19T01:58:27.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.595+0000 7f0a577fe640 1 -- 172.21.15.55:0/2907046772 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f0a680c1020 con 0x7f0a70150f50 2024-03-19T01:58:27.593 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.595+0000 7f0a577fe640 1 --2- 172.21.15.55:0/2907046772 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0a5c0676e0 0x7f0a5c069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:27.593 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.595+0000 7f0a6effd640 1 --2- 172.21.15.55:0/2907046772 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0a5c0676e0 0x7f0a5c069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:27.595 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.599+0000 7f0a6effd640 1 --2- 172.21.15.55:0/2907046772 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0a5c0676e0 0x7f0a5c069ba0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0a60001770 tx=0x7f0a60009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:27.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.619+0000 7f0a577fe640 1 -- 172.21.15.55:0/2907046772 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f0a680f13a0 con 0x7f0a70150f50 2024-03-19T01:58:27.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.619+0000 7f0a577fe640 1 -- 172.21.15.55:0/2907046772 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f0a680c8030 con 0x7f0a70150f50 2024-03-19T01:58:27.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.819+0000 7f0a754c3640 1 -- 172.21.15.55:0/2907046772 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a700c7600 con 0x7f0a70150f50 2024-03-19T01:58:27.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.843+0000 7f0a577fe640 1 -- 172.21.15.55:0/2907046772 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f0a68018820 con 0x7f0a70150f50 2024-03-19T01:58:27.840 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:27.840 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:27.840 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:27.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.847+0000 7f0a754c3640 1 -- 172.21.15.55:0/2907046772 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0a5c0676e0 msgr2=0x7f0a5c069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:27.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.847+0000 7f0a754c3640 1 --2- 172.21.15.55:0/2907046772 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0a5c0676e0 0x7f0a5c069ba0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0a60001770 tx=0x7f0a60009040 comp rx=0 tx=0).stop 2024-03-19T01:58:27.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.847+0000 7f0a754c3640 1 -- 172.21.15.55:0/2907046772 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a70150f50 msgr2=0x7f0a701f1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:27.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.847+0000 7f0a754c3640 1 --2- 172.21.15.55:0/2907046772 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a70150f50 0x7f0a701f1f20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0a680b1040 tx=0x7f0a6801b9d0 comp rx=0 tx=0).stop 2024-03-19T01:58:27.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.847+0000 7f0a754c3640 1 -- 172.21.15.55:0/2907046772 shutdown_connections 2024-03-19T01:58:27.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.847+0000 7f0a754c3640 1 --2- 172.21.15.55:0/2907046772 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0a5c0676e0 0x7f0a5c069ba0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:27.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.847+0000 7f0a754c3640 1 --2- 172.21.15.55:0/2907046772 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0a70150f50 0x7f0a701f1f20 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:27.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.847+0000 7f0a754c3640 1 --2- 172.21.15.55:0/2907046772 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0a700ca350 0x7f0a70070410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:27.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.847+0000 7f0a754c3640 1 --2- 172.21.15.55:0/2907046772 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a700c3320 0x7f0a700cba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:27.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.847+0000 7f0a754c3640 1 -- 172.21.15.55:0/2907046772 >> 172.21.15.55:0/2907046772 conn(0x7f0a7006e670 msgr2=0x7f0a7006cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:27.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.847+0000 7f0a754c3640 1 -- 172.21.15.55:0/2907046772 shutdown_connections 2024-03-19T01:58:27.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:27.847+0000 7f0a754c3640 1 -- 172.21.15.55:0/2907046772 wait complete. 2024-03-19T01:58:28.467 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:28.467 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:29.468 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:29.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.583+0000 7f0c5860a640 1 Processor -- start 2024-03-19T01:58:29.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.583+0000 7f0c5860a640 1 -- start start 2024-03-19T01:58:29.582 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.583+0000 7f0c5860a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f0c50154650 0x7f0c5015ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:29.582 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.583+0000 7f0c5860a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0c5015e370 0x7f0c500c9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:29.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.583+0000 7f0c5860a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c50073790 con 0x7f0c50154270 2024-03-19T01:58:29.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.583+0000 7f0c5860a640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f0c5015d7d0 con 0x7f0c50154650 2024-03-19T01:58:29.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.583+0000 7f0c5860a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f0c5015b090 con 0x7f0c5015e370 2024-03-19T01:58:29.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.587+0000 7f0c56ba7640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0c5015e370 0x7f0c500c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:29.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.587+0000 7f0c563a6640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f0c50154270 0x7f0c5015dc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:53690/0 (socket says 172.21.15.55:53690) 2024-03-19T01:58:29.584 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.587+0000 7f0c563a6640 1 -- 172.21.15.55:0/1392162181 learned_addr learned my addr 172.21.15.55:0/1392162181 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:58:29.584 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.587+0000 7f0c55ba5640 1 --2- 172.21.15.55:0/1392162181 >> v2:172.21.15.84:3300/0 conn(0x7f0c50154650 0x7f0c5015ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:29.586 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.587+0000 7f0c553a4640 1 -- 172.21.15.55:0/1392162181 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2573520256 0 0) 0x7f0c50073790 con 0x7f0c50154270 2024-03-19T01:58:29.586 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.587+0000 7f0c553a4640 1 -- 172.21.15.55:0/1392162181 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c3c003530 con 0x7f0c50154270 2024-03-19T01:58:29.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.591+0000 7f0c56ba7640 1 -- 172.21.15.55:0/1392162181 >> v2:172.21.15.84:3300/0 conn(0x7f0c50154650 msgr2=0x7f0c5015ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:29.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.591+0000 7f0c56ba7640 1 --2- 172.21.15.55:0/1392162181 >> v2:172.21.15.84:3300/0 conn(0x7f0c50154650 0x7f0c5015ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:29.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.591+0000 7f0c56ba7640 1 -- 172.21.15.55:0/1392162181 >> v1:172.21.15.55:6789/0 conn(0x7f0c50154270 legacy=0x7f0c5015dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:29.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.591+0000 7f0c56ba7640 1 -- 172.21.15.55:0/1392162181 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c50073370 con 0x7f0c5015e370 2024-03-19T01:58:29.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.595+0000 7f0c56ba7640 1 --2- 172.21.15.55:0/1392162181 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0c5015e370 0x7f0c500c9150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0c48009d90 tx=0x7f0c4801f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec9f290ad8064308 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:29.597 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.599+0000 7f0c553a4640 1 -- 172.21.15.55:0/1392162181 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c480b4070 con 0x7f0c5015e370 2024-03-19T01:58:29.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.603+0000 7f0c553a4640 1 -- 172.21.15.55:0/1392162181 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c480af600 con 0x7f0c5015e370 2024-03-19T01:58:29.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.603+0000 7f0c553a4640 1 -- 172.21.15.55:0/1392162181 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c480af9b0 con 0x7f0c5015e370 2024-03-19T01:58:29.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.607+0000 7f0c5860a640 1 -- 172.21.15.55:0/1392162181 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0c5015e370 msgr2=0x7f0c500c9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:29.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.607+0000 7f0c5860a640 1 --2- 172.21.15.55:0/1392162181 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0c5015e370 0x7f0c500c9150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0c48009d90 tx=0x7f0c4801f150 comp rx=0 tx=0).stop 2024-03-19T01:58:29.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.607+0000 7f0c5860a640 1 -- 172.21.15.55:0/1392162181 shutdown_connections 2024-03-19T01:58:29.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.607+0000 7f0c5860a640 1 --2- 172.21.15.55:0/1392162181 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0c5015e370 0x7f0c500c9150 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:29.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.607+0000 7f0c5860a640 1 --2- 172.21.15.55:0/1392162181 >> v2:172.21.15.84:3300/0 conn(0x7f0c50154650 0x7f0c5015ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:29.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.607+0000 7f0c5860a640 1 -- 172.21.15.55:0/1392162181 >> 172.21.15.55:0/1392162181 conn(0x7f0c500cc9c0 msgr2=0x7f0c500ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:29.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.607+0000 7f0c5860a640 1 -- 172.21.15.55:0/1392162181 shutdown_connections 2024-03-19T01:58:29.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.607+0000 7f0c5860a640 1 -- 172.21.15.55:0/1392162181 wait complete. 2024-03-19T01:58:29.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.607+0000 7f0c5860a640 1 Processor -- start 2024-03-19T01:58:29.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.607+0000 7f0c5860a640 1 -- start start 2024-03-19T01:58:29.605 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.607+0000 7f0c5860a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0c50154270 0x7f0c500c6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:29.605 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.607+0000 7f0c5860a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0c5015e370 0x7f0c501463a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:29.605 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.607+0000 7f0c5860a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c500c94e0 0x7f0c501468e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:29.605 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.607+0000 7f0c5860a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0c50070ea0 con 0x7f0c500c94e0 2024-03-19T01:58:29.605 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.607+0000 7f0c5860a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f0c50070bc0 con 0x7f0c50154270 2024-03-19T01:58:29.605 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.607+0000 7f0c5860a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f0c50072d50 con 0x7f0c5015e370 2024-03-19T01:58:29.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.607+0000 7f0c56ba7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c500c94e0 0x7f0c501468e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:29.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.607+0000 7f0c56ba7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c500c94e0 0x7f0c501468e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:36736/0 (socket says 172.21.15.55:36736) 2024-03-19T01:58:29.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.607+0000 7f0c56ba7640 1 -- 172.21.15.55:0/22734600 learned_addr learned my addr 172.21.15.55:0/22734600 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:29.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.607+0000 7f0c563a6640 1 --2- 172.21.15.55:0/22734600 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0c50154270 0x7f0c500c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:29.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.611+0000 7f0c55ba5640 1 --2- 172.21.15.55:0/22734600 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0c5015e370 0x7f0c501463a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:29.610 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.611+0000 7f0c56ba7640 1 -- 172.21.15.55:0/22734600 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0c5015e370 msgr2=0x7f0c501463a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:29.610 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.611+0000 7f0c56ba7640 1 --2- 172.21.15.55:0/22734600 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0c5015e370 0x7f0c501463a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:29.610 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.611+0000 7f0c56ba7640 1 -- 172.21.15.55:0/22734600 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0c50154270 msgr2=0x7f0c500c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:29.610 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.611+0000 7f0c56ba7640 1 --2- 172.21.15.55:0/22734600 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0c50154270 0x7f0c500c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:29.610 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.611+0000 7f0c56ba7640 1 -- 172.21.15.55:0/22734600 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c501efae0 con 0x7f0c500c94e0 2024-03-19T01:58:29.612 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.615+0000 7f0c56ba7640 1 --2- 172.21.15.55:0/22734600 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c500c94e0 0x7f0c501468e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0c48009d90 tx=0x7f0c4801bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:29.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.619+0000 7f0c46ffd640 1 -- 172.21.15.55:0/22734600 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c480b4040 con 0x7f0c500c94e0 2024-03-19T01:58:29.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.619+0000 7f0c46ffd640 1 -- 172.21.15.55:0/22734600 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c480bf070 con 0x7f0c500c94e0 2024-03-19T01:58:29.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.619+0000 7f0c5860a640 1 -- 172.21.15.55:0/22734600 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c501efd80 con 0x7f0c500c94e0 2024-03-19T01:58:29.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.619+0000 7f0c46ffd640 1 -- 172.21.15.55:0/22734600 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c48009450 con 0x7f0c500c94e0 2024-03-19T01:58:29.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.619+0000 7f0c5860a640 1 -- 172.21.15.55:0/22734600 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c501f0340 con 0x7f0c500c94e0 2024-03-19T01:58:29.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.619+0000 7f0c5860a640 1 -- 172.21.15.55:0/22734600 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c50068330 con 0x7f0c500c94e0 2024-03-19T01:58:29.916 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.919+0000 7f0c46ffd640 1 -- 172.21.15.55:0/22734600 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f0c480af9d0 con 0x7f0c500c94e0 2024-03-19T01:58:29.917 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.919+0000 7f0c46ffd640 1 --2- 172.21.15.55:0/22734600 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0c340676e0 0x7f0c34069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:29.917 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.919+0000 7f0c563a6640 1 --2- 172.21.15.55:0/22734600 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0c340676e0 0x7f0c34069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:29.918 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.919+0000 7f0c563a6640 1 --2- 172.21.15.55:0/22734600 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0c340676e0 0x7f0c34069ba0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c007fd0 tx=0x7f0c4c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:29.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.955+0000 7f0c46ffd640 1 -- 172.21.15.55:0/22734600 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f0c480f2a50 con 0x7f0c500c94e0 2024-03-19T01:58:29.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:29.959+0000 7f0c46ffd640 1 -- 172.21.15.55:0/22734600 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f0c48018cb0 con 0x7f0c500c94e0 2024-03-19T01:58:30.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:30.171+0000 7f0c5860a640 1 -- 172.21.15.55:0/22734600 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c50154b60 con 0x7f0c500c94e0 2024-03-19T01:58:30.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:30.195+0000 7f0c46ffd640 1 -- 172.21.15.55:0/22734600 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f0c480f49b0 con 0x7f0c500c94e0 2024-03-19T01:58:30.191 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:30.191 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:30.191 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:30.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:30.195+0000 7f0c5860a640 1 -- 172.21.15.55:0/22734600 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0c340676e0 msgr2=0x7f0c34069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:30.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:30.195+0000 7f0c5860a640 1 --2- 172.21.15.55:0/22734600 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0c340676e0 0x7f0c34069ba0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c007fd0 tx=0x7f0c4c009040 comp rx=0 tx=0).stop 2024-03-19T01:58:30.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:30.195+0000 7f0c5860a640 1 -- 172.21.15.55:0/22734600 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c500c94e0 msgr2=0x7f0c501468e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:30.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:30.195+0000 7f0c5860a640 1 --2- 172.21.15.55:0/22734600 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c500c94e0 0x7f0c501468e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0c48009d90 tx=0x7f0c4801bef0 comp rx=0 tx=0).stop 2024-03-19T01:58:30.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:30.195+0000 7f0c5860a640 1 -- 172.21.15.55:0/22734600 shutdown_connections 2024-03-19T01:58:30.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:30.195+0000 7f0c5860a640 1 --2- 172.21.15.55:0/22734600 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0c340676e0 0x7f0c34069ba0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:30.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:30.195+0000 7f0c5860a640 1 --2- 172.21.15.55:0/22734600 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c500c94e0 0x7f0c501468e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:30.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:30.195+0000 7f0c5860a640 1 --2- 172.21.15.55:0/22734600 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0c5015e370 0x7f0c501463a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:30.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:30.195+0000 7f0c5860a640 1 --2- 172.21.15.55:0/22734600 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0c50154270 0x7f0c500c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:30.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:30.195+0000 7f0c5860a640 1 -- 172.21.15.55:0/22734600 >> 172.21.15.55:0/22734600 conn(0x7f0c500cc9c0 msgr2=0x7f0c5014a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:30.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:30.195+0000 7f0c5860a640 1 -- 172.21.15.55:0/22734600 shutdown_connections 2024-03-19T01:58:30.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:30.195+0000 7f0c5860a640 1 -- 172.21.15.55:0/22734600 wait complete. 2024-03-19T01:58:30.815 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:30.815 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:31.816 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:31.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.935+0000 7f1ccc5ab640 1 Processor -- start 2024-03-19T01:58:31.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.935+0000 7f1ccc5ab640 1 -- start start 2024-03-19T01:58:31.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.939+0000 7f1ccc5ab640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1cc40c96b0 0x7f1cc4070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:31.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.939+0000 7f1ccc5ab640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1cc40c9a90 0x7f1cc40c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:31.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.939+0000 7f1ccc5ab640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f1cc41540c0 0x7f1cc4154570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:31.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.939+0000 7f1ccc5ab640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f1cc415e3a0 con 0x7f1cc40c96b0 2024-03-19T01:58:31.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.939+0000 7f1ccc5ab640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f1cc415e680 con 0x7f1cc41540c0 2024-03-19T01:58:31.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.939+0000 7f1ccc5ab640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1cc415a330 con 0x7f1cc40c9a90 2024-03-19T01:58:31.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.939+0000 7f1cca347640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1cc40c96b0 0x7f1cc4070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:31.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.939+0000 7f1ccab48640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f1cc41540c0 0x7f1cc4154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:31.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.939+0000 7f1cca347640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1cc40c96b0 0x7f1cc4070f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:36744/0 (socket says 172.21.15.55:36744) 2024-03-19T01:58:31.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.939+0000 7f1cca347640 1 -- 172.21.15.55:0/2870896274 learned_addr learned my addr 172.21.15.55:0/2870896274 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:31.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.939+0000 7f1cc9b46640 1 --2- 172.21.15.55:0/2870896274 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1cc40c9a90 0x7f1cc40c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:31.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.943+0000 7f1cca347640 1 -- 172.21.15.55:0/2870896274 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1cc40c9a90 msgr2=0x7f1cc40c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:31.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.943+0000 7f1cca347640 1 --2- 172.21.15.55:0/2870896274 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1cc40c9a90 0x7f1cc40c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:31.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.943+0000 7f1cca347640 1 -- 172.21.15.55:0/2870896274 >> v2:172.21.15.84:3300/0 conn(0x7f1cc41540c0 msgr2=0x7f1cc4154570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:31.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.943+0000 7f1cca347640 1 --2- 172.21.15.55:0/2870896274 >> v2:172.21.15.84:3300/0 conn(0x7f1cc41540c0 0x7f1cc4154570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:31.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.943+0000 7f1cca347640 1 -- 172.21.15.55:0/2870896274 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc415a610 con 0x7f1cc40c96b0 2024-03-19T01:58:31.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.943+0000 7f1cca347640 1 --2- 172.21.15.55:0/2870896274 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1cc40c96b0 0x7f1cc4070f20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1cc40ca870 tx=0x7f1cc001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a2d5ac6e691c7ae server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:31.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.947+0000 7f1cc9345640 1 -- 172.21.15.55:0/2870896274 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc00b0070 con 0x7f1cc40c96b0 2024-03-19T01:58:31.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.947+0000 7f1cc9345640 1 -- 172.21.15.55:0/2870896274 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cc00ab600 con 0x7f1cc40c96b0 2024-03-19T01:58:31.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.951+0000 7f1cc9345640 1 -- 172.21.15.55:0/2870896274 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc00ab9d0 con 0x7f1cc40c96b0 2024-03-19T01:58:31.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.951+0000 7f1ccc5ab640 1 -- 172.21.15.55:0/2870896274 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1cc40c96b0 msgr2=0x7f1cc4070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:31.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.951+0000 7f1ccc5ab640 1 --2- 172.21.15.55:0/2870896274 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1cc40c96b0 0x7f1cc4070f20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1cc40ca870 tx=0x7f1cc001c490 comp rx=0 tx=0).stop 2024-03-19T01:58:31.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.951+0000 7f1ccc5ab640 1 -- 172.21.15.55:0/2870896274 shutdown_connections 2024-03-19T01:58:31.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.951+0000 7f1ccc5ab640 1 --2- 172.21.15.55:0/2870896274 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1cc40c9a90 0x7f1cc40c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:31.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.951+0000 7f1ccc5ab640 1 --2- 172.21.15.55:0/2870896274 >> v2:172.21.15.84:3300/0 conn(0x7f1cc41540c0 0x7f1cc4154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:31.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.951+0000 7f1ccc5ab640 1 --2- 172.21.15.55:0/2870896274 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1cc40c96b0 0x7f1cc4070f20 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:31.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.951+0000 7f1ccc5ab640 1 -- 172.21.15.55:0/2870896274 >> 172.21.15.55:0/2870896274 conn(0x7f1cc40cc9c0 msgr2=0x7f1cc40ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:31.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.951+0000 7f1ccc5ab640 1 -- 172.21.15.55:0/2870896274 shutdown_connections 2024-03-19T01:58:31.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.951+0000 7f1ccc5ab640 1 -- 172.21.15.55:0/2870896274 wait complete. 2024-03-19T01:58:31.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.951+0000 7f1ccc5ab640 1 Processor -- start 2024-03-19T01:58:31.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.951+0000 7f1ccc5ab640 1 -- start start 2024-03-19T01:58:31.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.951+0000 7f1ccc5ab640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1cc40c96b0 0x7f1cc41490a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:31.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.951+0000 7f1ccc5ab640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1cc40c9a90 0x7f1cc4152d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:31.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.951+0000 7f1ccc5ab640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1cc41540c0 0x7f1cc41f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:31.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.951+0000 7f1ccc5ab640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc40c7a80 con 0x7f1cc40c96b0 2024-03-19T01:58:31.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.951+0000 7f1ccc5ab640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc40c7900 con 0x7f1cc41540c0 2024-03-19T01:58:31.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.951+0000 7f1ccc5ab640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1cc406b2a0 con 0x7f1cc40c9a90 2024-03-19T01:58:31.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.955+0000 7f1cca347640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1cc40c96b0 0x7f1cc41490a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:31.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.955+0000 7f1cca347640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1cc40c96b0 0x7f1cc41490a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:36760/0 (socket says 172.21.15.55:36760) 2024-03-19T01:58:31.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.955+0000 7f1cca347640 1 -- 172.21.15.55:0/3276664658 learned_addr learned my addr 172.21.15.55:0/3276664658 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:31.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.955+0000 7f1ccab48640 1 --2- 172.21.15.55:0/3276664658 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1cc41540c0 0x7f1cc41f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:31.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.955+0000 7f1cc9b46640 1 --2- 172.21.15.55:0/3276664658 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1cc40c9a90 0x7f1cc4152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:31.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.959+0000 7f1cca347640 1 -- 172.21.15.55:0/3276664658 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1cc40c9a90 msgr2=0x7f1cc4152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:31.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.959+0000 7f1cca347640 1 --2- 172.21.15.55:0/3276664658 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1cc40c9a90 0x7f1cc4152d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:31.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.959+0000 7f1cca347640 1 -- 172.21.15.55:0/3276664658 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1cc41540c0 msgr2=0x7f1cc41f2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:31.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.959+0000 7f1cca347640 1 --2- 172.21.15.55:0/3276664658 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1cc41540c0 0x7f1cc41f2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:31.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.959+0000 7f1cca347640 1 -- 172.21.15.55:0/3276664658 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc4149810 con 0x7f1cc40c96b0 2024-03-19T01:58:31.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.959+0000 7f1cca347640 1 --2- 172.21.15.55:0/3276664658 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1cc40c96b0 0x7f1cc41490a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1cc001cd30 tx=0x7f1cc001acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:31.961 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.963+0000 7f1cbaffd640 1 -- 172.21.15.55:0/3276664658 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc00ba020 con 0x7f1cc40c96b0 2024-03-19T01:58:31.961 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.963+0000 7f1cbaffd640 1 -- 172.21.15.55:0/3276664658 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cc00abe10 con 0x7f1cc40c96b0 2024-03-19T01:58:31.961 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.963+0000 7f1ccc5ab640 1 -- 172.21.15.55:0/3276664658 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cc4152b20 con 0x7f1cc40c96b0 2024-03-19T01:58:31.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.963+0000 7f1cbaffd640 1 -- 172.21.15.55:0/3276664658 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc00b0040 con 0x7f1cc40c96b0 2024-03-19T01:58:31.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.963+0000 7f1ccc5ab640 1 -- 172.21.15.55:0/3276664658 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cc41539b0 con 0x7f1cc40c96b0 2024-03-19T01:58:31.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:31.967+0000 7f1ccc5ab640 1 -- 172.21.15.55:0/3276664658 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cc4068a40 con 0x7f1cc40c96b0 2024-03-19T01:58:32.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:32.239+0000 7f1cbaffd640 1 -- 172.21.15.55:0/3276664658 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1cc00bd020 con 0x7f1cc40c96b0 2024-03-19T01:58:32.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:32.239+0000 7f1cbaffd640 1 --2- 172.21.15.55:0/3276664658 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ca8067690 0x7f1ca8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:32.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:32.239+0000 7f1cc9b46640 1 --2- 172.21.15.55:0/3276664658 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ca8067690 0x7f1ca8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:32.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:32.239+0000 7f1cc9b46640 1 --2- 172.21.15.55:0/3276664658 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ca8067690 0x7f1ca8069b50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4151ae0 tx=0x7f1cb4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:32.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:32.255+0000 7f1cbaffd640 1 -- 172.21.15.55:0/3276664658 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f1cc00ee780 con 0x7f1cc40c96b0 2024-03-19T01:58:32.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:32.255+0000 7f1cbaffd640 1 -- 172.21.15.55:0/3276664658 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1cc0016110 con 0x7f1cc40c96b0 2024-03-19T01:58:32.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:32.447+0000 7f1ccc5ab640 1 -- 172.21.15.55:0/3276664658 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1cc4072e70 con 0x7f1cc40c96b0 2024-03-19T01:58:32.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:32.483+0000 7f1cbaffd640 1 -- 172.21.15.55:0/3276664658 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f1cc00f06e0 con 0x7f1cc40c96b0 2024-03-19T01:58:32.479 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:32.479 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:32.480 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:32.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:32.483+0000 7f1ccc5ab640 1 -- 172.21.15.55:0/3276664658 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ca8067690 msgr2=0x7f1ca8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:32.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:32.483+0000 7f1ccc5ab640 1 --2- 172.21.15.55:0/3276664658 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ca8067690 0x7f1ca8069b50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4151ae0 tx=0x7f1cb4009190 comp rx=0 tx=0).stop 2024-03-19T01:58:32.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:32.483+0000 7f1ccc5ab640 1 -- 172.21.15.55:0/3276664658 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1cc40c96b0 msgr2=0x7f1cc41490a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:32.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:32.483+0000 7f1ccc5ab640 1 --2- 172.21.15.55:0/3276664658 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1cc40c96b0 0x7f1cc41490a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1cc001cd30 tx=0x7f1cc001acd0 comp rx=0 tx=0).stop 2024-03-19T01:58:32.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:32.483+0000 7f1ccc5ab640 1 -- 172.21.15.55:0/3276664658 shutdown_connections 2024-03-19T01:58:32.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:32.483+0000 7f1ccc5ab640 1 --2- 172.21.15.55:0/3276664658 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ca8067690 0x7f1ca8069b50 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:32.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:32.483+0000 7f1ccc5ab640 1 --2- 172.21.15.55:0/3276664658 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1cc41540c0 0x7f1cc41f2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:32.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:32.483+0000 7f1ccc5ab640 1 --2- 172.21.15.55:0/3276664658 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1cc40c9a90 0x7f1cc4152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:32.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:32.483+0000 7f1ccc5ab640 1 --2- 172.21.15.55:0/3276664658 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1cc40c96b0 0x7f1cc41490a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:32.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:32.483+0000 7f1ccc5ab640 1 -- 172.21.15.55:0/3276664658 >> 172.21.15.55:0/3276664658 conn(0x7f1cc40cc9c0 msgr2=0x7f1cc4070a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:32.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:32.483+0000 7f1ccc5ab640 1 -- 172.21.15.55:0/3276664658 shutdown_connections 2024-03-19T01:58:32.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:32.483+0000 7f1ccc5ab640 1 -- 172.21.15.55:0/3276664658 wait complete. 2024-03-19T01:58:33.101 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:33.102 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:34.103 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:34.229 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.231+0000 7f96b0cf1640 1 Processor -- start 2024-03-19T01:58:34.229 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.231+0000 7f96b0cf1640 1 -- start start 2024-03-19T01:58:34.230 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.231+0000 7f96b0cf1640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f96ac1510f0 0x7f96ac0c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:34.230 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.231+0000 7f96b0cf1640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f96ac15a410 0x7f96ac0708a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:34.230 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.231+0000 7f96b0cf1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96ac0c3f60 con 0x7f96ac0c73c0 2024-03-19T01:58:34.230 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.231+0000 7f96b0cf1640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f96ac1514d0 con 0x7f96ac1510f0 2024-03-19T01:58:34.230 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.231+0000 7f96b0cf1640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f96ac0c6870 con 0x7f96ac15a410 2024-03-19T01:58:34.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.235+0000 7f96aa59c640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f96ac1510f0 0x7f96ac0c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:34.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.235+0000 7f96a9d9b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f96ac15a410 0x7f96ac0708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:34.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.235+0000 7f96aa59c640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f96ac1510f0 0x7f96ac0c61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:49666/0 (socket says 172.21.15.55:49666) 2024-03-19T01:58:34.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.235+0000 7f96aa59c640 1 -- 172.21.15.55:0/3763206658 learned_addr learned my addr 172.21.15.55:0/3763206658 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:34.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.239+0000 7f96aa59c640 1 -- 172.21.15.55:0/3763206658 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f96ac15a410 msgr2=0x7f96ac0708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:34.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.239+0000 7f96aa59c640 1 --2- 172.21.15.55:0/3763206658 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f96ac15a410 0x7f96ac0708a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:34.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.239+0000 7f96aa59c640 1 -- 172.21.15.55:0/3763206658 >> v1:172.21.15.84:6789/0 conn(0x7f96ac0c73c0 legacy=0x7f96ac072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:34.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.239+0000 7f96aa59c640 1 -- 172.21.15.55:0/3763206658 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96ac0c6b50 con 0x7f96ac1510f0 2024-03-19T01:58:34.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.243+0000 7f96aa59c640 1 --2- 172.21.15.55:0/3763206658 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96ac1510f0 0x7f96ac0c61c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f969c009870 tx=0x7f969c01ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8aff2affc0c58679 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:34.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.251+0000 7f96a959a640 1 -- 172.21.15.55:0/3763206658 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969c0b5070 con 0x7f96ac1510f0 2024-03-19T01:58:34.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.251+0000 7f96a959a640 1 -- 172.21.15.55:0/3763206658 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f969c0a8a50 con 0x7f96ac1510f0 2024-03-19T01:58:34.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.251+0000 7f96a959a640 1 -- 172.21.15.55:0/3763206658 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969c0a8e20 con 0x7f96ac1510f0 2024-03-19T01:58:34.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.251+0000 7f96b0cf1640 1 -- 172.21.15.55:0/3763206658 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96ac1510f0 msgr2=0x7f96ac0c61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:34.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.251+0000 7f96b0cf1640 1 --2- 172.21.15.55:0/3763206658 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96ac1510f0 0x7f96ac0c61c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f969c009870 tx=0x7f969c01ed50 comp rx=0 tx=0).stop 2024-03-19T01:58:34.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.251+0000 7f96b0cf1640 1 -- 172.21.15.55:0/3763206658 shutdown_connections 2024-03-19T01:58:34.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.251+0000 7f96b0cf1640 1 --2- 172.21.15.55:0/3763206658 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f96ac15a410 0x7f96ac0708a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:34.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.251+0000 7f96b0cf1640 1 --2- 172.21.15.55:0/3763206658 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96ac1510f0 0x7f96ac0c61c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:34.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.251+0000 7f96b0cf1640 1 -- 172.21.15.55:0/3763206658 >> 172.21.15.55:0/3763206658 conn(0x7f96ac0cc9c0 msgr2=0x7f96ac0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:34.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.251+0000 7f96b0cf1640 1 -- 172.21.15.55:0/3763206658 shutdown_connections 2024-03-19T01:58:34.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.251+0000 7f96b0cf1640 1 -- 172.21.15.55:0/3763206658 wait complete. 2024-03-19T01:58:34.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.251+0000 7f96b0cf1640 1 Processor -- start 2024-03-19T01:58:34.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.251+0000 7f96b0cf1640 1 -- start start 2024-03-19T01:58:34.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.251+0000 7f96b0cf1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96ac0c73c0 0x7f96ac1efdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:34.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.251+0000 7f96b0cf1640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96ac1510f0 0x7f96ac1f3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:34.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.251+0000 7f96b0cf1640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f96ac15a410 0x7f96ac14d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:34.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.251+0000 7f96b0cf1640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f96ac0c6e30 con 0x7f96ac0c73c0 2024-03-19T01:58:34.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.251+0000 7f96b0cf1640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f96ac0c44e0 con 0x7f96ac1510f0 2024-03-19T01:58:34.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.251+0000 7f96b0cf1640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f96ac0c4a00 con 0x7f96ac15a410 2024-03-19T01:58:34.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.251+0000 7f96aa59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96ac0c73c0 0x7f96ac1efdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:34.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.255+0000 7f96a9d9b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96ac1510f0 0x7f96ac1f3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:34.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.255+0000 7f96aa59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96ac0c73c0 0x7f96ac1efdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:36762/0 (socket says 172.21.15.55:36762) 2024-03-19T01:58:34.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.255+0000 7f96aa59c640 1 -- 172.21.15.55:0/1703756771 learned_addr learned my addr 172.21.15.55:0/1703756771 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:34.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.259+0000 7f96aa59c640 1 -- 172.21.15.55:0/1703756771 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f96ac15a410 msgr2=0x7f96ac14d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:34.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.259+0000 7f96aa59c640 1 --2- 172.21.15.55:0/1703756771 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f96ac15a410 0x7f96ac14d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:34.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.259+0000 7f96aa59c640 1 -- 172.21.15.55:0/1703756771 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96ac1510f0 msgr2=0x7f96ac1f3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:34.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.259+0000 7f96aa59c640 1 --2- 172.21.15.55:0/1703756771 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96ac1510f0 0x7f96ac1f3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:34.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.259+0000 7f96aa59c640 1 -- 172.21.15.55:0/1703756771 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96ac1587e0 con 0x7f96ac0c73c0 2024-03-19T01:58:34.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.259+0000 7f96aa59c640 1 --2- 172.21.15.55:0/1703756771 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96ac0c73c0 0x7f96ac1efdc0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f969c01b960 tx=0x7f969c01baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:34.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.267+0000 7f969affd640 1 -- 172.21.15.55:0/1703756771 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969c0b5040 con 0x7f96ac0c73c0 2024-03-19T01:58:34.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.267+0000 7f969affd640 1 -- 172.21.15.55:0/1703756771 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f969c0a8850 con 0x7f96ac0c73c0 2024-03-19T01:58:34.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.267+0000 7f969affd640 1 -- 172.21.15.55:0/1703756771 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969c0b1d20 con 0x7f96ac0c73c0 2024-03-19T01:58:34.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.267+0000 7f96b0cf1640 1 -- 172.21.15.55:0/1703756771 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96ac158a50 con 0x7f96ac0c73c0 2024-03-19T01:58:34.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.267+0000 7f96b0cf1640 1 -- 172.21.15.55:0/1703756771 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96ac14dd90 con 0x7f96ac0c73c0 2024-03-19T01:58:34.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.267+0000 7f96b0cf1640 1 -- 172.21.15.55:0/1703756771 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96ac068a40 con 0x7f96ac0c73c0 2024-03-19T01:58:34.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.291+0000 7f969affd640 1 -- 172.21.15.55:0/1703756771 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f969c0c0030 con 0x7f96ac0c73c0 2024-03-19T01:58:34.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.291+0000 7f969affd640 1 --2- 172.21.15.55:0/1703756771 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f968c067690 0x7f968c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:34.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.291+0000 7f96a9d9b640 1 --2- 172.21.15.55:0/1703756771 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f968c067690 0x7f968c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:34.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.295+0000 7f96a9d9b640 1 --2- 172.21.15.55:0/1703756771 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f968c067690 0x7f968c069b50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f96ac157cf0 tx=0x7f96a0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:34.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.575+0000 7f969affd640 1 -- 172.21.15.55:0/1703756771 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f969c0a8a00 con 0x7f96ac0c73c0 2024-03-19T01:58:34.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.575+0000 7f969affd640 1 -- 172.21.15.55:0/1703756771 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f969c0c7060 con 0x7f96ac0c73c0 2024-03-19T01:58:34.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.799+0000 7f96b0cf1640 1 -- 172.21.15.55:0/1703756771 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96ac074d80 con 0x7f96ac0c73c0 2024-03-19T01:58:34.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.863+0000 7f969affd640 1 -- 172.21.15.55:0/1703756771 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f969c0188b0 con 0x7f96ac0c73c0 2024-03-19T01:58:34.862 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:34.862 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:34.863 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:34.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.867+0000 7f96b0cf1640 1 -- 172.21.15.55:0/1703756771 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f968c067690 msgr2=0x7f968c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:34.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.867+0000 7f96b0cf1640 1 --2- 172.21.15.55:0/1703756771 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f968c067690 0x7f968c069b50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f96ac157cf0 tx=0x7f96a0009190 comp rx=0 tx=0).stop 2024-03-19T01:58:34.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.867+0000 7f96b0cf1640 1 -- 172.21.15.55:0/1703756771 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96ac0c73c0 msgr2=0x7f96ac1efdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:34.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.867+0000 7f96b0cf1640 1 --2- 172.21.15.55:0/1703756771 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96ac0c73c0 0x7f96ac1efdc0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f969c01b960 tx=0x7f969c01baf0 comp rx=0 tx=0).stop 2024-03-19T01:58:34.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.867+0000 7f96b0cf1640 1 -- 172.21.15.55:0/1703756771 shutdown_connections 2024-03-19T01:58:34.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.867+0000 7f96b0cf1640 1 --2- 172.21.15.55:0/1703756771 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f968c067690 0x7f968c069b50 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:34.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.867+0000 7f96b0cf1640 1 --2- 172.21.15.55:0/1703756771 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f96ac15a410 0x7f96ac14d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:34.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.867+0000 7f96b0cf1640 1 --2- 172.21.15.55:0/1703756771 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96ac1510f0 0x7f96ac1f3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:34.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.867+0000 7f96b0cf1640 1 --2- 172.21.15.55:0/1703756771 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96ac0c73c0 0x7f96ac1efdc0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:34.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.867+0000 7f96b0cf1640 1 -- 172.21.15.55:0/1703756771 >> 172.21.15.55:0/1703756771 conn(0x7f96ac0cc9c0 msgr2=0x7f96ac0c7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:34.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.867+0000 7f96b0cf1640 1 -- 172.21.15.55:0/1703756771 shutdown_connections 2024-03-19T01:58:34.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:34.867+0000 7f96b0cf1640 1 -- 172.21.15.55:0/1703756771 wait complete. 2024-03-19T01:58:35.491 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:35.491 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:36.492 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:36.616 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.619+0000 7f83013c3640 1 Processor -- start 2024-03-19T01:58:36.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.619+0000 7f83013c3640 1 -- start start 2024-03-19T01:58:36.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.619+0000 7f83013c3640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f82fc0c7430 0x7f82fc0c5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:36.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.619+0000 7f83013c3640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82fc0c8f70 con 0x7f82fc0c9780 2024-03-19T01:58:36.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.619+0000 7f83013c3640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82fc0c9250 con 0x7f82fc13dd00 2024-03-19T01:58:36.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.619+0000 7f83013c3640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f82fc15bc60 con 0x7f82fc0c7430 2024-03-19T01:58:36.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.619+0000 7f82fa7fc640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f82fc0c9780 0x7f82fc15b550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:53692/0 (socket says 172.21.15.55:53692) 2024-03-19T01:58:36.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.619+0000 7f82fa7fc640 1 -- 172.21.15.55:0/2336074013 learned_addr learned my addr 172.21.15.55:0/2336074013 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:58:36.618 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.619+0000 7f82fb7fe640 1 --2- 172.21.15.55:0/2336074013 >> v2:172.21.15.84:3300/0 conn(0x7f82fc0c7430 0x7f82fc0c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:36.622 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.623+0000 7f82f9ffb640 1 -- 172.21.15.55:0/2336074013 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 888282074 0 0) 0x7f82fc0c8f70 con 0x7f82fc0c9780 2024-03-19T01:58:36.622 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.623+0000 7f82f9ffb640 1 -- 172.21.15.55:0/2336074013 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82e4003530 con 0x7f82fc0c9780 2024-03-19T01:58:36.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.631+0000 7f82f9ffb640 1 -- 172.21.15.55:0/2336074013 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 492510013 0 0) 0x7f82e4003530 con 0x7f82fc0c9780 2024-03-19T01:58:36.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.631+0000 7f82f9ffb640 1 -- 172.21.15.55:0/2336074013 >> v2:172.21.15.84:3300/0 conn(0x7f82fc0c7430 msgr2=0x7f82fc0c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:36.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.631+0000 7f82f9ffb640 1 --2- 172.21.15.55:0/2336074013 >> v2:172.21.15.84:3300/0 conn(0x7f82fc0c7430 0x7f82fc0c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:36.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.631+0000 7f82f9ffb640 1 -- 172.21.15.55:0/2336074013 >> v1:172.21.15.84:6789/0 conn(0x7f82fc13dd00 legacy=0x7f82fc0c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:36.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.631+0000 7f82f9ffb640 1 -- 172.21.15.55:0/2336074013 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82fc1582b0 con 0x7f82fc0c9780 2024-03-19T01:58:36.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.631+0000 7f82f9ffb640 1 -- 172.21.15.55:0/2336074013 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f82f0001680 con 0x7f82fc0c9780 2024-03-19T01:58:36.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.631+0000 7f82f9ffb640 1 -- 172.21.15.55:0/2336074013 >> v1:172.21.15.55:6789/0 conn(0x7f82fc0c9780 legacy=0x7f82fc15b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:36.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.631+0000 7f82f9ffb640 1 --2- 172.21.15.55:0/2336074013 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82e4003a10 0x7f82e4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:36.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.631+0000 7f82f9ffb640 1 --2- 172.21.15.55:0/2336074013 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82e4004810 0x7f82e4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:36.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.631+0000 7f82f9ffb640 1 --2- 172.21.15.55:0/2336074013 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f82e40051d0 0x7f82e400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:36.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.631+0000 7f82f9ffb640 1 -- 172.21.15.55:0/2336074013 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc15bc60 con 0x7f82e4003a10 2024-03-19T01:58:36.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.631+0000 7f82f9ffb640 1 -- 172.21.15.55:0/2336074013 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f82e400c0d0 con 0x7f82e4004810 2024-03-19T01:58:36.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.631+0000 7f82f9ffb640 1 -- 172.21.15.55:0/2336074013 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f82e400c2d0 con 0x7f82e40051d0 2024-03-19T01:58:36.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.631+0000 7f82fb7fe640 1 --2- 172.21.15.55:0/2336074013 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f82e40051d0 0x7f82e400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:36.631 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.631+0000 7f82faffd640 1 --2- 172.21.15.55:0/2336074013 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82e4003a10 0x7f82e4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:36.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.635+0000 7f82fb7fe640 1 -- 172.21.15.55:0/2336074013 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82e4004810 msgr2=0x7f82e4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:36.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.635+0000 7f82fb7fe640 1 --2- 172.21.15.55:0/2336074013 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82e4004810 0x7f82e4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:36.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.635+0000 7f82fb7fe640 1 -- 172.21.15.55:0/2336074013 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82e4003a10 msgr2=0x7f82e4003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:36.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.635+0000 7f82fb7fe640 1 --2- 172.21.15.55:0/2336074013 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82e4003a10 0x7f82e4003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:36.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.635+0000 7f82fb7fe640 1 -- 172.21.15.55:0/2336074013 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e4003530 con 0x7f82e40051d0 2024-03-19T01:58:36.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.639+0000 7f82fb7fe640 1 -- 172.21.15.55:0/2336074013 reap_dead start 2024-03-19T01:58:36.637 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.639+0000 7f82fb7fe640 1 --2- 172.21.15.55:0/2336074013 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f82e40051d0 0x7f82e400b9d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f82f4009fc0 tx=0x7f82f401ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63a95d9a79dccca2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:36.641 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.643+0000 7f82f9ffb640 1 -- 172.21.15.55:0/2336074013 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f4009c10 con 0x7f82e40051d0 2024-03-19T01:58:36.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.647+0000 7f82f9ffb640 1 -- 172.21.15.55:0/2336074013 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82f40a5c00 con 0x7f82e40051d0 2024-03-19T01:58:36.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.647+0000 7f82f9ffb640 1 -- 172.21.15.55:0/2336074013 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f40039a0 con 0x7f82e40051d0 2024-03-19T01:58:36.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.647+0000 7f83013c3640 1 -- 172.21.15.55:0/2336074013 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f82e40051d0 msgr2=0x7f82e400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:36.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.647+0000 7f83013c3640 1 --2- 172.21.15.55:0/2336074013 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f82e40051d0 0x7f82e400b9d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f82f4009fc0 tx=0x7f82f401ccc0 comp rx=0 tx=0).stop 2024-03-19T01:58:36.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.647+0000 7f83013c3640 1 -- 172.21.15.55:0/2336074013 shutdown_connections 2024-03-19T01:58:36.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.647+0000 7f83013c3640 1 --2- 172.21.15.55:0/2336074013 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f82e40051d0 0x7f82e400b9d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:36.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.647+0000 7f83013c3640 1 -- 172.21.15.55:0/2336074013 >> 172.21.15.55:0/2336074013 conn(0x7f82fc06e5a0 msgr2=0x7f82fc072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:36.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.647+0000 7f83013c3640 1 -- 172.21.15.55:0/2336074013 shutdown_connections 2024-03-19T01:58:36.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.647+0000 7f83013c3640 1 -- 172.21.15.55:0/2336074013 wait complete. 2024-03-19T01:58:36.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.647+0000 7f83013c3640 1 Processor -- start 2024-03-19T01:58:36.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.647+0000 7f83013c3640 1 -- start start 2024-03-19T01:58:36.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.651+0000 7f83013c3640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82e40051d0 0x7f82fc13c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:36.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.651+0000 7f83013c3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82fc153840 0x7f82fc1f6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:36.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.651+0000 7f83013c3640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f82fc152850 0x7f82fc139960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:36.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.651+0000 7f83013c3640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc158f10 con 0x7f82fc153840 2024-03-19T01:58:36.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.651+0000 7f83013c3640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc158d90 con 0x7f82e40051d0 2024-03-19T01:58:36.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.651+0000 7f83013c3640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f82fc0c5730 con 0x7f82fc152850 2024-03-19T01:58:36.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.651+0000 7f82fa7fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82fc153840 0x7f82fc1f6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:36.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.651+0000 7f82faffd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82e40051d0 0x7f82fc13c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:36.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.651+0000 7f82fb7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f82fc152850 0x7f82fc139960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:36.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.651+0000 7f82fa7fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82fc153840 0x7f82fc1f6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:36778/0 (socket says 172.21.15.55:36778) 2024-03-19T01:58:36.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.651+0000 7f82fa7fc640 1 -- 172.21.15.55:0/936153150 learned_addr learned my addr 172.21.15.55:0/936153150 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:36.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.655+0000 7f82fa7fc640 1 -- 172.21.15.55:0/936153150 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f82fc152850 msgr2=0x7f82fc139960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:36.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.655+0000 7f82fa7fc640 1 --2- 172.21.15.55:0/936153150 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f82fc152850 0x7f82fc139960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:36.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.655+0000 7f82fa7fc640 1 -- 172.21.15.55:0/936153150 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82e40051d0 msgr2=0x7f82fc13c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:36.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.655+0000 7f82fa7fc640 1 --2- 172.21.15.55:0/936153150 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82e40051d0 0x7f82fc13c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:36.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.655+0000 7f82fa7fc640 1 -- 172.21.15.55:0/936153150 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82fc155090 con 0x7f82fc153840 2024-03-19T01:58:36.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.659+0000 7f82fa7fc640 1 --2- 172.21.15.55:0/936153150 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82fc153840 0x7f82fc1f6450 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f82f000c760 tx=0x7f82f000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:36.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.663+0000 7f82e37fe640 1 -- 172.21.15.55:0/936153150 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f0018070 con 0x7f82fc153840 2024-03-19T01:58:36.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.663+0000 7f82e37fe640 1 -- 172.21.15.55:0/936153150 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82f00072d0 con 0x7f82fc153840 2024-03-19T01:58:36.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.663+0000 7f83013c3640 1 -- 172.21.15.55:0/936153150 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82fc155370 con 0x7f82fc153840 2024-03-19T01:58:36.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.663+0000 7f82e37fe640 1 -- 172.21.15.55:0/936153150 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f00047f0 con 0x7f82fc153840 2024-03-19T01:58:36.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.663+0000 7f83013c3640 1 -- 172.21.15.55:0/936153150 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82fc154520 con 0x7f82fc153840 2024-03-19T01:58:36.661 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.663+0000 7f83013c3640 1 -- 172.21.15.55:0/936153150 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82fc068a40 con 0x7f82fc153840 2024-03-19T01:58:36.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.951+0000 7f82e37fe640 1 -- 172.21.15.55:0/936153150 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f82f001b070 con 0x7f82fc153840 2024-03-19T01:58:36.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.955+0000 7f82e37fe640 1 --2- 172.21.15.55:0/936153150 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f82e80676b0 0x7f82e8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:36.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.955+0000 7f82faffd640 1 --2- 172.21.15.55:0/936153150 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f82e80676b0 0x7f82e8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:36.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.955+0000 7f82faffd640 1 --2- 172.21.15.55:0/936153150 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f82e80676b0 0x7f82e8069b70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f82ec007fd0 tx=0x7f82ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:36.991 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.995+0000 7f82e37fe640 1 -- 172.21.15.55:0/936153150 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f82f0053900 con 0x7f82fc153840 2024-03-19T01:58:36.997 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:36.999+0000 7f82e37fe640 1 -- 172.21.15.55:0/936153150 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f82f0022020 con 0x7f82fc153840 2024-03-19T01:58:37.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:37.251+0000 7f83013c3640 1 -- 172.21.15.55:0/936153150 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82fc0c9560 con 0x7f82fc153840 2024-03-19T01:58:37.269 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:37.271+0000 7f82e37fe640 1 -- 172.21.15.55:0/936153150 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f82f0055860 con 0x7f82fc153840 2024-03-19T01:58:37.269 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:37.269 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:37.270 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:37.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:37.275+0000 7f83013c3640 1 -- 172.21.15.55:0/936153150 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f82e80676b0 msgr2=0x7f82e8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:37.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:37.275+0000 7f83013c3640 1 --2- 172.21.15.55:0/936153150 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f82e80676b0 0x7f82e8069b70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f82ec007fd0 tx=0x7f82ec009040 comp rx=0 tx=0).stop 2024-03-19T01:58:37.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:37.275+0000 7f83013c3640 1 -- 172.21.15.55:0/936153150 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82fc153840 msgr2=0x7f82fc1f6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:37.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:37.275+0000 7f83013c3640 1 --2- 172.21.15.55:0/936153150 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82fc153840 0x7f82fc1f6450 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f82f000c760 tx=0x7f82f000cc20 comp rx=0 tx=0).stop 2024-03-19T01:58:37.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:37.275+0000 7f83013c3640 1 -- 172.21.15.55:0/936153150 shutdown_connections 2024-03-19T01:58:37.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:37.275+0000 7f83013c3640 1 --2- 172.21.15.55:0/936153150 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f82e80676b0 0x7f82e8069b70 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:37.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:37.275+0000 7f83013c3640 1 --2- 172.21.15.55:0/936153150 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f82fc152850 0x7f82fc139960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:37.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:37.275+0000 7f83013c3640 1 --2- 172.21.15.55:0/936153150 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82fc153840 0x7f82fc1f6450 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:37.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:37.275+0000 7f83013c3640 1 --2- 172.21.15.55:0/936153150 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82e40051d0 0x7f82fc13c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:37.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:37.275+0000 7f83013c3640 1 -- 172.21.15.55:0/936153150 >> 172.21.15.55:0/936153150 conn(0x7f82fc06e5a0 msgr2=0x7f82fc13bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:37.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:37.275+0000 7f83013c3640 1 -- 172.21.15.55:0/936153150 shutdown_connections 2024-03-19T01:58:37.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:37.275+0000 7f83013c3640 1 -- 172.21.15.55:0/936153150 wait complete. 2024-03-19T01:58:37.892 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:37.892 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:38.893 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:39.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.019+0000 7f519cd4f640 1 Processor -- start 2024-03-19T01:58:39.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.019+0000 7f519cd4f640 1 -- start start 2024-03-19T01:58:39.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.023+0000 7f519cd4f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51980c96b0 0x7f5198070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:39.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.023+0000 7f519cd4f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f51980c9a90 0x7f51980c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:39.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.023+0000 7f519cd4f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f519815e3a0 con 0x7f51981540c0 2024-03-19T01:58:39.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.023+0000 7f519cd4f640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f519815e680 con 0x7f51980c9a90 2024-03-19T01:58:39.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.023+0000 7f519cd4f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f519815a330 con 0x7f51980c96b0 2024-03-19T01:58:39.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.023+0000 7f5195d9b640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f51980c9a90 0x7f51980c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:39.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.023+0000 7f519659c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51980c96b0 0x7f5198070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:39.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.023+0000 7f519659c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51980c96b0 0x7f5198070f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:50724/0 (socket says 172.21.15.55:50724) 2024-03-19T01:58:39.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.023+0000 7f519659c640 1 -- 172.21.15.55:0/1421304387 learned_addr learned my addr 172.21.15.55:0/1421304387 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:39.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.023+0000 7f5196d9d640 1 --1- 172.21.15.55:0/1421304387 >> v1:172.21.15.84:6789/0 conn(0x7f51981540c0 0x7f519815d810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:37892/0 (socket says 172.21.15.55:37892) 2024-03-19T01:58:39.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.031+0000 7f519559a640 1 -- 172.21.15.55:0/1421304387 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2515751328 0 0) 0x7f519815e3a0 con 0x7f51981540c0 2024-03-19T01:58:39.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.031+0000 7f519559a640 1 -- 172.21.15.55:0/1421304387 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5180003530 con 0x7f51981540c0 2024-03-19T01:58:39.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.031+0000 7f519659c640 1 -- 172.21.15.55:0/1421304387 >> v2:172.21.15.84:3300/0 conn(0x7f51980c9a90 msgr2=0x7f51980c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:39.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.031+0000 7f519659c640 1 --2- 172.21.15.55:0/1421304387 >> v2:172.21.15.84:3300/0 conn(0x7f51980c9a90 0x7f51980c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:39.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.031+0000 7f519659c640 1 -- 172.21.15.55:0/1421304387 >> v1:172.21.15.84:6789/0 conn(0x7f51981540c0 legacy=0x7f519815d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:39.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.031+0000 7f519659c640 1 -- 172.21.15.55:0/1421304387 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f519815a610 con 0x7f51980c96b0 2024-03-19T01:58:39.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.035+0000 7f519659c640 1 --2- 172.21.15.55:0/1421304387 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51980c96b0 0x7f5198070f20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5188009870 tx=0x7f518801c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c77410040bd7117 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:39.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.043+0000 7f519559a640 1 -- 172.21.15.55:0/1421304387 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51880b1070 con 0x7f51980c96b0 2024-03-19T01:58:39.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.043+0000 7f519559a640 1 -- 172.21.15.55:0/1421304387 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51880ac620 con 0x7f51980c96b0 2024-03-19T01:58:39.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.043+0000 7f519559a640 1 -- 172.21.15.55:0/1421304387 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51880ac9f0 con 0x7f51980c96b0 2024-03-19T01:58:39.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.047+0000 7f519cd4f640 1 -- 172.21.15.55:0/1421304387 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51980c96b0 msgr2=0x7f5198070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:39.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.047+0000 7f519cd4f640 1 --2- 172.21.15.55:0/1421304387 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51980c96b0 0x7f5198070f20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5188009870 tx=0x7f518801c4d0 comp rx=0 tx=0).stop 2024-03-19T01:58:39.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.047+0000 7f519cd4f640 1 -- 172.21.15.55:0/1421304387 shutdown_connections 2024-03-19T01:58:39.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.047+0000 7f519cd4f640 1 --2- 172.21.15.55:0/1421304387 >> v2:172.21.15.84:3300/0 conn(0x7f51980c9a90 0x7f51980c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:39.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.047+0000 7f519cd4f640 1 --2- 172.21.15.55:0/1421304387 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51980c96b0 0x7f5198070f20 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:39.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.047+0000 7f519cd4f640 1 -- 172.21.15.55:0/1421304387 >> 172.21.15.55:0/1421304387 conn(0x7f51980cc9c0 msgr2=0x7f51980ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:39.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.047+0000 7f519cd4f640 1 -- 172.21.15.55:0/1421304387 shutdown_connections 2024-03-19T01:58:39.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.047+0000 7f519cd4f640 1 -- 172.21.15.55:0/1421304387 wait complete. 2024-03-19T01:58:39.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.047+0000 7f519cd4f640 1 Processor -- start 2024-03-19T01:58:39.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.047+0000 7f519cd4f640 1 -- start start 2024-03-19T01:58:39.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.047+0000 7f519cd4f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51980c96b0 0x7f51980cbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:39.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.047+0000 7f519cd4f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51980c9a90 0x7f5198154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:39.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.047+0000 7f519cd4f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51981540c0 0x7f51981f44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:39.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.047+0000 7f519cd4f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f51980c7900 con 0x7f51981540c0 2024-03-19T01:58:39.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.047+0000 7f519cd4f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f519815b0f0 con 0x7f51980c96b0 2024-03-19T01:58:39.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.047+0000 7f519cd4f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f51980c7a80 con 0x7f51980c9a90 2024-03-19T01:58:39.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.051+0000 7f5195d9b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51980c9a90 0x7f5198154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:39.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.051+0000 7f5196d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51981540c0 0x7f51981f44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:39.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.051+0000 7f519659c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51980c96b0 0x7f51980cbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:39.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.051+0000 7f5196d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51981540c0 0x7f51981f44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50790/0 (socket says 172.21.15.55:50790) 2024-03-19T01:58:39.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.051+0000 7f5196d9d640 1 -- 172.21.15.55:0/3121324562 learned_addr learned my addr 172.21.15.55:0/3121324562 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:39.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.055+0000 7f5196d9d640 1 -- 172.21.15.55:0/3121324562 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51980c9a90 msgr2=0x7f5198154b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:39.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.055+0000 7f5196d9d640 1 --2- 172.21.15.55:0/3121324562 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51980c9a90 0x7f5198154b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:39.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.055+0000 7f5196d9d640 1 -- 172.21.15.55:0/3121324562 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51980c96b0 msgr2=0x7f51980cbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:39.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.055+0000 7f5196d9d640 1 --2- 172.21.15.55:0/3121324562 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51980c96b0 0x7f51980cbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:39.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.055+0000 7f5196d9d640 1 -- 172.21.15.55:0/3121324562 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5198151bf0 con 0x7f51981540c0 2024-03-19T01:58:39.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.059+0000 7f5196d9d640 1 --2- 172.21.15.55:0/3121324562 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51981540c0 0x7f51981f44d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f519000c770 tx=0x7f519000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:39.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.063+0000 7f5186ffd640 1 -- 172.21.15.55:0/3121324562 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5190017070 con 0x7f51981540c0 2024-03-19T01:58:39.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.063+0000 7f5186ffd640 1 -- 172.21.15.55:0/3121324562 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f519000a640 con 0x7f51981540c0 2024-03-19T01:58:39.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.063+0000 7f519cd4f640 1 -- 172.21.15.55:0/3121324562 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5198151ed0 con 0x7f51981540c0 2024-03-19T01:58:39.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.063+0000 7f5186ffd640 1 -- 172.21.15.55:0/3121324562 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519000aa30 con 0x7f51981540c0 2024-03-19T01:58:39.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.063+0000 7f519cd4f640 1 -- 172.21.15.55:0/3121324562 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5198152490 con 0x7f51981540c0 2024-03-19T01:58:39.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.067+0000 7f519cd4f640 1 -- 172.21.15.55:0/3121324562 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5198068a40 con 0x7f51981540c0 2024-03-19T01:58:39.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.099+0000 7f5186ffd640 1 -- 172.21.15.55:0/3121324562 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f519001a070 con 0x7f51981540c0 2024-03-19T01:58:39.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.099+0000 7f5186ffd640 1 --2- 172.21.15.55:0/3121324562 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f51780676b0 0x7f5178069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:39.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.099+0000 7f519659c640 1 --2- 172.21.15.55:0/3121324562 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f51780676b0 0x7f5178069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:39.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.099+0000 7f519659c640 1 --2- 172.21.15.55:0/3121324562 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f51780676b0 0x7f5178069b70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f51880ae000 tx=0x7f51880a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:39.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.103+0000 7f5186ffd640 1 -- 172.21.15.55:0/3121324562 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f5190053a30 con 0x7f51981540c0 2024-03-19T01:58:39.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.383+0000 7f5186ffd640 1 -- 172.21.15.55:0/3121324562 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f5190055990 con 0x7f51981540c0 2024-03-19T01:58:39.621 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.623+0000 7f519cd4f640 1 -- 172.21.15.55:0/3121324562 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51980c70a0 con 0x7f51981540c0 2024-03-19T01:58:39.701 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.703+0000 7f5186ffd640 1 -- 172.21.15.55:0/3121324562 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f5190059a30 con 0x7f51981540c0 2024-03-19T01:58:39.702 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:39.702 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:39.702 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:39.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.707+0000 7f519cd4f640 1 -- 172.21.15.55:0/3121324562 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f51780676b0 msgr2=0x7f5178069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:39.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.707+0000 7f519cd4f640 1 --2- 172.21.15.55:0/3121324562 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f51780676b0 0x7f5178069b70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f51880ae000 tx=0x7f51880a7000 comp rx=0 tx=0).stop 2024-03-19T01:58:39.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.707+0000 7f519cd4f640 1 -- 172.21.15.55:0/3121324562 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51981540c0 msgr2=0x7f51981f44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:39.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.707+0000 7f519cd4f640 1 --2- 172.21.15.55:0/3121324562 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51981540c0 0x7f51981f44d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f519000c770 tx=0x7f519000cc30 comp rx=0 tx=0).stop 2024-03-19T01:58:39.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.707+0000 7f519cd4f640 1 -- 172.21.15.55:0/3121324562 shutdown_connections 2024-03-19T01:58:39.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.707+0000 7f519cd4f640 1 --2- 172.21.15.55:0/3121324562 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f51780676b0 0x7f5178069b70 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:39.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.707+0000 7f519cd4f640 1 --2- 172.21.15.55:0/3121324562 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51981540c0 0x7f51981f44d0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:39.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.707+0000 7f519cd4f640 1 --2- 172.21.15.55:0/3121324562 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51980c9a90 0x7f5198154b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:39.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.707+0000 7f519cd4f640 1 --2- 172.21.15.55:0/3121324562 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51980c96b0 0x7f51980cbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:39.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.707+0000 7f519cd4f640 1 -- 172.21.15.55:0/3121324562 >> 172.21.15.55:0/3121324562 conn(0x7f51980cc9c0 msgr2=0x7f5198070a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:39.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.707+0000 7f519cd4f640 1 -- 172.21.15.55:0/3121324562 shutdown_connections 2024-03-19T01:58:39.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:39.707+0000 7f519cd4f640 1 -- 172.21.15.55:0/3121324562 wait complete. 2024-03-19T01:58:39.801 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:58:39.803 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:58:40.327 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:40.327 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:41.327 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:41.441 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.443+0000 7f8fab8be640 1 Processor -- start 2024-03-19T01:58:41.441 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.443+0000 7f8fab8be640 1 -- start start 2024-03-19T01:58:41.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.443+0000 7f8fab8be640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f8fa40c38c0 0x7f8fa4158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:41.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.443+0000 7f8fab8be640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f8fa415b550 0x7f8fa40c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:41.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.443+0000 7f8fab8be640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fa40c7900 con 0x7f8fa40c73c0 2024-03-19T01:58:41.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.443+0000 7f8fab8be640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f8fa40c7be0 con 0x7f8fa40c38c0 2024-03-19T01:58:41.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.443+0000 7f8fab8be640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f8fa40c7ec0 con 0x7f8fa415b550 2024-03-19T01:58:41.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.443+0000 7f8fa8e59640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f8fa40c73c0 0x7f8fa406b980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:57932/0 (socket says 172.21.15.55:57932) 2024-03-19T01:58:41.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.443+0000 7f8fa8e59640 1 -- 172.21.15.55:0/452526000 learned_addr learned my addr 172.21.15.55:0/452526000 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:58:41.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.447+0000 7f8fa9e5b640 1 --2- 172.21.15.55:0/452526000 >> v2:172.21.15.84:3300/0 conn(0x7f8fa415b550 0x7f8fa40c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:41.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.447+0000 7f8fa965a640 1 --2- 172.21.15.55:0/452526000 >> v2:172.21.15.55:3300/0 conn(0x7f8fa40c38c0 0x7f8fa4158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:41.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.451+0000 7f8fa965a640 1 -- 172.21.15.55:0/452526000 >> v2:172.21.15.84:3300/0 conn(0x7f8fa415b550 msgr2=0x7f8fa40c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:41.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.451+0000 7f8fa965a640 1 --2- 172.21.15.55:0/452526000 >> v2:172.21.15.84:3300/0 conn(0x7f8fa415b550 0x7f8fa40c60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:41.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.451+0000 7f8fa965a640 1 -- 172.21.15.55:0/452526000 >> v1:172.21.15.55:6789/0 conn(0x7f8fa40c73c0 legacy=0x7f8fa406b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:41.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.451+0000 7f8fa965a640 1 -- 172.21.15.55:0/452526000 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fa41582b0 con 0x7f8fa40c38c0 2024-03-19T01:58:41.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.455+0000 7f8fa965a640 1 --2- 172.21.15.55:0/452526000 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fa40c38c0 0x7f8fa4158ab0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8fa40c7830 tx=0x7f8fa001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd628e65ad5f3188 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:41.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.459+0000 7f8f9bfff640 1 -- 172.21.15.55:0/452526000 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa00b5070 con 0x7f8fa40c38c0 2024-03-19T01:58:41.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.459+0000 7f8f9bfff640 1 -- 172.21.15.55:0/452526000 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fa00a8970 con 0x7f8fa40c38c0 2024-03-19T01:58:41.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.459+0000 7f8f9bfff640 1 -- 172.21.15.55:0/452526000 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa00a8d40 con 0x7f8fa40c38c0 2024-03-19T01:58:41.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.459+0000 7f8fab8be640 1 -- 172.21.15.55:0/452526000 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fa40c38c0 msgr2=0x7f8fa4158ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:41.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.459+0000 7f8fab8be640 1 --2- 172.21.15.55:0/452526000 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fa40c38c0 0x7f8fa4158ab0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8fa40c7830 tx=0x7f8fa001ec70 comp rx=0 tx=0).stop 2024-03-19T01:58:41.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.459+0000 7f8fab8be640 1 -- 172.21.15.55:0/452526000 shutdown_connections 2024-03-19T01:58:41.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.459+0000 7f8fab8be640 1 --2- 172.21.15.55:0/452526000 >> v2:172.21.15.84:3300/0 conn(0x7f8fa415b550 0x7f8fa40c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:41.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.459+0000 7f8fab8be640 1 --2- 172.21.15.55:0/452526000 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fa40c38c0 0x7f8fa4158ab0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:41.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.459+0000 7f8fab8be640 1 -- 172.21.15.55:0/452526000 >> 172.21.15.55:0/452526000 conn(0x7f8fa40cc9c0 msgr2=0x7f8fa40ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:41.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.459+0000 7f8fab8be640 1 -- 172.21.15.55:0/452526000 shutdown_connections 2024-03-19T01:58:41.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.459+0000 7f8fab8be640 1 -- 172.21.15.55:0/452526000 wait complete. 2024-03-19T01:58:41.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.459+0000 7f8fab8be640 1 Processor -- start 2024-03-19T01:58:41.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.463+0000 7f8fab8be640 1 -- start start 2024-03-19T01:58:41.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.463+0000 7f8fab8be640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fa40c38c0 0x7f8fa41f3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:41.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.463+0000 7f8fab8be640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fa40c73c0 0x7f8fa40c5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:41.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.463+0000 7f8fab8be640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8fa415b550 0x7f8fa413cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:41.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.463+0000 7f8fab8be640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa406c090 con 0x7f8fa40c73c0 2024-03-19T01:58:41.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.463+0000 7f8fab8be640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa4157f60 con 0x7f8fa40c38c0 2024-03-19T01:58:41.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.463+0000 7f8fab8be640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f8fa406c210 con 0x7f8fa415b550 2024-03-19T01:58:41.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.463+0000 7f8fa9e5b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8fa415b550 0x7f8fa413cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:41.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.463+0000 7f8fa8e59640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fa40c73c0 0x7f8fa40c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:41.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.463+0000 7f8fa8e59640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fa40c73c0 0x7f8fa40c5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50798/0 (socket says 172.21.15.55:50798) 2024-03-19T01:58:41.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.463+0000 7f8fa8e59640 1 -- 172.21.15.55:0/973143171 learned_addr learned my addr 172.21.15.55:0/973143171 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:41.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.463+0000 7f8fa965a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fa40c38c0 0x7f8fa41f3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:41.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.467+0000 7f8fa8e59640 1 -- 172.21.15.55:0/973143171 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8fa415b550 msgr2=0x7f8fa413cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:41.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.467+0000 7f8fa8e59640 1 --2- 172.21.15.55:0/973143171 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8fa415b550 0x7f8fa413cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:41.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.467+0000 7f8fa8e59640 1 -- 172.21.15.55:0/973143171 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fa40c38c0 msgr2=0x7f8fa41f3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:41.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.467+0000 7f8fa8e59640 1 --2- 172.21.15.55:0/973143171 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fa40c38c0 0x7f8fa41f3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:41.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.467+0000 7f8fa8e59640 1 -- 172.21.15.55:0/973143171 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fa41f4190 con 0x7f8fa40c73c0 2024-03-19T01:58:41.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.471+0000 7f8fa8e59640 1 --2- 172.21.15.55:0/973143171 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fa40c73c0 0x7f8fa40c5820 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8f9400c420 tx=0x7f8f9400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:41.472 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.475+0000 7f8f99ffb640 1 -- 172.21.15.55:0/973143171 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f94017070 con 0x7f8fa40c73c0 2024-03-19T01:58:41.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.475+0000 7f8f99ffb640 1 -- 172.21.15.55:0/973143171 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f94005cf0 con 0x7f8fa40c73c0 2024-03-19T01:58:41.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.475+0000 7f8fab8be640 1 -- 172.21.15.55:0/973143171 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fa41e6990 con 0x7f8fa40c73c0 2024-03-19T01:58:41.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.475+0000 7f8f99ffb640 1 -- 172.21.15.55:0/973143171 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f940072f0 con 0x7f8fa40c73c0 2024-03-19T01:58:41.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.475+0000 7f8fab8be640 1 -- 172.21.15.55:0/973143171 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fa41e6f50 con 0x7f8fa40c73c0 2024-03-19T01:58:41.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.475+0000 7f8f8f7fe640 1 -- 172.21.15.55:0/973143171 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fa4068a40 con 0x7f8fa40c73c0 2024-03-19T01:58:41.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.499+0000 7f8f99ffb640 1 -- 172.21.15.55:0/973143171 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f8f9401a070 con 0x7f8fa40c73c0 2024-03-19T01:58:41.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.499+0000 7f8f99ffb640 1 --2- 172.21.15.55:0/973143171 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8f880676b0 0x7f8f88069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:41.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.499+0000 7f8fa965a640 1 --2- 172.21.15.55:0/973143171 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8f880676b0 0x7f8f88069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:41.499 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.503+0000 7f8fa965a640 1 --2- 172.21.15.55:0/973143171 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8f880676b0 0x7f8f88069b70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8fa00b2000 tx=0x7f8fa0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:41.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.799+0000 7f8f99ffb640 1 -- 172.21.15.55:0/973143171 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f8f940539e0 con 0x7f8fa40c73c0 2024-03-19T01:58:41.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:41.799+0000 7f8f99ffb640 1 -- 172.21.15.55:0/973143171 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f8f94021070 con 0x7f8fa40c73c0 2024-03-19T01:58:42.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:42.055+0000 7f8f8f7fe640 1 -- 172.21.15.55:0/973143171 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8fa415c270 con 0x7f8fa40c73c0 2024-03-19T01:58:42.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:42.075+0000 7f8f99ffb640 1 -- 172.21.15.55:0/973143171 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f8f94055940 con 0x7f8fa40c73c0 2024-03-19T01:58:42.074 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:42.074 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:42.074 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:42.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:42.079+0000 7f8f8f7fe640 1 -- 172.21.15.55:0/973143171 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8f880676b0 msgr2=0x7f8f88069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:42.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:42.079+0000 7f8f8f7fe640 1 --2- 172.21.15.55:0/973143171 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8f880676b0 0x7f8f88069b70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8fa00b2000 tx=0x7f8fa0008040 comp rx=0 tx=0).stop 2024-03-19T01:58:42.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:42.079+0000 7f8f8f7fe640 1 -- 172.21.15.55:0/973143171 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fa40c73c0 msgr2=0x7f8fa40c5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:42.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:42.079+0000 7f8f8f7fe640 1 --2- 172.21.15.55:0/973143171 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fa40c73c0 0x7f8fa40c5820 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8f9400c420 tx=0x7f8f9400c8e0 comp rx=0 tx=0).stop 2024-03-19T01:58:42.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:42.079+0000 7f8f8f7fe640 1 -- 172.21.15.55:0/973143171 shutdown_connections 2024-03-19T01:58:42.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:42.079+0000 7f8f8f7fe640 1 --2- 172.21.15.55:0/973143171 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8f880676b0 0x7f8f88069b70 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:42.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:42.079+0000 7f8f8f7fe640 1 --2- 172.21.15.55:0/973143171 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8fa415b550 0x7f8fa413cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:42.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:42.079+0000 7f8f8f7fe640 1 --2- 172.21.15.55:0/973143171 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fa40c73c0 0x7f8fa40c5820 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:42.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:42.079+0000 7f8f8f7fe640 1 --2- 172.21.15.55:0/973143171 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fa40c38c0 0x7f8fa41f3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:42.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:42.079+0000 7f8f8f7fe640 1 -- 172.21.15.55:0/973143171 >> 172.21.15.55:0/973143171 conn(0x7f8fa40cc9c0 msgr2=0x7f8fa40cbf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:42.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:42.079+0000 7f8f8f7fe640 1 -- 172.21.15.55:0/973143171 shutdown_connections 2024-03-19T01:58:42.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:42.079+0000 7f8f8f7fe640 1 -- 172.21.15.55:0/973143171 wait complete. 2024-03-19T01:58:42.700 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:42.700 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:43.701 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:43.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.815+0000 7fc28d881640 1 Processor -- start 2024-03-19T01:58:43.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.815+0000 7fc28d881640 1 -- start start 2024-03-19T01:58:43.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.819+0000 7fc28d881640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc2880c96b0 0x7fc288070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:43.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.819+0000 7fc28d881640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fc2880c9a90 0x7fc2880c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:43.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.819+0000 7fc28d881640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc2881540c0 0x7fc288154570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:43.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.819+0000 7fc28d881640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fc28815e3a0 con 0x7fc2881540c0 2024-03-19T01:58:43.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.819+0000 7fc28d881640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fc28815e680 con 0x7fc2880c9a90 2024-03-19T01:58:43.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.819+0000 7fc28d881640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc28815a330 con 0x7fc2880c96b0 2024-03-19T01:58:43.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.819+0000 7fc2877fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc2881540c0 0x7fc288154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:43.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.819+0000 7fc286ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc2880c96b0 0x7fc288070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:43.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.819+0000 7fc2867fc640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fc2880c9a90 0x7fc2880c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:43.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.819+0000 7fc286ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc2880c96b0 0x7fc288070f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:50752/0 (socket says 172.21.15.55:50752) 2024-03-19T01:58:43.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.819+0000 7fc286ffd640 1 -- 172.21.15.55:0/2766152874 learned_addr learned my addr 172.21.15.55:0/2766152874 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:43.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.819+0000 7fc2877fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc2881540c0 0x7fc288154570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50814/0 (socket says 172.21.15.55:50814) 2024-03-19T01:58:43.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.823+0000 7fc2877fe640 1 -- 172.21.15.55:0/2766152874 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc2880c96b0 msgr2=0x7fc288070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:43.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.823+0000 7fc2877fe640 1 --2- 172.21.15.55:0/2766152874 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc2880c96b0 0x7fc288070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:43.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.823+0000 7fc2877fe640 1 -- 172.21.15.55:0/2766152874 >> v2:172.21.15.84:3300/0 conn(0x7fc2880c9a90 msgr2=0x7fc2880c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:43.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.823+0000 7fc2877fe640 1 --2- 172.21.15.55:0/2766152874 >> v2:172.21.15.84:3300/0 conn(0x7fc2880c9a90 0x7fc2880c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:43.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.823+0000 7fc2877fe640 1 -- 172.21.15.55:0/2766152874 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc28815a610 con 0x7fc2881540c0 2024-03-19T01:58:43.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.827+0000 7fc2877fe640 1 --2- 172.21.15.55:0/2766152874 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2881540c0 0x7fc288154570 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc288071e50 tx=0x7fc28001c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9eadb9ecc4b66a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:43.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.831+0000 7fc285ffb640 1 -- 172.21.15.55:0/2766152874 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2800b2070 con 0x7fc2881540c0 2024-03-19T01:58:43.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.831+0000 7fc285ffb640 1 -- 172.21.15.55:0/2766152874 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2800a5810 con 0x7fc2881540c0 2024-03-19T01:58:43.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.831+0000 7fc285ffb640 1 -- 172.21.15.55:0/2766152874 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2800a5be0 con 0x7fc2881540c0 2024-03-19T01:58:43.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.831+0000 7fc28d881640 1 -- 172.21.15.55:0/2766152874 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2881540c0 msgr2=0x7fc288154570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:43.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.831+0000 7fc28d881640 1 --2- 172.21.15.55:0/2766152874 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2881540c0 0x7fc288154570 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc288071e50 tx=0x7fc28001c8a0 comp rx=0 tx=0).stop 2024-03-19T01:58:43.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.831+0000 7fc28d881640 1 -- 172.21.15.55:0/2766152874 shutdown_connections 2024-03-19T01:58:43.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.831+0000 7fc28d881640 1 --2- 172.21.15.55:0/2766152874 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2881540c0 0x7fc288154570 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:43.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.831+0000 7fc28d881640 1 --2- 172.21.15.55:0/2766152874 >> v2:172.21.15.84:3300/0 conn(0x7fc2880c9a90 0x7fc2880c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:43.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.831+0000 7fc28d881640 1 --2- 172.21.15.55:0/2766152874 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc2880c96b0 0x7fc288070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:43.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.831+0000 7fc28d881640 1 -- 172.21.15.55:0/2766152874 >> 172.21.15.55:0/2766152874 conn(0x7fc2880cc9c0 msgr2=0x7fc2880ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:43.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.831+0000 7fc28d881640 1 -- 172.21.15.55:0/2766152874 shutdown_connections 2024-03-19T01:58:43.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.831+0000 7fc28d881640 1 -- 172.21.15.55:0/2766152874 wait complete. 2024-03-19T01:58:43.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.831+0000 7fc28d881640 1 Processor -- start 2024-03-19T01:58:43.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.831+0000 7fc28d881640 1 -- start start 2024-03-19T01:58:43.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.835+0000 7fc28d881640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2880c96b0 0x7fc288159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:43.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.835+0000 7fc28d881640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2880c9a90 0x7fc288158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:43.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.835+0000 7fc28d881640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc2881540c0 0x7fc2880cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:43.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.835+0000 7fc28d881640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc2880c7a80 con 0x7fc2880c96b0 2024-03-19T01:58:43.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.835+0000 7fc28d881640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc2880c7900 con 0x7fc2880c9a90 2024-03-19T01:58:43.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.835+0000 7fc28d881640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc28806b2a0 con 0x7fc2881540c0 2024-03-19T01:58:43.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.835+0000 7fc286ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2880c96b0 0x7fc288159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:43.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.835+0000 7fc2867fc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2880c9a90 0x7fc288158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:43.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.835+0000 7fc286ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2880c96b0 0x7fc288159d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50826/0 (socket says 172.21.15.55:50826) 2024-03-19T01:58:43.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.835+0000 7fc286ffd640 1 -- 172.21.15.55:0/735320956 learned_addr learned my addr 172.21.15.55:0/735320956 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:43.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.835+0000 7fc2877fe640 1 --2- 172.21.15.55:0/735320956 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc2881540c0 0x7fc2880cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:43.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.839+0000 7fc286ffd640 1 -- 172.21.15.55:0/735320956 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc2881540c0 msgr2=0x7fc2880cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:43.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.839+0000 7fc286ffd640 1 --2- 172.21.15.55:0/735320956 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc2881540c0 0x7fc2880cbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:43.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.839+0000 7fc286ffd640 1 -- 172.21.15.55:0/735320956 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2880c9a90 msgr2=0x7fc288158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:43.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.839+0000 7fc286ffd640 1 --2- 172.21.15.55:0/735320956 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2880c9a90 0x7fc288158090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:43.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.839+0000 7fc286ffd640 1 -- 172.21.15.55:0/735320956 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc28806c0e0 con 0x7fc2880c96b0 2024-03-19T01:58:43.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.839+0000 7fc286ffd640 1 --2- 172.21.15.55:0/735320956 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2880c96b0 0x7fc288159d40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc27800e890 tx=0x7fc27800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:43.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.843+0000 7fc26ffff640 1 -- 172.21.15.55:0/735320956 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc278017070 con 0x7fc2880c96b0 2024-03-19T01:58:43.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.843+0000 7fc26ffff640 1 -- 172.21.15.55:0/735320956 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc27800c940 con 0x7fc2880c96b0 2024-03-19T01:58:43.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.843+0000 7fc26ffff640 1 -- 172.21.15.55:0/735320956 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27800cd30 con 0x7fc2880c96b0 2024-03-19T01:58:43.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.847+0000 7fc28d881640 1 -- 172.21.15.55:0/735320956 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc28814e120 con 0x7fc2880c96b0 2024-03-19T01:58:43.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.847+0000 7fc28d881640 1 -- 172.21.15.55:0/735320956 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc288148bf0 con 0x7fc2880c96b0 2024-03-19T01:58:43.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:43.847+0000 7fc28d881640 1 -- 172.21.15.55:0/735320956 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc288068a40 con 0x7fc2880c96b0 2024-03-19T01:58:44.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:44.155+0000 7fc26ffff640 1 -- 172.21.15.55:0/735320956 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc27801a070 con 0x7fc2880c96b0 2024-03-19T01:58:44.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:44.155+0000 7fc26ffff640 1 --2- 172.21.15.55:0/735320956 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc274067660 0x7fc274069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:44.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:44.155+0000 7fc2867fc640 1 --2- 172.21.15.55:0/735320956 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc274067660 0x7fc274069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:44.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:44.155+0000 7fc2867fc640 1 --2- 172.21.15.55:0/735320956 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc274067660 0x7fc274069b20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc2880c89b0 tx=0x7fc27c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:44.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:44.191+0000 7fc26ffff640 1 -- 172.21.15.55:0/735320956 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fc278054680 con 0x7fc2880c96b0 2024-03-19T01:58:44.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:44.191+0000 7fc26ffff640 1 -- 172.21.15.55:0/735320956 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc2780565e0 con 0x7fc2880c96b0 2024-03-19T01:58:44.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:44.403+0000 7fc28d881640 1 -- 172.21.15.55:0/735320956 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc288159960 con 0x7fc2880c96b0 2024-03-19T01:58:44.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:44.455+0000 7fc26ffff640 1 -- 172.21.15.55:0/735320956 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fc27805a680 con 0x7fc2880c96b0 2024-03-19T01:58:44.454 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:44.455 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:44.455 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:44.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:44.459+0000 7fc28d881640 1 -- 172.21.15.55:0/735320956 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc274067660 msgr2=0x7fc274069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:44.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:44.459+0000 7fc28d881640 1 --2- 172.21.15.55:0/735320956 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc274067660 0x7fc274069b20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc2880c89b0 tx=0x7fc27c009190 comp rx=0 tx=0).stop 2024-03-19T01:58:44.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:44.459+0000 7fc28d881640 1 -- 172.21.15.55:0/735320956 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2880c96b0 msgr2=0x7fc288159d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:44.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:44.459+0000 7fc28d881640 1 --2- 172.21.15.55:0/735320956 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2880c96b0 0x7fc288159d40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc27800e890 tx=0x7fc27800ed50 comp rx=0 tx=0).stop 2024-03-19T01:58:44.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:44.459+0000 7fc28d881640 1 -- 172.21.15.55:0/735320956 shutdown_connections 2024-03-19T01:58:44.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:44.459+0000 7fc28d881640 1 --2- 172.21.15.55:0/735320956 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc274067660 0x7fc274069b20 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:44.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:44.459+0000 7fc28d881640 1 --2- 172.21.15.55:0/735320956 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc2881540c0 0x7fc2880cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:44.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:44.459+0000 7fc28d881640 1 --2- 172.21.15.55:0/735320956 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2880c9a90 0x7fc288158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:44.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:44.459+0000 7fc28d881640 1 --2- 172.21.15.55:0/735320956 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2880c96b0 0x7fc288159d40 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:44.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:44.459+0000 7fc28d881640 1 -- 172.21.15.55:0/735320956 >> 172.21.15.55:0/735320956 conn(0x7fc2880cc9c0 msgr2=0x7fc288070a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:44.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:44.459+0000 7fc28d881640 1 -- 172.21.15.55:0/735320956 shutdown_connections 2024-03-19T01:58:44.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:44.459+0000 7fc28d881640 1 -- 172.21.15.55:0/735320956 wait complete. 2024-03-19T01:58:45.084 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:45.084 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:46.085 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:46.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.207+0000 7fa23ac0f640 1 Processor -- start 2024-03-19T01:58:46.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.207+0000 7fa23ac0f640 1 -- start start 2024-03-19T01:58:46.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.207+0000 7fa23ac0f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa23406be30 0x7fa23406c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:46.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.211+0000 7fa23ac0f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa234158220 con 0x7fa23413dec0 2024-03-19T01:58:46.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.211+0000 7fa23ac0f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa234158500 con 0x7fa2340c76f0 2024-03-19T01:58:46.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.211+0000 7fa23ac0f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fa2340c7b30 con 0x7fa23406be30 2024-03-19T01:58:46.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.211+0000 7fa2391ac640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa23406be30 0x7fa23406c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:46.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.211+0000 7fa2391ac640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa23406be30 0x7fa23406c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50840/0 (socket says 172.21.15.55:50840) 2024-03-19T01:58:46.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.211+0000 7fa2391ac640 1 -- 172.21.15.55:0/2612906652 learned_addr learned my addr 172.21.15.55:0/2612906652 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:46.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.211+0000 7fa2391ac640 1 -- 172.21.15.55:0/2612906652 >> v1:172.21.15.84:6789/0 conn(0x7fa2340c76f0 legacy=0x7fa2340c5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:46.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.211+0000 7fa2391ac640 1 -- 172.21.15.55:0/2612906652 >> v1:172.21.15.55:6789/0 conn(0x7fa23413dec0 legacy=0x7fa234067000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:46.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.211+0000 7fa2391ac640 1 -- 172.21.15.55:0/2612906652 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2341587e0 con 0x7fa23406be30 2024-03-19T01:58:46.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.215+0000 7fa2391ac640 1 --2- 172.21.15.55:0/2612906652 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa23406be30 0x7fa23406c240 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa224009e50 tx=0x7fa22401f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=318e0186fdd53c27 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:46.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.219+0000 7fa2337fe640 1 -- 172.21.15.55:0/2612906652 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2240b4070 con 0x7fa23406be30 2024-03-19T01:58:46.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.223+0000 7fa2337fe640 1 -- 172.21.15.55:0/2612906652 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2240af400 con 0x7fa23406be30 2024-03-19T01:58:46.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.223+0000 7fa2337fe640 1 -- 172.21.15.55:0/2612906652 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2240af7d0 con 0x7fa23406be30 2024-03-19T01:58:46.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.223+0000 7fa23ac0f640 1 -- 172.21.15.55:0/2612906652 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa23406be30 msgr2=0x7fa23406c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:46.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.223+0000 7fa23ac0f640 1 --2- 172.21.15.55:0/2612906652 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa23406be30 0x7fa23406c240 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa224009e50 tx=0x7fa22401f1e0 comp rx=0 tx=0).stop 2024-03-19T01:58:46.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.223+0000 7fa23ac0f640 1 -- 172.21.15.55:0/2612906652 shutdown_connections 2024-03-19T01:58:46.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.223+0000 7fa23ac0f640 1 --2- 172.21.15.55:0/2612906652 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa23406be30 0x7fa23406c240 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:46.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.223+0000 7fa23ac0f640 1 -- 172.21.15.55:0/2612906652 >> 172.21.15.55:0/2612906652 conn(0x7fa2340cc270 msgr2=0x7fa2340cc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:46.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.223+0000 7fa23ac0f640 1 -- 172.21.15.55:0/2612906652 shutdown_connections 2024-03-19T01:58:46.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.223+0000 7fa23ac0f640 1 -- 172.21.15.55:0/2612906652 wait complete. 2024-03-19T01:58:46.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.223+0000 7fa23ac0f640 1 Processor -- start 2024-03-19T01:58:46.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.223+0000 7fa23ac0f640 1 -- start start 2024-03-19T01:58:46.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.227+0000 7fa23ac0f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa23406be30 0x7fa2341572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:46.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.227+0000 7fa23ac0f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa2340c76f0 0x7fa234159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:46.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.227+0000 7fa23ac0f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa23413dec0 0x7fa234159ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:46.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.227+0000 7fa23ac0f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa23406c780 con 0x7fa23413dec0 2024-03-19T01:58:46.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.227+0000 7fa23ac0f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa234158da0 con 0x7fa23406be30 2024-03-19T01:58:46.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.227+0000 7fa23ac0f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa23413e2a0 con 0x7fa2340c76f0 2024-03-19T01:58:46.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.227+0000 7fa2391ac640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa23413dec0 0x7fa234159ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:46.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.227+0000 7fa2389ab640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa23406be30 0x7fa2341572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:46.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.227+0000 7fa233fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa2340c76f0 0x7fa234159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:46.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.227+0000 7fa2391ac640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa23413dec0 0x7fa234159ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50850/0 (socket says 172.21.15.55:50850) 2024-03-19T01:58:46.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.227+0000 7fa2391ac640 1 -- 172.21.15.55:0/2874078342 learned_addr learned my addr 172.21.15.55:0/2874078342 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:46.229 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.231+0000 7fa2391ac640 1 -- 172.21.15.55:0/2874078342 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa2340c76f0 msgr2=0x7fa234159660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:46.230 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.231+0000 7fa2391ac640 1 --2- 172.21.15.55:0/2874078342 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa2340c76f0 0x7fa234159660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:46.230 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.231+0000 7fa2391ac640 1 -- 172.21.15.55:0/2874078342 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa23406be30 msgr2=0x7fa2341572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:46.230 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.231+0000 7fa2391ac640 1 --2- 172.21.15.55:0/2874078342 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa23406be30 0x7fa2341572b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:46.230 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.231+0000 7fa2391ac640 1 -- 172.21.15.55:0/2874078342 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa23415c080 con 0x7fa23413dec0 2024-03-19T01:58:46.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.235+0000 7fa2391ac640 1 --2- 172.21.15.55:0/2874078342 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa23413dec0 0x7fa234159ba0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa224009e50 tx=0x7fa22401bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:46.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.239+0000 7fa2317fa640 1 -- 172.21.15.55:0/2874078342 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2240b4040 con 0x7fa23413dec0 2024-03-19T01:58:46.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.239+0000 7fa2317fa640 1 -- 172.21.15.55:0/2874078342 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2240c1070 con 0x7fa23413dec0 2024-03-19T01:58:46.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.239+0000 7fa2317fa640 1 -- 172.21.15.55:0/2874078342 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2240ba470 con 0x7fa23413dec0 2024-03-19T01:58:46.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.239+0000 7fa23ac0f640 1 -- 172.21.15.55:0/2874078342 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2340c54e0 con 0x7fa23413dec0 2024-03-19T01:58:46.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.239+0000 7fa23ac0f640 1 -- 172.21.15.55:0/2874078342 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa23415ba30 con 0x7fa23413dec0 2024-03-19T01:58:46.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.243+0000 7fa23ac0f640 1 -- 172.21.15.55:0/2874078342 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa234068a40 con 0x7fa23413dec0 2024-03-19T01:58:46.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.259+0000 7fa2317fa640 1 -- 172.21.15.55:0/2874078342 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa2240ba610 con 0x7fa23413dec0 2024-03-19T01:58:46.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.259+0000 7fa2317fa640 1 --2- 172.21.15.55:0/2874078342 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa214067660 0x7fa214069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:46.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.263+0000 7fa2389ab640 1 --2- 172.21.15.55:0/2874078342 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa214067660 0x7fa214069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:46.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.263+0000 7fa2389ab640 1 --2- 172.21.15.55:0/2874078342 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa214067660 0x7fa214069b20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa22c0098e0 tx=0x7fa22c009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:46.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.267+0000 7fa2317fa640 1 -- 172.21.15.55:0/2874078342 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fa224002020 con 0x7fa23413dec0 2024-03-19T01:58:46.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.531+0000 7fa2317fa640 1 -- 172.21.15.55:0/2874078342 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa224018d40 con 0x7fa23413dec0 2024-03-19T01:58:46.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.731+0000 7fa23ac0f640 1 -- 172.21.15.55:0/2874078342 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa23413cdf0 con 0x7fa23413dec0 2024-03-19T01:58:46.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.791+0000 7fa2317fa640 1 -- 172.21.15.55:0/2874078342 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fa2240f4ea0 con 0x7fa23413dec0 2024-03-19T01:58:46.790 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:46.790 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:46.790 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:46.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.795+0000 7fa23ac0f640 1 -- 172.21.15.55:0/2874078342 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa214067660 msgr2=0x7fa214069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:46.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.795+0000 7fa23ac0f640 1 --2- 172.21.15.55:0/2874078342 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa214067660 0x7fa214069b20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa22c0098e0 tx=0x7fa22c009060 comp rx=0 tx=0).stop 2024-03-19T01:58:46.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.795+0000 7fa23ac0f640 1 -- 172.21.15.55:0/2874078342 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa23413dec0 msgr2=0x7fa234159ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:46.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.795+0000 7fa23ac0f640 1 --2- 172.21.15.55:0/2874078342 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa23413dec0 0x7fa234159ba0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa224009e50 tx=0x7fa22401bf80 comp rx=0 tx=0).stop 2024-03-19T01:58:46.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.795+0000 7fa23ac0f640 1 -- 172.21.15.55:0/2874078342 shutdown_connections 2024-03-19T01:58:46.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.795+0000 7fa23ac0f640 1 --2- 172.21.15.55:0/2874078342 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa214067660 0x7fa214069b20 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:46.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.795+0000 7fa23ac0f640 1 --2- 172.21.15.55:0/2874078342 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa23413dec0 0x7fa234159ba0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:46.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.795+0000 7fa23ac0f640 1 --2- 172.21.15.55:0/2874078342 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa2340c76f0 0x7fa234159660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:46.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.795+0000 7fa23ac0f640 1 --2- 172.21.15.55:0/2874078342 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa23406be30 0x7fa2341572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:46.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.795+0000 7fa23ac0f640 1 -- 172.21.15.55:0/2874078342 >> 172.21.15.55:0/2874078342 conn(0x7fa2340cc270 msgr2=0x7fa234067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:46.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.795+0000 7fa23ac0f640 1 -- 172.21.15.55:0/2874078342 shutdown_connections 2024-03-19T01:58:46.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:46.795+0000 7fa23ac0f640 1 -- 172.21.15.55:0/2874078342 wait complete. 2024-03-19T01:58:47.413 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:47.413 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:48.414 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:48.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.543+0000 7f553de7a640 1 Processor -- start 2024-03-19T01:58:48.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.543+0000 7f553de7a640 1 -- start start 2024-03-19T01:58:48.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.543+0000 7f553de7a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f553806b2a0 0x7f5538073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:48.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.543+0000 7f553de7a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55380c7fa0 con 0x7f553815acb0 2024-03-19T01:58:48.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.543+0000 7f553de7a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55380c8240 con 0x7f55381544f0 2024-03-19T01:58:48.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.543+0000 7f553de7a640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f553815b210 con 0x7f553806b2a0 2024-03-19T01:58:48.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.543+0000 7f5537fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f553815acb0 0x7f55380cce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:57946/0 (socket says 172.21.15.55:57946) 2024-03-19T01:58:48.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.543+0000 7f5537fff640 1 -- 172.21.15.55:0/348969817 learned_addr learned my addr 172.21.15.55:0/348969817 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:58:48.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.547+0000 7f55367fc640 1 -- 172.21.15.55:0/348969817 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3129235309 0 0) 0x7f55380c7fa0 con 0x7f553815acb0 2024-03-19T01:58:48.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.547+0000 7f55367fc640 1 -- 172.21.15.55:0/348969817 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5520003530 con 0x7f553815acb0 2024-03-19T01:58:48.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.551+0000 7f55367fc640 1 -- 172.21.15.55:0/348969817 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1525742476 0 0) 0x7f5520003530 con 0x7f553815acb0 2024-03-19T01:58:48.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.551+0000 7f55367fc640 1 -- 172.21.15.55:0/348969817 >> v2:172.21.15.84:3300/0 conn(0x7f553806b2a0 msgr2=0x7f5538073540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:48.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.551+0000 7f55367fc640 1 --2- 172.21.15.55:0/348969817 >> v2:172.21.15.84:3300/0 conn(0x7f553806b2a0 0x7f5538073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:48.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.551+0000 7f55367fc640 1 -- 172.21.15.55:0/348969817 >> v1:172.21.15.84:6789/0 conn(0x7f55381544f0 legacy=0x7f553815e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:48.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.551+0000 7f55367fc640 1 -- 172.21.15.55:0/348969817 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f553815d920 con 0x7f553815acb0 2024-03-19T01:58:48.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.551+0000 7f55367fc640 1 -- 172.21.15.55:0/348969817 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f5530001a20 con 0x7f553815acb0 2024-03-19T01:58:48.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.551+0000 7f55367fc640 1 -- 172.21.15.55:0/348969817 >> v1:172.21.15.55:6789/0 conn(0x7f553815acb0 legacy=0x7f55380cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:48.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.551+0000 7f55367fc640 1 --2- 172.21.15.55:0/348969817 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5520003a10 0x7f5520003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:48.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.551+0000 7f55367fc640 1 --2- 172.21.15.55:0/348969817 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5520004810 0x7f5520004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:48.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.551+0000 7f55367fc640 1 --2- 172.21.15.55:0/348969817 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f55200052f0 0x7f552000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:48.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.551+0000 7f55367fc640 1 -- 172.21.15.55:0/348969817 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f553815b210 con 0x7f5520004810 2024-03-19T01:58:48.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.551+0000 7f55367fc640 1 -- 172.21.15.55:0/348969817 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f552000c230 con 0x7f5520003a10 2024-03-19T01:58:48.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.551+0000 7f55367fc640 1 -- 172.21.15.55:0/348969817 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f552000c430 con 0x7f55200052f0 2024-03-19T01:58:48.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.551+0000 7f5537fff640 1 --2- 172.21.15.55:0/348969817 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f55200052f0 0x7f552000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:48.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.551+0000 7f5536ffd640 1 --2- 172.21.15.55:0/348969817 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5520004810 0x7f5520004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:48.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.555+0000 7f55377fe640 1 --2- 172.21.15.55:0/348969817 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5520003a10 0x7f5520003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:48.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.559+0000 7f5536ffd640 1 -- 172.21.15.55:0/348969817 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f55200052f0 msgr2=0x7f552000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:48.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.559+0000 7f5536ffd640 1 --2- 172.21.15.55:0/348969817 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f55200052f0 0x7f552000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:48.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.559+0000 7f5536ffd640 1 -- 172.21.15.55:0/348969817 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5520003a10 msgr2=0x7f5520003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:48.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.559+0000 7f5536ffd640 1 --2- 172.21.15.55:0/348969817 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5520003a10 0x7f5520003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:48.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.559+0000 7f5536ffd640 1 -- 172.21.15.55:0/348969817 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5520003530 con 0x7f5520004810 2024-03-19T01:58:48.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.559+0000 7f5537fff640 1 -- 172.21.15.55:0/348969817 reap_dead start 2024-03-19T01:58:48.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.563+0000 7f5536ffd640 1 --2- 172.21.15.55:0/348969817 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5520004810 0x7f5520004c90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f552c009860 tx=0x7f552c01c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b84a3a07c129a88a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:48.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.567+0000 7f55367fc640 1 -- 172.21.15.55:0/348969817 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552c0b0070 con 0x7f5520004810 2024-03-19T01:58:48.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.567+0000 7f55367fc640 1 -- 172.21.15.55:0/348969817 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f552c0ab620 con 0x7f5520004810 2024-03-19T01:58:48.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.567+0000 7f55367fc640 1 -- 172.21.15.55:0/348969817 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552c009460 con 0x7f5520004810 2024-03-19T01:58:48.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.567+0000 7f553de7a640 1 -- 172.21.15.55:0/348969817 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5520004810 msgr2=0x7f5520004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:48.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.567+0000 7f553de7a640 1 --2- 172.21.15.55:0/348969817 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5520004810 0x7f5520004c90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f552c009860 tx=0x7f552c01c320 comp rx=0 tx=0).stop 2024-03-19T01:58:48.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.567+0000 7f553de7a640 1 -- 172.21.15.55:0/348969817 shutdown_connections 2024-03-19T01:58:48.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.567+0000 7f553de7a640 1 --2- 172.21.15.55:0/348969817 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5520004810 0x7f5520004c90 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:48.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.567+0000 7f553de7a640 1 -- 172.21.15.55:0/348969817 >> 172.21.15.55:0/348969817 conn(0x7f553806e670 msgr2=0x7f553806ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:48.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.567+0000 7f553de7a640 1 -- 172.21.15.55:0/348969817 shutdown_connections 2024-03-19T01:58:48.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.567+0000 7f553de7a640 1 -- 172.21.15.55:0/348969817 wait complete. 2024-03-19T01:58:48.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.567+0000 7f553de7a640 1 Processor -- start 2024-03-19T01:58:48.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.567+0000 7f553de7a640 1 -- start start 2024-03-19T01:58:48.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.571+0000 7f553de7a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5520004810 0x7f55380cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:48.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.571+0000 7f553de7a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f55380c3db0 0x7f5538070410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:48.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.571+0000 7f553de7a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f55381f01e0 0x7f55381f0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:48.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.571+0000 7f553de7a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5538073540 con 0x7f5520004810 2024-03-19T01:58:48.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.571+0000 7f553de7a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f55381548d0 con 0x7f55381f01e0 2024-03-19T01:58:48.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.571+0000 7f553de7a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f55380736c0 con 0x7f55380c3db0 2024-03-19T01:58:48.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.571+0000 7f55377fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5520004810 0x7f55380cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:48.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.571+0000 7f5536ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f55380c3db0 0x7f5538070410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:48.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.571+0000 7f55377fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5520004810 0x7f55380cbb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50868/0 (socket says 172.21.15.55:50868) 2024-03-19T01:58:48.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.571+0000 7f55377fe640 1 -- 172.21.15.55:0/2646988721 learned_addr learned my addr 172.21.15.55:0/2646988721 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:48.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.575+0000 7f55377fe640 1 -- 172.21.15.55:0/2646988721 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f55380c3db0 msgr2=0x7f5538070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:48.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.575+0000 7f55377fe640 1 --2- 172.21.15.55:0/2646988721 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f55380c3db0 0x7f5538070410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:48.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.575+0000 7f55377fe640 1 -- 172.21.15.55:0/2646988721 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f55381f01e0 msgr2=0x7f55381f0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:48.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.575+0000 7f55377fe640 1 --2- 172.21.15.55:0/2646988721 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f55381f01e0 0x7f55381f0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:48.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.575+0000 7f55377fe640 1 -- 172.21.15.55:0/2646988721 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55380c5780 con 0x7f5520004810 2024-03-19T01:58:48.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.595+0000 7f55377fe640 1 --2- 172.21.15.55:0/2646988721 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5520004810 0x7f55380cbb80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f552800e4b0 tx=0x7f552800e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:48.610 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.611+0000 7f551ffff640 1 -- 172.21.15.55:0/2646988721 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5528019070 con 0x7f5520004810 2024-03-19T01:58:48.610 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.611+0000 7f551ffff640 1 -- 172.21.15.55:0/2646988721 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55280052e0 con 0x7f5520004810 2024-03-19T01:58:48.610 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.611+0000 7f553de7a640 1 -- 172.21.15.55:0/2646988721 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55381f0ba0 con 0x7f5520004810 2024-03-19T01:58:48.610 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.611+0000 7f551ffff640 1 -- 172.21.15.55:0/2646988721 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5528003820 con 0x7f5520004810 2024-03-19T01:58:48.610 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.611+0000 7f553de7a640 1 -- 172.21.15.55:0/2646988721 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55380c59f0 con 0x7f5520004810 2024-03-19T01:58:48.611 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.615+0000 7f553de7a640 1 -- 172.21.15.55:0/2646988721 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5538068a40 con 0x7f5520004810 2024-03-19T01:58:48.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.647+0000 7f551ffff640 1 -- 172.21.15.55:0/2646988721 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f552801c070 con 0x7f5520004810 2024-03-19T01:58:48.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.647+0000 7f551ffff640 1 --2- 172.21.15.55:0/2646988721 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5524067660 0x7f5524069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:48.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.647+0000 7f5536ffd640 1 --2- 172.21.15.55:0/2646988721 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5524067660 0x7f5524069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:48.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.647+0000 7f5536ffd640 1 --2- 172.21.15.55:0/2646988721 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5524067660 0x7f5524069b20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f552c0ad000 tx=0x7f552c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:48.919 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.919+0000 7f551ffff640 1 -- 172.21.15.55:0/2646988721 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f5528053a60 con 0x7f5520004810 2024-03-19T01:58:48.919 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:48.923+0000 7f551ffff640 1 -- 172.21.15.55:0/2646988721 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f5528023020 con 0x7f5520004810 2024-03-19T01:58:49.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:49.135+0000 7f553de7a640 1 -- 172.21.15.55:0/2646988721 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55380c9e50 con 0x7f5520004810 2024-03-19T01:58:49.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:49.159+0000 7f551ffff640 1 -- 172.21.15.55:0/2646988721 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f55381f0ba0 con 0x7f5520004810 2024-03-19T01:58:49.155 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:49.155 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:49.156 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:49.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:49.159+0000 7f553de7a640 1 -- 172.21.15.55:0/2646988721 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5524067660 msgr2=0x7f5524069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:49.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:49.159+0000 7f553de7a640 1 --2- 172.21.15.55:0/2646988721 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5524067660 0x7f5524069b20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f552c0ad000 tx=0x7f552c008040 comp rx=0 tx=0).stop 2024-03-19T01:58:49.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:49.159+0000 7f553de7a640 1 -- 172.21.15.55:0/2646988721 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5520004810 msgr2=0x7f55380cbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:49.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:49.159+0000 7f553de7a640 1 --2- 172.21.15.55:0/2646988721 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5520004810 0x7f55380cbb80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f552800e4b0 tx=0x7f552800e970 comp rx=0 tx=0).stop 2024-03-19T01:58:49.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:49.159+0000 7f553de7a640 1 -- 172.21.15.55:0/2646988721 shutdown_connections 2024-03-19T01:58:49.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:49.159+0000 7f553de7a640 1 --2- 172.21.15.55:0/2646988721 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5524067660 0x7f5524069b20 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:49.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:49.159+0000 7f553de7a640 1 --2- 172.21.15.55:0/2646988721 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f55381f01e0 0x7f55381f0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:49.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:49.159+0000 7f553de7a640 1 --2- 172.21.15.55:0/2646988721 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f55380c3db0 0x7f5538070410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:49.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:49.159+0000 7f553de7a640 1 --2- 172.21.15.55:0/2646988721 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5520004810 0x7f55380cbb80 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:49.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:49.159+0000 7f553de7a640 1 -- 172.21.15.55:0/2646988721 >> 172.21.15.55:0/2646988721 conn(0x7f553806e670 msgr2=0x7f55380729f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:49.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:49.163+0000 7f553de7a640 1 -- 172.21.15.55:0/2646988721 shutdown_connections 2024-03-19T01:58:49.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:49.163+0000 7f553de7a640 1 -- 172.21.15.55:0/2646988721 wait complete. 2024-03-19T01:58:49.785 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:49.785 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:50.786 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:50.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.915+0000 7f166d578640 1 Processor -- start 2024-03-19T01:58:50.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.915+0000 7f166d578640 1 -- start start 2024-03-19T01:58:50.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.919+0000 7f166d578640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f16681511f0 0x7f16680c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:50.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.919+0000 7f166d578640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16680c3330 con 0x7f16680c2e40 2024-03-19T01:58:50.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.919+0000 7f166d578640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16680c3610 con 0x7f16680c8260 2024-03-19T01:58:50.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.919+0000 7f166d578640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f16680c5040 con 0x7f16681511f0 2024-03-19T01:58:50.916 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.919+0000 7f16667fc640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f16680c2e40 0x7f16680c7450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:51770/0 (socket says 172.21.15.55:51770) 2024-03-19T01:58:50.916 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.919+0000 7f16667fc640 1 -- 172.21.15.55:0/1480195898 learned_addr learned my addr 172.21.15.55:0/1480195898 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:58:50.916 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.919+0000 7f1666ffd640 1 --2- 172.21.15.55:0/1480195898 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f16681511f0 0x7f16680c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:50.919 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.923+0000 7f1665ffb640 1 -- 172.21.15.55:0/1480195898 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3104251954 0 0) 0x7f16680c3330 con 0x7f16680c2e40 2024-03-19T01:58:50.919 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.923+0000 7f1665ffb640 1 -- 172.21.15.55:0/1480195898 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1650003530 con 0x7f16680c2e40 2024-03-19T01:58:50.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.923+0000 7f1666ffd640 1 -- 172.21.15.55:0/1480195898 >> v1:172.21.15.84:6789/0 conn(0x7f16680c8260 legacy=0x7f1668161810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:50.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.923+0000 7f1666ffd640 1 -- 172.21.15.55:0/1480195898 >> v1:172.21.15.55:6789/0 conn(0x7f16680c2e40 legacy=0x7f16680c7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:50.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.923+0000 7f1666ffd640 1 -- 172.21.15.55:0/1480195898 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16680c38f0 con 0x7f16681511f0 2024-03-19T01:58:50.925 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.927+0000 7f1666ffd640 1 --2- 172.21.15.55:0/1480195898 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f16681511f0 0x7f16680c5a90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f16580097f0 tx=0x7f165801ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90f4adc99baab8fa server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:50.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.935+0000 7f1665ffb640 1 -- 172.21.15.55:0/1480195898 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16580b5070 con 0x7f16681511f0 2024-03-19T01:58:50.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.935+0000 7f1665ffb640 1 -- 172.21.15.55:0/1480195898 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16580a89d0 con 0x7f16681511f0 2024-03-19T01:58:50.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.935+0000 7f1665ffb640 1 -- 172.21.15.55:0/1480195898 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16580a8da0 con 0x7f16681511f0 2024-03-19T01:58:50.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.935+0000 7f166d578640 1 -- 172.21.15.55:0/1480195898 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f16681511f0 msgr2=0x7f16680c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:50.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.935+0000 7f166d578640 1 --2- 172.21.15.55:0/1480195898 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f16681511f0 0x7f16680c5a90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f16580097f0 tx=0x7f165801ecd0 comp rx=0 tx=0).stop 2024-03-19T01:58:50.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.935+0000 7f166d578640 1 -- 172.21.15.55:0/1480195898 shutdown_connections 2024-03-19T01:58:50.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.935+0000 7f166d578640 1 --2- 172.21.15.55:0/1480195898 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f16681511f0 0x7f16680c5a90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:50.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.935+0000 7f166d578640 1 -- 172.21.15.55:0/1480195898 >> 172.21.15.55:0/1480195898 conn(0x7f166806f3b0 msgr2=0x7f166806f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:50.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.935+0000 7f166d578640 1 -- 172.21.15.55:0/1480195898 shutdown_connections 2024-03-19T01:58:50.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.935+0000 7f166d578640 1 -- 172.21.15.55:0/1480195898 wait complete. 2024-03-19T01:58:50.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.939+0000 7f166d578640 1 Processor -- start 2024-03-19T01:58:50.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.939+0000 7f166d578640 1 -- start start 2024-03-19T01:58:50.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.939+0000 7f166d578640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f16680c2e40 0x7f1668143c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:50.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.939+0000 7f166d578640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f16680c8260 0x7f16681458e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:50.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.939+0000 7f166d578640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f16681511f0 0x7f1668153c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:50.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.939+0000 7f166d578640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f16680c29c0 con 0x7f16681511f0 2024-03-19T01:58:50.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.939+0000 7f166d578640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f16681515d0 con 0x7f16680c8260 2024-03-19T01:58:50.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.939+0000 7f166d578640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f16680c2b40 con 0x7f16680c2e40 2024-03-19T01:58:50.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.939+0000 7f16677fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f16681511f0 0x7f1668153c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:50.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.939+0000 7f16667fc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f16680c8260 0x7f16681458e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:50.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.939+0000 7f1666ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f16680c2e40 0x7f1668143c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:50.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.939+0000 7f16677fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f16681511f0 0x7f1668153c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53518/0 (socket says 172.21.15.55:53518) 2024-03-19T01:58:50.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.939+0000 7f16677fe640 1 -- 172.21.15.55:0/2380266570 learned_addr learned my addr 172.21.15.55:0/2380266570 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:50.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.943+0000 7f16677fe640 1 -- 172.21.15.55:0/2380266570 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f16680c2e40 msgr2=0x7f1668143c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:50.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.943+0000 7f16677fe640 1 --2- 172.21.15.55:0/2380266570 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f16680c2e40 0x7f1668143c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:50.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.943+0000 7f16677fe640 1 -- 172.21.15.55:0/2380266570 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f16680c8260 msgr2=0x7f16681458e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:50.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.943+0000 7f16677fe640 1 --2- 172.21.15.55:0/2380266570 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f16680c8260 0x7f16681458e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:50.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.943+0000 7f16677fe640 1 -- 172.21.15.55:0/2380266570 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16681edf40 con 0x7f16681511f0 2024-03-19T01:58:50.945 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.947+0000 7f16677fe640 1 --2- 172.21.15.55:0/2380266570 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f16681511f0 0x7f1668153c70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f166000c730 tx=0x7f166000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:50.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.951+0000 7f164f7fe640 1 -- 172.21.15.55:0/2380266570 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1660003b80 con 0x7f16681511f0 2024-03-19T01:58:50.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.951+0000 7f164f7fe640 1 -- 172.21.15.55:0/2380266570 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1660003d20 con 0x7f16681511f0 2024-03-19T01:58:50.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.951+0000 7f166d578640 1 -- 172.21.15.55:0/2380266570 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1668146720 con 0x7f16681511f0 2024-03-19T01:58:50.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.951+0000 7f164f7fe640 1 -- 172.21.15.55:0/2380266570 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1660005310 con 0x7f16681511f0 2024-03-19T01:58:50.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.951+0000 7f166d578640 1 -- 172.21.15.55:0/2380266570 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16681ee200 con 0x7f16681511f0 2024-03-19T01:58:50.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:50.955+0000 7f166d578640 1 -- 172.21.15.55:0/2380266570 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16680c9160 con 0x7f16681511f0 2024-03-19T01:58:51.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:51.231+0000 7f164f7fe640 1 -- 172.21.15.55:0/2380266570 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f16600056c0 con 0x7f16681511f0 2024-03-19T01:58:51.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:51.231+0000 7f164f7fe640 1 --2- 172.21.15.55:0/2380266570 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f16540676e0 0x7f1654069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:51.229 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:51.231+0000 7f1666ffd640 1 --2- 172.21.15.55:0/2380266570 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f16540676e0 0x7f1654069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:51.229 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:51.231+0000 7f1666ffd640 1 --2- 172.21.15.55:0/2380266570 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f16540676e0 0x7f1654069ba0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f16580b2000 tx=0x7f1658008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:51.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:51.255+0000 7f164f7fe640 1 -- 172.21.15.55:0/2380266570 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f16600548d0 con 0x7f16681511f0 2024-03-19T01:58:51.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:51.255+0000 7f164f7fe640 1 -- 172.21.15.55:0/2380266570 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1660056830 con 0x7f16681511f0 2024-03-19T01:58:51.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:51.467+0000 7f166d578640 1 -- 172.21.15.55:0/2380266570 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1668068330 con 0x7f16681511f0 2024-03-19T01:58:51.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:51.551+0000 7f164f7fe640 1 -- 172.21.15.55:0/2380266570 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f166005a8d0 con 0x7f16681511f0 2024-03-19T01:58:51.548 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:51.549 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:51.549 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:51.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:51.555+0000 7f166d578640 1 -- 172.21.15.55:0/2380266570 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f16540676e0 msgr2=0x7f1654069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:51.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:51.555+0000 7f166d578640 1 --2- 172.21.15.55:0/2380266570 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f16540676e0 0x7f1654069ba0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f16580b2000 tx=0x7f1658008040 comp rx=0 tx=0).stop 2024-03-19T01:58:51.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:51.555+0000 7f166d578640 1 -- 172.21.15.55:0/2380266570 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f16681511f0 msgr2=0x7f1668153c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:51.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:51.555+0000 7f166d578640 1 --2- 172.21.15.55:0/2380266570 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f16681511f0 0x7f1668153c70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f166000c730 tx=0x7f166000cbf0 comp rx=0 tx=0).stop 2024-03-19T01:58:51.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:51.555+0000 7f166d578640 1 -- 172.21.15.55:0/2380266570 shutdown_connections 2024-03-19T01:58:51.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:51.555+0000 7f166d578640 1 --2- 172.21.15.55:0/2380266570 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f16540676e0 0x7f1654069ba0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:51.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:51.555+0000 7f166d578640 1 --2- 172.21.15.55:0/2380266570 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f16681511f0 0x7f1668153c70 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:51.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:51.555+0000 7f166d578640 1 --2- 172.21.15.55:0/2380266570 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f16680c8260 0x7f16681458e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:51.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:51.555+0000 7f166d578640 1 --2- 172.21.15.55:0/2380266570 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f16680c2e40 0x7f1668143c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:51.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:51.555+0000 7f166d578640 1 -- 172.21.15.55:0/2380266570 >> 172.21.15.55:0/2380266570 conn(0x7f166806f3b0 msgr2=0x7f16680cad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:51.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:51.555+0000 7f166d578640 1 -- 172.21.15.55:0/2380266570 shutdown_connections 2024-03-19T01:58:51.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:51.555+0000 7f166d578640 1 -- 172.21.15.55:0/2380266570 wait complete. 2024-03-19T01:58:52.175 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:52.175 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:53.176 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:53.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.291+0000 7fe7cbfff640 1 Processor -- start 2024-03-19T01:58:53.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.291+0000 7fe7cbfff640 1 -- start start 2024-03-19T01:58:53.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.291+0000 7fe7cbfff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe7c406c610 0x7fe7c415ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:53.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.291+0000 7fe7cbfff640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7c4070e80 con 0x7fe7c406ceb0 2024-03-19T01:58:53.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.291+0000 7fe7cbfff640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7c415b250 con 0x7fe7c406c9f0 2024-03-19T01:58:53.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.291+0000 7fe7cbfff640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fe7c406fb00 con 0x7fe7c406c610 2024-03-19T01:58:53.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.295+0000 7fe7c9d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe7c406c610 0x7fe7c415ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:53.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.295+0000 7fe7c959a640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fe7c406c9f0 0x7fe7c40c92c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:46444/0 (socket says 172.21.15.55:46444) 2024-03-19T01:58:53.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.295+0000 7fe7c959a640 1 -- 172.21.15.55:0/2699873334 learned_addr learned my addr 172.21.15.55:0/2699873334 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:58:53.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.299+0000 7fe7c9d9b640 1 -- 172.21.15.55:0/2699873334 >> v1:172.21.15.84:6789/0 conn(0x7fe7c406c9f0 legacy=0x7fe7c40c92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:53.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.299+0000 7fe7c9d9b640 1 -- 172.21.15.55:0/2699873334 >> v1:172.21.15.55:6789/0 conn(0x7fe7c406ceb0 legacy=0x7fe7c41427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:53.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.299+0000 7fe7c9d9b640 1 -- 172.21.15.55:0/2699873334 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7c406fde0 con 0x7fe7c406c610 2024-03-19T01:58:53.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.303+0000 7fe7c9d9b640 1 --2- 172.21.15.55:0/2699873334 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7c406c610 0x7fe7c415ad10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe7c00097c0 tx=0x7fe7c001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a3d6f14059abaf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:53.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.307+0000 7fe7c8d99640 1 -- 172.21.15.55:0/2699873334 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c00b0070 con 0x7fe7c406c610 2024-03-19T01:58:53.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.307+0000 7fe7c8d99640 1 -- 172.21.15.55:0/2699873334 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7c0004230 con 0x7fe7c406c610 2024-03-19T01:58:53.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.307+0000 7fe7c8d99640 1 -- 172.21.15.55:0/2699873334 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c0003620 con 0x7fe7c406c610 2024-03-19T01:58:53.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.307+0000 7fe7cbfff640 1 -- 172.21.15.55:0/2699873334 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7c406c610 msgr2=0x7fe7c415ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:53.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.307+0000 7fe7cbfff640 1 --2- 172.21.15.55:0/2699873334 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7c406c610 0x7fe7c415ad10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe7c00097c0 tx=0x7fe7c001c460 comp rx=0 tx=0).stop 2024-03-19T01:58:53.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.307+0000 7fe7cbfff640 1 -- 172.21.15.55:0/2699873334 shutdown_connections 2024-03-19T01:58:53.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.307+0000 7fe7cbfff640 1 --2- 172.21.15.55:0/2699873334 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7c406c610 0x7fe7c415ad10 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:53.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.307+0000 7fe7cbfff640 1 -- 172.21.15.55:0/2699873334 >> 172.21.15.55:0/2699873334 conn(0x7fe7c4067600 msgr2=0x7fe7c4071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:53.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.307+0000 7fe7cbfff640 1 -- 172.21.15.55:0/2699873334 shutdown_connections 2024-03-19T01:58:53.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.307+0000 7fe7cbfff640 1 -- 172.21.15.55:0/2699873334 wait complete. 2024-03-19T01:58:53.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.311+0000 7fe7cbfff640 1 Processor -- start 2024-03-19T01:58:53.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.311+0000 7fe7cbfff640 1 -- start start 2024-03-19T01:58:53.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.311+0000 7fe7cbfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7c406c610 0x7fe7c413ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:53.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.311+0000 7fe7cbfff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7c406c9f0 0x7fe7c40c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:53.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.311+0000 7fe7cbfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7c406ceb0 0x7fe7c41f66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:53.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.311+0000 7fe7cbfff640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c40c99d0 con 0x7fe7c406c610 2024-03-19T01:58:53.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.311+0000 7fe7cbfff640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c4070960 con 0x7fe7c406c9f0 2024-03-19T01:58:53.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.311+0000 7fe7cbfff640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fe7c4142f00 con 0x7fe7c406ceb0 2024-03-19T01:58:53.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.311+0000 7fe7c9d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7c406c610 0x7fe7c413ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:53.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.311+0000 7fe7c959a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7c406c9f0 0x7fe7c40c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:53.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.311+0000 7fe7c9d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7c406c610 0x7fe7c413ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53530/0 (socket says 172.21.15.55:53530) 2024-03-19T01:58:53.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.311+0000 7fe7c9d9b640 1 -- 172.21.15.55:0/763612323 learned_addr learned my addr 172.21.15.55:0/763612323 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:53.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.311+0000 7fe7ca59c640 1 --2- 172.21.15.55:0/763612323 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7c406ceb0 0x7fe7c41f66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:53.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.319+0000 7fe7c9d9b640 1 -- 172.21.15.55:0/763612323 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7c406ceb0 msgr2=0x7fe7c41f66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:53.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.319+0000 7fe7c9d9b640 1 --2- 172.21.15.55:0/763612323 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7c406ceb0 0x7fe7c41f66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:53.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.319+0000 7fe7c9d9b640 1 -- 172.21.15.55:0/763612323 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7c406c9f0 msgr2=0x7fe7c40c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:53.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.319+0000 7fe7c9d9b640 1 --2- 172.21.15.55:0/763612323 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7c406c9f0 0x7fe7c40c7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:53.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.319+0000 7fe7c9d9b640 1 -- 172.21.15.55:0/763612323 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7c40c6490 con 0x7fe7c406c610 2024-03-19T01:58:53.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.323+0000 7fe7c9d9b640 1 --2- 172.21.15.55:0/763612323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7c406c610 0x7fe7c413ea70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe7c001a090 tx=0x7fe7c0019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:53.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.327+0000 7fe7ba7fc640 1 -- 172.21.15.55:0/763612323 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c00b9070 con 0x7fe7c406c610 2024-03-19T01:58:53.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.327+0000 7fe7ba7fc640 1 -- 172.21.15.55:0/763612323 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7c0003e20 con 0x7fe7c406c610 2024-03-19T01:58:53.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.327+0000 7fe7cbfff640 1 -- 172.21.15.55:0/763612323 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7c40c6770 con 0x7fe7c406c610 2024-03-19T01:58:53.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.327+0000 7fe7ba7fc640 1 -- 172.21.15.55:0/763612323 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c00b0040 con 0x7fe7c406c610 2024-03-19T01:58:53.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.327+0000 7fe7cbfff640 1 -- 172.21.15.55:0/763612323 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7c40c6d30 con 0x7fe7c406c610 2024-03-19T01:58:53.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.327+0000 7fe7cbfff640 1 -- 172.21.15.55:0/763612323 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7c4068a40 con 0x7fe7c406c610 2024-03-19T01:58:53.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.359+0000 7fe7ba7fc640 1 -- 172.21.15.55:0/763612323 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fe7c00bc070 con 0x7fe7c406c610 2024-03-19T01:58:53.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.359+0000 7fe7ba7fc640 1 --2- 172.21.15.55:0/763612323 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe7a80676b0 0x7fe7a8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:53.356 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.359+0000 7fe7c959a640 1 --2- 172.21.15.55:0/763612323 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe7a80676b0 0x7fe7a8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:53.356 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.359+0000 7fe7c959a640 1 --2- 172.21.15.55:0/763612323 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe7a80676b0 0x7fe7a8069b70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe7c41381a0 tx=0x7fe7b4007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:53.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.363+0000 7fe7ba7fc640 1 -- 172.21.15.55:0/763612323 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fe7c00041d0 con 0x7fe7c406c610 2024-03-19T01:58:53.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.667+0000 7fe7ba7fc640 1 -- 172.21.15.55:0/763612323 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fe7c0016050 con 0x7fe7c406c610 2024-03-19T01:58:53.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.875+0000 7fe7cbfff640 1 -- 172.21.15.55:0/763612323 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7c413cfc0 con 0x7fe7c406c610 2024-03-19T01:58:53.911 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.915+0000 7fe7ba7fc640 1 -- 172.21.15.55:0/763612323 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fe7c00f0460 con 0x7fe7c406c610 2024-03-19T01:58:53.911 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:53.911 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:53.911 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:53.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.915+0000 7fe7cbfff640 1 -- 172.21.15.55:0/763612323 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe7a80676b0 msgr2=0x7fe7a8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:53.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.915+0000 7fe7cbfff640 1 --2- 172.21.15.55:0/763612323 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe7a80676b0 0x7fe7a8069b70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe7c41381a0 tx=0x7fe7b4007e90 comp rx=0 tx=0).stop 2024-03-19T01:58:53.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.915+0000 7fe7cbfff640 1 -- 172.21.15.55:0/763612323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7c406c610 msgr2=0x7fe7c413ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:53.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.915+0000 7fe7cbfff640 1 --2- 172.21.15.55:0/763612323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7c406c610 0x7fe7c413ea70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe7c001a090 tx=0x7fe7c0019200 comp rx=0 tx=0).stop 2024-03-19T01:58:53.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.915+0000 7fe7cbfff640 1 -- 172.21.15.55:0/763612323 shutdown_connections 2024-03-19T01:58:53.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.915+0000 7fe7cbfff640 1 --2- 172.21.15.55:0/763612323 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe7a80676b0 0x7fe7a8069b70 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:53.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.915+0000 7fe7cbfff640 1 --2- 172.21.15.55:0/763612323 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7c406ceb0 0x7fe7c41f66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:53.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.915+0000 7fe7cbfff640 1 --2- 172.21.15.55:0/763612323 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7c406c9f0 0x7fe7c40c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:53.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.915+0000 7fe7cbfff640 1 --2- 172.21.15.55:0/763612323 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7c406c610 0x7fe7c413ea70 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:53.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.915+0000 7fe7cbfff640 1 -- 172.21.15.55:0/763612323 >> 172.21.15.55:0/763612323 conn(0x7fe7c4067600 msgr2=0x7fe7c40c4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:53.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.915+0000 7fe7cbfff640 1 -- 172.21.15.55:0/763612323 shutdown_connections 2024-03-19T01:58:53.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:53.915+0000 7fe7cbfff640 1 -- 172.21.15.55:0/763612323 wait complete. 2024-03-19T01:58:54.538 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:54.538 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:55.539 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:55.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.659+0000 7f27bc41e640 1 Processor -- start 2024-03-19T01:58:55.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.659+0000 7f27bc41e640 1 -- start start 2024-03-19T01:58:55.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.663+0000 7f27bc41e640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f27b4152120 0x7f27b40c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:55.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.663+0000 7f27bc41e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f27b40c9610 0x7f27b4157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:55.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.663+0000 7f27bc41e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27b406fff0 con 0x7f27b4070780 2024-03-19T01:58:55.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.663+0000 7f27bc41e640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f27b4152500 con 0x7f27b40c9610 2024-03-19T01:58:55.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.663+0000 7f27bc41e640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f27b4070590 con 0x7f27b4152120 2024-03-19T01:58:55.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.663+0000 7f27b99b9640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f27b40c9610 0x7f27b4157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:55.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.663+0000 7f27b99b9640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f27b40c9610 0x7f27b4157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53532/0 (socket says 172.21.15.55:53532) 2024-03-19T01:58:55.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.663+0000 7f27b99b9640 1 -- 172.21.15.55:0/1702220164 learned_addr learned my addr 172.21.15.55:0/1702220164 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:55.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.663+0000 7f27ba1ba640 1 --2- 172.21.15.55:0/1702220164 >> v2:172.21.15.84:3300/0 conn(0x7f27b4152120 0x7f27b40c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:55.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.667+0000 7f27b99b9640 1 -- 172.21.15.55:0/1702220164 >> v2:172.21.15.84:3300/0 conn(0x7f27b4152120 msgr2=0x7f27b40c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:55.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.667+0000 7f27b99b9640 1 --2- 172.21.15.55:0/1702220164 >> v2:172.21.15.84:3300/0 conn(0x7f27b4152120 0x7f27b40c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:55.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.667+0000 7f27b99b9640 1 -- 172.21.15.55:0/1702220164 >> v1:172.21.15.55:6789/0 conn(0x7f27b4070780 legacy=0x7f27b4158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:55.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.667+0000 7f27b99b9640 1 -- 172.21.15.55:0/1702220164 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b40c39c0 con 0x7f27b40c9610 2024-03-19T01:58:55.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.671+0000 7f27b99b9640 1 --2- 172.21.15.55:0/1702220164 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27b40c9610 0x7f27b4157fd0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f27b4072430 tx=0x7f27a401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=465a34a47f0578d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:55.672 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.675+0000 7f27b91b8640 1 -- 172.21.15.55:0/1702220164 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a40b0070 con 0x7f27b40c9610 2024-03-19T01:58:55.672 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.675+0000 7f27b91b8640 1 -- 172.21.15.55:0/1702220164 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27a40ab600 con 0x7f27b40c9610 2024-03-19T01:58:55.672 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.675+0000 7f27b91b8640 1 -- 172.21.15.55:0/1702220164 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a40ab9d0 con 0x7f27b40c9610 2024-03-19T01:58:55.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.675+0000 7f27bc41e640 1 -- 172.21.15.55:0/1702220164 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27b40c9610 msgr2=0x7f27b4157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:55.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.675+0000 7f27bc41e640 1 --2- 172.21.15.55:0/1702220164 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27b40c9610 0x7f27b4157fd0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f27b4072430 tx=0x7f27a401c4e0 comp rx=0 tx=0).stop 2024-03-19T01:58:55.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.675+0000 7f27bc41e640 1 -- 172.21.15.55:0/1702220164 shutdown_connections 2024-03-19T01:58:55.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.675+0000 7f27bc41e640 1 --2- 172.21.15.55:0/1702220164 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27b40c9610 0x7f27b4157fd0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:55.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.675+0000 7f27bc41e640 1 --2- 172.21.15.55:0/1702220164 >> v2:172.21.15.84:3300/0 conn(0x7f27b4152120 0x7f27b40c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:55.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.675+0000 7f27bc41e640 1 -- 172.21.15.55:0/1702220164 >> 172.21.15.55:0/1702220164 conn(0x7f27b4067600 msgr2=0x7f27b4071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:55.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.675+0000 7f27bc41e640 1 -- 172.21.15.55:0/1702220164 shutdown_connections 2024-03-19T01:58:55.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.675+0000 7f27bc41e640 1 -- 172.21.15.55:0/1702220164 wait complete. 2024-03-19T01:58:55.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.679+0000 7f27bc41e640 1 Processor -- start 2024-03-19T01:58:55.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.679+0000 7f27bc41e640 1 -- start start 2024-03-19T01:58:55.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.679+0000 7f27bc41e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27b4070780 0x7f27b41f7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:55.675 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.679+0000 7f27bc41e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27b40c9610 0x7f27b41f3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:55.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.679+0000 7f27bc41e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f27b4152120 0x7f27b4156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:55.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.679+0000 7f27bc41e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f27b40c86d0 con 0x7f27b40c9610 2024-03-19T01:58:55.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.679+0000 7f27bc41e640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f27b40c31c0 con 0x7f27b4070780 2024-03-19T01:58:55.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.679+0000 7f27bc41e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f27b4165170 con 0x7f27b4152120 2024-03-19T01:58:55.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.679+0000 7f27b99b9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27b40c9610 0x7f27b41f3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:55.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.679+0000 7f27b99b9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27b40c9610 0x7f27b41f3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53546/0 (socket says 172.21.15.55:53546) 2024-03-19T01:58:55.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.679+0000 7f27b99b9640 1 -- 172.21.15.55:0/283516094 learned_addr learned my addr 172.21.15.55:0/283516094 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:55.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.679+0000 7f27ba9bb640 1 --2- 172.21.15.55:0/283516094 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f27b4152120 0x7f27b4156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:55.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.679+0000 7f27ba1ba640 1 --2- 172.21.15.55:0/283516094 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27b4070780 0x7f27b41f7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:55.680 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.683+0000 7f27b99b9640 1 -- 172.21.15.55:0/283516094 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f27b4152120 msgr2=0x7f27b4156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:55.680 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.683+0000 7f27b99b9640 1 --2- 172.21.15.55:0/283516094 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f27b4152120 0x7f27b4156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:55.680 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.683+0000 7f27b99b9640 1 -- 172.21.15.55:0/283516094 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27b4070780 msgr2=0x7f27b41f7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:55.680 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.683+0000 7f27b99b9640 1 --2- 172.21.15.55:0/283516094 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27b4070780 0x7f27b41f7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:55.680 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.683+0000 7f27b99b9640 1 -- 172.21.15.55:0/283516094 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b40b0eb0 con 0x7f27b40c9610 2024-03-19T01:58:55.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.683+0000 7f27b99b9640 1 --2- 172.21.15.55:0/283516094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27b40c9610 0x7f27b41f3690 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f27a401a110 tx=0x7f27a4019280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:55.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.691+0000 7f27aaffd640 1 -- 172.21.15.55:0/283516094 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a40ba070 con 0x7f27b40c9610 2024-03-19T01:58:55.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.691+0000 7f27aaffd640 1 -- 172.21.15.55:0/283516094 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27a40abe50 con 0x7f27b40c9610 2024-03-19T01:58:55.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.691+0000 7f27aaffd640 1 -- 172.21.15.55:0/283516094 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a40b0040 con 0x7f27b40c9610 2024-03-19T01:58:55.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.691+0000 7f27bc41e640 1 -- 172.21.15.55:0/283516094 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27b40b1190 con 0x7f27b40c9610 2024-03-19T01:58:55.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.691+0000 7f27bc41e640 1 -- 172.21.15.55:0/283516094 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27b41f6ca0 con 0x7f27b40c9610 2024-03-19T01:58:55.688 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.691+0000 7f27bc41e640 1 -- 172.21.15.55:0/283516094 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27b4068a40 con 0x7f27b40c9610 2024-03-19T01:58:55.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.727+0000 7f27aaffd640 1 -- 172.21.15.55:0/283516094 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f27a40b4640 con 0x7f27b40c9610 2024-03-19T01:58:55.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.727+0000 7f27aaffd640 1 --2- 172.21.15.55:0/283516094 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2798067660 0x7f2798069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:55.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.727+0000 7f27ba1ba640 1 --2- 172.21.15.55:0/283516094 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2798067660 0x7f2798069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:55.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:55.727+0000 7f27ba1ba640 1 --2- 172.21.15.55:0/283516094 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2798067660 0x7f2798069b20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f27b0009ac0 tx=0x7f27b0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:56.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:56.055+0000 7f27aaffd640 1 -- 172.21.15.55:0/283516094 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f27a40ed400 con 0x7f27b40c9610 2024-03-19T01:58:56.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:56.055+0000 7f27aaffd640 1 -- 172.21.15.55:0/283516094 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f27a40c3020 con 0x7f27b40c9610 2024-03-19T01:58:56.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:56.263+0000 7f27bc41e640 1 -- 172.21.15.55:0/283516094 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27b40cba70 con 0x7f27b40c9610 2024-03-19T01:58:56.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:56.283+0000 7f27aaffd640 1 -- 172.21.15.55:0/283516094 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f27a40160d0 con 0x7f27b40c9610 2024-03-19T01:58:56.279 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:56.279 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:56.279 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:56.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:56.283+0000 7f27bc41e640 1 -- 172.21.15.55:0/283516094 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2798067660 msgr2=0x7f2798069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:56.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:56.283+0000 7f27bc41e640 1 --2- 172.21.15.55:0/283516094 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2798067660 0x7f2798069b20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f27b0009ac0 tx=0x7f27b0009190 comp rx=0 tx=0).stop 2024-03-19T01:58:56.282 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:56.283+0000 7f27bc41e640 1 -- 172.21.15.55:0/283516094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27b40c9610 msgr2=0x7f27b41f3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:56.282 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:56.283+0000 7f27bc41e640 1 --2- 172.21.15.55:0/283516094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27b40c9610 0x7f27b41f3690 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f27a401a110 tx=0x7f27a4019280 comp rx=0 tx=0).stop 2024-03-19T01:58:56.282 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:56.283+0000 7f27bc41e640 1 -- 172.21.15.55:0/283516094 shutdown_connections 2024-03-19T01:58:56.282 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:56.283+0000 7f27bc41e640 1 --2- 172.21.15.55:0/283516094 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2798067660 0x7f2798069b20 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:56.282 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:56.283+0000 7f27bc41e640 1 --2- 172.21.15.55:0/283516094 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f27b4152120 0x7f27b4156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:56.283 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:56.283+0000 7f27bc41e640 1 --2- 172.21.15.55:0/283516094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27b40c9610 0x7f27b41f3690 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:56.283 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:56.283+0000 7f27bc41e640 1 --2- 172.21.15.55:0/283516094 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27b4070780 0x7f27b41f7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:56.283 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:56.283+0000 7f27bc41e640 1 -- 172.21.15.55:0/283516094 >> 172.21.15.55:0/283516094 conn(0x7f27b4067600 msgr2=0x7f27b406f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:56.283 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:56.283+0000 7f27bc41e640 1 -- 172.21.15.55:0/283516094 shutdown_connections 2024-03-19T01:58:56.283 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:56.283+0000 7f27bc41e640 1 -- 172.21.15.55:0/283516094 wait complete. 2024-03-19T01:58:56.903 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:56.904 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:58:57.904 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:58:58.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.035+0000 7f3edfa54640 1 Processor -- start 2024-03-19T01:58:58.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.035+0000 7f3edfa54640 1 -- start start 2024-03-19T01:58:58.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.035+0000 7f3edfa54640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3ed813dd00 0x7f3ed80c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:58.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.035+0000 7f3edfa54640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f3ed80c9780 0x7f3ed80c8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:58.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.035+0000 7f3edfa54640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3ed815b550 0x7f3ed80c5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:58.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.035+0000 7f3edfa54640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f3ed80c8f70 con 0x7f3ed813dd00 2024-03-19T01:58:58.035 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.035+0000 7f3edfa54640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f3ed80c9250 con 0x7f3ed80c9780 2024-03-19T01:58:58.035 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.035+0000 7f3edfa54640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3ed81582b0 con 0x7f3ed815b550 2024-03-19T01:58:58.035 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.039+0000 7f3edcfef640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f3ed80c9780 0x7f3ed80c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:58.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.039+0000 7f3edd7f0640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3ed813dd00 0x7f3ed80c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:58.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.039+0000 7f3eddff1640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3ed815b550 0x7f3ed80c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:58.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.039+0000 7f3edd7f0640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3ed813dd00 0x7f3ed80c7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53560/0 (socket says 172.21.15.55:53560) 2024-03-19T01:58:58.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.039+0000 7f3edd7f0640 1 -- 172.21.15.55:0/844403625 learned_addr learned my addr 172.21.15.55:0/844403625 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:58.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.043+0000 7f3edd7f0640 1 -- 172.21.15.55:0/844403625 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3ed815b550 msgr2=0x7f3ed80c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:58.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.043+0000 7f3edd7f0640 1 --2- 172.21.15.55:0/844403625 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3ed815b550 0x7f3ed80c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:58.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.043+0000 7f3edd7f0640 1 -- 172.21.15.55:0/844403625 >> v2:172.21.15.84:3300/0 conn(0x7f3ed80c9780 msgr2=0x7f3ed80c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:58:58.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.043+0000 7f3edd7f0640 1 --2- 172.21.15.55:0/844403625 >> v2:172.21.15.84:3300/0 conn(0x7f3ed80c9780 0x7f3ed80c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:58.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.043+0000 7f3edd7f0640 1 -- 172.21.15.55:0/844403625 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed8158590 con 0x7f3ed813dd00 2024-03-19T01:58:58.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.043+0000 7f3edd7f0640 1 --2- 172.21.15.55:0/844403625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ed813dd00 0x7f3ed80c7430 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3ed806ec30 tx=0x7f3ed401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=badbc88e7b2bb18d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:58.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.047+0000 7f3ecffff640 1 -- 172.21.15.55:0/844403625 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed40b1070 con 0x7f3ed813dd00 2024-03-19T01:58:58.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.047+0000 7f3ecffff640 1 -- 172.21.15.55:0/844403625 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ed40ac410 con 0x7f3ed813dd00 2024-03-19T01:58:58.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.047+0000 7f3ecffff640 1 -- 172.21.15.55:0/844403625 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed40ac7e0 con 0x7f3ed813dd00 2024-03-19T01:58:58.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.051+0000 7f3edfa54640 1 -- 172.21.15.55:0/844403625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ed813dd00 msgr2=0x7f3ed80c7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:58.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.051+0000 7f3edfa54640 1 --2- 172.21.15.55:0/844403625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ed813dd00 0x7f3ed80c7430 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3ed806ec30 tx=0x7f3ed401c520 comp rx=0 tx=0).stop 2024-03-19T01:58:58.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.051+0000 7f3edfa54640 1 -- 172.21.15.55:0/844403625 shutdown_connections 2024-03-19T01:58:58.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.051+0000 7f3edfa54640 1 --2- 172.21.15.55:0/844403625 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3ed815b550 0x7f3ed80c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:58.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.051+0000 7f3edfa54640 1 --2- 172.21.15.55:0/844403625 >> v2:172.21.15.84:3300/0 conn(0x7f3ed80c9780 0x7f3ed80c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:58.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.051+0000 7f3edfa54640 1 --2- 172.21.15.55:0/844403625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ed813dd00 0x7f3ed80c7430 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:58.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.051+0000 7f3edfa54640 1 -- 172.21.15.55:0/844403625 >> 172.21.15.55:0/844403625 conn(0x7f3ed806e5a0 msgr2=0x7f3ed8072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:58.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.051+0000 7f3edfa54640 1 -- 172.21.15.55:0/844403625 shutdown_connections 2024-03-19T01:58:58.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.051+0000 7f3edfa54640 1 -- 172.21.15.55:0/844403625 wait complete. 2024-03-19T01:58:58.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.051+0000 7f3edfa54640 1 Processor -- start 2024-03-19T01:58:58.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.051+0000 7f3edfa54640 1 -- start start 2024-03-19T01:58:58.050 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.055+0000 7f3edfa54640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3ed80c9780 0x7f3ed81fafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:58.050 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.055+0000 7f3edfa54640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ed813dd00 0x7f3ed80c3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:58.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.055+0000 7f3edfa54640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ed815b550 0x7f3ed8138060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:58.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.055+0000 7f3edfa54640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed8158f10 con 0x7f3ed815b550 2024-03-19T01:58:58.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.055+0000 7f3edfa54640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed8158d90 con 0x7f3ed813dd00 2024-03-19T01:58:58.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.055+0000 7f3edfa54640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3ed80cc7c0 con 0x7f3ed80c9780 2024-03-19T01:58:58.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.055+0000 7f3eddff1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ed815b550 0x7f3ed8138060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:58.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.055+0000 7f3edd7f0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3ed80c9780 0x7f3ed81fafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:58.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.055+0000 7f3eddff1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ed815b550 0x7f3ed8138060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:53570/0 (socket says 172.21.15.55:53570) 2024-03-19T01:58:58.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.055+0000 7f3eddff1640 1 -- 172.21.15.55:0/2905013675 learned_addr learned my addr 172.21.15.55:0/2905013675 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:58:58.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.055+0000 7f3edcfef640 1 --2- 172.21.15.55:0/2905013675 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ed813dd00 0x7f3ed80c3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:58.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.059+0000 7f3eddff1640 1 -- 172.21.15.55:0/2905013675 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3ed80c9780 msgr2=0x7f3ed81fafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:58.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.059+0000 7f3eddff1640 1 --2- 172.21.15.55:0/2905013675 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3ed80c9780 0x7f3ed81fafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:58.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.059+0000 7f3eddff1640 1 -- 172.21.15.55:0/2905013675 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ed813dd00 msgr2=0x7f3ed80c3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:58.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.059+0000 7f3eddff1640 1 --2- 172.21.15.55:0/2905013675 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ed813dd00 0x7f3ed80c3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:58.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.059+0000 7f3eddff1640 1 -- 172.21.15.55:0/2905013675 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed81334b0 con 0x7f3ed815b550 2024-03-19T01:58:58.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.059+0000 7f3eddff1640 1 --2- 172.21.15.55:0/2905013675 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ed815b550 0x7f3ed8138060 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3ed000efc0 tx=0x7f3ed000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:58.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.067+0000 7f3ecdffb640 1 -- 172.21.15.55:0/2905013675 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0018020 con 0x7f3ed815b550 2024-03-19T01:58:58.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.067+0000 7f3ecdffb640 1 -- 172.21.15.55:0/2905013675 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ed0014480 con 0x7f3ed815b550 2024-03-19T01:58:58.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.067+0000 7f3edfa54640 1 -- 172.21.15.55:0/2905013675 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ed80c5020 con 0x7f3ed815b550 2024-03-19T01:58:58.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.067+0000 7f3ecdffb640 1 -- 172.21.15.55:0/2905013675 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0014870 con 0x7f3ed815b550 2024-03-19T01:58:58.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.067+0000 7f3edfa54640 1 -- 172.21.15.55:0/2905013675 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ed80cb2a0 con 0x7f3ed815b550 2024-03-19T01:58:58.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.067+0000 7f3edfa54640 1 -- 172.21.15.55:0/2905013675 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ed8068a40 con 0x7f3ed815b550 2024-03-19T01:58:58.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.087+0000 7f3ecdffb640 1 -- 172.21.15.55:0/2905013675 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f3ed001b020 con 0x7f3ed815b550 2024-03-19T01:58:58.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.087+0000 7f3ecdffb640 1 --2- 172.21.15.55:0/2905013675 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3ebc0676b0 0x7f3ebc069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:58:58.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.087+0000 7f3edd7f0640 1 --2- 172.21.15.55:0/2905013675 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3ebc0676b0 0x7f3ebc069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:58:58.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.091+0000 7f3edd7f0640 1 --2- 172.21.15.55:0/2905013675 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3ebc0676b0 0x7f3ebc069b70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3ed40ae000 tx=0x7f3ed4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:58:58.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.355+0000 7f3ecdffb640 1 -- 172.21.15.55:0/2905013675 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f3ed00550e0 con 0x7f3ed815b550 2024-03-19T01:58:58.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.359+0000 7f3ecdffb640 1 -- 172.21.15.55:0/2905013675 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f3ed0022030 con 0x7f3ed815b550 2024-03-19T01:58:58.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.575+0000 7f3edfa54640 1 -- 172.21.15.55:0/2905013675 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ed81626c0 con 0x7f3ed815b550 2024-03-19T01:58:58.624 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.627+0000 7f3ecdffb640 1 -- 172.21.15.55:0/2905013675 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f3ed81334b0 con 0x7f3ed815b550 2024-03-19T01:58:58.624 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:58:58.624 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:58:58.625 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:58:58.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.627+0000 7f3edfa54640 1 -- 172.21.15.55:0/2905013675 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3ebc0676b0 msgr2=0x7f3ebc069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:58.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.627+0000 7f3edfa54640 1 --2- 172.21.15.55:0/2905013675 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3ebc0676b0 0x7f3ebc069b70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3ed40ae000 tx=0x7f3ed4008040 comp rx=0 tx=0).stop 2024-03-19T01:58:58.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.627+0000 7f3edfa54640 1 -- 172.21.15.55:0/2905013675 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ed815b550 msgr2=0x7f3ed8138060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:58:58.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.627+0000 7f3edfa54640 1 --2- 172.21.15.55:0/2905013675 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ed815b550 0x7f3ed8138060 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3ed000efc0 tx=0x7f3ed000c490 comp rx=0 tx=0).stop 2024-03-19T01:58:58.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.627+0000 7f3edfa54640 1 -- 172.21.15.55:0/2905013675 shutdown_connections 2024-03-19T01:58:58.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.627+0000 7f3edfa54640 1 --2- 172.21.15.55:0/2905013675 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3ebc0676b0 0x7f3ebc069b70 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:58.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.627+0000 7f3edfa54640 1 --2- 172.21.15.55:0/2905013675 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ed815b550 0x7f3ed8138060 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:58.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.627+0000 7f3edfa54640 1 --2- 172.21.15.55:0/2905013675 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ed813dd00 0x7f3ed80c3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:58.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.627+0000 7f3edfa54640 1 --2- 172.21.15.55:0/2905013675 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3ed80c9780 0x7f3ed81fafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:58:58.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.627+0000 7f3edfa54640 1 -- 172.21.15.55:0/2905013675 >> 172.21.15.55:0/2905013675 conn(0x7f3ed806e5a0 msgr2=0x7f3ed8138ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:58:58.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.627+0000 7f3edfa54640 1 -- 172.21.15.55:0/2905013675 shutdown_connections 2024-03-19T01:58:58.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:58:58.627+0000 7f3edfa54640 1 -- 172.21.15.55:0/2905013675 wait complete. 2024-03-19T01:58:59.251 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:58:59.251 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:00.253 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:00.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.383+0000 7feef4ae9640 1 Processor -- start 2024-03-19T01:59:00.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.383+0000 7feef4ae9640 1 -- start start 2024-03-19T01:59:00.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.383+0000 7feef4ae9640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7feef013dc50 0x7feef00c5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:00.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.383+0000 7feef4ae9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7feef013e030 0x7feef00c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:00.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.383+0000 7feef4ae9640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7feef015b980 0x7feef015be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:00.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.383+0000 7feef4ae9640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7feef015c350 con 0x7feef013dc50 2024-03-19T01:59:00.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.383+0000 7feef4ae9640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7feef01581c0 con 0x7feef015b980 2024-03-19T01:59:00.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.383+0000 7feef4ae9640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7feef0158460 con 0x7feef013e030 2024-03-19T01:59:00.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.383+0000 7feeee59c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7feef013dc50 0x7feef00c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:00.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.383+0000 7feeeed9d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7feef015b980 0x7feef015be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:00.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.387+0000 7feeedd9b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7feef013e030 0x7feef00c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:00.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.387+0000 7feeee59c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7feef013dc50 0x7feef00c5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:46492/0 (socket says 172.21.15.55:46492) 2024-03-19T01:59:00.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.387+0000 7feeee59c640 1 -- 172.21.15.55:0/3643893101 learned_addr learned my addr 172.21.15.55:0/3643893101 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:00.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.387+0000 7feeee59c640 1 -- 172.21.15.55:0/3643893101 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7feef013e030 msgr2=0x7feef00c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:00.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.387+0000 7feeee59c640 1 --2- 172.21.15.55:0/3643893101 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7feef013e030 0x7feef00c5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:00.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.387+0000 7feeee59c640 1 -- 172.21.15.55:0/3643893101 >> v2:172.21.15.84:3300/0 conn(0x7feef015b980 msgr2=0x7feef015be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:00.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.387+0000 7feeee59c640 1 --2- 172.21.15.55:0/3643893101 >> v2:172.21.15.84:3300/0 conn(0x7feef015b980 0x7feef015be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:00.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.387+0000 7feeee59c640 1 -- 172.21.15.55:0/3643893101 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef0158740 con 0x7feef013dc50 2024-03-19T01:59:00.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.391+0000 7feeee59c640 1 --2- 172.21.15.55:0/3643893101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feef013dc50 0x7feef00c5810 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7feef006cc60 tx=0x7feee001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91845d224bedf448 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:00.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.395+0000 7feeed59a640 1 -- 172.21.15.55:0/3643893101 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee00b5070 con 0x7feef013dc50 2024-03-19T01:59:00.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.395+0000 7feeed59a640 1 -- 172.21.15.55:0/3643893101 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feee00a89d0 con 0x7feef013dc50 2024-03-19T01:59:00.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.395+0000 7feeed59a640 1 -- 172.21.15.55:0/3643893101 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee00a8da0 con 0x7feef013dc50 2024-03-19T01:59:00.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.399+0000 7feef4ae9640 1 -- 172.21.15.55:0/3643893101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feef013dc50 msgr2=0x7feef00c5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:00.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.399+0000 7feef4ae9640 1 --2- 172.21.15.55:0/3643893101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feef013dc50 0x7feef00c5810 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7feef006cc60 tx=0x7feee001ed10 comp rx=0 tx=0).stop 2024-03-19T01:59:00.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.399+0000 7feef4ae9640 1 -- 172.21.15.55:0/3643893101 shutdown_connections 2024-03-19T01:59:00.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.399+0000 7feef4ae9640 1 --2- 172.21.15.55:0/3643893101 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7feef013e030 0x7feef00c5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:00.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.399+0000 7feef4ae9640 1 --2- 172.21.15.55:0/3643893101 >> v2:172.21.15.84:3300/0 conn(0x7feef015b980 0x7feef015be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:00.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.399+0000 7feef4ae9640 1 --2- 172.21.15.55:0/3643893101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feef013dc50 0x7feef00c5810 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:00.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.399+0000 7feef4ae9640 1 -- 172.21.15.55:0/3643893101 >> 172.21.15.55:0/3643893101 conn(0x7feef006fb40 msgr2=0x7feef006ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:00.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.399+0000 7feef4ae9640 1 -- 172.21.15.55:0/3643893101 shutdown_connections 2024-03-19T01:59:00.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.399+0000 7feef4ae9640 1 -- 172.21.15.55:0/3643893101 wait complete. 2024-03-19T01:59:00.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.399+0000 7feef4ae9640 1 Processor -- start 2024-03-19T01:59:00.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.399+0000 7feef4ae9640 1 -- start start 2024-03-19T01:59:00.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.399+0000 7feef4ae9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feef013dc50 0x7feef01528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:00.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.399+0000 7feef4ae9640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feef013e030 0x7feef00c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:00.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.403+0000 7feef4ae9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7feef015b980 0x7feef01f0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:00.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.403+0000 7feef4ae9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7feef00c6b00 con 0x7feef013dc50 2024-03-19T01:59:00.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.403+0000 7feef4ae9640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7feef0158d00 con 0x7feef013e030 2024-03-19T01:59:00.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.403+0000 7feef4ae9640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7feef00c6c80 con 0x7feef015b980 2024-03-19T01:59:00.400 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.403+0000 7feeee59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feef013dc50 0x7feef01528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:00.400 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.403+0000 7feeee59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feef013dc50 0x7feef01528c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:46496/0 (socket says 172.21.15.55:46496) 2024-03-19T01:59:00.400 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.403+0000 7feeee59c640 1 -- 172.21.15.55:0/2392625584 learned_addr learned my addr 172.21.15.55:0/2392625584 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:00.400 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.403+0000 7feeedd9b640 1 --2- 172.21.15.55:0/2392625584 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feef013e030 0x7feef00c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:00.400 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.403+0000 7feeeed9d640 1 --2- 172.21.15.55:0/2392625584 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7feef015b980 0x7feef01f0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:00.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.407+0000 7feeee59c640 1 -- 172.21.15.55:0/2392625584 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7feef015b980 msgr2=0x7feef01f0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:00.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.407+0000 7feeee59c640 1 --2- 172.21.15.55:0/2392625584 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7feef015b980 0x7feef01f0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:00.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.407+0000 7feeee59c640 1 -- 172.21.15.55:0/2392625584 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feef013e030 msgr2=0x7feef00c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:00.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.407+0000 7feeee59c640 1 --2- 172.21.15.55:0/2392625584 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feef013e030 0x7feef00c8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:00.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.407+0000 7feeee59c640 1 -- 172.21.15.55:0/2392625584 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef0152e00 con 0x7feef013dc50 2024-03-19T01:59:00.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.407+0000 7feeee59c640 1 --2- 172.21.15.55:0/2392625584 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feef013dc50 0x7feef01528c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7feee00b3040 tx=0x7feee001bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:00.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.411+0000 7feedeffd640 1 -- 172.21.15.55:0/2392625584 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee00b5070 con 0x7feef013dc50 2024-03-19T01:59:00.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.411+0000 7feedeffd640 1 -- 172.21.15.55:0/2392625584 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feee00b0950 con 0x7feef013dc50 2024-03-19T01:59:00.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.411+0000 7feef4ae9640 1 -- 172.21.15.55:0/2392625584 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feef01548d0 con 0x7feef013dc50 2024-03-19T01:59:00.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.411+0000 7feedeffd640 1 -- 172.21.15.55:0/2392625584 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee00b0d40 con 0x7feef013dc50 2024-03-19T01:59:00.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.411+0000 7feef4ae9640 1 -- 172.21.15.55:0/2392625584 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feef015aa50 con 0x7feef013dc50 2024-03-19T01:59:00.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.415+0000 7feef4ae9640 1 -- 172.21.15.55:0/2392625584 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feef0068a40 con 0x7feef013dc50 2024-03-19T01:59:00.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.439+0000 7feedeffd640 1 -- 172.21.15.55:0/2392625584 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7feee00c2060 con 0x7feef013dc50 2024-03-19T01:59:00.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.439+0000 7feedeffd640 1 --2- 172.21.15.55:0/2392625584 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7feed00676e0 0x7feed0069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:00.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.439+0000 7feeedd9b640 1 --2- 172.21.15.55:0/2392625584 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7feed00676e0 0x7feed0069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:00.437 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.439+0000 7feeedd9b640 1 --2- 172.21.15.55:0/2392625584 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7feed00676e0 0x7feed0069ba0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7feef00ca9c0 tx=0x7feee4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:00.439 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.443+0000 7feedeffd640 1 -- 172.21.15.55:0/2392625584 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7feee00f11f0 con 0x7feef013dc50 2024-03-19T01:59:00.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.731+0000 7feedeffd640 1 -- 172.21.15.55:0/2392625584 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7feee0018870 con 0x7feef013dc50 2024-03-19T01:59:00.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:00.951+0000 7feef4ae9640 1 -- 172.21.15.55:0/2392625584 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feef0070380 con 0x7feef013dc50 2024-03-19T01:59:01.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:01.027+0000 7feedeffd640 1 -- 172.21.15.55:0/2392625584 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7feee00f3150 con 0x7feef013dc50 2024-03-19T01:59:01.024 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:01.024 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:01.024 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:01.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:01.031+0000 7feef4ae9640 1 -- 172.21.15.55:0/2392625584 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7feed00676e0 msgr2=0x7feed0069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:01.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:01.031+0000 7feef4ae9640 1 --2- 172.21.15.55:0/2392625584 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7feed00676e0 0x7feed0069ba0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7feef00ca9c0 tx=0x7feee4009190 comp rx=0 tx=0).stop 2024-03-19T01:59:01.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:01.031+0000 7feef4ae9640 1 -- 172.21.15.55:0/2392625584 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feef013dc50 msgr2=0x7feef01528c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:01.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:01.031+0000 7feef4ae9640 1 --2- 172.21.15.55:0/2392625584 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feef013dc50 0x7feef01528c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7feee00b3040 tx=0x7feee001bab0 comp rx=0 tx=0).stop 2024-03-19T01:59:01.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:01.031+0000 7feef4ae9640 1 -- 172.21.15.55:0/2392625584 shutdown_connections 2024-03-19T01:59:01.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:01.031+0000 7feef4ae9640 1 --2- 172.21.15.55:0/2392625584 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7feed00676e0 0x7feed0069ba0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:01.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:01.031+0000 7feef4ae9640 1 --2- 172.21.15.55:0/2392625584 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7feef015b980 0x7feef01f0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:01.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:01.031+0000 7feef4ae9640 1 --2- 172.21.15.55:0/2392625584 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feef013e030 0x7feef00c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:01.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:01.031+0000 7feef4ae9640 1 --2- 172.21.15.55:0/2392625584 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feef013dc50 0x7feef01528c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:01.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:01.031+0000 7feef4ae9640 1 -- 172.21.15.55:0/2392625584 >> 172.21.15.55:0/2392625584 conn(0x7feef006fb40 msgr2=0x7feef00cb320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:01.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:01.031+0000 7feef4ae9640 1 -- 172.21.15.55:0/2392625584 shutdown_connections 2024-03-19T01:59:01.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:01.031+0000 7feef4ae9640 1 -- 172.21.15.55:0/2392625584 wait complete. 2024-03-19T01:59:01.649 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:01.649 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:02.650 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:02.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.783+0000 7fdfc8a3f640 1 Processor -- start 2024-03-19T01:59:02.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.783+0000 7fdfc8a3f640 1 -- start start 2024-03-19T01:59:02.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.783+0000 7fdfc8a3f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdfc4154720 0x7fdfc415e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:02.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.783+0000 7fdfc8a3f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fdfc40c73c0 0x7fdfc415aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:02.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.783+0000 7fdfc8a3f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfc4071280 con 0x7fdfc4154340 2024-03-19T01:59:02.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.783+0000 7fdfc8a3f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fdfc4073840 con 0x7fdfc40c73c0 2024-03-19T01:59:02.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.783+0000 7fdfc8a3f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fdfc415df10 con 0x7fdfc4154720 2024-03-19T01:59:02.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.783+0000 7fdfc2d9d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fdfc40c73c0 0x7fdfc415aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:02.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.783+0000 7fdfc1d9b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdfc4154720 0x7fdfc415e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:02.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.783+0000 7fdfc259c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fdfc4154340 0x7fdfc4072e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:58132/0 (socket says 172.21.15.55:58132) 2024-03-19T01:59:02.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.783+0000 7fdfc259c640 1 -- 172.21.15.55:0/3910115003 learned_addr learned my addr 172.21.15.55:0/3910115003 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:59:02.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.787+0000 7fdfc2d9d640 1 -- 172.21.15.55:0/3910115003 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdfc4154720 msgr2=0x7fdfc415e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:02.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.787+0000 7fdfc2d9d640 1 --2- 172.21.15.55:0/3910115003 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdfc4154720 0x7fdfc415e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:02.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.787+0000 7fdfc2d9d640 1 -- 172.21.15.55:0/3910115003 >> v1:172.21.15.84:6789/0 conn(0x7fdfc4154340 legacy=0x7fdfc4072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:02.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.787+0000 7fdfc2d9d640 1 -- 172.21.15.55:0/3910115003 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc415d900 con 0x7fdfc40c73c0 2024-03-19T01:59:02.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.791+0000 7fdfc2d9d640 1 --2- 172.21.15.55:0/3910115003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfc40c73c0 0x7fdfc415aca0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009d90 tx=0x7fdfbc01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a2f75c4bafceafe server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:02.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.795+0000 7fdfc159a640 1 -- 172.21.15.55:0/3910115003 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc0b2070 con 0x7fdfc40c73c0 2024-03-19T01:59:02.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.795+0000 7fdfc159a640 1 -- 172.21.15.55:0/3910115003 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfbc0a5830 con 0x7fdfc40c73c0 2024-03-19T01:59:02.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.795+0000 7fdfc159a640 1 -- 172.21.15.55:0/3910115003 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc0a5c00 con 0x7fdfc40c73c0 2024-03-19T01:59:02.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.795+0000 7fdfc8a3f640 1 -- 172.21.15.55:0/3910115003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfc40c73c0 msgr2=0x7fdfc415aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:02.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.795+0000 7fdfc8a3f640 1 --2- 172.21.15.55:0/3910115003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfc40c73c0 0x7fdfc415aca0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009d90 tx=0x7fdfbc01c970 comp rx=0 tx=0).stop 2024-03-19T01:59:02.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.795+0000 7fdfc8a3f640 1 -- 172.21.15.55:0/3910115003 shutdown_connections 2024-03-19T01:59:02.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.795+0000 7fdfc8a3f640 1 --2- 172.21.15.55:0/3910115003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfc40c73c0 0x7fdfc415aca0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:02.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.795+0000 7fdfc8a3f640 1 --2- 172.21.15.55:0/3910115003 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdfc4154720 0x7fdfc415e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:02.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.795+0000 7fdfc8a3f640 1 -- 172.21.15.55:0/3910115003 >> 172.21.15.55:0/3910115003 conn(0x7fdfc40cc9c0 msgr2=0x7fdfc40ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:02.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.795+0000 7fdfc8a3f640 1 -- 172.21.15.55:0/3910115003 shutdown_connections 2024-03-19T01:59:02.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.795+0000 7fdfc8a3f640 1 -- 172.21.15.55:0/3910115003 wait complete. 2024-03-19T01:59:02.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.799+0000 7fdfc8a3f640 1 Processor -- start 2024-03-19T01:59:02.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.799+0000 7fdfc8a3f640 1 -- start start 2024-03-19T01:59:02.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.799+0000 7fdfc8a3f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfc40c73c0 0x7fdfc41f7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:02.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.799+0000 7fdfc8a3f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdfc4154340 0x7fdfc41f3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:02.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.799+0000 7fdfc8a3f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdfc4154720 0x7fdfc414a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:02.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.799+0000 7fdfc8a3f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc4166b70 con 0x7fdfc40c73c0 2024-03-19T01:59:02.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.799+0000 7fdfc8a3f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc415b1e0 con 0x7fdfc4154720 2024-03-19T01:59:02.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.799+0000 7fdfc8a3f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fdfc4166cf0 con 0x7fdfc4154340 2024-03-19T01:59:02.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.799+0000 7fdfc259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfc40c73c0 0x7fdfc41f7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:02.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.799+0000 7fdfc259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfc40c73c0 0x7fdfc41f7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:46516/0 (socket says 172.21.15.55:46516) 2024-03-19T01:59:02.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.799+0000 7fdfc259c640 1 -- 172.21.15.55:0/1227852039 learned_addr learned my addr 172.21.15.55:0/1227852039 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:02.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.799+0000 7fdfc2d9d640 1 --2- 172.21.15.55:0/1227852039 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdfc4154720 0x7fdfc414a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:02.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.803+0000 7fdfc1d9b640 1 --2- 172.21.15.55:0/1227852039 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdfc4154340 0x7fdfc41f3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:02.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.803+0000 7fdfc259c640 1 -- 172.21.15.55:0/1227852039 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdfc4154340 msgr2=0x7fdfc41f3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:02.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.803+0000 7fdfc259c640 1 --2- 172.21.15.55:0/1227852039 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdfc4154340 0x7fdfc41f3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:02.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.803+0000 7fdfc259c640 1 -- 172.21.15.55:0/1227852039 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdfc4154720 msgr2=0x7fdfc414a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:02.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.803+0000 7fdfc259c640 1 --2- 172.21.15.55:0/1227852039 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdfc4154720 0x7fdfc414a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:02.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.803+0000 7fdfc259c640 1 -- 172.21.15.55:0/1227852039 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc41f6590 con 0x7fdfc40c73c0 2024-03-19T01:59:02.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.807+0000 7fdfc259c640 1 --2- 172.21.15.55:0/1227852039 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfc40c73c0 0x7fdfc41f7030 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fdfb400c420 tx=0x7fdfb400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:02.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.811+0000 7fdfb2ffd640 1 -- 172.21.15.55:0/1227852039 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb4017070 con 0x7fdfc40c73c0 2024-03-19T01:59:02.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.811+0000 7fdfb2ffd640 1 -- 172.21.15.55:0/1227852039 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfb4005cf0 con 0x7fdfc40c73c0 2024-03-19T01:59:02.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.811+0000 7fdfb2ffd640 1 -- 172.21.15.55:0/1227852039 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb40072f0 con 0x7fdfc40c73c0 2024-03-19T01:59:02.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.811+0000 7fdfc8a3f640 1 -- 172.21.15.55:0/1227852039 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfc41f3890 con 0x7fdfc40c73c0 2024-03-19T01:59:02.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.811+0000 7fdfc8a3f640 1 -- 172.21.15.55:0/1227852039 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfc41f3e50 con 0x7fdfc40c73c0 2024-03-19T01:59:02.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.811+0000 7fdfc8a3f640 1 -- 172.21.15.55:0/1227852039 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfc4068a40 con 0x7fdfc40c73c0 2024-03-19T01:59:02.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.847+0000 7fdfb2ffd640 1 -- 172.21.15.55:0/1227852039 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fdfb401a070 con 0x7fdfc40c73c0 2024-03-19T01:59:02.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.847+0000 7fdfb2ffd640 1 --2- 172.21.15.55:0/1227852039 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdfa4067690 0x7fdfa4069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:02.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.847+0000 7fdfc1d9b640 1 --2- 172.21.15.55:0/1227852039 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdfa4067690 0x7fdfa4069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:02.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.851+0000 7fdfc1d9b640 1 --2- 172.21.15.55:0/1227852039 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdfa4067690 0x7fdfa4069b50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdfc414da30 tx=0x7fdfb8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:02.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:02.851+0000 7fdfb2ffd640 1 -- 172.21.15.55:0/1227852039 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fdfb4053700 con 0x7fdfc40c73c0 2024-03-19T01:59:03.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:03.139+0000 7fdfb2ffd640 1 -- 172.21.15.55:0/1227852039 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fdfb4021020 con 0x7fdfc40c73c0 2024-03-19T01:59:03.342 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:03.343+0000 7fdfc8a3f640 1 -- 172.21.15.55:0/1227852039 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfc41f35b0 con 0x7fdfc40c73c0 2024-03-19T01:59:03.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:03.455+0000 7fdfb2ffd640 1 -- 172.21.15.55:0/1227852039 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fdfb4055660 con 0x7fdfc40c73c0 2024-03-19T01:59:03.452 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:03.452 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:03.453 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:03.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:03.459+0000 7fdfc8a3f640 1 -- 172.21.15.55:0/1227852039 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdfa4067690 msgr2=0x7fdfa4069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:03.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:03.459+0000 7fdfc8a3f640 1 --2- 172.21.15.55:0/1227852039 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdfa4067690 0x7fdfa4069b50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdfc414da30 tx=0x7fdfb8008040 comp rx=0 tx=0).stop 2024-03-19T01:59:03.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:03.459+0000 7fdfc8a3f640 1 -- 172.21.15.55:0/1227852039 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfc40c73c0 msgr2=0x7fdfc41f7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:03.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:03.459+0000 7fdfc8a3f640 1 --2- 172.21.15.55:0/1227852039 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfc40c73c0 0x7fdfc41f7030 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fdfb400c420 tx=0x7fdfb400c8e0 comp rx=0 tx=0).stop 2024-03-19T01:59:03.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:03.459+0000 7fdfc8a3f640 1 -- 172.21.15.55:0/1227852039 shutdown_connections 2024-03-19T01:59:03.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:03.459+0000 7fdfc8a3f640 1 --2- 172.21.15.55:0/1227852039 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdfa4067690 0x7fdfa4069b50 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:03.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:03.459+0000 7fdfc8a3f640 1 --2- 172.21.15.55:0/1227852039 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdfc4154720 0x7fdfc414a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:03.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:03.459+0000 7fdfc8a3f640 1 --2- 172.21.15.55:0/1227852039 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdfc4154340 0x7fdfc41f3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:03.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:03.459+0000 7fdfc8a3f640 1 --2- 172.21.15.55:0/1227852039 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdfc40c73c0 0x7fdfc41f7030 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:03.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:03.459+0000 7fdfc8a3f640 1 -- 172.21.15.55:0/1227852039 >> 172.21.15.55:0/1227852039 conn(0x7fdfc40cc9c0 msgr2=0x7fdfc40ca8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:03.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:03.459+0000 7fdfc8a3f640 1 -- 172.21.15.55:0/1227852039 shutdown_connections 2024-03-19T01:59:03.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:03.459+0000 7fdfc8a3f640 1 -- 172.21.15.55:0/1227852039 wait complete. 2024-03-19T01:59:04.080 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:04.080 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:05.081 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:05.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.195+0000 7f2a7f8c9640 1 Processor -- start 2024-03-19T01:59:05.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.195+0000 7f2a7f8c9640 1 -- start start 2024-03-19T01:59:05.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.199+0000 7f2a7f8c9640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2a780c96b0 0x7f2a78070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:05.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.199+0000 7f2a7f8c9640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f2a780c9a90 0x7f2a780c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:05.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.199+0000 7f2a7f8c9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a7815e3a0 con 0x7f2a781540c0 2024-03-19T01:59:05.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.199+0000 7f2a7f8c9640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f2a7815e680 con 0x7f2a780c96b0 2024-03-19T01:59:05.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.199+0000 7f2a7f8c9640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f2a7815a330 con 0x7f2a780c9a90 2024-03-19T01:59:05.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.199+0000 7f2a7ce64640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f2a780c9a90 0x7f2a780c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:05.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.199+0000 7f2a7d665640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2a780c96b0 0x7f2a78070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:05.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.199+0000 7f2a7de66640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f2a781540c0 0x7f2a7815d810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:58142/0 (socket says 172.21.15.55:58142) 2024-03-19T01:59:05.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.199+0000 7f2a7de66640 1 -- 172.21.15.55:0/637562887 learned_addr learned my addr 172.21.15.55:0/637562887 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:59:05.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.203+0000 7f2a7d665640 1 -- 172.21.15.55:0/637562887 >> v2:172.21.15.84:3300/0 conn(0x7f2a780c9a90 msgr2=0x7f2a780c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:05.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.203+0000 7f2a7d665640 1 --2- 172.21.15.55:0/637562887 >> v2:172.21.15.84:3300/0 conn(0x7f2a780c9a90 0x7f2a780c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:05.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.203+0000 7f2a7d665640 1 -- 172.21.15.55:0/637562887 >> v1:172.21.15.84:6789/0 conn(0x7f2a781540c0 legacy=0x7f2a7815d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:05.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.203+0000 7f2a7d665640 1 -- 172.21.15.55:0/637562887 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a7815a610 con 0x7f2a780c96b0 2024-03-19T01:59:05.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.203+0000 7f2a7d665640 1 --2- 172.21.15.55:0/637562887 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a780c96b0 0x7f2a78070f20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2a780c9420 tx=0x7f2a7401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcdefd0c33f55a0a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:05.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.211+0000 7f2a6ffff640 1 -- 172.21.15.55:0/637562887 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a740b0070 con 0x7f2a780c96b0 2024-03-19T01:59:05.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.211+0000 7f2a6ffff640 1 -- 172.21.15.55:0/637562887 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a740ab600 con 0x7f2a780c96b0 2024-03-19T01:59:05.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.211+0000 7f2a6ffff640 1 -- 172.21.15.55:0/637562887 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a740ab9d0 con 0x7f2a780c96b0 2024-03-19T01:59:05.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.211+0000 7f2a7f8c9640 1 -- 172.21.15.55:0/637562887 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a780c96b0 msgr2=0x7f2a78070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:05.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.211+0000 7f2a7f8c9640 1 --2- 172.21.15.55:0/637562887 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a780c96b0 0x7f2a78070f20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2a780c9420 tx=0x7f2a7401c3f0 comp rx=0 tx=0).stop 2024-03-19T01:59:05.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.211+0000 7f2a7f8c9640 1 -- 172.21.15.55:0/637562887 shutdown_connections 2024-03-19T01:59:05.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.211+0000 7f2a7f8c9640 1 --2- 172.21.15.55:0/637562887 >> v2:172.21.15.84:3300/0 conn(0x7f2a780c9a90 0x7f2a780c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:05.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.211+0000 7f2a7f8c9640 1 --2- 172.21.15.55:0/637562887 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a780c96b0 0x7f2a78070f20 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:05.209 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.211+0000 7f2a7f8c9640 1 -- 172.21.15.55:0/637562887 >> 172.21.15.55:0/637562887 conn(0x7f2a780cc9c0 msgr2=0x7f2a780ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:05.209 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.211+0000 7f2a7f8c9640 1 -- 172.21.15.55:0/637562887 shutdown_connections 2024-03-19T01:59:05.209 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.211+0000 7f2a7f8c9640 1 -- 172.21.15.55:0/637562887 wait complete. 2024-03-19T01:59:05.209 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.211+0000 7f2a7f8c9640 1 Processor -- start 2024-03-19T01:59:05.209 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.211+0000 7f2a7f8c9640 1 -- start start 2024-03-19T01:59:05.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.211+0000 7f2a7f8c9640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a780c96b0 0x7f2a78146740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:05.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.211+0000 7f2a7f8c9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2a780c9a90 0x7f2a7814f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:05.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.211+0000 7f2a7f8c9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a781540c0 0x7f2a7814fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:05.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.211+0000 7f2a7f8c9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2a780c7900 con 0x7f2a781540c0 2024-03-19T01:59:05.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.211+0000 7f2a7f8c9640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2a781548e0 con 0x7f2a780c96b0 2024-03-19T01:59:05.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.211+0000 7f2a7f8c9640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2a780c7a80 con 0x7f2a780c9a90 2024-03-19T01:59:05.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.215+0000 7f2a7de66640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a781540c0 0x7f2a7814fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:05.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.215+0000 7f2a7d665640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a780c96b0 0x7f2a78146740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:05.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.215+0000 7f2a7ce64640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2a780c9a90 0x7f2a7814f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:05.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.215+0000 7f2a7de66640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a781540c0 0x7f2a7814fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:46554/0 (socket says 172.21.15.55:46554) 2024-03-19T01:59:05.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.215+0000 7f2a7de66640 1 -- 172.21.15.55:0/2303726083 learned_addr learned my addr 172.21.15.55:0/2303726083 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:05.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.219+0000 7f2a7de66640 1 -- 172.21.15.55:0/2303726083 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2a780c9a90 msgr2=0x7f2a7814f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:05.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.219+0000 7f2a7de66640 1 --2- 172.21.15.55:0/2303726083 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2a780c9a90 0x7f2a7814f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:05.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.219+0000 7f2a7de66640 1 -- 172.21.15.55:0/2303726083 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a780c96b0 msgr2=0x7f2a78146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:05.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.219+0000 7f2a7de66640 1 --2- 172.21.15.55:0/2303726083 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a780c96b0 0x7f2a78146740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:05.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.219+0000 7f2a7de66640 1 -- 172.21.15.55:0/2303726083 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a7815a0d0 con 0x7f2a781540c0 2024-03-19T01:59:05.217 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.219+0000 7f2a7de66640 1 --2- 172.21.15.55:0/2303726083 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a781540c0 0x7f2a7814fb30 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2a7000ca80 tx=0x7f2a7000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:05.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.223+0000 7f2a6dffb640 1 -- 172.21.15.55:0/2303726083 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a70017070 con 0x7f2a781540c0 2024-03-19T01:59:05.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.223+0000 7f2a6dffb640 1 -- 172.21.15.55:0/2303726083 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a7000ac30 con 0x7f2a781540c0 2024-03-19T01:59:05.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.223+0000 7f2a7f8c9640 1 -- 172.21.15.55:0/2303726083 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a781f7220 con 0x7f2a781540c0 2024-03-19T01:59:05.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.223+0000 7f2a6dffb640 1 -- 172.21.15.55:0/2303726083 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a70012600 con 0x7f2a781540c0 2024-03-19T01:59:05.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.223+0000 7f2a7f8c9640 1 -- 172.21.15.55:0/2303726083 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a781f34e0 con 0x7f2a781540c0 2024-03-19T01:59:05.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.227+0000 7f2a7f8c9640 1 -- 172.21.15.55:0/2303726083 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a78068a40 con 0x7f2a781540c0 2024-03-19T01:59:05.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.515+0000 7f2a6dffb640 1 -- 172.21.15.55:0/2303726083 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f2a7001a070 con 0x7f2a781540c0 2024-03-19T01:59:05.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.515+0000 7f2a6dffb640 1 --2- 172.21.15.55:0/2303726083 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2a5c0676e0 0x7f2a5c069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:05.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.515+0000 7f2a7d665640 1 --2- 172.21.15.55:0/2303726083 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2a5c0676e0 0x7f2a5c069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:05.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.515+0000 7f2a7d665640 1 --2- 172.21.15.55:0/2303726083 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2a5c0676e0 0x7f2a5c069ba0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2a740ad000 tx=0x7f2a740a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:05.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.555+0000 7f2a6dffb640 1 -- 172.21.15.55:0/2303726083 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f2a70054040 con 0x7f2a781540c0 2024-03-19T01:59:05.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.559+0000 7f2a6dffb640 1 -- 172.21.15.55:0/2303726083 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2a7000ae20 con 0x7f2a781540c0 2024-03-19T01:59:05.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.815+0000 7f2a7f8c9640 1 -- 172.21.15.55:0/2303726083 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a780c7f80 con 0x7f2a781540c0 2024-03-19T01:59:05.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.943+0000 7f2a6dffb640 1 -- 172.21.15.55:0/2303726083 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f2a70055fa0 con 0x7f2a781540c0 2024-03-19T01:59:05.940 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:05.940 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:05.940 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:05.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.947+0000 7f2a7f8c9640 1 -- 172.21.15.55:0/2303726083 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2a5c0676e0 msgr2=0x7f2a5c069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:05.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.947+0000 7f2a7f8c9640 1 --2- 172.21.15.55:0/2303726083 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2a5c0676e0 0x7f2a5c069ba0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2a740ad000 tx=0x7f2a740a6000 comp rx=0 tx=0).stop 2024-03-19T01:59:05.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.947+0000 7f2a7f8c9640 1 -- 172.21.15.55:0/2303726083 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a781540c0 msgr2=0x7f2a7814fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:05.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.947+0000 7f2a7f8c9640 1 --2- 172.21.15.55:0/2303726083 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a781540c0 0x7f2a7814fb30 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2a7000ca80 tx=0x7f2a7000cf40 comp rx=0 tx=0).stop 2024-03-19T01:59:05.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.947+0000 7f2a7f8c9640 1 -- 172.21.15.55:0/2303726083 shutdown_connections 2024-03-19T01:59:05.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.947+0000 7f2a7f8c9640 1 --2- 172.21.15.55:0/2303726083 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2a5c0676e0 0x7f2a5c069ba0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:05.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.947+0000 7f2a7f8c9640 1 --2- 172.21.15.55:0/2303726083 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2a781540c0 0x7f2a7814fb30 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:05.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.947+0000 7f2a7f8c9640 1 --2- 172.21.15.55:0/2303726083 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2a780c9a90 0x7f2a7814f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:05.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.947+0000 7f2a7f8c9640 1 --2- 172.21.15.55:0/2303726083 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a780c96b0 0x7f2a78146740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:05.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.947+0000 7f2a7f8c9640 1 -- 172.21.15.55:0/2303726083 >> 172.21.15.55:0/2303726083 conn(0x7f2a780cc9c0 msgr2=0x7f2a780c54d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:05.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.947+0000 7f2a7f8c9640 1 -- 172.21.15.55:0/2303726083 shutdown_connections 2024-03-19T01:59:05.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:05.947+0000 7f2a7f8c9640 1 -- 172.21.15.55:0/2303726083 wait complete. 2024-03-19T01:59:06.564 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:06.565 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:07.565 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:07.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.695+0000 7fc27afce640 1 Processor -- start 2024-03-19T01:59:07.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.695+0000 7fc27afce640 1 -- start start 2024-03-19T01:59:07.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.699+0000 7fc27afce640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc2740c39c0 0x7fc274158a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:07.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.699+0000 7fc27afce640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fc274072c60 0x7fc27415b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:07.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.699+0000 7fc27afce640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc274152660 con 0x7fc27406b9f0 2024-03-19T01:59:07.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.699+0000 7fc27afce640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fc274073130 con 0x7fc2740c39c0 2024-03-19T01:59:07.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.699+0000 7fc27afce640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fc2740733f0 con 0x7fc274072c60 2024-03-19T01:59:07.696 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.699+0000 7fc278d6a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc2740c39c0 0x7fc274158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:07.696 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.699+0000 7fc273fff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fc274072c60 0x7fc27415b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:07.696 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.699+0000 7fc278d6a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc2740c39c0 0x7fc274158a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:46560/0 (socket says 172.21.15.55:46560) 2024-03-19T01:59:07.696 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.699+0000 7fc278d6a640 1 -- 172.21.15.55:0/3600353877 learned_addr learned my addr 172.21.15.55:0/3600353877 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:07.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.703+0000 7fc273fff640 1 -- 172.21.15.55:0/3600353877 >> v2:172.21.15.55:3300/0 conn(0x7fc2740c39c0 msgr2=0x7fc274158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:07.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.703+0000 7fc273fff640 1 --2- 172.21.15.55:0/3600353877 >> v2:172.21.15.55:3300/0 conn(0x7fc2740c39c0 0x7fc274158a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:07.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.703+0000 7fc273fff640 1 -- 172.21.15.55:0/3600353877 >> v1:172.21.15.55:6789/0 conn(0x7fc27406b9f0 legacy=0x7fc2740c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:07.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.703+0000 7fc273fff640 1 -- 172.21.15.55:0/3600353877 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2740736d0 con 0x7fc274072c60 2024-03-19T01:59:07.703 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.707+0000 7fc273fff640 1 --2- 172.21.15.55:0/3600353877 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc274072c60 0x7fc27415b540 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc274071fa0 tx=0x7fc26001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae7575930920f5a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:07.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.711+0000 7fc2737fe640 1 -- 172.21.15.55:0/3600353877 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2600b5070 con 0x7fc274072c60 2024-03-19T01:59:07.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.711+0000 7fc2737fe640 1 -- 172.21.15.55:0/3600353877 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2600a8970 con 0x7fc274072c60 2024-03-19T01:59:07.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.711+0000 7fc2737fe640 1 -- 172.21.15.55:0/3600353877 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2600a8d40 con 0x7fc274072c60 2024-03-19T01:59:07.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.711+0000 7fc27afce640 1 -- 172.21.15.55:0/3600353877 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc274072c60 msgr2=0x7fc27415b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:07.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.711+0000 7fc27afce640 1 --2- 172.21.15.55:0/3600353877 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc274072c60 0x7fc27415b540 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc274071fa0 tx=0x7fc26001ec70 comp rx=0 tx=0).stop 2024-03-19T01:59:07.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.715+0000 7fc27afce640 1 -- 172.21.15.55:0/3600353877 shutdown_connections 2024-03-19T01:59:07.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.715+0000 7fc27afce640 1 --2- 172.21.15.55:0/3600353877 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc274072c60 0x7fc27415b540 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:07.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.715+0000 7fc27afce640 1 --2- 172.21.15.55:0/3600353877 >> v2:172.21.15.55:3300/0 conn(0x7fc2740c39c0 0x7fc274158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:07.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.715+0000 7fc27afce640 1 -- 172.21.15.55:0/3600353877 >> 172.21.15.55:0/3600353877 conn(0x7fc2740cc9c0 msgr2=0x7fc2740ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:07.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.715+0000 7fc27afce640 1 -- 172.21.15.55:0/3600353877 shutdown_connections 2024-03-19T01:59:07.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.715+0000 7fc27afce640 1 -- 172.21.15.55:0/3600353877 wait complete. 2024-03-19T01:59:07.712 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.715+0000 7fc27afce640 1 Processor -- start 2024-03-19T01:59:07.712 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.715+0000 7fc27afce640 1 -- start start 2024-03-19T01:59:07.712 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.715+0000 7fc27afce640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc27406b9f0 0x7fc274139a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:07.713 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.715+0000 7fc27afce640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc274072c60 0x7fc2741ef5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:07.713 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.715+0000 7fc27afce640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2740c39c0 0x7fc2741efaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:07.713 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.715+0000 7fc27afce640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc274157f50 con 0x7fc27406b9f0 2024-03-19T01:59:07.713 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.715+0000 7fc27afce640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc274158560 con 0x7fc2740c39c0 2024-03-19T01:59:07.713 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.715+0000 7fc27afce640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc2740c67e0 con 0x7fc274072c60 2024-03-19T01:59:07.713 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.715+0000 7fc278d6a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc27406b9f0 0x7fc274139a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:07.714 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.715+0000 7fc27956b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2740c39c0 0x7fc2741efaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:07.714 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.715+0000 7fc278d6a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc27406b9f0 0x7fc274139a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:46576/0 (socket says 172.21.15.55:46576) 2024-03-19T01:59:07.714 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.715+0000 7fc278d6a640 1 -- 172.21.15.55:0/1435050630 learned_addr learned my addr 172.21.15.55:0/1435050630 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:07.714 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.715+0000 7fc273fff640 1 --2- 172.21.15.55:0/1435050630 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc274072c60 0x7fc2741ef5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:07.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.723+0000 7fc278d6a640 1 -- 172.21.15.55:0/1435050630 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc274072c60 msgr2=0x7fc2741ef5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:07.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.723+0000 7fc278d6a640 1 --2- 172.21.15.55:0/1435050630 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc274072c60 0x7fc2741ef5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:07.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.723+0000 7fc278d6a640 1 -- 172.21.15.55:0/1435050630 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2740c39c0 msgr2=0x7fc2741efaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:07.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.723+0000 7fc278d6a640 1 --2- 172.21.15.55:0/1435050630 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2740c39c0 0x7fc2741efaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:07.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.723+0000 7fc278d6a640 1 -- 172.21.15.55:0/1435050630 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2741eecf0 con 0x7fc27406b9f0 2024-03-19T01:59:07.722 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.723+0000 7fc278d6a640 1 --2- 172.21.15.55:0/1435050630 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc27406b9f0 0x7fc274139a40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc26800e7f0 tx=0x7fc26800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:07.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.731+0000 7fc2717fa640 1 -- 172.21.15.55:0/1435050630 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc268017070 con 0x7fc27406b9f0 2024-03-19T01:59:07.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.731+0000 7fc2717fa640 1 -- 172.21.15.55:0/1435050630 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc26800c940 con 0x7fc27406b9f0 2024-03-19T01:59:07.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.731+0000 7fc27afce640 1 -- 172.21.15.55:0/1435050630 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2741eef90 con 0x7fc27406b9f0 2024-03-19T01:59:07.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.731+0000 7fc2717fa640 1 -- 172.21.15.55:0/1435050630 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc26800cd30 con 0x7fc27406b9f0 2024-03-19T01:59:07.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.731+0000 7fc27afce640 1 -- 172.21.15.55:0/1435050630 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2741e20f0 con 0x7fc27406b9f0 2024-03-19T01:59:07.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:07.731+0000 7fc27afce640 1 -- 172.21.15.55:0/1435050630 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc274068a40 con 0x7fc27406b9f0 2024-03-19T01:59:08.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:08.035+0000 7fc2717fa640 1 -- 172.21.15.55:0/1435050630 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc26801a070 con 0x7fc27406b9f0 2024-03-19T01:59:08.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:08.035+0000 7fc2717fa640 1 --2- 172.21.15.55:0/1435050630 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc2580676b0 0x7fc258069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:08.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:08.035+0000 7fc273fff640 1 --2- 172.21.15.55:0/1435050630 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc2580676b0 0x7fc258069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:08.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:08.035+0000 7fc273fff640 1 --2- 172.21.15.55:0/1435050630 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc2580676b0 0x7fc258069b70 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc2600b2000 tx=0x7fc260008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:08.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:08.071+0000 7fc2717fa640 1 -- 172.21.15.55:0/1435050630 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fc2680545a0 con 0x7fc27406b9f0 2024-03-19T01:59:08.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:08.071+0000 7fc2717fa640 1 -- 172.21.15.55:0/1435050630 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc268056500 con 0x7fc27406b9f0 2024-03-19T01:59:08.284 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:08.287+0000 7fc27afce640 1 -- 172.21.15.55:0/1435050630 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc274072a40 con 0x7fc27406b9f0 2024-03-19T01:59:08.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:08.303+0000 7fc2717fa640 1 -- 172.21.15.55:0/1435050630 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fc26805a5a0 con 0x7fc27406b9f0 2024-03-19T01:59:08.299 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:08.299 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:08.300 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:08.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:08.303+0000 7fc27afce640 1 -- 172.21.15.55:0/1435050630 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc2580676b0 msgr2=0x7fc258069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:08.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:08.303+0000 7fc27afce640 1 --2- 172.21.15.55:0/1435050630 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc2580676b0 0x7fc258069b70 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc2600b2000 tx=0x7fc260008040 comp rx=0 tx=0).stop 2024-03-19T01:59:08.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:08.303+0000 7fc27afce640 1 -- 172.21.15.55:0/1435050630 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc27406b9f0 msgr2=0x7fc274139a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:08.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:08.303+0000 7fc27afce640 1 --2- 172.21.15.55:0/1435050630 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc27406b9f0 0x7fc274139a40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc26800e7f0 tx=0x7fc26800ecb0 comp rx=0 tx=0).stop 2024-03-19T01:59:08.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:08.303+0000 7fc27afce640 1 -- 172.21.15.55:0/1435050630 shutdown_connections 2024-03-19T01:59:08.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:08.303+0000 7fc27afce640 1 --2- 172.21.15.55:0/1435050630 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc2580676b0 0x7fc258069b70 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:08.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:08.303+0000 7fc27afce640 1 --2- 172.21.15.55:0/1435050630 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2740c39c0 0x7fc2741efaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:08.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:08.303+0000 7fc27afce640 1 --2- 172.21.15.55:0/1435050630 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc274072c60 0x7fc2741ef5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:08.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:08.303+0000 7fc27afce640 1 --2- 172.21.15.55:0/1435050630 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc27406b9f0 0x7fc274139a40 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:08.303 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:08.303+0000 7fc27afce640 1 -- 172.21.15.55:0/1435050630 >> 172.21.15.55:0/1435050630 conn(0x7fc2740cc9c0 msgr2=0x7fc2740c60d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:08.303 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:08.303+0000 7fc27afce640 1 -- 172.21.15.55:0/1435050630 shutdown_connections 2024-03-19T01:59:08.303 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:08.303+0000 7fc27afce640 1 -- 172.21.15.55:0/1435050630 wait complete. 2024-03-19T01:59:08.921 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:08.921 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:09.922 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:09.955 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:59:09.972 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:59:10.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.039+0000 7f5544851640 1 Processor -- start 2024-03-19T01:59:10.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.039+0000 7f5544851640 1 -- start start 2024-03-19T01:59:10.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.039+0000 7f5544851640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f554013de70 0x7f554006e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:10.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.039+0000 7f5544851640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f554015b740 con 0x7f554006eb70 2024-03-19T01:59:10.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.039+0000 7f5544851640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f554015ba20 con 0x7f554006f1b0 2024-03-19T01:59:10.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.039+0000 7f5544851640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f554013e250 con 0x7f554013de70 2024-03-19T01:59:10.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.039+0000 7f553dd9b640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f554006eb70 0x7f55400c5810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:33404/0 (socket says 172.21.15.55:33404) 2024-03-19T01:59:10.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.039+0000 7f553e59c640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f554013de70 0x7f554006e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:10.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.039+0000 7f553dd9b640 1 -- 172.21.15.55:0/2502340643 learned_addr learned my addr 172.21.15.55:0/2502340643 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:59:10.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.043+0000 7f553d59a640 1 -- 172.21.15.55:0/2502340643 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2865229581 0 0) 0x7f554015b740 con 0x7f554006eb70 2024-03-19T01:59:10.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.043+0000 7f553d59a640 1 -- 172.21.15.55:0/2502340643 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5528003530 con 0x7f554006eb70 2024-03-19T01:59:10.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.043+0000 7f553e59c640 1 -- 172.21.15.55:0/2502340643 >> v1:172.21.15.84:6789/0 conn(0x7f554006f1b0 legacy=0x7f5540158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:10.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.043+0000 7f553e59c640 1 -- 172.21.15.55:0/2502340643 >> v1:172.21.15.55:6789/0 conn(0x7f554006eb70 legacy=0x7f55400c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:10.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.043+0000 7f553e59c640 1 -- 172.21.15.55:0/2502340643 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f554015bd00 con 0x7f554013de70 2024-03-19T01:59:10.050 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.051+0000 7f553e59c640 1 --2- 172.21.15.55:0/2502340643 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f554013de70 0x7f554006e630 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f55300097c0 tx=0x7f553001ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b27ed1fa7e1f6b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:10.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.059+0000 7f553d59a640 1 -- 172.21.15.55:0/2502340643 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55300b5070 con 0x7f554013de70 2024-03-19T01:59:10.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.059+0000 7f553d59a640 1 -- 172.21.15.55:0/2502340643 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55300a8990 con 0x7f554013de70 2024-03-19T01:59:10.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.059+0000 7f553d59a640 1 -- 172.21.15.55:0/2502340643 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55300a8d60 con 0x7f554013de70 2024-03-19T01:59:10.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.059+0000 7f5544851640 1 -- 172.21.15.55:0/2502340643 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f554013de70 msgr2=0x7f554006e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:10.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.059+0000 7f5544851640 1 --2- 172.21.15.55:0/2502340643 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f554013de70 0x7f554006e630 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f55300097c0 tx=0x7f553001ec90 comp rx=0 tx=0).stop 2024-03-19T01:59:10.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.059+0000 7f5544851640 1 -- 172.21.15.55:0/2502340643 shutdown_connections 2024-03-19T01:59:10.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.059+0000 7f5544851640 1 --2- 172.21.15.55:0/2502340643 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f554013de70 0x7f554006e630 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:10.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.059+0000 7f5544851640 1 -- 172.21.15.55:0/2502340643 >> 172.21.15.55:0/2502340643 conn(0x7f554006fb40 msgr2=0x7f554006ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:10.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.059+0000 7f5544851640 1 -- 172.21.15.55:0/2502340643 shutdown_connections 2024-03-19T01:59:10.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.059+0000 7f5544851640 1 -- 172.21.15.55:0/2502340643 wait complete. 2024-03-19T01:59:10.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.059+0000 7f5544851640 1 Processor -- start 2024-03-19T01:59:10.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.059+0000 7f5544851640 1 -- start start 2024-03-19T01:59:10.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.063+0000 7f5544851640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f554006eb70 0x7f55401e3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:10.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.063+0000 7f5544851640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f554006f1b0 0x7f55401e35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:10.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.063+0000 7f5544851640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f554013de70 0x7f55401385c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:10.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.063+0000 7f5544851640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f55400c5f20 con 0x7f554006eb70 2024-03-19T01:59:10.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.063+0000 7f5544851640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f554015c2c0 con 0x7f554006f1b0 2024-03-19T01:59:10.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.063+0000 7f5544851640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f55400c60a0 con 0x7f554013de70 2024-03-19T01:59:10.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.063+0000 7f553e59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f554006eb70 0x7f55401e3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:10.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.063+0000 7f553dd9b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f554006f1b0 0x7f55401e35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:10.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.063+0000 7f553ed9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f554013de70 0x7f55401385c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:10.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.063+0000 7f553e59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f554006eb70 0x7f55401e3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41380/0 (socket says 172.21.15.55:41380) 2024-03-19T01:59:10.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.063+0000 7f553e59c640 1 -- 172.21.15.55:0/1135859199 learned_addr learned my addr 172.21.15.55:0/1135859199 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:10.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.067+0000 7f553e59c640 1 -- 172.21.15.55:0/1135859199 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f554013de70 msgr2=0x7f55401385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:10.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.067+0000 7f553e59c640 1 --2- 172.21.15.55:0/1135859199 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f554013de70 0x7f55401385c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:10.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.067+0000 7f553e59c640 1 -- 172.21.15.55:0/1135859199 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f554006f1b0 msgr2=0x7f55401e35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:10.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.067+0000 7f553e59c640 1 --2- 172.21.15.55:0/1135859199 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f554006f1b0 0x7f55401e35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:10.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.067+0000 7f553e59c640 1 -- 172.21.15.55:0/1135859199 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55401533a0 con 0x7f554006eb70 2024-03-19T01:59:10.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.071+0000 7f553e59c640 1 --2- 172.21.15.55:0/1135859199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f554006eb70 0x7f55401e3070 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f553001b8a0 tx=0x7f553001ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:10.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.075+0000 7f552effd640 1 -- 172.21.15.55:0/1135859199 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55300bf070 con 0x7f554006eb70 2024-03-19T01:59:10.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.075+0000 7f552effd640 1 -- 172.21.15.55:0/1135859199 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55300b0720 con 0x7f554006eb70 2024-03-19T01:59:10.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.075+0000 7f5544851640 1 -- 172.21.15.55:0/1135859199 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55401579f0 con 0x7f554006eb70 2024-03-19T01:59:10.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.075+0000 7f552effd640 1 -- 172.21.15.55:0/1135859199 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55300b5040 con 0x7f554006eb70 2024-03-19T01:59:10.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.075+0000 7f5544851640 1 -- 172.21.15.55:0/1135859199 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55401e2590 con 0x7f554006eb70 2024-03-19T01:59:10.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.075+0000 7f5544851640 1 -- 172.21.15.55:0/1135859199 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5540068a40 con 0x7f554006eb70 2024-03-19T01:59:10.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.099+0000 7f552effd640 1 -- 172.21.15.55:0/1135859199 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f55300c2070 con 0x7f554006eb70 2024-03-19T01:59:10.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.099+0000 7f552effd640 1 --2- 172.21.15.55:0/1135859199 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5520067690 0x7f5520069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:10.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.099+0000 7f553dd9b640 1 --2- 172.21.15.55:0/1135859199 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5520067690 0x7f5520069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:10.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.103+0000 7f553dd9b640 1 --2- 172.21.15.55:0/1135859199 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5520067690 0x7f5520069b50 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5534007fd0 tx=0x7f5534009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:10.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.419+0000 7f552effd640 1 -- 172.21.15.55:0/1135859199 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f55300f0740 con 0x7f554006eb70 2024-03-19T01:59:10.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.419+0000 7f552effd640 1 -- 172.21.15.55:0/1135859199 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f55300187f0 con 0x7f554006eb70 2024-03-19T01:59:10.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.627+0000 7f5544851640 1 -- 172.21.15.55:0/1135859199 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f554006ef90 con 0x7f554006eb70 2024-03-19T01:59:10.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.695+0000 7f552effd640 1 -- 172.21.15.55:0/1135859199 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f55300a8960 con 0x7f554006eb70 2024-03-19T01:59:10.695 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:10.695 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:10.695 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:10.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.699+0000 7f5544851640 1 -- 172.21.15.55:0/1135859199 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5520067690 msgr2=0x7f5520069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:10.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.699+0000 7f5544851640 1 --2- 172.21.15.55:0/1135859199 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5520067690 0x7f5520069b50 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5534007fd0 tx=0x7f5534009040 comp rx=0 tx=0).stop 2024-03-19T01:59:10.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.699+0000 7f5544851640 1 -- 172.21.15.55:0/1135859199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f554006eb70 msgr2=0x7f55401e3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:10.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.699+0000 7f5544851640 1 --2- 172.21.15.55:0/1135859199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f554006eb70 0x7f55401e3070 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f553001b8a0 tx=0x7f553001ba30 comp rx=0 tx=0).stop 2024-03-19T01:59:10.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.699+0000 7f5544851640 1 -- 172.21.15.55:0/1135859199 shutdown_connections 2024-03-19T01:59:10.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.699+0000 7f5544851640 1 --2- 172.21.15.55:0/1135859199 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5520067690 0x7f5520069b50 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:10.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.699+0000 7f5544851640 1 --2- 172.21.15.55:0/1135859199 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f554013de70 0x7f55401385c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:10.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.699+0000 7f5544851640 1 --2- 172.21.15.55:0/1135859199 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f554006f1b0 0x7f55401e35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:10.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.699+0000 7f5544851640 1 --2- 172.21.15.55:0/1135859199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f554006eb70 0x7f55401e3070 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:10.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.699+0000 7f5544851640 1 -- 172.21.15.55:0/1135859199 >> 172.21.15.55:0/1135859199 conn(0x7f554006fb40 msgr2=0x7f55400c9390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:10.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.699+0000 7f5544851640 1 -- 172.21.15.55:0/1135859199 shutdown_connections 2024-03-19T01:59:10.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:10.699+0000 7f5544851640 1 -- 172.21.15.55:0/1135859199 wait complete. 2024-03-19T01:59:11.322 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:11.323 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:12.323 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:12.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.455+0000 7fce06121640 1 Processor -- start 2024-03-19T01:59:12.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.455+0000 7fce06121640 1 -- start start 2024-03-19T01:59:12.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.455+0000 7fce06121640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fce00154340 0x7fce0015e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:12.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.455+0000 7fce06121640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fce00154720 0x7fce000c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:12.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.455+0000 7fce06121640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce00071280 con 0x7fce0015aca0 2024-03-19T01:59:12.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.455+0000 7fce06121640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fce00073840 con 0x7fce00154340 2024-03-19T01:59:12.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.455+0000 7fce06121640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fce0015df10 con 0x7fce00154720 2024-03-19T01:59:12.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.459+0000 7fcdff7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fce00154340 0x7fce0015e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:12.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.459+0000 7fcdfeffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fce00154720 0x7fce000c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:12.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.459+0000 7fcdff7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fce00154340 0x7fce0015e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41396/0 (socket says 172.21.15.55:41396) 2024-03-19T01:59:12.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.459+0000 7fcdff7fe640 1 -- 172.21.15.55:0/901309725 learned_addr learned my addr 172.21.15.55:0/901309725 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:12.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.463+0000 7fcdfeffd640 1 -- 172.21.15.55:0/901309725 >> v2:172.21.15.55:3300/0 conn(0x7fce00154340 msgr2=0x7fce0015e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:12.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.463+0000 7fcdfeffd640 1 --2- 172.21.15.55:0/901309725 >> v2:172.21.15.55:3300/0 conn(0x7fce00154340 0x7fce0015e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:12.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.463+0000 7fcdfeffd640 1 -- 172.21.15.55:0/901309725 >> v1:172.21.15.55:6789/0 conn(0x7fce0015aca0 legacy=0x7fce00072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:12.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.463+0000 7fcdfeffd640 1 -- 172.21.15.55:0/901309725 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce0015d900 con 0x7fce00154720 2024-03-19T01:59:12.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.467+0000 7fcdfeffd640 1 --2- 172.21.15.55:0/901309725 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fce00154720 0x7fce000c73c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcdf40097c0 tx=0x7fcdf401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9885a8ebfc160036 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:12.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.471+0000 7fcdfe7fc640 1 -- 172.21.15.55:0/901309725 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf40b2070 con 0x7fce00154720 2024-03-19T01:59:12.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.479+0000 7fcdfe7fc640 1 -- 172.21.15.55:0/901309725 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdf40ad640 con 0x7fce00154720 2024-03-19T01:59:12.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.479+0000 7fcdfe7fc640 1 -- 172.21.15.55:0/901309725 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf40ada10 con 0x7fce00154720 2024-03-19T01:59:12.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.479+0000 7fce06121640 1 -- 172.21.15.55:0/901309725 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fce00154720 msgr2=0x7fce000c73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:12.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.479+0000 7fce06121640 1 --2- 172.21.15.55:0/901309725 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fce00154720 0x7fce000c73c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcdf40097c0 tx=0x7fcdf401c4a0 comp rx=0 tx=0).stop 2024-03-19T01:59:12.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.479+0000 7fce06121640 1 -- 172.21.15.55:0/901309725 shutdown_connections 2024-03-19T01:59:12.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.479+0000 7fce06121640 1 --2- 172.21.15.55:0/901309725 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fce00154720 0x7fce000c73c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:12.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.479+0000 7fce06121640 1 --2- 172.21.15.55:0/901309725 >> v2:172.21.15.55:3300/0 conn(0x7fce00154340 0x7fce0015e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:12.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.479+0000 7fce06121640 1 -- 172.21.15.55:0/901309725 >> 172.21.15.55:0/901309725 conn(0x7fce000cc9c0 msgr2=0x7fce000ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:12.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.479+0000 7fce06121640 1 -- 172.21.15.55:0/901309725 shutdown_connections 2024-03-19T01:59:12.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.479+0000 7fce06121640 1 -- 172.21.15.55:0/901309725 wait complete. 2024-03-19T01:59:12.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.479+0000 7fce06121640 1 Processor -- start 2024-03-19T01:59:12.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.479+0000 7fce06121640 1 -- start start 2024-03-19T01:59:12.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.479+0000 7fce06121640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fce00154340 0x7fce0014d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:12.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.479+0000 7fce06121640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fce00154720 0x7fce00155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:12.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.479+0000 7fce06121640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fce0015aca0 0x7fce00155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:12.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.479+0000 7fce06121640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fce0015b200 con 0x7fce0015aca0 2024-03-19T01:59:12.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.479+0000 7fce06121640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fce0015b080 con 0x7fce00154340 2024-03-19T01:59:12.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.479+0000 7fce06121640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fce00166b70 con 0x7fce00154720 2024-03-19T01:59:12.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.483+0000 7fcdff7fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fce00154340 0x7fce0014d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:12.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.483+0000 7fcdff7fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fce00154340 0x7fce0014d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:55196/0 (socket says 172.21.15.55:55196) 2024-03-19T01:59:12.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.483+0000 7fcdff7fe640 1 -- 172.21.15.55:0/4057296322 learned_addr learned my addr 172.21.15.55:0/4057296322 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:12.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.483+0000 7fcdfeffd640 1 --2- 172.21.15.55:0/4057296322 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fce00154720 0x7fce00155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:12.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.487+0000 7fcdff7fe640 1 -- 172.21.15.55:0/4057296322 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fce00154720 msgr2=0x7fce00155520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:12.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.487+0000 7fcdff7fe640 1 --2- 172.21.15.55:0/4057296322 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fce00154720 0x7fce00155520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:12.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.487+0000 7fcdff7fe640 1 -- 172.21.15.55:0/4057296322 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fce0015aca0 msgr2=0x7fce00155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:12.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.487+0000 7fcdff7fe640 1 --2- 172.21.15.55:0/4057296322 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fce0015aca0 0x7fce00155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:12.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.487+0000 7fcdff7fe640 1 -- 172.21.15.55:0/4057296322 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce0014e080 con 0x7fce00154340 2024-03-19T01:59:12.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.495+0000 7fcdff7fe640 1 --2- 172.21.15.55:0/4057296322 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fce00154340 0x7fce0014d220 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcdf000e8f0 tx=0x7fcdf000edb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:12.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.499+0000 7fcde7fff640 1 -- 172.21.15.55:0/4057296322 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf0017070 con 0x7fce00154340 2024-03-19T01:59:12.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.499+0000 7fcde7fff640 1 -- 172.21.15.55:0/4057296322 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdf000c940 con 0x7fce00154340 2024-03-19T01:59:12.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.499+0000 7fce06121640 1 -- 172.21.15.55:0/4057296322 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce001f3b00 con 0x7fce00154340 2024-03-19T01:59:12.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.499+0000 7fcde7fff640 1 -- 172.21.15.55:0/4057296322 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf000cd30 con 0x7fce00154340 2024-03-19T01:59:12.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.499+0000 7fce06121640 1 -- 172.21.15.55:0/4057296322 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce001504b0 con 0x7fce00154340 2024-03-19T01:59:12.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.499+0000 7fce06121640 1 -- 172.21.15.55:0/4057296322 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce00068a40 con 0x7fce00154340 2024-03-19T01:59:12.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.783+0000 7fcde7fff640 1 -- 172.21.15.55:0/4057296322 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fcdf001a070 con 0x7fce00154340 2024-03-19T01:59:12.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.783+0000 7fcde7fff640 1 --2- 172.21.15.55:0/4057296322 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcdec0676e0 0x7fcdec069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:12.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.783+0000 7fcdfeffd640 1 --2- 172.21.15.55:0/4057296322 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcdec0676e0 0x7fcdec069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:12.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.783+0000 7fcdfeffd640 1 --2- 172.21.15.55:0/4057296322 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcdec0676e0 0x7fcdec069ba0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcdf40af000 tx=0x7fcdf4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:12.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.791+0000 7fcde7fff640 1 -- 172.21.15.55:0/4057296322 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fcdf00545f0 con 0x7fce00154340 2024-03-19T01:59:12.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:12.807+0000 7fcde7fff640 1 -- 172.21.15.55:0/4057296322 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fcdf0056550 con 0x7fce00154340 2024-03-19T01:59:13.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:13.047+0000 7fce06121640 1 -- 172.21.15.55:0/4057296322 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce00150ec0 con 0x7fce00154340 2024-03-19T01:59:13.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:13.103+0000 7fcde7fff640 1 -- 172.21.15.55:0/4057296322 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fcdf005a5f0 con 0x7fce00154340 2024-03-19T01:59:13.100 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:13.100 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:13.100 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:13.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:13.103+0000 7fce06121640 1 -- 172.21.15.55:0/4057296322 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcdec0676e0 msgr2=0x7fcdec069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:13.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:13.103+0000 7fce06121640 1 --2- 172.21.15.55:0/4057296322 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcdec0676e0 0x7fcdec069ba0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcdf40af000 tx=0x7fcdf4008040 comp rx=0 tx=0).stop 2024-03-19T01:59:13.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:13.103+0000 7fce06121640 1 -- 172.21.15.55:0/4057296322 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fce00154340 msgr2=0x7fce0014d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:13.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:13.103+0000 7fce06121640 1 --2- 172.21.15.55:0/4057296322 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fce00154340 0x7fce0014d220 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcdf000e8f0 tx=0x7fcdf000edb0 comp rx=0 tx=0).stop 2024-03-19T01:59:13.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:13.103+0000 7fce06121640 1 -- 172.21.15.55:0/4057296322 shutdown_connections 2024-03-19T01:59:13.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:13.103+0000 7fce06121640 1 --2- 172.21.15.55:0/4057296322 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcdec0676e0 0x7fcdec069ba0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:13.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:13.103+0000 7fce06121640 1 --2- 172.21.15.55:0/4057296322 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fce0015aca0 0x7fce00155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:13.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:13.103+0000 7fce06121640 1 --2- 172.21.15.55:0/4057296322 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fce00154720 0x7fce00155520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:13.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:13.103+0000 7fce06121640 1 --2- 172.21.15.55:0/4057296322 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fce00154340 0x7fce0014d220 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:13.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:13.103+0000 7fce06121640 1 -- 172.21.15.55:0/4057296322 >> 172.21.15.55:0/4057296322 conn(0x7fce000cc9c0 msgr2=0x7fce000c8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:13.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:13.103+0000 7fce06121640 1 -- 172.21.15.55:0/4057296322 shutdown_connections 2024-03-19T01:59:13.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:13.103+0000 7fce06121640 1 -- 172.21.15.55:0/4057296322 wait complete. 2024-03-19T01:59:13.728 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:13.728 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:14.729 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:14.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.850+0000 7fc9093a8640 1 Processor -- start 2024-03-19T01:59:14.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.850+0000 7fc9093a8640 1 -- start start 2024-03-19T01:59:14.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.850+0000 7fc9093a8640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fc904070970 0x7fc9040c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:14.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.850+0000 7fc9093a8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc90406f6d0 con 0x7fc904152150 2024-03-19T01:59:14.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.850+0000 7fc9093a8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc904070450 con 0x7fc9040c9600 2024-03-19T01:59:14.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.850+0000 7fc9093a8640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fc904071130 con 0x7fc904070970 2024-03-19T01:59:14.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.850+0000 7fc9037fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fc9040c9600 0x7fc9040c2b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:36294/0 (socket says 172.21.15.55:36294) 2024-03-19T01:59:14.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.850+0000 7fc9037fe640 1 -- 172.21.15.55:0/1127222933 learned_addr learned my addr 172.21.15.55:0/1127222933 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:59:14.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.850+0000 7fc9027fc640 1 --2- 172.21.15.55:0/1127222933 >> v2:172.21.15.84:3300/0 conn(0x7fc904070970 0x7fc9040c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:14.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.862+0000 7fc9027fc640 1 -- 172.21.15.55:0/1127222933 >> v1:172.21.15.84:6789/0 conn(0x7fc9040c9600 legacy=0x7fc9040c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:14.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.862+0000 7fc9027fc640 1 -- 172.21.15.55:0/1127222933 >> v1:172.21.15.55:6789/0 conn(0x7fc904152150 legacy=0x7fc904158a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:14.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.862+0000 7fc9027fc640 1 -- 172.21.15.55:0/1127222933 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc904070e50 con 0x7fc904070970 2024-03-19T01:59:14.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.870+0000 7fc9027fc640 1 --2- 172.21.15.55:0/1127222933 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc904070970 0x7fc9040c90c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc9040b31c0 tx=0x7fc8ec01ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13bd0a70f0fd2d90 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:14.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.878+0000 7fc901ffb640 1 -- 172.21.15.55:0/1127222933 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec0b5070 con 0x7fc904070970 2024-03-19T01:59:14.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.878+0000 7fc901ffb640 1 -- 172.21.15.55:0/1127222933 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8ec0a8950 con 0x7fc904070970 2024-03-19T01:59:14.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.878+0000 7fc901ffb640 1 -- 172.21.15.55:0/1127222933 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec0a8d20 con 0x7fc904070970 2024-03-19T01:59:14.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.878+0000 7fc9093a8640 1 -- 172.21.15.55:0/1127222933 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc904070970 msgr2=0x7fc9040c90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:14.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.878+0000 7fc9093a8640 1 --2- 172.21.15.55:0/1127222933 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc904070970 0x7fc9040c90c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc9040b31c0 tx=0x7fc8ec01ec50 comp rx=0 tx=0).stop 2024-03-19T01:59:14.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.882+0000 7fc9093a8640 1 -- 172.21.15.55:0/1127222933 shutdown_connections 2024-03-19T01:59:14.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.882+0000 7fc9093a8640 1 --2- 172.21.15.55:0/1127222933 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc904070970 0x7fc9040c90c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:14.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.882+0000 7fc9093a8640 1 -- 172.21.15.55:0/1127222933 >> 172.21.15.55:0/1127222933 conn(0x7fc904067600 msgr2=0x7fc9040714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:14.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.882+0000 7fc9093a8640 1 -- 172.21.15.55:0/1127222933 shutdown_connections 2024-03-19T01:59:14.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.882+0000 7fc9093a8640 1 -- 172.21.15.55:0/1127222933 wait complete. 2024-03-19T01:59:14.881 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.882+0000 7fc9093a8640 1 Processor -- start 2024-03-19T01:59:14.881 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.882+0000 7fc9093a8640 1 -- start start 2024-03-19T01:59:14.882 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.882+0000 7fc9093a8640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc904070970 0x7fc9041e3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:14.882 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.882+0000 7fc9093a8640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9040c9600 0x7fc9041463a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:14.882 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.882+0000 7fc9093a8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc904152150 0x7fc9041468e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:14.882 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.882+0000 7fc9093a8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc90406ff30 con 0x7fc904152150 2024-03-19T01:59:14.882 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.882+0000 7fc9093a8640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc90406fc50 con 0x7fc9040c9600 2024-03-19T01:59:14.882 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.882+0000 7fc9093a8640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc904152530 con 0x7fc904070970 2024-03-19T01:59:14.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.886+0000 7fc9037fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc904152150 0x7fc9041468e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:14.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.886+0000 7fc902ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc904070970 0x7fc9041e3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:14.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.886+0000 7fc9037fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc904152150 0x7fc9041468e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41422/0 (socket says 172.21.15.55:41422) 2024-03-19T01:59:14.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.886+0000 7fc9037fe640 1 -- 172.21.15.55:0/3791273497 learned_addr learned my addr 172.21.15.55:0/3791273497 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:14.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.886+0000 7fc9027fc640 1 --2- 172.21.15.55:0/3791273497 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9040c9600 0x7fc9041463a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:14.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.890+0000 7fc9037fe640 1 -- 172.21.15.55:0/3791273497 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc904070970 msgr2=0x7fc9041e3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:14.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.890+0000 7fc9037fe640 1 --2- 172.21.15.55:0/3791273497 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc904070970 0x7fc9041e3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:14.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.890+0000 7fc9037fe640 1 -- 172.21.15.55:0/3791273497 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9040c9600 msgr2=0x7fc9041463a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:14.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.890+0000 7fc9037fe640 1 --2- 172.21.15.55:0/3791273497 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9040c9600 0x7fc9041463a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:14.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.890+0000 7fc9037fe640 1 -- 172.21.15.55:0/3791273497 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc90406de40 con 0x7fc904152150 2024-03-19T01:59:14.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.890+0000 7fc9037fe640 1 --2- 172.21.15.55:0/3791273497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc904152150 0x7fc9041468e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc00ca30 tx=0x7fc8fc00cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:14.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.894+0000 7fc8fb7fe640 1 -- 172.21.15.55:0/3791273497 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc017070 con 0x7fc904152150 2024-03-19T01:59:14.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.894+0000 7fc8fb7fe640 1 -- 172.21.15.55:0/3791273497 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8fc00abd0 con 0x7fc904152150 2024-03-19T01:59:14.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.894+0000 7fc9093a8640 1 -- 172.21.15.55:0/3791273497 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc90406e3c0 con 0x7fc904152150 2024-03-19T01:59:14.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.894+0000 7fc8fb7fe640 1 -- 172.21.15.55:0/3791273497 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc012400 con 0x7fc904152150 2024-03-19T01:59:14.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.894+0000 7fc9093a8640 1 -- 172.21.15.55:0/3791273497 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9040cba80 con 0x7fc904152150 2024-03-19T01:59:14.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:14.898+0000 7fc9093a8640 1 -- 172.21.15.55:0/3791273497 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc904068a40 con 0x7fc904152150 2024-03-19T01:59:15.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:15.194+0000 7fc8fb7fe640 1 -- 172.21.15.55:0/3791273497 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc8fc01a070 con 0x7fc904152150 2024-03-19T01:59:15.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:15.194+0000 7fc8fb7fe640 1 --2- 172.21.15.55:0/3791273497 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc8e40676b0 0x7fc8e4069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:15.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:15.198+0000 7fc902ffd640 1 --2- 172.21.15.55:0/3791273497 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc8e40676b0 0x7fc8e4069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:15.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:15.198+0000 7fc902ffd640 1 --2- 172.21.15.55:0/3791273497 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc8e40676b0 0x7fc8e4069b70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc8f40098e0 tx=0x7fc8f4009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:15.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:15.218+0000 7fc8fb7fe640 1 -- 172.21.15.55:0/3791273497 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fc8fc054090 con 0x7fc904152150 2024-03-19T01:59:15.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:15.222+0000 7fc8fb7fe640 1 -- 172.21.15.55:0/3791273497 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc8fc021020 con 0x7fc904152150 2024-03-19T01:59:15.437 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:15.438+0000 7fc9093a8640 1 -- 172.21.15.55:0/3791273497 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9041559a0 con 0x7fc904152150 2024-03-19T01:59:15.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:15.478+0000 7fc8fb7fe640 1 -- 172.21.15.55:0/3791273497 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fc8fc055ff0 con 0x7fc904152150 2024-03-19T01:59:15.478 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:15.478 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:15.478 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:15.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:15.482+0000 7fc9093a8640 1 -- 172.21.15.55:0/3791273497 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc8e40676b0 msgr2=0x7fc8e4069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:15.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:15.482+0000 7fc9093a8640 1 --2- 172.21.15.55:0/3791273497 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc8e40676b0 0x7fc8e4069b70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc8f40098e0 tx=0x7fc8f4009060 comp rx=0 tx=0).stop 2024-03-19T01:59:15.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:15.482+0000 7fc9093a8640 1 -- 172.21.15.55:0/3791273497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc904152150 msgr2=0x7fc9041468e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:15.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:15.482+0000 7fc9093a8640 1 --2- 172.21.15.55:0/3791273497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc904152150 0x7fc9041468e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc00ca30 tx=0x7fc8fc00cef0 comp rx=0 tx=0).stop 2024-03-19T01:59:15.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:15.482+0000 7fc9093a8640 1 -- 172.21.15.55:0/3791273497 shutdown_connections 2024-03-19T01:59:15.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:15.482+0000 7fc9093a8640 1 --2- 172.21.15.55:0/3791273497 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc8e40676b0 0x7fc8e4069b70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:15.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:15.482+0000 7fc9093a8640 1 --2- 172.21.15.55:0/3791273497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc904152150 0x7fc9041468e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:15.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:15.482+0000 7fc9093a8640 1 --2- 172.21.15.55:0/3791273497 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9040c9600 0x7fc9041463a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:15.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:15.482+0000 7fc9093a8640 1 --2- 172.21.15.55:0/3791273497 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc904070970 0x7fc9041e3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:15.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:15.482+0000 7fc9093a8640 1 -- 172.21.15.55:0/3791273497 >> 172.21.15.55:0/3791273497 conn(0x7fc904067600 msgr2=0x7fc904156e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:15.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:15.482+0000 7fc9093a8640 1 -- 172.21.15.55:0/3791273497 shutdown_connections 2024-03-19T01:59:15.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:15.482+0000 7fc9093a8640 1 -- 172.21.15.55:0/3791273497 wait complete. 2024-03-19T01:59:16.101 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:16.101 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:17.103 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:17.232 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.234+0000 7f447018e640 1 Processor -- start 2024-03-19T01:59:17.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.234+0000 7f447018e640 1 -- start start 2024-03-19T01:59:17.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.234+0000 7f447018e640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f4468068350 0x7f446815e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:17.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.234+0000 7f447018e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f44680c6dc0 0x7f4468072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:17.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.234+0000 7f447018e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f446806d8f0 con 0x7f44681546e0 2024-03-19T01:59:17.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.234+0000 7f447018e640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f446815a3d0 con 0x7f44680c6dc0 2024-03-19T01:59:17.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.234+0000 7f447018e640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f44680c7280 con 0x7f4468068350 2024-03-19T01:59:17.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.234+0000 7f446e72b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f44680c6dc0 0x7f4468072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:17.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.234+0000 7f446d729640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f4468068350 0x7f446815e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:17.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.234+0000 7f446e72b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f44680c6dc0 0x7f4468072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41432/0 (socket says 172.21.15.55:41432) 2024-03-19T01:59:17.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.234+0000 7f446e72b640 1 -- 172.21.15.55:0/260341282 learned_addr learned my addr 172.21.15.55:0/260341282 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:17.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.238+0000 7f446e72b640 1 -- 172.21.15.55:0/260341282 >> v2:172.21.15.84:3300/0 conn(0x7f4468068350 msgr2=0x7f446815e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:17.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.238+0000 7f446e72b640 1 --2- 172.21.15.55:0/260341282 >> v2:172.21.15.84:3300/0 conn(0x7f4468068350 0x7f446815e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:17.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.238+0000 7f446e72b640 1 -- 172.21.15.55:0/260341282 >> v1:172.21.15.55:6789/0 conn(0x7f44681546e0 legacy=0x7f446815abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:17.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.238+0000 7f446e72b640 1 -- 172.21.15.55:0/260341282 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44680c7560 con 0x7f44680c6dc0 2024-03-19T01:59:17.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.246+0000 7f446e72b640 1 --2- 172.21.15.55:0/260341282 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44680c6dc0 0x7f4468072ee0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4468068040 tx=0x7f446001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f186022d63f0ad9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:17.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.246+0000 7f446cf28640 1 -- 172.21.15.55:0/260341282 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44600a57d0 con 0x7f44680c6dc0 2024-03-19T01:59:17.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.246+0000 7f446cf28640 1 -- 172.21.15.55:0/260341282 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44600a5970 con 0x7f44680c6dc0 2024-03-19T01:59:17.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.246+0000 7f446cf28640 1 -- 172.21.15.55:0/260341282 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44600a5d40 con 0x7f44680c6dc0 2024-03-19T01:59:17.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.250+0000 7f447018e640 1 -- 172.21.15.55:0/260341282 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44680c6dc0 msgr2=0x7f4468072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:17.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.250+0000 7f447018e640 1 --2- 172.21.15.55:0/260341282 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44680c6dc0 0x7f4468072ee0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4468068040 tx=0x7f446001c9f0 comp rx=0 tx=0).stop 2024-03-19T01:59:17.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.250+0000 7f447018e640 1 -- 172.21.15.55:0/260341282 shutdown_connections 2024-03-19T01:59:17.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.250+0000 7f447018e640 1 --2- 172.21.15.55:0/260341282 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44680c6dc0 0x7f4468072ee0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:17.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.250+0000 7f447018e640 1 --2- 172.21.15.55:0/260341282 >> v2:172.21.15.84:3300/0 conn(0x7f4468068350 0x7f446815e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:17.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.250+0000 7f447018e640 1 -- 172.21.15.55:0/260341282 >> 172.21.15.55:0/260341282 conn(0x7f44680c8990 msgr2=0x7f44680c8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:17.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.250+0000 7f447018e640 1 -- 172.21.15.55:0/260341282 shutdown_connections 2024-03-19T01:59:17.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.250+0000 7f447018e640 1 -- 172.21.15.55:0/260341282 wait complete. 2024-03-19T01:59:17.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.250+0000 7f447018e640 1 Processor -- start 2024-03-19T01:59:17.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.250+0000 7f447018e640 1 -- start start 2024-03-19T01:59:17.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.250+0000 7f447018e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4468068350 0x7f44681f35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:17.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.250+0000 7f447018e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44680c6dc0 0x7f4468074290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:17.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.250+0000 7f447018e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f44681546e0 0x7f44680747d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:17.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.250+0000 7f447018e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4468154ac0 con 0x7f4468068350 2024-03-19T01:59:17.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.250+0000 7f447018e640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f446815b2c0 con 0x7f44680c6dc0 2024-03-19T01:59:17.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.250+0000 7f447018e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4468167490 con 0x7f44681546e0 2024-03-19T01:59:17.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.250+0000 7f446df2a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4468068350 0x7f44681f35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:17.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.250+0000 7f446d729640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44680c6dc0 0x7f4468074290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:17.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.250+0000 7f446e72b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f44681546e0 0x7f44680747d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:17.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.250+0000 7f446df2a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4468068350 0x7f44681f35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41446/0 (socket says 172.21.15.55:41446) 2024-03-19T01:59:17.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.254+0000 7f446df2a640 1 -- 172.21.15.55:0/3624530416 learned_addr learned my addr 172.21.15.55:0/3624530416 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:17.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.258+0000 7f446df2a640 1 -- 172.21.15.55:0/3624530416 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f44681546e0 msgr2=0x7f44680747d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:17.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.258+0000 7f446df2a640 1 --2- 172.21.15.55:0/3624530416 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f44681546e0 0x7f44680747d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:17.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.258+0000 7f446df2a640 1 -- 172.21.15.55:0/3624530416 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44680c6dc0 msgr2=0x7f4468074290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:17.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.258+0000 7f446df2a640 1 --2- 172.21.15.55:0/3624530416 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44680c6dc0 0x7f4468074290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:17.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.258+0000 7f446df2a640 1 -- 172.21.15.55:0/3624530416 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44680ca300 con 0x7f4468068350 2024-03-19T01:59:17.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.262+0000 7f446df2a640 1 --2- 172.21.15.55:0/3624530416 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4468068350 0x7f44681f35a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f446400c3a0 tx=0x7f446400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:17.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.266+0000 7f445e7fc640 1 -- 172.21.15.55:0/3624530416 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44640052d0 con 0x7f4468068350 2024-03-19T01:59:17.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.266+0000 7f445e7fc640 1 -- 172.21.15.55:0/3624530416 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4464005470 con 0x7f4468068350 2024-03-19T01:59:17.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.266+0000 7f447018e640 1 -- 172.21.15.55:0/3624530416 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4468144bf0 con 0x7f4468068350 2024-03-19T01:59:17.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.266+0000 7f445e7fc640 1 -- 172.21.15.55:0/3624530416 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4464007260 con 0x7f4468068350 2024-03-19T01:59:17.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.266+0000 7f447018e640 1 -- 172.21.15.55:0/3624530416 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44680ca5a0 con 0x7f4468068350 2024-03-19T01:59:17.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.266+0000 7f447018e640 1 -- 172.21.15.55:0/3624530416 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4468068a40 con 0x7f4468068350 2024-03-19T01:59:17.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.294+0000 7f445e7fc640 1 -- 172.21.15.55:0/3624530416 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f4464007400 con 0x7f4468068350 2024-03-19T01:59:17.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.294+0000 7f445e7fc640 1 --2- 172.21.15.55:0/3624530416 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f444c0676e0 0x7f444c069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:17.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.298+0000 7f446d729640 1 --2- 172.21.15.55:0/3624530416 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f444c0676e0 0x7f444c069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:17.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.298+0000 7f446d729640 1 --2- 172.21.15.55:0/3624530416 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f444c0676e0 0x7f444c069ba0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f44680cbb20 tx=0x7f4458009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:17.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.302+0000 7f445e7fc640 1 -- 172.21.15.55:0/3624530416 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f4464053b70 con 0x7f4468068350 2024-03-19T01:59:17.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.574+0000 7f445e7fc640 1 -- 172.21.15.55:0/3624530416 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4464055ad0 con 0x7f4468068350 2024-03-19T01:59:17.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.786+0000 7f447018e640 1 -- 172.21.15.55:0/3624530416 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44680b1980 con 0x7f4468068350 2024-03-19T01:59:17.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.902+0000 7f445e7fc640 1 -- 172.21.15.55:0/3624530416 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f4464059b70 con 0x7f4468068350 2024-03-19T01:59:17.902 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:17.902 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:17.903 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:17.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.906+0000 7f447018e640 1 -- 172.21.15.55:0/3624530416 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f444c0676e0 msgr2=0x7f444c069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:17.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.906+0000 7f447018e640 1 --2- 172.21.15.55:0/3624530416 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f444c0676e0 0x7f444c069ba0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f44680cbb20 tx=0x7f4458009190 comp rx=0 tx=0).stop 2024-03-19T01:59:17.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.906+0000 7f447018e640 1 -- 172.21.15.55:0/3624530416 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4468068350 msgr2=0x7f44681f35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:17.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.906+0000 7f447018e640 1 --2- 172.21.15.55:0/3624530416 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4468068350 0x7f44681f35a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f446400c3a0 tx=0x7f446400c860 comp rx=0 tx=0).stop 2024-03-19T01:59:17.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.906+0000 7f447018e640 1 -- 172.21.15.55:0/3624530416 shutdown_connections 2024-03-19T01:59:17.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.906+0000 7f447018e640 1 --2- 172.21.15.55:0/3624530416 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f444c0676e0 0x7f444c069ba0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:17.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.906+0000 7f447018e640 1 --2- 172.21.15.55:0/3624530416 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f44681546e0 0x7f44680747d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:17.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.906+0000 7f447018e640 1 --2- 172.21.15.55:0/3624530416 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44680c6dc0 0x7f4468074290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:17.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.906+0000 7f447018e640 1 --2- 172.21.15.55:0/3624530416 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4468068350 0x7f44681f35a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:17.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.906+0000 7f447018e640 1 -- 172.21.15.55:0/3624530416 >> 172.21.15.55:0/3624530416 conn(0x7f44680c8990 msgr2=0x7f4468158df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:17.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.906+0000 7f447018e640 1 -- 172.21.15.55:0/3624530416 shutdown_connections 2024-03-19T01:59:17.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:17.906+0000 7f447018e640 1 -- 172.21.15.55:0/3624530416 wait complete. 2024-03-19T01:59:18.528 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:18.528 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:19.530 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:19.633 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:09:31.793 27555== Warning: unimplemented fcntl command: 1036 2024-03-19T01:59:19.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:19.646+0000 7fa8bbee0640 1 Processor -- start 2024-03-19T01:59:19.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:19.646+0000 7fa8bbee0640 1 -- start start 2024-03-19T01:59:19.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:19.646+0000 7fa8bbee0640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa8b4150f50 0x7fa8b415b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:19.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:19.646+0000 7fa8bbee0640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa8b40ca350 0x7fa8b40728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:19.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:19.650+0000 7fa8bbee0640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8b4151330 con 0x7fa8b40c3320 2024-03-19T01:59:19.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:19.650+0000 7fa8bbee0640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fa8b40c3e00 con 0x7fa8b4150f50 2024-03-19T01:59:19.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:19.650+0000 7fa8bbee0640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fa8b415ac10 con 0x7fa8b40ca350 2024-03-19T01:59:19.724 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:09:31.884 27555== Warning: unimplemented fcntl command: 1036 2024-03-19T01:59:19.817 INFO:tasks.ceph.mon.b.smithi084.stderr:==00:00:09:31.915 27626== Warning: unimplemented fcntl command: 1036 2024-03-19T01:59:19.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:19.822+0000 7fa8b947b640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa8b40ca350 0x7fa8b40728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:19.822 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:09:31.919 27627== Warning: unimplemented fcntl command: 1036 2024-03-19T01:59:19.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:19.850+0000 7fa8b947b640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa8b40ca350 0x7fa8b40728b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:51278/0 (socket says 172.21.15.55:51278) 2024-03-19T01:59:19.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:19.850+0000 7fa8b947b640 1 -- 172.21.15.55:0/2778537187 learned_addr learned my addr 172.21.15.55:0/2778537187 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:19.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:19.874+0000 7fa8b9c7c640 1 --2- 172.21.15.55:0/2778537187 >> v2:172.21.15.55:3300/0 conn(0x7fa8b4150f50 0x7fa8b415b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:19.922 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:09:32.019 27627== Warning: unimplemented fcntl command: 1036 2024-03-19T01:59:19.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:19.934+0000 7fa8b947b640 1 -- 172.21.15.55:0/2778537187 >> v2:172.21.15.55:3300/0 conn(0x7fa8b4150f50 msgr2=0x7fa8b415b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:19.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:19.934+0000 7fa8b947b640 1 --2- 172.21.15.55:0/2778537187 >> v2:172.21.15.55:3300/0 conn(0x7fa8b4150f50 0x7fa8b415b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:19.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:19.934+0000 7fa8b947b640 1 -- 172.21.15.55:0/2778537187 >> v1:172.21.15.55:6789/0 conn(0x7fa8b40c3320 legacy=0x7fa8b4157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:19.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:19.934+0000 7fa8b947b640 1 -- 172.21.15.55:0/2778537187 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b40c3840 con 0x7fa8b40ca350 2024-03-19T01:59:19.934 INFO:tasks.ceph.mon.b.smithi084.stderr:==00:00:09:32.031 27626== Warning: unimplemented fcntl command: 1036 2024-03-19T01:59:20.216 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:09:32.376 27555== Warning: unimplemented fcntl command: 1036 2024-03-19T01:59:20.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.338+0000 7fa8b947b640 1 --2- 172.21.15.55:0/2778537187 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8b40ca350 0x7fa8b40728b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4151760 tx=0x7fa8a401c4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8e2a7e72d736e61 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:20.346 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.346+0000 7fa8b8c7a640 1 -- 172.21.15.55:0/2778537187 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a40a6040 con 0x7fa8b40ca350 2024-03-19T01:59:20.346 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.346+0000 7fa8b8c7a640 1 -- 172.21.15.55:0/2778537187 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8a40af070 con 0x7fa8b40ca350 2024-03-19T01:59:20.346 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.346+0000 7fa8b8c7a640 1 -- 172.21.15.55:0/2778537187 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a40aa860 con 0x7fa8b40ca350 2024-03-19T01:59:20.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.350+0000 7fa8bbee0640 1 -- 172.21.15.55:0/2778537187 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8b40ca350 msgr2=0x7fa8b40728b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:20.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.350+0000 7fa8bbee0640 1 --2- 172.21.15.55:0/2778537187 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8b40ca350 0x7fa8b40728b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4151760 tx=0x7fa8a401c4e0 comp rx=0 tx=0).stop 2024-03-19T01:59:20.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.350+0000 7fa8bbee0640 1 -- 172.21.15.55:0/2778537187 shutdown_connections 2024-03-19T01:59:20.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.350+0000 7fa8bbee0640 1 --2- 172.21.15.55:0/2778537187 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8b40ca350 0x7fa8b40728b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:20.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.350+0000 7fa8bbee0640 1 --2- 172.21.15.55:0/2778537187 >> v2:172.21.15.55:3300/0 conn(0x7fa8b4150f50 0x7fa8b415b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:20.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.350+0000 7fa8bbee0640 1 -- 172.21.15.55:0/2778537187 >> 172.21.15.55:0/2778537187 conn(0x7fa8b406e670 msgr2=0x7fa8b406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:20.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.350+0000 7fa8bbee0640 1 -- 172.21.15.55:0/2778537187 shutdown_connections 2024-03-19T01:59:20.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.350+0000 7fa8bbee0640 1 -- 172.21.15.55:0/2778537187 wait complete. 2024-03-19T01:59:20.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.350+0000 7fa8bbee0640 1 Processor -- start 2024-03-19T01:59:20.351 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.354+0000 7fa8bbee0640 1 -- start start 2024-03-19T01:59:20.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.354+0000 7fa8bbee0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8b40c3320 0x7fa8b4143080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:20.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.354+0000 7fa8bbee0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa8b40ca350 0x7fa8b41f7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:20.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.354+0000 7fa8bbee0640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8b4150f50 0x7fa8b41f7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:20.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.354+0000 7fa8bbee0640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b41619a0 con 0x7fa8b40c3320 2024-03-19T01:59:20.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.354+0000 7fa8bbee0640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b4161820 con 0x7fa8b4150f50 2024-03-19T01:59:20.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.354+0000 7fa8bbee0640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa8b4161b20 con 0x7fa8b40ca350 2024-03-19T01:59:20.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.374+0000 7fa8b9c7c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8b40c3320 0x7fa8b4143080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:20.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.386+0000 7fa8b9c7c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8b40c3320 0x7fa8b4143080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33112/0 (socket says 172.21.15.55:33112) 2024-03-19T01:59:20.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.386+0000 7fa8b9c7c640 1 -- 172.21.15.55:0/2463593947 learned_addr learned my addr 172.21.15.55:0/2463593947 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:20.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.414+0000 7fa8b9c7c640 1 -- 172.21.15.55:0/2463593947 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa8b40ca350 msgr2=0x7fa8b41f7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:20.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.414+0000 7fa8b9c7c640 1 --2- 172.21.15.55:0/2463593947 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa8b40ca350 0x7fa8b41f7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:20.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.414+0000 7fa8b9c7c640 1 -- 172.21.15.55:0/2463593947 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8b4150f50 msgr2=0x7fa8b41f7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:20.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.414+0000 7fa8b9c7c640 1 --2- 172.21.15.55:0/2463593947 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8b4150f50 0x7fa8b41f7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:20.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.414+0000 7fa8b9c7c640 1 -- 172.21.15.55:0/2463593947 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b406bd40 con 0x7fa8b40c3320 2024-03-19T01:59:20.439 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:09:32.537 27627== Warning: unimplemented fcntl command: 1036 2024-03-19T01:59:20.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.446+0000 7fa8b9c7c640 1 --2- 172.21.15.55:0/2463593947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8b40c3320 0x7fa8b4143080 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa8b000c7f0 tx=0x7fa8b000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:20.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.514+0000 7fa8aa7fc640 1 -- 172.21.15.55:0/2463593947 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b0015070 con 0x7fa8b40c3320 2024-03-19T01:59:20.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.514+0000 7fa8aa7fc640 1 -- 172.21.15.55:0/2463593947 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8b0007820 con 0x7fa8b40c3320 2024-03-19T01:59:20.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.514+0000 7fa8bbee0640 1 -- 172.21.15.55:0/2463593947 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8b41f6c30 con 0x7fa8b40c3320 2024-03-19T01:59:20.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.514+0000 7fa8aa7fc640 1 -- 172.21.15.55:0/2463593947 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b0007c10 con 0x7fa8b40c3320 2024-03-19T01:59:20.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.514+0000 7fa8bbee0640 1 -- 172.21.15.55:0/2463593947 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8b406ce80 con 0x7fa8b40c3320 2024-03-19T01:59:20.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.514+0000 7fa8bbee0640 1 -- 172.21.15.55:0/2463593947 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8b4068a40 con 0x7fa8b40c3320 2024-03-19T01:59:20.595 INFO:tasks.ceph.mon.b.smithi084.stderr:==00:00:09:32.688 27626== Warning: unimplemented fcntl command: 1036 2024-03-19T01:59:20.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.974+0000 7fa8aa7fc640 1 -- 172.21.15.55:0/2463593947 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa8b0018070 con 0x7fa8b40c3320 2024-03-19T01:59:20.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.974+0000 7fa8aa7fc640 1 --2- 172.21.15.55:0/2463593947 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa8980676b0 0x7fa898069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:20.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.974+0000 7fa8b947b640 1 --2- 172.21.15.55:0/2463593947 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa8980676b0 0x7fa898069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:20.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:20.974+0000 7fa8b947b640 1 --2- 172.21.15.55:0/2463593947 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa8980676b0 0x7fa898069b70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa8a40ae000 tx=0x7fa8a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:21.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:21.058+0000 7fa8aa7fc640 1 -- 172.21.15.55:0/2463593947 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fa8b00543b0 con 0x7fa8b40c3320 2024-03-19T01:59:21.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:21.058+0000 7fa8aa7fc640 1 -- 172.21.15.55:0/2463593947 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa8b0056a10 con 0x7fa8b40c3320 2024-03-19T01:59:21.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:21.274+0000 7fa8bbee0640 1 -- 172.21.15.55:0/2463593947 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8b414ebe0 con 0x7fa8b40c3320 2024-03-19T01:59:21.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:21.298+0000 7fa8aa7fc640 1 -- 172.21.15.55:0/2463593947 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fa8b0056310 con 0x7fa8b40c3320 2024-03-19T01:59:21.297 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:21.297 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:21.297 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:21.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:21.302+0000 7fa8bbee0640 1 -- 172.21.15.55:0/2463593947 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa8980676b0 msgr2=0x7fa898069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:21.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:21.302+0000 7fa8bbee0640 1 --2- 172.21.15.55:0/2463593947 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa8980676b0 0x7fa898069b70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa8a40ae000 tx=0x7fa8a4008040 comp rx=0 tx=0).stop 2024-03-19T01:59:21.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:21.302+0000 7fa8bbee0640 1 -- 172.21.15.55:0/2463593947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8b40c3320 msgr2=0x7fa8b4143080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:21.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:21.302+0000 7fa8bbee0640 1 --2- 172.21.15.55:0/2463593947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8b40c3320 0x7fa8b4143080 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa8b000c7f0 tx=0x7fa8b000ccb0 comp rx=0 tx=0).stop 2024-03-19T01:59:21.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:21.302+0000 7fa8bbee0640 1 -- 172.21.15.55:0/2463593947 shutdown_connections 2024-03-19T01:59:21.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:21.302+0000 7fa8bbee0640 1 --2- 172.21.15.55:0/2463593947 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa8980676b0 0x7fa898069b70 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:21.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:21.302+0000 7fa8bbee0640 1 --2- 172.21.15.55:0/2463593947 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8b4150f50 0x7fa8b41f7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:21.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:21.302+0000 7fa8bbee0640 1 --2- 172.21.15.55:0/2463593947 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa8b40ca350 0x7fa8b41f7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:21.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:21.302+0000 7fa8bbee0640 1 --2- 172.21.15.55:0/2463593947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8b40c3320 0x7fa8b4143080 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:21.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:21.302+0000 7fa8bbee0640 1 -- 172.21.15.55:0/2463593947 >> 172.21.15.55:0/2463593947 conn(0x7fa8b406e670 msgr2=0x7fa8b406ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:21.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:21.302+0000 7fa8bbee0640 1 -- 172.21.15.55:0/2463593947 shutdown_connections 2024-03-19T01:59:21.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:21.302+0000 7fa8bbee0640 1 -- 172.21.15.55:0/2463593947 wait complete. 2024-03-19T01:59:21.920 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:21.920 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:22.922 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:23.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.046+0000 7f6b4a7f7640 1 Processor -- start 2024-03-19T01:59:23.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.046+0000 7f6b4a7f7640 1 -- start start 2024-03-19T01:59:23.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.046+0000 7f6b4a7f7640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f6b44071050 0x7f6b440ca210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:23.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.046+0000 7f6b4a7f7640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f6b440c7d60 0x7f6b4415ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:23.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.046+0000 7f6b4a7f7640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b440c8170 con 0x7f6b441545f0 2024-03-19T01:59:23.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.046+0000 7f6b4a7f7640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f6b4415b340 con 0x7f6b44071050 2024-03-19T01:59:23.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.046+0000 7f6b4a7f7640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f6b44072c90 con 0x7f6b440c7d60 2024-03-19T01:59:23.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.050+0000 7f6b437fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f6b44071050 0x7f6b440ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:23.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.050+0000 7f6b48d94640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f6b440c7d60 0x7f6b4415ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:23.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.050+0000 7f6b437fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f6b44071050 0x7f6b440ca210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33120/0 (socket says 172.21.15.55:33120) 2024-03-19T01:59:23.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.050+0000 7f6b437fe640 1 -- 172.21.15.55:0/3507288253 learned_addr learned my addr 172.21.15.55:0/3507288253 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:23.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.050+0000 7f6b437fe640 1 -- 172.21.15.55:0/3507288253 >> v2:172.21.15.84:3300/0 conn(0x7f6b440c7d60 msgr2=0x7f6b4415ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:23.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.050+0000 7f6b437fe640 1 --2- 172.21.15.55:0/3507288253 >> v2:172.21.15.84:3300/0 conn(0x7f6b440c7d60 0x7f6b4415ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:23.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.050+0000 7f6b437fe640 1 -- 172.21.15.55:0/3507288253 >> v1:172.21.15.84:6789/0 conn(0x7f6b441545f0 legacy=0x7f6b4406b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:23.050 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.050+0000 7f6b437fe640 1 -- 172.21.15.55:0/3507288253 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b44073210 con 0x7f6b44071050 2024-03-19T01:59:23.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.054+0000 7f6b437fe640 1 --2- 172.21.15.55:0/3507288253 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6b44071050 0x7f6b440ca210 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6b44075070 tx=0x7f6b3801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13e2d495d25a8b69 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:23.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.058+0000 7f6b42ffd640 1 -- 172.21.15.55:0/3507288253 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b380b0070 con 0x7f6b44071050 2024-03-19T01:59:23.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.058+0000 7f6b42ffd640 1 -- 172.21.15.55:0/3507288253 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b380ab600 con 0x7f6b44071050 2024-03-19T01:59:23.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.058+0000 7f6b42ffd640 1 -- 172.21.15.55:0/3507288253 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b380ab9d0 con 0x7f6b44071050 2024-03-19T01:59:23.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.058+0000 7f6b4a7f7640 1 -- 172.21.15.55:0/3507288253 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6b44071050 msgr2=0x7f6b440ca210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:23.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.058+0000 7f6b4a7f7640 1 --2- 172.21.15.55:0/3507288253 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6b44071050 0x7f6b440ca210 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6b44075070 tx=0x7f6b3801c520 comp rx=0 tx=0).stop 2024-03-19T01:59:23.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.058+0000 7f6b4a7f7640 1 -- 172.21.15.55:0/3507288253 shutdown_connections 2024-03-19T01:59:23.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.058+0000 7f6b4a7f7640 1 --2- 172.21.15.55:0/3507288253 >> v2:172.21.15.84:3300/0 conn(0x7f6b440c7d60 0x7f6b4415ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:23.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.058+0000 7f6b4a7f7640 1 --2- 172.21.15.55:0/3507288253 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6b44071050 0x7f6b440ca210 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:23.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.058+0000 7f6b4a7f7640 1 -- 172.21.15.55:0/3507288253 >> 172.21.15.55:0/3507288253 conn(0x7f6b44070410 msgr2=0x7f6b44070820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:23.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.058+0000 7f6b4a7f7640 1 -- 172.21.15.55:0/3507288253 shutdown_connections 2024-03-19T01:59:23.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.058+0000 7f6b4a7f7640 1 -- 172.21.15.55:0/3507288253 wait complete. 2024-03-19T01:59:23.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.062+0000 7f6b4a7f7640 1 Processor -- start 2024-03-19T01:59:23.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.062+0000 7f6b4a7f7640 1 -- start start 2024-03-19T01:59:23.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.062+0000 7f6b4a7f7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6b44071050 0x7f6b44150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:23.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.062+0000 7f6b4a7f7640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6b440c7d60 0x7f6b4414f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:23.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.062+0000 7f6b4a7f7640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6b441545f0 0x7f6b4414fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:23.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.062+0000 7f6b4a7f7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6b44154b50 con 0x7f6b44071050 2024-03-19T01:59:23.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.062+0000 7f6b4a7f7640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f6b441549d0 con 0x7f6b440c7d60 2024-03-19T01:59:23.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.062+0000 7f6b4a7f7640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f6b44166c70 con 0x7f6b441545f0 2024-03-19T01:59:23.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.062+0000 7f6b43fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6b44071050 0x7f6b44150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:23.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.062+0000 7f6b437fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6b440c7d60 0x7f6b4414f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:23.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.062+0000 7f6b43fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6b44071050 0x7f6b44150b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33126/0 (socket says 172.21.15.55:33126) 2024-03-19T01:59:23.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.062+0000 7f6b43fff640 1 -- 172.21.15.55:0/2993758927 learned_addr learned my addr 172.21.15.55:0/2993758927 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:23.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.062+0000 7f6b48d94640 1 --2- 172.21.15.55:0/2993758927 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6b441545f0 0x7f6b4414fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:23.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.066+0000 7f6b43fff640 1 -- 172.21.15.55:0/2993758927 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6b441545f0 msgr2=0x7f6b4414fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:23.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.066+0000 7f6b43fff640 1 --2- 172.21.15.55:0/2993758927 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6b441545f0 0x7f6b4414fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:23.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.066+0000 7f6b43fff640 1 -- 172.21.15.55:0/2993758927 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6b440c7d60 msgr2=0x7f6b4414f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:23.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.066+0000 7f6b43fff640 1 --2- 172.21.15.55:0/2993758927 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6b440c7d60 0x7f6b4414f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:23.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.066+0000 7f6b43fff640 1 -- 172.21.15.55:0/2993758927 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b441f3bf0 con 0x7f6b44071050 2024-03-19T01:59:23.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.070+0000 7f6b43fff640 1 --2- 172.21.15.55:0/2993758927 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6b44071050 0x7f6b44150b00 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6b3400c3a0 tx=0x7f6b3400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:23.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.074+0000 7f6b40ff9640 1 -- 172.21.15.55:0/2993758927 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b340052d0 con 0x7f6b44071050 2024-03-19T01:59:23.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.074+0000 7f6b40ff9640 1 -- 172.21.15.55:0/2993758927 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b34005470 con 0x7f6b44071050 2024-03-19T01:59:23.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.074+0000 7f6b4a7f7640 1 -- 172.21.15.55:0/2993758927 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b441f3eb0 con 0x7f6b44071050 2024-03-19T01:59:23.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.074+0000 7f6b40ff9640 1 -- 172.21.15.55:0/2993758927 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b34007260 con 0x7f6b44071050 2024-03-19T01:59:23.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.074+0000 7f6b4a7f7640 1 -- 172.21.15.55:0/2993758927 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b4414eeb0 con 0x7f6b44071050 2024-03-19T01:59:23.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.074+0000 7f6b4a7f7640 1 -- 172.21.15.55:0/2993758927 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b44068a40 con 0x7f6b44071050 2024-03-19T01:59:23.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.106+0000 7f6b40ff9640 1 -- 172.21.15.55:0/2993758927 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f6b34007400 con 0x7f6b44071050 2024-03-19T01:59:23.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.106+0000 7f6b40ff9640 1 --2- 172.21.15.55:0/2993758927 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6b300676b0 0x7f6b30069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:23.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.106+0000 7f6b437fe640 1 --2- 172.21.15.55:0/2993758927 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6b300676b0 0x7f6b30069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:23.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.106+0000 7f6b437fe640 1 --2- 172.21.15.55:0/2993758927 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6b300676b0 0x7f6b30069b70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6b380ad000 tx=0x7f6b380a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:23.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.110+0000 7f6b40ff9640 1 -- 172.21.15.55:0/2993758927 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f6b34053ba0 con 0x7f6b44071050 2024-03-19T01:59:23.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.386+0000 7f6b40ff9640 1 -- 172.21.15.55:0/2993758927 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f6b440c8170 con 0x7f6b44071050 2024-03-19T01:59:23.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.606+0000 7f6b4a7f7640 1 -- 172.21.15.55:0/2993758927 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b44164a40 con 0x7f6b44071050 2024-03-19T01:59:23.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.726+0000 7f6b40ff9640 1 -- 172.21.15.55:0/2993758927 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f6b34055b00 con 0x7f6b44071050 2024-03-19T01:59:23.724 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:23.724 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:23.724 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:23.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.726+0000 7f6b4a7f7640 1 -- 172.21.15.55:0/2993758927 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6b300676b0 msgr2=0x7f6b30069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:23.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.726+0000 7f6b4a7f7640 1 --2- 172.21.15.55:0/2993758927 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6b300676b0 0x7f6b30069b70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6b380ad000 tx=0x7f6b380a6000 comp rx=0 tx=0).stop 2024-03-19T01:59:23.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.726+0000 7f6b4a7f7640 1 -- 172.21.15.55:0/2993758927 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6b44071050 msgr2=0x7f6b44150b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:23.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.726+0000 7f6b4a7f7640 1 --2- 172.21.15.55:0/2993758927 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6b44071050 0x7f6b44150b00 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6b3400c3a0 tx=0x7f6b3400c860 comp rx=0 tx=0).stop 2024-03-19T01:59:23.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.726+0000 7f6b4a7f7640 1 -- 172.21.15.55:0/2993758927 shutdown_connections 2024-03-19T01:59:23.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.726+0000 7f6b4a7f7640 1 --2- 172.21.15.55:0/2993758927 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6b300676b0 0x7f6b30069b70 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:23.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.726+0000 7f6b4a7f7640 1 --2- 172.21.15.55:0/2993758927 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6b441545f0 0x7f6b4414fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:23.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.726+0000 7f6b4a7f7640 1 --2- 172.21.15.55:0/2993758927 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6b440c7d60 0x7f6b4414f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:23.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.726+0000 7f6b4a7f7640 1 --2- 172.21.15.55:0/2993758927 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6b44071050 0x7f6b44150b00 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:23.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.726+0000 7f6b4a7f7640 1 -- 172.21.15.55:0/2993758927 >> 172.21.15.55:0/2993758927 conn(0x7f6b44070410 msgr2=0x7f6b441581f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:23.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.726+0000 7f6b4a7f7640 1 -- 172.21.15.55:0/2993758927 shutdown_connections 2024-03-19T01:59:23.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:23.726+0000 7f6b4a7f7640 1 -- 172.21.15.55:0/2993758927 wait complete. 2024-03-19T01:59:24.347 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:24.347 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:25.348 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:25.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.478+0000 7fb9b01da640 1 Processor -- start 2024-03-19T01:59:25.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.478+0000 7fb9b01da640 1 -- start start 2024-03-19T01:59:25.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.478+0000 7fb9b01da640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb9a8150f50 0x7fb9a815b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:25.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.478+0000 7fb9b01da640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fb9a80ca350 0x7fb9a80728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:25.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.478+0000 7fb9b01da640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9a8151330 con 0x7fb9a80c3320 2024-03-19T01:59:25.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.478+0000 7fb9b01da640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fb9a80c3e00 con 0x7fb9a8150f50 2024-03-19T01:59:25.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.478+0000 7fb9b01da640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fb9a815ac10 con 0x7fb9a80ca350 2024-03-19T01:59:25.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.482+0000 7fb9adf76640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb9a8150f50 0x7fb9a815b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:25.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.482+0000 7fb9ad775640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fb9a80ca350 0x7fb9a80728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:25.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.482+0000 7fb9adf76640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb9a8150f50 0x7fb9a815b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33134/0 (socket says 172.21.15.55:33134) 2024-03-19T01:59:25.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.482+0000 7fb9adf76640 1 -- 172.21.15.55:0/3888731833 learned_addr learned my addr 172.21.15.55:0/3888731833 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:25.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.482+0000 7fb9adf76640 1 -- 172.21.15.55:0/3888731833 >> v2:172.21.15.84:3300/0 conn(0x7fb9a80ca350 msgr2=0x7fb9a80728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:25.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.482+0000 7fb9adf76640 1 --2- 172.21.15.55:0/3888731833 >> v2:172.21.15.84:3300/0 conn(0x7fb9a80ca350 0x7fb9a80728b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:25.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.482+0000 7fb9adf76640 1 -- 172.21.15.55:0/3888731833 >> v1:172.21.15.84:6789/0 conn(0x7fb9a80c3320 legacy=0x7fb9a8157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:25.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.482+0000 7fb9adf76640 1 -- 172.21.15.55:0/3888731833 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a80c3840 con 0x7fb9a8150f50 2024-03-19T01:59:25.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.486+0000 7fb9adf76640 1 --2- 172.21.15.55:0/3888731833 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9a8150f50 0x7fb9a815b0a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8151760 tx=0x7fb99801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25973185bc0c4af2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:25.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.490+0000 7fb9acf74640 1 -- 172.21.15.55:0/3888731833 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9980a6040 con 0x7fb9a8150f50 2024-03-19T01:59:25.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.490+0000 7fb9acf74640 1 -- 172.21.15.55:0/3888731833 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9980af070 con 0x7fb9a8150f50 2024-03-19T01:59:25.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.490+0000 7fb9acf74640 1 -- 172.21.15.55:0/3888731833 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9980aa860 con 0x7fb9a8150f50 2024-03-19T01:59:25.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.490+0000 7fb9b01da640 1 -- 172.21.15.55:0/3888731833 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9a8150f50 msgr2=0x7fb9a815b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:25.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.490+0000 7fb9b01da640 1 --2- 172.21.15.55:0/3888731833 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9a8150f50 0x7fb9a815b0a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8151760 tx=0x7fb99801c4e0 comp rx=0 tx=0).stop 2024-03-19T01:59:25.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.490+0000 7fb9b01da640 1 -- 172.21.15.55:0/3888731833 shutdown_connections 2024-03-19T01:59:25.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.490+0000 7fb9b01da640 1 --2- 172.21.15.55:0/3888731833 >> v2:172.21.15.84:3300/0 conn(0x7fb9a80ca350 0x7fb9a80728b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:25.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.490+0000 7fb9b01da640 1 --2- 172.21.15.55:0/3888731833 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9a8150f50 0x7fb9a815b0a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:25.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.490+0000 7fb9b01da640 1 -- 172.21.15.55:0/3888731833 >> 172.21.15.55:0/3888731833 conn(0x7fb9a806e670 msgr2=0x7fb9a806ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:25.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.490+0000 7fb9b01da640 1 -- 172.21.15.55:0/3888731833 shutdown_connections 2024-03-19T01:59:25.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.490+0000 7fb9b01da640 1 -- 172.21.15.55:0/3888731833 wait complete. 2024-03-19T01:59:25.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.494+0000 7fb9b01da640 1 Processor -- start 2024-03-19T01:59:25.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.494+0000 7fb9b01da640 1 -- start start 2024-03-19T01:59:25.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.494+0000 7fb9b01da640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb9a80c3320 0x7fb9a80c6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:25.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.494+0000 7fb9b01da640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb9a80ca350 0x7fb9a80c6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:25.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.494+0000 7fb9b01da640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9a8150f50 0x7fb9a814bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:25.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.494+0000 7fb9b01da640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a81639b0 con 0x7fb9a8150f50 2024-03-19T01:59:25.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.494+0000 7fb9b01da640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a8163830 con 0x7fb9a80c3320 2024-03-19T01:59:25.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.494+0000 7fb9b01da640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fb9a8163b30 con 0x7fb9a80ca350 2024-03-19T01:59:25.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.494+0000 7fb9ae777640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9a8150f50 0x7fb9a814bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:25.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.494+0000 7fb9adf76640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb9a80c3320 0x7fb9a80c6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:25.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.494+0000 7fb9ae777640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9a8150f50 0x7fb9a814bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33136/0 (socket says 172.21.15.55:33136) 2024-03-19T01:59:25.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.494+0000 7fb9ae777640 1 -- 172.21.15.55:0/3300216976 learned_addr learned my addr 172.21.15.55:0/3300216976 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:25.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.494+0000 7fb9ad775640 1 --2- 172.21.15.55:0/3300216976 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb9a80ca350 0x7fb9a80c6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:25.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.498+0000 7fb9ae777640 1 -- 172.21.15.55:0/3300216976 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb9a80ca350 msgr2=0x7fb9a80c6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:25.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.498+0000 7fb9ae777640 1 --2- 172.21.15.55:0/3300216976 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb9a80ca350 0x7fb9a80c6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:25.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.498+0000 7fb9ae777640 1 -- 172.21.15.55:0/3300216976 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb9a80c3320 msgr2=0x7fb9a80c6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:25.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.498+0000 7fb9ae777640 1 --2- 172.21.15.55:0/3300216976 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb9a80c3320 0x7fb9a80c6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:25.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.498+0000 7fb9ae777640 1 -- 172.21.15.55:0/3300216976 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a80c4f40 con 0x7fb9a8150f50 2024-03-19T01:59:25.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.502+0000 7fb9ae777640 1 --2- 172.21.15.55:0/3300216976 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9a8150f50 0x7fb9a814bec0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb9a000c780 tx=0x7fb9a000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:25.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.506+0000 7fb9a67fc640 1 -- 172.21.15.55:0/3300216976 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a0016020 con 0x7fb9a8150f50 2024-03-19T01:59:25.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.506+0000 7fb9a67fc640 1 -- 172.21.15.55:0/3300216976 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9a000a9f0 con 0x7fb9a8150f50 2024-03-19T01:59:25.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.506+0000 7fb9b01da640 1 -- 172.21.15.55:0/3300216976 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9a80c5220 con 0x7fb9a8150f50 2024-03-19T01:59:25.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.506+0000 7fb9a67fc640 1 -- 172.21.15.55:0/3300216976 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a000ae20 con 0x7fb9a8150f50 2024-03-19T01:59:25.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.506+0000 7fb9b01da640 1 -- 172.21.15.55:0/3300216976 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9a80c66b0 con 0x7fb9a8150f50 2024-03-19T01:59:25.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.506+0000 7fb9b01da640 1 -- 172.21.15.55:0/3300216976 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9a8068a40 con 0x7fb9a8150f50 2024-03-19T01:59:25.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.534+0000 7fb9a67fc640 1 -- 172.21.15.55:0/3300216976 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb9a0019020 con 0x7fb9a8150f50 2024-03-19T01:59:25.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.534+0000 7fb9a67fc640 1 --2- 172.21.15.55:0/3300216976 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb9940676e0 0x7fb994069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:25.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.534+0000 7fb9adf76640 1 --2- 172.21.15.55:0/3300216976 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb9940676e0 0x7fb994069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:25.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.534+0000 7fb9adf76640 1 --2- 172.21.15.55:0/3300216976 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb9940676e0 0x7fb994069ba0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb9980ae000 tx=0x7fb9980a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:25.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.538+0000 7fb9a67fc640 1 -- 172.21.15.55:0/3300216976 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fb9a0054130 con 0x7fb9a8150f50 2024-03-19T01:59:25.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:25.814+0000 7fb9a67fc640 1 -- 172.21.15.55:0/3300216976 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fb9a0020030 con 0x7fb9a8150f50 2024-03-19T01:59:26.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:26.034+0000 7fb9b01da640 1 -- 172.21.15.55:0/3300216976 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9a80c7600 con 0x7fb9a8150f50 2024-03-19T01:59:26.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:26.058+0000 7fb9a67fc640 1 -- 172.21.15.55:0/3300216976 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fb9a0056090 con 0x7fb9a8150f50 2024-03-19T01:59:26.057 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:26.057 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:26.057 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:26.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:26.062+0000 7fb9b01da640 1 -- 172.21.15.55:0/3300216976 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb9940676e0 msgr2=0x7fb994069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:26.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:26.062+0000 7fb9b01da640 1 --2- 172.21.15.55:0/3300216976 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb9940676e0 0x7fb994069ba0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb9980ae000 tx=0x7fb9980a7000 comp rx=0 tx=0).stop 2024-03-19T01:59:26.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:26.062+0000 7fb9b01da640 1 -- 172.21.15.55:0/3300216976 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9a8150f50 msgr2=0x7fb9a814bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:26.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:26.062+0000 7fb9b01da640 1 --2- 172.21.15.55:0/3300216976 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9a8150f50 0x7fb9a814bec0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb9a000c780 tx=0x7fb9a000cc40 comp rx=0 tx=0).stop 2024-03-19T01:59:26.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:26.062+0000 7fb9b01da640 1 -- 172.21.15.55:0/3300216976 shutdown_connections 2024-03-19T01:59:26.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:26.062+0000 7fb9b01da640 1 --2- 172.21.15.55:0/3300216976 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb9940676e0 0x7fb994069ba0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:26.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:26.062+0000 7fb9b01da640 1 --2- 172.21.15.55:0/3300216976 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9a8150f50 0x7fb9a814bec0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:26.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:26.062+0000 7fb9b01da640 1 --2- 172.21.15.55:0/3300216976 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb9a80ca350 0x7fb9a80c6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:26.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:26.062+0000 7fb9b01da640 1 --2- 172.21.15.55:0/3300216976 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb9a80c3320 0x7fb9a80c6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:26.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:26.062+0000 7fb9b01da640 1 -- 172.21.15.55:0/3300216976 >> 172.21.15.55:0/3300216976 conn(0x7fb9a806e670 msgr2=0x7fb9a80701b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:26.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:26.062+0000 7fb9b01da640 1 -- 172.21.15.55:0/3300216976 shutdown_connections 2024-03-19T01:59:26.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:26.062+0000 7fb9b01da640 1 -- 172.21.15.55:0/3300216976 wait complete. 2024-03-19T01:59:26.681 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:26.681 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:27.682 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:27.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.814+0000 7f686e8ef640 1 Processor -- start 2024-03-19T01:59:27.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.814+0000 7f686e8ef640 1 -- start start 2024-03-19T01:59:27.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.814+0000 7f686e8ef640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f686806e850 0x7f68680c71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:27.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.814+0000 7f686e8ef640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f686815af80 con 0x7f686806e470 2024-03-19T01:59:27.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.814+0000 7f686e8ef640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68680c79b0 con 0x7f68680c8770 2024-03-19T01:59:27.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.814+0000 7f686e8ef640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f686815ad30 con 0x7f686806e850 2024-03-19T01:59:27.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.814+0000 7f6867fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f686806e470 0x7f686806b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:35812/0 (socket says 172.21.15.55:35812) 2024-03-19T01:59:27.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.814+0000 7f6867fff640 1 -- 172.21.15.55:0/1274240061 learned_addr learned my addr 172.21.15.55:0/1274240061 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:59:27.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.814+0000 7f68677fe640 1 --2- 172.21.15.55:0/1274240061 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f686806e850 0x7f68680c71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:27.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.818+0000 7f6866ffd640 1 -- 172.21.15.55:0/1274240061 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 107498618 0 0) 0x7f686815af80 con 0x7f686806e470 2024-03-19T01:59:27.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.818+0000 7f6866ffd640 1 -- 172.21.15.55:0/1274240061 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6850003530 con 0x7f686806e470 2024-03-19T01:59:27.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.818+0000 7f68677fe640 1 -- 172.21.15.55:0/1274240061 >> v1:172.21.15.84:6789/0 conn(0x7f68680c8770 legacy=0x7f6868154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:27.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.818+0000 7f68677fe640 1 -- 172.21.15.55:0/1274240061 >> v1:172.21.15.55:6789/0 conn(0x7f686806e470 legacy=0x7f686806b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:27.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.818+0000 7f68677fe640 1 -- 172.21.15.55:0/1274240061 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68680c7c90 con 0x7f686806e850 2024-03-19T01:59:27.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.826+0000 7f68677fe640 1 --2- 172.21.15.55:0/1274240061 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f686806e850 0x7f68680c71a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f685c0097f0 tx=0x7f685c01c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48bfbb9aa46d9b61 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:27.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.830+0000 7f6866ffd640 1 -- 172.21.15.55:0/1274240061 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685c0ab5f0 con 0x7f686806e850 2024-03-19T01:59:27.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.830+0000 7f6866ffd640 1 -- 172.21.15.55:0/1274240061 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f685c0b0070 con 0x7f686806e850 2024-03-19T01:59:27.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.830+0000 7f6866ffd640 1 -- 172.21.15.55:0/1274240061 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685c0ab9b0 con 0x7f686806e850 2024-03-19T01:59:27.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.830+0000 7f686e8ef640 1 -- 172.21.15.55:0/1274240061 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f686806e850 msgr2=0x7f68680c71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:27.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.830+0000 7f686e8ef640 1 --2- 172.21.15.55:0/1274240061 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f686806e850 0x7f68680c71a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f685c0097f0 tx=0x7f685c01c4e0 comp rx=0 tx=0).stop 2024-03-19T01:59:27.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.830+0000 7f686e8ef640 1 -- 172.21.15.55:0/1274240061 shutdown_connections 2024-03-19T01:59:27.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.830+0000 7f686e8ef640 1 --2- 172.21.15.55:0/1274240061 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f686806e850 0x7f68680c71a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:27.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.830+0000 7f686e8ef640 1 -- 172.21.15.55:0/1274240061 >> 172.21.15.55:0/1274240061 conn(0x7f68680cc270 msgr2=0x7f68680cc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:27.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.830+0000 7f686e8ef640 1 -- 172.21.15.55:0/1274240061 shutdown_connections 2024-03-19T01:59:27.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.830+0000 7f686e8ef640 1 -- 172.21.15.55:0/1274240061 wait complete. 2024-03-19T01:59:27.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.834+0000 7f686e8ef640 1 Processor -- start 2024-03-19T01:59:27.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.834+0000 7f686e8ef640 1 -- start start 2024-03-19T01:59:27.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.834+0000 7f686e8ef640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f686806e470 0x7f68680cb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:27.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.834+0000 7f686e8ef640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f686806e850 0x7f686815b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:27.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.834+0000 7f686e8ef640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68680c8770 0x7f686815b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:27.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.834+0000 7f686e8ef640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6868154880 con 0x7f686806e470 2024-03-19T01:59:27.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.834+0000 7f686e8ef640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f686815b260 con 0x7f68680c8770 2024-03-19T01:59:27.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.834+0000 7f686e8ef640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f6868154a00 con 0x7f686806e850 2024-03-19T01:59:27.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.834+0000 7f6867fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f686806e470 0x7f68680cb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:27.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.834+0000 7f6867fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f686806e470 0x7f68680cb780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33144/0 (socket says 172.21.15.55:33144) 2024-03-19T01:59:27.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.834+0000 7f6867fff640 1 -- 172.21.15.55:0/2721397126 learned_addr learned my addr 172.21.15.55:0/2721397126 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:27.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.834+0000 7f686ce8c640 1 --2- 172.21.15.55:0/2721397126 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68680c8770 0x7f686815b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:27.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.834+0000 7f68677fe640 1 --2- 172.21.15.55:0/2721397126 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f686806e850 0x7f686815b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:27.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.838+0000 7f6867fff640 1 -- 172.21.15.55:0/2721397126 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f686806e850 msgr2=0x7f686815b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:27.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.838+0000 7f6867fff640 1 --2- 172.21.15.55:0/2721397126 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f686806e850 0x7f686815b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:27.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.838+0000 7f6867fff640 1 -- 172.21.15.55:0/2721397126 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68680c8770 msgr2=0x7f686815b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:27.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.838+0000 7f6867fff640 1 --2- 172.21.15.55:0/2721397126 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68680c8770 0x7f686815b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:27.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.838+0000 7f6867fff640 1 -- 172.21.15.55:0/2721397126 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68680cbde0 con 0x7f686806e470 2024-03-19T01:59:27.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.842+0000 7f6867fff640 1 --2- 172.21.15.55:0/2721397126 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f686806e470 0x7f68680cb780 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f685800c3c0 tx=0x7f685800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:27.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.846+0000 7f6864ff9640 1 -- 172.21.15.55:0/2721397126 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6858017070 con 0x7f686806e470 2024-03-19T01:59:27.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.846+0000 7f6864ff9640 1 -- 172.21.15.55:0/2721397126 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6858005cf0 con 0x7f686806e470 2024-03-19T01:59:27.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.846+0000 7f686e8ef640 1 -- 172.21.15.55:0/2721397126 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f686815cc20 con 0x7f686806e470 2024-03-19T01:59:27.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.846+0000 7f6864ff9640 1 -- 172.21.15.55:0/2721397126 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68580072f0 con 0x7f686806e470 2024-03-19T01:59:27.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.846+0000 7f686e8ef640 1 -- 172.21.15.55:0/2721397126 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f686815d7a0 con 0x7f686806e470 2024-03-19T01:59:27.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.846+0000 7f686e8ef640 1 -- 172.21.15.55:0/2721397126 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6868068a40 con 0x7f686806e470 2024-03-19T01:59:27.876 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.878+0000 7f6864ff9640 1 -- 172.21.15.55:0/2721397126 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f685801a070 con 0x7f686806e470 2024-03-19T01:59:27.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.878+0000 7f6864ff9640 1 --2- 172.21.15.55:0/2721397126 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6854067660 0x7f6854069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:27.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.878+0000 7f68677fe640 1 --2- 172.21.15.55:0/2721397126 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6854067660 0x7f6854069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:27.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:27.878+0000 7f68677fe640 1 --2- 172.21.15.55:0/2721397126 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6854067660 0x7f6854069b20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f685c0af000 tx=0x7f685c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:28.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:28.214+0000 7f6864ff9640 1 -- 172.21.15.55:0/2721397126 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f6858053730 con 0x7f686806e470 2024-03-19T01:59:28.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:28.214+0000 7f6864ff9640 1 -- 172.21.15.55:0/2721397126 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f6858021020 con 0x7f686806e470 2024-03-19T01:59:28.422 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:28.422+0000 7f686e8ef640 1 -- 172.21.15.55:0/2721397126 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68680c6d70 con 0x7f686806e470 2024-03-19T01:59:28.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:28.450+0000 7f6864ff9640 1 -- 172.21.15.55:0/2721397126 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f6858055690 con 0x7f686806e470 2024-03-19T01:59:28.449 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:28.449 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:28.449 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:28.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:28.450+0000 7f686e8ef640 1 -- 172.21.15.55:0/2721397126 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6854067660 msgr2=0x7f6854069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:28.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:28.450+0000 7f686e8ef640 1 --2- 172.21.15.55:0/2721397126 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6854067660 0x7f6854069b20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f685c0af000 tx=0x7f685c0a7000 comp rx=0 tx=0).stop 2024-03-19T01:59:28.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:28.450+0000 7f686e8ef640 1 -- 172.21.15.55:0/2721397126 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f686806e470 msgr2=0x7f68680cb780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:28.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:28.450+0000 7f686e8ef640 1 --2- 172.21.15.55:0/2721397126 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f686806e470 0x7f68680cb780 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f685800c3c0 tx=0x7f685800c880 comp rx=0 tx=0).stop 2024-03-19T01:59:28.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:28.450+0000 7f686e8ef640 1 -- 172.21.15.55:0/2721397126 shutdown_connections 2024-03-19T01:59:28.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:28.450+0000 7f686e8ef640 1 --2- 172.21.15.55:0/2721397126 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6854067660 0x7f6854069b20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:28.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:28.450+0000 7f686e8ef640 1 --2- 172.21.15.55:0/2721397126 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68680c8770 0x7f686815b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:28.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:28.450+0000 7f686e8ef640 1 --2- 172.21.15.55:0/2721397126 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f686806e850 0x7f686815b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:28.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:28.450+0000 7f686e8ef640 1 --2- 172.21.15.55:0/2721397126 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f686806e470 0x7f68680cb780 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:28.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:28.450+0000 7f686e8ef640 1 -- 172.21.15.55:0/2721397126 >> 172.21.15.55:0/2721397126 conn(0x7f68680cc270 msgr2=0x7f686814a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:28.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:28.450+0000 7f686e8ef640 1 -- 172.21.15.55:0/2721397126 shutdown_connections 2024-03-19T01:59:28.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:28.450+0000 7f686e8ef640 1 -- 172.21.15.55:0/2721397126 wait complete. 2024-03-19T01:59:29.071 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:29.071 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:30.073 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:30.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.186+0000 7fc6f941c640 1 Processor -- start 2024-03-19T01:59:30.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.186+0000 7fc6f941c640 1 -- start start 2024-03-19T01:59:30.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.190+0000 7fc6f941c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc6f415d2d0 0x7fc6f415aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:30.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.190+0000 7fc6f941c640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6f40cdde0 con 0x7fc6f415bf60 2024-03-19T01:59:30.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.190+0000 7fc6f941c640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6f40ce0c0 con 0x7fc6f40ceb10 2024-03-19T01:59:30.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.190+0000 7fc6f941c640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc6f415b3f0 con 0x7fc6f415d2d0 2024-03-19T01:59:30.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.190+0000 7fc6f37fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fc6f415bf60 0x7fc6f415c3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:41772/0 (socket says 172.21.15.55:41772) 2024-03-19T01:59:30.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.190+0000 7fc6f37fe640 1 -- 172.21.15.55:0/3612850995 learned_addr learned my addr 172.21.15.55:0/3612850995 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:59:30.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.190+0000 7fc6f2ffd640 1 --2- 172.21.15.55:0/3612850995 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc6f415d2d0 0x7fc6f415aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:30.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.194+0000 7fc6f27fc640 1 -- 172.21.15.55:0/3612850995 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 601059414 0 0) 0x7fc6f40cdde0 con 0x7fc6f415bf60 2024-03-19T01:59:30.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.194+0000 7fc6f27fc640 1 -- 172.21.15.55:0/3612850995 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6e0003530 con 0x7fc6f415bf60 2024-03-19T01:59:30.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.194+0000 7fc6f2ffd640 1 -- 172.21.15.55:0/3612850995 >> v1:172.21.15.84:6789/0 conn(0x7fc6f40ceb10 legacy=0x7fc6f406c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:30.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.194+0000 7fc6f2ffd640 1 -- 172.21.15.55:0/3612850995 >> v1:172.21.15.55:6789/0 conn(0x7fc6f415bf60 legacy=0x7fc6f415c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:30.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.194+0000 7fc6f2ffd640 1 -- 172.21.15.55:0/3612850995 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6f40ce3a0 con 0x7fc6f415d2d0 2024-03-19T01:59:30.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.198+0000 7fc6f2ffd640 1 --2- 172.21.15.55:0/3612850995 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc6f415d2d0 0x7fc6f415aeb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc6e40097f0 tx=0x7fc6e401ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f862d6bc607858e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:30.209 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.210+0000 7fc6f27fc640 1 -- 172.21.15.55:0/3612850995 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e40b5070 con 0x7fc6f415d2d0 2024-03-19T01:59:30.209 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.210+0000 7fc6f27fc640 1 -- 172.21.15.55:0/3612850995 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6e40a89d0 con 0x7fc6f415d2d0 2024-03-19T01:59:30.209 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.210+0000 7fc6f27fc640 1 -- 172.21.15.55:0/3612850995 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e40a8da0 con 0x7fc6f415d2d0 2024-03-19T01:59:30.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.210+0000 7fc6f941c640 1 -- 172.21.15.55:0/3612850995 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc6f415d2d0 msgr2=0x7fc6f415aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:30.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.210+0000 7fc6f941c640 1 --2- 172.21.15.55:0/3612850995 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc6f415d2d0 0x7fc6f415aeb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc6e40097f0 tx=0x7fc6e401ecd0 comp rx=0 tx=0).stop 2024-03-19T01:59:30.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.210+0000 7fc6f941c640 1 -- 172.21.15.55:0/3612850995 shutdown_connections 2024-03-19T01:59:30.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.210+0000 7fc6f941c640 1 --2- 172.21.15.55:0/3612850995 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc6f415d2d0 0x7fc6f415aeb0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:30.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.210+0000 7fc6f941c640 1 -- 172.21.15.55:0/3612850995 >> 172.21.15.55:0/3612850995 conn(0x7fc6f40cfd90 msgr2=0x7fc6f40d01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:30.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.210+0000 7fc6f941c640 1 -- 172.21.15.55:0/3612850995 shutdown_connections 2024-03-19T01:59:30.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.210+0000 7fc6f941c640 1 -- 172.21.15.55:0/3612850995 wait complete. 2024-03-19T01:59:30.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.214+0000 7fc6f941c640 1 Processor -- start 2024-03-19T01:59:30.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.214+0000 7fc6f941c640 1 -- start start 2024-03-19T01:59:30.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.214+0000 7fc6f941c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc6f40ceb10 0x7fc6f40c3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:30.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.214+0000 7fc6f941c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6f415bf60 0x7fc6f41f03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:30.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.214+0000 7fc6f941c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc6f415d2d0 0x7fc6f414e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:30.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.214+0000 7fc6f941c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f415d6b0 con 0x7fc6f40ceb10 2024-03-19T01:59:30.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.214+0000 7fc6f941c640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f415bad0 con 0x7fc6f415bf60 2024-03-19T01:59:30.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.214+0000 7fc6f941c640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc6f406caf0 con 0x7fc6f415d2d0 2024-03-19T01:59:30.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.214+0000 7fc6f2ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc6f40ceb10 0x7fc6f40c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:30.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.214+0000 7fc6ea7fc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6f415bf60 0x7fc6f41f03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:30.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.214+0000 7fc6f2ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc6f40ceb10 0x7fc6f40c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48018/0 (socket says 172.21.15.55:48018) 2024-03-19T01:59:30.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.214+0000 7fc6f2ffd640 1 -- 172.21.15.55:0/1783236332 learned_addr learned my addr 172.21.15.55:0/1783236332 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:30.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.214+0000 7fc6f37fe640 1 --2- 172.21.15.55:0/1783236332 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc6f415d2d0 0x7fc6f414e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:30.217 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.218+0000 7fc6f2ffd640 1 -- 172.21.15.55:0/1783236332 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc6f415d2d0 msgr2=0x7fc6f414e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:30.217 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.218+0000 7fc6f2ffd640 1 --2- 172.21.15.55:0/1783236332 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc6f415d2d0 0x7fc6f414e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:30.217 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.218+0000 7fc6f2ffd640 1 -- 172.21.15.55:0/1783236332 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6f415bf60 msgr2=0x7fc6f41f03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:30.217 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.218+0000 7fc6f2ffd640 1 --2- 172.21.15.55:0/1783236332 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6f415bf60 0x7fc6f41f03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:30.217 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.218+0000 7fc6f2ffd640 1 -- 172.21.15.55:0/1783236332 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6f41efc10 con 0x7fc6f40ceb10 2024-03-19T01:59:30.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.222+0000 7fc6f2ffd640 1 --2- 172.21.15.55:0/1783236332 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc6f40ceb10 0x7fc6f40c3900 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc6e401b8e0 tx=0x7fc6e401ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:30.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.226+0000 7fc6ebfff640 1 -- 172.21.15.55:0/1783236332 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e40b5040 con 0x7fc6f40ceb10 2024-03-19T01:59:30.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.226+0000 7fc6ebfff640 1 -- 172.21.15.55:0/1783236332 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6e40bf070 con 0x7fc6f40ceb10 2024-03-19T01:59:30.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.226+0000 7fc6f941c640 1 -- 172.21.15.55:0/1783236332 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6f40c64e0 con 0x7fc6f40ceb10 2024-03-19T01:59:30.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.226+0000 7fc6ebfff640 1 -- 172.21.15.55:0/1783236332 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e40b19f0 con 0x7fc6f40ceb10 2024-03-19T01:59:30.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.226+0000 7fc6f941c640 1 -- 172.21.15.55:0/1783236332 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6f41f0910 con 0x7fc6f40ceb10 2024-03-19T01:59:30.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.226+0000 7fc6f941c640 1 -- 172.21.15.55:0/1783236332 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6f41efa00 con 0x7fc6f40ceb10 2024-03-19T01:59:30.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.246+0000 7fc6ebfff640 1 -- 172.21.15.55:0/1783236332 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc6e40b1b90 con 0x7fc6f40ceb10 2024-03-19T01:59:30.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.246+0000 7fc6ebfff640 1 --2- 172.21.15.55:0/1783236332 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc6d40676b0 0x7fc6d4069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:30.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.246+0000 7fc6ea7fc640 1 --2- 172.21.15.55:0/1783236332 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc6d40676b0 0x7fc6d4069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:30.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.246+0000 7fc6ea7fc640 1 --2- 172.21.15.55:0/1783236332 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc6d40676b0 0x7fc6d4069b70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc6f40af3f0 tx=0x7fc6dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:30.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.250+0000 7fc6ebfff640 1 -- 172.21.15.55:0/1783236332 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fc6e40f2060 con 0x7fc6f40ceb10 2024-03-19T01:59:30.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.530+0000 7fc6ebfff640 1 -- 172.21.15.55:0/1783236332 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc6e40c9020 con 0x7fc6f40ceb10 2024-03-19T01:59:30.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.750+0000 7fc6f941c640 1 -- 172.21.15.55:0/1783236332 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6f4068a40 con 0x7fc6f40ceb10 2024-03-19T01:59:30.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.830+0000 7fc6ebfff640 1 -- 172.21.15.55:0/1783236332 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fc6e4018830 con 0x7fc6f40ceb10 2024-03-19T01:59:30.828 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:30.828 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:30.829 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:30.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.834+0000 7fc6f941c640 1 -- 172.21.15.55:0/1783236332 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc6d40676b0 msgr2=0x7fc6d4069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:30.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.834+0000 7fc6f941c640 1 --2- 172.21.15.55:0/1783236332 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc6d40676b0 0x7fc6d4069b70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc6f40af3f0 tx=0x7fc6dc009040 comp rx=0 tx=0).stop 2024-03-19T01:59:30.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.834+0000 7fc6f941c640 1 -- 172.21.15.55:0/1783236332 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc6f40ceb10 msgr2=0x7fc6f40c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:30.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.834+0000 7fc6f941c640 1 --2- 172.21.15.55:0/1783236332 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc6f40ceb10 0x7fc6f40c3900 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc6e401b8e0 tx=0x7fc6e401ba70 comp rx=0 tx=0).stop 2024-03-19T01:59:30.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.834+0000 7fc6f941c640 1 -- 172.21.15.55:0/1783236332 shutdown_connections 2024-03-19T01:59:30.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.834+0000 7fc6f941c640 1 --2- 172.21.15.55:0/1783236332 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc6d40676b0 0x7fc6d4069b70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:30.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.834+0000 7fc6f941c640 1 --2- 172.21.15.55:0/1783236332 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc6f415d2d0 0x7fc6f414e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:30.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.834+0000 7fc6f941c640 1 --2- 172.21.15.55:0/1783236332 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6f415bf60 0x7fc6f41f03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:30.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.834+0000 7fc6f941c640 1 --2- 172.21.15.55:0/1783236332 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc6f40ceb10 0x7fc6f40c3900 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:30.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.834+0000 7fc6f941c640 1 -- 172.21.15.55:0/1783236332 >> 172.21.15.55:0/1783236332 conn(0x7fc6f40cfd90 msgr2=0x7fc6f40d0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:30.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.834+0000 7fc6f941c640 1 -- 172.21.15.55:0/1783236332 shutdown_connections 2024-03-19T01:59:30.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:30.834+0000 7fc6f941c640 1 -- 172.21.15.55:0/1783236332 wait complete. 2024-03-19T01:59:31.459 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:31.459 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:32.460 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:32.576 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.578+0000 7f22221dc640 1 Processor -- start 2024-03-19T01:59:32.576 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.578+0000 7f22221dc640 1 -- start start 2024-03-19T01:59:32.577 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.578+0000 7f22221dc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f221c14e930 0x7f221c06b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:32.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.578+0000 7f22221dc640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f221c06be50 0x7f221c1585a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:32.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.578+0000 7f22221dc640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f221c0c7540 con 0x7f221c158ae0 2024-03-19T01:59:32.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.578+0000 7f22221dc640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f221c0c80a0 con 0x7f221c06be50 2024-03-19T01:59:32.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.578+0000 7f22221dc640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f221c0c7800 con 0x7f221c14e930 2024-03-19T01:59:32.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.578+0000 7f221affd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f221c06be50 0x7f221c1585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:32.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.578+0000 7f221b7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f221c14e930 0x7f221c06b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:32.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.582+0000 7f221b7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f221c14e930 0x7f221c06b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:46062/0 (socket says 172.21.15.55:46062) 2024-03-19T01:59:32.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.582+0000 7f221b7fe640 1 -- 172.21.15.55:0/83726756 learned_addr learned my addr 172.21.15.55:0/83726756 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:32.582 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.582+0000 7f221affd640 1 -- 172.21.15.55:0/83726756 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f221c14e930 msgr2=0x7f221c06b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:32.582 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.582+0000 7f221affd640 1 --2- 172.21.15.55:0/83726756 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f221c14e930 0x7f221c06b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:32.582 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.582+0000 7f221affd640 1 -- 172.21.15.55:0/83726756 >> v1:172.21.15.84:6789/0 conn(0x7f221c158ae0 legacy=0x7f221c0c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:32.582 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.582+0000 7f221affd640 1 -- 172.21.15.55:0/83726756 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f221c0c7ae0 con 0x7f221c06be50 2024-03-19T01:59:32.586 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.586+0000 7f221affd640 1 --2- 172.21.15.55:0/83726756 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f221c06be50 0x7f221c1585a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f221c0c8220 tx=0x7f221001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=320430f69a45431 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:32.591 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.594+0000 7f221a7fc640 1 -- 172.21.15.55:0/83726756 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22100b5070 con 0x7f221c06be50 2024-03-19T01:59:32.591 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.594+0000 7f221a7fc640 1 -- 172.21.15.55:0/83726756 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22100a8990 con 0x7f221c06be50 2024-03-19T01:59:32.591 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.594+0000 7f221a7fc640 1 -- 172.21.15.55:0/83726756 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22100a8d60 con 0x7f221c06be50 2024-03-19T01:59:32.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.594+0000 7f22221dc640 1 -- 172.21.15.55:0/83726756 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f221c06be50 msgr2=0x7f221c1585a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:32.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.594+0000 7f22221dc640 1 --2- 172.21.15.55:0/83726756 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f221c06be50 0x7f221c1585a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f221c0c8220 tx=0x7f221001ec90 comp rx=0 tx=0).stop 2024-03-19T01:59:32.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.594+0000 7f22221dc640 1 -- 172.21.15.55:0/83726756 shutdown_connections 2024-03-19T01:59:32.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.594+0000 7f22221dc640 1 --2- 172.21.15.55:0/83726756 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f221c06be50 0x7f221c1585a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:32.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.594+0000 7f22221dc640 1 --2- 172.21.15.55:0/83726756 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f221c14e930 0x7f221c06b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:32.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.594+0000 7f22221dc640 1 -- 172.21.15.55:0/83726756 >> 172.21.15.55:0/83726756 conn(0x7f221c0cc9c0 msgr2=0x7f221c0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:32.592 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.594+0000 7f22221dc640 1 -- 172.21.15.55:0/83726756 shutdown_connections 2024-03-19T01:59:32.593 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.594+0000 7f22221dc640 1 -- 172.21.15.55:0/83726756 wait complete. 2024-03-19T01:59:32.593 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.594+0000 7f22221dc640 1 Processor -- start 2024-03-19T01:59:32.593 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.594+0000 7f22221dc640 1 -- start start 2024-03-19T01:59:32.594 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.594+0000 7f22221dc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f221c06be50 0x7f221c15b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:32.594 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.594+0000 7f22221dc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f221c14e930 0x7f221c156f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:32.594 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.594+0000 7f22221dc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f221c158ae0 0x7f221c159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:32.594 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.594+0000 7f22221dc640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f221c14ed10 con 0x7f221c158ae0 2024-03-19T01:59:32.594 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.594+0000 7f22221dc640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f221c158080 con 0x7f221c14e930 2024-03-19T01:59:32.594 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.594+0000 7f22221dc640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f221c073600 con 0x7f221c06be50 2024-03-19T01:59:32.595 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.594+0000 7f221bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f221c158ae0 0x7f221c159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:32.595 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.598+0000 7f221b7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f221c06be50 0x7f221c15b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:32.595 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.598+0000 7f221affd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f221c14e930 0x7f221c156f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:32.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.598+0000 7f221bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f221c158ae0 0x7f221c159a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48050/0 (socket says 172.21.15.55:48050) 2024-03-19T01:59:32.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.598+0000 7f221bfff640 1 -- 172.21.15.55:0/1477725618 learned_addr learned my addr 172.21.15.55:0/1477725618 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:32.600 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.602+0000 7f221bfff640 1 -- 172.21.15.55:0/1477725618 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f221c06be50 msgr2=0x7f221c15b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:32.600 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.602+0000 7f221bfff640 1 --2- 172.21.15.55:0/1477725618 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f221c06be50 0x7f221c15b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:32.600 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.602+0000 7f221bfff640 1 -- 172.21.15.55:0/1477725618 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f221c14e930 msgr2=0x7f221c156f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:32.600 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.602+0000 7f221bfff640 1 --2- 172.21.15.55:0/1477725618 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f221c14e930 0x7f221c156f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:32.600 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.602+0000 7f221bfff640 1 -- 172.21.15.55:0/1477725618 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f221c0c5340 con 0x7f221c158ae0 2024-03-19T01:59:32.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.606+0000 7f221bfff640 1 --2- 172.21.15.55:0/1477725618 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f221c158ae0 0x7f221c159a80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f221400c740 tx=0x7f221400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:32.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.610+0000 7f2203fff640 1 -- 172.21.15.55:0/1477725618 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2214016020 con 0x7f221c158ae0 2024-03-19T01:59:32.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.610+0000 7f2203fff640 1 -- 172.21.15.55:0/1477725618 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f221400ce50 con 0x7f221c158ae0 2024-03-19T01:59:32.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.610+0000 7f2203fff640 1 -- 172.21.15.55:0/1477725618 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2214003980 con 0x7f221c158ae0 2024-03-19T01:59:32.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.610+0000 7f22221dc640 1 -- 172.21.15.55:0/1477725618 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f221c0c5ea0 con 0x7f221c158ae0 2024-03-19T01:59:32.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.610+0000 7f22221dc640 1 -- 172.21.15.55:0/1477725618 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f221c0c58e0 con 0x7f221c158ae0 2024-03-19T01:59:32.609 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.610+0000 7f22221dc640 1 -- 172.21.15.55:0/1477725618 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f221c068a40 con 0x7f221c158ae0 2024-03-19T01:59:32.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.642+0000 7f2203fff640 1 -- 172.21.15.55:0/1477725618 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f2214003b20 con 0x7f221c158ae0 2024-03-19T01:59:32.642 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.642+0000 7f2203fff640 1 --2- 172.21.15.55:0/1477725618 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2208067660 0x7f2208069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:32.642 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.642+0000 7f221b7fe640 1 --2- 172.21.15.55:0/1477725618 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2208067660 0x7f2208069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:32.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.646+0000 7f221b7fe640 1 --2- 172.21.15.55:0/1477725618 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2208067660 0x7f2208069b20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f220c009b50 tx=0x7f220c009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:32.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.650+0000 7f2203fff640 1 -- 172.21.15.55:0/1477725618 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f22140548f0 con 0x7f221c158ae0 2024-03-19T01:59:32.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:32.938+0000 7f2203fff640 1 -- 172.21.15.55:0/1477725618 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2214020020 con 0x7f221c158ae0 2024-03-19T01:59:33.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:33.150+0000 7f22221dc640 1 -- 172.21.15.55:0/1477725618 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f221c06b2a0 con 0x7f221c158ae0 2024-03-19T01:59:33.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:33.194+0000 7f2203fff640 1 -- 172.21.15.55:0/1477725618 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f2214056850 con 0x7f221c158ae0 2024-03-19T01:59:33.195 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:33.195 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:33.195 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:33.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:33.198+0000 7f22221dc640 1 -- 172.21.15.55:0/1477725618 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2208067660 msgr2=0x7f2208069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:33.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:33.198+0000 7f22221dc640 1 --2- 172.21.15.55:0/1477725618 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2208067660 0x7f2208069b20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f220c009b50 tx=0x7f220c009240 comp rx=0 tx=0).stop 2024-03-19T01:59:33.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:33.198+0000 7f22221dc640 1 -- 172.21.15.55:0/1477725618 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f221c158ae0 msgr2=0x7f221c159a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:33.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:33.198+0000 7f22221dc640 1 --2- 172.21.15.55:0/1477725618 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f221c158ae0 0x7f221c159a80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f221400c740 tx=0x7f221400cc00 comp rx=0 tx=0).stop 2024-03-19T01:59:33.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:33.198+0000 7f22221dc640 1 -- 172.21.15.55:0/1477725618 shutdown_connections 2024-03-19T01:59:33.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:33.198+0000 7f22221dc640 1 --2- 172.21.15.55:0/1477725618 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2208067660 0x7f2208069b20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:33.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:33.198+0000 7f22221dc640 1 --2- 172.21.15.55:0/1477725618 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f221c158ae0 0x7f221c159a80 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:33.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:33.198+0000 7f22221dc640 1 --2- 172.21.15.55:0/1477725618 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f221c14e930 0x7f221c156f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:33.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:33.198+0000 7f22221dc640 1 --2- 172.21.15.55:0/1477725618 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f221c06be50 0x7f221c15b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:33.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:33.198+0000 7f22221dc640 1 -- 172.21.15.55:0/1477725618 >> 172.21.15.55:0/1477725618 conn(0x7f221c0cc9c0 msgr2=0x7f221c0c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:33.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:33.198+0000 7f22221dc640 1 -- 172.21.15.55:0/1477725618 shutdown_connections 2024-03-19T01:59:33.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:33.198+0000 7f22221dc640 1 -- 172.21.15.55:0/1477725618 wait complete. 2024-03-19T01:59:33.818 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:33.819 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:34.820 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:34.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.934+0000 7f5381dbd640 1 Processor -- start 2024-03-19T01:59:34.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.934+0000 7f5381dbd640 1 -- start start 2024-03-19T01:59:34.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.934+0000 7f5381dbd640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f537c154290 0x7f537c15e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:34.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.934+0000 7f5381dbd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f537c15ad40 0x7f537c154670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:34.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.934+0000 7f5381dbd640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f537c06bc90 con 0x7f537c0ca310 2024-03-19T01:59:34.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.934+0000 7f5381dbd640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f537c15b2c0 con 0x7f537c15ad40 2024-03-19T01:59:34.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.934+0000 7f5381dbd640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f537c06c1d0 con 0x7f537c154290 2024-03-19T01:59:34.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.938+0000 7f537bfff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f537c15ad40 0x7f537c154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:34.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.938+0000 7f537b7fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f537c154290 0x7f537c15e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:34.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.938+0000 7f537bfff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f537c15ad40 0x7f537c154670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48058/0 (socket says 172.21.15.55:48058) 2024-03-19T01:59:34.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.938+0000 7f537bfff640 1 -- 172.21.15.55:0/559790476 learned_addr learned my addr 172.21.15.55:0/559790476 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:34.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.938+0000 7f537bfff640 1 -- 172.21.15.55:0/559790476 >> v2:172.21.15.84:3300/0 conn(0x7f537c154290 msgr2=0x7f537c15e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:34.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.938+0000 7f537bfff640 1 --2- 172.21.15.55:0/559790476 >> v2:172.21.15.84:3300/0 conn(0x7f537c154290 0x7f537c15e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:34.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.938+0000 7f537bfff640 1 -- 172.21.15.55:0/559790476 >> v1:172.21.15.84:6789/0 conn(0x7f537c0ca310 legacy=0x7f537c070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:34.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.938+0000 7f537bfff640 1 -- 172.21.15.55:0/559790476 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f537c06bf30 con 0x7f537c15ad40 2024-03-19T01:59:34.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.942+0000 7f537bfff640 1 --2- 172.21.15.55:0/559790476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f537c15ad40 0x7f537c154670 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f537c071360 tx=0x7f537401c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3552e1ca53809e08 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:34.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.946+0000 7f537a7fc640 1 -- 172.21.15.55:0/559790476 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53740a5780 con 0x7f537c15ad40 2024-03-19T01:59:34.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.946+0000 7f537a7fc640 1 -- 172.21.15.55:0/559790476 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53740a5920 con 0x7f537c15ad40 2024-03-19T01:59:34.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.946+0000 7f537a7fc640 1 -- 172.21.15.55:0/559790476 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53740a5cf0 con 0x7f537c15ad40 2024-03-19T01:59:34.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.950+0000 7f5381dbd640 1 -- 172.21.15.55:0/559790476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f537c15ad40 msgr2=0x7f537c154670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:34.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.950+0000 7f5381dbd640 1 --2- 172.21.15.55:0/559790476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f537c15ad40 0x7f537c154670 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f537c071360 tx=0x7f537401c9b0 comp rx=0 tx=0).stop 2024-03-19T01:59:34.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.950+0000 7f5381dbd640 1 -- 172.21.15.55:0/559790476 shutdown_connections 2024-03-19T01:59:34.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.950+0000 7f5381dbd640 1 --2- 172.21.15.55:0/559790476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f537c15ad40 0x7f537c154670 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:34.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.950+0000 7f5381dbd640 1 --2- 172.21.15.55:0/559790476 >> v2:172.21.15.84:3300/0 conn(0x7f537c154290 0x7f537c15e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:34.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.950+0000 7f5381dbd640 1 -- 172.21.15.55:0/559790476 >> 172.21.15.55:0/559790476 conn(0x7f537c06c580 msgr2=0x7f537c06c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:34.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.950+0000 7f5381dbd640 1 -- 172.21.15.55:0/559790476 shutdown_connections 2024-03-19T01:59:34.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.950+0000 7f5381dbd640 1 -- 172.21.15.55:0/559790476 wait complete. 2024-03-19T01:59:34.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.950+0000 7f5381dbd640 1 Processor -- start 2024-03-19T01:59:34.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.950+0000 7f5381dbd640 1 -- start start 2024-03-19T01:59:34.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.950+0000 7f5381dbd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f537c0ca310 0x7f537c0c4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:34.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.950+0000 7f5381dbd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f537c154290 0x7f537c0c52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:34.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.950+0000 7f5381dbd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f537c15ad40 0x7f537c0cab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:34.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.950+0000 7f5381dbd640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f537c0b2460 con 0x7f537c154290 2024-03-19T01:59:34.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.950+0000 7f5381dbd640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f537c0b22e0 con 0x7f537c0ca310 2024-03-19T01:59:34.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.950+0000 7f5381dbd640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f537c070d50 con 0x7f537c15ad40 2024-03-19T01:59:34.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.950+0000 7f537affd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f537c154290 0x7f537c0c52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:34.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.950+0000 7f537bfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f537c15ad40 0x7f537c0cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:34.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.950+0000 7f537b7fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f537c0ca310 0x7f537c0c4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:34.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.950+0000 7f537affd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f537c154290 0x7f537c0c52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48072/0 (socket says 172.21.15.55:48072) 2024-03-19T01:59:34.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.950+0000 7f537affd640 1 -- 172.21.15.55:0/778739297 learned_addr learned my addr 172.21.15.55:0/778739297 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:34.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.954+0000 7f537affd640 1 -- 172.21.15.55:0/778739297 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f537c15ad40 msgr2=0x7f537c0cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:34.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.954+0000 7f537affd640 1 --2- 172.21.15.55:0/778739297 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f537c15ad40 0x7f537c0cab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:34.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.954+0000 7f537affd640 1 -- 172.21.15.55:0/778739297 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f537c0ca310 msgr2=0x7f537c0c4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:34.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.954+0000 7f537affd640 1 --2- 172.21.15.55:0/778739297 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f537c0ca310 0x7f537c0c4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:34.954 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.954+0000 7f537affd640 1 -- 172.21.15.55:0/778739297 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f537c152ba0 con 0x7f537c154290 2024-03-19T01:59:34.956 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.958+0000 7f537affd640 1 --2- 172.21.15.55:0/778739297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f537c154290 0x7f537c0c52c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f537000b3e0 tx=0x7f537000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:34.960 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.962+0000 7f5363fff640 1 -- 172.21.15.55:0/778739297 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5370015020 con 0x7f537c154290 2024-03-19T01:59:34.960 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.962+0000 7f5363fff640 1 -- 172.21.15.55:0/778739297 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5370011850 con 0x7f537c154290 2024-03-19T01:59:34.960 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.962+0000 7f5363fff640 1 -- 172.21.15.55:0/778739297 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5370011c40 con 0x7f537c154290 2024-03-19T01:59:34.960 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.962+0000 7f5381dbd640 1 -- 172.21.15.55:0/778739297 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f537c1539c0 con 0x7f537c154290 2024-03-19T01:59:34.960 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.962+0000 7f5381dbd640 1 -- 172.21.15.55:0/778739297 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f537c159640 con 0x7f537c154290 2024-03-19T01:59:34.961 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:34.962+0000 7f5381dbd640 1 -- 172.21.15.55:0/778739297 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f537c068a40 con 0x7f537c154290 2024-03-19T01:59:35.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:35.238+0000 7f5363fff640 1 -- 172.21.15.55:0/778739297 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f5370018020 con 0x7f537c154290 2024-03-19T01:59:35.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:35.238+0000 7f5363fff640 1 --2- 172.21.15.55:0/778739297 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5368067660 0x7f5368069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:35.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:35.238+0000 7f537b7fe640 1 --2- 172.21.15.55:0/778739297 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5368067660 0x7f5368069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:35.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:35.238+0000 7f537b7fe640 1 --2- 172.21.15.55:0/778739297 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5368067660 0x7f5368069b20 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f536c009ac0 tx=0x7f536c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:35.268 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:35.270+0000 7f5363fff640 1 -- 172.21.15.55:0/778739297 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f5370011de0 con 0x7f537c154290 2024-03-19T01:59:35.268 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:35.270+0000 7f5363fff640 1 -- 172.21.15.55:0/778739297 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f537001f020 con 0x7f537c154290 2024-03-19T01:59:35.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:35.498+0000 7f5381dbd640 1 -- 172.21.15.55:0/778739297 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f537c06b960 con 0x7f537c154290 2024-03-19T01:59:35.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:35.514+0000 7f5363fff640 1 -- 172.21.15.55:0/778739297 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f5370056290 con 0x7f537c154290 2024-03-19T01:59:35.513 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:35.513 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:35.513 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:35.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:35.518+0000 7f5381dbd640 1 -- 172.21.15.55:0/778739297 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5368067660 msgr2=0x7f5368069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:35.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:35.518+0000 7f5381dbd640 1 --2- 172.21.15.55:0/778739297 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5368067660 0x7f5368069b20 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f536c009ac0 tx=0x7f536c009190 comp rx=0 tx=0).stop 2024-03-19T01:59:35.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:35.518+0000 7f5381dbd640 1 -- 172.21.15.55:0/778739297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f537c154290 msgr2=0x7f537c0c52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:35.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:35.518+0000 7f5381dbd640 1 --2- 172.21.15.55:0/778739297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f537c154290 0x7f537c0c52c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f537000b3e0 tx=0x7f537000b8a0 comp rx=0 tx=0).stop 2024-03-19T01:59:35.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:35.518+0000 7f5381dbd640 1 -- 172.21.15.55:0/778739297 shutdown_connections 2024-03-19T01:59:35.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:35.518+0000 7f5381dbd640 1 --2- 172.21.15.55:0/778739297 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5368067660 0x7f5368069b20 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:35.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:35.518+0000 7f5381dbd640 1 --2- 172.21.15.55:0/778739297 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f537c15ad40 0x7f537c0cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:35.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:35.518+0000 7f5381dbd640 1 --2- 172.21.15.55:0/778739297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f537c154290 0x7f537c0c52c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:35.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:35.518+0000 7f5381dbd640 1 --2- 172.21.15.55:0/778739297 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f537c0ca310 0x7f537c0c4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:35.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:35.518+0000 7f5381dbd640 1 -- 172.21.15.55:0/778739297 >> 172.21.15.55:0/778739297 conn(0x7f537c06c580 msgr2=0x7f537c158970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:35.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:35.518+0000 7f5381dbd640 1 -- 172.21.15.55:0/778739297 shutdown_connections 2024-03-19T01:59:35.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:35.518+0000 7f5381dbd640 1 -- 172.21.15.55:0/778739297 wait complete. 2024-03-19T01:59:36.136 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:36.136 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:37.137 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:37.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.254+0000 7f2b7dc3e640 1 Processor -- start 2024-03-19T01:59:37.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.254+0000 7f2b7dc3e640 1 -- start start 2024-03-19T01:59:37.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.258+0000 7f2b7dc3e640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f2b7806b2a0 0x7f2b7815ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:37.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.258+0000 7f2b7dc3e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b780c8840 con 0x7f2b7806c110 2024-03-19T01:59:37.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.258+0000 7f2b7dc3e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b780c8ab0 con 0x7f2b78154300 2024-03-19T01:59:37.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.258+0000 7f2b7dc3e640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f2b7815b260 con 0x7f2b7806b2a0 2024-03-19T01:59:37.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.258+0000 7f2b77fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f2b7806c110 0x7f2b78164c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:41776/0 (socket says 172.21.15.55:41776) 2024-03-19T01:59:37.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.258+0000 7f2b77fff640 1 -- 172.21.15.55:0/1408861983 learned_addr learned my addr 172.21.15.55:0/1408861983 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:59:37.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.258+0000 7f2b6ffff640 1 --2- 172.21.15.55:0/1408861983 >> v2:172.21.15.84:3300/0 conn(0x7f2b7806b2a0 0x7f2b7815ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:37.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.262+0000 7f2b76ffd640 1 -- 172.21.15.55:0/1408861983 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1687416618 0 0) 0x7f2b780c8840 con 0x7f2b7806c110 2024-03-19T01:59:37.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.262+0000 7f2b76ffd640 1 -- 172.21.15.55:0/1408861983 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b64003530 con 0x7f2b7806c110 2024-03-19T01:59:37.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.262+0000 7f2b6ffff640 1 -- 172.21.15.55:0/1408861983 >> v1:172.21.15.84:6789/0 conn(0x7f2b78154300 legacy=0x7f2b7815e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:37.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.262+0000 7f2b6ffff640 1 -- 172.21.15.55:0/1408861983 >> v1:172.21.15.55:6789/0 conn(0x7f2b7806c110 legacy=0x7f2b78164c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:37.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.262+0000 7f2b6ffff640 1 -- 172.21.15.55:0/1408861983 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b780c8d90 con 0x7f2b7806b2a0 2024-03-19T01:59:37.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.266+0000 7f2b6ffff640 1 --2- 172.21.15.55:0/1408861983 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b7806b2a0 0x7f2b7815ad20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2b600097c0 tx=0x7f2b6001c430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c979c0c60ada6a63 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:37.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.270+0000 7f2b76ffd640 1 -- 172.21.15.55:0/1408861983 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b600ab430 con 0x7f2b7806b2a0 2024-03-19T01:59:37.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.274+0000 7f2b76ffd640 1 -- 172.21.15.55:0/1408861983 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b600b0070 con 0x7f2b7806b2a0 2024-03-19T01:59:37.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.274+0000 7f2b76ffd640 1 -- 172.21.15.55:0/1408861983 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b600ab7f0 con 0x7f2b7806b2a0 2024-03-19T01:59:37.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.274+0000 7f2b7dc3e640 1 -- 172.21.15.55:0/1408861983 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b7806b2a0 msgr2=0x7f2b7815ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:37.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.274+0000 7f2b7dc3e640 1 --2- 172.21.15.55:0/1408861983 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b7806b2a0 0x7f2b7815ad20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2b600097c0 tx=0x7f2b6001c430 comp rx=0 tx=0).stop 2024-03-19T01:59:37.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.274+0000 7f2b7dc3e640 1 -- 172.21.15.55:0/1408861983 shutdown_connections 2024-03-19T01:59:37.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.274+0000 7f2b7dc3e640 1 --2- 172.21.15.55:0/1408861983 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b7806b2a0 0x7f2b7815ad20 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:37.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.274+0000 7f2b7dc3e640 1 -- 172.21.15.55:0/1408861983 >> 172.21.15.55:0/1408861983 conn(0x7f2b7806fb40 msgr2=0x7f2b7806ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:37.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.274+0000 7f2b7dc3e640 1 -- 172.21.15.55:0/1408861983 shutdown_connections 2024-03-19T01:59:37.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.274+0000 7f2b7dc3e640 1 -- 172.21.15.55:0/1408861983 wait complete. 2024-03-19T01:59:37.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.274+0000 7f2b7dc3e640 1 Processor -- start 2024-03-19T01:59:37.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.274+0000 7f2b7dc3e640 1 -- start start 2024-03-19T01:59:37.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.274+0000 7f2b7dc3e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2b7806b2a0 0x7f2b781f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:37.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.274+0000 7f2b7dc3e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b7806c110 0x7f2b7814d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:37.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.274+0000 7f2b7dc3e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b78154300 0x7f2b78147e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:37.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.274+0000 7f2b7dc3e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7815e7d0 con 0x7f2b78154300 2024-03-19T01:59:37.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.274+0000 7f2b7dc3e640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7815dba0 con 0x7f2b7806c110 2024-03-19T01:59:37.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.274+0000 7f2b7dc3e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2b780c74c0 con 0x7f2b7806b2a0 2024-03-19T01:59:37.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.278+0000 7f2b77fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b78154300 0x7f2b78147e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:37.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.278+0000 7f2b777fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2b7806b2a0 0x7f2b781f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:37.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.278+0000 7f2b777fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2b7806b2a0 0x7f2b781f2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:46080/0 (socket says 172.21.15.55:46080) 2024-03-19T01:59:37.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.278+0000 7f2b777fe640 1 -- 172.21.15.55:0/2323228945 learned_addr learned my addr 172.21.15.55:0/2323228945 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:37.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.278+0000 7f2b6ffff640 1 --2- 172.21.15.55:0/2323228945 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b7806c110 0x7f2b7814d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:37.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.282+0000 7f2b77fff640 1 -- 172.21.15.55:0/2323228945 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2b7806b2a0 msgr2=0x7f2b781f2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:37.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.282+0000 7f2b77fff640 1 --2- 172.21.15.55:0/2323228945 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2b7806b2a0 0x7f2b781f2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:37.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.282+0000 7f2b77fff640 1 -- 172.21.15.55:0/2323228945 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b7806c110 msgr2=0x7f2b7814d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:37.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.282+0000 7f2b77fff640 1 --2- 172.21.15.55:0/2323228945 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b7806c110 0x7f2b7814d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:37.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.282+0000 7f2b77fff640 1 -- 172.21.15.55:0/2323228945 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b7815cce0 con 0x7f2b78154300 2024-03-19T01:59:37.284 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.286+0000 7f2b77fff640 1 --2- 172.21.15.55:0/2323228945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b78154300 0x7f2b78147e40 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f2b7000ca80 tx=0x7f2b7000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:37.288 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.290+0000 7f2b74ff9640 1 -- 172.21.15.55:0/2323228945 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b70017070 con 0x7f2b78154300 2024-03-19T01:59:37.288 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.290+0000 7f2b74ff9640 1 -- 172.21.15.55:0/2323228945 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b7000abd0 con 0x7f2b78154300 2024-03-19T01:59:37.288 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.290+0000 7f2b74ff9640 1 -- 172.21.15.55:0/2323228945 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b70012400 con 0x7f2b78154300 2024-03-19T01:59:37.288 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.290+0000 7f2b7dc3e640 1 -- 172.21.15.55:0/2323228945 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b7815cfc0 con 0x7f2b78154300 2024-03-19T01:59:37.288 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.290+0000 7f2b7dc3e640 1 -- 172.21.15.55:0/2323228945 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b7815d580 con 0x7f2b78154300 2024-03-19T01:59:37.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.290+0000 7f2b7dc3e640 1 -- 172.21.15.55:0/2323228945 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b7814ac30 con 0x7f2b78154300 2024-03-19T01:59:37.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.566+0000 7f2b74ff9640 1 -- 172.21.15.55:0/2323228945 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f2b7001a070 con 0x7f2b78154300 2024-03-19T01:59:37.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.566+0000 7f2b74ff9640 1 --2- 172.21.15.55:0/2323228945 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2b580676e0 0x7f2b58069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:37.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.566+0000 7f2b777fe640 1 --2- 172.21.15.55:0/2323228945 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2b580676e0 0x7f2b58069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:37.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.566+0000 7f2b777fe640 1 --2- 172.21.15.55:0/2323228945 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2b580676e0 0x7f2b58069ba0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2b68001960 tx=0x7f2b68009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:37.584 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.586+0000 7f2b74ff9640 1 -- 172.21.15.55:0/2323228945 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f2b70054110 con 0x7f2b78154300 2024-03-19T01:59:37.585 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.586+0000 7f2b74ff9640 1 -- 172.21.15.55:0/2323228945 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2b70056070 con 0x7f2b78154300 2024-03-19T01:59:37.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.830+0000 7f2b7dc3e640 1 -- 172.21.15.55:0/2323228945 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b78068a40 con 0x7f2b78154300 2024-03-19T01:59:37.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.850+0000 7f2b74ff9640 1 -- 172.21.15.55:0/2323228945 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f2b7005a110 con 0x7f2b78154300 2024-03-19T01:59:37.851 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:37.851 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:37.851 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:37.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.854+0000 7f2b7dc3e640 1 -- 172.21.15.55:0/2323228945 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2b580676e0 msgr2=0x7f2b58069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:37.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.854+0000 7f2b7dc3e640 1 --2- 172.21.15.55:0/2323228945 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2b580676e0 0x7f2b58069ba0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2b68001960 tx=0x7f2b68009040 comp rx=0 tx=0).stop 2024-03-19T01:59:37.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.854+0000 7f2b7dc3e640 1 -- 172.21.15.55:0/2323228945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b78154300 msgr2=0x7f2b78147e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:37.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.854+0000 7f2b7dc3e640 1 --2- 172.21.15.55:0/2323228945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b78154300 0x7f2b78147e40 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f2b7000ca80 tx=0x7f2b7000cf40 comp rx=0 tx=0).stop 2024-03-19T01:59:37.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.854+0000 7f2b7dc3e640 1 -- 172.21.15.55:0/2323228945 shutdown_connections 2024-03-19T01:59:37.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.854+0000 7f2b7dc3e640 1 --2- 172.21.15.55:0/2323228945 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2b580676e0 0x7f2b58069ba0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:37.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.854+0000 7f2b7dc3e640 1 --2- 172.21.15.55:0/2323228945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b78154300 0x7f2b78147e40 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:37.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.854+0000 7f2b7dc3e640 1 --2- 172.21.15.55:0/2323228945 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b7806c110 0x7f2b7814d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:37.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.854+0000 7f2b7dc3e640 1 --2- 172.21.15.55:0/2323228945 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2b7806b2a0 0x7f2b781f2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:37.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.854+0000 7f2b7dc3e640 1 -- 172.21.15.55:0/2323228945 >> 172.21.15.55:0/2323228945 conn(0x7f2b7806fb40 msgr2=0x7f2b780c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:37.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.854+0000 7f2b7dc3e640 1 -- 172.21.15.55:0/2323228945 shutdown_connections 2024-03-19T01:59:37.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:37.854+0000 7f2b7dc3e640 1 -- 172.21.15.55:0/2323228945 wait complete. 2024-03-19T01:59:38.474 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:38.474 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:39.475 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:39.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.602+0000 7f46d3168640 1 Processor -- start 2024-03-19T01:59:39.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.602+0000 7f46d3168640 1 -- start start 2024-03-19T01:59:39.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.606+0000 7f46d3168640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f46cc06c610 0x7f46cc15ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:39.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.606+0000 7f46d3168640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46cc06c9f0 0x7f46cc0c92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:39.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.606+0000 7f46d3168640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46cc070e80 con 0x7f46cc0c9800 2024-03-19T01:59:39.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.606+0000 7f46d3168640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f46cc15b250 con 0x7f46cc06c610 2024-03-19T01:59:39.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.606+0000 7f46d3168640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f46cc06fb00 con 0x7f46cc06c9f0 2024-03-19T01:59:39.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.606+0000 7f46cbfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46cc06c9f0 0x7f46cc0c92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:39.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.606+0000 7f46d0f04640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f46cc06c610 0x7f46cc15ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:39.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.606+0000 7f46cbfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46cc06c9f0 0x7f46cc0c92c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:56110/0 (socket says 172.21.15.55:56110) 2024-03-19T01:59:39.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.606+0000 7f46cbfff640 1 -- 172.21.15.55:0/2793997620 learned_addr learned my addr 172.21.15.55:0/2793997620 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:39.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.610+0000 7f46cbfff640 1 -- 172.21.15.55:0/2793997620 >> v2:172.21.15.84:3300/0 conn(0x7f46cc06c610 msgr2=0x7f46cc15ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:39.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.610+0000 7f46cbfff640 1 --2- 172.21.15.55:0/2793997620 >> v2:172.21.15.84:3300/0 conn(0x7f46cc06c610 0x7f46cc15ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:39.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.610+0000 7f46cbfff640 1 -- 172.21.15.55:0/2793997620 >> v1:172.21.15.84:6789/0 conn(0x7f46cc0c9800 legacy=0x7f46cc1427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:39.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.610+0000 7f46cbfff640 1 -- 172.21.15.55:0/2793997620 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46cc06fde0 con 0x7f46cc06c9f0 2024-03-19T01:59:39.612 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.614+0000 7f46cbfff640 1 --2- 172.21.15.55:0/2793997620 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46cc06c9f0 0x7f46cc0c92c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f46b8009870 tx=0x7f46b801ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=359a7eb58d803b3f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:39.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.622+0000 7f46cb7fe640 1 -- 172.21.15.55:0/2793997620 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b80b5070 con 0x7f46cc06c9f0 2024-03-19T01:59:39.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.622+0000 7f46cb7fe640 1 -- 172.21.15.55:0/2793997620 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46b80a89d0 con 0x7f46cc06c9f0 2024-03-19T01:59:39.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.622+0000 7f46cb7fe640 1 -- 172.21.15.55:0/2793997620 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b80a8da0 con 0x7f46cc06c9f0 2024-03-19T01:59:39.621 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.622+0000 7f46d3168640 1 -- 172.21.15.55:0/2793997620 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46cc06c9f0 msgr2=0x7f46cc0c92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:39.621 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.622+0000 7f46d3168640 1 --2- 172.21.15.55:0/2793997620 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46cc06c9f0 0x7f46cc0c92c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f46b8009870 tx=0x7f46b801ed10 comp rx=0 tx=0).stop 2024-03-19T01:59:39.621 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.622+0000 7f46d3168640 1 -- 172.21.15.55:0/2793997620 shutdown_connections 2024-03-19T01:59:39.621 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.622+0000 7f46d3168640 1 --2- 172.21.15.55:0/2793997620 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46cc06c9f0 0x7f46cc0c92c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:39.622 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.622+0000 7f46d3168640 1 --2- 172.21.15.55:0/2793997620 >> v2:172.21.15.84:3300/0 conn(0x7f46cc06c610 0x7f46cc15ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:39.622 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.622+0000 7f46d3168640 1 -- 172.21.15.55:0/2793997620 >> 172.21.15.55:0/2793997620 conn(0x7f46cc067600 msgr2=0x7f46cc071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:39.622 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.622+0000 7f46d3168640 1 -- 172.21.15.55:0/2793997620 shutdown_connections 2024-03-19T01:59:39.622 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.622+0000 7f46d3168640 1 -- 172.21.15.55:0/2793997620 wait complete. 2024-03-19T01:59:39.623 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.626+0000 7f46d3168640 1 Processor -- start 2024-03-19T01:59:39.623 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.626+0000 7f46d3168640 1 -- start start 2024-03-19T01:59:39.624 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.626+0000 7f46d3168640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46cc06c610 0x7f46cc0c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:39.624 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.626+0000 7f46d3168640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46cc06c9f0 0x7f46cc141c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:39.624 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.626+0000 7f46d3168640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46cc0c9800 0x7f46cc0c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:39.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.626+0000 7f46d3168640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc070680 con 0x7f46cc06c9f0 2024-03-19T01:59:39.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.626+0000 7f46d3168640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc0703a0 con 0x7f46cc06c610 2024-03-19T01:59:39.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.626+0000 7f46d3168640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f46cc070960 con 0x7f46cc0c9800 2024-03-19T01:59:39.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.626+0000 7f46d1705640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46cc0c9800 0x7f46cc0c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:39.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.626+0000 7f46cbfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46cc06c9f0 0x7f46cc141c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:39.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.626+0000 7f46d1705640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46cc0c9800 0x7f46cc0c8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:56116/0 (socket says 172.21.15.55:56116) 2024-03-19T01:59:39.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.626+0000 7f46d1705640 1 -- 172.21.15.55:0/2874673328 learned_addr learned my addr 172.21.15.55:0/2874673328 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:39.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.626+0000 7f46d0f04640 1 --2- 172.21.15.55:0/2874673328 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46cc06c610 0x7f46cc0c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:39.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.630+0000 7f46d1705640 1 -- 172.21.15.55:0/2874673328 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46cc06c610 msgr2=0x7f46cc0c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:39.631 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.630+0000 7f46d1705640 1 --2- 172.21.15.55:0/2874673328 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46cc06c610 0x7f46cc0c45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:39.631 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.630+0000 7f46d1705640 1 -- 172.21.15.55:0/2874673328 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46cc06c9f0 msgr2=0x7f46cc141c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:39.631 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.630+0000 7f46d1705640 1 --2- 172.21.15.55:0/2874673328 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46cc06c9f0 0x7f46cc141c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:39.631 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.630+0000 7f46d1705640 1 -- 172.21.15.55:0/2874673328 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46cc15b960 con 0x7f46cc0c9800 2024-03-19T01:59:39.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.634+0000 7f46d1705640 1 --2- 172.21.15.55:0/2874673328 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46cc0c9800 0x7f46cc0c8dc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f46c400b960 tx=0x7f46c400be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:39.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.642+0000 7f46c97fa640 1 -- 172.21.15.55:0/2874673328 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c4019070 con 0x7f46cc0c9800 2024-03-19T01:59:39.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.642+0000 7f46c97fa640 1 -- 172.21.15.55:0/2874673328 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46c400cb20 con 0x7f46cc0c9800 2024-03-19T01:59:39.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.642+0000 7f46c97fa640 1 -- 172.21.15.55:0/2874673328 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c4014440 con 0x7f46cc0c9800 2024-03-19T01:59:39.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.642+0000 7f46d3168640 1 -- 172.21.15.55:0/2874673328 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46cc0c2db0 con 0x7f46cc0c9800 2024-03-19T01:59:39.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.646+0000 7f46d3168640 1 -- 172.21.15.55:0/2874673328 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f46cc140230 con 0x7f46cc0c9800 2024-03-19T01:59:39.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.646+0000 7f46d3168640 1 -- 172.21.15.55:0/2874673328 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46cc068a40 con 0x7f46cc0c9800 2024-03-19T01:59:39.686 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.686+0000 7f46c97fa640 1 -- 172.21.15.55:0/2874673328 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f46c401c070 con 0x7f46cc0c9800 2024-03-19T01:59:39.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.686+0000 7f46c97fa640 1 --2- 172.21.15.55:0/2874673328 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f46b00676b0 0x7f46b0069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:39.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.686+0000 7f46d0f04640 1 --2- 172.21.15.55:0/2874673328 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f46b00676b0 0x7f46b0069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:39.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.690+0000 7f46d0f04640 1 --2- 172.21.15.55:0/2874673328 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f46b00676b0 0x7f46b0069b70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f46c0009b60 tx=0x7f46c0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:39.979 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.982+0000 7f46c97fa640 1 -- 172.21.15.55:0/2874673328 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f46c4053fe0 con 0x7f46cc0c9800 2024-03-19T01:59:39.991 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:39.994+0000 7f46c97fa640 1 -- 172.21.15.55:0/2874673328 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f46c4023020 con 0x7f46cc0c9800 2024-03-19T01:59:40.024 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:59:40.027 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T01:59:40.257 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:40.258+0000 7f46d3168640 1 -- 172.21.15.55:0/2874673328 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46cc13c800 con 0x7f46cc0c9800 2024-03-19T01:59:40.283 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:40.286+0000 7f46c97fa640 1 -- 172.21.15.55:0/2874673328 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f46c4055f40 con 0x7f46cc0c9800 2024-03-19T01:59:40.283 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:40.283 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:40.283 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:40.285 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:40.286+0000 7f46d3168640 1 -- 172.21.15.55:0/2874673328 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f46b00676b0 msgr2=0x7f46b0069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:40.285 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:40.286+0000 7f46d3168640 1 --2- 172.21.15.55:0/2874673328 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f46b00676b0 0x7f46b0069b70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f46c0009b60 tx=0x7f46c0009190 comp rx=0 tx=0).stop 2024-03-19T01:59:40.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:40.286+0000 7f46d3168640 1 -- 172.21.15.55:0/2874673328 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46cc0c9800 msgr2=0x7f46cc0c8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:40.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:40.286+0000 7f46d3168640 1 --2- 172.21.15.55:0/2874673328 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46cc0c9800 0x7f46cc0c8dc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f46c400b960 tx=0x7f46c400be20 comp rx=0 tx=0).stop 2024-03-19T01:59:40.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:40.286+0000 7f46d3168640 1 -- 172.21.15.55:0/2874673328 shutdown_connections 2024-03-19T01:59:40.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:40.286+0000 7f46d3168640 1 --2- 172.21.15.55:0/2874673328 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f46b00676b0 0x7f46b0069b70 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:40.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:40.286+0000 7f46d3168640 1 --2- 172.21.15.55:0/2874673328 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46cc0c9800 0x7f46cc0c8dc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:40.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:40.286+0000 7f46d3168640 1 --2- 172.21.15.55:0/2874673328 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46cc06c9f0 0x7f46cc141c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:40.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:40.286+0000 7f46d3168640 1 --2- 172.21.15.55:0/2874673328 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46cc06c610 0x7f46cc0c45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:40.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:40.286+0000 7f46d3168640 1 -- 172.21.15.55:0/2874673328 >> 172.21.15.55:0/2874673328 conn(0x7f46cc067600 msgr2=0x7f46cc13cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:40.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:40.286+0000 7f46d3168640 1 -- 172.21.15.55:0/2874673328 shutdown_connections 2024-03-19T01:59:40.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:40.286+0000 7f46d3168640 1 -- 172.21.15.55:0/2874673328 wait complete. 2024-03-19T01:59:40.907 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:40.907 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:41.908 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:42.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.022+0000 7fcf1de3f640 1 Processor -- start 2024-03-19T01:59:42.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.022+0000 7fcf1de3f640 1 -- start start 2024-03-19T01:59:42.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.022+0000 7fcf1de3f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf1806c630 0x7fcf18161f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:42.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.022+0000 7fcf1de3f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1806c230 con 0x7fcf18150ae0 2024-03-19T01:59:42.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.022+0000 7fcf1de3f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1806bc70 con 0x7fcf180c3520 2024-03-19T01:59:42.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.022+0000 7fcf1de3f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fcf180c3a40 con 0x7fcf1806c630 2024-03-19T01:59:42.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.026+0000 7fcf177fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fcf18150ae0 0x7fcf180c6e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:51546/0 (socket says 172.21.15.55:51546) 2024-03-19T01:59:42.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.026+0000 7fcf177fe640 1 -- 172.21.15.55:0/2018505930 learned_addr learned my addr 172.21.15.55:0/2018505930 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:59:42.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.026+0000 7fcf17fff640 1 --2- 172.21.15.55:0/2018505930 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf1806c630 0x7fcf18161f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:42.026 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.026+0000 7fcf167fc640 1 -- 172.21.15.55:0/2018505930 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4235625026 0 0) 0x7fcf08001120 con 0x7fcf18150ae0 2024-03-19T01:59:42.026 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.026+0000 7fcf167fc640 1 -- 172.21.15.55:0/2018505930 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf00003450 con 0x7fcf18150ae0 2024-03-19T01:59:42.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.030+0000 7fcf17fff640 1 -- 172.21.15.55:0/2018505930 >> v1:172.21.15.84:6789/0 conn(0x7fcf180c3520 legacy=0x7fcf180cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:42.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.030+0000 7fcf17fff640 1 -- 172.21.15.55:0/2018505930 >> v1:172.21.15.55:6789/0 conn(0x7fcf18150ae0 legacy=0x7fcf180c6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:42.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.030+0000 7fcf17fff640 1 -- 172.21.15.55:0/2018505930 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1806bf50 con 0x7fcf1806c630 2024-03-19T01:59:42.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.034+0000 7fcf17fff640 1 --2- 172.21.15.55:0/2018505930 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf1806c630 0x7fcf18161f40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcf10009dc0 tx=0x7fcf1001c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e55ed173529a21a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:42.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.042+0000 7fcf167fc640 1 -- 172.21.15.55:0/2018505930 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf100b2070 con 0x7fcf1806c630 2024-03-19T01:59:42.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.042+0000 7fcf167fc640 1 -- 172.21.15.55:0/2018505930 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf100a5740 con 0x7fcf1806c630 2024-03-19T01:59:42.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.042+0000 7fcf167fc640 1 -- 172.21.15.55:0/2018505930 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf100a5b10 con 0x7fcf1806c630 2024-03-19T01:59:42.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.042+0000 7fcf1de3f640 1 -- 172.21.15.55:0/2018505930 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf1806c630 msgr2=0x7fcf18161f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:42.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.042+0000 7fcf1de3f640 1 --2- 172.21.15.55:0/2018505930 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf1806c630 0x7fcf18161f40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcf10009dc0 tx=0x7fcf1001c900 comp rx=0 tx=0).stop 2024-03-19T01:59:42.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.042+0000 7fcf1de3f640 1 -- 172.21.15.55:0/2018505930 shutdown_connections 2024-03-19T01:59:42.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.042+0000 7fcf1de3f640 1 --2- 172.21.15.55:0/2018505930 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf1806c630 0x7fcf18161f40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:42.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.042+0000 7fcf1de3f640 1 -- 172.21.15.55:0/2018505930 >> 172.21.15.55:0/2018505930 conn(0x7fcf1806e5a0 msgr2=0x7fcf18072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:42.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.042+0000 7fcf1de3f640 1 -- 172.21.15.55:0/2018505930 shutdown_connections 2024-03-19T01:59:42.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.042+0000 7fcf1de3f640 1 -- 172.21.15.55:0/2018505930 wait complete. 2024-03-19T01:59:42.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.042+0000 7fcf1de3f640 1 Processor -- start 2024-03-19T01:59:42.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.042+0000 7fcf1de3f640 1 -- start start 2024-03-19T01:59:42.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.042+0000 7fcf1de3f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf1806c630 0x7fcf181f42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:42.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.042+0000 7fcf1de3f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcf180c3520 0x7fcf18142e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:42.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.042+0000 7fcf1de3f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf18150ae0 0x7fcf180cb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:42.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.042+0000 7fcf1de3f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcf180cc820 con 0x7fcf1806c630 2024-03-19T01:59:42.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.042+0000 7fcf1de3f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fcf180cc6a0 con 0x7fcf180c3520 2024-03-19T01:59:42.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.042+0000 7fcf1de3f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fcf180c8670 con 0x7fcf18150ae0 2024-03-19T01:59:42.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.046+0000 7fcf177fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf1806c630 0x7fcf181f42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:42.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.046+0000 7fcf17fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf18150ae0 0x7fcf180cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:42.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.046+0000 7fcf177fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf1806c630 0x7fcf181f42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55924/0 (socket says 172.21.15.55:55924) 2024-03-19T01:59:42.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.046+0000 7fcf177fe640 1 -- 172.21.15.55:0/2745722436 learned_addr learned my addr 172.21.15.55:0/2745722436 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:42.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.046+0000 7fcf16ffd640 1 --2- 172.21.15.55:0/2745722436 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcf180c3520 0x7fcf18142e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:42.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.050+0000 7fcf177fe640 1 -- 172.21.15.55:0/2745722436 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf18150ae0 msgr2=0x7fcf180cb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:42.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.050+0000 7fcf177fe640 1 --2- 172.21.15.55:0/2745722436 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf18150ae0 0x7fcf180cb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:42.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.050+0000 7fcf177fe640 1 -- 172.21.15.55:0/2745722436 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcf180c3520 msgr2=0x7fcf18142e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:42.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.050+0000 7fcf177fe640 1 --2- 172.21.15.55:0/2745722436 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcf180c3520 0x7fcf18142e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:42.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.050+0000 7fcf177fe640 1 -- 172.21.15.55:0/2745722436 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf181f7140 con 0x7fcf1806c630 2024-03-19T01:59:42.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.054+0000 7fcf177fe640 1 --2- 172.21.15.55:0/2745722436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf1806c630 0x7fcf181f42d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fcf0800c3f0 tx=0x7fcf0800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:42.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.058+0000 7fcefffff640 1 -- 172.21.15.55:0/2745722436 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf08017070 con 0x7fcf1806c630 2024-03-19T01:59:42.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.058+0000 7fcefffff640 1 -- 172.21.15.55:0/2745722436 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf08005cf0 con 0x7fcf1806c630 2024-03-19T01:59:42.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.058+0000 7fcefffff640 1 -- 172.21.15.55:0/2745722436 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf08007310 con 0x7fcf1806c630 2024-03-19T01:59:42.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.058+0000 7fcf1de3f640 1 -- 172.21.15.55:0/2745722436 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf181420a0 con 0x7fcf1806c630 2024-03-19T01:59:42.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.058+0000 7fcf1de3f640 1 -- 172.21.15.55:0/2745722436 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf18142640 con 0x7fcf1806c630 2024-03-19T01:59:42.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.058+0000 7fcf1de3f640 1 -- 172.21.15.55:0/2745722436 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf18068a40 con 0x7fcf1806c630 2024-03-19T01:59:42.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.090+0000 7fcefffff640 1 -- 172.21.15.55:0/2745722436 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fcf0801a070 con 0x7fcf1806c630 2024-03-19T01:59:42.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.090+0000 7fcefffff640 1 --2- 172.21.15.55:0/2745722436 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcf040676b0 0x7fcf04069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:42.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.090+0000 7fcf16ffd640 1 --2- 172.21.15.55:0/2745722436 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcf040676b0 0x7fcf04069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:42.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.090+0000 7fcf16ffd640 1 --2- 172.21.15.55:0/2745722436 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcf040676b0 0x7fcf04069b70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcf181478f0 tx=0x7fcf0c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:42.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.094+0000 7fcefffff640 1 -- 172.21.15.55:0/2745722436 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fcf08053770 con 0x7fcf1806c630 2024-03-19T01:59:42.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.374+0000 7fcefffff640 1 -- 172.21.15.55:0/2745722436 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fcf08021020 con 0x7fcf1806c630 2024-03-19T01:59:42.612 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.614+0000 7fcf1de3f640 1 -- 172.21.15.55:0/2745722436 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf180cc250 con 0x7fcf1806c630 2024-03-19T01:59:42.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.706+0000 7fcefffff640 1 -- 172.21.15.55:0/2745722436 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fcf1806c230 con 0x7fcf1806c630 2024-03-19T01:59:42.706 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:42.707 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:42.707 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:42.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.710+0000 7fcf1de3f640 1 -- 172.21.15.55:0/2745722436 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcf040676b0 msgr2=0x7fcf04069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:42.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.710+0000 7fcf1de3f640 1 --2- 172.21.15.55:0/2745722436 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcf040676b0 0x7fcf04069b70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcf181478f0 tx=0x7fcf0c009040 comp rx=0 tx=0).stop 2024-03-19T01:59:42.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.710+0000 7fcf1de3f640 1 -- 172.21.15.55:0/2745722436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf1806c630 msgr2=0x7fcf181f42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:42.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.710+0000 7fcf1de3f640 1 --2- 172.21.15.55:0/2745722436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf1806c630 0x7fcf181f42d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fcf0800c3f0 tx=0x7fcf0800c8b0 comp rx=0 tx=0).stop 2024-03-19T01:59:42.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.710+0000 7fcf1de3f640 1 -- 172.21.15.55:0/2745722436 shutdown_connections 2024-03-19T01:59:42.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.710+0000 7fcf1de3f640 1 --2- 172.21.15.55:0/2745722436 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcf040676b0 0x7fcf04069b70 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:42.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.710+0000 7fcf1de3f640 1 --2- 172.21.15.55:0/2745722436 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcf18150ae0 0x7fcf180cb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:42.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.710+0000 7fcf1de3f640 1 --2- 172.21.15.55:0/2745722436 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcf180c3520 0x7fcf18142e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:42.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.710+0000 7fcf1de3f640 1 --2- 172.21.15.55:0/2745722436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf1806c630 0x7fcf181f42d0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:42.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.710+0000 7fcf1de3f640 1 -- 172.21.15.55:0/2745722436 >> 172.21.15.55:0/2745722436 conn(0x7fcf1806e5a0 msgr2=0x7fcf180c8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:42.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.710+0000 7fcf1de3f640 1 -- 172.21.15.55:0/2745722436 shutdown_connections 2024-03-19T01:59:42.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:42.710+0000 7fcf1de3f640 1 -- 172.21.15.55:0/2745722436 wait complete. 2024-03-19T01:59:43.331 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:43.331 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:44.332 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:44.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.446+0000 7fa53ee0f640 1 Processor -- start 2024-03-19T01:59:44.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.446+0000 7fa53ee0f640 1 -- start start 2024-03-19T01:59:44.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.450+0000 7fa53ee0f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa538152120 0x7fa5380c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:44.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.450+0000 7fa53ee0f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa5380c9610 0x7fa538157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:44.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.450+0000 7fa53ee0f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa53806fff0 con 0x7fa538070780 2024-03-19T01:59:44.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.450+0000 7fa53ee0f640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fa538152500 con 0x7fa5380c9610 2024-03-19T01:59:44.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.450+0000 7fa53ee0f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa538070590 con 0x7fa538152120 2024-03-19T01:59:44.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.450+0000 7fa537fff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa5380c9610 0x7fa538157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:44.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.450+0000 7fa53cbab640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa538152120 0x7fa5380c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:44.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.450+0000 7fa537fff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa5380c9610 0x7fa538157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:59392/0 (socket says 172.21.15.55:59392) 2024-03-19T01:59:44.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.450+0000 7fa537fff640 1 -- 172.21.15.55:0/4160252331 learned_addr learned my addr 172.21.15.55:0/4160252331 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:44.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.454+0000 7fa537fff640 1 -- 172.21.15.55:0/4160252331 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa538152120 msgr2=0x7fa5380c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:44.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.454+0000 7fa537fff640 1 --2- 172.21.15.55:0/4160252331 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa538152120 0x7fa5380c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:44.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.454+0000 7fa537fff640 1 -- 172.21.15.55:0/4160252331 >> v1:172.21.15.84:6789/0 conn(0x7fa538070780 legacy=0x7fa538158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:44.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.454+0000 7fa537fff640 1 -- 172.21.15.55:0/4160252331 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5380c39c0 con 0x7fa5380c9610 2024-03-19T01:59:44.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.458+0000 7fa537fff640 1 --2- 172.21.15.55:0/4160252331 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5380c9610 0x7fa538157fd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa524009870 tx=0x7fa52401c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2027af031a07c426 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:44.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.466+0000 7fa5377fe640 1 -- 172.21.15.55:0/4160252331 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5240b0070 con 0x7fa5380c9610 2024-03-19T01:59:44.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.466+0000 7fa5377fe640 1 -- 172.21.15.55:0/4160252331 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5240ab600 con 0x7fa5380c9610 2024-03-19T01:59:44.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.466+0000 7fa5377fe640 1 -- 172.21.15.55:0/4160252331 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5240ab9d0 con 0x7fa5380c9610 2024-03-19T01:59:44.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.466+0000 7fa53ee0f640 1 -- 172.21.15.55:0/4160252331 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5380c9610 msgr2=0x7fa538157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:44.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.466+0000 7fa53ee0f640 1 --2- 172.21.15.55:0/4160252331 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5380c9610 0x7fa538157fd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa524009870 tx=0x7fa52401c520 comp rx=0 tx=0).stop 2024-03-19T01:59:44.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.466+0000 7fa53ee0f640 1 -- 172.21.15.55:0/4160252331 shutdown_connections 2024-03-19T01:59:44.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.466+0000 7fa53ee0f640 1 --2- 172.21.15.55:0/4160252331 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5380c9610 0x7fa538157fd0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:44.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.466+0000 7fa53ee0f640 1 --2- 172.21.15.55:0/4160252331 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa538152120 0x7fa5380c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:44.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.466+0000 7fa53ee0f640 1 -- 172.21.15.55:0/4160252331 >> 172.21.15.55:0/4160252331 conn(0x7fa538067600 msgr2=0x7fa538071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:44.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.466+0000 7fa53ee0f640 1 -- 172.21.15.55:0/4160252331 shutdown_connections 2024-03-19T01:59:44.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.466+0000 7fa53ee0f640 1 -- 172.21.15.55:0/4160252331 wait complete. 2024-03-19T01:59:44.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.470+0000 7fa53ee0f640 1 Processor -- start 2024-03-19T01:59:44.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.470+0000 7fa53ee0f640 1 -- start start 2024-03-19T01:59:44.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.470+0000 7fa53ee0f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa538070780 0x7fa53806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:44.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.470+0000 7fa53ee0f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5380c9610 0x7fa53813d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:44.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.470+0000 7fa53ee0f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa538152120 0x7fa538141420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:44.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.470+0000 7fa53ee0f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa5380c86d0 con 0x7fa538070780 2024-03-19T01:59:44.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.470+0000 7fa53ee0f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa5380c31c0 con 0x7fa538152120 2024-03-19T01:59:44.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.470+0000 7fa53ee0f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa538165170 con 0x7fa5380c9610 2024-03-19T01:59:44.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.470+0000 7fa53cbab640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa538070780 0x7fa53806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:44.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.470+0000 7fa53cbab640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa538070780 0x7fa53806d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55934/0 (socket says 172.21.15.55:55934) 2024-03-19T01:59:44.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.470+0000 7fa53cbab640 1 -- 172.21.15.55:0/3578211300 learned_addr learned my addr 172.21.15.55:0/3578211300 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:44.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.470+0000 7fa53d3ac640 1 --2- 172.21.15.55:0/3578211300 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa538152120 0x7fa538141420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:44.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.470+0000 7fa537fff640 1 --2- 172.21.15.55:0/3578211300 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5380c9610 0x7fa53813d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:44.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.474+0000 7fa53cbab640 1 -- 172.21.15.55:0/3578211300 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5380c9610 msgr2=0x7fa53813d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:44.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.474+0000 7fa53cbab640 1 --2- 172.21.15.55:0/3578211300 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5380c9610 0x7fa53813d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:44.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.474+0000 7fa53cbab640 1 -- 172.21.15.55:0/3578211300 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa538152120 msgr2=0x7fa538141420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:44.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.474+0000 7fa53cbab640 1 --2- 172.21.15.55:0/3578211300 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa538152120 0x7fa538141420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:44.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.474+0000 7fa53cbab640 1 -- 172.21.15.55:0/3578211300 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5381f4040 con 0x7fa538070780 2024-03-19T01:59:44.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.478+0000 7fa53cbab640 1 --2- 172.21.15.55:0/3578211300 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa538070780 0x7fa53806d9c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa52c00e8a0 tx=0x7fa52c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:44.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.482+0000 7fa5357fa640 1 -- 172.21.15.55:0/3578211300 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c017070 con 0x7fa538070780 2024-03-19T01:59:44.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.482+0000 7fa5357fa640 1 -- 172.21.15.55:0/3578211300 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa52c00c940 con 0x7fa538070780 2024-03-19T01:59:44.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.482+0000 7fa53ee0f640 1 -- 172.21.15.55:0/3578211300 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa538142e50 con 0x7fa538070780 2024-03-19T01:59:44.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.482+0000 7fa5357fa640 1 -- 172.21.15.55:0/3578211300 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c00ccf0 con 0x7fa538070780 2024-03-19T01:59:44.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.482+0000 7fa53ee0f640 1 -- 172.21.15.55:0/3578211300 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa53806c550 con 0x7fa538070780 2024-03-19T01:59:44.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.482+0000 7fa53ee0f640 1 -- 172.21.15.55:0/3578211300 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5380c84b0 con 0x7fa538070780 2024-03-19T01:59:44.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.790+0000 7fa5357fa640 1 -- 172.21.15.55:0/3578211300 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa52c01a070 con 0x7fa538070780 2024-03-19T01:59:44.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.794+0000 7fa5357fa640 1 --2- 172.21.15.55:0/3578211300 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa51c0676b0 0x7fa51c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:44.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.794+0000 7fa537fff640 1 --2- 172.21.15.55:0/3578211300 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa51c0676b0 0x7fa51c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:44.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.794+0000 7fa537fff640 1 --2- 172.21.15.55:0/3578211300 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa51c0676b0 0x7fa51c069b70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa5240ad000 tx=0x7fa5240a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:44.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.814+0000 7fa5357fa640 1 -- 172.21.15.55:0/3578211300 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fa52c054590 con 0x7fa538070780 2024-03-19T01:59:44.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:44.814+0000 7fa5357fa640 1 -- 172.21.15.55:0/3578211300 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa52c0564f0 con 0x7fa538070780 2024-03-19T01:59:45.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:45.030+0000 7fa53ee0f640 1 -- 172.21.15.55:0/3578211300 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa538070db0 con 0x7fa538070780 2024-03-19T01:59:45.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:45.042+0000 7fa5357fa640 1 -- 172.21.15.55:0/3578211300 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fa52c05a590 con 0x7fa538070780 2024-03-19T01:59:45.042 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:45.042 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:45.042 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:45.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:45.046+0000 7fa53ee0f640 1 -- 172.21.15.55:0/3578211300 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa51c0676b0 msgr2=0x7fa51c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:45.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:45.046+0000 7fa53ee0f640 1 --2- 172.21.15.55:0/3578211300 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa51c0676b0 0x7fa51c069b70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa5240ad000 tx=0x7fa5240a6000 comp rx=0 tx=0).stop 2024-03-19T01:59:45.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:45.046+0000 7fa53ee0f640 1 -- 172.21.15.55:0/3578211300 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa538070780 msgr2=0x7fa53806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:45.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:45.046+0000 7fa53ee0f640 1 --2- 172.21.15.55:0/3578211300 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa538070780 0x7fa53806d9c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa52c00e8a0 tx=0x7fa52c00eca0 comp rx=0 tx=0).stop 2024-03-19T01:59:45.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:45.046+0000 7fa53ee0f640 1 -- 172.21.15.55:0/3578211300 shutdown_connections 2024-03-19T01:59:45.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:45.046+0000 7fa53ee0f640 1 --2- 172.21.15.55:0/3578211300 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa51c0676b0 0x7fa51c069b70 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:45.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:45.046+0000 7fa53ee0f640 1 --2- 172.21.15.55:0/3578211300 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa538152120 0x7fa538141420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:45.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:45.046+0000 7fa53ee0f640 1 --2- 172.21.15.55:0/3578211300 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5380c9610 0x7fa53813d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:45.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:45.046+0000 7fa53ee0f640 1 --2- 172.21.15.55:0/3578211300 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa538070780 0x7fa53806d9c0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:45.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:45.046+0000 7fa53ee0f640 1 -- 172.21.15.55:0/3578211300 >> 172.21.15.55:0/3578211300 conn(0x7fa538067600 msgr2=0x7fa5380c8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:45.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:45.046+0000 7fa53ee0f640 1 -- 172.21.15.55:0/3578211300 shutdown_connections 2024-03-19T01:59:45.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:45.046+0000 7fa53ee0f640 1 -- 172.21.15.55:0/3578211300 wait complete. 2024-03-19T01:59:45.664 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:45.664 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:46.665 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:46.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.782+0000 7ffaa535a640 1 Processor -- start 2024-03-19T01:59:46.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.782+0000 7ffaa535a640 1 -- start start 2024-03-19T01:59:46.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.782+0000 7ffaa535a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffaa00c9000 0x7ffaa00c9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:46.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.782+0000 7ffaa535a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffaa0158d50 con 0x7ffaa01520b0 2024-03-19T01:59:46.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.782+0000 7ffaa535a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffaa015b720 con 0x7ffaa00c2ad0 2024-03-19T01:59:46.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.782+0000 7ffaa535a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ffaa015c4e0 con 0x7ffaa00c9000 2024-03-19T01:59:46.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.782+0000 7ffa9effd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7ffaa01520b0 0x7ffaa00709a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:51558/0 (socket says 172.21.15.55:51558) 2024-03-19T01:59:46.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.782+0000 7ffa9effd640 1 -- 172.21.15.55:0/227156578 learned_addr learned my addr 172.21.15.55:0/227156578 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:59:46.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.782+0000 7ffa9e7fc640 1 --2- 172.21.15.55:0/227156578 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffaa00c9000 0x7ffaa00c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:46.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.786+0000 7ffa9dffb640 1 -- 172.21.15.55:0/227156578 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 286316607 0 0) 0x7ffaa0158d50 con 0x7ffaa01520b0 2024-03-19T01:59:46.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.786+0000 7ffa9dffb640 1 -- 172.21.15.55:0/227156578 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa88003530 con 0x7ffaa01520b0 2024-03-19T01:59:46.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.786+0000 7ffa9e7fc640 1 -- 172.21.15.55:0/227156578 >> v1:172.21.15.84:6789/0 conn(0x7ffaa00c2ad0 legacy=0x7ffaa00c2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:46.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.786+0000 7ffa9e7fc640 1 -- 172.21.15.55:0/227156578 >> v1:172.21.15.55:6789/0 conn(0x7ffaa01520b0 legacy=0x7ffaa00709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:46.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.786+0000 7ffa9e7fc640 1 -- 172.21.15.55:0/227156578 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaa015ba00 con 0x7ffaa00c9000 2024-03-19T01:59:46.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.794+0000 7ffa9e7fc640 1 --2- 172.21.15.55:0/227156578 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffaa00c9000 0x7ffaa00c9420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ffa940097f0 tx=0x7ffa9401ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66fc03653a975c13 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:46.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.798+0000 7ffa9dffb640 1 -- 172.21.15.55:0/227156578 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa940b5070 con 0x7ffaa00c9000 2024-03-19T01:59:46.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.798+0000 7ffa9dffb640 1 -- 172.21.15.55:0/227156578 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa940a89d0 con 0x7ffaa00c9000 2024-03-19T01:59:46.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.798+0000 7ffa9dffb640 1 -- 172.21.15.55:0/227156578 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa940a8da0 con 0x7ffaa00c9000 2024-03-19T01:59:46.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.798+0000 7ffaa535a640 1 -- 172.21.15.55:0/227156578 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffaa00c9000 msgr2=0x7ffaa00c9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:46.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.798+0000 7ffaa535a640 1 --2- 172.21.15.55:0/227156578 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffaa00c9000 0x7ffaa00c9420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ffa940097f0 tx=0x7ffa9401ecd0 comp rx=0 tx=0).stop 2024-03-19T01:59:46.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.798+0000 7ffaa535a640 1 -- 172.21.15.55:0/227156578 shutdown_connections 2024-03-19T01:59:46.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.798+0000 7ffaa535a640 1 --2- 172.21.15.55:0/227156578 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffaa00c9000 0x7ffaa00c9420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:46.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.798+0000 7ffaa535a640 1 -- 172.21.15.55:0/227156578 >> 172.21.15.55:0/227156578 conn(0x7ffaa0067600 msgr2=0x7ffaa0071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:46.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.798+0000 7ffaa535a640 1 -- 172.21.15.55:0/227156578 shutdown_connections 2024-03-19T01:59:46.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.798+0000 7ffaa535a640 1 -- 172.21.15.55:0/227156578 wait complete. 2024-03-19T01:59:46.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.798+0000 7ffaa535a640 1 Processor -- start 2024-03-19T01:59:46.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.798+0000 7ffaa535a640 1 -- start start 2024-03-19T01:59:46.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.802+0000 7ffaa535a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffaa00c2ad0 0x7ffaa0151520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:46.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.802+0000 7ffaa535a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffaa00c9000 0x7ffaa01ed6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:46.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.802+0000 7ffaa535a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffaa01520b0 0x7ffaa0153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:46.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.802+0000 7ffaa535a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa00710b0 con 0x7ffaa00c2ad0 2024-03-19T01:59:46.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.802+0000 7ffaa535a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa015c050 con 0x7ffaa00c9000 2024-03-19T01:59:46.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.802+0000 7ffaa535a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ffaa01589a0 con 0x7ffaa01520b0 2024-03-19T01:59:46.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.802+0000 7ffa9effd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffaa00c2ad0 0x7ffaa0151520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:46.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.802+0000 7ffa9f7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffaa01520b0 0x7ffaa0153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:46.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.802+0000 7ffa9e7fc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffaa00c9000 0x7ffaa01ed6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:46.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.802+0000 7ffa9e7fc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffaa00c9000 0x7ffaa01ed6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:59412/0 (socket says 172.21.15.55:59412) 2024-03-19T01:59:46.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.802+0000 7ffa9e7fc640 1 -- 172.21.15.55:0/415300934 learned_addr learned my addr 172.21.15.55:0/415300934 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:46.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.806+0000 7ffa9effd640 1 -- 172.21.15.55:0/415300934 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffaa01520b0 msgr2=0x7ffaa0153260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:46.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.806+0000 7ffa9effd640 1 --2- 172.21.15.55:0/415300934 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffaa01520b0 0x7ffaa0153260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:46.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.806+0000 7ffa9effd640 1 -- 172.21.15.55:0/415300934 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffaa00c9000 msgr2=0x7ffaa01ed6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:46.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.806+0000 7ffa9effd640 1 --2- 172.21.15.55:0/415300934 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffaa00c9000 0x7ffaa01ed6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:46.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.806+0000 7ffa9effd640 1 -- 172.21.15.55:0/415300934 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaa014f8d0 con 0x7ffaa00c2ad0 2024-03-19T01:59:46.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.806+0000 7ffa9effd640 1 --2- 172.21.15.55:0/415300934 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffaa00c2ad0 0x7ffaa0151520 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ffa9000c390 tx=0x7ffa90001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:46.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.814+0000 7ffa877fe640 1 -- 172.21.15.55:0/415300934 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa90017070 con 0x7ffaa00c2ad0 2024-03-19T01:59:46.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.814+0000 7ffa877fe640 1 -- 172.21.15.55:0/415300934 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa90005cf0 con 0x7ffaa00c2ad0 2024-03-19T01:59:46.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.814+0000 7ffaa535a640 1 -- 172.21.15.55:0/415300934 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaa014fb90 con 0x7ffaa00c2ad0 2024-03-19T01:59:46.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.814+0000 7ffa877fe640 1 -- 172.21.15.55:0/415300934 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa900072f0 con 0x7ffaa00c2ad0 2024-03-19T01:59:46.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.814+0000 7ffaa535a640 1 -- 172.21.15.55:0/415300934 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaa0150d20 con 0x7ffaa00c2ad0 2024-03-19T01:59:46.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:46.814+0000 7ffaa535a640 1 -- 172.21.15.55:0/415300934 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaa0150090 con 0x7ffaa00c2ad0 2024-03-19T01:59:47.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:47.090+0000 7ffa877fe640 1 -- 172.21.15.55:0/415300934 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ffa9001a070 con 0x7ffaa00c2ad0 2024-03-19T01:59:47.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:47.094+0000 7ffa877fe640 1 --2- 172.21.15.55:0/415300934 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffa8c067660 0x7ffa8c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:47.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:47.094+0000 7ffa9e7fc640 1 --2- 172.21.15.55:0/415300934 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffa8c067660 0x7ffa8c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:47.092 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:47.094+0000 7ffa9e7fc640 1 --2- 172.21.15.55:0/415300934 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffa8c067660 0x7ffa8c069b20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ffa94009740 tx=0x7ffa940aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:47.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:47.114+0000 7ffa877fe640 1 -- 172.21.15.55:0/415300934 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7ffa90053620 con 0x7ffaa00c2ad0 2024-03-19T01:59:47.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:47.118+0000 7ffa877fe640 1 -- 172.21.15.55:0/415300934 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ffa90021020 con 0x7ffaa00c2ad0 2024-03-19T01:59:47.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:47.338+0000 7ffaa535a640 1 -- 172.21.15.55:0/415300934 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffaa0068a40 con 0x7ffaa00c2ad0 2024-03-19T01:59:47.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:47.362+0000 7ffa877fe640 1 -- 172.21.15.55:0/415300934 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7ffa90055580 con 0x7ffaa00c2ad0 2024-03-19T01:59:47.361 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:47.362 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:47.362 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:47.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:47.366+0000 7ffaa535a640 1 -- 172.21.15.55:0/415300934 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffa8c067660 msgr2=0x7ffa8c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:47.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:47.366+0000 7ffaa535a640 1 --2- 172.21.15.55:0/415300934 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffa8c067660 0x7ffa8c069b20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ffa94009740 tx=0x7ffa940aa070 comp rx=0 tx=0).stop 2024-03-19T01:59:47.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:47.366+0000 7ffaa535a640 1 -- 172.21.15.55:0/415300934 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffaa00c2ad0 msgr2=0x7ffaa0151520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:47.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:47.366+0000 7ffaa535a640 1 --2- 172.21.15.55:0/415300934 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffaa00c2ad0 0x7ffaa0151520 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ffa9000c390 tx=0x7ffa90001e80 comp rx=0 tx=0).stop 2024-03-19T01:59:47.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:47.366+0000 7ffaa535a640 1 -- 172.21.15.55:0/415300934 shutdown_connections 2024-03-19T01:59:47.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:47.366+0000 7ffaa535a640 1 --2- 172.21.15.55:0/415300934 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffa8c067660 0x7ffa8c069b20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:47.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:47.366+0000 7ffaa535a640 1 --2- 172.21.15.55:0/415300934 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffaa01520b0 0x7ffaa0153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:47.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:47.366+0000 7ffaa535a640 1 --2- 172.21.15.55:0/415300934 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffaa00c9000 0x7ffaa01ed6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:47.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:47.366+0000 7ffaa535a640 1 --2- 172.21.15.55:0/415300934 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffaa00c2ad0 0x7ffaa0151520 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:47.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:47.366+0000 7ffaa535a640 1 -- 172.21.15.55:0/415300934 >> 172.21.15.55:0/415300934 conn(0x7ffaa0067600 msgr2=0x7ffaa00b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:47.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:47.366+0000 7ffaa535a640 1 -- 172.21.15.55:0/415300934 shutdown_connections 2024-03-19T01:59:47.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:47.366+0000 7ffaa535a640 1 -- 172.21.15.55:0/415300934 wait complete. 2024-03-19T01:59:47.984 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:47.984 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:48.986 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:49.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.118+0000 7f416fc85640 1 Processor -- start 2024-03-19T01:59:49.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.118+0000 7f416fc85640 1 -- start start 2024-03-19T01:59:49.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.118+0000 7f416fc85640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f41681544d0 0x7f4168070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:49.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.118+0000 7f416fc85640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f41680c73c0 0x7f41680c77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:49.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.118+0000 7f416fc85640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f416815a410 con 0x7f41680c9450 2024-03-19T01:59:49.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.118+0000 7f416fc85640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f41680c4a90 con 0x7f41680c73c0 2024-03-19T01:59:49.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.118+0000 7f416fc85640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f416815a6f0 con 0x7f41681544d0 2024-03-19T01:59:49.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.118+0000 7f416e222640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f41680c9450 0x7f416815d870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:33472/0 (socket says 172.21.15.55:33472) 2024-03-19T01:59:49.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.118+0000 7f416e222640 1 -- 172.21.15.55:0/828654818 learned_addr learned my addr 172.21.15.55:0/828654818 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T01:59:49.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.118+0000 7f416da21640 1 --2- 172.21.15.55:0/828654818 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f41681544d0 0x7f4168070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:49.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.118+0000 7f416d220640 1 --2- 172.21.15.55:0/828654818 >> v2:172.21.15.84:3300/0 conn(0x7f41680c73c0 0x7f41680c77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:49.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.122+0000 7f416ca1f640 1 -- 172.21.15.55:0/828654818 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1607469059 0 0) 0x7f416815a410 con 0x7f41680c9450 2024-03-19T01:59:49.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.122+0000 7f416ca1f640 1 -- 172.21.15.55:0/828654818 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4154003530 con 0x7f41680c9450 2024-03-19T01:59:49.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.122+0000 7f416d220640 1 -- 172.21.15.55:0/828654818 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f41681544d0 msgr2=0x7f4168070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:49.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.122+0000 7f416d220640 1 --2- 172.21.15.55:0/828654818 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f41681544d0 0x7f4168070ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:49.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.122+0000 7f416d220640 1 -- 172.21.15.55:0/828654818 >> v1:172.21.15.55:6789/0 conn(0x7f41680c9450 legacy=0x7f416815d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:49.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.122+0000 7f416d220640 1 -- 172.21.15.55:0/828654818 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f416815a9d0 con 0x7f41680c73c0 2024-03-19T01:59:49.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.126+0000 7f416d220640 1 --2- 172.21.15.55:0/828654818 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41680c73c0 0x7f41680c77c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f41580097c0 tx=0x7f415801c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=709afe74dd052892 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:49.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.158+0000 7f416ca1f640 1 -- 172.21.15.55:0/828654818 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41580b0070 con 0x7f41680c73c0 2024-03-19T01:59:49.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.158+0000 7f416ca1f640 1 -- 172.21.15.55:0/828654818 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41580ab600 con 0x7f41680c73c0 2024-03-19T01:59:49.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.158+0000 7f416ca1f640 1 -- 172.21.15.55:0/828654818 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41580ab9d0 con 0x7f41680c73c0 2024-03-19T01:59:49.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.158+0000 7f416fc85640 1 -- 172.21.15.55:0/828654818 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41680c73c0 msgr2=0x7f41680c77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:49.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.158+0000 7f416fc85640 1 --2- 172.21.15.55:0/828654818 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41680c73c0 0x7f41680c77c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f41580097c0 tx=0x7f415801c410 comp rx=0 tx=0).stop 2024-03-19T01:59:49.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.158+0000 7f416fc85640 1 -- 172.21.15.55:0/828654818 shutdown_connections 2024-03-19T01:59:49.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.158+0000 7f416fc85640 1 --2- 172.21.15.55:0/828654818 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41680c73c0 0x7f41680c77c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:49.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.158+0000 7f416fc85640 1 --2- 172.21.15.55:0/828654818 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f41681544d0 0x7f4168070ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:49.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.158+0000 7f416fc85640 1 -- 172.21.15.55:0/828654818 >> 172.21.15.55:0/828654818 conn(0x7f41680cc9c0 msgr2=0x7f41680ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:49.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.158+0000 7f416fc85640 1 -- 172.21.15.55:0/828654818 shutdown_connections 2024-03-19T01:59:49.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.158+0000 7f416fc85640 1 -- 172.21.15.55:0/828654818 wait complete. 2024-03-19T01:59:49.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.162+0000 7f416fc85640 1 Processor -- start 2024-03-19T01:59:49.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.162+0000 7f416fc85640 1 -- start start 2024-03-19T01:59:49.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.162+0000 7f416fc85640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41680c73c0 0x7f416815bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:49.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.162+0000 7f416fc85640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f41680c9450 0x7f416815ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:49.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.162+0000 7f416fc85640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f41681544d0 0x7f4168151360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:49.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.162+0000 7f416fc85640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4168167480 con 0x7f41680c9450 2024-03-19T01:59:49.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.162+0000 7f416fc85640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4168167300 con 0x7f41680c73c0 2024-03-19T01:59:49.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.162+0000 7f416fc85640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4168167600 con 0x7f41681544d0 2024-03-19T01:59:49.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.162+0000 7f416d220640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f41680c9450 0x7f416815ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:49.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.162+0000 7f416e222640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f41681544d0 0x7f4168151360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:49.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.162+0000 7f416d220640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f41680c9450 0x7f416815ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48086/0 (socket says 172.21.15.55:48086) 2024-03-19T01:59:49.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.162+0000 7f416d220640 1 -- 172.21.15.55:0/2024881062 learned_addr learned my addr 172.21.15.55:0/2024881062 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:49.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.166+0000 7f416d220640 1 -- 172.21.15.55:0/2024881062 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f41681544d0 msgr2=0x7f4168151360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:49.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.166+0000 7f416d220640 1 --2- 172.21.15.55:0/2024881062 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f41681544d0 0x7f4168151360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:49.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.166+0000 7f416d220640 1 -- 172.21.15.55:0/2024881062 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41680c73c0 msgr2=0x7f416815bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:49.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.166+0000 7f416d220640 1 --2- 172.21.15.55:0/2024881062 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41680c73c0 0x7f416815bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:49.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.166+0000 7f416d220640 1 -- 172.21.15.55:0/2024881062 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f416815d860 con 0x7f41680c9450 2024-03-19T01:59:49.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.170+0000 7f416d220640 1 --2- 172.21.15.55:0/2024881062 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f41680c9450 0x7f416815ce50 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4158019020 tx=0x7f41580191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:49.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.214+0000 7f415e7fc640 1 -- 172.21.15.55:0/2024881062 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41580b9070 con 0x7f41680c9450 2024-03-19T01:59:49.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.214+0000 7f415e7fc640 1 -- 172.21.15.55:0/2024881062 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41580b5480 con 0x7f41680c9450 2024-03-19T01:59:49.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.214+0000 7f415e7fc640 1 -- 172.21.15.55:0/2024881062 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41580b0040 con 0x7f41680c9450 2024-03-19T01:59:49.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.214+0000 7f416fc85640 1 -- 172.21.15.55:0/2024881062 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f416815db40 con 0x7f41680c9450 2024-03-19T01:59:49.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.214+0000 7f416fc85640 1 -- 172.21.15.55:0/2024881062 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41681578e0 con 0x7f41680c9450 2024-03-19T01:59:49.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.218+0000 7f416fc85640 1 -- 172.21.15.55:0/2024881062 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4168068a40 con 0x7f41680c9450 2024-03-19T01:59:49.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.266+0000 7f415e7fc640 1 -- 172.21.15.55:0/2024881062 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f41580bc070 con 0x7f41680c9450 2024-03-19T01:59:49.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.266+0000 7f415e7fc640 1 --2- 172.21.15.55:0/2024881062 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f414c0676e0 0x7f414c069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:49.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.266+0000 7f416da21640 1 --2- 172.21.15.55:0/2024881062 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f414c0676e0 0x7f414c069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:49.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.266+0000 7f416da21640 1 --2- 172.21.15.55:0/2024881062 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f414c0676e0 0x7f414c069ba0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4164006aa0 tx=0x7f4164006a30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:49.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.274+0000 7f415e7fc640 1 -- 172.21.15.55:0/2024881062 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f41580ee2d0 con 0x7f41680c9450 2024-03-19T01:59:49.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.558+0000 7f415e7fc640 1 -- 172.21.15.55:0/2024881062 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4158016090 con 0x7f41680c9450 2024-03-19T01:59:49.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.838+0000 7f416fc85640 1 -- 172.21.15.55:0/2024881062 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41680c7840 con 0x7f41680c9450 2024-03-19T01:59:49.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.882+0000 7f415e7fc640 1 -- 172.21.15.55:0/2024881062 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f41580f0230 con 0x7f41680c9450 2024-03-19T01:59:49.880 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:49.880 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:49.880 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:49.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.882+0000 7f416fc85640 1 -- 172.21.15.55:0/2024881062 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f414c0676e0 msgr2=0x7f414c069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:49.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.882+0000 7f416fc85640 1 --2- 172.21.15.55:0/2024881062 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f414c0676e0 0x7f414c069ba0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4164006aa0 tx=0x7f4164006a30 comp rx=0 tx=0).stop 2024-03-19T01:59:49.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.882+0000 7f416fc85640 1 -- 172.21.15.55:0/2024881062 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f41680c9450 msgr2=0x7f416815ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:49.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.882+0000 7f416fc85640 1 --2- 172.21.15.55:0/2024881062 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f41680c9450 0x7f416815ce50 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4158019020 tx=0x7f41580191b0 comp rx=0 tx=0).stop 2024-03-19T01:59:49.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.882+0000 7f416fc85640 1 -- 172.21.15.55:0/2024881062 shutdown_connections 2024-03-19T01:59:49.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.882+0000 7f416fc85640 1 --2- 172.21.15.55:0/2024881062 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f414c0676e0 0x7f414c069ba0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:49.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.882+0000 7f416fc85640 1 --2- 172.21.15.55:0/2024881062 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f41681544d0 0x7f4168151360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:49.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.882+0000 7f416fc85640 1 --2- 172.21.15.55:0/2024881062 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f41680c9450 0x7f416815ce50 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:49.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.882+0000 7f416fc85640 1 --2- 172.21.15.55:0/2024881062 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41680c73c0 0x7f416815bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:49.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.882+0000 7f416fc85640 1 -- 172.21.15.55:0/2024881062 >> 172.21.15.55:0/2024881062 conn(0x7f41680cc9c0 msgr2=0x7f4168158c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:49.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.882+0000 7f416fc85640 1 -- 172.21.15.55:0/2024881062 shutdown_connections 2024-03-19T01:59:49.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:49.882+0000 7f416fc85640 1 -- 172.21.15.55:0/2024881062 wait complete. 2024-03-19T01:59:50.509 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:50.509 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:51.229 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:10:03.389 27555== Warning: unimplemented fcntl command: 1036 2024-03-19T01:59:51.238 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:10:03.398 27555== Warning: unimplemented fcntl command: 1036 2024-03-19T01:59:51.264 INFO:tasks.ceph.mon.b.smithi084.stderr:==00:00:10:03.361 27626== Warning: unimplemented fcntl command: 1036 2024-03-19T01:59:51.269 INFO:tasks.ceph.mon.b.smithi084.stderr:==00:00:10:03.366 27626== Warning: unimplemented fcntl command: 1036 2024-03-19T01:59:51.273 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:10:03.370 27627== Warning: unimplemented fcntl command: 1036 2024-03-19T01:59:51.277 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:10:03.374 27627== Warning: unimplemented fcntl command: 1036 2024-03-19T01:59:51.332 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:10:03.492 27555== Warning: unimplemented fcntl command: 1036 2024-03-19T01:59:51.350 INFO:tasks.ceph.mon.b.smithi084.stderr:==00:00:10:03.447 27626== Warning: unimplemented fcntl command: 1036 2024-03-19T01:59:51.365 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:10:03.462 27627== Warning: unimplemented fcntl command: 1036 2024-03-19T01:59:51.510 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:51.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.629+0000 7ff968da8640 1 Processor -- start 2024-03-19T01:59:51.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.629+0000 7ff968da8640 1 -- start start 2024-03-19T01:59:51.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.629+0000 7ff968da8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff9640c52b0 0x7ff964151d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:51.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.629+0000 7ff968da8640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff9640c89f0 0x7ff964152260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:51.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.629+0000 7ff968da8640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff9640c5690 0x7ff964158070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:51.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.629+0000 7ff968da8640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7ff9640724c0 con 0x7ff9640c52b0 2024-03-19T01:59:51.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.629+0000 7ff968da8640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7ff96413d290 con 0x7ff9640c5690 2024-03-19T01:59:51.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.629+0000 7ff968da8640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff96413d570 con 0x7ff9640c89f0 2024-03-19T01:59:51.636 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.637+0000 7ff962d9d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff9640c5690 0x7ff964158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:51.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.637+0000 7ff962d9d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff9640c5690 0x7ff964158070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:44906/0 (socket says 172.21.15.55:44906) 2024-03-19T01:59:51.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.637+0000 7ff962d9d640 1 -- 172.21.15.55:0/2692475612 learned_addr learned my addr 172.21.15.55:0/2692475612 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:51.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.637+0000 7ff961d9b640 1 --2- 172.21.15.55:0/2692475612 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff9640c89f0 0x7ff964152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:51.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.645+0000 7ff962d9d640 1 -- 172.21.15.55:0/2692475612 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff9640c89f0 msgr2=0x7ff964152260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:51.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.645+0000 7ff962d9d640 1 --2- 172.21.15.55:0/2692475612 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff9640c89f0 0x7ff964152260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:51.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.645+0000 7ff962d9d640 1 -- 172.21.15.55:0/2692475612 >> v2:172.21.15.55:3300/0 conn(0x7ff9640c52b0 msgr2=0x7ff964151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:51.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.645+0000 7ff962d9d640 1 --2- 172.21.15.55:0/2692475612 >> v2:172.21.15.55:3300/0 conn(0x7ff9640c52b0 0x7ff964151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:51.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.645+0000 7ff962d9d640 1 -- 172.21.15.55:0/2692475612 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff96413d850 con 0x7ff9640c5690 2024-03-19T01:59:51.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.657+0000 7ff962d9d640 1 --2- 172.21.15.55:0/2692475612 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9640c5690 0x7ff964158070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff9640b7fc0 tx=0x7ff95c01f1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7739dd3b92f1283d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:51.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.665+0000 7ff96159a640 1 -- 172.21.15.55:0/2692475612 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff95c0b4070 con 0x7ff9640c5690 2024-03-19T01:59:51.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.665+0000 7ff96159a640 1 -- 172.21.15.55:0/2692475612 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff95c0af400 con 0x7ff9640c5690 2024-03-19T01:59:51.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.665+0000 7ff96159a640 1 -- 172.21.15.55:0/2692475612 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff95c0af7d0 con 0x7ff9640c5690 2024-03-19T01:59:51.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.665+0000 7ff968da8640 1 -- 172.21.15.55:0/2692475612 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9640c5690 msgr2=0x7ff964158070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:51.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.665+0000 7ff968da8640 1 --2- 172.21.15.55:0/2692475612 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9640c5690 0x7ff964158070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff9640b7fc0 tx=0x7ff95c01f1e0 comp rx=0 tx=0).stop 2024-03-19T01:59:51.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.665+0000 7ff968da8640 1 -- 172.21.15.55:0/2692475612 shutdown_connections 2024-03-19T01:59:51.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.665+0000 7ff968da8640 1 --2- 172.21.15.55:0/2692475612 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff9640c89f0 0x7ff964152260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:51.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.665+0000 7ff968da8640 1 --2- 172.21.15.55:0/2692475612 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9640c5690 0x7ff964158070 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:51.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.665+0000 7ff968da8640 1 --2- 172.21.15.55:0/2692475612 >> v2:172.21.15.55:3300/0 conn(0x7ff9640c52b0 0x7ff964151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:51.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.665+0000 7ff968da8640 1 -- 172.21.15.55:0/2692475612 >> 172.21.15.55:0/2692475612 conn(0x7ff96406f3b0 msgr2=0x7ff96406f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:51.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.665+0000 7ff968da8640 1 -- 172.21.15.55:0/2692475612 shutdown_connections 2024-03-19T01:59:51.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.665+0000 7ff968da8640 1 -- 172.21.15.55:0/2692475612 wait complete. 2024-03-19T01:59:51.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.669+0000 7ff968da8640 1 Processor -- start 2024-03-19T01:59:51.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.669+0000 7ff968da8640 1 -- start start 2024-03-19T01:59:51.668 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.669+0000 7ff968da8640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9640c52b0 0x7ff9641ed940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:51.668 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.669+0000 7ff968da8640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff9640c5690 0x7ff964137790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:51.668 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.669+0000 7ff968da8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9640c89f0 0x7ff9640c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:51.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.669+0000 7ff968da8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff9641585b0 con 0x7ff9640c89f0 2024-03-19T01:59:51.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.669+0000 7ff968da8640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff96413de10 con 0x7ff9640c52b0 2024-03-19T01:59:51.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.669+0000 7ff968da8640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff964158730 con 0x7ff9640c5690 2024-03-19T01:59:51.672 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.673+0000 7ff961d9b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff9640c5690 0x7ff964137790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:51.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.673+0000 7ff961d9b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff9640c5690 0x7ff964137790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:45664/0 (socket says 172.21.15.55:45664) 2024-03-19T01:59:51.674 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.673+0000 7ff961d9b640 1 -- 172.21.15.55:0/2090466753 learned_addr learned my addr 172.21.15.55:0/2090466753 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:51.676 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.677+0000 7ff96259c640 1 --2- 172.21.15.55:0/2090466753 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9640c52b0 0x7ff9641ed940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:51.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.685+0000 7ff961d9b640 1 -- 172.21.15.55:0/2090466753 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9640c52b0 msgr2=0x7ff9641ed940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:51.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.685+0000 7ff961d9b640 1 --2- 172.21.15.55:0/2090466753 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9640c52b0 0x7ff9641ed940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:51.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.685+0000 7ff961d9b640 1 -- 172.21.15.55:0/2090466753 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9640c89f0 msgr2=0x7ff9640c99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:51.684 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.685+0000 7ff961d9b640 1 --2- 172.21.15.55:0/2090466753 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9640c89f0 0x7ff9640c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:51.684 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.685+0000 7ff961d9b640 1 -- 172.21.15.55:0/2090466753 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff96413b880 con 0x7ff9640c5690 2024-03-19T01:59:51.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.689+0000 7ff961d9b640 1 --2- 172.21.15.55:0/2090466753 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff9640c5690 0x7ff964137790 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff95800e840 tx=0x7ff95800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:51.716 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.717+0000 7ff952ffd640 1 -- 172.21.15.55:0/2090466753 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff958017070 con 0x7ff9640c5690 2024-03-19T01:59:51.716 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.717+0000 7ff952ffd640 1 -- 172.21.15.55:0/2090466753 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff958004020 con 0x7ff9640c5690 2024-03-19T01:59:51.717 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.717+0000 7ff968da8640 1 -- 172.21.15.55:0/2090466753 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff96413bb60 con 0x7ff9640c5690 2024-03-19T01:59:51.717 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.717+0000 7ff952ffd640 1 -- 172.21.15.55:0/2090466753 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9580043f0 con 0x7ff9640c5690 2024-03-19T01:59:51.717 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.717+0000 7ff968da8640 1 -- 172.21.15.55:0/2090466753 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff96413c120 con 0x7ff9640c5690 2024-03-19T01:59:51.717 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:51.717+0000 7ff968da8640 1 -- 172.21.15.55:0/2090466753 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff964068330 con 0x7ff9640c5690 2024-03-19T01:59:52.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:52.029+0000 7ff952ffd640 1 -- 172.21.15.55:0/2090466753 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ff95800c8c0 con 0x7ff9640c5690 2024-03-19T01:59:52.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:52.029+0000 7ff952ffd640 1 --2- 172.21.15.55:0/2090466753 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff9440676b0 0x7ff944069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:52.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:52.029+0000 7ff96259c640 1 --2- 172.21.15.55:0/2090466753 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff9440676b0 0x7ff944069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:52.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:52.029+0000 7ff96259c640 1 --2- 172.21.15.55:0/2090466753 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff9440676b0 0x7ff944069b70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff95400a830 tx=0x7ff954008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:52.031 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:52.033+0000 7ff952ffd640 1 -- 172.21.15.55:0/2090466753 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7ff9580544d0 con 0x7ff9640c5690 2024-03-19T01:59:52.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:52.057+0000 7ff952ffd640 1 -- 172.21.15.55:0/2090466753 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ff958056b30 con 0x7ff9640c5690 2024-03-19T01:59:52.262 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:52.261+0000 7ff968da8640 1 -- 172.21.15.55:0/2090466753 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff964156880 con 0x7ff9640c5690 2024-03-19T01:59:52.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:52.297+0000 7ff952ffd640 1 -- 172.21.15.55:0/2090466753 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7ff958056430 con 0x7ff9640c5690 2024-03-19T01:59:52.298 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:52.298 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:52.298 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:52.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:52.301+0000 7ff968da8640 1 -- 172.21.15.55:0/2090466753 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff9440676b0 msgr2=0x7ff944069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:52.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:52.301+0000 7ff968da8640 1 --2- 172.21.15.55:0/2090466753 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff9440676b0 0x7ff944069b70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff95400a830 tx=0x7ff954008040 comp rx=0 tx=0).stop 2024-03-19T01:59:52.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:52.301+0000 7ff968da8640 1 -- 172.21.15.55:0/2090466753 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff9640c5690 msgr2=0x7ff964137790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:52.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:52.301+0000 7ff968da8640 1 --2- 172.21.15.55:0/2090466753 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff9640c5690 0x7ff964137790 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff95800e840 tx=0x7ff95800ed00 comp rx=0 tx=0).stop 2024-03-19T01:59:52.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:52.301+0000 7ff968da8640 1 -- 172.21.15.55:0/2090466753 shutdown_connections 2024-03-19T01:59:52.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:52.301+0000 7ff968da8640 1 --2- 172.21.15.55:0/2090466753 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff9440676b0 0x7ff944069b70 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:52.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:52.301+0000 7ff968da8640 1 --2- 172.21.15.55:0/2090466753 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9640c89f0 0x7ff9640c99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:52.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:52.301+0000 7ff968da8640 1 --2- 172.21.15.55:0/2090466753 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff9640c5690 0x7ff964137790 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:52.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:52.301+0000 7ff968da8640 1 --2- 172.21.15.55:0/2090466753 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9640c52b0 0x7ff9641ed940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:52.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:52.301+0000 7ff968da8640 1 -- 172.21.15.55:0/2090466753 >> 172.21.15.55:0/2090466753 conn(0x7ff96406f3b0 msgr2=0x7ff964157360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:52.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:52.301+0000 7ff968da8640 1 -- 172.21.15.55:0/2090466753 shutdown_connections 2024-03-19T01:59:52.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:52.301+0000 7ff968da8640 1 -- 172.21.15.55:0/2090466753 wait complete. 2024-03-19T01:59:52.922 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:52.922 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:53.923 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:54.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.037+0000 7f7139d26640 1 Processor -- start 2024-03-19T01:59:54.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.037+0000 7f7139d26640 1 -- start start 2024-03-19T01:59:54.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.037+0000 7f7139d26640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f71340c5e10 0x7f7134067000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:54.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.037+0000 7f7139d26640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7134150d30 0x7f7134151180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:54.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.037+0000 7f7139d26640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71340c3770 con 0x7f71340c61f0 2024-03-19T01:59:54.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.037+0000 7f7139d26640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f71340c4340 con 0x7f7134150d30 2024-03-19T01:59:54.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.037+0000 7f7139d26640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f71341516c0 con 0x7f71340c5e10 2024-03-19T01:59:54.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.041+0000 7f7133fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7134150d30 0x7f7134151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:54.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.041+0000 7f71337fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f71340c5e10 0x7f7134067000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:54.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.041+0000 7f7133fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7134150d30 0x7f7134151180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48112/0 (socket says 172.21.15.55:48112) 2024-03-19T01:59:54.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.041+0000 7f7133fff640 1 -- 172.21.15.55:0/2379564030 learned_addr learned my addr 172.21.15.55:0/2379564030 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:54.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.045+0000 7f7133fff640 1 -- 172.21.15.55:0/2379564030 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f71340c5e10 msgr2=0x7f7134067000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:54.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.045+0000 7f7133fff640 1 --2- 172.21.15.55:0/2379564030 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f71340c5e10 0x7f7134067000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:54.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.045+0000 7f7133fff640 1 -- 172.21.15.55:0/2379564030 >> v1:172.21.15.55:6789/0 conn(0x7f71340c61f0 legacy=0x7f71340c6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:54.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.045+0000 7f7133fff640 1 -- 172.21.15.55:0/2379564030 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7134067690 con 0x7f7134150d30 2024-03-19T01:59:54.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.049+0000 7f7133fff640 1 --2- 172.21.15.55:0/2379564030 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7134150d30 0x7f7134151180 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f712c009e50 tx=0x7f712c01f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6e7e1f0e7ba4e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:54.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.053+0000 7f71327fc640 1 -- 172.21.15.55:0/2379564030 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c0b4070 con 0x7f7134150d30 2024-03-19T01:59:54.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.053+0000 7f71327fc640 1 -- 172.21.15.55:0/2379564030 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f712c0af400 con 0x7f7134150d30 2024-03-19T01:59:54.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.053+0000 7f71327fc640 1 -- 172.21.15.55:0/2379564030 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c0af7d0 con 0x7f7134150d30 2024-03-19T01:59:54.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.053+0000 7f7139d26640 1 -- 172.21.15.55:0/2379564030 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7134150d30 msgr2=0x7f7134151180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:54.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.053+0000 7f7139d26640 1 --2- 172.21.15.55:0/2379564030 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7134150d30 0x7f7134151180 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f712c009e50 tx=0x7f712c01f210 comp rx=0 tx=0).stop 2024-03-19T01:59:54.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.053+0000 7f7139d26640 1 -- 172.21.15.55:0/2379564030 shutdown_connections 2024-03-19T01:59:54.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.053+0000 7f7139d26640 1 --2- 172.21.15.55:0/2379564030 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7134150d30 0x7f7134151180 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:54.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.053+0000 7f7139d26640 1 --2- 172.21.15.55:0/2379564030 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f71340c5e10 0x7f7134067000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:54.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.053+0000 7f7139d26640 1 -- 172.21.15.55:0/2379564030 >> 172.21.15.55:0/2379564030 conn(0x7f71340cc270 msgr2=0x7f71340cc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:54.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.053+0000 7f7139d26640 1 -- 172.21.15.55:0/2379564030 shutdown_connections 2024-03-19T01:59:54.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.053+0000 7f7139d26640 1 -- 172.21.15.55:0/2379564030 wait complete. 2024-03-19T01:59:54.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.057+0000 7f7139d26640 1 Processor -- start 2024-03-19T01:59:54.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.057+0000 7f7139d26640 1 -- start start 2024-03-19T01:59:54.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.057+0000 7f7139d26640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f71340c5e10 0x7f7134152d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:54.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.057+0000 7f7139d26640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7134150d30 0x7f71341571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:54.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.057+0000 7f7139d26640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71341efb20 0x7f71341effb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:54.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.057+0000 7f7139d26640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f71340c3cf0 con 0x7f71341efb20 2024-03-19T01:59:54.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.057+0000 7f7139d26640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f71340c7380 con 0x7f7134150d30 2024-03-19T01:59:54.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.057+0000 7f7139d26640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f71340c3fd0 con 0x7f71340c5e10 2024-03-19T01:59:54.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.057+0000 7f7133fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71341efb20 0x7f71341effb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:54.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.057+0000 7f7133fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71341efb20 0x7f71341effb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48124/0 (socket says 172.21.15.55:48124) 2024-03-19T01:59:54.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.057+0000 7f7133fff640 1 -- 172.21.15.55:0/2706265709 learned_addr learned my addr 172.21.15.55:0/2706265709 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:54.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.061+0000 7f71337fe640 1 --2- 172.21.15.55:0/2706265709 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f71340c5e10 0x7f7134152d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:54.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.061+0000 7f7132ffd640 1 --2- 172.21.15.55:0/2706265709 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7134150d30 0x7f71341571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:54.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.061+0000 7f7133fff640 1 -- 172.21.15.55:0/2706265709 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f71340c5e10 msgr2=0x7f7134152d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:54.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.061+0000 7f7133fff640 1 --2- 172.21.15.55:0/2706265709 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f71340c5e10 0x7f7134152d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:54.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.061+0000 7f7133fff640 1 -- 172.21.15.55:0/2706265709 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7134150d30 msgr2=0x7f71341571b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:54.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.061+0000 7f7133fff640 1 --2- 172.21.15.55:0/2706265709 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7134150d30 0x7f71341571b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:54.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.061+0000 7f7133fff640 1 -- 172.21.15.55:0/2706265709 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7134152030 con 0x7f71341efb20 2024-03-19T01:59:54.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.065+0000 7f7133fff640 1 --2- 172.21.15.55:0/2706265709 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71341efb20 0x7f71341effb0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f712c01ce40 tx=0x7f712c01bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:54.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.069+0000 7f711bfff640 1 -- 172.21.15.55:0/2706265709 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c0b4040 con 0x7f71341efb20 2024-03-19T01:59:54.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.069+0000 7f711bfff640 1 -- 172.21.15.55:0/2706265709 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f712c0bf070 con 0x7f71341efb20 2024-03-19T01:59:54.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.069+0000 7f7139d26640 1 -- 172.21.15.55:0/2706265709 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71341f3ef0 con 0x7f71341efb20 2024-03-19T01:59:54.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.069+0000 7f711bfff640 1 -- 172.21.15.55:0/2706265709 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c0b9410 con 0x7f71341efb20 2024-03-19T01:59:54.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.069+0000 7f7139d26640 1 -- 172.21.15.55:0/2706265709 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71341f3130 con 0x7f71341efb20 2024-03-19T01:59:54.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.073+0000 7f7139d26640 1 -- 172.21.15.55:0/2706265709 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7134068a40 con 0x7f71341efb20 2024-03-19T01:59:54.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.101+0000 7f711bfff640 1 -- 172.21.15.55:0/2706265709 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f712c0a8d10 con 0x7f71341efb20 2024-03-19T01:59:54.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.105+0000 7f711bfff640 1 --2- 172.21.15.55:0/2706265709 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f71200676b0 0x7f7120069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:54.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.105+0000 7f71337fe640 1 --2- 172.21.15.55:0/2706265709 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f71200676b0 0x7f7120069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:54.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.105+0000 7f71337fe640 1 --2- 172.21.15.55:0/2706265709 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f71200676b0 0x7f7120069b70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7124009af0 tx=0x7f7124008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:54.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.109+0000 7f711bfff640 1 -- 172.21.15.55:0/2706265709 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f712c0f2740 con 0x7f71341efb20 2024-03-19T01:59:54.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.405+0000 7f711bfff640 1 -- 172.21.15.55:0/2706265709 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f712c0c9020 con 0x7f71341efb20 2024-03-19T01:59:54.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.629+0000 7f7139d26640 1 -- 172.21.15.55:0/2706265709 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71340ccad0 con 0x7f71341efb20 2024-03-19T01:59:54.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.649+0000 7f711bfff640 1 -- 172.21.15.55:0/2706265709 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f712c018d70 con 0x7f71341efb20 2024-03-19T01:59:54.648 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:54.648 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:54.648 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:54.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.649+0000 7f7139d26640 1 -- 172.21.15.55:0/2706265709 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f71200676b0 msgr2=0x7f7120069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:54.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.649+0000 7f7139d26640 1 --2- 172.21.15.55:0/2706265709 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f71200676b0 0x7f7120069b70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7124009af0 tx=0x7f7124008040 comp rx=0 tx=0).stop 2024-03-19T01:59:54.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.649+0000 7f7139d26640 1 -- 172.21.15.55:0/2706265709 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71341efb20 msgr2=0x7f71341effb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:54.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.649+0000 7f7139d26640 1 --2- 172.21.15.55:0/2706265709 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71341efb20 0x7f71341effb0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f712c01ce40 tx=0x7f712c01bfb0 comp rx=0 tx=0).stop 2024-03-19T01:59:54.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.649+0000 7f7139d26640 1 -- 172.21.15.55:0/2706265709 shutdown_connections 2024-03-19T01:59:54.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.649+0000 7f7139d26640 1 --2- 172.21.15.55:0/2706265709 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f71200676b0 0x7f7120069b70 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:54.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.649+0000 7f7139d26640 1 --2- 172.21.15.55:0/2706265709 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71341efb20 0x7f71341effb0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:54.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.649+0000 7f7139d26640 1 --2- 172.21.15.55:0/2706265709 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7134150d30 0x7f71341571b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:54.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.649+0000 7f7139d26640 1 --2- 172.21.15.55:0/2706265709 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f71340c5e10 0x7f7134152d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:54.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.649+0000 7f7139d26640 1 -- 172.21.15.55:0/2706265709 >> 172.21.15.55:0/2706265709 conn(0x7f71340cc270 msgr2=0x7f713406b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:54.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.649+0000 7f7139d26640 1 -- 172.21.15.55:0/2706265709 shutdown_connections 2024-03-19T01:59:54.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:54.653+0000 7f7139d26640 1 -- 172.21.15.55:0/2706265709 wait complete. 2024-03-19T01:59:55.271 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:55.271 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:56.272 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:56.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.405+0000 7f922304b640 1 Processor -- start 2024-03-19T01:59:56.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.405+0000 7f922304b640 1 -- start start 2024-03-19T01:59:56.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.405+0000 7f922304b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f921c150f50 0x7f921c15b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:56.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.405+0000 7f922304b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f921c0c3320 0x7f921c0728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:56.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.405+0000 7f922304b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f921c151330 con 0x7f921c0ca350 2024-03-19T01:59:56.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.405+0000 7f922304b640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f921c0c3e00 con 0x7f921c150f50 2024-03-19T01:59:56.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.405+0000 7f922304b640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f921c15ac10 con 0x7f921c0c3320 2024-03-19T01:59:56.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.405+0000 7f9220de7640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f921c150f50 0x7f921c15b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:56.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.405+0000 7f92215e8640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f921c0c3320 0x7f921c0728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:56.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.405+0000 7f9220de7640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f921c150f50 0x7f921c15b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48134/0 (socket says 172.21.15.55:48134) 2024-03-19T01:59:56.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.405+0000 7f9220de7640 1 -- 172.21.15.55:0/36037353 learned_addr learned my addr 172.21.15.55:0/36037353 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:56.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.409+0000 7f9220de7640 1 -- 172.21.15.55:0/36037353 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f921c0c3320 msgr2=0x7f921c0728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:56.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.409+0000 7f9220de7640 1 --2- 172.21.15.55:0/36037353 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f921c0c3320 0x7f921c0728b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:56.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.409+0000 7f9220de7640 1 -- 172.21.15.55:0/36037353 >> v1:172.21.15.55:6789/0 conn(0x7f921c0ca350 legacy=0x7f921c157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:56.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.409+0000 7f9220de7640 1 -- 172.21.15.55:0/36037353 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f921c0c3840 con 0x7f921c150f50 2024-03-19T01:59:56.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.413+0000 7f9220de7640 1 --2- 172.21.15.55:0/36037353 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f921c150f50 0x7f921c15b0a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f920c00bb10 tx=0x7f920c01ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f1f603d6d799ca server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:56.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.413+0000 7f921b7fe640 1 -- 172.21.15.55:0/36037353 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f920c0b4070 con 0x7f921c150f50 2024-03-19T01:59:56.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.417+0000 7f921b7fe640 1 -- 172.21.15.55:0/36037353 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f920c0a7c80 con 0x7f921c150f50 2024-03-19T01:59:56.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.417+0000 7f921b7fe640 1 -- 172.21.15.55:0/36037353 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f920c0af660 con 0x7f921c150f50 2024-03-19T01:59:56.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.417+0000 7f922304b640 1 -- 172.21.15.55:0/36037353 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f921c150f50 msgr2=0x7f921c15b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:56.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.417+0000 7f922304b640 1 --2- 172.21.15.55:0/36037353 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f921c150f50 0x7f921c15b0a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f920c00bb10 tx=0x7f920c01ebf0 comp rx=0 tx=0).stop 2024-03-19T01:59:56.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.417+0000 7f922304b640 1 -- 172.21.15.55:0/36037353 shutdown_connections 2024-03-19T01:59:56.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.417+0000 7f922304b640 1 --2- 172.21.15.55:0/36037353 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f921c0c3320 0x7f921c0728b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:56.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.417+0000 7f922304b640 1 --2- 172.21.15.55:0/36037353 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f921c150f50 0x7f921c15b0a0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:56.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.417+0000 7f922304b640 1 -- 172.21.15.55:0/36037353 >> 172.21.15.55:0/36037353 conn(0x7f921c06e670 msgr2=0x7f921c06ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:56.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.417+0000 7f922304b640 1 -- 172.21.15.55:0/36037353 shutdown_connections 2024-03-19T01:59:56.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.417+0000 7f922304b640 1 -- 172.21.15.55:0/36037353 wait complete. 2024-03-19T01:59:56.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.417+0000 7f922304b640 1 Processor -- start 2024-03-19T01:59:56.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.417+0000 7f922304b640 1 -- start start 2024-03-19T01:59:56.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.421+0000 7f922304b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f921c0c3320 0x7f921c14f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:56.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.421+0000 7f922304b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f921c0ca350 0x7f921c070410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:56.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.421+0000 7f922304b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f921c150f50 0x7f921c158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:56.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.421+0000 7f922304b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f921c06b420 con 0x7f921c150f50 2024-03-19T01:59:56.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.421+0000 7f922304b640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f921c06b2a0 con 0x7f921c0c3320 2024-03-19T01:59:56.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.421+0000 7f922304b640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f921c06b5a0 con 0x7f921c0ca350 2024-03-19T01:59:56.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.421+0000 7f92215e8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f921c150f50 0x7f921c158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:56.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.421+0000 7f921bfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f921c0ca350 0x7f921c070410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:56.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.421+0000 7f9220de7640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f921c0c3320 0x7f921c14f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:56.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.421+0000 7f92215e8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f921c150f50 0x7f921c158060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48142/0 (socket says 172.21.15.55:48142) 2024-03-19T01:59:56.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.421+0000 7f92215e8640 1 -- 172.21.15.55:0/458295567 learned_addr learned my addr 172.21.15.55:0/458295567 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:56.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.425+0000 7f92215e8640 1 -- 172.21.15.55:0/458295567 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f921c0ca350 msgr2=0x7f921c070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:56.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.425+0000 7f92215e8640 1 --2- 172.21.15.55:0/458295567 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f921c0ca350 0x7f921c070410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:56.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.425+0000 7f92215e8640 1 -- 172.21.15.55:0/458295567 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f921c0c3320 msgr2=0x7f921c14f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:56.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.425+0000 7f92215e8640 1 --2- 172.21.15.55:0/458295567 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f921c0c3320 0x7f921c14f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:56.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.425+0000 7f92215e8640 1 -- 172.21.15.55:0/458295567 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f921c14fcb0 con 0x7f921c150f50 2024-03-19T01:59:56.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.425+0000 7f92215e8640 1 --2- 172.21.15.55:0/458295567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f921c150f50 0x7f921c158060 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f9214009d80 tx=0x7f921400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:56.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.429+0000 7f92197fa640 1 -- 172.21.15.55:0/458295567 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9214018020 con 0x7f921c150f50 2024-03-19T01:59:56.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.429+0000 7f92197fa640 1 -- 172.21.15.55:0/458295567 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9214014480 con 0x7f921c150f50 2024-03-19T01:59:56.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.429+0000 7f92197fa640 1 -- 172.21.15.55:0/458295567 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9214014870 con 0x7f921c150f50 2024-03-19T01:59:56.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.429+0000 7f922304b640 1 -- 172.21.15.55:0/458295567 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f921c14ff90 con 0x7f921c150f50 2024-03-19T01:59:56.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.429+0000 7f922304b640 1 -- 172.21.15.55:0/458295567 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f921c1f29a0 con 0x7f921c150f50 2024-03-19T01:59:56.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.433+0000 7f922304b640 1 -- 172.21.15.55:0/458295567 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f921c068a40 con 0x7f921c150f50 2024-03-19T01:59:56.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.461+0000 7f92197fa640 1 -- 172.21.15.55:0/458295567 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f921401b020 con 0x7f921c150f50 2024-03-19T01:59:56.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.465+0000 7f92197fa640 1 --2- 172.21.15.55:0/458295567 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92080676b0 0x7f9208069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:56.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.465+0000 7f9220de7640 1 --2- 172.21.15.55:0/458295567 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92080676b0 0x7f9208069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:56.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.465+0000 7f9220de7640 1 --2- 172.21.15.55:0/458295567 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92080676b0 0x7f9208069b70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f920c0b1000 tx=0x7f920c0a9070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:56.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.469+0000 7f92197fa640 1 -- 172.21.15.55:0/458295567 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f9214055200 con 0x7f921c150f50 2024-03-19T01:59:56.737 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.737+0000 7f92197fa640 1 -- 172.21.15.55:0/458295567 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f9214022030 con 0x7f921c150f50 2024-03-19T01:59:56.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:56.977+0000 7f922304b640 1 -- 172.21.15.55:0/458295567 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f921c0c7600 con 0x7f921c150f50 2024-03-19T01:59:57.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:57.069+0000 7f92197fa640 1 -- 172.21.15.55:0/458295567 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f9214010040 con 0x7f921c150f50 2024-03-19T01:59:57.068 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:57.069 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:57.069 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:57.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:57.073+0000 7f922304b640 1 -- 172.21.15.55:0/458295567 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92080676b0 msgr2=0x7f9208069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:57.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:57.073+0000 7f922304b640 1 --2- 172.21.15.55:0/458295567 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92080676b0 0x7f9208069b70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f920c0b1000 tx=0x7f920c0a9070 comp rx=0 tx=0).stop 2024-03-19T01:59:57.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:57.073+0000 7f922304b640 1 -- 172.21.15.55:0/458295567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f921c150f50 msgr2=0x7f921c158060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:57.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:57.073+0000 7f922304b640 1 --2- 172.21.15.55:0/458295567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f921c150f50 0x7f921c158060 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f9214009d80 tx=0x7f921400c490 comp rx=0 tx=0).stop 2024-03-19T01:59:57.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:57.073+0000 7f922304b640 1 -- 172.21.15.55:0/458295567 shutdown_connections 2024-03-19T01:59:57.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:57.073+0000 7f922304b640 1 --2- 172.21.15.55:0/458295567 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92080676b0 0x7f9208069b70 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:57.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:57.073+0000 7f922304b640 1 --2- 172.21.15.55:0/458295567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f921c150f50 0x7f921c158060 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:57.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:57.073+0000 7f922304b640 1 --2- 172.21.15.55:0/458295567 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f921c0ca350 0x7f921c070410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:57.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:57.073+0000 7f922304b640 1 --2- 172.21.15.55:0/458295567 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f921c0c3320 0x7f921c14f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:57.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:57.073+0000 7f922304b640 1 -- 172.21.15.55:0/458295567 >> 172.21.15.55:0/458295567 conn(0x7f921c06e670 msgr2=0x7f921c1473f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:57.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:57.073+0000 7f922304b640 1 -- 172.21.15.55:0/458295567 shutdown_connections 2024-03-19T01:59:57.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:57.073+0000 7f922304b640 1 -- 172.21.15.55:0/458295567 wait complete. 2024-03-19T01:59:57.694 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T01:59:57.694 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T01:59:58.695 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:59:58.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.825+0000 7f46cb812640 1 Processor -- start 2024-03-19T01:59:58.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.825+0000 7f46cb812640 1 -- start start 2024-03-19T01:59:58.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.825+0000 7f46cb812640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46c415a6e0 0x7f46c40ceb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:58.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.825+0000 7f46cb812640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f46c406c3e0 0x7f46c406c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:58.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.825+0000 7f46cb812640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46c40cda20 con 0x7f46c415aac0 2024-03-19T01:59:58.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.825+0000 7f46cb812640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f46c40cdcc0 con 0x7f46c406c3e0 2024-03-19T01:59:58.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.825+0000 7f46cb812640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f46c40cdfa0 con 0x7f46c415a6e0 2024-03-19T01:59:58.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.825+0000 7f46c95ae640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46c415a6e0 0x7f46c40ceb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:58.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.825+0000 7f46c95ae640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46c415a6e0 0x7f46c40ceb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:54446/0 (socket says 172.21.15.55:54446) 2024-03-19T01:59:58.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.825+0000 7f46c95ae640 1 -- 172.21.15.55:0/1587991215 learned_addr learned my addr 172.21.15.55:0/1587991215 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:58.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.829+0000 7f46c8dad640 1 --2- 172.21.15.55:0/1587991215 >> v2:172.21.15.84:3300/0 conn(0x7f46c406c3e0 0x7f46c406c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:58.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.829+0000 7f46c95ae640 1 -- 172.21.15.55:0/1587991215 >> v2:172.21.15.84:3300/0 conn(0x7f46c406c3e0 msgr2=0x7f46c406c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:59:58.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.829+0000 7f46c95ae640 1 --2- 172.21.15.55:0/1587991215 >> v2:172.21.15.84:3300/0 conn(0x7f46c406c3e0 0x7f46c406c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:58.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.829+0000 7f46c95ae640 1 -- 172.21.15.55:0/1587991215 >> v1:172.21.15.84:6789/0 conn(0x7f46c415aac0 legacy=0x7f46c40ca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:58.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.829+0000 7f46c95ae640 1 -- 172.21.15.55:0/1587991215 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c40ce280 con 0x7f46c415a6e0 2024-03-19T01:59:58.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.833+0000 7f46c95ae640 1 --2- 172.21.15.55:0/1587991215 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46c415a6e0 0x7f46c40ceb60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f46c0009870 tx=0x7f46c001ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9db25d6bdbb82c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:58.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.841+0000 7f46bbfff640 1 -- 172.21.15.55:0/1587991215 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c00b5070 con 0x7f46c415a6e0 2024-03-19T01:59:58.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.841+0000 7f46bbfff640 1 -- 172.21.15.55:0/1587991215 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46c00a8a50 con 0x7f46c415a6e0 2024-03-19T01:59:58.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.841+0000 7f46bbfff640 1 -- 172.21.15.55:0/1587991215 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c00a8e20 con 0x7f46c415a6e0 2024-03-19T01:59:58.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.841+0000 7f46cb812640 1 -- 172.21.15.55:0/1587991215 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46c415a6e0 msgr2=0x7f46c40ceb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:58.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.841+0000 7f46cb812640 1 --2- 172.21.15.55:0/1587991215 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46c415a6e0 0x7f46c40ceb60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f46c0009870 tx=0x7f46c001ed50 comp rx=0 tx=0).stop 2024-03-19T01:59:58.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.845+0000 7f46cb812640 1 -- 172.21.15.55:0/1587991215 shutdown_connections 2024-03-19T01:59:58.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.845+0000 7f46cb812640 1 --2- 172.21.15.55:0/1587991215 >> v2:172.21.15.84:3300/0 conn(0x7f46c406c3e0 0x7f46c406c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:58.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.845+0000 7f46cb812640 1 --2- 172.21.15.55:0/1587991215 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46c415a6e0 0x7f46c40ceb60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:58.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.845+0000 7f46cb812640 1 -- 172.21.15.55:0/1587991215 >> 172.21.15.55:0/1587991215 conn(0x7f46c40cfd70 msgr2=0x7f46c40d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:58.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.845+0000 7f46cb812640 1 -- 172.21.15.55:0/1587991215 shutdown_connections 2024-03-19T01:59:58.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.845+0000 7f46cb812640 1 -- 172.21.15.55:0/1587991215 wait complete. 2024-03-19T01:59:58.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.845+0000 7f46cb812640 1 Processor -- start 2024-03-19T01:59:58.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.845+0000 7f46cb812640 1 -- start start 2024-03-19T01:59:58.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.845+0000 7f46cb812640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46c406c3e0 0x7f46c40c6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:58.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.845+0000 7f46cb812640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46c415a6e0 0x7f46c41512c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:58.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.845+0000 7f46cb812640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46c415aac0 0x7f46c40aff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:58.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.845+0000 7f46cb812640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f46c40cf0a0 con 0x7f46c406c3e0 2024-03-19T01:59:58.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.845+0000 7f46cb812640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f46c40ce840 con 0x7f46c415aac0 2024-03-19T01:59:58.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.845+0000 7f46cb812640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f46c40cf220 con 0x7f46c415a6e0 2024-03-19T01:59:58.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.845+0000 7f46c8dad640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46c415a6e0 0x7f46c41512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:58.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.849+0000 7f46c95ae640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46c406c3e0 0x7f46c40c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:58.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.849+0000 7f46c8dad640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46c415a6e0 0x7f46c41512c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:54462/0 (socket says 172.21.15.55:54462) 2024-03-19T01:59:58.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.849+0000 7f46c8dad640 1 -- 172.21.15.55:0/2880417108 learned_addr learned my addr 172.21.15.55:0/2880417108 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T01:59:58.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.849+0000 7f46c9daf640 1 --2- 172.21.15.55:0/2880417108 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46c415aac0 0x7f46c40aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:58.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.853+0000 7f46c95ae640 1 -- 172.21.15.55:0/2880417108 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46c415a6e0 msgr2=0x7f46c41512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:58.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.853+0000 7f46c95ae640 1 --2- 172.21.15.55:0/2880417108 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46c415a6e0 0x7f46c41512c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:58.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.853+0000 7f46c95ae640 1 -- 172.21.15.55:0/2880417108 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46c415aac0 msgr2=0x7f46c40aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:58.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.853+0000 7f46c95ae640 1 --2- 172.21.15.55:0/2880417108 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46c415aac0 0x7f46c40aff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:58.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.853+0000 7f46c95ae640 1 -- 172.21.15.55:0/2880417108 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c4151800 con 0x7f46c406c3e0 2024-03-19T01:59:58.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.853+0000 7f46c95ae640 1 --2- 172.21.15.55:0/2880417108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46c406c3e0 0x7f46c40c6220 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f46c00af000 tx=0x7f46c001baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:58.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.857+0000 7f46b9ffb640 1 -- 172.21.15.55:0/2880417108 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c00b5040 con 0x7f46c406c3e0 2024-03-19T01:59:58.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.857+0000 7f46b9ffb640 1 -- 172.21.15.55:0/2880417108 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46c00a8850 con 0x7f46c406c3e0 2024-03-19T01:59:58.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.857+0000 7f46cb812640 1 -- 172.21.15.55:0/2880417108 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46c41f00c0 con 0x7f46c406c3e0 2024-03-19T01:59:58.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.857+0000 7f46b9ffb640 1 -- 172.21.15.55:0/2880417108 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c00b1e30 con 0x7f46c406c3e0 2024-03-19T01:59:58.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.857+0000 7f46cb812640 1 -- 172.21.15.55:0/2880417108 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46c41f0380 con 0x7f46c406c3e0 2024-03-19T01:59:58.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:58.857+0000 7f46cb812640 1 -- 172.21.15.55:0/2880417108 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46c4068330 con 0x7f46c406c3e0 2024-03-19T01:59:59.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:59.141+0000 7f46b9ffb640 1 -- 172.21.15.55:0/2880417108 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f46c00c0030 con 0x7f46c406c3e0 2024-03-19T01:59:59.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:59.141+0000 7f46b9ffb640 1 --2- 172.21.15.55:0/2880417108 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f46a8067690 0x7f46a8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:59:59.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:59.141+0000 7f46c8dad640 1 --2- 172.21.15.55:0/2880417108 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f46a8067690 0x7f46a8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:59:59.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:59.141+0000 7f46c8dad640 1 --2- 172.21.15.55:0/2880417108 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f46a8067690 0x7f46a8069b50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f46c40cb850 tx=0x7f46b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:59:59.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:59.169+0000 7f46b9ffb640 1 -- 172.21.15.55:0/2880417108 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f46c00a8a00 con 0x7f46c406c3e0 2024-03-19T01:59:59.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:59.169+0000 7f46b9ffb640 1 -- 172.21.15.55:0/2880417108 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f46c00c7060 con 0x7f46c406c3e0 2024-03-19T01:59:59.388 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:59.389+0000 7f46cb812640 1 -- 172.21.15.55:0/2880417108 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46c40afd50 con 0x7f46c406c3e0 2024-03-19T01:59:59.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:59.409+0000 7f46b9ffb640 1 -- 172.21.15.55:0/2880417108 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f46c00188b0 con 0x7f46c406c3e0 2024-03-19T01:59:59.407 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T01:59:59.408 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T01:59:59.408 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T01:59:59.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:59.409+0000 7f46cb812640 1 -- 172.21.15.55:0/2880417108 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f46a8067690 msgr2=0x7f46a8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:59.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:59.409+0000 7f46cb812640 1 --2- 172.21.15.55:0/2880417108 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f46a8067690 0x7f46a8069b50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f46c40cb850 tx=0x7f46b4008040 comp rx=0 tx=0).stop 2024-03-19T01:59:59.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:59.409+0000 7f46cb812640 1 -- 172.21.15.55:0/2880417108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46c406c3e0 msgr2=0x7f46c40c6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:59:59.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:59.409+0000 7f46cb812640 1 --2- 172.21.15.55:0/2880417108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46c406c3e0 0x7f46c40c6220 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f46c00af000 tx=0x7f46c001baf0 comp rx=0 tx=0).stop 2024-03-19T01:59:59.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:59.409+0000 7f46cb812640 1 -- 172.21.15.55:0/2880417108 shutdown_connections 2024-03-19T01:59:59.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:59.409+0000 7f46cb812640 1 --2- 172.21.15.55:0/2880417108 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f46a8067690 0x7f46a8069b50 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:59.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:59.409+0000 7f46cb812640 1 --2- 172.21.15.55:0/2880417108 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46c415aac0 0x7f46c40aff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:59.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:59.409+0000 7f46cb812640 1 --2- 172.21.15.55:0/2880417108 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f46c415a6e0 0x7f46c41512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:59.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:59.409+0000 7f46cb812640 1 --2- 172.21.15.55:0/2880417108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f46c406c3e0 0x7f46c40c6220 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:59:59.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:59.409+0000 7f46cb812640 1 -- 172.21.15.55:0/2880417108 >> 172.21.15.55:0/2880417108 conn(0x7f46c40cfd70 msgr2=0x7f46c40c36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:59:59.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:59.409+0000 7f46cb812640 1 -- 172.21.15.55:0/2880417108 shutdown_connections 2024-03-19T01:59:59.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T01:59:59.409+0000 7f46cb812640 1 -- 172.21.15.55:0/2880417108 wait complete. 2024-03-19T02:00:00.034 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:00.034 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:01.036 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:01.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.165+0000 7f7846024640 1 Processor -- start 2024-03-19T02:00:01.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.165+0000 7f7846024640 1 -- start start 2024-03-19T02:00:01.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.165+0000 7f7846024640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7840154290 0x7f784015e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:01.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.165+0000 7f7846024640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f78400ca310 0x7f7840154670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:01.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.165+0000 7f7846024640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f784015ad40 0x7f7840072900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:01.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.165+0000 7f7846024640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f784015b2c0 con 0x7f7840154290 2024-03-19T02:00:01.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.165+0000 7f7846024640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f784006c1d0 con 0x7f784015ad40 2024-03-19T02:00:01.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.165+0000 7f7846024640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f784015e030 con 0x7f78400ca310 2024-03-19T02:00:01.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.165+0000 7f783f7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7840154290 0x7f784015e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:01.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.165+0000 7f783ffff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f784015ad40 0x7f7840072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:01.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.165+0000 7f783effd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f78400ca310 0x7f7840154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:01.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.165+0000 7f783ffff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f784015ad40 0x7f7840072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:35378/0 (socket says 172.21.15.55:35378) 2024-03-19T02:00:01.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.165+0000 7f783f7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7840154290 0x7f784015e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58660/0 (socket says 172.21.15.55:58660) 2024-03-19T02:00:01.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.165+0000 7f783ffff640 1 -- 172.21.15.55:0/4002237698 learned_addr learned my addr 172.21.15.55:0/4002237698 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:01.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.169+0000 7f783f7fe640 1 -- 172.21.15.55:0/4002237698 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f78400ca310 msgr2=0x7f7840154670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:01.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.169+0000 7f783f7fe640 1 --2- 172.21.15.55:0/4002237698 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f78400ca310 0x7f7840154670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:01.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.169+0000 7f783f7fe640 1 -- 172.21.15.55:0/4002237698 >> v2:172.21.15.84:3300/0 conn(0x7f784015ad40 msgr2=0x7f7840072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:01.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.169+0000 7f783f7fe640 1 --2- 172.21.15.55:0/4002237698 >> v2:172.21.15.84:3300/0 conn(0x7f784015ad40 0x7f7840072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:01.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.169+0000 7f783f7fe640 1 -- 172.21.15.55:0/4002237698 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f784006bc90 con 0x7f7840154290 2024-03-19T02:00:01.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.173+0000 7f783f7fe640 1 --2- 172.21.15.55:0/4002237698 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7840154290 0x7f784015e4c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f784006ff90 tx=0x7f783001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2786ce7de8a2d644 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:01.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.177+0000 7f783e7fc640 1 -- 172.21.15.55:0/4002237698 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78300ab660 con 0x7f7840154290 2024-03-19T02:00:01.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.177+0000 7f783e7fc640 1 -- 172.21.15.55:0/4002237698 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78300b0070 con 0x7f7840154290 2024-03-19T02:00:01.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.177+0000 7f783e7fc640 1 -- 172.21.15.55:0/4002237698 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78300aba20 con 0x7f7840154290 2024-03-19T02:00:01.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.177+0000 7f7846024640 1 -- 172.21.15.55:0/4002237698 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7840154290 msgr2=0x7f784015e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:01.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.177+0000 7f7846024640 1 --2- 172.21.15.55:0/4002237698 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7840154290 0x7f784015e4c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f784006ff90 tx=0x7f783001c480 comp rx=0 tx=0).stop 2024-03-19T02:00:01.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.177+0000 7f7846024640 1 -- 172.21.15.55:0/4002237698 shutdown_connections 2024-03-19T02:00:01.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.177+0000 7f7846024640 1 --2- 172.21.15.55:0/4002237698 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f78400ca310 0x7f7840154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:01.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.177+0000 7f7846024640 1 --2- 172.21.15.55:0/4002237698 >> v2:172.21.15.84:3300/0 conn(0x7f784015ad40 0x7f7840072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:01.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.177+0000 7f7846024640 1 --2- 172.21.15.55:0/4002237698 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7840154290 0x7f784015e4c0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:01.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.177+0000 7f7846024640 1 -- 172.21.15.55:0/4002237698 >> 172.21.15.55:0/4002237698 conn(0x7f784006c580 msgr2=0x7f784006c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:01.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.177+0000 7f7846024640 1 -- 172.21.15.55:0/4002237698 shutdown_connections 2024-03-19T02:00:01.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.177+0000 7f7846024640 1 -- 172.21.15.55:0/4002237698 wait complete. 2024-03-19T02:00:01.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.181+0000 7f7846024640 1 Processor -- start 2024-03-19T02:00:01.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.181+0000 7f7846024640 1 -- start start 2024-03-19T02:00:01.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.181+0000 7f7846024640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78400ca310 0x7f78401f3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:01.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.181+0000 7f7846024640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7840154290 0x7f7840153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:01.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.181+0000 7f7846024640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f784015ad40 0x7f784014f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:01.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.181+0000 7f7846024640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f78400ccb50 con 0x7f78400ca310 2024-03-19T02:00:01.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.181+0000 7f7846024640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f784006bf30 con 0x7f784015ad40 2024-03-19T02:00:01.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.181+0000 7f7846024640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f78400cccd0 con 0x7f7840154290 2024-03-19T02:00:01.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.181+0000 7f783ffff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f784015ad40 0x7f784014f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:01.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.181+0000 7f783effd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7840154290 0x7f7840153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:01.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.181+0000 7f783f7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78400ca310 0x7f78401f3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:01.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.181+0000 7f783ffff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f784015ad40 0x7f784014f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:35390/0 (socket says 172.21.15.55:35390) 2024-03-19T02:00:01.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.181+0000 7f783ffff640 1 -- 172.21.15.55:0/2691989980 learned_addr learned my addr 172.21.15.55:0/2691989980 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:01.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.185+0000 7f783f7fe640 1 -- 172.21.15.55:0/2691989980 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7840154290 msgr2=0x7f7840153880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:01.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.185+0000 7f783f7fe640 1 --2- 172.21.15.55:0/2691989980 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7840154290 0x7f7840153880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:01.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.185+0000 7f783f7fe640 1 -- 172.21.15.55:0/2691989980 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f784015ad40 msgr2=0x7f784014f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:01.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.185+0000 7f783f7fe640 1 --2- 172.21.15.55:0/2691989980 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f784015ad40 0x7f784014f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:01.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.185+0000 7f783f7fe640 1 -- 172.21.15.55:0/2691989980 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78400c4130 con 0x7f78400ca310 2024-03-19T02:00:01.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.189+0000 7f783f7fe640 1 --2- 172.21.15.55:0/2691989980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78400ca310 0x7f78401f3880 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7840071390 tx=0x7f783001acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:01.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.193+0000 7f7827fff640 1 -- 172.21.15.55:0/2691989980 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78300b5600 con 0x7f78400ca310 2024-03-19T02:00:01.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.193+0000 7f7827fff640 1 -- 172.21.15.55:0/2691989980 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78300b57c0 con 0x7f78400ca310 2024-03-19T02:00:01.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.193+0000 7f7846024640 1 -- 172.21.15.55:0/2691989980 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78401f2dc0 con 0x7f78400ca310 2024-03-19T02:00:01.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.193+0000 7f7827fff640 1 -- 172.21.15.55:0/2691989980 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78300b5bb0 con 0x7f78400ca310 2024-03-19T02:00:01.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.193+0000 7f7846024640 1 -- 172.21.15.55:0/2691989980 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7840155000 con 0x7f78400ca310 2024-03-19T02:00:01.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.193+0000 7f7846024640 1 -- 172.21.15.55:0/2691989980 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78401551d0 con 0x7f78400ca310 2024-03-19T02:00:01.217 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.217+0000 7f7827fff640 1 -- 172.21.15.55:0/2691989980 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f78300bd030 con 0x7f78400ca310 2024-03-19T02:00:01.217 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.217+0000 7f7827fff640 1 --2- 172.21.15.55:0/2691989980 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f782c0676b0 0x7f782c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:01.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.217+0000 7f783effd640 1 --2- 172.21.15.55:0/2691989980 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f782c0676b0 0x7f782c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:01.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.217+0000 7f783effd640 1 --2- 172.21.15.55:0/2691989980 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f782c0676b0 0x7f782c069b70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f784015cc00 tx=0x7f7834009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:01.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.221+0000 7f7827fff640 1 -- 172.21.15.55:0/2691989980 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f78300ee610 con 0x7f78400ca310 2024-03-19T02:00:01.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.581+0000 7f7827fff640 1 -- 172.21.15.55:0/2691989980 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f7830016070 con 0x7f78400ca310 2024-03-19T02:00:01.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.817+0000 7f7846024640 1 -- 172.21.15.55:0/2691989980 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7840068a40 con 0x7f78400ca310 2024-03-19T02:00:01.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.841+0000 7f7827fff640 1 -- 172.21.15.55:0/2691989980 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f78300f0570 con 0x7f78400ca310 2024-03-19T02:00:01.841 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:01.841 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:01.841 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:01.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.845+0000 7f7846024640 1 -- 172.21.15.55:0/2691989980 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f782c0676b0 msgr2=0x7f782c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:01.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.845+0000 7f7846024640 1 --2- 172.21.15.55:0/2691989980 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f782c0676b0 0x7f782c069b70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f784015cc00 tx=0x7f7834009190 comp rx=0 tx=0).stop 2024-03-19T02:00:01.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.845+0000 7f7846024640 1 -- 172.21.15.55:0/2691989980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78400ca310 msgr2=0x7f78401f3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:01.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.845+0000 7f7846024640 1 --2- 172.21.15.55:0/2691989980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78400ca310 0x7f78401f3880 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7840071390 tx=0x7f783001acc0 comp rx=0 tx=0).stop 2024-03-19T02:00:01.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.845+0000 7f7846024640 1 -- 172.21.15.55:0/2691989980 shutdown_connections 2024-03-19T02:00:01.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.845+0000 7f7846024640 1 --2- 172.21.15.55:0/2691989980 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f782c0676b0 0x7f782c069b70 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:01.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.845+0000 7f7846024640 1 --2- 172.21.15.55:0/2691989980 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f784015ad40 0x7f784014f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:01.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.845+0000 7f7846024640 1 --2- 172.21.15.55:0/2691989980 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7840154290 0x7f7840153880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:01.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.845+0000 7f7846024640 1 --2- 172.21.15.55:0/2691989980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78400ca310 0x7f78401f3880 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:01.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.845+0000 7f7846024640 1 -- 172.21.15.55:0/2691989980 >> 172.21.15.55:0/2691989980 conn(0x7f784006c580 msgr2=0x7f784006c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:01.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.845+0000 7f7846024640 1 -- 172.21.15.55:0/2691989980 shutdown_connections 2024-03-19T02:00:01.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:01.845+0000 7f7846024640 1 -- 172.21.15.55:0/2691989980 wait complete. 2024-03-19T02:00:02.465 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:02.465 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:03.466 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:03.577 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.577+0000 7f767f744640 1 Processor -- start 2024-03-19T02:00:03.577 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.577+0000 7f767f744640 1 -- start start 2024-03-19T02:00:03.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.577+0000 7f767f744640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f7678152120 0x7f76780c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:03.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.577+0000 7f767f744640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f76780c9610 0x7f7678157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:03.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.577+0000 7f767f744640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f767806fff0 con 0x7f7678070780 2024-03-19T02:00:03.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.577+0000 7f767f744640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f7678152500 con 0x7f76780c9610 2024-03-19T02:00:03.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.577+0000 7f767f744640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f7678070590 con 0x7f7678152120 2024-03-19T02:00:03.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.577+0000 7f767d4e0640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f7678152120 0x7f76780c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:03.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.577+0000 7f767ccdf640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f76780c9610 0x7f7678157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:03.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.581+0000 7f767ccdf640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f76780c9610 0x7f7678157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58676/0 (socket says 172.21.15.55:58676) 2024-03-19T02:00:03.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.581+0000 7f767ccdf640 1 -- 172.21.15.55:0/552215360 learned_addr learned my addr 172.21.15.55:0/552215360 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:03.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.581+0000 7f767d4e0640 1 -- 172.21.15.55:0/552215360 >> v2:172.21.15.55:3300/0 conn(0x7f76780c9610 msgr2=0x7f7678157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:03.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.581+0000 7f767d4e0640 1 --2- 172.21.15.55:0/552215360 >> v2:172.21.15.55:3300/0 conn(0x7f76780c9610 0x7f7678157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:03.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.585+0000 7f767d4e0640 1 -- 172.21.15.55:0/552215360 >> v1:172.21.15.55:6789/0 conn(0x7f7678070780 legacy=0x7f7678158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:03.584 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.585+0000 7f767d4e0640 1 -- 172.21.15.55:0/552215360 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76780c39c0 con 0x7f7678152120 2024-03-19T02:00:03.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.589+0000 7f767d4e0640 1 --2- 172.21.15.55:0/552215360 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7678152120 0x7f76780c90d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7678072430 tx=0x7f767401c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a497ff335bb00c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:03.591 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.593+0000 7f766ffff640 1 -- 172.21.15.55:0/552215360 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76740b0070 con 0x7f7678152120 2024-03-19T02:00:03.595 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.597+0000 7f766ffff640 1 -- 172.21.15.55:0/552215360 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76740ab600 con 0x7f7678152120 2024-03-19T02:00:03.595 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.597+0000 7f766ffff640 1 -- 172.21.15.55:0/552215360 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76740ab9d0 con 0x7f7678152120 2024-03-19T02:00:03.597 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.597+0000 7f767f744640 1 -- 172.21.15.55:0/552215360 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7678152120 msgr2=0x7f76780c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:03.597 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.597+0000 7f767f744640 1 --2- 172.21.15.55:0/552215360 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7678152120 0x7f76780c90d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7678072430 tx=0x7f767401c480 comp rx=0 tx=0).stop 2024-03-19T02:00:03.597 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.597+0000 7f767f744640 1 -- 172.21.15.55:0/552215360 shutdown_connections 2024-03-19T02:00:03.597 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.597+0000 7f767f744640 1 --2- 172.21.15.55:0/552215360 >> v2:172.21.15.55:3300/0 conn(0x7f76780c9610 0x7f7678157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:03.597 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.597+0000 7f767f744640 1 --2- 172.21.15.55:0/552215360 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7678152120 0x7f76780c90d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:03.597 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.597+0000 7f767f744640 1 -- 172.21.15.55:0/552215360 >> 172.21.15.55:0/552215360 conn(0x7f7678067600 msgr2=0x7f7678071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:03.597 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.597+0000 7f767f744640 1 -- 172.21.15.55:0/552215360 shutdown_connections 2024-03-19T02:00:03.597 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.597+0000 7f767f744640 1 -- 172.21.15.55:0/552215360 wait complete. 2024-03-19T02:00:03.598 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.597+0000 7f767f744640 1 Processor -- start 2024-03-19T02:00:03.598 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.597+0000 7f767f744640 1 -- start start 2024-03-19T02:00:03.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.601+0000 7f767f744640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7678070780 0x7f767815a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:03.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.601+0000 7f767f744640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f76780c9610 0x7f7678141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:03.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.601+0000 7f767f744640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7678152120 0x7f767815b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:03.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.601+0000 7f767f744640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f76780c86d0 con 0x7f7678152120 2024-03-19T02:00:03.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.601+0000 7f767f744640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f76780c31c0 con 0x7f76780c9610 2024-03-19T02:00:03.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.601+0000 7f767f744640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f7678165170 con 0x7f7678070780 2024-03-19T02:00:03.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.601+0000 7f767dce1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7678152120 0x7f767815b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:03.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.601+0000 7f767ccdf640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f76780c9610 0x7f7678141aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:03.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.601+0000 7f767d4e0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7678070780 0x7f767815a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:03.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.601+0000 7f767dce1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7678152120 0x7f767815b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58684/0 (socket says 172.21.15.55:58684) 2024-03-19T02:00:03.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.601+0000 7f767dce1640 1 -- 172.21.15.55:0/410120151 learned_addr learned my addr 172.21.15.55:0/410120151 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:03.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.605+0000 7f767dce1640 1 -- 172.21.15.55:0/410120151 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7678070780 msgr2=0x7f767815a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:03.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.605+0000 7f767dce1640 1 --2- 172.21.15.55:0/410120151 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7678070780 0x7f767815a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:03.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.605+0000 7f767dce1640 1 -- 172.21.15.55:0/410120151 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f76780c9610 msgr2=0x7f7678141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:03.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.605+0000 7f767dce1640 1 --2- 172.21.15.55:0/410120151 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f76780c9610 0x7f7678141aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:03.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.605+0000 7f767dce1640 1 -- 172.21.15.55:0/410120151 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f767815bce0 con 0x7f7678152120 2024-03-19T02:00:03.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.609+0000 7f767dce1640 1 --2- 172.21.15.55:0/410120151 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7678152120 0x7f767815b7a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f767000c7a0 tx=0x7f767000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:03.611 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.613+0000 7f766dffb640 1 -- 172.21.15.55:0/410120151 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7670003b80 con 0x7f7678152120 2024-03-19T02:00:03.611 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.613+0000 7f766dffb640 1 -- 172.21.15.55:0/410120151 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7670003d20 con 0x7f7678152120 2024-03-19T02:00:03.612 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.613+0000 7f767f744640 1 -- 172.21.15.55:0/410120151 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7678159230 con 0x7f7678152120 2024-03-19T02:00:03.612 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.613+0000 7f766dffb640 1 -- 172.21.15.55:0/410120151 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767000aa30 con 0x7f7678152120 2024-03-19T02:00:03.612 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.613+0000 7f767f744640 1 -- 172.21.15.55:0/410120151 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7678140fe0 con 0x7f7678152120 2024-03-19T02:00:03.612 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.613+0000 7f767f744640 1 -- 172.21.15.55:0/410120151 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7678068a40 con 0x7f7678152120 2024-03-19T02:00:03.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.633+0000 7f766dffb640 1 -- 172.21.15.55:0/410120151 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f767000abd0 con 0x7f7678152120 2024-03-19T02:00:03.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.633+0000 7f766dffb640 1 --2- 172.21.15.55:0/410120151 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f765c0676b0 0x7f765c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:03.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.633+0000 7f767d4e0640 1 --2- 172.21.15.55:0/410120151 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f765c0676b0 0x7f765c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:03.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.637+0000 7f767d4e0640 1 --2- 172.21.15.55:0/410120151 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f765c0676b0 0x7f765c069b70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f76740ad000 tx=0x7f7674008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:03.636 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.637+0000 7f766dffb640 1 -- 172.21.15.55:0/410120151 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f76700547f0 con 0x7f7678152120 2024-03-19T02:00:03.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:03.913+0000 7f766dffb640 1 -- 172.21.15.55:0/410120151 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f7670056750 con 0x7f7678152120 2024-03-19T02:00:04.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:04.157+0000 7f767f744640 1 -- 172.21.15.55:0/410120151 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7678070db0 con 0x7f7678152120 2024-03-19T02:00:04.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:04.173+0000 7f766dffb640 1 -- 172.21.15.55:0/410120151 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f767005a7f0 con 0x7f7678152120 2024-03-19T02:00:04.174 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:04.174 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:04.174 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:04.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:04.177+0000 7f767f744640 1 -- 172.21.15.55:0/410120151 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f765c0676b0 msgr2=0x7f765c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:04.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:04.177+0000 7f767f744640 1 --2- 172.21.15.55:0/410120151 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f765c0676b0 0x7f765c069b70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f76740ad000 tx=0x7f7674008040 comp rx=0 tx=0).stop 2024-03-19T02:00:04.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:04.177+0000 7f767f744640 1 -- 172.21.15.55:0/410120151 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7678152120 msgr2=0x7f767815b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:04.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:04.177+0000 7f767f744640 1 --2- 172.21.15.55:0/410120151 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7678152120 0x7f767815b7a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f767000c7a0 tx=0x7f767000cc60 comp rx=0 tx=0).stop 2024-03-19T02:00:04.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:04.177+0000 7f767f744640 1 -- 172.21.15.55:0/410120151 shutdown_connections 2024-03-19T02:00:04.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:04.177+0000 7f767f744640 1 --2- 172.21.15.55:0/410120151 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f765c0676b0 0x7f765c069b70 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:04.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:04.177+0000 7f767f744640 1 --2- 172.21.15.55:0/410120151 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7678152120 0x7f767815b7a0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:04.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:04.177+0000 7f767f744640 1 --2- 172.21.15.55:0/410120151 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f76780c9610 0x7f7678141aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:04.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:04.177+0000 7f767f744640 1 --2- 172.21.15.55:0/410120151 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7678070780 0x7f767815a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:04.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:04.177+0000 7f767f744640 1 -- 172.21.15.55:0/410120151 >> 172.21.15.55:0/410120151 conn(0x7f7678067600 msgr2=0x7f76780c8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:04.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:04.177+0000 7f767f744640 1 -- 172.21.15.55:0/410120151 shutdown_connections 2024-03-19T02:00:04.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:04.177+0000 7f767f744640 1 -- 172.21.15.55:0/410120151 wait complete. 2024-03-19T02:00:04.797 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:04.797 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:05.798 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:05.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.913+0000 7fbd4e1b8640 1 Processor -- start 2024-03-19T02:00:05.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.913+0000 7fbd4e1b8640 1 -- start start 2024-03-19T02:00:05.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.917+0000 7fbd4e1b8640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd480c96b0 0x7fbd48070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:05.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.917+0000 7fbd4e1b8640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fbd480c9a90 0x7fbd480c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:05.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.917+0000 7fbd4e1b8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fbd481540c0 0x7fbd48154570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:05.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.917+0000 7fbd4e1b8640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fbd4815e3a0 con 0x7fbd481540c0 2024-03-19T02:00:05.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.917+0000 7fbd4e1b8640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fbd4815e680 con 0x7fbd480c9a90 2024-03-19T02:00:05.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.917+0000 7fbd4e1b8640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbd4815a330 con 0x7fbd480c96b0 2024-03-19T02:00:05.916 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.917+0000 7fbd46ffd640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fbd480c9a90 0x7fbd480c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:05.916 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.917+0000 7fbd47fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fbd481540c0 0x7fbd48154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:05.916 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.917+0000 7fbd477fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd480c96b0 0x7fbd48070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:05.917 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.917+0000 7fbd47fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fbd481540c0 0x7fbd48154570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58698/0 (socket says 172.21.15.55:58698) 2024-03-19T02:00:05.917 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.917+0000 7fbd47fff640 1 -- 172.21.15.55:0/2360483668 learned_addr learned my addr 172.21.15.55:0/2360483668 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:05.920 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.921+0000 7fbd47fff640 1 -- 172.21.15.55:0/2360483668 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd480c96b0 msgr2=0x7fbd48070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:05.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.921+0000 7fbd47fff640 1 --2- 172.21.15.55:0/2360483668 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd480c96b0 0x7fbd48070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:05.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.921+0000 7fbd47fff640 1 -- 172.21.15.55:0/2360483668 >> v2:172.21.15.84:3300/0 conn(0x7fbd480c9a90 msgr2=0x7fbd480c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:05.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.921+0000 7fbd47fff640 1 --2- 172.21.15.55:0/2360483668 >> v2:172.21.15.84:3300/0 conn(0x7fbd480c9a90 0x7fbd480c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:05.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.921+0000 7fbd47fff640 1 -- 172.21.15.55:0/2360483668 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd4815a610 con 0x7fbd481540c0 2024-03-19T02:00:05.924 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.925+0000 7fbd47fff640 1 --2- 172.21.15.55:0/2360483668 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd481540c0 0x7fbd48154570 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fbd48071e50 tx=0x7fbd4001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab86aaaf7e4a314 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:05.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.929+0000 7fbd467fc640 1 -- 172.21.15.55:0/2360483668 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd400a56d0 con 0x7fbd481540c0 2024-03-19T02:00:05.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.929+0000 7fbd467fc640 1 -- 172.21.15.55:0/2360483668 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd400a5870 con 0x7fbd481540c0 2024-03-19T02:00:05.930 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.929+0000 7fbd467fc640 1 -- 172.21.15.55:0/2360483668 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd400a5c40 con 0x7fbd481540c0 2024-03-19T02:00:05.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.929+0000 7fbd4e1b8640 1 -- 172.21.15.55:0/2360483668 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd481540c0 msgr2=0x7fbd48154570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:05.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.929+0000 7fbd4e1b8640 1 --2- 172.21.15.55:0/2360483668 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd481540c0 0x7fbd48154570 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fbd48071e50 tx=0x7fbd4001c960 comp rx=0 tx=0).stop 2024-03-19T02:00:05.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.933+0000 7fbd4e1b8640 1 -- 172.21.15.55:0/2360483668 shutdown_connections 2024-03-19T02:00:05.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.933+0000 7fbd4e1b8640 1 --2- 172.21.15.55:0/2360483668 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd481540c0 0x7fbd48154570 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:05.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.933+0000 7fbd4e1b8640 1 --2- 172.21.15.55:0/2360483668 >> v2:172.21.15.84:3300/0 conn(0x7fbd480c9a90 0x7fbd480c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:05.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.933+0000 7fbd4e1b8640 1 --2- 172.21.15.55:0/2360483668 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd480c96b0 0x7fbd48070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:05.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.933+0000 7fbd4e1b8640 1 -- 172.21.15.55:0/2360483668 >> 172.21.15.55:0/2360483668 conn(0x7fbd480cc9c0 msgr2=0x7fbd480ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:05.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.933+0000 7fbd4e1b8640 1 -- 172.21.15.55:0/2360483668 shutdown_connections 2024-03-19T02:00:05.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.933+0000 7fbd4e1b8640 1 -- 172.21.15.55:0/2360483668 wait complete. 2024-03-19T02:00:05.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.933+0000 7fbd4e1b8640 1 Processor -- start 2024-03-19T02:00:05.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.933+0000 7fbd4e1b8640 1 -- start start 2024-03-19T02:00:05.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.933+0000 7fbd4e1b8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd480c96b0 0x7fbd481f6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:05.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.933+0000 7fbd4e1b8640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd480c9a90 0x7fbd481432f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:05.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.933+0000 7fbd4e1b8640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd481540c0 0x7fbd481401a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:05.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.933+0000 7fbd4e1b8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbd480c7a80 con 0x7fbd480c96b0 2024-03-19T02:00:05.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.933+0000 7fbd4e1b8640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fbd480c7900 con 0x7fbd480c9a90 2024-03-19T02:00:05.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.933+0000 7fbd4e1b8640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbd4806b2a0 con 0x7fbd481540c0 2024-03-19T02:00:05.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.933+0000 7fbd477fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd480c96b0 0x7fbd481f6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:05.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.937+0000 7fbd46ffd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd480c9a90 0x7fbd481432f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:05.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.937+0000 7fbd47fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd481540c0 0x7fbd481401a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:05.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.937+0000 7fbd46ffd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd480c9a90 0x7fbd481432f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:35424/0 (socket says 172.21.15.55:35424) 2024-03-19T02:00:05.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.937+0000 7fbd46ffd640 1 -- 172.21.15.55:0/442417269 learned_addr learned my addr 172.21.15.55:0/442417269 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:05.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.941+0000 7fbd477fe640 1 -- 172.21.15.55:0/442417269 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd481540c0 msgr2=0x7fbd481401a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:05.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.941+0000 7fbd477fe640 1 --2- 172.21.15.55:0/442417269 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd481540c0 0x7fbd481401a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:05.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.941+0000 7fbd477fe640 1 -- 172.21.15.55:0/442417269 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd480c9a90 msgr2=0x7fbd481432f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:05.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.941+0000 7fbd477fe640 1 --2- 172.21.15.55:0/442417269 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd480c9a90 0x7fbd481432f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:05.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.941+0000 7fbd477fe640 1 -- 172.21.15.55:0/442417269 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd480cc7a0 con 0x7fbd480c96b0 2024-03-19T02:00:05.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.941+0000 7fbd477fe640 1 --2- 172.21.15.55:0/442417269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd480c96b0 0x7fbd481f6270 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fbd3800e870 tx=0x7fbd3800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:05.945 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.945+0000 7fbd2ffff640 1 -- 172.21.15.55:0/442417269 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd38017070 con 0x7fbd480c96b0 2024-03-19T02:00:05.945 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.945+0000 7fbd4e1b8640 1 -- 172.21.15.55:0/442417269 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd4814a150 con 0x7fbd480c96b0 2024-03-19T02:00:05.945 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.945+0000 7fbd4e1b8640 1 -- 172.21.15.55:0/442417269 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd480cbf00 con 0x7fbd480c96b0 2024-03-19T02:00:05.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.945+0000 7fbd4e1b8640 1 -- 172.21.15.55:0/442417269 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd48068a40 con 0x7fbd480c96b0 2024-03-19T02:00:05.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.949+0000 7fbd2ffff640 1 -- 172.21.15.55:0/442417269 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd380059a0 con 0x7fbd480c96b0 2024-03-19T02:00:05.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:05.949+0000 7fbd2ffff640 1 -- 172.21.15.55:0/442417269 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd38021060 con 0x7fbd480c96b0 2024-03-19T02:00:06.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:06.253+0000 7fbd2ffff640 1 -- 172.21.15.55:0/442417269 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fbd3801a070 con 0x7fbd480c96b0 2024-03-19T02:00:06.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:06.253+0000 7fbd2ffff640 1 --2- 172.21.15.55:0/442417269 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbd34067660 0x7fbd34069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:06.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:06.253+0000 7fbd46ffd640 1 --2- 172.21.15.55:0/442417269 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbd34067660 0x7fbd34069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:06.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:06.253+0000 7fbd46ffd640 1 --2- 172.21.15.55:0/442417269 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbd34067660 0x7fbd34069b20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbd48144590 tx=0x7fbd3c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:06.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:06.273+0000 7fbd2ffff640 1 -- 172.21.15.55:0/442417269 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fbd380546f0 con 0x7fbd480c96b0 2024-03-19T02:00:06.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:06.273+0000 7fbd2ffff640 1 -- 172.21.15.55:0/442417269 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fbd38056650 con 0x7fbd480c96b0 2024-03-19T02:00:06.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:06.477+0000 7fbd4e1b8640 1 -- 172.21.15.55:0/442417269 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd480cc1e0 con 0x7fbd480c96b0 2024-03-19T02:00:06.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:06.493+0000 7fbd2ffff640 1 -- 172.21.15.55:0/442417269 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fbd3805a6f0 con 0x7fbd480c96b0 2024-03-19T02:00:06.493 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:06.493 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:06.494 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:06.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:06.497+0000 7fbd4e1b8640 1 -- 172.21.15.55:0/442417269 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbd34067660 msgr2=0x7fbd34069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:06.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:06.497+0000 7fbd4e1b8640 1 --2- 172.21.15.55:0/442417269 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbd34067660 0x7fbd34069b20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbd48144590 tx=0x7fbd3c009190 comp rx=0 tx=0).stop 2024-03-19T02:00:06.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:06.497+0000 7fbd4e1b8640 1 -- 172.21.15.55:0/442417269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd480c96b0 msgr2=0x7fbd481f6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:06.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:06.497+0000 7fbd4e1b8640 1 --2- 172.21.15.55:0/442417269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd480c96b0 0x7fbd481f6270 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fbd3800e870 tx=0x7fbd3800ed30 comp rx=0 tx=0).stop 2024-03-19T02:00:06.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:06.497+0000 7fbd4e1b8640 1 -- 172.21.15.55:0/442417269 shutdown_connections 2024-03-19T02:00:06.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:06.497+0000 7fbd4e1b8640 1 --2- 172.21.15.55:0/442417269 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbd34067660 0x7fbd34069b20 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:06.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:06.497+0000 7fbd4e1b8640 1 --2- 172.21.15.55:0/442417269 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbd481540c0 0x7fbd481401a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:06.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:06.497+0000 7fbd4e1b8640 1 --2- 172.21.15.55:0/442417269 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd480c9a90 0x7fbd481432f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:06.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:06.497+0000 7fbd4e1b8640 1 --2- 172.21.15.55:0/442417269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd480c96b0 0x7fbd481f6270 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:06.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:06.497+0000 7fbd4e1b8640 1 -- 172.21.15.55:0/442417269 >> 172.21.15.55:0/442417269 conn(0x7fbd480cc9c0 msgr2=0x7fbd480c3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:06.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:06.497+0000 7fbd4e1b8640 1 -- 172.21.15.55:0/442417269 shutdown_connections 2024-03-19T02:00:06.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:06.497+0000 7fbd4e1b8640 1 -- 172.21.15.55:0/442417269 wait complete. 2024-03-19T02:00:07.116 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:07.117 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:08.118 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:08.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.237+0000 7fafac4c4640 1 Processor -- start 2024-03-19T02:00:08.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.237+0000 7fafac4c4640 1 -- start start 2024-03-19T02:00:08.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.237+0000 7fafac4c4640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fafa4152120 0x7fafa40c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:08.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.237+0000 7fafac4c4640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fafa4070780 0x7fafa41586e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:08.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.237+0000 7fafac4c4640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafa406fff0 con 0x7fafa40c9610 2024-03-19T02:00:08.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.237+0000 7fafac4c4640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fafa4152500 con 0x7fafa4152120 2024-03-19T02:00:08.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.237+0000 7fafac4c4640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fafa4070590 con 0x7fafa4070780 2024-03-19T02:00:08.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.241+0000 7fafaa260640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fafa4152120 0x7fafa40c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:08.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.241+0000 7fafaaa61640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fafa4070780 0x7fafa41586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:08.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.241+0000 7fafaa260640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fafa4152120 0x7fafa40c90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58722/0 (socket says 172.21.15.55:58722) 2024-03-19T02:00:08.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.241+0000 7fafaa260640 1 -- 172.21.15.55:0/2196163965 learned_addr learned my addr 172.21.15.55:0/2196163965 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:08.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.241+0000 7fafaa260640 1 -- 172.21.15.55:0/2196163965 >> v2:172.21.15.84:3300/0 conn(0x7fafa4070780 msgr2=0x7fafa41586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:08.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.241+0000 7fafaa260640 1 --2- 172.21.15.55:0/2196163965 >> v2:172.21.15.84:3300/0 conn(0x7fafa4070780 0x7fafa41586e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:08.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.241+0000 7fafaa260640 1 -- 172.21.15.55:0/2196163965 >> v1:172.21.15.84:6789/0 conn(0x7fafa40c9610 legacy=0x7fafa4157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:08.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.241+0000 7fafaa260640 1 -- 172.21.15.55:0/2196163965 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa40c39c0 con 0x7fafa4152120 2024-03-19T02:00:08.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.245+0000 7fafaa260640 1 --2- 172.21.15.55:0/2196163965 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fafa4152120 0x7fafa40c90d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fafa40cb940 tx=0x7fafa001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd972296649b08c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:08.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.253+0000 7fafa925e640 1 -- 172.21.15.55:0/2196163965 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa00b0070 con 0x7fafa4152120 2024-03-19T02:00:08.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.253+0000 7fafa925e640 1 -- 172.21.15.55:0/2196163965 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafa00ab600 con 0x7fafa4152120 2024-03-19T02:00:08.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.253+0000 7fafa925e640 1 -- 172.21.15.55:0/2196163965 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa00ab9d0 con 0x7fafa4152120 2024-03-19T02:00:08.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.253+0000 7fafac4c4640 1 -- 172.21.15.55:0/2196163965 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fafa4152120 msgr2=0x7fafa40c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:08.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.253+0000 7fafac4c4640 1 --2- 172.21.15.55:0/2196163965 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fafa4152120 0x7fafa40c90d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fafa40cb940 tx=0x7fafa001c520 comp rx=0 tx=0).stop 2024-03-19T02:00:08.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.253+0000 7fafac4c4640 1 -- 172.21.15.55:0/2196163965 shutdown_connections 2024-03-19T02:00:08.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.253+0000 7fafac4c4640 1 --2- 172.21.15.55:0/2196163965 >> v2:172.21.15.84:3300/0 conn(0x7fafa4070780 0x7fafa41586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:08.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.253+0000 7fafac4c4640 1 --2- 172.21.15.55:0/2196163965 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fafa4152120 0x7fafa40c90d0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:08.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.253+0000 7fafac4c4640 1 -- 172.21.15.55:0/2196163965 >> 172.21.15.55:0/2196163965 conn(0x7fafa4067600 msgr2=0x7fafa4071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:08.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.253+0000 7fafac4c4640 1 -- 172.21.15.55:0/2196163965 shutdown_connections 2024-03-19T02:00:08.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.253+0000 7fafac4c4640 1 -- 172.21.15.55:0/2196163965 wait complete. 2024-03-19T02:00:08.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.253+0000 7fafac4c4640 1 Processor -- start 2024-03-19T02:00:08.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.253+0000 7fafac4c4640 1 -- start start 2024-03-19T02:00:08.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.253+0000 7fafac4c4640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafa4070780 0x7fafa40ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:08.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.253+0000 7fafac4c4640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fafa40c9610 0x7fafa40b2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:08.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.253+0000 7fafac4c4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fafa4152120 0x7fafa40c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:08.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.253+0000 7fafac4c4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fafa40c86d0 con 0x7fafa4152120 2024-03-19T02:00:08.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.253+0000 7fafac4c4640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fafa40c31c0 con 0x7fafa4070780 2024-03-19T02:00:08.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.253+0000 7fafac4c4640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fafa4165170 con 0x7fafa40c9610 2024-03-19T02:00:08.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.253+0000 7fafaaa61640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fafa4152120 0x7fafa40c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:08.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.253+0000 7fafaa260640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafa4070780 0x7fafa40ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:08.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.257+0000 7fafa9a5f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fafa40c9610 0x7fafa40b2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:08.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.257+0000 7fafaaa61640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fafa4152120 0x7fafa40c5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58724/0 (socket says 172.21.15.55:58724) 2024-03-19T02:00:08.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.257+0000 7fafaaa61640 1 -- 172.21.15.55:0/3425754929 learned_addr learned my addr 172.21.15.55:0/3425754929 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:08.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.261+0000 7fafaaa61640 1 -- 172.21.15.55:0/3425754929 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fafa40c9610 msgr2=0x7fafa40b2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:08.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.261+0000 7fafaaa61640 1 --2- 172.21.15.55:0/3425754929 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fafa40c9610 0x7fafa40b2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:08.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.261+0000 7fafaaa61640 1 -- 172.21.15.55:0/3425754929 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafa4070780 msgr2=0x7fafa40ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:08.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.261+0000 7fafaaa61640 1 --2- 172.21.15.55:0/3425754929 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafa4070780 0x7fafa40ccd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:08.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.261+0000 7fafaaa61640 1 -- 172.21.15.55:0/3425754929 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa40cd790 con 0x7fafa4152120 2024-03-19T02:00:08.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.265+0000 7fafaaa61640 1 --2- 172.21.15.55:0/3425754929 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fafa4152120 0x7fafa40c5a70 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7faf9c009e30 tx=0x7faf9c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:08.268 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.269+0000 7faf9affd640 1 -- 172.21.15.55:0/3425754929 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9c018020 con 0x7fafa4152120 2024-03-19T02:00:08.268 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.269+0000 7faf9affd640 1 -- 172.21.15.55:0/3425754929 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf9c014480 con 0x7fafa4152120 2024-03-19T02:00:08.268 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.269+0000 7fafac4c4640 1 -- 172.21.15.55:0/3425754929 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafa40b14d0 con 0x7fafa4152120 2024-03-19T02:00:08.268 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.269+0000 7faf9affd640 1 -- 172.21.15.55:0/3425754929 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9c014870 con 0x7fafa4152120 2024-03-19T02:00:08.268 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.269+0000 7fafac4c4640 1 -- 172.21.15.55:0/3425754929 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafa40b1a90 con 0x7fafa4152120 2024-03-19T02:00:08.269 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.269+0000 7fafac4c4640 1 -- 172.21.15.55:0/3425754929 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafa4068a40 con 0x7fafa4152120 2024-03-19T02:00:08.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.301+0000 7faf9affd640 1 -- 172.21.15.55:0/3425754929 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7faf9c01b020 con 0x7fafa4152120 2024-03-19T02:00:08.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.301+0000 7faf9affd640 1 --2- 172.21.15.55:0/3425754929 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7faf88067660 0x7faf88069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:08.303 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.305+0000 7fafaa260640 1 --2- 172.21.15.55:0/3425754929 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7faf88067660 0x7faf88069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:08.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.305+0000 7fafaa260640 1 --2- 172.21.15.55:0/3425754929 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7faf88067660 0x7faf88069b20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fafa00ad000 tx=0x7fafa00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:08.594 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.593+0000 7faf9affd640 1 -- 172.21.15.55:0/3425754929 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7faf9c0552b0 con 0x7fafa4152120 2024-03-19T02:00:08.594 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.593+0000 7faf9affd640 1 -- 172.21.15.55:0/3425754929 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7faf9c022030 con 0x7fafa4152120 2024-03-19T02:00:08.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.809+0000 7fafac4c4640 1 -- 172.21.15.55:0/3425754929 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafa40c84b0 con 0x7fafa4152120 2024-03-19T02:00:08.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.893+0000 7faf9affd640 1 -- 172.21.15.55:0/3425754929 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7faf9c010040 con 0x7fafa4152120 2024-03-19T02:00:08.894 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:08.894 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:08.894 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:08.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.897+0000 7fafac4c4640 1 -- 172.21.15.55:0/3425754929 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7faf88067660 msgr2=0x7faf88069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:08.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.897+0000 7fafac4c4640 1 --2- 172.21.15.55:0/3425754929 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7faf88067660 0x7faf88069b20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fafa00ad000 tx=0x7fafa00a6000 comp rx=0 tx=0).stop 2024-03-19T02:00:08.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.897+0000 7fafac4c4640 1 -- 172.21.15.55:0/3425754929 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fafa4152120 msgr2=0x7fafa40c5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:08.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.897+0000 7fafac4c4640 1 --2- 172.21.15.55:0/3425754929 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fafa4152120 0x7fafa40c5a70 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7faf9c009e30 tx=0x7faf9c00c490 comp rx=0 tx=0).stop 2024-03-19T02:00:08.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.897+0000 7fafac4c4640 1 -- 172.21.15.55:0/3425754929 shutdown_connections 2024-03-19T02:00:08.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.897+0000 7fafac4c4640 1 --2- 172.21.15.55:0/3425754929 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7faf88067660 0x7faf88069b20 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:08.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.897+0000 7fafac4c4640 1 --2- 172.21.15.55:0/3425754929 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fafa4152120 0x7fafa40c5a70 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:08.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.897+0000 7fafac4c4640 1 --2- 172.21.15.55:0/3425754929 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fafa40c9610 0x7fafa40b2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:08.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.897+0000 7fafac4c4640 1 --2- 172.21.15.55:0/3425754929 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafa4070780 0x7fafa40ccd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:08.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.897+0000 7fafac4c4640 1 -- 172.21.15.55:0/3425754929 >> 172.21.15.55:0/3425754929 conn(0x7fafa4067600 msgr2=0x7fafa40c8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:08.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.897+0000 7fafac4c4640 1 -- 172.21.15.55:0/3425754929 shutdown_connections 2024-03-19T02:00:08.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:08.897+0000 7fafac4c4640 1 -- 172.21.15.55:0/3425754929 wait complete. 2024-03-19T02:00:09.521 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:09.521 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:10.071 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:00:10.073 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:00:10.522 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:10.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.633+0000 7f88d5bdb640 1 Processor -- start 2024-03-19T02:00:10.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.633+0000 7f88d5bdb640 1 -- start start 2024-03-19T02:00:10.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.637+0000 7f88d5bdb640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f88d0158d00 0x7f88d00c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:10.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.637+0000 7f88d5bdb640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88d014ee80 con 0x7f88d014eaa0 2024-03-19T02:00:10.636 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.637+0000 7f88d5bdb640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88d00c7540 con 0x7f88d00c60d0 2024-03-19T02:00:10.636 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.637+0000 7f88d5bdb640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f88d00c77c0 con 0x7f88d0158d00 2024-03-19T02:00:10.636 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.637+0000 7f88cffff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f88d0158d00 0x7f88d00c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:10.636 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.637+0000 7f88ceffd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f88d00c60d0 0x7f88d01585f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:60410/0 (socket says 172.21.15.55:60410) 2024-03-19T02:00:10.636 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.637+0000 7f88ceffd640 1 -- 172.21.15.55:0/2193101619 learned_addr learned my addr 172.21.15.55:0/2193101619 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:00:10.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.641+0000 7f88cffff640 1 -- 172.21.15.55:0/2193101619 >> v1:172.21.15.84:6789/0 conn(0x7f88d00c60d0 legacy=0x7f88d01585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:10.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.641+0000 7f88cffff640 1 -- 172.21.15.55:0/2193101619 >> v1:172.21.15.55:6789/0 conn(0x7f88d014eaa0 legacy=0x7f88d006b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:10.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.641+0000 7f88cffff640 1 -- 172.21.15.55:0/2193101619 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d00c7aa0 con 0x7f88d0158d00 2024-03-19T02:00:10.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.645+0000 7f88cffff640 1 --2- 172.21.15.55:0/2193101619 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88d0158d00 0x7f88d00c6540 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f88c8009d90 tx=0x7f88c801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a36bafd5581d8763 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:10.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.649+0000 7f88ce7fc640 1 -- 172.21.15.55:0/2193101619 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c80b2070 con 0x7f88d0158d00 2024-03-19T02:00:10.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.649+0000 7f88ce7fc640 1 -- 172.21.15.55:0/2193101619 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88c80a59a0 con 0x7f88d0158d00 2024-03-19T02:00:10.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.649+0000 7f88ce7fc640 1 -- 172.21.15.55:0/2193101619 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c80a5d70 con 0x7f88d0158d00 2024-03-19T02:00:10.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.653+0000 7f88d5bdb640 1 -- 172.21.15.55:0/2193101619 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88d0158d00 msgr2=0x7f88d00c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:10.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.653+0000 7f88d5bdb640 1 --2- 172.21.15.55:0/2193101619 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88d0158d00 0x7f88d00c6540 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f88c8009d90 tx=0x7f88c801c8e0 comp rx=0 tx=0).stop 2024-03-19T02:00:10.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.653+0000 7f88d5bdb640 1 -- 172.21.15.55:0/2193101619 shutdown_connections 2024-03-19T02:00:10.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.653+0000 7f88d5bdb640 1 --2- 172.21.15.55:0/2193101619 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88d0158d00 0x7f88d00c6540 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:10.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.653+0000 7f88d5bdb640 1 -- 172.21.15.55:0/2193101619 >> 172.21.15.55:0/2193101619 conn(0x7f88d00cc9c0 msgr2=0x7f88d00ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:10.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.653+0000 7f88d5bdb640 1 -- 172.21.15.55:0/2193101619 shutdown_connections 2024-03-19T02:00:10.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.653+0000 7f88d5bdb640 1 -- 172.21.15.55:0/2193101619 wait complete. 2024-03-19T02:00:10.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.653+0000 7f88d5bdb640 1 Processor -- start 2024-03-19T02:00:10.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.653+0000 7f88d5bdb640 1 -- start start 2024-03-19T02:00:10.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.653+0000 7f88d5bdb640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f88d00c60d0 0x7f88d0163340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:10.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.653+0000 7f88d5bdb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88d014eaa0 0x7f88d0143d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:10.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.653+0000 7f88d5bdb640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f88d0155280 0x7f88d00caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:10.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.653+0000 7f88d5bdb640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f88d006c210 con 0x7f88d014eaa0 2024-03-19T02:00:10.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.653+0000 7f88d5bdb640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f88d006c090 con 0x7f88d0155280 2024-03-19T02:00:10.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.653+0000 7f88d5bdb640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f88d00c6bf0 con 0x7f88d00c60d0 2024-03-19T02:00:10.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.653+0000 7f88ceffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88d014eaa0 0x7f88d0143d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:10.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.653+0000 7f88cf7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f88d00c60d0 0x7f88d0163340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:10.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.653+0000 7f88ceffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88d014eaa0 0x7f88d0143d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56126/0 (socket says 172.21.15.55:56126) 2024-03-19T02:00:10.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.653+0000 7f88ceffd640 1 -- 172.21.15.55:0/1878813657 learned_addr learned my addr 172.21.15.55:0/1878813657 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:10.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.657+0000 7f88cffff640 1 --2- 172.21.15.55:0/1878813657 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f88d0155280 0x7f88d00caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:10.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.657+0000 7f88ceffd640 1 -- 172.21.15.55:0/1878813657 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f88d00c60d0 msgr2=0x7f88d0163340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:10.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.657+0000 7f88ceffd640 1 --2- 172.21.15.55:0/1878813657 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f88d00c60d0 0x7f88d0163340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:10.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.657+0000 7f88ceffd640 1 -- 172.21.15.55:0/1878813657 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f88d0155280 msgr2=0x7f88d00caf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:10.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.657+0000 7f88ceffd640 1 --2- 172.21.15.55:0/1878813657 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f88d0155280 0x7f88d00caf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:10.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.657+0000 7f88ceffd640 1 -- 172.21.15.55:0/1878813657 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d0155010 con 0x7f88d014eaa0 2024-03-19T02:00:10.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.661+0000 7f88ceffd640 1 --2- 172.21.15.55:0/1878813657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88d014eaa0 0x7f88d0143d20 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f88c400c400 tx=0x7f88c400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:10.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.665+0000 7f88b7fff640 1 -- 172.21.15.55:0/1878813657 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c4017070 con 0x7f88d014eaa0 2024-03-19T02:00:10.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.665+0000 7f88b7fff640 1 -- 172.21.15.55:0/1878813657 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88c4005cf0 con 0x7f88d014eaa0 2024-03-19T02:00:10.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.665+0000 7f88d5bdb640 1 -- 172.21.15.55:0/1878813657 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88d00c4c10 con 0x7f88d014eaa0 2024-03-19T02:00:10.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.665+0000 7f88b7fff640 1 -- 172.21.15.55:0/1878813657 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c40072d0 con 0x7f88d014eaa0 2024-03-19T02:00:10.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.665+0000 7f88d5bdb640 1 -- 172.21.15.55:0/1878813657 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88d01627f0 con 0x7f88d014eaa0 2024-03-19T02:00:10.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.665+0000 7f88d5bdb640 1 -- 172.21.15.55:0/1878813657 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88d0068a40 con 0x7f88d014eaa0 2024-03-19T02:00:10.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.697+0000 7f88b7fff640 1 -- 172.21.15.55:0/1878813657 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f88c401a070 con 0x7f88d014eaa0 2024-03-19T02:00:10.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.697+0000 7f88b7fff640 1 --2- 172.21.15.55:0/1878813657 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f88bc0676e0 0x7f88bc069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:10.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.701+0000 7f88cf7fe640 1 --2- 172.21.15.55:0/1878813657 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f88bc0676e0 0x7f88bc069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:10.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.701+0000 7f88cf7fe640 1 --2- 172.21.15.55:0/1878813657 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f88bc0676e0 0x7f88bc069ba0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f88c0001770 tx=0x7f88c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:10.979 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.981+0000 7f88b7fff640 1 -- 172.21.15.55:0/1878813657 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f88c4053720 con 0x7f88d014eaa0 2024-03-19T02:00:10.980 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:10.981+0000 7f88b7fff640 1 -- 172.21.15.55:0/1878813657 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f88d00c7540 con 0x7f88d014eaa0 2024-03-19T02:00:11.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:11.189+0000 7f88d5bdb640 1 -- 172.21.15.55:0/1878813657 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88d01558e0 con 0x7f88d014eaa0 2024-03-19T02:00:11.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:11.217+0000 7f88b7fff640 1 -- 172.21.15.55:0/1878813657 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f88c4055680 con 0x7f88d014eaa0 2024-03-19T02:00:11.219 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:11.219 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:11.220 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:11.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:11.221+0000 7f88d5bdb640 1 -- 172.21.15.55:0/1878813657 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f88bc0676e0 msgr2=0x7f88bc069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:11.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:11.221+0000 7f88d5bdb640 1 --2- 172.21.15.55:0/1878813657 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f88bc0676e0 0x7f88bc069ba0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f88c0001770 tx=0x7f88c0009040 comp rx=0 tx=0).stop 2024-03-19T02:00:11.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:11.221+0000 7f88d5bdb640 1 -- 172.21.15.55:0/1878813657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88d014eaa0 msgr2=0x7f88d0143d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:11.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:11.221+0000 7f88d5bdb640 1 --2- 172.21.15.55:0/1878813657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88d014eaa0 0x7f88d0143d20 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f88c400c400 tx=0x7f88c400c8c0 comp rx=0 tx=0).stop 2024-03-19T02:00:11.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:11.221+0000 7f88d5bdb640 1 -- 172.21.15.55:0/1878813657 shutdown_connections 2024-03-19T02:00:11.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:11.221+0000 7f88d5bdb640 1 --2- 172.21.15.55:0/1878813657 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f88bc0676e0 0x7f88bc069ba0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:11.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:11.221+0000 7f88d5bdb640 1 --2- 172.21.15.55:0/1878813657 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f88d0155280 0x7f88d00caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:11.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:11.221+0000 7f88d5bdb640 1 --2- 172.21.15.55:0/1878813657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88d014eaa0 0x7f88d0143d20 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:11.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:11.221+0000 7f88d5bdb640 1 --2- 172.21.15.55:0/1878813657 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f88d00c60d0 0x7f88d0163340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:11.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:11.221+0000 7f88d5bdb640 1 -- 172.21.15.55:0/1878813657 >> 172.21.15.55:0/1878813657 conn(0x7f88d00cc9c0 msgr2=0x7f88d0156730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:11.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:11.221+0000 7f88d5bdb640 1 -- 172.21.15.55:0/1878813657 shutdown_connections 2024-03-19T02:00:11.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:11.221+0000 7f88d5bdb640 1 -- 172.21.15.55:0/1878813657 wait complete. 2024-03-19T02:00:11.841 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:11.842 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:12.843 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:12.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.957+0000 7f3936725640 1 Processor -- start 2024-03-19T02:00:12.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.957+0000 7f3936725640 1 -- start start 2024-03-19T02:00:12.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.957+0000 7f3936725640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f39300cd470 0x7f393014e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:12.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.957+0000 7f3936725640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f39300c8010 0x7f393014ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:12.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.957+0000 7f3936725640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3930158330 con 0x7f39300cd090 2024-03-19T02:00:12.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.957+0000 7f3936725640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f3930158e90 con 0x7f39300cd470 2024-03-19T02:00:12.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.957+0000 7f3936725640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f39301585f0 con 0x7f39300c8010 2024-03-19T02:00:12.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.961+0000 7f392f7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f39300cd470 0x7f393014e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:12.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.961+0000 7f392ffff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f39300cd090 0x7f39300732b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:34288/0 (socket says 172.21.15.55:34288) 2024-03-19T02:00:12.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.961+0000 7f3934cc2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f39300c8010 0x7f393014ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:12.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.961+0000 7f392ffff640 1 -- 172.21.15.55:0/1067025458 learned_addr learned my addr 172.21.15.55:0/1067025458 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:00:12.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.965+0000 7f392f7fe640 1 -- 172.21.15.55:0/1067025458 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f39300c8010 msgr2=0x7f393014ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:12.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.965+0000 7f392f7fe640 1 --2- 172.21.15.55:0/1067025458 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f39300c8010 0x7f393014ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:12.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.965+0000 7f392f7fe640 1 -- 172.21.15.55:0/1067025458 >> v1:172.21.15.55:6789/0 conn(0x7f39300cd090 legacy=0x7f39300732b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:12.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.965+0000 7f392f7fe640 1 -- 172.21.15.55:0/1067025458 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39301588d0 con 0x7f39300cd470 2024-03-19T02:00:12.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.969+0000 7f392f7fe640 1 --2- 172.21.15.55:0/1067025458 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39300cd470 0x7f393014e550 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f39240097c0 tx=0x7f392401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eda304531c597ee2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:12.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.973+0000 7f392effd640 1 -- 172.21.15.55:0/1067025458 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39240b0070 con 0x7f39300cd470 2024-03-19T02:00:12.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.973+0000 7f392effd640 1 -- 172.21.15.55:0/1067025458 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39240ab600 con 0x7f39300cd470 2024-03-19T02:00:12.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.973+0000 7f392effd640 1 -- 172.21.15.55:0/1067025458 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39240ab9d0 con 0x7f39300cd470 2024-03-19T02:00:12.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.973+0000 7f3936725640 1 -- 172.21.15.55:0/1067025458 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39300cd470 msgr2=0x7f393014e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:12.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.973+0000 7f3936725640 1 --2- 172.21.15.55:0/1067025458 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39300cd470 0x7f393014e550 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f39240097c0 tx=0x7f392401c4a0 comp rx=0 tx=0).stop 2024-03-19T02:00:12.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.973+0000 7f3936725640 1 -- 172.21.15.55:0/1067025458 shutdown_connections 2024-03-19T02:00:12.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.973+0000 7f3936725640 1 --2- 172.21.15.55:0/1067025458 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f39300c8010 0x7f393014ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:12.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.973+0000 7f3936725640 1 --2- 172.21.15.55:0/1067025458 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39300cd470 0x7f393014e550 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:12.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.973+0000 7f3936725640 1 -- 172.21.15.55:0/1067025458 >> 172.21.15.55:0/1067025458 conn(0x7f393006e670 msgr2=0x7f393006ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:12.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.973+0000 7f3936725640 1 -- 172.21.15.55:0/1067025458 shutdown_connections 2024-03-19T02:00:12.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.973+0000 7f3936725640 1 -- 172.21.15.55:0/1067025458 wait complete. 2024-03-19T02:00:12.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.977+0000 7f3936725640 1 Processor -- start 2024-03-19T02:00:12.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.977+0000 7f3936725640 1 -- start start 2024-03-19T02:00:12.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.977+0000 7f3936725640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f39300c8010 0x7f3930147d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:12.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.977+0000 7f3936725640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39300cd090 0x7f3930148b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:12.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.977+0000 7f3936725640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f39300cd470 0x7f393015b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:12.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.977+0000 7f3936725640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3930164a90 con 0x7f39300cd090 2024-03-19T02:00:12.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.977+0000 7f3936725640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3930164910 con 0x7f39300cd470 2024-03-19T02:00:12.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.977+0000 7f3936725640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3930164c10 con 0x7f39300c8010 2024-03-19T02:00:12.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.977+0000 7f392f7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39300cd090 0x7f3930148b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:12.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.977+0000 7f392f7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39300cd090 0x7f3930148b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56148/0 (socket says 172.21.15.55:56148) 2024-03-19T02:00:12.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.977+0000 7f392f7fe640 1 -- 172.21.15.55:0/3636024054 learned_addr learned my addr 172.21.15.55:0/3636024054 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:12.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.977+0000 7f392ffff640 1 --2- 172.21.15.55:0/3636024054 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f39300c8010 0x7f3930147d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:12.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.977+0000 7f3934cc2640 1 --2- 172.21.15.55:0/3636024054 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f39300cd470 0x7f393015b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:12.983 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.985+0000 7f392f7fe640 1 -- 172.21.15.55:0/3636024054 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f39300c8010 msgr2=0x7f3930147d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:12.983 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.985+0000 7f392f7fe640 1 --2- 172.21.15.55:0/3636024054 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f39300c8010 0x7f3930147d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:12.983 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.985+0000 7f392f7fe640 1 -- 172.21.15.55:0/3636024054 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f39300cd470 msgr2=0x7f393015b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:12.983 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.985+0000 7f392f7fe640 1 --2- 172.21.15.55:0/3636024054 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f39300cd470 0x7f393015b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:12.983 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.985+0000 7f392f7fe640 1 -- 172.21.15.55:0/3636024054 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39301473a0 con 0x7f39300cd090 2024-03-19T02:00:12.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.985+0000 7f392f7fe640 1 --2- 172.21.15.55:0/3636024054 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39300cd090 0x7f3930148b70 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f39240ae040 tx=0x7f3924019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:12.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.993+0000 7f392cff9640 1 -- 172.21.15.55:0/3636024054 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39240ba070 con 0x7f39300cd090 2024-03-19T02:00:12.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.993+0000 7f392cff9640 1 -- 172.21.15.55:0/3636024054 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39240b43f0 con 0x7f39300cd090 2024-03-19T02:00:12.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.993+0000 7f3936725640 1 -- 172.21.15.55:0/3636024054 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3930146580 con 0x7f39300cd090 2024-03-19T02:00:12.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.993+0000 7f392cff9640 1 -- 172.21.15.55:0/3636024054 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39240b0040 con 0x7f39300cd090 2024-03-19T02:00:12.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.993+0000 7f3936725640 1 -- 172.21.15.55:0/3636024054 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3930146b00 con 0x7f39300cd090 2024-03-19T02:00:12.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:12.993+0000 7f3936725640 1 -- 172.21.15.55:0/3636024054 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3930068a40 con 0x7f39300cd090 2024-03-19T02:00:13.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:13.261+0000 7f392cff9640 1 -- 172.21.15.55:0/3636024054 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f39240bd070 con 0x7f39300cd090 2024-03-19T02:00:13.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:13.261+0000 7f392cff9640 1 --2- 172.21.15.55:0/3636024054 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f391c067690 0x7f391c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:13.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:13.261+0000 7f392ffff640 1 --2- 172.21.15.55:0/3636024054 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f391c067690 0x7f391c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:13.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:13.261+0000 7f392ffff640 1 --2- 172.21.15.55:0/3636024054 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f391c067690 0x7f391c069b50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3920000f70 tx=0x7f3920007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:13.284 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:13.285+0000 7f392cff9640 1 -- 172.21.15.55:0/3636024054 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f39240ecfd0 con 0x7f39300cd090 2024-03-19T02:00:13.284 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:13.285+0000 7f392cff9640 1 -- 172.21.15.55:0/3636024054 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f3924016090 con 0x7f39300cd090 2024-03-19T02:00:13.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:13.493+0000 7f3936725640 1 -- 172.21.15.55:0/3636024054 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3930156d20 con 0x7f39300cd090 2024-03-19T02:00:13.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:13.521+0000 7f392cff9640 1 -- 172.21.15.55:0/3636024054 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f39240f28a0 con 0x7f39300cd090 2024-03-19T02:00:13.523 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:13.523 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:13.523 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:13.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:13.525+0000 7f3936725640 1 -- 172.21.15.55:0/3636024054 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f391c067690 msgr2=0x7f391c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:13.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:13.525+0000 7f3936725640 1 --2- 172.21.15.55:0/3636024054 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f391c067690 0x7f391c069b50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3920000f70 tx=0x7f3920007e90 comp rx=0 tx=0).stop 2024-03-19T02:00:13.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:13.525+0000 7f3936725640 1 -- 172.21.15.55:0/3636024054 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39300cd090 msgr2=0x7f3930148b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:13.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:13.525+0000 7f3936725640 1 --2- 172.21.15.55:0/3636024054 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39300cd090 0x7f3930148b70 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f39240ae040 tx=0x7f3924019240 comp rx=0 tx=0).stop 2024-03-19T02:00:13.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:13.525+0000 7f3936725640 1 -- 172.21.15.55:0/3636024054 shutdown_connections 2024-03-19T02:00:13.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:13.525+0000 7f3936725640 1 --2- 172.21.15.55:0/3636024054 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f391c067690 0x7f391c069b50 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:13.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:13.525+0000 7f3936725640 1 --2- 172.21.15.55:0/3636024054 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f39300cd470 0x7f393015b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:13.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:13.525+0000 7f3936725640 1 --2- 172.21.15.55:0/3636024054 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39300cd090 0x7f3930148b70 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:13.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:13.525+0000 7f3936725640 1 --2- 172.21.15.55:0/3636024054 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f39300c8010 0x7f3930147d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:13.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:13.525+0000 7f3936725640 1 -- 172.21.15.55:0/3636024054 >> 172.21.15.55:0/3636024054 conn(0x7f393006e670 msgr2=0x7f39300c7170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:13.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:13.525+0000 7f3936725640 1 -- 172.21.15.55:0/3636024054 shutdown_connections 2024-03-19T02:00:13.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:13.525+0000 7f3936725640 1 -- 172.21.15.55:0/3636024054 wait complete. 2024-03-19T02:00:14.148 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:14.148 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:15.149 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:15.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.261+0000 7f52b4818640 1 Processor -- start 2024-03-19T02:00:15.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.261+0000 7f52b4818640 1 -- start start 2024-03-19T02:00:15.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.265+0000 7f52b4818640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f52b00ca350 0x7f52b015b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:15.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.265+0000 7f52b4818640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f52b00c3320 0x7f52b00728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:15.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.265+0000 7f52b4818640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52b0151330 con 0x7f52b0150f50 2024-03-19T02:00:15.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.265+0000 7f52b4818640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f52b00c3e00 con 0x7f52b00ca350 2024-03-19T02:00:15.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.265+0000 7f52b4818640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f52b015ac10 con 0x7f52b00c3320 2024-03-19T02:00:15.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.265+0000 7f52add9b640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f52b00ca350 0x7f52b015b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:15.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.265+0000 7f52ae59c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f52b0150f50 0x7f52b0157920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:60414/0 (socket says 172.21.15.55:60414) 2024-03-19T02:00:15.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.265+0000 7f52ae59c640 1 -- 172.21.15.55:0/801199401 learned_addr learned my addr 172.21.15.55:0/801199401 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:00:15.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.277+0000 7f52ad59a640 1 -- 172.21.15.55:0/801199401 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3079902441 0 0) 0x7f52b0151330 con 0x7f52b0150f50 2024-03-19T02:00:15.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.277+0000 7f52ad59a640 1 -- 172.21.15.55:0/801199401 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5298005740 con 0x7f52b0150f50 2024-03-19T02:00:15.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.277+0000 7f52add9b640 1 -- 172.21.15.55:0/801199401 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f52b00c3320 msgr2=0x7f52b00728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:15.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.277+0000 7f52add9b640 1 --2- 172.21.15.55:0/801199401 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f52b00c3320 0x7f52b00728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:15.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.277+0000 7f52add9b640 1 -- 172.21.15.55:0/801199401 >> v1:172.21.15.84:6789/0 conn(0x7f52b0150f50 legacy=0x7f52b0157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:15.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.277+0000 7f52add9b640 1 -- 172.21.15.55:0/801199401 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52b00c3840 con 0x7f52b00ca350 2024-03-19T02:00:15.285 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.285+0000 7f52add9b640 1 --2- 172.21.15.55:0/801199401 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f52b00ca350 0x7f52b015b0a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f52a40097c0 tx=0x7f52a401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c895b70597f21654 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:15.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.297+0000 7f52ad59a640 1 -- 172.21.15.55:0/801199401 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a40aa600 con 0x7f52b00ca350 2024-03-19T02:00:15.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.297+0000 7f52ad59a640 1 -- 172.21.15.55:0/801199401 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52a40af070 con 0x7f52b00ca350 2024-03-19T02:00:15.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.297+0000 7f52ad59a640 1 -- 172.21.15.55:0/801199401 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a40aa9c0 con 0x7f52b00ca350 2024-03-19T02:00:15.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.297+0000 7f52b4818640 1 -- 172.21.15.55:0/801199401 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f52b00ca350 msgr2=0x7f52b015b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:15.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.297+0000 7f52b4818640 1 --2- 172.21.15.55:0/801199401 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f52b00ca350 0x7f52b015b0a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f52a40097c0 tx=0x7f52a401c4a0 comp rx=0 tx=0).stop 2024-03-19T02:00:15.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.297+0000 7f52b4818640 1 -- 172.21.15.55:0/801199401 shutdown_connections 2024-03-19T02:00:15.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.297+0000 7f52b4818640 1 --2- 172.21.15.55:0/801199401 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f52b00c3320 0x7f52b00728b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:15.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.297+0000 7f52b4818640 1 --2- 172.21.15.55:0/801199401 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f52b00ca350 0x7f52b015b0a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:15.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.297+0000 7f52b4818640 1 -- 172.21.15.55:0/801199401 >> 172.21.15.55:0/801199401 conn(0x7f52b006e670 msgr2=0x7f52b006ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:15.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.297+0000 7f52b4818640 1 -- 172.21.15.55:0/801199401 shutdown_connections 2024-03-19T02:00:15.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.297+0000 7f52b4818640 1 -- 172.21.15.55:0/801199401 wait complete. 2024-03-19T02:00:15.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.297+0000 7f52b4818640 1 Processor -- start 2024-03-19T02:00:15.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.297+0000 7f52b4818640 1 -- start start 2024-03-19T02:00:15.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.297+0000 7f52b4818640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f52b00c3320 0x7f52b0158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:15.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.297+0000 7f52b4818640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f52b00ca350 0x7f52b0159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:15.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.301+0000 7f52b4818640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52b0150f50 0x7f52b0070410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:15.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.301+0000 7f52b4818640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f52b006b420 con 0x7f52b0150f50 2024-03-19T02:00:15.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.301+0000 7f52b4818640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f52b006b2a0 con 0x7f52b00c3320 2024-03-19T02:00:15.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.301+0000 7f52b4818640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f52b006b5a0 con 0x7f52b00ca350 2024-03-19T02:00:15.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.301+0000 7f52ae59c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f52b00c3320 0x7f52b0158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:15.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.301+0000 7f52ae59c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f52b00c3320 0x7f52b0158060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:52636/0 (socket says 172.21.15.55:52636) 2024-03-19T02:00:15.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.301+0000 7f52ae59c640 1 -- 172.21.15.55:0/1625743944 learned_addr learned my addr 172.21.15.55:0/1625743944 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:15.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.301+0000 7f52add9b640 1 --2- 172.21.15.55:0/1625743944 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f52b00ca350 0x7f52b0159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:15.302 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.301+0000 7f52aed9d640 1 --2- 172.21.15.55:0/1625743944 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52b0150f50 0x7f52b0070410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:15.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.305+0000 7f52ae59c640 1 -- 172.21.15.55:0/1625743944 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f52b00ca350 msgr2=0x7f52b0159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:15.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.305+0000 7f52ae59c640 1 --2- 172.21.15.55:0/1625743944 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f52b00ca350 0x7f52b0159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:15.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.305+0000 7f52ae59c640 1 -- 172.21.15.55:0/1625743944 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52b0150f50 msgr2=0x7f52b0070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:15.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.305+0000 7f52ae59c640 1 --2- 172.21.15.55:0/1625743944 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52b0150f50 0x7f52b0070410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:15.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.305+0000 7f52ae59c640 1 -- 172.21.15.55:0/1625743944 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52b00cbc20 con 0x7f52b00c3320 2024-03-19T02:00:15.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.309+0000 7f52ae59c640 1 --2- 172.21.15.55:0/1625743944 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f52b00c3320 0x7f52b0158060 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f52a000c480 tx=0x7f52a000c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:15.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.313+0000 7f529effd640 1 -- 172.21.15.55:0/1625743944 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a0017070 con 0x7f52b00c3320 2024-03-19T02:00:15.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.313+0000 7f529effd640 1 -- 172.21.15.55:0/1625743944 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52a0005cf0 con 0x7f52b00c3320 2024-03-19T02:00:15.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.313+0000 7f529effd640 1 -- 172.21.15.55:0/1625743944 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a0007270 con 0x7f52b00c3320 2024-03-19T02:00:15.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.313+0000 7f52b4818640 1 -- 172.21.15.55:0/1625743944 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52b00cbf00 con 0x7f52b00c3320 2024-03-19T02:00:15.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.317+0000 7f52b4818640 1 -- 172.21.15.55:0/1625743944 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52b0159930 con 0x7f52b00c3320 2024-03-19T02:00:15.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.317+0000 7f52b4818640 1 -- 172.21.15.55:0/1625743944 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52b0159760 con 0x7f52b00c3320 2024-03-19T02:00:15.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.605+0000 7f529effd640 1 -- 172.21.15.55:0/1625743944 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f52a001a070 con 0x7f52b00c3320 2024-03-19T02:00:15.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.609+0000 7f529effd640 1 --2- 172.21.15.55:0/1625743944 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f52900676e0 0x7f5290069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:15.607 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.609+0000 7f52add9b640 1 --2- 172.21.15.55:0/1625743944 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f52900676e0 0x7f5290069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:15.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.609+0000 7f52add9b640 1 --2- 172.21.15.55:0/1625743944 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f52900676e0 0x7f5290069ba0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f52a40ae000 tx=0x7f52a40a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:15.611 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.613+0000 7f529effd640 1 -- 172.21.15.55:0/1625743944 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f52a0053760 con 0x7f52b00c3320 2024-03-19T02:00:15.639 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.641+0000 7f529effd640 1 -- 172.21.15.55:0/1625743944 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f52a00556c0 con 0x7f52b00c3320 2024-03-19T02:00:15.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.857+0000 7f52b4818640 1 -- 172.21.15.55:0/1625743944 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52b00c7600 con 0x7f52b00c3320 2024-03-19T02:00:15.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.885+0000 7f529effd640 1 -- 172.21.15.55:0/1625743944 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f52a0059760 con 0x7f52b00c3320 2024-03-19T02:00:15.886 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:15.886 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:15.886 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:15.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.889+0000 7f52b4818640 1 -- 172.21.15.55:0/1625743944 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f52900676e0 msgr2=0x7f5290069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:15.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.889+0000 7f52b4818640 1 --2- 172.21.15.55:0/1625743944 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f52900676e0 0x7f5290069ba0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f52a40ae000 tx=0x7f52a40a7000 comp rx=0 tx=0).stop 2024-03-19T02:00:15.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.889+0000 7f52b4818640 1 -- 172.21.15.55:0/1625743944 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f52b00c3320 msgr2=0x7f52b0158060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:15.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.889+0000 7f52b4818640 1 --2- 172.21.15.55:0/1625743944 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f52b00c3320 0x7f52b0158060 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f52a000c480 tx=0x7f52a000c940 comp rx=0 tx=0).stop 2024-03-19T02:00:15.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.889+0000 7f52b4818640 1 -- 172.21.15.55:0/1625743944 shutdown_connections 2024-03-19T02:00:15.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.889+0000 7f52b4818640 1 --2- 172.21.15.55:0/1625743944 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f52900676e0 0x7f5290069ba0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:15.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.889+0000 7f52b4818640 1 --2- 172.21.15.55:0/1625743944 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52b0150f50 0x7f52b0070410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:15.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.889+0000 7f52b4818640 1 --2- 172.21.15.55:0/1625743944 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f52b00ca350 0x7f52b0159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:15.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.889+0000 7f52b4818640 1 --2- 172.21.15.55:0/1625743944 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f52b00c3320 0x7f52b0158060 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:15.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.889+0000 7f52b4818640 1 -- 172.21.15.55:0/1625743944 >> 172.21.15.55:0/1625743944 conn(0x7f52b006e670 msgr2=0x7f52b0143d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:15.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.889+0000 7f52b4818640 1 -- 172.21.15.55:0/1625743944 shutdown_connections 2024-03-19T02:00:15.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:15.889+0000 7f52b4818640 1 -- 172.21.15.55:0/1625743944 wait complete. 2024-03-19T02:00:16.508 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:16.508 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:17.509 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:17.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.633+0000 7fc30f287640 1 Processor -- start 2024-03-19T02:00:17.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.633+0000 7fc30f287640 1 -- start start 2024-03-19T02:00:17.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.633+0000 7fc30f287640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fc30806c610 0x7fc30815ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:17.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.633+0000 7fc30f287640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc308070e80 con 0x7fc30806c9f0 2024-03-19T02:00:17.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.633+0000 7fc30f287640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc30815b250 con 0x7fc30806ceb0 2024-03-19T02:00:17.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.633+0000 7fc30f287640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fc30806fb00 con 0x7fc30806c610 2024-03-19T02:00:17.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.637+0000 7fc30d023640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fc30806c610 0x7fc30815ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:17.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.637+0000 7fc30c822640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fc30806c9f0 0x7fc3080c92c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:34294/0 (socket says 172.21.15.55:34294) 2024-03-19T02:00:17.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.637+0000 7fc30c822640 1 -- 172.21.15.55:0/438664522 learned_addr learned my addr 172.21.15.55:0/438664522 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:00:17.639 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.637+0000 7fc2fffff640 1 -- 172.21.15.55:0/438664522 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 666112077 0 0) 0x7fc308070e80 con 0x7fc30806c9f0 2024-03-19T02:00:17.639 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.637+0000 7fc2fffff640 1 -- 172.21.15.55:0/438664522 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2f4003530 con 0x7fc30806c9f0 2024-03-19T02:00:17.639 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.641+0000 7fc30d023640 1 -- 172.21.15.55:0/438664522 >> v1:172.21.15.84:6789/0 conn(0x7fc30806ceb0 legacy=0x7fc3081427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:17.639 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.641+0000 7fc30d023640 1 -- 172.21.15.55:0/438664522 >> v1:172.21.15.55:6789/0 conn(0x7fc30806c9f0 legacy=0x7fc3080c92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:17.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.641+0000 7fc30d023640 1 -- 172.21.15.55:0/438664522 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30806fde0 con 0x7fc30806c610 2024-03-19T02:00:17.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.645+0000 7fc30d023640 1 --2- 172.21.15.55:0/438664522 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc30806c610 0x7fc30815ad10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc3040097c0 tx=0x7fc30401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29f9bb1decf11f07 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:17.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.653+0000 7fc2fffff640 1 -- 172.21.15.55:0/438664522 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3040b1070 con 0x7fc30806c610 2024-03-19T02:00:17.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.653+0000 7fc2fffff640 1 -- 172.21.15.55:0/438664522 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3040ac5f0 con 0x7fc30806c610 2024-03-19T02:00:17.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.653+0000 7fc2fffff640 1 -- 172.21.15.55:0/438664522 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3040ac9c0 con 0x7fc30806c610 2024-03-19T02:00:17.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.653+0000 7fc30f287640 1 -- 172.21.15.55:0/438664522 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc30806c610 msgr2=0x7fc30815ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:17.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.653+0000 7fc30f287640 1 --2- 172.21.15.55:0/438664522 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc30806c610 0x7fc30815ad10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc3040097c0 tx=0x7fc30401c4a0 comp rx=0 tx=0).stop 2024-03-19T02:00:17.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.653+0000 7fc30f287640 1 -- 172.21.15.55:0/438664522 shutdown_connections 2024-03-19T02:00:17.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.653+0000 7fc30f287640 1 --2- 172.21.15.55:0/438664522 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc30806c610 0x7fc30815ad10 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:17.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.653+0000 7fc30f287640 1 -- 172.21.15.55:0/438664522 >> 172.21.15.55:0/438664522 conn(0x7fc308067600 msgr2=0x7fc308071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:17.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.653+0000 7fc30f287640 1 -- 172.21.15.55:0/438664522 shutdown_connections 2024-03-19T02:00:17.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.653+0000 7fc30f287640 1 -- 172.21.15.55:0/438664522 wait complete. 2024-03-19T02:00:17.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.653+0000 7fc30f287640 1 Processor -- start 2024-03-19T02:00:17.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.653+0000 7fc30f287640 1 -- start start 2024-03-19T02:00:17.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.653+0000 7fc30f287640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc30806c610 0x7fc30813c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:17.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.653+0000 7fc30f287640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc30806c9f0 0x7fc308159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:17.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.653+0000 7fc30f287640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc30806ceb0 0x7fc3081f2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:17.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.657+0000 7fc30f287640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc3080c99d0 con 0x7fc30806c610 2024-03-19T02:00:17.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.657+0000 7fc30f287640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc308070960 con 0x7fc30806ceb0 2024-03-19T02:00:17.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.657+0000 7fc30f287640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc308142f00 con 0x7fc30806c9f0 2024-03-19T02:00:17.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.657+0000 7fc30d824640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc30806ceb0 0x7fc3081f2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:17.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.657+0000 7fc30d023640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc30806c610 0x7fc30813c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:17.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.657+0000 7fc30c822640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc30806c9f0 0x7fc308159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:17.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.657+0000 7fc30d023640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc30806c610 0x7fc30813c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56170/0 (socket says 172.21.15.55:56170) 2024-03-19T02:00:17.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.657+0000 7fc30d023640 1 -- 172.21.15.55:0/2463722978 learned_addr learned my addr 172.21.15.55:0/2463722978 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:17.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.661+0000 7fc30d023640 1 -- 172.21.15.55:0/2463722978 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc30806c9f0 msgr2=0x7fc308159b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:17.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.661+0000 7fc30d023640 1 --2- 172.21.15.55:0/2463722978 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc30806c9f0 0x7fc308159b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:17.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.661+0000 7fc30d023640 1 -- 172.21.15.55:0/2463722978 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc30806ceb0 msgr2=0x7fc3081f2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:17.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.661+0000 7fc30d023640 1 --2- 172.21.15.55:0/2463722978 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc30806ceb0 0x7fc3081f2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:17.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.661+0000 7fc30d023640 1 -- 172.21.15.55:0/2463722978 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30815df20 con 0x7fc30806c610 2024-03-19T02:00:17.661 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.661+0000 7fc30d023640 1 --2- 172.21.15.55:0/2463722978 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc30806c610 0x7fc30813c890 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc30401a0d0 tx=0x7fc304019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:17.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.665+0000 7fc2fdffb640 1 -- 172.21.15.55:0/2463722978 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3040b1040 con 0x7fc30806c610 2024-03-19T02:00:17.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.665+0000 7fc2fdffb640 1 -- 172.21.15.55:0/2463722978 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3040bc070 con 0x7fc30806c610 2024-03-19T02:00:17.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.665+0000 7fc2fdffb640 1 -- 172.21.15.55:0/2463722978 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3040b66c0 con 0x7fc30806c610 2024-03-19T02:00:17.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.665+0000 7fc30f287640 1 -- 172.21.15.55:0/2463722978 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30813ced0 con 0x7fc30806c610 2024-03-19T02:00:17.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.665+0000 7fc30f287640 1 -- 172.21.15.55:0/2463722978 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc30815d960 con 0x7fc30806c610 2024-03-19T02:00:17.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.665+0000 7fc30f287640 1 -- 172.21.15.55:0/2463722978 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc308068a40 con 0x7fc30806c610 2024-03-19T02:00:17.686 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.685+0000 7fc2fdffb640 1 -- 172.21.15.55:0/2463722978 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc3040b4070 con 0x7fc30806c610 2024-03-19T02:00:17.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.685+0000 7fc2fdffb640 1 --2- 172.21.15.55:0/2463722978 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc2ec067660 0x7fc2ec069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:17.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.689+0000 7fc30c822640 1 --2- 172.21.15.55:0/2463722978 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc2ec067660 0x7fc2ec069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:17.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.689+0000 7fc30c822640 1 --2- 172.21.15.55:0/2463722978 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc2ec067660 0x7fc2ec069b20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8007fd0 tx=0x7fc2f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:17.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.689+0000 7fc2fdffb640 1 -- 172.21.15.55:0/2463722978 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fc3040ee370 con 0x7fc30806c610 2024-03-19T02:00:17.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:17.973+0000 7fc2fdffb640 1 -- 172.21.15.55:0/2463722978 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc3040a9070 con 0x7fc30806c610 2024-03-19T02:00:18.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:18.197+0000 7fc30f287640 1 -- 172.21.15.55:0/2463722978 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3080c39d0 con 0x7fc30806c610 2024-03-19T02:00:18.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:18.217+0000 7fc2fdffb640 1 -- 172.21.15.55:0/2463722978 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fc304016090 con 0x7fc30806c610 2024-03-19T02:00:18.218 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:18.218 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:18.219 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:18.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:18.221+0000 7fc30f287640 1 -- 172.21.15.55:0/2463722978 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc2ec067660 msgr2=0x7fc2ec069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:18.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:18.221+0000 7fc30f287640 1 --2- 172.21.15.55:0/2463722978 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc2ec067660 0x7fc2ec069b20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8007fd0 tx=0x7fc2f8009040 comp rx=0 tx=0).stop 2024-03-19T02:00:18.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:18.221+0000 7fc30f287640 1 -- 172.21.15.55:0/2463722978 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc30806c610 msgr2=0x7fc30813c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:18.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:18.221+0000 7fc30f287640 1 --2- 172.21.15.55:0/2463722978 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc30806c610 0x7fc30813c890 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc30401a0d0 tx=0x7fc304019240 comp rx=0 tx=0).stop 2024-03-19T02:00:18.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:18.221+0000 7fc30f287640 1 -- 172.21.15.55:0/2463722978 shutdown_connections 2024-03-19T02:00:18.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:18.221+0000 7fc30f287640 1 --2- 172.21.15.55:0/2463722978 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc2ec067660 0x7fc2ec069b20 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:18.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:18.221+0000 7fc30f287640 1 --2- 172.21.15.55:0/2463722978 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc30806ceb0 0x7fc3081f2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:18.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:18.221+0000 7fc30f287640 1 --2- 172.21.15.55:0/2463722978 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc30806c9f0 0x7fc308159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:18.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:18.221+0000 7fc30f287640 1 --2- 172.21.15.55:0/2463722978 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc30806c610 0x7fc30813c890 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:18.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:18.221+0000 7fc30f287640 1 -- 172.21.15.55:0/2463722978 >> 172.21.15.55:0/2463722978 conn(0x7fc308067600 msgr2=0x7fc3080c3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:18.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:18.221+0000 7fc30f287640 1 -- 172.21.15.55:0/2463722978 shutdown_connections 2024-03-19T02:00:18.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:18.221+0000 7fc30f287640 1 -- 172.21.15.55:0/2463722978 wait complete. 2024-03-19T02:00:18.840 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:18.841 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:19.842 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:19.956 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.957+0000 7fc998094640 1 Processor -- start 2024-03-19T02:00:19.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.957+0000 7fc998094640 1 -- start start 2024-03-19T02:00:19.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.957+0000 7fc998094640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc990158d00 0x7fc9900c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:19.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.957+0000 7fc998094640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc99014ee80 con 0x7fc9900c60d0 2024-03-19T02:00:19.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.957+0000 7fc998094640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9900c7540 con 0x7fc99014eaa0 2024-03-19T02:00:19.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.957+0000 7fc998094640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fc9900c77c0 con 0x7fc990158d00 2024-03-19T02:00:19.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.957+0000 7fc996631640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc990158d00 0x7fc9900c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:19.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.961+0000 7fc995e30640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fc99014eaa0 0x7fc99006b980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:52874/0 (socket says 172.21.15.55:52874) 2024-03-19T02:00:19.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.961+0000 7fc995e30640 1 -- 172.21.15.55:0/4223746415 learned_addr learned my addr 172.21.15.55:0/4223746415 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:00:19.961 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.961+0000 7fc996631640 1 -- 172.21.15.55:0/4223746415 >> v1:172.21.15.84:6789/0 conn(0x7fc99014eaa0 legacy=0x7fc99006b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:19.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.961+0000 7fc996631640 1 -- 172.21.15.55:0/4223746415 >> v1:172.21.15.55:6789/0 conn(0x7fc9900c60d0 legacy=0x7fc9901585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:19.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.961+0000 7fc996631640 1 -- 172.21.15.55:0/4223746415 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9900c7aa0 con 0x7fc990158d00 2024-03-19T02:00:19.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.965+0000 7fc996631640 1 --2- 172.21.15.55:0/4223746415 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc990158d00 0x7fc9900c6540 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc988009d90 tx=0x7fc98801c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53bb8d97adacd0c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:19.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.973+0000 7fc994e2e640 1 -- 172.21.15.55:0/4223746415 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9880b2070 con 0x7fc990158d00 2024-03-19T02:00:19.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.973+0000 7fc994e2e640 1 -- 172.21.15.55:0/4223746415 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9880a5960 con 0x7fc990158d00 2024-03-19T02:00:19.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.973+0000 7fc994e2e640 1 -- 172.21.15.55:0/4223746415 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9880a5d30 con 0x7fc990158d00 2024-03-19T02:00:19.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.973+0000 7fc998094640 1 -- 172.21.15.55:0/4223746415 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc990158d00 msgr2=0x7fc9900c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:19.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.973+0000 7fc998094640 1 --2- 172.21.15.55:0/4223746415 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc990158d00 0x7fc9900c6540 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc988009d90 tx=0x7fc98801c8a0 comp rx=0 tx=0).stop 2024-03-19T02:00:19.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.973+0000 7fc998094640 1 -- 172.21.15.55:0/4223746415 shutdown_connections 2024-03-19T02:00:19.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.973+0000 7fc998094640 1 --2- 172.21.15.55:0/4223746415 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc990158d00 0x7fc9900c6540 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:19.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.973+0000 7fc998094640 1 -- 172.21.15.55:0/4223746415 >> 172.21.15.55:0/4223746415 conn(0x7fc9900cc9c0 msgr2=0x7fc9900ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:19.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.973+0000 7fc998094640 1 -- 172.21.15.55:0/4223746415 shutdown_connections 2024-03-19T02:00:19.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.973+0000 7fc998094640 1 -- 172.21.15.55:0/4223746415 wait complete. 2024-03-19T02:00:19.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.973+0000 7fc998094640 1 Processor -- start 2024-03-19T02:00:19.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.973+0000 7fc998094640 1 -- start start 2024-03-19T02:00:19.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.973+0000 7fc998094640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9900c60d0 0x7fc990143210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:19.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.973+0000 7fc998094640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc99014eaa0 0x7fc990153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:19.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.973+0000 7fc998094640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc990159b90 0x7fc9901543e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:19.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.973+0000 7fc998094640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc99006c210 con 0x7fc990159b90 2024-03-19T02:00:19.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.973+0000 7fc998094640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc99006c090 con 0x7fc9900c60d0 2024-03-19T02:00:19.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.973+0000 7fc998094640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc9900c6bf0 con 0x7fc99014eaa0 2024-03-19T02:00:19.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.977+0000 7fc996631640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc990159b90 0x7fc9901543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:19.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.977+0000 7fc996631640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc990159b90 0x7fc9901543e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39386/0 (socket says 172.21.15.55:39386) 2024-03-19T02:00:19.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.977+0000 7fc996631640 1 -- 172.21.15.55:0/3766192907 learned_addr learned my addr 172.21.15.55:0/3766192907 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:19.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.977+0000 7fc99562f640 1 --2- 172.21.15.55:0/3766192907 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc99014eaa0 0x7fc990153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:19.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.977+0000 7fc995e30640 1 --2- 172.21.15.55:0/3766192907 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9900c60d0 0x7fc990143210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:19.979 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.981+0000 7fc996631640 1 -- 172.21.15.55:0/3766192907 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc99014eaa0 msgr2=0x7fc990153ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:19.979 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.981+0000 7fc996631640 1 --2- 172.21.15.55:0/3766192907 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc99014eaa0 0x7fc990153ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:19.979 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.981+0000 7fc996631640 1 -- 172.21.15.55:0/3766192907 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9900c60d0 msgr2=0x7fc990143210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:19.979 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.981+0000 7fc996631640 1 --2- 172.21.15.55:0/3766192907 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9900c60d0 0x7fc990143210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:19.980 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.981+0000 7fc996631640 1 -- 172.21.15.55:0/3766192907 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc99015ae90 con 0x7fc990159b90 2024-03-19T02:00:19.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.981+0000 7fc996631640 1 --2- 172.21.15.55:0/3766192907 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc990159b90 0x7fc9901543e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc98801a4d0 tx=0x7fc988019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:19.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.985+0000 7fc9867fc640 1 -- 172.21.15.55:0/3766192907 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9880b2050 con 0x7fc990159b90 2024-03-19T02:00:19.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.985+0000 7fc9867fc640 1 -- 172.21.15.55:0/3766192907 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9880be070 con 0x7fc990159b90 2024-03-19T02:00:19.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.985+0000 7fc998094640 1 -- 172.21.15.55:0/3766192907 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc990155300 con 0x7fc990159b90 2024-03-19T02:00:19.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.985+0000 7fc9867fc640 1 -- 172.21.15.55:0/3766192907 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9880adb50 con 0x7fc990159b90 2024-03-19T02:00:19.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.985+0000 7fc998094640 1 -- 172.21.15.55:0/3766192907 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc99015b780 con 0x7fc990159b90 2024-03-19T02:00:19.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:19.989+0000 7fc998094640 1 -- 172.21.15.55:0/3766192907 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc990068a40 con 0x7fc990159b90 2024-03-19T02:00:20.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:20.013+0000 7fc9867fc640 1 -- 172.21.15.55:0/3766192907 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc9880adcf0 con 0x7fc990159b90 2024-03-19T02:00:20.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:20.013+0000 7fc9867fc640 1 --2- 172.21.15.55:0/3766192907 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc9740676b0 0x7fc974069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:20.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:20.013+0000 7fc995e30640 1 --2- 172.21.15.55:0/3766192907 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc9740676b0 0x7fc974069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:20.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:20.013+0000 7fc995e30640 1 --2- 172.21.15.55:0/3766192907 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc9740676b0 0x7fc974069b70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc98c000f70 tx=0x7fc98c007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:20.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:20.313+0000 7fc9867fc640 1 -- 172.21.15.55:0/3766192907 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fc9880efb80 con 0x7fc990159b90 2024-03-19T02:00:20.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:20.313+0000 7fc9867fc640 1 -- 172.21.15.55:0/3766192907 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc988016520 con 0x7fc990159b90 2024-03-19T02:00:20.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:20.533+0000 7fc998094640 1 -- 172.21.15.55:0/3766192907 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9900c2170 con 0x7fc990159b90 2024-03-19T02:00:20.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:20.549+0000 7fc9867fc640 1 -- 172.21.15.55:0/3766192907 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fc9880f1ae0 con 0x7fc990159b90 2024-03-19T02:00:20.550 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:20.550 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:20.550 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:20.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:20.553+0000 7fc998094640 1 -- 172.21.15.55:0/3766192907 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc9740676b0 msgr2=0x7fc974069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:20.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:20.553+0000 7fc998094640 1 --2- 172.21.15.55:0/3766192907 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc9740676b0 0x7fc974069b70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc98c000f70 tx=0x7fc98c007e90 comp rx=0 tx=0).stop 2024-03-19T02:00:20.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:20.553+0000 7fc998094640 1 -- 172.21.15.55:0/3766192907 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc990159b90 msgr2=0x7fc9901543e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:20.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:20.553+0000 7fc998094640 1 --2- 172.21.15.55:0/3766192907 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc990159b90 0x7fc9901543e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc98801a4d0 tx=0x7fc988019640 comp rx=0 tx=0).stop 2024-03-19T02:00:20.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:20.553+0000 7fc998094640 1 -- 172.21.15.55:0/3766192907 shutdown_connections 2024-03-19T02:00:20.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:20.553+0000 7fc998094640 1 --2- 172.21.15.55:0/3766192907 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc9740676b0 0x7fc974069b70 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:20.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:20.553+0000 7fc998094640 1 --2- 172.21.15.55:0/3766192907 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc990159b90 0x7fc9901543e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:20.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:20.553+0000 7fc998094640 1 --2- 172.21.15.55:0/3766192907 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc99014eaa0 0x7fc990153ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:20.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:20.553+0000 7fc998094640 1 --2- 172.21.15.55:0/3766192907 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9900c60d0 0x7fc990143210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:20.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:20.553+0000 7fc998094640 1 -- 172.21.15.55:0/3766192907 >> 172.21.15.55:0/3766192907 conn(0x7fc9900cc9c0 msgr2=0x7fc9901565f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:20.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:20.553+0000 7fc998094640 1 -- 172.21.15.55:0/3766192907 shutdown_connections 2024-03-19T02:00:20.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:20.553+0000 7fc998094640 1 -- 172.21.15.55:0/3766192907 wait complete. 2024-03-19T02:00:21.173 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:21.174 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:22.175 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:22.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.305+0000 7f42d68c8640 1 Processor -- start 2024-03-19T02:00:22.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.305+0000 7f42d68c8640 1 -- start start 2024-03-19T02:00:22.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.305+0000 7f42d68c8640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f42d0154720 0x7f42d015e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:22.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.305+0000 7f42d68c8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f42d00c73c0 0x7f42d015aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:22.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.305+0000 7f42d68c8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42d0071280 con 0x7f42d0154340 2024-03-19T02:00:22.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.305+0000 7f42d68c8640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f42d0073840 con 0x7f42d00c73c0 2024-03-19T02:00:22.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.305+0000 7f42d68c8640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f42d00c7850 con 0x7f42d0154720 2024-03-19T02:00:22.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.305+0000 7f42d4e65640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f42d00c73c0 0x7f42d015aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:22.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.305+0000 7f42d4e65640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f42d00c73c0 0x7f42d015aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39396/0 (socket says 172.21.15.55:39396) 2024-03-19T02:00:22.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.305+0000 7f42d4e65640 1 -- 172.21.15.55:0/2304236743 learned_addr learned my addr 172.21.15.55:0/2304236743 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:22.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.305+0000 7f42cf7fe640 1 --2- 172.21.15.55:0/2304236743 >> v2:172.21.15.84:3300/0 conn(0x7f42d0154720 0x7f42d015e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:22.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.309+0000 7f42d4e65640 1 -- 172.21.15.55:0/2304236743 >> v2:172.21.15.84:3300/0 conn(0x7f42d0154720 msgr2=0x7f42d015e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:22.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.309+0000 7f42d4e65640 1 --2- 172.21.15.55:0/2304236743 >> v2:172.21.15.84:3300/0 conn(0x7f42d0154720 0x7f42d015e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:22.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.309+0000 7f42d4e65640 1 -- 172.21.15.55:0/2304236743 >> v1:172.21.15.55:6789/0 conn(0x7f42d0154340 legacy=0x7f42d0072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:22.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.309+0000 7f42d4e65640 1 -- 172.21.15.55:0/2304236743 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d015d900 con 0x7f42d00c73c0 2024-03-19T02:00:22.313 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.313+0000 7f42d4e65640 1 --2- 172.21.15.55:0/2304236743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42d00c73c0 0x7f42d015aca0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f42d00709e0 tx=0x7f42c801c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ef9058081ce12b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:22.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.317+0000 7f42ceffd640 1 -- 172.21.15.55:0/2304236743 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c80a5670 con 0x7f42d00c73c0 2024-03-19T02:00:22.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.317+0000 7f42ceffd640 1 -- 172.21.15.55:0/2304236743 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42c80a5810 con 0x7f42d00c73c0 2024-03-19T02:00:22.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.317+0000 7f42ceffd640 1 -- 172.21.15.55:0/2304236743 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c80a5be0 con 0x7f42d00c73c0 2024-03-19T02:00:22.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.317+0000 7f42d68c8640 1 -- 172.21.15.55:0/2304236743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42d00c73c0 msgr2=0x7f42d015aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:22.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.317+0000 7f42d68c8640 1 --2- 172.21.15.55:0/2304236743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42d00c73c0 0x7f42d015aca0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f42d00709e0 tx=0x7f42c801c9b0 comp rx=0 tx=0).stop 2024-03-19T02:00:22.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.317+0000 7f42d68c8640 1 -- 172.21.15.55:0/2304236743 shutdown_connections 2024-03-19T02:00:22.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.317+0000 7f42d68c8640 1 --2- 172.21.15.55:0/2304236743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42d00c73c0 0x7f42d015aca0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:22.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.317+0000 7f42d68c8640 1 --2- 172.21.15.55:0/2304236743 >> v2:172.21.15.84:3300/0 conn(0x7f42d0154720 0x7f42d015e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:22.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.317+0000 7f42d68c8640 1 -- 172.21.15.55:0/2304236743 >> 172.21.15.55:0/2304236743 conn(0x7f42d00cc9c0 msgr2=0x7f42d00ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:22.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.317+0000 7f42d68c8640 1 -- 172.21.15.55:0/2304236743 shutdown_connections 2024-03-19T02:00:22.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.317+0000 7f42d68c8640 1 -- 172.21.15.55:0/2304236743 wait complete. 2024-03-19T02:00:22.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.321+0000 7f42d68c8640 1 Processor -- start 2024-03-19T02:00:22.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.321+0000 7f42d68c8640 1 -- start start 2024-03-19T02:00:22.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.321+0000 7f42d68c8640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f42d00c73c0 0x7f42d014ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:22.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.321+0000 7f42d68c8640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42d0154340 0x7f42d015c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:22.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.321+0000 7f42d68c8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42d0154720 0x7f42d0158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:22.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.321+0000 7f42d68c8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f42d0166b70 con 0x7f42d0154720 2024-03-19T02:00:22.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.321+0000 7f42d68c8640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f42d015b1e0 con 0x7f42d0154340 2024-03-19T02:00:22.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.321+0000 7f42d68c8640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f42d0166cf0 con 0x7f42d00c73c0 2024-03-19T02:00:22.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.321+0000 7f42d4e65640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42d0154720 0x7f42d0158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:22.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.321+0000 7f42d4e65640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42d0154720 0x7f42d0158060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39410/0 (socket says 172.21.15.55:39410) 2024-03-19T02:00:22.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.321+0000 7f42d4e65640 1 -- 172.21.15.55:0/684857711 learned_addr learned my addr 172.21.15.55:0/684857711 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:22.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.321+0000 7f42cf7fe640 1 --2- 172.21.15.55:0/684857711 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42d0154340 0x7f42d015c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:22.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.325+0000 7f42cffff640 1 --2- 172.21.15.55:0/684857711 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f42d00c73c0 0x7f42d014ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:22.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.325+0000 7f42d4e65640 1 -- 172.21.15.55:0/684857711 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f42d00c73c0 msgr2=0x7f42d014ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:22.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.325+0000 7f42d4e65640 1 --2- 172.21.15.55:0/684857711 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f42d00c73c0 0x7f42d014ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:22.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.325+0000 7f42d4e65640 1 -- 172.21.15.55:0/684857711 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42d0154340 msgr2=0x7f42d015c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:22.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.325+0000 7f42d4e65640 1 --2- 172.21.15.55:0/684857711 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42d0154340 0x7f42d015c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:22.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.325+0000 7f42d4e65640 1 -- 172.21.15.55:0/684857711 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d01f3210 con 0x7f42d0154720 2024-03-19T02:00:22.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.329+0000 7f42d4e65640 1 --2- 172.21.15.55:0/684857711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42d0154720 0x7f42d0158060 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f42c8009e50 tx=0x7f42c8019750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:22.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.333+0000 7f42ccff9640 1 -- 172.21.15.55:0/684857711 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c8009510 con 0x7f42d0154720 2024-03-19T02:00:22.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.333+0000 7f42ccff9640 1 -- 172.21.15.55:0/684857711 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42c80be070 con 0x7f42d0154720 2024-03-19T02:00:22.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.333+0000 7f42ccff9640 1 -- 172.21.15.55:0/684857711 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c80b88b0 con 0x7f42d0154720 2024-03-19T02:00:22.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.333+0000 7f42d68c8640 1 -- 172.21.15.55:0/684857711 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42d01f7220 con 0x7f42d0154720 2024-03-19T02:00:22.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.333+0000 7f42d68c8640 1 -- 172.21.15.55:0/684857711 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42d01f3790 con 0x7f42d0154720 2024-03-19T02:00:22.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.333+0000 7f42d68c8640 1 -- 172.21.15.55:0/684857711 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42d0068a40 con 0x7f42d0154720 2024-03-19T02:00:22.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.357+0000 7f42ccff9640 1 -- 172.21.15.55:0/684857711 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f42c80b8a50 con 0x7f42d0154720 2024-03-19T02:00:22.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.357+0000 7f42ccff9640 1 --2- 172.21.15.55:0/684857711 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f42bc0676e0 0x7f42bc069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:22.356 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.357+0000 7f42cffff640 1 --2- 172.21.15.55:0/684857711 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f42bc0676e0 0x7f42bc069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:22.356 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.357+0000 7f42cffff640 1 --2- 172.21.15.55:0/684857711 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f42bc0676e0 0x7f42bc069ba0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f42c00016c0 tx=0x7f42c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:22.613 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.613+0000 7f42ccff9640 1 -- 172.21.15.55:0/684857711 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f42c80ef6b0 con 0x7f42d0154720 2024-03-19T02:00:22.613 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.613+0000 7f42ccff9640 1 -- 172.21.15.55:0/684857711 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f42c80c7020 con 0x7f42d0154720 2024-03-19T02:00:22.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.821+0000 7f42d68c8640 1 -- 172.21.15.55:0/684857711 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42d00cb190 con 0x7f42d0154720 2024-03-19T02:00:22.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.897+0000 7f42ccff9640 1 -- 172.21.15.55:0/684857711 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f42c80165a0 con 0x7f42d0154720 2024-03-19T02:00:22.899 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:22.899 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:22.899 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:22.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.901+0000 7f42d68c8640 1 -- 172.21.15.55:0/684857711 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f42bc0676e0 msgr2=0x7f42bc069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:22.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.901+0000 7f42d68c8640 1 --2- 172.21.15.55:0/684857711 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f42bc0676e0 0x7f42bc069ba0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f42c00016c0 tx=0x7f42c0009040 comp rx=0 tx=0).stop 2024-03-19T02:00:22.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.901+0000 7f42d68c8640 1 -- 172.21.15.55:0/684857711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42d0154720 msgr2=0x7f42d0158060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:22.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.901+0000 7f42d68c8640 1 --2- 172.21.15.55:0/684857711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42d0154720 0x7f42d0158060 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f42c8009e50 tx=0x7f42c8019750 comp rx=0 tx=0).stop 2024-03-19T02:00:22.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.901+0000 7f42d68c8640 1 -- 172.21.15.55:0/684857711 shutdown_connections 2024-03-19T02:00:22.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.901+0000 7f42d68c8640 1 --2- 172.21.15.55:0/684857711 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f42bc0676e0 0x7f42bc069ba0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:22.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.901+0000 7f42d68c8640 1 --2- 172.21.15.55:0/684857711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f42d0154720 0x7f42d0158060 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:22.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.901+0000 7f42d68c8640 1 --2- 172.21.15.55:0/684857711 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42d0154340 0x7f42d015c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:22.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.901+0000 7f42d68c8640 1 --2- 172.21.15.55:0/684857711 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f42d00c73c0 0x7f42d014ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:22.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.901+0000 7f42d68c8640 1 -- 172.21.15.55:0/684857711 >> 172.21.15.55:0/684857711 conn(0x7f42d00cc9c0 msgr2=0x7f42d00c5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:22.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.901+0000 7f42d68c8640 1 -- 172.21.15.55:0/684857711 shutdown_connections 2024-03-19T02:00:22.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:22.901+0000 7f42d68c8640 1 -- 172.21.15.55:0/684857711 wait complete. 2024-03-19T02:00:23.523 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:23.523 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:24.524 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:24.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.637+0000 7f1ad1f1e640 1 Processor -- start 2024-03-19T02:00:24.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.637+0000 7f1ad1f1e640 1 -- start start 2024-03-19T02:00:24.639 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.641+0000 7f1ad1f1e640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f1acc1544b0 0x7f1acc06b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:24.639 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.641+0000 7f1ad1f1e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1acc0c9050 0x7f1acc0c94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:24.639 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.641+0000 7f1ad1f1e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1acc15a6b0 con 0x7f1acc15a0e0 2024-03-19T02:00:24.639 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.641+0000 7f1ad1f1e640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f1acc15a990 con 0x7f1acc1544b0 2024-03-19T02:00:24.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.641+0000 7f1ad1f1e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1acc15ac70 con 0x7f1acc0c9050 2024-03-19T02:00:24.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.641+0000 7f1acaffd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f1acc15a0e0 0x7f1acc0cbfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:45830/0 (socket says 172.21.15.55:45830) 2024-03-19T02:00:24.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.641+0000 7f1acaffd640 1 -- 172.21.15.55:0/3668009562 learned_addr learned my addr 172.21.15.55:0/3668009562 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:00:24.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.641+0000 7f1acb7fe640 1 --2- 172.21.15.55:0/3668009562 >> v2:172.21.15.84:3300/0 conn(0x7f1acc1544b0 0x7f1acc06b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:24.640 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.641+0000 7f1acbfff640 1 --2- 172.21.15.55:0/3668009562 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1acc0c9050 0x7f1acc0c94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:24.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.645+0000 7f1aca7fc640 1 -- 172.21.15.55:0/3668009562 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1441802973 0 0) 0x7f1acc15a6b0 con 0x7f1acc15a0e0 2024-03-19T02:00:24.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.645+0000 7f1aca7fc640 1 -- 172.21.15.55:0/3668009562 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ab4003530 con 0x7f1acc15a0e0 2024-03-19T02:00:24.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.645+0000 7f1acb7fe640 1 -- 172.21.15.55:0/3668009562 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1acc0c9050 msgr2=0x7f1acc0c94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:24.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.645+0000 7f1acb7fe640 1 --2- 172.21.15.55:0/3668009562 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1acc0c9050 0x7f1acc0c94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:24.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.645+0000 7f1acb7fe640 1 -- 172.21.15.55:0/3668009562 >> v1:172.21.15.55:6789/0 conn(0x7f1acc15a0e0 legacy=0x7f1acc0cbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:24.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.645+0000 7f1acb7fe640 1 -- 172.21.15.55:0/3668009562 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1acc15af50 con 0x7f1acc1544b0 2024-03-19T02:00:24.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.649+0000 7f1acb7fe640 1 --2- 172.21.15.55:0/3668009562 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1acc1544b0 0x7f1acc06b2a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0097c0 tx=0x7f1abc01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=872c4c42bdd76f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:24.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.653+0000 7f1aca7fc640 1 -- 172.21.15.55:0/3668009562 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc0b5070 con 0x7f1acc1544b0 2024-03-19T02:00:24.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.653+0000 7f1aca7fc640 1 -- 172.21.15.55:0/3668009562 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1abc0a8990 con 0x7f1acc1544b0 2024-03-19T02:00:24.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.653+0000 7f1aca7fc640 1 -- 172.21.15.55:0/3668009562 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc0a8d60 con 0x7f1acc1544b0 2024-03-19T02:00:24.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.657+0000 7f1ad1f1e640 1 -- 172.21.15.55:0/3668009562 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1acc1544b0 msgr2=0x7f1acc06b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:24.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.657+0000 7f1ad1f1e640 1 --2- 172.21.15.55:0/3668009562 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1acc1544b0 0x7f1acc06b2a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0097c0 tx=0x7f1abc01ec90 comp rx=0 tx=0).stop 2024-03-19T02:00:24.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.657+0000 7f1ad1f1e640 1 -- 172.21.15.55:0/3668009562 shutdown_connections 2024-03-19T02:00:24.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.657+0000 7f1ad1f1e640 1 --2- 172.21.15.55:0/3668009562 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1acc0c9050 0x7f1acc0c94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:24.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.657+0000 7f1ad1f1e640 1 --2- 172.21.15.55:0/3668009562 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1acc1544b0 0x7f1acc06b2a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:24.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.657+0000 7f1ad1f1e640 1 -- 172.21.15.55:0/3668009562 >> 172.21.15.55:0/3668009562 conn(0x7f1acc06e5a0 msgr2=0x7f1acc072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:24.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.657+0000 7f1ad1f1e640 1 -- 172.21.15.55:0/3668009562 shutdown_connections 2024-03-19T02:00:24.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.657+0000 7f1ad1f1e640 1 -- 172.21.15.55:0/3668009562 wait complete. 2024-03-19T02:00:24.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.657+0000 7f1ad1f1e640 1 Processor -- start 2024-03-19T02:00:24.657 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.657+0000 7f1ad1f1e640 1 -- start start 2024-03-19T02:00:24.657 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.657+0000 7f1ad1f1e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1acc0c9050 0x7f1acc14fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:24.657 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.657+0000 7f1ad1f1e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1acc1544b0 0x7f1acc1e3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:24.657 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.657+0000 7f1ad1f1e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1acc15a0e0 0x7f1acc0ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:24.657 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.657+0000 7f1ad1f1e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc06b7e0 con 0x7f1acc0c9050 2024-03-19T02:00:24.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.657+0000 7f1ad1f1e640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc0c6be0 con 0x7f1acc1544b0 2024-03-19T02:00:24.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.657+0000 7f1ad1f1e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1acc0c8760 con 0x7f1acc15a0e0 2024-03-19T02:00:24.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.657+0000 7f1acb7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1acc0c9050 0x7f1acc14fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:24.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.657+0000 7f1acaffd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1acc1544b0 0x7f1acc1e3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:24.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.657+0000 7f1acbfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1acc15a0e0 0x7f1acc0ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:24.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.657+0000 7f1acb7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1acc0c9050 0x7f1acc14fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39418/0 (socket says 172.21.15.55:39418) 2024-03-19T02:00:24.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.657+0000 7f1acb7fe640 1 -- 172.21.15.55:0/3383130439 learned_addr learned my addr 172.21.15.55:0/3383130439 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:24.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.661+0000 7f1acb7fe640 1 -- 172.21.15.55:0/3383130439 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1acc15a0e0 msgr2=0x7f1acc0ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:24.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.661+0000 7f1acb7fe640 1 --2- 172.21.15.55:0/3383130439 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1acc15a0e0 0x7f1acc0ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:24.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.661+0000 7f1acb7fe640 1 -- 172.21.15.55:0/3383130439 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1acc1544b0 msgr2=0x7f1acc1e3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:24.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.661+0000 7f1acb7fe640 1 --2- 172.21.15.55:0/3383130439 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1acc1544b0 0x7f1acc1e3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:24.662 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.661+0000 7f1acb7fe640 1 -- 172.21.15.55:0/3383130439 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1acc0cad70 con 0x7f1acc0c9050 2024-03-19T02:00:24.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.665+0000 7f1acb7fe640 1 --2- 172.21.15.55:0/3383130439 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1acc0c9050 0x7f1acc14fc50 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0b3040 tx=0x7f1abc01ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:24.668 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.669+0000 7f1ab3fff640 1 -- 172.21.15.55:0/3383130439 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc0bf070 con 0x7f1acc0c9050 2024-03-19T02:00:24.668 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.669+0000 7f1ab3fff640 1 -- 172.21.15.55:0/3383130439 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1abc0b0700 con 0x7f1acc0c9050 2024-03-19T02:00:24.668 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.669+0000 7f1ab3fff640 1 -- 172.21.15.55:0/3383130439 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc0b5040 con 0x7f1acc0c9050 2024-03-19T02:00:24.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.669+0000 7f1ad1f1e640 1 -- 172.21.15.55:0/3383130439 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1acc0cb030 con 0x7f1acc0c9050 2024-03-19T02:00:24.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.669+0000 7f1ad1f1e640 1 -- 172.21.15.55:0/3383130439 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1acc14f170 con 0x7f1acc0c9050 2024-03-19T02:00:24.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.669+0000 7f1ad1f1e640 1 -- 172.21.15.55:0/3383130439 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1acc068a40 con 0x7f1acc0c9050 2024-03-19T02:00:24.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.693+0000 7f1ab3fff640 1 -- 172.21.15.55:0/3383130439 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1abc0c2070 con 0x7f1acc0c9050 2024-03-19T02:00:24.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.697+0000 7f1ab3fff640 1 --2- 172.21.15.55:0/3383130439 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ab80676b0 0x7f1ab8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:24.696 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.697+0000 7f1acaffd640 1 --2- 172.21.15.55:0/3383130439 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ab80676b0 0x7f1ab8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:24.696 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.697+0000 7f1acaffd640 1 --2- 172.21.15.55:0/3383130439 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ab80676b0 0x7f1ab8069b70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0007fd0 tx=0x7f1ac0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:24.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.977+0000 7f1ab3fff640 1 -- 172.21.15.55:0/3383130439 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f1abc0a8910 con 0x7f1acc0c9050 2024-03-19T02:00:24.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:24.977+0000 7f1ab3fff640 1 -- 172.21.15.55:0/3383130439 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1abc0c9080 con 0x7f1acc0c9050 2024-03-19T02:00:25.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:25.173+0000 7f1ad1f1e640 1 -- 172.21.15.55:0/3383130439 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1acc06bb10 con 0x7f1acc0c9050 2024-03-19T02:00:25.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:25.197+0000 7f1ab3fff640 1 -- 172.21.15.55:0/3383130439 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f1abc0187f0 con 0x7f1acc0c9050 2024-03-19T02:00:25.198 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:25.198 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:25.198 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:25.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:25.201+0000 7f1ad1f1e640 1 -- 172.21.15.55:0/3383130439 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ab80676b0 msgr2=0x7f1ab8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:25.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:25.201+0000 7f1ad1f1e640 1 --2- 172.21.15.55:0/3383130439 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ab80676b0 0x7f1ab8069b70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0007fd0 tx=0x7f1ac0009040 comp rx=0 tx=0).stop 2024-03-19T02:00:25.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:25.201+0000 7f1ad1f1e640 1 -- 172.21.15.55:0/3383130439 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1acc0c9050 msgr2=0x7f1acc14fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:25.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:25.201+0000 7f1ad1f1e640 1 --2- 172.21.15.55:0/3383130439 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1acc0c9050 0x7f1acc14fc50 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0b3040 tx=0x7f1abc01ba30 comp rx=0 tx=0).stop 2024-03-19T02:00:25.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:25.201+0000 7f1ad1f1e640 1 -- 172.21.15.55:0/3383130439 shutdown_connections 2024-03-19T02:00:25.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:25.201+0000 7f1ad1f1e640 1 --2- 172.21.15.55:0/3383130439 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ab80676b0 0x7f1ab8069b70 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:25.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:25.201+0000 7f1ad1f1e640 1 --2- 172.21.15.55:0/3383130439 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1acc15a0e0 0x7f1acc0ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:25.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:25.201+0000 7f1ad1f1e640 1 --2- 172.21.15.55:0/3383130439 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1acc1544b0 0x7f1acc1e3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:25.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:25.201+0000 7f1ad1f1e640 1 --2- 172.21.15.55:0/3383130439 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1acc0c9050 0x7f1acc14fc50 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:25.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:25.201+0000 7f1ad1f1e640 1 -- 172.21.15.55:0/3383130439 >> 172.21.15.55:0/3383130439 conn(0x7f1acc06e5a0 msgr2=0x7f1acc158630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:25.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:25.201+0000 7f1ad1f1e640 1 -- 172.21.15.55:0/3383130439 shutdown_connections 2024-03-19T02:00:25.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:25.201+0000 7f1ad1f1e640 1 -- 172.21.15.55:0/3383130439 wait complete. 2024-03-19T02:00:25.820 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:25.821 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:26.821 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:26.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.933+0000 7fea0097d640 1 Processor -- start 2024-03-19T02:00:26.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.933+0000 7fea0097d640 1 -- start start 2024-03-19T02:00:26.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.933+0000 7fea0097d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe9fc0c96b0 0x7fe9fc070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:26.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.933+0000 7fea0097d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fe9fc0c9a90 0x7fe9fc0c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:26.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.933+0000 7fea0097d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9fc15e3a0 con 0x7fe9fc1540c0 2024-03-19T02:00:26.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.933+0000 7fea0097d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fe9fc15e680 con 0x7fe9fc0c96b0 2024-03-19T02:00:26.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.933+0000 7fea0097d640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fe9fc15a330 con 0x7fe9fc0c9a90 2024-03-19T02:00:26.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.937+0000 7fe9fa59c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe9fc0c96b0 0x7fe9fc070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:26.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.937+0000 7fe9f9d9b640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fe9fc0c9a90 0x7fe9fc0c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:26.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.937+0000 7fe9fa59c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe9fc0c96b0 0x7fe9fc070f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39426/0 (socket says 172.21.15.55:39426) 2024-03-19T02:00:26.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.937+0000 7fe9fa59c640 1 -- 172.21.15.55:0/2511965668 learned_addr learned my addr 172.21.15.55:0/2511965668 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:26.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.937+0000 7fe9fa59c640 1 -- 172.21.15.55:0/2511965668 >> v2:172.21.15.84:3300/0 conn(0x7fe9fc0c9a90 msgr2=0x7fe9fc0c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:26.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.937+0000 7fe9fa59c640 1 --2- 172.21.15.55:0/2511965668 >> v2:172.21.15.84:3300/0 conn(0x7fe9fc0c9a90 0x7fe9fc0c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:26.939 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.937+0000 7fe9fa59c640 1 -- 172.21.15.55:0/2511965668 >> v1:172.21.15.84:6789/0 conn(0x7fe9fc1540c0 legacy=0x7fe9fc15d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:26.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.937+0000 7fe9fa59c640 1 -- 172.21.15.55:0/2511965668 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9fc15a610 con 0x7fe9fc0c96b0 2024-03-19T02:00:26.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.945+0000 7fe9fa59c640 1 --2- 172.21.15.55:0/2511965668 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9fc0c96b0 0x7fe9fc070f20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc0c9420 tx=0x7fe9ec01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2301d67a0f73a62 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:26.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.949+0000 7fe9f959a640 1 -- 172.21.15.55:0/2511965668 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec0b0070 con 0x7fe9fc0c96b0 2024-03-19T02:00:26.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.949+0000 7fe9f959a640 1 -- 172.21.15.55:0/2511965668 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9ec0ab600 con 0x7fe9fc0c96b0 2024-03-19T02:00:26.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.949+0000 7fe9f959a640 1 -- 172.21.15.55:0/2511965668 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec0ab9d0 con 0x7fe9fc0c96b0 2024-03-19T02:00:26.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.949+0000 7fea0097d640 1 -- 172.21.15.55:0/2511965668 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9fc0c96b0 msgr2=0x7fe9fc070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:26.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.949+0000 7fea0097d640 1 --2- 172.21.15.55:0/2511965668 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9fc0c96b0 0x7fe9fc070f20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc0c9420 tx=0x7fe9ec01c450 comp rx=0 tx=0).stop 2024-03-19T02:00:26.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.949+0000 7fea0097d640 1 -- 172.21.15.55:0/2511965668 shutdown_connections 2024-03-19T02:00:26.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.949+0000 7fea0097d640 1 --2- 172.21.15.55:0/2511965668 >> v2:172.21.15.84:3300/0 conn(0x7fe9fc0c9a90 0x7fe9fc0c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:26.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.949+0000 7fea0097d640 1 --2- 172.21.15.55:0/2511965668 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9fc0c96b0 0x7fe9fc070f20 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:26.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.949+0000 7fea0097d640 1 -- 172.21.15.55:0/2511965668 >> 172.21.15.55:0/2511965668 conn(0x7fe9fc0cc9c0 msgr2=0x7fe9fc0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:26.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.949+0000 7fea0097d640 1 -- 172.21.15.55:0/2511965668 shutdown_connections 2024-03-19T02:00:26.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.949+0000 7fea0097d640 1 -- 172.21.15.55:0/2511965668 wait complete. 2024-03-19T02:00:26.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.949+0000 7fea0097d640 1 Processor -- start 2024-03-19T02:00:26.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.949+0000 7fea0097d640 1 -- start start 2024-03-19T02:00:26.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.953+0000 7fea0097d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9fc0c96b0 0x7fe9fc1f6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:26.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.953+0000 7fea0097d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe9fc0c9a90 0x7fe9fc14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:26.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.953+0000 7fea0097d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9fc1540c0 0x7fe9fc159470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:26.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.953+0000 7fea0097d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc0c7900 con 0x7fe9fc0c96b0 2024-03-19T02:00:26.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.953+0000 7fea0097d640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc15b0f0 con 0x7fe9fc1540c0 2024-03-19T02:00:26.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.953+0000 7fea0097d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fe9fc0c7a80 con 0x7fe9fc0c9a90 2024-03-19T02:00:26.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.953+0000 7fe9fa59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9fc0c96b0 0x7fe9fc1f6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:26.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.953+0000 7fe9f9d9b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe9fc0c9a90 0x7fe9fc14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:26.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.953+0000 7fe9fa59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9fc0c96b0 0x7fe9fc1f6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39434/0 (socket says 172.21.15.55:39434) 2024-03-19T02:00:26.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.953+0000 7fe9fa59c640 1 -- 172.21.15.55:0/3388561771 learned_addr learned my addr 172.21.15.55:0/3388561771 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:26.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.953+0000 7fe9fad9d640 1 --2- 172.21.15.55:0/3388561771 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9fc1540c0 0x7fe9fc159470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:26.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.957+0000 7fe9fa59c640 1 -- 172.21.15.55:0/3388561771 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe9fc0c9a90 msgr2=0x7fe9fc14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:26.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.957+0000 7fe9fa59c640 1 --2- 172.21.15.55:0/3388561771 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe9fc0c9a90 0x7fe9fc14efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:26.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.957+0000 7fe9fa59c640 1 -- 172.21.15.55:0/3388561771 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9fc1540c0 msgr2=0x7fe9fc159470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:26.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.957+0000 7fe9fa59c640 1 --2- 172.21.15.55:0/3388561771 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9fc1540c0 0x7fe9fc159470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:26.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.957+0000 7fe9fa59c640 1 -- 172.21.15.55:0/3388561771 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9fc150630 con 0x7fe9fc0c96b0 2024-03-19T02:00:26.959 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.961+0000 7fe9fa59c640 1 --2- 172.21.15.55:0/3388561771 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9fc0c96b0 0x7fe9fc1f6620 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec01ab00 tx=0x7fe9ec0191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:26.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.965+0000 7fe9eaffd640 1 -- 172.21.15.55:0/3388561771 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec0bb070 con 0x7fe9fc0c96b0 2024-03-19T02:00:26.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.965+0000 7fe9eaffd640 1 -- 172.21.15.55:0/3388561771 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9ec0b6440 con 0x7fe9fc0c96b0 2024-03-19T02:00:26.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.965+0000 7fe9eaffd640 1 -- 172.21.15.55:0/3388561771 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec0b0040 con 0x7fe9fc0c96b0 2024-03-19T02:00:26.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.965+0000 7fea0097d640 1 -- 172.21.15.55:0/3388561771 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9fc150910 con 0x7fe9fc0c96b0 2024-03-19T02:00:26.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.965+0000 7fea0097d640 1 -- 172.21.15.55:0/3388561771 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9fc14e7e0 con 0x7fe9fc0c96b0 2024-03-19T02:00:26.964 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:26.965+0000 7fea0097d640 1 -- 172.21.15.55:0/3388561771 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9fc068a40 con 0x7fe9fc0c96b0 2024-03-19T02:00:27.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:27.245+0000 7fe9eaffd640 1 -- 172.21.15.55:0/3388561771 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fe9ec0be070 con 0x7fe9fc0c96b0 2024-03-19T02:00:27.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:27.245+0000 7fe9eaffd640 1 --2- 172.21.15.55:0/3388561771 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe9dc0676b0 0x7fe9dc069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:27.246 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:27.245+0000 7fe9f9d9b640 1 --2- 172.21.15.55:0/3388561771 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe9dc0676b0 0x7fe9dc069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:27.247 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:27.249+0000 7fe9f9d9b640 1 --2- 172.21.15.55:0/3388561771 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe9dc0676b0 0x7fe9dc069b70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc157f50 tx=0x7fe9f0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:27.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:27.269+0000 7fe9eaffd640 1 -- 172.21.15.55:0/3388561771 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fe9ec0ed860 con 0x7fe9fc0c96b0 2024-03-19T02:00:27.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:27.273+0000 7fe9eaffd640 1 -- 172.21.15.55:0/3388561771 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fe9ec0160d0 con 0x7fe9fc0c96b0 2024-03-19T02:00:27.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:27.481+0000 7fea0097d640 1 -- 172.21.15.55:0/3388561771 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9fc0c6b80 con 0x7fe9fc0c96b0 2024-03-19T02:00:27.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:27.517+0000 7fe9eaffd640 1 -- 172.21.15.55:0/3388561771 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fe9ec0ef7c0 con 0x7fe9fc0c96b0 2024-03-19T02:00:27.517 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:27.517 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:27.518 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:27.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:27.521+0000 7fea0097d640 1 -- 172.21.15.55:0/3388561771 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe9dc0676b0 msgr2=0x7fe9dc069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:27.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:27.521+0000 7fea0097d640 1 --2- 172.21.15.55:0/3388561771 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe9dc0676b0 0x7fe9dc069b70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc157f50 tx=0x7fe9f0009190 comp rx=0 tx=0).stop 2024-03-19T02:00:27.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:27.521+0000 7fea0097d640 1 -- 172.21.15.55:0/3388561771 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9fc0c96b0 msgr2=0x7fe9fc1f6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:27.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:27.521+0000 7fea0097d640 1 --2- 172.21.15.55:0/3388561771 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9fc0c96b0 0x7fe9fc1f6620 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec01ab00 tx=0x7fe9ec0191f0 comp rx=0 tx=0).stop 2024-03-19T02:00:27.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:27.521+0000 7fea0097d640 1 -- 172.21.15.55:0/3388561771 shutdown_connections 2024-03-19T02:00:27.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:27.521+0000 7fea0097d640 1 --2- 172.21.15.55:0/3388561771 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe9dc0676b0 0x7fe9dc069b70 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:27.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:27.521+0000 7fea0097d640 1 --2- 172.21.15.55:0/3388561771 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9fc1540c0 0x7fe9fc159470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:27.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:27.521+0000 7fea0097d640 1 --2- 172.21.15.55:0/3388561771 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe9fc0c9a90 0x7fe9fc14efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:27.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:27.521+0000 7fea0097d640 1 --2- 172.21.15.55:0/3388561771 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9fc0c96b0 0x7fe9fc1f6620 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:27.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:27.521+0000 7fea0097d640 1 -- 172.21.15.55:0/3388561771 >> 172.21.15.55:0/3388561771 conn(0x7fe9fc0cc9c0 msgr2=0x7fe9fc0c5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:27.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:27.521+0000 7fea0097d640 1 -- 172.21.15.55:0/3388561771 shutdown_connections 2024-03-19T02:00:27.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:27.521+0000 7fea0097d640 1 -- 172.21.15.55:0/3388561771 wait complete. 2024-03-19T02:00:28.142 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:28.142 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:29.143 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:29.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.273+0000 7f4e10090640 1 Processor -- start 2024-03-19T02:00:29.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.273+0000 7f4e10090640 1 -- start start 2024-03-19T02:00:29.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.273+0000 7f4e10090640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4e080ca350 0x7f4e0815b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:29.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.273+0000 7f4e10090640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f4e080c3320 0x7f4e080728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:29.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.273+0000 7f4e10090640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e08151330 con 0x7f4e08150f50 2024-03-19T02:00:29.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.273+0000 7f4e10090640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f4e080c3e00 con 0x7f4e080ca350 2024-03-19T02:00:29.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.273+0000 7f4e10090640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f4e0815ac10 con 0x7f4e080c3320 2024-03-19T02:00:29.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.277+0000 7f4e0d62b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4e080ca350 0x7f4e0815b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:29.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.277+0000 7f4e0e62d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f4e080c3320 0x7f4e080728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:29.276 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.277+0000 7f4e0de2c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f4e08150f50 0x7f4e08157920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:54076/0 (socket says 172.21.15.55:54076) 2024-03-19T02:00:29.276 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.277+0000 7f4e0de2c640 1 -- 172.21.15.55:0/3214815249 learned_addr learned my addr 172.21.15.55:0/3214815249 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:00:29.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.281+0000 7f4e0e62d640 1 -- 172.21.15.55:0/3214815249 >> v2:172.21.15.55:3300/0 conn(0x7f4e080ca350 msgr2=0x7f4e0815b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:29.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.281+0000 7f4e0e62d640 1 --2- 172.21.15.55:0/3214815249 >> v2:172.21.15.55:3300/0 conn(0x7f4e080ca350 0x7f4e0815b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:29.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.281+0000 7f4e0e62d640 1 -- 172.21.15.55:0/3214815249 >> v1:172.21.15.55:6789/0 conn(0x7f4e08150f50 legacy=0x7f4e08157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:29.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.281+0000 7f4e0e62d640 1 -- 172.21.15.55:0/3214815249 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e080c3840 con 0x7f4e080c3320 2024-03-19T02:00:29.284 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.285+0000 7f4e0e62d640 1 --2- 172.21.15.55:0/3214815249 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e080c3320 0x7f4e080728b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4e0815ab80 tx=0x7f4e0001c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b25c733eddbf100 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:29.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.289+0000 7f4e0ce2a640 1 -- 172.21.15.55:0/3214815249 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e000b2070 con 0x7f4e080c3320 2024-03-19T02:00:29.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.289+0000 7f4e0ce2a640 1 -- 172.21.15.55:0/3214815249 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e000a59f0 con 0x7f4e080c3320 2024-03-19T02:00:29.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.289+0000 7f4e0ce2a640 1 -- 172.21.15.55:0/3214815249 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e000a5dc0 con 0x7f4e080c3320 2024-03-19T02:00:29.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.289+0000 7f4e10090640 1 -- 172.21.15.55:0/3214815249 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e080c3320 msgr2=0x7f4e080728b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:29.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.289+0000 7f4e10090640 1 --2- 172.21.15.55:0/3214815249 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e080c3320 0x7f4e080728b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4e0815ab80 tx=0x7f4e0001c930 comp rx=0 tx=0).stop 2024-03-19T02:00:29.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.289+0000 7f4e10090640 1 -- 172.21.15.55:0/3214815249 shutdown_connections 2024-03-19T02:00:29.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.289+0000 7f4e10090640 1 --2- 172.21.15.55:0/3214815249 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e080c3320 0x7f4e080728b0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:29.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.289+0000 7f4e10090640 1 --2- 172.21.15.55:0/3214815249 >> v2:172.21.15.55:3300/0 conn(0x7f4e080ca350 0x7f4e0815b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:29.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.289+0000 7f4e10090640 1 -- 172.21.15.55:0/3214815249 >> 172.21.15.55:0/3214815249 conn(0x7f4e0806e670 msgr2=0x7f4e0806ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:29.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.289+0000 7f4e10090640 1 -- 172.21.15.55:0/3214815249 shutdown_connections 2024-03-19T02:00:29.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.289+0000 7f4e10090640 1 -- 172.21.15.55:0/3214815249 wait complete. 2024-03-19T02:00:29.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.293+0000 7f4e10090640 1 Processor -- start 2024-03-19T02:00:29.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.293+0000 7f4e10090640 1 -- start start 2024-03-19T02:00:29.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.293+0000 7f4e10090640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e080c3320 0x7f4e08159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:29.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.293+0000 7f4e10090640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4e080ca350 0x7f4e081f1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:29.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.293+0000 7f4e10090640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e08150f50 0x7f4e081f2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:29.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.293+0000 7f4e10090640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4e081639b0 con 0x7f4e08150f50 2024-03-19T02:00:29.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.293+0000 7f4e10090640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4e08163830 con 0x7f4e080c3320 2024-03-19T02:00:29.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.293+0000 7f4e10090640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4e08163b30 con 0x7f4e080ca350 2024-03-19T02:00:29.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.293+0000 7f4e0de2c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e080c3320 0x7f4e08159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:29.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.293+0000 7f4e0e62d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e08150f50 0x7f4e081f2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:29.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.293+0000 7f4e0e62d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e08150f50 0x7f4e081f2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41952/0 (socket says 172.21.15.55:41952) 2024-03-19T02:00:29.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.293+0000 7f4e0e62d640 1 -- 172.21.15.55:0/4292078915 learned_addr learned my addr 172.21.15.55:0/4292078915 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:29.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.293+0000 7f4e0d62b640 1 --2- 172.21.15.55:0/4292078915 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4e080ca350 0x7f4e081f1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:29.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.297+0000 7f4e0e62d640 1 -- 172.21.15.55:0/4292078915 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4e080ca350 msgr2=0x7f4e081f1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:29.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.297+0000 7f4e0e62d640 1 --2- 172.21.15.55:0/4292078915 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4e080ca350 0x7f4e081f1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:29.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.297+0000 7f4e0e62d640 1 -- 172.21.15.55:0/4292078915 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e080c3320 msgr2=0x7f4e08159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:29.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.297+0000 7f4e0e62d640 1 --2- 172.21.15.55:0/4292078915 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e080c3320 0x7f4e08159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:29.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.297+0000 7f4e0e62d640 1 -- 172.21.15.55:0/4292078915 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e081456b0 con 0x7f4e08150f50 2024-03-19T02:00:29.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.301+0000 7f4e0e62d640 1 --2- 172.21.15.55:0/4292078915 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e08150f50 0x7f4e081f2400 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f4e00019540 tx=0x7f4e000196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:29.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.305+0000 7f4dfe7fc640 1 -- 172.21.15.55:0/4292078915 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e000bb030 con 0x7f4e08150f50 2024-03-19T02:00:29.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.305+0000 7f4dfe7fc640 1 -- 172.21.15.55:0/4292078915 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e000ad440 con 0x7f4e08150f50 2024-03-19T02:00:29.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.305+0000 7f4e10090640 1 -- 172.21.15.55:0/4292078915 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e08146620 con 0x7f4e08150f50 2024-03-19T02:00:29.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.305+0000 7f4dfe7fc640 1 -- 172.21.15.55:0/4292078915 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e000b2040 con 0x7f4e08150f50 2024-03-19T02:00:29.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.305+0000 7f4e10090640 1 -- 172.21.15.55:0/4292078915 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e08146be0 con 0x7f4e08150f50 2024-03-19T02:00:29.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.305+0000 7f4e10090640 1 -- 172.21.15.55:0/4292078915 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e08068a40 con 0x7f4e08150f50 2024-03-19T02:00:29.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.337+0000 7f4dfe7fc640 1 -- 172.21.15.55:0/4292078915 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f4e000be070 con 0x7f4e08150f50 2024-03-19T02:00:29.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.341+0000 7f4dfe7fc640 1 --2- 172.21.15.55:0/4292078915 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4dec067660 0x7f4dec069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:29.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.341+0000 7f4e0de2c640 1 --2- 172.21.15.55:0/4292078915 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4dec067660 0x7f4dec069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:29.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.341+0000 7f4e0de2c640 1 --2- 172.21.15.55:0/4292078915 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4dec067660 0x7f4dec069b20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4e04000f70 tx=0x7f4e04007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:29.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.625+0000 7f4dfe7fc640 1 -- 172.21.15.55:0/4292078915 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f4e000a5970 con 0x7f4e08150f50 2024-03-19T02:00:29.625 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.625+0000 7f4dfe7fc640 1 -- 172.21.15.55:0/4292078915 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4e00016520 con 0x7f4e08150f50 2024-03-19T02:00:29.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.893+0000 7f4e10090640 1 -- 172.21.15.55:0/4292078915 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e0814ead0 con 0x7f4e08150f50 2024-03-19T02:00:29.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.933+0000 7f4dfe7fc640 1 -- 172.21.15.55:0/4292078915 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f4e000f0d00 con 0x7f4e08150f50 2024-03-19T02:00:29.932 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:29.932 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:29.932 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:29.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.933+0000 7f4e10090640 1 -- 172.21.15.55:0/4292078915 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4dec067660 msgr2=0x7f4dec069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:29.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.933+0000 7f4e10090640 1 --2- 172.21.15.55:0/4292078915 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4dec067660 0x7f4dec069b20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4e04000f70 tx=0x7f4e04007e90 comp rx=0 tx=0).stop 2024-03-19T02:00:29.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.933+0000 7f4e10090640 1 -- 172.21.15.55:0/4292078915 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e08150f50 msgr2=0x7f4e081f2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:29.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.933+0000 7f4e10090640 1 --2- 172.21.15.55:0/4292078915 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e08150f50 0x7f4e081f2400 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f4e00019540 tx=0x7f4e000196d0 comp rx=0 tx=0).stop 2024-03-19T02:00:29.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.933+0000 7f4e10090640 1 -- 172.21.15.55:0/4292078915 shutdown_connections 2024-03-19T02:00:29.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.933+0000 7f4e10090640 1 --2- 172.21.15.55:0/4292078915 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4dec067660 0x7f4dec069b20 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:29.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.933+0000 7f4e10090640 1 --2- 172.21.15.55:0/4292078915 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e08150f50 0x7f4e081f2400 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:29.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.933+0000 7f4e10090640 1 --2- 172.21.15.55:0/4292078915 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4e080ca350 0x7f4e081f1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:29.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.933+0000 7f4e10090640 1 --2- 172.21.15.55:0/4292078915 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e080c3320 0x7f4e08159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:29.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.933+0000 7f4e10090640 1 -- 172.21.15.55:0/4292078915 >> 172.21.15.55:0/4292078915 conn(0x7f4e0806e670 msgr2=0x7f4e08148120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:29.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.937+0000 7f4e10090640 1 -- 172.21.15.55:0/4292078915 shutdown_connections 2024-03-19T02:00:29.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:29.937+0000 7f4e10090640 1 -- 172.21.15.55:0/4292078915 wait complete. 2024-03-19T02:00:30.558 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:30.558 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:31.560 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:31.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.676+0000 7f94c6f51640 1 Processor -- start 2024-03-19T02:00:31.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.676+0000 7f94c6f51640 1 -- start start 2024-03-19T02:00:31.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.676+0000 7f94c6f51640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f94c006c610 0x7f94c015ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:31.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.676+0000 7f94c6f51640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f94c006c9f0 0x7f94c00c92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:31.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.676+0000 7f94c6f51640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94c0070e80 con 0x7f94c00c9800 2024-03-19T02:00:31.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.676+0000 7f94c6f51640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f94c015b250 con 0x7f94c006c610 2024-03-19T02:00:31.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.676+0000 7f94c6f51640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f94c006fb00 con 0x7f94c006c9f0 2024-03-19T02:00:31.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.680+0000 7f94c4ced640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f94c006c610 0x7f94c015ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:31.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.680+0000 7f94bffff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f94c006c9f0 0x7f94c00c92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:31.681 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.680+0000 7f94bffff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f94c006c9f0 0x7f94c00c92c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:34714/0 (socket says 172.21.15.55:34714) 2024-03-19T02:00:31.681 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.680+0000 7f94bffff640 1 -- 172.21.15.55:0/3929756648 learned_addr learned my addr 172.21.15.55:0/3929756648 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:31.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.684+0000 7f94bffff640 1 -- 172.21.15.55:0/3929756648 >> v2:172.21.15.55:3300/0 conn(0x7f94c006c610 msgr2=0x7f94c015ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:31.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.684+0000 7f94bffff640 1 --2- 172.21.15.55:0/3929756648 >> v2:172.21.15.55:3300/0 conn(0x7f94c006c610 0x7f94c015ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:31.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.684+0000 7f94bffff640 1 -- 172.21.15.55:0/3929756648 >> v1:172.21.15.55:6789/0 conn(0x7f94c00c9800 legacy=0x7f94c01427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:31.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.684+0000 7f94bffff640 1 -- 172.21.15.55:0/3929756648 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94c006fde0 con 0x7f94c006c9f0 2024-03-19T02:00:31.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.684+0000 7f94bffff640 1 --2- 172.21.15.55:0/3929756648 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f94c006c9f0 0x7f94c00c92c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f94ac009870 tx=0x7f94ac01ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90b012e96c9553f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:31.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.692+0000 7f94bf7fe640 1 -- 172.21.15.55:0/3929756648 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac0b5070 con 0x7f94c006c9f0 2024-03-19T02:00:31.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.692+0000 7f94bf7fe640 1 -- 172.21.15.55:0/3929756648 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94ac0a8a50 con 0x7f94c006c9f0 2024-03-19T02:00:31.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.692+0000 7f94bf7fe640 1 -- 172.21.15.55:0/3929756648 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac0a8e20 con 0x7f94c006c9f0 2024-03-19T02:00:31.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.692+0000 7f94c6f51640 1 -- 172.21.15.55:0/3929756648 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f94c006c9f0 msgr2=0x7f94c00c92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:31.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.692+0000 7f94c6f51640 1 --2- 172.21.15.55:0/3929756648 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f94c006c9f0 0x7f94c00c92c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f94ac009870 tx=0x7f94ac01ed50 comp rx=0 tx=0).stop 2024-03-19T02:00:31.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.692+0000 7f94c6f51640 1 -- 172.21.15.55:0/3929756648 shutdown_connections 2024-03-19T02:00:31.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.692+0000 7f94c6f51640 1 --2- 172.21.15.55:0/3929756648 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f94c006c9f0 0x7f94c00c92c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:31.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.692+0000 7f94c6f51640 1 --2- 172.21.15.55:0/3929756648 >> v2:172.21.15.55:3300/0 conn(0x7f94c006c610 0x7f94c015ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:31.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.692+0000 7f94c6f51640 1 -- 172.21.15.55:0/3929756648 >> 172.21.15.55:0/3929756648 conn(0x7f94c0067600 msgr2=0x7f94c0071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:31.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.692+0000 7f94c6f51640 1 -- 172.21.15.55:0/3929756648 shutdown_connections 2024-03-19T02:00:31.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.692+0000 7f94c6f51640 1 -- 172.21.15.55:0/3929756648 wait complete. 2024-03-19T02:00:31.696 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.696+0000 7f94c6f51640 1 Processor -- start 2024-03-19T02:00:31.696 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.696+0000 7f94c6f51640 1 -- start start 2024-03-19T02:00:31.697 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.696+0000 7f94c6f51640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f94c006c610 0x7f94c0141160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:31.697 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.696+0000 7f94c6f51640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f94c006c9f0 0x7f94c01f07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:31.697 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.696+0000 7f94c6f51640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94c00c9800 0x7f94c00c4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:31.697 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.696+0000 7f94c6f51640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f94c0070680 con 0x7f94c00c9800 2024-03-19T02:00:31.697 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.696+0000 7f94c6f51640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f94c00703a0 con 0x7f94c006c610 2024-03-19T02:00:31.697 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.696+0000 7f94c6f51640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f94c0070960 con 0x7f94c006c9f0 2024-03-19T02:00:31.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.696+0000 7f94c54ee640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94c00c9800 0x7f94c00c4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:31.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.696+0000 7f94c54ee640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94c00c9800 0x7f94c00c4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41972/0 (socket says 172.21.15.55:41972) 2024-03-19T02:00:31.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.696+0000 7f94c54ee640 1 -- 172.21.15.55:0/3444693819 learned_addr learned my addr 172.21.15.55:0/3444693819 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:31.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.696+0000 7f94c4ced640 1 --2- 172.21.15.55:0/3444693819 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f94c006c610 0x7f94c0141160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:31.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.700+0000 7f94bffff640 1 --2- 172.21.15.55:0/3444693819 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f94c006c9f0 0x7f94c01f07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:31.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.700+0000 7f94c54ee640 1 -- 172.21.15.55:0/3444693819 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f94c006c9f0 msgr2=0x7f94c01f07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:31.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.700+0000 7f94c54ee640 1 --2- 172.21.15.55:0/3444693819 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f94c006c9f0 0x7f94c01f07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:31.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.700+0000 7f94c54ee640 1 -- 172.21.15.55:0/3444693819 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f94c006c610 msgr2=0x7f94c0141160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:31.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.700+0000 7f94c54ee640 1 --2- 172.21.15.55:0/3444693819 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f94c006c610 0x7f94c0141160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:31.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.700+0000 7f94c54ee640 1 -- 172.21.15.55:0/3444693819 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94c01416a0 con 0x7f94c00c9800 2024-03-19T02:00:31.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.704+0000 7f94c54ee640 1 --2- 172.21.15.55:0/3444693819 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94c00c9800 0x7f94c00c4060 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f94b800c760 tx=0x7f94b800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:31.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.708+0000 7f94bd7fa640 1 -- 172.21.15.55:0/3444693819 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b8003b80 con 0x7f94c00c9800 2024-03-19T02:00:31.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.708+0000 7f94bd7fa640 1 -- 172.21.15.55:0/3444693819 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94b8003d20 con 0x7f94c00c9800 2024-03-19T02:00:31.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.708+0000 7f94bd7fa640 1 -- 172.21.15.55:0/3444693819 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b800aa90 con 0x7f94c00c9800 2024-03-19T02:00:31.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.712+0000 7f94c6f51640 1 -- 172.21.15.55:0/3444693819 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94c01f0cf0 con 0x7f94c00c9800 2024-03-19T02:00:31.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.712+0000 7f94c6f51640 1 -- 172.21.15.55:0/3444693819 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94c00c38f0 con 0x7f94c00c9800 2024-03-19T02:00:31.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.712+0000 7f94c6f51640 1 -- 172.21.15.55:0/3444693819 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94c0068a40 con 0x7f94c00c9800 2024-03-19T02:00:31.991 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.988+0000 7f94bd7fa640 1 -- 172.21.15.55:0/3444693819 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f94b800ac30 con 0x7f94c00c9800 2024-03-19T02:00:31.991 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.992+0000 7f94bd7fa640 1 --2- 172.21.15.55:0/3444693819 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f94a40676b0 0x7f94a4069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:31.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.992+0000 7f94c4ced640 1 --2- 172.21.15.55:0/3444693819 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f94a40676b0 0x7f94a4069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:31.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:31.992+0000 7f94c4ced640 1 --2- 172.21.15.55:0/3444693819 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f94a40676b0 0x7f94a4069b70 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f94b4009b60 tx=0x7f94b4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:32.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:32.012+0000 7f94bd7fa640 1 -- 172.21.15.55:0/3444693819 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f94b8054710 con 0x7f94c00c9800 2024-03-19T02:00:32.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:32.012+0000 7f94bd7fa640 1 -- 172.21.15.55:0/3444693819 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f94b8056670 con 0x7f94c00c9800 2024-03-19T02:00:32.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:32.220+0000 7f94c6f51640 1 -- 172.21.15.55:0/3444693819 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94c0074440 con 0x7f94c00c9800 2024-03-19T02:00:32.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:32.252+0000 7f94bd7fa640 1 -- 172.21.15.55:0/3444693819 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f94b805a710 con 0x7f94c00c9800 2024-03-19T02:00:32.255 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:32.255 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:32.255 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:32.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:32.256+0000 7f94c6f51640 1 -- 172.21.15.55:0/3444693819 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f94a40676b0 msgr2=0x7f94a4069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:32.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:32.256+0000 7f94c6f51640 1 --2- 172.21.15.55:0/3444693819 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f94a40676b0 0x7f94a4069b70 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f94b4009b60 tx=0x7f94b4009190 comp rx=0 tx=0).stop 2024-03-19T02:00:32.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:32.256+0000 7f94c6f51640 1 -- 172.21.15.55:0/3444693819 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94c00c9800 msgr2=0x7f94c00c4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:32.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:32.256+0000 7f94c6f51640 1 --2- 172.21.15.55:0/3444693819 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94c00c9800 0x7f94c00c4060 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f94b800c760 tx=0x7f94b800cc20 comp rx=0 tx=0).stop 2024-03-19T02:00:32.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:32.256+0000 7f94c6f51640 1 -- 172.21.15.55:0/3444693819 shutdown_connections 2024-03-19T02:00:32.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:32.256+0000 7f94c6f51640 1 --2- 172.21.15.55:0/3444693819 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f94a40676b0 0x7f94a4069b70 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:32.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:32.256+0000 7f94c6f51640 1 --2- 172.21.15.55:0/3444693819 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94c00c9800 0x7f94c00c4060 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:32.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:32.256+0000 7f94c6f51640 1 --2- 172.21.15.55:0/3444693819 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f94c006c9f0 0x7f94c01f07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:32.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:32.256+0000 7f94c6f51640 1 --2- 172.21.15.55:0/3444693819 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f94c006c610 0x7f94c0141160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:32.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:32.256+0000 7f94c6f51640 1 -- 172.21.15.55:0/3444693819 >> 172.21.15.55:0/3444693819 conn(0x7f94c0067600 msgr2=0x7f94c0158940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:32.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:32.256+0000 7f94c6f51640 1 -- 172.21.15.55:0/3444693819 shutdown_connections 2024-03-19T02:00:32.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:32.256+0000 7f94c6f51640 1 -- 172.21.15.55:0/3444693819 wait complete. 2024-03-19T02:00:32.877 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:32.878 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:33.878 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:34.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.000+0000 7f9f6d8d6640 1 Processor -- start 2024-03-19T02:00:34.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.000+0000 7f9f6d8d6640 1 -- start start 2024-03-19T02:00:34.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.000+0000 7f9f6d8d6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9f68152120 0x7f9f680c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:34.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.000+0000 7f9f6d8d6640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f9f68070780 0x7f9f681586e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:34.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.000+0000 7f9f6d8d6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f6806fff0 con 0x7f9f680c9610 2024-03-19T02:00:34.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.000+0000 7f9f6d8d6640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f9f68152500 con 0x7f9f68070780 2024-03-19T02:00:34.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.000+0000 7f9f6d8d6640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f9f68070590 con 0x7f9f68152120 2024-03-19T02:00:34.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.000+0000 7f9f66ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9f68152120 0x7f9f680c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:34.003 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.000+0000 7f9f667fc640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f9f680c9610 0x7f9f68157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:54100/0 (socket says 172.21.15.55:54100) 2024-03-19T02:00:34.003 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.000+0000 7f9f667fc640 1 -- 172.21.15.55:0/1937599454 learned_addr learned my addr 172.21.15.55:0/1937599454 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:00:34.003 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.004+0000 7f9f677fe640 1 --2- 172.21.15.55:0/1937599454 >> v2:172.21.15.84:3300/0 conn(0x7f9f68070780 0x7f9f681586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:34.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.004+0000 7f9f66ffd640 1 -- 172.21.15.55:0/1937599454 >> v2:172.21.15.84:3300/0 conn(0x7f9f68070780 msgr2=0x7f9f681586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:34.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.004+0000 7f9f66ffd640 1 --2- 172.21.15.55:0/1937599454 >> v2:172.21.15.84:3300/0 conn(0x7f9f68070780 0x7f9f681586e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:34.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.004+0000 7f9f66ffd640 1 -- 172.21.15.55:0/1937599454 >> v1:172.21.15.55:6789/0 conn(0x7f9f680c9610 legacy=0x7f9f68157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:34.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.004+0000 7f9f65ffb640 1 -- 172.21.15.55:0/1937599454 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1819593655 0 0) 0x7f9f6806fff0 con 0x7f9f680c9610 2024-03-19T02:00:34.007 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.004+0000 7f9f66ffd640 1 -- 172.21.15.55:0/1937599454 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f680c39c0 con 0x7f9f68152120 2024-03-19T02:00:34.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.012+0000 7f9f66ffd640 1 --2- 172.21.15.55:0/1937599454 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9f68152120 0x7f9f680c90d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9f580097c0 tx=0x7f9f5801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51a963d0c90c27c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:34.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.016+0000 7f9f65ffb640 1 -- 172.21.15.55:0/1937599454 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f580b0070 con 0x7f9f68152120 2024-03-19T02:00:34.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.016+0000 7f9f65ffb640 1 -- 172.21.15.55:0/1937599454 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f580ab600 con 0x7f9f68152120 2024-03-19T02:00:34.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.016+0000 7f9f65ffb640 1 -- 172.21.15.55:0/1937599454 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f580ab9d0 con 0x7f9f68152120 2024-03-19T02:00:34.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.016+0000 7f9f6d8d6640 1 -- 172.21.15.55:0/1937599454 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9f68152120 msgr2=0x7f9f680c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:34.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.016+0000 7f9f6d8d6640 1 --2- 172.21.15.55:0/1937599454 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9f68152120 0x7f9f680c90d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9f580097c0 tx=0x7f9f5801c4a0 comp rx=0 tx=0).stop 2024-03-19T02:00:34.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.016+0000 7f9f6d8d6640 1 -- 172.21.15.55:0/1937599454 shutdown_connections 2024-03-19T02:00:34.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.016+0000 7f9f6d8d6640 1 --2- 172.21.15.55:0/1937599454 >> v2:172.21.15.84:3300/0 conn(0x7f9f68070780 0x7f9f681586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:34.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.016+0000 7f9f6d8d6640 1 --2- 172.21.15.55:0/1937599454 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9f68152120 0x7f9f680c90d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:34.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.016+0000 7f9f6d8d6640 1 -- 172.21.15.55:0/1937599454 >> 172.21.15.55:0/1937599454 conn(0x7f9f68067600 msgr2=0x7f9f68071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:34.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.016+0000 7f9f6d8d6640 1 -- 172.21.15.55:0/1937599454 shutdown_connections 2024-03-19T02:00:34.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.016+0000 7f9f6d8d6640 1 -- 172.21.15.55:0/1937599454 wait complete. 2024-03-19T02:00:34.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.020+0000 7f9f6d8d6640 1 Processor -- start 2024-03-19T02:00:34.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.020+0000 7f9f6d8d6640 1 -- start start 2024-03-19T02:00:34.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.020+0000 7f9f6d8d6640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9f68070780 0x7f9f680c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:34.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.020+0000 7f9f6d8d6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9f680c9610 0x7f9f68143c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:34.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.020+0000 7f9f6d8d6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9f68152120 0x7f9f6806c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:34.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.020+0000 7f9f6d8d6640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9f680c86d0 con 0x7f9f68152120 2024-03-19T02:00:34.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.020+0000 7f9f6d8d6640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f9f680c31c0 con 0x7f9f68070780 2024-03-19T02:00:34.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.020+0000 7f9f6d8d6640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f9f68165170 con 0x7f9f680c9610 2024-03-19T02:00:34.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.020+0000 7f9f667fc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9f680c9610 0x7f9f68143c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:34.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.020+0000 7f9f677fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9f68152120 0x7f9f6806c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:34.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.020+0000 7f9f667fc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9f680c9610 0x7f9f68143c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:34740/0 (socket says 172.21.15.55:34740) 2024-03-19T02:00:34.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.020+0000 7f9f667fc640 1 -- 172.21.15.55:0/4180214042 learned_addr learned my addr 172.21.15.55:0/4180214042 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:34.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.020+0000 7f9f66ffd640 1 --2- 172.21.15.55:0/4180214042 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9f68070780 0x7f9f680c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:34.026 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.024+0000 7f9f677fe640 1 -- 172.21.15.55:0/4180214042 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9f680c9610 msgr2=0x7f9f68143c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:34.026 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.024+0000 7f9f677fe640 1 --2- 172.21.15.55:0/4180214042 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9f680c9610 0x7f9f68143c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:34.026 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.024+0000 7f9f677fe640 1 -- 172.21.15.55:0/4180214042 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9f68070780 msgr2=0x7f9f680c6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:34.026 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.024+0000 7f9f677fe640 1 --2- 172.21.15.55:0/4180214042 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9f68070780 0x7f9f680c6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:34.026 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.024+0000 7f9f677fe640 1 -- 172.21.15.55:0/4180214042 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f6806d7a0 con 0x7f9f68152120 2024-03-19T02:00:34.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.028+0000 7f9f677fe640 1 --2- 172.21.15.55:0/4180214042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9f68152120 0x7f9f6806c610 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009d90 tx=0x7f9f6000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:34.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.032+0000 7f9f4ffff640 1 -- 172.21.15.55:0/4180214042 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f60018020 con 0x7f9f68152120 2024-03-19T02:00:34.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.032+0000 7f9f4ffff640 1 -- 172.21.15.55:0/4180214042 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f60014480 con 0x7f9f68152120 2024-03-19T02:00:34.035 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.032+0000 7f9f6d8d6640 1 -- 172.21.15.55:0/4180214042 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f6806e3d0 con 0x7f9f68152120 2024-03-19T02:00:34.035 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.032+0000 7f9f6d8d6640 1 -- 172.21.15.55:0/4180214042 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f6806de10 con 0x7f9f68152120 2024-03-19T02:00:34.035 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.036+0000 7f9f4ffff640 1 -- 172.21.15.55:0/4180214042 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f60014870 con 0x7f9f68152120 2024-03-19T02:00:34.035 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.036+0000 7f9f6d8d6640 1 -- 172.21.15.55:0/4180214042 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f68068a40 con 0x7f9f68152120 2024-03-19T02:00:34.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.432+0000 7f9f4ffff640 1 -- 172.21.15.55:0/4180214042 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f9f6001b020 con 0x7f9f68152120 2024-03-19T02:00:34.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.432+0000 7f9f4ffff640 1 --2- 172.21.15.55:0/4180214042 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9f540676e0 0x7f9f54069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:34.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.432+0000 7f9f66ffd640 1 --2- 172.21.15.55:0/4180214042 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9f540676e0 0x7f9f54069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:34.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.432+0000 7f9f66ffd640 1 --2- 172.21.15.55:0/4180214042 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9f540676e0 0x7f9f54069ba0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9f580ad000 tx=0x7f9f580a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:34.461 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.460+0000 7f9f4ffff640 1 -- 172.21.15.55:0/4180214042 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f9f60055170 con 0x7f9f68152120 2024-03-19T02:00:34.461 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.460+0000 7f9f4ffff640 1 -- 172.21.15.55:0/4180214042 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f9f60022030 con 0x7f9f68152120 2024-03-19T02:00:34.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.684+0000 7f9f6d8d6640 1 -- 172.21.15.55:0/4180214042 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f681439e0 con 0x7f9f68152120 2024-03-19T02:00:34.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.704+0000 7f9f4ffff640 1 -- 172.21.15.55:0/4180214042 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f9f60010040 con 0x7f9f68152120 2024-03-19T02:00:34.705 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:34.705 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:34.706 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:34.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.708+0000 7f9f6d8d6640 1 -- 172.21.15.55:0/4180214042 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9f540676e0 msgr2=0x7f9f54069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:34.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.708+0000 7f9f6d8d6640 1 --2- 172.21.15.55:0/4180214042 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9f540676e0 0x7f9f54069ba0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9f580ad000 tx=0x7f9f580a6000 comp rx=0 tx=0).stop 2024-03-19T02:00:34.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.708+0000 7f9f6d8d6640 1 -- 172.21.15.55:0/4180214042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9f68152120 msgr2=0x7f9f6806c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:34.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.708+0000 7f9f6d8d6640 1 --2- 172.21.15.55:0/4180214042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9f68152120 0x7f9f6806c610 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009d90 tx=0x7f9f6000c490 comp rx=0 tx=0).stop 2024-03-19T02:00:34.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.708+0000 7f9f6d8d6640 1 -- 172.21.15.55:0/4180214042 shutdown_connections 2024-03-19T02:00:34.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.708+0000 7f9f6d8d6640 1 --2- 172.21.15.55:0/4180214042 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f9f540676e0 0x7f9f54069ba0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:34.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.708+0000 7f9f6d8d6640 1 --2- 172.21.15.55:0/4180214042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9f68152120 0x7f9f6806c610 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:34.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.708+0000 7f9f6d8d6640 1 --2- 172.21.15.55:0/4180214042 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9f680c9610 0x7f9f68143c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:34.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.708+0000 7f9f6d8d6640 1 --2- 172.21.15.55:0/4180214042 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9f68070780 0x7f9f680c6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:34.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.708+0000 7f9f6d8d6640 1 -- 172.21.15.55:0/4180214042 >> 172.21.15.55:0/4180214042 conn(0x7f9f68067600 msgr2=0x7f9f6806f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:34.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.708+0000 7f9f6d8d6640 1 -- 172.21.15.55:0/4180214042 shutdown_connections 2024-03-19T02:00:34.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:34.708+0000 7f9f6d8d6640 1 -- 172.21.15.55:0/4180214042 wait complete. 2024-03-19T02:00:35.334 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:35.334 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:36.336 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:36.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.448+0000 7f6099769640 1 Processor -- start 2024-03-19T02:00:36.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.448+0000 7f6099769640 1 -- start start 2024-03-19T02:00:36.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.448+0000 7f6099769640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f6094150c70 0x7f6094156770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:36.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.448+0000 7f6099769640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6094156e70 con 0x7f60940c2420 2024-03-19T02:00:36.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.448+0000 7f6099769640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6094157130 con 0x7f60940c2800 2024-03-19T02:00:36.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.448+0000 7f6099769640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f60941576f0 con 0x7f6094150c70 2024-03-19T02:00:36.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.448+0000 7f60937fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f6094150c70 0x7f6094156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:36.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.448+0000 7f60927fc640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f60940c2800 0x7f6094158510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:48008/0 (socket says 172.21.15.55:48008) 2024-03-19T02:00:36.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.448+0000 7f60927fc640 1 -- 172.21.15.55:0/1675639093 learned_addr learned my addr 172.21.15.55:0/1675639093 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:00:36.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.456+0000 7f60937fe640 1 -- 172.21.15.55:0/1675639093 >> v1:172.21.15.84:6789/0 conn(0x7f60940c2800 legacy=0x7f6094158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:36.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.456+0000 7f60937fe640 1 -- 172.21.15.55:0/1675639093 >> v1:172.21.15.55:6789/0 conn(0x7f60940c2420 legacy=0x7f6094150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:36.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.456+0000 7f60937fe640 1 -- 172.21.15.55:0/1675639093 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6094157410 con 0x7f6094150c70 2024-03-19T02:00:36.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.460+0000 7f60937fe640 1 --2- 172.21.15.55:0/1675639093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6094150c70 0x7f6094156770 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f6094157870 tx=0x7f608c01da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3720704708d5855e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:36.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.468+0000 7f6091ffb640 1 -- 172.21.15.55:0/1675639093 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608c0b3070 con 0x7f6094150c70 2024-03-19T02:00:36.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.468+0000 7f6091ffb640 1 -- 172.21.15.55:0/1675639093 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f608c0a6a50 con 0x7f6094150c70 2024-03-19T02:00:36.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.468+0000 7f6091ffb640 1 -- 172.21.15.55:0/1675639093 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608c0a6e20 con 0x7f6094150c70 2024-03-19T02:00:36.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.468+0000 7f6099769640 1 -- 172.21.15.55:0/1675639093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6094150c70 msgr2=0x7f6094156770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:36.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.468+0000 7f6099769640 1 --2- 172.21.15.55:0/1675639093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6094150c70 0x7f6094156770 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f6094157870 tx=0x7f608c01da60 comp rx=0 tx=0).stop 2024-03-19T02:00:36.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.468+0000 7f6099769640 1 -- 172.21.15.55:0/1675639093 shutdown_connections 2024-03-19T02:00:36.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.468+0000 7f6099769640 1 --2- 172.21.15.55:0/1675639093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6094150c70 0x7f6094156770 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:36.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.468+0000 7f6099769640 1 -- 172.21.15.55:0/1675639093 >> 172.21.15.55:0/1675639093 conn(0x7f609406f3b0 msgr2=0x7f609406f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:36.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.468+0000 7f6099769640 1 -- 172.21.15.55:0/1675639093 shutdown_connections 2024-03-19T02:00:36.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.468+0000 7f6099769640 1 -- 172.21.15.55:0/1675639093 wait complete. 2024-03-19T02:00:36.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.468+0000 7f6099769640 1 Processor -- start 2024-03-19T02:00:36.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.468+0000 7f6099769640 1 -- start start 2024-03-19T02:00:36.471 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.472+0000 7f6099769640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60940c2420 0x7f60941433b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:36.472 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.472+0000 7f6099769640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60940c2800 0x7f6094139c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:36.472 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.472+0000 7f6099769640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6094150c70 0x7f609413a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:36.472 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.472+0000 7f6099769640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6094164030 con 0x7f60940c2420 2024-03-19T02:00:36.472 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.472+0000 7f6099769640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f60940c5590 con 0x7f60940c2800 2024-03-19T02:00:36.472 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.472+0000 7f6099769640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f60941641b0 con 0x7f6094150c70 2024-03-19T02:00:36.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.472+0000 7f6092ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60940c2420 0x7f60941433b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:36.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.472+0000 7f60927fc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60940c2800 0x7f6094139c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:36.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.472+0000 7f6092ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60940c2420 0x7f60941433b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:42006/0 (socket says 172.21.15.55:42006) 2024-03-19T02:00:36.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.472+0000 7f6092ffd640 1 -- 172.21.15.55:0/365896286 learned_addr learned my addr 172.21.15.55:0/365896286 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:36.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.472+0000 7f60937fe640 1 --2- 172.21.15.55:0/365896286 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6094150c70 0x7f609413a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:36.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.476+0000 7f6092ffd640 1 -- 172.21.15.55:0/365896286 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6094150c70 msgr2=0x7f609413a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:36.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.476+0000 7f6092ffd640 1 --2- 172.21.15.55:0/365896286 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6094150c70 0x7f609413a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:36.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.476+0000 7f6092ffd640 1 -- 172.21.15.55:0/365896286 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60940c2800 msgr2=0x7f6094139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:36.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.476+0000 7f6092ffd640 1 --2- 172.21.15.55:0/365896286 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60940c2800 0x7f6094139c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:36.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.476+0000 7f6092ffd640 1 -- 172.21.15.55:0/365896286 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f609413afe0 con 0x7f60940c2420 2024-03-19T02:00:36.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.476+0000 7f6092ffd640 1 --2- 172.21.15.55:0/365896286 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60940c2420 0x7f60941433b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f608400c420 tx=0x7f608400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:36.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.480+0000 7f607b7fe640 1 -- 172.21.15.55:0/365896286 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60840052d0 con 0x7f60940c2420 2024-03-19T02:00:36.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.480+0000 7f607b7fe640 1 -- 172.21.15.55:0/365896286 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6084005470 con 0x7f60940c2420 2024-03-19T02:00:36.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.480+0000 7f607b7fe640 1 -- 172.21.15.55:0/365896286 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6084007260 con 0x7f60940c2420 2024-03-19T02:00:36.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.480+0000 7f6099769640 1 -- 172.21.15.55:0/365896286 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60941efef0 con 0x7f60940c2420 2024-03-19T02:00:36.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.480+0000 7f6099769640 1 -- 172.21.15.55:0/365896286 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6094140260 con 0x7f60940c2420 2024-03-19T02:00:36.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.484+0000 7f6099769640 1 -- 172.21.15.55:0/365896286 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60941404a0 con 0x7f60940c2420 2024-03-19T02:00:36.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.760+0000 7f607b7fe640 1 -- 172.21.15.55:0/365896286 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f6084007400 con 0x7f60940c2420 2024-03-19T02:00:36.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.760+0000 7f607b7fe640 1 --2- 172.21.15.55:0/365896286 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f60800676e0 0x7f6080069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:36.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.764+0000 7f60927fc640 1 --2- 172.21.15.55:0/365896286 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f60800676e0 0x7f6080069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:36.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.764+0000 7f60927fc640 1 --2- 172.21.15.55:0/365896286 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f60800676e0 0x7f6080069ba0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f60941542c0 tx=0x7f6088009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:36.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.800+0000 7f607b7fe640 1 -- 172.21.15.55:0/365896286 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f6084053850 con 0x7f60940c2420 2024-03-19T02:00:36.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:36.800+0000 7f607b7fe640 1 -- 172.21.15.55:0/365896286 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f60941404a0 con 0x7f60940c2420 2024-03-19T02:00:37.014 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:37.012+0000 7f6099769640 1 -- 172.21.15.55:0/365896286 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6094068330 con 0x7f60940c2420 2024-03-19T02:00:37.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:37.040+0000 7f607b7fe640 1 -- 172.21.15.55:0/365896286 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f60840557b0 con 0x7f60940c2420 2024-03-19T02:00:37.042 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:37.042 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:37.042 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:37.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:37.044+0000 7f6099769640 1 -- 172.21.15.55:0/365896286 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f60800676e0 msgr2=0x7f6080069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:37.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:37.044+0000 7f6099769640 1 --2- 172.21.15.55:0/365896286 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f60800676e0 0x7f6080069ba0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f60941542c0 tx=0x7f6088009040 comp rx=0 tx=0).stop 2024-03-19T02:00:37.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:37.044+0000 7f6099769640 1 -- 172.21.15.55:0/365896286 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60940c2420 msgr2=0x7f60941433b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:37.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:37.044+0000 7f6099769640 1 --2- 172.21.15.55:0/365896286 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60940c2420 0x7f60941433b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f608400c420 tx=0x7f608400c8e0 comp rx=0 tx=0).stop 2024-03-19T02:00:37.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:37.044+0000 7f6099769640 1 -- 172.21.15.55:0/365896286 shutdown_connections 2024-03-19T02:00:37.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:37.044+0000 7f6099769640 1 --2- 172.21.15.55:0/365896286 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f60800676e0 0x7f6080069ba0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:37.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:37.044+0000 7f6099769640 1 --2- 172.21.15.55:0/365896286 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6094150c70 0x7f609413a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:37.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:37.044+0000 7f6099769640 1 --2- 172.21.15.55:0/365896286 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60940c2800 0x7f6094139c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:37.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:37.044+0000 7f6099769640 1 --2- 172.21.15.55:0/365896286 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60940c2420 0x7f60941433b0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:37.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:37.044+0000 7f6099769640 1 -- 172.21.15.55:0/365896286 >> 172.21.15.55:0/365896286 conn(0x7f609406f3b0 msgr2=0x7f609406f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:37.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:37.044+0000 7f6099769640 1 -- 172.21.15.55:0/365896286 shutdown_connections 2024-03-19T02:00:37.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:37.044+0000 7f6099769640 1 -- 172.21.15.55:0/365896286 wait complete. 2024-03-19T02:00:37.667 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:37.667 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:38.668 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:38.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.796+0000 7f442dae6640 1 Processor -- start 2024-03-19T02:00:38.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.796+0000 7f442dae6640 1 -- start start 2024-03-19T02:00:38.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.796+0000 7f442dae6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f44280cf1e0 0x7f442815dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:38.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.796+0000 7f442dae6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f442806cb90 con 0x7f442806c6f0 2024-03-19T02:00:38.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.796+0000 7f442dae6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44280cda30 con 0x7f44280cbf30 2024-03-19T02:00:38.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.796+0000 7f442dae6640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f44280ce640 con 0x7f44280cf1e0 2024-03-19T02:00:38.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.796+0000 7f4426ffd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f442806c6f0 0x7f442815d8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:39540/0 (socket says 172.21.15.55:39540) 2024-03-19T02:00:38.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.796+0000 7f4426ffd640 1 -- 172.21.15.55:0/3177037936 learned_addr learned my addr 172.21.15.55:0/3177037936 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:00:38.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.796+0000 7f4427fff640 1 --2- 172.21.15.55:0/3177037936 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f44280cf1e0 0x7f442815dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:38.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.800+0000 7f44267fc640 1 -- 172.21.15.55:0/3177037936 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1725806493 0 0) 0x7f442806cb90 con 0x7f442806c6f0 2024-03-19T02:00:38.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.800+0000 7f44267fc640 1 -- 172.21.15.55:0/3177037936 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4410003530 con 0x7f442806c6f0 2024-03-19T02:00:38.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.800+0000 7f4427fff640 1 -- 172.21.15.55:0/3177037936 >> v1:172.21.15.84:6789/0 conn(0x7f44280cbf30 legacy=0x7f44280cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:38.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.800+0000 7f4427fff640 1 -- 172.21.15.55:0/3177037936 >> v1:172.21.15.55:6789/0 conn(0x7f442806c6f0 legacy=0x7f442815d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:38.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.800+0000 7f4427fff640 1 -- 172.21.15.55:0/3177037936 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44280cdd10 con 0x7f44280cf1e0 2024-03-19T02:00:38.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.804+0000 7f4427fff640 1 --2- 172.21.15.55:0/3177037936 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f44280cf1e0 0x7f442815dfe0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f44280ce7c0 tx=0x7f442001c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd3f1dce0084eab server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:38.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.812+0000 7f44267fc640 1 -- 172.21.15.55:0/3177037936 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44200b2070 con 0x7f44280cf1e0 2024-03-19T02:00:38.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.812+0000 7f44267fc640 1 -- 172.21.15.55:0/3177037936 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44200a5990 con 0x7f44280cf1e0 2024-03-19T02:00:38.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.812+0000 7f44267fc640 1 -- 172.21.15.55:0/3177037936 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44200a5d60 con 0x7f44280cf1e0 2024-03-19T02:00:38.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.812+0000 7f442dae6640 1 -- 172.21.15.55:0/3177037936 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f44280cf1e0 msgr2=0x7f442815dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:38.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.812+0000 7f442dae6640 1 --2- 172.21.15.55:0/3177037936 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f44280cf1e0 0x7f442815dfe0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f44280ce7c0 tx=0x7f442001c8d0 comp rx=0 tx=0).stop 2024-03-19T02:00:38.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.812+0000 7f442dae6640 1 -- 172.21.15.55:0/3177037936 shutdown_connections 2024-03-19T02:00:38.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.812+0000 7f442dae6640 1 --2- 172.21.15.55:0/3177037936 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f44280cf1e0 0x7f442815dfe0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:38.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.812+0000 7f442dae6640 1 -- 172.21.15.55:0/3177037936 >> 172.21.15.55:0/3177037936 conn(0x7f44280b16a0 msgr2=0x7f44280b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:38.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.812+0000 7f442dae6640 1 -- 172.21.15.55:0/3177037936 shutdown_connections 2024-03-19T02:00:38.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.812+0000 7f442dae6640 1 -- 172.21.15.55:0/3177037936 wait complete. 2024-03-19T02:00:38.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.816+0000 7f442dae6640 1 Processor -- start 2024-03-19T02:00:38.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.816+0000 7f442dae6640 1 -- start start 2024-03-19T02:00:38.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.816+0000 7f442dae6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f442806c6f0 0x7f44281f6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:38.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.816+0000 7f442dae6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f44280cbf30 0x7f44281f4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:38.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.816+0000 7f442dae6640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44280cf1e0 0x7f44281f5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:38.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.816+0000 7f442dae6640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f44280cc310 con 0x7f442806c6f0 2024-03-19T02:00:38.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.816+0000 7f442dae6640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f44280b23b0 con 0x7f44280cf1e0 2024-03-19T02:00:38.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.816+0000 7f442dae6640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f44280cc490 con 0x7f44280cbf30 2024-03-19T02:00:38.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.816+0000 7f4427fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44280cf1e0 0x7f44281f5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:38.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.816+0000 7f4427fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44280cf1e0 0x7f44281f5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:42824/0 (socket says 172.21.15.55:42824) 2024-03-19T02:00:38.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.816+0000 7f4427fff640 1 -- 172.21.15.55:0/4274152582 learned_addr learned my addr 172.21.15.55:0/4274152582 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:38.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.816+0000 7f44277fe640 1 --2- 172.21.15.55:0/4274152582 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f442806c6f0 0x7f44281f6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:38.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.816+0000 7f4426ffd640 1 --2- 172.21.15.55:0/4274152582 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f44280cbf30 0x7f44281f4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:38.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.820+0000 7f44277fe640 1 -- 172.21.15.55:0/4274152582 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f44280cbf30 msgr2=0x7f44281f4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:38.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.820+0000 7f44277fe640 1 --2- 172.21.15.55:0/4274152582 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f44280cbf30 0x7f44281f4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:38.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.820+0000 7f44277fe640 1 -- 172.21.15.55:0/4274152582 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44280cf1e0 msgr2=0x7f44281f5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:38.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.820+0000 7f44277fe640 1 --2- 172.21.15.55:0/4274152582 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44280cf1e0 0x7f44281f5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:38.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.820+0000 7f44277fe640 1 -- 172.21.15.55:0/4274152582 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4428153ac0 con 0x7f442806c6f0 2024-03-19T02:00:38.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.824+0000 7f44277fe640 1 --2- 172.21.15.55:0/4274152582 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f442806c6f0 0x7f44281f6870 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f441800c3f0 tx=0x7f441800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:38.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.828+0000 7f440ffff640 1 -- 172.21.15.55:0/4274152582 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44180052d0 con 0x7f442806c6f0 2024-03-19T02:00:38.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.828+0000 7f440ffff640 1 -- 172.21.15.55:0/4274152582 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4418005470 con 0x7f442806c6f0 2024-03-19T02:00:38.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.828+0000 7f442dae6640 1 -- 172.21.15.55:0/4274152582 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f442815df00 con 0x7f442806c6f0 2024-03-19T02:00:38.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.828+0000 7f440ffff640 1 -- 172.21.15.55:0/4274152582 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4418007260 con 0x7f442806c6f0 2024-03-19T02:00:38.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.828+0000 7f442dae6640 1 -- 172.21.15.55:0/4274152582 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44281f4640 con 0x7f442806c6f0 2024-03-19T02:00:38.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:38.832+0000 7f442dae6640 1 -- 172.21.15.55:0/4274152582 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44281f7380 con 0x7f442806c6f0 2024-03-19T02:00:39.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:39.100+0000 7f440ffff640 1 -- 172.21.15.55:0/4274152582 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f4418007400 con 0x7f442806c6f0 2024-03-19T02:00:39.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:39.100+0000 7f440ffff640 1 --2- 172.21.15.55:0/4274152582 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f44140676b0 0x7f4414069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:39.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:39.100+0000 7f4426ffd640 1 --2- 172.21.15.55:0/4274152582 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f44140676b0 0x7f4414069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:39.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:39.100+0000 7f4426ffd640 1 --2- 172.21.15.55:0/4274152582 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f44140676b0 0x7f4414069b70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f44280c67f0 tx=0x7f441c007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:39.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:39.124+0000 7f440ffff640 1 -- 172.21.15.55:0/4274152582 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f4418053b30 con 0x7f442806c6f0 2024-03-19T02:00:39.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:39.124+0000 7f440ffff640 1 -- 172.21.15.55:0/4274152582 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f44281f7380 con 0x7f442806c6f0 2024-03-19T02:00:39.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:39.352+0000 7f442dae6640 1 -- 172.21.15.55:0/4274152582 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4428068330 con 0x7f442806c6f0 2024-03-19T02:00:39.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:39.444+0000 7f440ffff640 1 -- 172.21.15.55:0/4274152582 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f4428153ac0 con 0x7f442806c6f0 2024-03-19T02:00:39.445 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:39.445 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:39.445 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:39.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:39.448+0000 7f442dae6640 1 -- 172.21.15.55:0/4274152582 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f44140676b0 msgr2=0x7f4414069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:39.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:39.448+0000 7f442dae6640 1 --2- 172.21.15.55:0/4274152582 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f44140676b0 0x7f4414069b70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f44280c67f0 tx=0x7f441c007d40 comp rx=0 tx=0).stop 2024-03-19T02:00:39.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:39.448+0000 7f442dae6640 1 -- 172.21.15.55:0/4274152582 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f442806c6f0 msgr2=0x7f44281f6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:39.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:39.448+0000 7f442dae6640 1 --2- 172.21.15.55:0/4274152582 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f442806c6f0 0x7f44281f6870 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f441800c3f0 tx=0x7f441800c8b0 comp rx=0 tx=0).stop 2024-03-19T02:00:39.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:39.448+0000 7f442dae6640 1 -- 172.21.15.55:0/4274152582 shutdown_connections 2024-03-19T02:00:39.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:39.448+0000 7f442dae6640 1 --2- 172.21.15.55:0/4274152582 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f44140676b0 0x7f4414069b70 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:39.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:39.448+0000 7f442dae6640 1 --2- 172.21.15.55:0/4274152582 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44280cf1e0 0x7f44281f5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:39.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:39.448+0000 7f442dae6640 1 --2- 172.21.15.55:0/4274152582 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f44280cbf30 0x7f44281f4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:39.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:39.448+0000 7f442dae6640 1 --2- 172.21.15.55:0/4274152582 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f442806c6f0 0x7f44281f6870 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:39.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:39.448+0000 7f442dae6640 1 -- 172.21.15.55:0/4274152582 >> 172.21.15.55:0/4274152582 conn(0x7f44280b16a0 msgr2=0x7f4428154260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:39.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:39.448+0000 7f442dae6640 1 -- 172.21.15.55:0/4274152582 shutdown_connections 2024-03-19T02:00:39.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:39.448+0000 7f442dae6640 1 -- 172.21.15.55:0/4274152582 wait complete. 2024-03-19T02:00:40.071 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:40.071 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:40.141 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:00:40.144 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:00:41.072 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:41.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.184+0000 7ff2c7f1d640 1 Processor -- start 2024-03-19T02:00:41.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.184+0000 7ff2c7f1d640 1 -- start start 2024-03-19T02:00:41.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.188+0000 7ff2c7f1d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff2c01520b0 0x7ff2c00709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:41.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.188+0000 7ff2c7f1d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2c0158d50 con 0x7ff2c00c9450 2024-03-19T02:00:41.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.188+0000 7ff2c7f1d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2c015b720 con 0x7ff2c00c9000 2024-03-19T02:00:41.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.188+0000 7ff2c7f1d640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7ff2c015c4e0 con 0x7ff2c01520b0 2024-03-19T02:00:41.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.188+0000 7ff2c5cb9640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff2c01520b0 0x7ff2c00709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:41.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.188+0000 7ff2c5cb9640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff2c01520b0 0x7ff2c00709a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:42832/0 (socket says 172.21.15.55:42832) 2024-03-19T02:00:41.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.188+0000 7ff2c5cb9640 1 -- 172.21.15.55:0/3356591705 learned_addr learned my addr 172.21.15.55:0/3356591705 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:41.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.192+0000 7ff2c4cb7640 1 -- 172.21.15.55:0/3356591705 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4286389250 0 0) 0x7ff2c0158d50 con 0x7ff2c00c9450 2024-03-19T02:00:41.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.192+0000 7ff2c4cb7640 1 -- 172.21.15.55:0/3356591705 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2ac003530 con 0x7ff2c00c9450 2024-03-19T02:00:41.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.192+0000 7ff2c5cb9640 1 -- 172.21.15.55:0/3356591705 >> v1:172.21.15.84:6789/0 conn(0x7ff2c00c9000 legacy=0x7ff2c00c2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:41.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.192+0000 7ff2c5cb9640 1 -- 172.21.15.55:0/3356591705 >> v1:172.21.15.55:6789/0 conn(0x7ff2c00c9450 legacy=0x7ff2c00c31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:41.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.192+0000 7ff2c5cb9640 1 -- 172.21.15.55:0/3356591705 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2c015ba00 con 0x7ff2c01520b0 2024-03-19T02:00:41.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.204+0000 7ff2c5cb9640 1 --2- 172.21.15.55:0/3356591705 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2c01520b0 0x7ff2c00709a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc009870 tx=0x7ff2bc01ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b38bf186830e69bd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:41.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.212+0000 7ff2c4cb7640 1 -- 172.21.15.55:0/3356591705 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc0b5070 con 0x7ff2c01520b0 2024-03-19T02:00:41.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.212+0000 7ff2c4cb7640 1 -- 172.21.15.55:0/3356591705 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2bc0a8a50 con 0x7ff2c01520b0 2024-03-19T02:00:41.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.212+0000 7ff2c4cb7640 1 -- 172.21.15.55:0/3356591705 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc0a8e20 con 0x7ff2c01520b0 2024-03-19T02:00:41.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.212+0000 7ff2c7f1d640 1 -- 172.21.15.55:0/3356591705 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2c01520b0 msgr2=0x7ff2c00709a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:41.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.212+0000 7ff2c7f1d640 1 --2- 172.21.15.55:0/3356591705 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2c01520b0 0x7ff2c00709a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc009870 tx=0x7ff2bc01ed50 comp rx=0 tx=0).stop 2024-03-19T02:00:41.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.212+0000 7ff2c7f1d640 1 -- 172.21.15.55:0/3356591705 shutdown_connections 2024-03-19T02:00:41.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.212+0000 7ff2c7f1d640 1 --2- 172.21.15.55:0/3356591705 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2c01520b0 0x7ff2c00709a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:41.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.212+0000 7ff2c7f1d640 1 -- 172.21.15.55:0/3356591705 >> 172.21.15.55:0/3356591705 conn(0x7ff2c0067600 msgr2=0x7ff2c0071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:41.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.212+0000 7ff2c7f1d640 1 -- 172.21.15.55:0/3356591705 shutdown_connections 2024-03-19T02:00:41.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.212+0000 7ff2c7f1d640 1 -- 172.21.15.55:0/3356591705 wait complete. 2024-03-19T02:00:41.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.212+0000 7ff2c7f1d640 1 Processor -- start 2024-03-19T02:00:41.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.212+0000 7ff2c7f1d640 1 -- start start 2024-03-19T02:00:41.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.216+0000 7ff2c7f1d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2c00c9000 0x7ff2c00cca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:41.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.216+0000 7ff2c7f1d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff2c00c9450 0x7ff2c01f3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:41.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.216+0000 7ff2c7f1d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2c01520b0 0x7ff2c01f3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:41.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.216+0000 7ff2c7f1d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c015c050 con 0x7ff2c01520b0 2024-03-19T02:00:41.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.216+0000 7ff2c7f1d640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c00c9880 con 0x7ff2c00c9000 2024-03-19T02:00:41.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.216+0000 7ff2c7f1d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff2c01589a0 con 0x7ff2c00c9450 2024-03-19T02:00:41.217 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.216+0000 7ff2c64ba640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2c01520b0 0x7ff2c01f3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:41.217 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.216+0000 7ff2c54b8640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff2c00c9450 0x7ff2c01f3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:41.217 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.216+0000 7ff2c5cb9640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2c00c9000 0x7ff2c00cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:41.217 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.216+0000 7ff2c64ba640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2c01520b0 0x7ff2c01f3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:54642/0 (socket says 172.21.15.55:54642) 2024-03-19T02:00:41.217 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.216+0000 7ff2c64ba640 1 -- 172.21.15.55:0/3101962106 learned_addr learned my addr 172.21.15.55:0/3101962106 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:41.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.220+0000 7ff2c64ba640 1 -- 172.21.15.55:0/3101962106 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff2c00c9450 msgr2=0x7ff2c01f3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:41.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.220+0000 7ff2c64ba640 1 --2- 172.21.15.55:0/3101962106 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff2c00c9450 0x7ff2c01f3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:41.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.220+0000 7ff2c64ba640 1 -- 172.21.15.55:0/3101962106 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2c00c9000 msgr2=0x7ff2c00cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:41.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.220+0000 7ff2c64ba640 1 --2- 172.21.15.55:0/3101962106 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2c00c9000 0x7ff2c00cca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:41.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.220+0000 7ff2c64ba640 1 -- 172.21.15.55:0/3101962106 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2c014ccd0 con 0x7ff2c01520b0 2024-03-19T02:00:41.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.224+0000 7ff2c64ba640 1 --2- 172.21.15.55:0/3101962106 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2c01520b0 0x7ff2c01f3840 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff2b800c7c0 tx=0x7ff2b800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:41.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.228+0000 7ff2b67fc640 1 -- 172.21.15.55:0/3101962106 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b8003b80 con 0x7ff2c01520b0 2024-03-19T02:00:41.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.228+0000 7ff2b67fc640 1 -- 172.21.15.55:0/3101962106 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2b8003d20 con 0x7ff2c01520b0 2024-03-19T02:00:41.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.228+0000 7ff2b67fc640 1 -- 172.21.15.55:0/3101962106 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b800aa50 con 0x7ff2c01520b0 2024-03-19T02:00:41.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.228+0000 7ff2c7f1d640 1 -- 172.21.15.55:0/3101962106 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2c006dfc0 con 0x7ff2c01520b0 2024-03-19T02:00:41.229 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.228+0000 7ff2c7f1d640 1 -- 172.21.15.55:0/3101962106 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2c006e280 con 0x7ff2c01520b0 2024-03-19T02:00:41.229 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.228+0000 7ff2c7f1d640 1 -- 172.21.15.55:0/3101962106 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2c00cc530 con 0x7ff2c01520b0 2024-03-19T02:00:41.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.248+0000 7ff2b67fc640 1 -- 172.21.15.55:0/3101962106 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ff2b800abf0 con 0x7ff2c01520b0 2024-03-19T02:00:41.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.248+0000 7ff2b67fc640 1 --2- 172.21.15.55:0/3101962106 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff2a40676e0 0x7ff2a4069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:41.249 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.248+0000 7ff2c5cb9640 1 --2- 172.21.15.55:0/3101962106 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff2a40676e0 0x7ff2a4069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:41.250 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.248+0000 7ff2c5cb9640 1 --2- 172.21.15.55:0/3101962106 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff2a40676e0 0x7ff2a4069ba0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc0b2000 tx=0x7ff2bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:41.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.252+0000 7ff2b67fc640 1 -- 172.21.15.55:0/3101962106 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7ff2b8054860 con 0x7ff2c01520b0 2024-03-19T02:00:41.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.552+0000 7ff2b67fc640 1 -- 172.21.15.55:0/3101962106 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ff2b80567c0 con 0x7ff2c01520b0 2024-03-19T02:00:41.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.760+0000 7ff2c7f1d640 1 -- 172.21.15.55:0/3101962106 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2c0068a40 con 0x7ff2c01520b0 2024-03-19T02:00:41.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.776+0000 7ff2b67fc640 1 -- 172.21.15.55:0/3101962106 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7ff2b805a860 con 0x7ff2c01520b0 2024-03-19T02:00:41.777 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:41.777 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:41.777 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:41.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.776+0000 7ff2c7f1d640 1 -- 172.21.15.55:0/3101962106 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff2a40676e0 msgr2=0x7ff2a4069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:41.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.780+0000 7ff2c7f1d640 1 --2- 172.21.15.55:0/3101962106 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff2a40676e0 0x7ff2a4069ba0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc0b2000 tx=0x7ff2bc009040 comp rx=0 tx=0).stop 2024-03-19T02:00:41.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.780+0000 7ff2c7f1d640 1 -- 172.21.15.55:0/3101962106 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2c01520b0 msgr2=0x7ff2c01f3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:41.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.780+0000 7ff2c7f1d640 1 --2- 172.21.15.55:0/3101962106 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2c01520b0 0x7ff2c01f3840 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff2b800c7c0 tx=0x7ff2b800cc80 comp rx=0 tx=0).stop 2024-03-19T02:00:41.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.780+0000 7ff2c7f1d640 1 -- 172.21.15.55:0/3101962106 shutdown_connections 2024-03-19T02:00:41.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.780+0000 7ff2c7f1d640 1 --2- 172.21.15.55:0/3101962106 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff2a40676e0 0x7ff2a4069ba0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:41.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.780+0000 7ff2c7f1d640 1 --2- 172.21.15.55:0/3101962106 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2c01520b0 0x7ff2c01f3840 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:41.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.780+0000 7ff2c7f1d640 1 --2- 172.21.15.55:0/3101962106 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff2c00c9450 0x7ff2c01f3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:41.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.780+0000 7ff2c7f1d640 1 --2- 172.21.15.55:0/3101962106 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2c00c9000 0x7ff2c00cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:41.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.780+0000 7ff2c7f1d640 1 -- 172.21.15.55:0/3101962106 >> 172.21.15.55:0/3101962106 conn(0x7ff2c0067600 msgr2=0x7ff2c00c9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:41.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.780+0000 7ff2c7f1d640 1 -- 172.21.15.55:0/3101962106 shutdown_connections 2024-03-19T02:00:41.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:41.780+0000 7ff2c7f1d640 1 -- 172.21.15.55:0/3101962106 wait complete. 2024-03-19T02:00:42.400 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:42.400 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:43.401 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:43.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.516+0000 7f2b0b6a9640 1 Processor -- start 2024-03-19T02:00:43.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.516+0000 7f2b0b6a9640 1 -- start start 2024-03-19T02:00:43.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.516+0000 7f2b0b6a9640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2b041511f0 0x7f2b040c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:43.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.516+0000 7f2b0b6a9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2b040c2e40 0x7f2b04156f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:43.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.516+0000 7f2b0b6a9640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b040c3330 con 0x7f2b040c8260 2024-03-19T02:00:43.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.516+0000 7f2b0b6a9640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f2b04157490 con 0x7f2b041511f0 2024-03-19T02:00:43.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.516+0000 7f2b0b6a9640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2b040c5040 con 0x7f2b040c2e40 2024-03-19T02:00:43.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.516+0000 7f2b09445640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2b041511f0 0x7f2b040c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:43.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.520+0000 7f2b09445640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2b041511f0 0x7f2b040c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:54654/0 (socket says 172.21.15.55:54654) 2024-03-19T02:00:43.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.520+0000 7f2b09445640 1 -- 172.21.15.55:0/1919018393 learned_addr learned my addr 172.21.15.55:0/1919018393 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:43.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.520+0000 7f2b09445640 1 -- 172.21.15.55:0/1919018393 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2b040c2e40 msgr2=0x7f2b04156f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:43.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.520+0000 7f2b09445640 1 --2- 172.21.15.55:0/1919018393 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2b040c2e40 0x7f2b04156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:43.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.520+0000 7f2b09445640 1 -- 172.21.15.55:0/1919018393 >> v1:172.21.15.55:6789/0 conn(0x7f2b040c8260 legacy=0x7f2b040c7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:43.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.520+0000 7f2b09445640 1 -- 172.21.15.55:0/1919018393 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b040c3610 con 0x7f2b041511f0 2024-03-19T02:00:43.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.528+0000 7f2b09445640 1 --2- 172.21.15.55:0/1919018393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b041511f0 0x7f2b040c5a90 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2b00009870 tx=0x7f2b0001ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b260bb6d123d62 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:43.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.532+0000 7f2afbfff640 1 -- 172.21.15.55:0/1919018393 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b000b5070 con 0x7f2b041511f0 2024-03-19T02:00:43.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.532+0000 7f2afbfff640 1 -- 172.21.15.55:0/1919018393 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b000a8a50 con 0x7f2b041511f0 2024-03-19T02:00:43.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.532+0000 7f2afbfff640 1 -- 172.21.15.55:0/1919018393 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b000a8e20 con 0x7f2b041511f0 2024-03-19T02:00:43.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.536+0000 7f2b0b6a9640 1 -- 172.21.15.55:0/1919018393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b041511f0 msgr2=0x7f2b040c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:43.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.536+0000 7f2b0b6a9640 1 --2- 172.21.15.55:0/1919018393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b041511f0 0x7f2b040c5a90 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2b00009870 tx=0x7f2b0001ed50 comp rx=0 tx=0).stop 2024-03-19T02:00:43.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.536+0000 7f2b0b6a9640 1 -- 172.21.15.55:0/1919018393 shutdown_connections 2024-03-19T02:00:43.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.536+0000 7f2b0b6a9640 1 --2- 172.21.15.55:0/1919018393 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2b040c2e40 0x7f2b04156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:43.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.536+0000 7f2b0b6a9640 1 --2- 172.21.15.55:0/1919018393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b041511f0 0x7f2b040c5a90 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:43.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.536+0000 7f2b0b6a9640 1 -- 172.21.15.55:0/1919018393 >> 172.21.15.55:0/1919018393 conn(0x7f2b0406f3b0 msgr2=0x7f2b0406f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:43.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.536+0000 7f2b0b6a9640 1 -- 172.21.15.55:0/1919018393 shutdown_connections 2024-03-19T02:00:43.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.536+0000 7f2b0b6a9640 1 -- 172.21.15.55:0/1919018393 wait complete. 2024-03-19T02:00:43.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.536+0000 7f2b0b6a9640 1 Processor -- start 2024-03-19T02:00:43.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.536+0000 7f2b0b6a9640 1 -- start start 2024-03-19T02:00:43.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.536+0000 7f2b0b6a9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b040c2e40 0x7f2b041ee410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:43.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.536+0000 7f2b0b6a9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2b040c8260 0x7f2b040cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:43.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.536+0000 7f2b0b6a9640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b041511f0 0x7f2b04142150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:43.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.536+0000 7f2b0b6a9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2b041515d0 con 0x7f2b040c2e40 2024-03-19T02:00:43.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.536+0000 7f2b0b6a9640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2b040c5290 con 0x7f2b041511f0 2024-03-19T02:00:43.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.536+0000 7f2b0b6a9640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2b040c29c0 con 0x7f2b040c8260 2024-03-19T02:00:43.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.536+0000 7f2b09445640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b040c2e40 0x7f2b041ee410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:43.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.536+0000 7f2b09445640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b040c2e40 0x7f2b041ee410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:54668/0 (socket says 172.21.15.55:54668) 2024-03-19T02:00:43.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.536+0000 7f2b09445640 1 -- 172.21.15.55:0/1829060445 learned_addr learned my addr 172.21.15.55:0/1829060445 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:43.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.540+0000 7f2b09c46640 1 --2- 172.21.15.55:0/1829060445 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b041511f0 0x7f2b04142150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:43.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.544+0000 7f2b09445640 1 -- 172.21.15.55:0/1829060445 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2b040c8260 msgr2=0x7f2b040cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:43.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.544+0000 7f2b09445640 1 --2- 172.21.15.55:0/1829060445 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2b040c8260 0x7f2b040cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:43.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.544+0000 7f2b09445640 1 -- 172.21.15.55:0/1829060445 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b041511f0 msgr2=0x7f2b04142150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:43.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.544+0000 7f2b09445640 1 --2- 172.21.15.55:0/1829060445 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b041511f0 0x7f2b04142150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:43.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.544+0000 7f2b09445640 1 -- 172.21.15.55:0/1829060445 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b04154770 con 0x7f2b040c2e40 2024-03-19T02:00:43.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.548+0000 7f2b09445640 1 --2- 172.21.15.55:0/1829060445 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b040c2e40 0x7f2b041ee410 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2b0001b960 tx=0x7f2b0001baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:43.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.552+0000 7f2af9ffb640 1 -- 172.21.15.55:0/1829060445 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b000b5050 con 0x7f2b040c2e40 2024-03-19T02:00:43.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.552+0000 7f2af9ffb640 1 -- 172.21.15.55:0/1829060445 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b000bf070 con 0x7f2b040c2e40 2024-03-19T02:00:43.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.552+0000 7f2b0b6a9640 1 -- 172.21.15.55:0/1829060445 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b04153ed0 con 0x7f2b040c2e40 2024-03-19T02:00:43.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.552+0000 7f2af9ffb640 1 -- 172.21.15.55:0/1829060445 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b000ba880 con 0x7f2b040c2e40 2024-03-19T02:00:43.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.552+0000 7f2b0b6a9640 1 -- 172.21.15.55:0/1829060445 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b04154490 con 0x7f2b040c2e40 2024-03-19T02:00:43.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.552+0000 7f2b0b6a9640 1 -- 172.21.15.55:0/1829060445 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b04068330 con 0x7f2b040c2e40 2024-03-19T02:00:43.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.584+0000 7f2af9ffb640 1 -- 172.21.15.55:0/1829060445 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f2b000baa20 con 0x7f2b040c2e40 2024-03-19T02:00:43.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.584+0000 7f2af9ffb640 1 --2- 172.21.15.55:0/1829060445 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2ae80676b0 0x7f2ae8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:43.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.584+0000 7f2b08c44640 1 --2- 172.21.15.55:0/1829060445 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2ae80676b0 0x7f2ae8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:43.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.588+0000 7f2b08c44640 1 --2- 172.21.15.55:0/1829060445 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2ae80676b0 0x7f2ae8069b70 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2b04072240 tx=0x7f2af4006ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:43.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.856+0000 7f2af9ffb640 1 -- 172.21.15.55:0/1829060445 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f2b000f2b00 con 0x7f2b040c2e40 2024-03-19T02:00:43.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:43.856+0000 7f2af9ffb640 1 -- 172.21.15.55:0/1829060445 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2b000c9020 con 0x7f2b040c2e40 2024-03-19T02:00:44.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:44.064+0000 7f2b0b6a9640 1 -- 172.21.15.55:0/1829060445 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b040c5c30 con 0x7f2b040c2e40 2024-03-19T02:00:44.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:44.092+0000 7f2af9ffb640 1 -- 172.21.15.55:0/1829060445 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f2b000188b0 con 0x7f2b040c2e40 2024-03-19T02:00:44.093 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:44.093 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:44.094 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:44.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:44.096+0000 7f2b0b6a9640 1 -- 172.21.15.55:0/1829060445 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2ae80676b0 msgr2=0x7f2ae8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:44.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:44.096+0000 7f2b0b6a9640 1 --2- 172.21.15.55:0/1829060445 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2ae80676b0 0x7f2ae8069b70 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2b04072240 tx=0x7f2af4006ca0 comp rx=0 tx=0).stop 2024-03-19T02:00:44.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:44.096+0000 7f2b0b6a9640 1 -- 172.21.15.55:0/1829060445 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b040c2e40 msgr2=0x7f2b041ee410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:44.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:44.096+0000 7f2b0b6a9640 1 --2- 172.21.15.55:0/1829060445 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b040c2e40 0x7f2b041ee410 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2b0001b960 tx=0x7f2b0001baf0 comp rx=0 tx=0).stop 2024-03-19T02:00:44.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:44.096+0000 7f2b0b6a9640 1 -- 172.21.15.55:0/1829060445 shutdown_connections 2024-03-19T02:00:44.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:44.096+0000 7f2b0b6a9640 1 --2- 172.21.15.55:0/1829060445 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2ae80676b0 0x7f2ae8069b70 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:44.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:44.096+0000 7f2b0b6a9640 1 --2- 172.21.15.55:0/1829060445 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b041511f0 0x7f2b04142150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:44.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:44.096+0000 7f2b0b6a9640 1 --2- 172.21.15.55:0/1829060445 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2b040c8260 0x7f2b040cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:44.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:44.096+0000 7f2b0b6a9640 1 --2- 172.21.15.55:0/1829060445 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b040c2e40 0x7f2b041ee410 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:44.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:44.096+0000 7f2b0b6a9640 1 -- 172.21.15.55:0/1829060445 >> 172.21.15.55:0/1829060445 conn(0x7f2b0406f3b0 msgr2=0x7f2b040c6340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:44.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:44.096+0000 7f2b0b6a9640 1 -- 172.21.15.55:0/1829060445 shutdown_connections 2024-03-19T02:00:44.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:44.096+0000 7f2b0b6a9640 1 -- 172.21.15.55:0/1829060445 wait complete. 2024-03-19T02:00:44.717 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:44.718 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:45.719 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:45.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.848+0000 7ff9897e4640 1 Processor -- start 2024-03-19T02:00:45.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.848+0000 7ff9897e4640 1 -- start start 2024-03-19T02:00:45.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.848+0000 7ff9897e4640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff984154340 0x7ff98415e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:45.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.848+0000 7ff9897e4640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff9840c73c0 0x7ff98415aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:45.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.848+0000 7ff9897e4640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff984071280 con 0x7ff984154720 2024-03-19T02:00:45.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.848+0000 7ff9897e4640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7ff984073840 con 0x7ff9840c73c0 2024-03-19T02:00:45.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.848+0000 7ff9897e4640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff98415df10 con 0x7ff984154340 2024-03-19T02:00:45.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.848+0000 7ff982ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff984154340 0x7ff98415e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:45.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.852+0000 7ff9837fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff9840c73c0 0x7ff98415aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:45.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.852+0000 7ff982ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff984154340 0x7ff98415e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:33530/0 (socket says 172.21.15.55:33530) 2024-03-19T02:00:45.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.852+0000 7ff9827fc640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7ff984154720 0x7ff984072e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:44364/0 (socket says 172.21.15.55:44364) 2024-03-19T02:00:45.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.852+0000 7ff982ffd640 1 -- 172.21.15.55:0/306357725 learned_addr learned my addr 172.21.15.55:0/306357725 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:45.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.856+0000 7ff982ffd640 1 -- 172.21.15.55:0/306357725 >> v2:172.21.15.84:3300/0 conn(0x7ff9840c73c0 msgr2=0x7ff98415aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:45.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.856+0000 7ff982ffd640 1 --2- 172.21.15.55:0/306357725 >> v2:172.21.15.84:3300/0 conn(0x7ff9840c73c0 0x7ff98415aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:45.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.856+0000 7ff982ffd640 1 -- 172.21.15.55:0/306357725 >> v1:172.21.15.84:6789/0 conn(0x7ff984154720 legacy=0x7ff984072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:45.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.856+0000 7ff982ffd640 1 -- 172.21.15.55:0/306357725 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff98415d900 con 0x7ff984154340 2024-03-19T02:00:45.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.856+0000 7ff982ffd640 1 --2- 172.21.15.55:0/306357725 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff984154340 0x7ff98415e3a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff974009870 tx=0x7ff97401c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8d2da8bd2cbdcf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:45.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.864+0000 7ff981ffb640 1 -- 172.21.15.55:0/306357725 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9740b2070 con 0x7ff984154340 2024-03-19T02:00:45.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.864+0000 7ff981ffb640 1 -- 172.21.15.55:0/306357725 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9740a9070 con 0x7ff984154340 2024-03-19T02:00:45.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.864+0000 7ff981ffb640 1 -- 172.21.15.55:0/306357725 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9740ad880 con 0x7ff984154340 2024-03-19T02:00:45.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.864+0000 7ff9897e4640 1 -- 172.21.15.55:0/306357725 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff984154340 msgr2=0x7ff98415e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:45.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.864+0000 7ff9897e4640 1 --2- 172.21.15.55:0/306357725 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff984154340 0x7ff98415e3a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff974009870 tx=0x7ff97401c560 comp rx=0 tx=0).stop 2024-03-19T02:00:45.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.864+0000 7ff9897e4640 1 -- 172.21.15.55:0/306357725 shutdown_connections 2024-03-19T02:00:45.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.864+0000 7ff9897e4640 1 --2- 172.21.15.55:0/306357725 >> v2:172.21.15.84:3300/0 conn(0x7ff9840c73c0 0x7ff98415aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:45.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.864+0000 7ff9897e4640 1 --2- 172.21.15.55:0/306357725 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff984154340 0x7ff98415e3a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:45.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.864+0000 7ff9897e4640 1 -- 172.21.15.55:0/306357725 >> 172.21.15.55:0/306357725 conn(0x7ff9840cc9c0 msgr2=0x7ff9840ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:45.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.864+0000 7ff9897e4640 1 -- 172.21.15.55:0/306357725 shutdown_connections 2024-03-19T02:00:45.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.864+0000 7ff9897e4640 1 -- 172.21.15.55:0/306357725 wait complete. 2024-03-19T02:00:45.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.868+0000 7ff9897e4640 1 Processor -- start 2024-03-19T02:00:45.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.868+0000 7ff9897e4640 1 -- start start 2024-03-19T02:00:45.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.868+0000 7ff9897e4640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9840c73c0 0x7ff9841f38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:45.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.868+0000 7ff9897e4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff984154340 0x7ff98415c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:45.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.868+0000 7ff9897e4640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff984154720 0x7ff984158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:45.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.868+0000 7ff9897e4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff984166b70 con 0x7ff984154340 2024-03-19T02:00:45.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.868+0000 7ff9897e4640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff9840c77a0 con 0x7ff9840c73c0 2024-03-19T02:00:45.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.868+0000 7ff9897e4640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff984166cf0 con 0x7ff984154720 2024-03-19T02:00:45.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.868+0000 7ff9837fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff984154720 0x7ff984158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:45.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.868+0000 7ff982ffd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9840c73c0 0x7ff9841f38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:45.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.868+0000 7ff9827fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff984154340 0x7ff98415c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:45.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.868+0000 7ff9827fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff984154340 0x7ff98415c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:54684/0 (socket says 172.21.15.55:54684) 2024-03-19T02:00:45.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.868+0000 7ff9827fc640 1 -- 172.21.15.55:0/2039055412 learned_addr learned my addr 172.21.15.55:0/2039055412 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:45.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.876+0000 7ff9827fc640 1 -- 172.21.15.55:0/2039055412 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff984154720 msgr2=0x7ff984158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:45.876 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.876+0000 7ff9827fc640 1 --2- 172.21.15.55:0/2039055412 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff984154720 0x7ff984158200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:45.876 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.876+0000 7ff9827fc640 1 -- 172.21.15.55:0/2039055412 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9840c73c0 msgr2=0x7ff9841f38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:45.876 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.876+0000 7ff9827fc640 1 --2- 172.21.15.55:0/2039055412 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9840c73c0 0x7ff9841f38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:45.876 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.876+0000 7ff9827fc640 1 -- 172.21.15.55:0/2039055412 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff984159b60 con 0x7ff984154340 2024-03-19T02:00:45.876 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.876+0000 7ff9837fe640 1 --2- 172.21.15.55:0/2039055412 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff984154720 0x7ff984158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:00:45.880 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.880+0000 7ff9827fc640 1 --2- 172.21.15.55:0/2039055412 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff984154340 0x7ff98415c990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff97800c3a0 tx=0x7ff97800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:45.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.884+0000 7ff96b7fe640 1 -- 172.21.15.55:0/2039055412 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9780074e0 con 0x7ff984154340 2024-03-19T02:00:45.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.884+0000 7ff96b7fe640 1 -- 172.21.15.55:0/2039055412 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff978007680 con 0x7ff984154340 2024-03-19T02:00:45.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.884+0000 7ff9897e4640 1 -- 172.21.15.55:0/2039055412 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff984157d20 con 0x7ff984154340 2024-03-19T02:00:45.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.884+0000 7ff96b7fe640 1 -- 172.21.15.55:0/2039055412 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978004a30 con 0x7ff984154340 2024-03-19T02:00:45.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.884+0000 7ff9897e4640 1 -- 172.21.15.55:0/2039055412 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9841595a0 con 0x7ff984154340 2024-03-19T02:00:45.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:45.888+0000 7ff9897e4640 1 -- 172.21.15.55:0/2039055412 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff984068a40 con 0x7ff984154340 2024-03-19T02:00:46.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:46.164+0000 7ff96b7fe640 1 -- 172.21.15.55:0/2039055412 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ff97801a020 con 0x7ff984154340 2024-03-19T02:00:46.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:46.164+0000 7ff96b7fe640 1 --2- 172.21.15.55:0/2039055412 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff9700676b0 0x7ff970069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:46.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:46.164+0000 7ff982ffd640 1 --2- 172.21.15.55:0/2039055412 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff9700676b0 0x7ff970069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:46.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:46.164+0000 7ff982ffd640 1 --2- 172.21.15.55:0/2039055412 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff9700676b0 0x7ff970069b70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff9740af000 tx=0x7ff9740a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:46.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:46.184+0000 7ff96b7fe640 1 -- 172.21.15.55:0/2039055412 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7ff978053dc0 con 0x7ff984154340 2024-03-19T02:00:46.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:46.184+0000 7ff96b7fe640 1 -- 172.21.15.55:0/2039055412 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ff978021070 con 0x7ff984154340 2024-03-19T02:00:46.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:46.404+0000 7ff9897e4640 1 -- 172.21.15.55:0/2039055412 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff984157ef0 con 0x7ff984154340 2024-03-19T02:00:46.439 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:46.436+0000 7ff96b7fe640 1 -- 172.21.15.55:0/2039055412 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7ff978055d20 con 0x7ff984154340 2024-03-19T02:00:46.439 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:46.439 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:46.440 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:46.441 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:46.440+0000 7ff9897e4640 1 -- 172.21.15.55:0/2039055412 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff9700676b0 msgr2=0x7ff970069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:46.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:46.440+0000 7ff9897e4640 1 --2- 172.21.15.55:0/2039055412 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff9700676b0 0x7ff970069b70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff9740af000 tx=0x7ff9740a7000 comp rx=0 tx=0).stop 2024-03-19T02:00:46.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:46.440+0000 7ff9897e4640 1 -- 172.21.15.55:0/2039055412 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff984154340 msgr2=0x7ff98415c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:46.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:46.440+0000 7ff9897e4640 1 --2- 172.21.15.55:0/2039055412 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff984154340 0x7ff98415c990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff97800c3a0 tx=0x7ff97800c860 comp rx=0 tx=0).stop 2024-03-19T02:00:46.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:46.440+0000 7ff9897e4640 1 -- 172.21.15.55:0/2039055412 shutdown_connections 2024-03-19T02:00:46.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:46.440+0000 7ff9897e4640 1 --2- 172.21.15.55:0/2039055412 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff9700676b0 0x7ff970069b70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:46.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:46.440+0000 7ff9897e4640 1 --2- 172.21.15.55:0/2039055412 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff984154720 0x7ff984158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:46.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:46.440+0000 7ff9897e4640 1 --2- 172.21.15.55:0/2039055412 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff984154340 0x7ff98415c990 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:46.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:46.440+0000 7ff9897e4640 1 --2- 172.21.15.55:0/2039055412 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9840c73c0 0x7ff9841f38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:46.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:46.440+0000 7ff9897e4640 1 -- 172.21.15.55:0/2039055412 >> 172.21.15.55:0/2039055412 conn(0x7ff9840cc9c0 msgr2=0x7ff9840cb640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:46.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:46.440+0000 7ff9897e4640 1 -- 172.21.15.55:0/2039055412 shutdown_connections 2024-03-19T02:00:46.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:46.440+0000 7ff9897e4640 1 -- 172.21.15.55:0/2039055412 wait complete. 2024-03-19T02:00:47.061 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:47.062 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:48.062 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:48.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.180+0000 7f7b72fdc640 1 Processor -- start 2024-03-19T02:00:48.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.180+0000 7f7b72fdc640 1 -- start start 2024-03-19T02:00:48.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.184+0000 7f7b72fdc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b6c06b2a0 0x7f7b6c15ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:48.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.184+0000 7f7b72fdc640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b6c0c8840 con 0x7f7b6c154300 2024-03-19T02:00:48.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.184+0000 7f7b72fdc640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b6c0c8ab0 con 0x7f7b6c06c110 2024-03-19T02:00:48.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.184+0000 7f7b72fdc640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f7b6c15b260 con 0x7f7b6c06b2a0 2024-03-19T02:00:48.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.184+0000 7f7b70d78640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f7b6c154300 0x7f7b6c15e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:39562/0 (socket says 172.21.15.55:39562) 2024-03-19T02:00:48.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.184+0000 7f7b70d78640 1 -- 172.21.15.55:0/3509689540 learned_addr learned my addr 172.21.15.55:0/3509689540 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:00:48.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.184+0000 7f7b6b7fe640 1 -- 172.21.15.55:0/3509689540 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1591939842 0 0) 0x7f7b6c0c8840 con 0x7f7b6c154300 2024-03-19T02:00:48.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.184+0000 7f7b6b7fe640 1 -- 172.21.15.55:0/3509689540 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b54003530 con 0x7f7b6c154300 2024-03-19T02:00:48.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.188+0000 7f7b6b7fe640 1 -- 172.21.15.55:0/3509689540 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 191982945 0 0) 0x7f7b6c0c8ab0 con 0x7f7b6c06c110 2024-03-19T02:00:48.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.188+0000 7f7b6b7fe640 1 -- 172.21.15.55:0/3509689540 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b6c0c8840 con 0x7f7b6c06c110 2024-03-19T02:00:48.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.188+0000 7f7b6b7fe640 1 -- 172.21.15.55:0/3509689540 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3764825807 0 0) 0x7f7b54003530 con 0x7f7b6c154300 2024-03-19T02:00:48.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.188+0000 7f7b6b7fe640 1 -- 172.21.15.55:0/3509689540 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b6c06b2a0 msgr2=0x7f7b6c15ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:48.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.188+0000 7f7b6b7fe640 1 --2- 172.21.15.55:0/3509689540 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b6c06b2a0 0x7f7b6c15ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:48.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.188+0000 7f7b6b7fe640 1 -- 172.21.15.55:0/3509689540 >> v1:172.21.15.84:6789/0 conn(0x7f7b6c06c110 legacy=0x7f7b6c164c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:48.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.188+0000 7f7b6b7fe640 1 -- 172.21.15.55:0/3509689540 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b6c0c8d90 con 0x7f7b6c154300 2024-03-19T02:00:48.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.188+0000 7f7b6b7fe640 1 -- 172.21.15.55:0/3509689540 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f7b5c001680 con 0x7f7b6c154300 2024-03-19T02:00:48.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.188+0000 7f7b6b7fe640 1 -- 172.21.15.55:0/3509689540 >> v1:172.21.15.55:6789/0 conn(0x7f7b6c154300 legacy=0x7f7b6c15e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:48.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.188+0000 7f7b6b7fe640 1 --2- 172.21.15.55:0/3509689540 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7b54003b80 0x7f7b54003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:48.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.188+0000 7f7b6b7fe640 1 --2- 172.21.15.55:0/3509689540 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b54004810 0x7f7b54004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:48.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.188+0000 7f7b6b7fe640 1 --2- 172.21.15.55:0/3509689540 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b6c06b2a0 0x7f7b5400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:48.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.188+0000 7f7b6b7fe640 1 -- 172.21.15.55:0/3509689540 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6c15b260 con 0x7f7b54004810 2024-03-19T02:00:48.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.188+0000 7f7b6b7fe640 1 -- 172.21.15.55:0/3509689540 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f7b54005560 con 0x7f7b54003b80 2024-03-19T02:00:48.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.188+0000 7f7b6b7fe640 1 -- 172.21.15.55:0/3509689540 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f7b5400bf10 con 0x7f7b6c06b2a0 2024-03-19T02:00:48.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.192+0000 7f7b70d78640 1 --2- 172.21.15.55:0/3509689540 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7b54003b80 0x7f7b54003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:48.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.192+0000 7f7b6bfff640 1 --2- 172.21.15.55:0/3509689540 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b54004810 0x7f7b54004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:48.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.196+0000 7f7b71579640 1 --2- 172.21.15.55:0/3509689540 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b6c06b2a0 0x7f7b5400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:48.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.196+0000 7f7b6bfff640 1 -- 172.21.15.55:0/3509689540 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b6c06b2a0 msgr2=0x7f7b5400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:48.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.196+0000 7f7b6bfff640 1 --2- 172.21.15.55:0/3509689540 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b6c06b2a0 0x7f7b5400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:48.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.196+0000 7f7b6bfff640 1 -- 172.21.15.55:0/3509689540 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7b54003b80 msgr2=0x7f7b54003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:48.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.196+0000 7f7b6bfff640 1 --2- 172.21.15.55:0/3509689540 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7b54003b80 0x7f7b54003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:48.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.196+0000 7f7b6bfff640 1 -- 172.21.15.55:0/3509689540 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b54003530 con 0x7f7b54004810 2024-03-19T02:00:48.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.200+0000 7f7b6bfff640 1 --2- 172.21.15.55:0/3509689540 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b54004810 0x7f7b54004c90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7b60009890 tx=0x7f7b6001c060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95055027cb9007f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:48.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.204+0000 7f7b6b7fe640 1 -- 172.21.15.55:0/3509689540 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b600b1070 con 0x7f7b54004810 2024-03-19T02:00:48.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.204+0000 7f7b6b7fe640 1 -- 172.21.15.55:0/3509689540 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b600a47e0 con 0x7f7b54004810 2024-03-19T02:00:48.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.204+0000 7f7b6b7fe640 1 -- 172.21.15.55:0/3509689540 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b60009460 con 0x7f7b54004810 2024-03-19T02:00:48.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.208+0000 7f7b72fdc640 1 -- 172.21.15.55:0/3509689540 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b54004810 msgr2=0x7f7b54004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:48.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.208+0000 7f7b72fdc640 1 --2- 172.21.15.55:0/3509689540 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b54004810 0x7f7b54004c90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7b60009890 tx=0x7f7b6001c060 comp rx=0 tx=0).stop 2024-03-19T02:00:48.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.208+0000 7f7b71579640 1 -- 172.21.15.55:0/3509689540 reap_dead start 2024-03-19T02:00:48.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.208+0000 7f7b72fdc640 1 -- 172.21.15.55:0/3509689540 shutdown_connections 2024-03-19T02:00:48.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.208+0000 7f7b72fdc640 1 -- 172.21.15.55:0/3509689540 >> 172.21.15.55:0/3509689540 conn(0x7f7b6c06fb40 msgr2=0x7f7b6c06ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:48.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.208+0000 7f7b72fdc640 1 -- 172.21.15.55:0/3509689540 shutdown_connections 2024-03-19T02:00:48.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.208+0000 7f7b72fdc640 1 -- 172.21.15.55:0/3509689540 wait complete. 2024-03-19T02:00:48.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.208+0000 7f7b72fdc640 1 Processor -- start 2024-03-19T02:00:48.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.208+0000 7f7b72fdc640 1 -- start start 2024-03-19T02:00:48.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.208+0000 7f7b72fdc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b6c14e810 0x7f7b6c159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:48.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.208+0000 7f7b72fdc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7b6c159f90 0x7f7b6c157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:48.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.208+0000 7f7b72fdc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b6c151200 0x7f7b6c1577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:48.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.208+0000 7f7b72fdc640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6c15d8c0 con 0x7f7b6c14e810 2024-03-19T02:00:48.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.208+0000 7f7b72fdc640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6c0c9350 con 0x7f7b6c159f90 2024-03-19T02:00:48.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.208+0000 7f7b72fdc640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f7b6c15dba0 con 0x7f7b6c151200 2024-03-19T02:00:48.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.208+0000 7f7b70d78640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b6c14e810 0x7f7b6c159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:48.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.208+0000 7f7b6bfff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7b6c159f90 0x7f7b6c157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:48.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.208+0000 7f7b70d78640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b6c14e810 0x7f7b6c159220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:54706/0 (socket says 172.21.15.55:54706) 2024-03-19T02:00:48.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.208+0000 7f7b70d78640 1 -- 172.21.15.55:0/3551544255 learned_addr learned my addr 172.21.15.55:0/3551544255 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:48.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.208+0000 7f7b71579640 1 --2- 172.21.15.55:0/3551544255 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b6c151200 0x7f7b6c1577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:48.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.212+0000 7f7b70d78640 1 -- 172.21.15.55:0/3551544255 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b6c151200 msgr2=0x7f7b6c1577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:48.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.212+0000 7f7b70d78640 1 --2- 172.21.15.55:0/3551544255 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b6c151200 0x7f7b6c1577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:48.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.212+0000 7f7b70d78640 1 -- 172.21.15.55:0/3551544255 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7b6c159f90 msgr2=0x7f7b6c157280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:48.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.212+0000 7f7b70d78640 1 --2- 172.21.15.55:0/3551544255 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7b6c159f90 0x7f7b6c157280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:48.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.212+0000 7f7b70d78640 1 -- 172.21.15.55:0/3551544255 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b6c159760 con 0x7f7b6c14e810 2024-03-19T02:00:48.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.216+0000 7f7b70d78640 1 --2- 172.21.15.55:0/3551544255 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b6c14e810 0x7f7b6c159220 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c011bb0 tx=0x7f7b5c00f540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:48.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.220+0000 7f7b697fa640 1 -- 172.21.15.55:0/3551544255 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c010040 con 0x7f7b6c14e810 2024-03-19T02:00:48.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.220+0000 7f7b697fa640 1 -- 172.21.15.55:0/3551544255 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b5c01a070 con 0x7f7b6c14e810 2024-03-19T02:00:48.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.220+0000 7f7b697fa640 1 -- 172.21.15.55:0/3551544255 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c015650 con 0x7f7b6c14e810 2024-03-19T02:00:48.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.220+0000 7f7b72fdc640 1 -- 172.21.15.55:0/3551544255 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b6c1f0ba0 con 0x7f7b6c14e810 2024-03-19T02:00:48.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.220+0000 7f7b72fdc640 1 -- 172.21.15.55:0/3551544255 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b6c14f3c0 con 0x7f7b6c14e810 2024-03-19T02:00:48.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.220+0000 7f7b72fdc640 1 -- 172.21.15.55:0/3551544255 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b6c068a40 con 0x7f7b6c14e810 2024-03-19T02:00:48.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.240+0000 7f7b697fa640 1 -- 172.21.15.55:0/3551544255 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f7b5c0157f0 con 0x7f7b6c14e810 2024-03-19T02:00:48.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.240+0000 7f7b697fa640 1 --2- 172.21.15.55:0/3551544255 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7b58067690 0x7f7b58069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:48.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.240+0000 7f7b6bfff640 1 --2- 172.21.15.55:0/3551544255 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7b58067690 0x7f7b58069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:48.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.240+0000 7f7b6bfff640 1 --2- 172.21.15.55:0/3551544255 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7b58067690 0x7f7b58069b50 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7b600ae000 tx=0x7f7b60004850 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:48.245 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.244+0000 7f7b697fa640 1 -- 172.21.15.55:0/3551544255 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f7b5c057c90 con 0x7f7b6c14e810 2024-03-19T02:00:48.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.508+0000 7f7b697fa640 1 -- 172.21.15.55:0/3551544255 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f7b5c059bf0 con 0x7f7b6c14e810 2024-03-19T02:00:48.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.720+0000 7f7b72fdc640 1 -- 172.21.15.55:0/3551544255 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b6c068e20 con 0x7f7b6c14e810 2024-03-19T02:00:48.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.732+0000 7f7b697fa640 1 -- 172.21.15.55:0/3551544255 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f7b5c05dc90 con 0x7f7b6c14e810 2024-03-19T02:00:48.734 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:48.734 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:48.734 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:48.736 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.736+0000 7f7b72fdc640 1 -- 172.21.15.55:0/3551544255 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7b58067690 msgr2=0x7f7b58069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:48.736 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.736+0000 7f7b72fdc640 1 --2- 172.21.15.55:0/3551544255 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7b58067690 0x7f7b58069b50 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7b600ae000 tx=0x7f7b60004850 comp rx=0 tx=0).stop 2024-03-19T02:00:48.736 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.736+0000 7f7b72fdc640 1 -- 172.21.15.55:0/3551544255 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b6c14e810 msgr2=0x7f7b6c159220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:48.736 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.736+0000 7f7b72fdc640 1 --2- 172.21.15.55:0/3551544255 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b6c14e810 0x7f7b6c159220 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c011bb0 tx=0x7f7b5c00f540 comp rx=0 tx=0).stop 2024-03-19T02:00:48.736 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.736+0000 7f7b72fdc640 1 -- 172.21.15.55:0/3551544255 shutdown_connections 2024-03-19T02:00:48.736 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.736+0000 7f7b72fdc640 1 --2- 172.21.15.55:0/3551544255 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7b58067690 0x7f7b58069b50 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:48.736 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.736+0000 7f7b72fdc640 1 --2- 172.21.15.55:0/3551544255 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b6c151200 0x7f7b6c1577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:48.736 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.736+0000 7f7b72fdc640 1 --2- 172.21.15.55:0/3551544255 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7b6c159f90 0x7f7b6c157280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:48.736 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.736+0000 7f7b72fdc640 1 --2- 172.21.15.55:0/3551544255 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b6c14e810 0x7f7b6c159220 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:48.737 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.736+0000 7f7b72fdc640 1 -- 172.21.15.55:0/3551544255 >> 172.21.15.55:0/3551544255 conn(0x7f7b6c06fb40 msgr2=0x7f7b6c0c69c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:48.737 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.736+0000 7f7b72fdc640 1 -- 172.21.15.55:0/3551544255 shutdown_connections 2024-03-19T02:00:48.737 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:48.736+0000 7f7b72fdc640 1 -- 172.21.15.55:0/3551544255 wait complete. 2024-03-19T02:00:49.357 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:49.357 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:50.358 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:50.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.472+0000 7f54ae3cc640 1 Processor -- start 2024-03-19T02:00:50.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.472+0000 7f54ae3cc640 1 -- start start 2024-03-19T02:00:50.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.472+0000 7f54ae3cc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f54a8072c60 0x7f54a8158a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:50.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.472+0000 7f54ae3cc640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f54a815b540 0x7f54a80c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:50.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.472+0000 7f54ae3cc640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54a8152660 con 0x7f54a80c39c0 2024-03-19T02:00:50.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.472+0000 7f54ae3cc640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f54a815b9b0 con 0x7f54a815b540 2024-03-19T02:00:50.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.472+0000 7f54ae3cc640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f54a8073130 con 0x7f54a8072c60 2024-03-19T02:00:50.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.476+0000 7f54ac969640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f54a815b540 0x7f54a80c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:50.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.476+0000 7f54ac969640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f54a815b540 0x7f54a80c60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57912/0 (socket says 172.21.15.55:57912) 2024-03-19T02:00:50.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.476+0000 7f54ac969640 1 -- 172.21.15.55:0/2226484809 learned_addr learned my addr 172.21.15.55:0/2226484809 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:50.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.476+0000 7f54a77fe640 1 --2- 172.21.15.55:0/2226484809 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f54a8072c60 0x7f54a8158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:50.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.476+0000 7f54ac969640 1 -- 172.21.15.55:0/2226484809 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f54a8072c60 msgr2=0x7f54a8158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:50.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.476+0000 7f54ac969640 1 --2- 172.21.15.55:0/2226484809 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f54a8072c60 0x7f54a8158a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:50.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.476+0000 7f54ac969640 1 -- 172.21.15.55:0/2226484809 >> v1:172.21.15.55:6789/0 conn(0x7f54a80c39c0 legacy=0x7f54a806b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:50.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.476+0000 7f54ac969640 1 -- 172.21.15.55:0/2226484809 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a80733f0 con 0x7f54a815b540 2024-03-19T02:00:50.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.484+0000 7f54ac969640 1 --2- 172.21.15.55:0/2226484809 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f54a815b540 0x7f54a80c60d0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f54a0009e50 tx=0x7f54a001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6029914d11874e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:50.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.488+0000 7f54a6ffd640 1 -- 172.21.15.55:0/2226484809 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a00a8bd0 con 0x7f54a815b540 2024-03-19T02:00:50.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.488+0000 7f54a6ffd640 1 -- 172.21.15.55:0/2226484809 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54a00a8d70 con 0x7f54a815b540 2024-03-19T02:00:50.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.488+0000 7f54a6ffd640 1 -- 172.21.15.55:0/2226484809 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a00b0610 con 0x7f54a815b540 2024-03-19T02:00:50.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.488+0000 7f54ae3cc640 1 -- 172.21.15.55:0/2226484809 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f54a815b540 msgr2=0x7f54a80c60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:50.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.488+0000 7f54ae3cc640 1 --2- 172.21.15.55:0/2226484809 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f54a815b540 0x7f54a80c60d0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f54a0009e50 tx=0x7f54a001f000 comp rx=0 tx=0).stop 2024-03-19T02:00:50.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.488+0000 7f54ae3cc640 1 -- 172.21.15.55:0/2226484809 shutdown_connections 2024-03-19T02:00:50.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.488+0000 7f54ae3cc640 1 --2- 172.21.15.55:0/2226484809 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f54a8072c60 0x7f54a8158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:50.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.488+0000 7f54ae3cc640 1 --2- 172.21.15.55:0/2226484809 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f54a815b540 0x7f54a80c60d0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:50.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.488+0000 7f54ae3cc640 1 -- 172.21.15.55:0/2226484809 >> 172.21.15.55:0/2226484809 conn(0x7f54a80cc9c0 msgr2=0x7f54a80ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:50.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.488+0000 7f54ae3cc640 1 -- 172.21.15.55:0/2226484809 shutdown_connections 2024-03-19T02:00:50.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.488+0000 7f54ae3cc640 1 -- 172.21.15.55:0/2226484809 wait complete. 2024-03-19T02:00:50.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.488+0000 7f54ae3cc640 1 Processor -- start 2024-03-19T02:00:50.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.488+0000 7f54ae3cc640 1 -- start start 2024-03-19T02:00:50.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.492+0000 7f54ae3cc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f54a8072c60 0x7f54a8136560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:50.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.492+0000 7f54ae3cc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f54a80c39c0 0x7f54a81f1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:50.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.492+0000 7f54ae3cc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54a815b540 0x7f54a8139600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:50.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.492+0000 7f54ae3cc640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f54a8158560 con 0x7f54a8072c60 2024-03-19T02:00:50.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.492+0000 7f54ae3cc640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f54a8158280 con 0x7f54a815b540 2024-03-19T02:00:50.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.492+0000 7f54ae3cc640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f54a8157f50 con 0x7f54a80c39c0 2024-03-19T02:00:50.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.492+0000 7f54a7fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f54a8072c60 0x7f54a8136560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:50.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.492+0000 7f54a7fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f54a8072c60 0x7f54a8136560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57914/0 (socket says 172.21.15.55:57914) 2024-03-19T02:00:50.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.492+0000 7f54a7fff640 1 -- 172.21.15.55:0/2528779606 learned_addr learned my addr 172.21.15.55:0/2528779606 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:50.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.492+0000 7f54a77fe640 1 --2- 172.21.15.55:0/2528779606 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f54a80c39c0 0x7f54a81f1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:50.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.492+0000 7f54ac969640 1 --2- 172.21.15.55:0/2528779606 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54a815b540 0x7f54a8139600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:50.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.496+0000 7f54a7fff640 1 -- 172.21.15.55:0/2528779606 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f54a80c39c0 msgr2=0x7f54a81f1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:50.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.496+0000 7f54a7fff640 1 --2- 172.21.15.55:0/2528779606 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f54a80c39c0 0x7f54a81f1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:50.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.496+0000 7f54a7fff640 1 -- 172.21.15.55:0/2528779606 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54a815b540 msgr2=0x7f54a8139600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:50.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.496+0000 7f54a7fff640 1 --2- 172.21.15.55:0/2528779606 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54a815b540 0x7f54a8139600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:50.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.496+0000 7f54a7fff640 1 -- 172.21.15.55:0/2528779606 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a8137ea0 con 0x7f54a8072c60 2024-03-19T02:00:50.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.500+0000 7f54a7fff640 1 --2- 172.21.15.55:0/2528779606 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f54a8072c60 0x7f54a8136560 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f549800c450 tx=0x7f549800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:50.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.504+0000 7f54a4ff9640 1 -- 172.21.15.55:0/2528779606 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5498012830 con 0x7f54a8072c60 2024-03-19T02:00:50.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.504+0000 7f54a4ff9640 1 -- 172.21.15.55:0/2528779606 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5498017070 con 0x7f54a8072c60 2024-03-19T02:00:50.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.508+0000 7f54ae3cc640 1 -- 172.21.15.55:0/2528779606 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54a81f1df0 con 0x7f54a8072c60 2024-03-19T02:00:50.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.508+0000 7f54a4ff9640 1 -- 172.21.15.55:0/2528779606 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5498012c30 con 0x7f54a8072c60 2024-03-19T02:00:50.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.508+0000 7f54ae3cc640 1 -- 172.21.15.55:0/2528779606 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54a8138140 con 0x7f54a8072c60 2024-03-19T02:00:50.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.508+0000 7f54ae3cc640 1 -- 172.21.15.55:0/2528779606 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54a8068a40 con 0x7f54a8072c60 2024-03-19T02:00:50.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.528+0000 7f54a4ff9640 1 -- 172.21.15.55:0/2528779606 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f549801c460 con 0x7f54a8072c60 2024-03-19T02:00:50.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.528+0000 7f54a4ff9640 1 --2- 172.21.15.55:0/2528779606 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f54940676b0 0x7f5494069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:50.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.528+0000 7f54a77fe640 1 --2- 172.21.15.55:0/2528779606 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f54940676b0 0x7f5494069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:50.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.528+0000 7f54a77fe640 1 --2- 172.21.15.55:0/2528779606 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f54940676b0 0x7f5494069b70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f54a81ef100 tx=0x7f549c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:50.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.800+0000 7f54a4ff9640 1 -- 172.21.15.55:0/2528779606 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f5498054760 con 0x7f54a8072c60 2024-03-19T02:00:50.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:50.800+0000 7f54a4ff9640 1 -- 172.21.15.55:0/2528779606 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f54980566c0 con 0x7f54a8072c60 2024-03-19T02:00:51.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:51.020+0000 7f54ae3cc640 1 -- 172.21.15.55:0/2528779606 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54a806cc30 con 0x7f54a8072c60 2024-03-19T02:00:51.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:51.036+0000 7f54a4ff9640 1 -- 172.21.15.55:0/2528779606 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f549805a760 con 0x7f54a8072c60 2024-03-19T02:00:51.037 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:51.037 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:51.037 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:51.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:51.036+0000 7f54ae3cc640 1 -- 172.21.15.55:0/2528779606 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f54940676b0 msgr2=0x7f5494069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:51.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:51.036+0000 7f54ae3cc640 1 --2- 172.21.15.55:0/2528779606 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f54940676b0 0x7f5494069b70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f54a81ef100 tx=0x7f549c009190 comp rx=0 tx=0).stop 2024-03-19T02:00:51.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:51.036+0000 7f54ae3cc640 1 -- 172.21.15.55:0/2528779606 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f54a8072c60 msgr2=0x7f54a8136560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:51.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:51.036+0000 7f54ae3cc640 1 --2- 172.21.15.55:0/2528779606 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f54a8072c60 0x7f54a8136560 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f549800c450 tx=0x7f549800c910 comp rx=0 tx=0).stop 2024-03-19T02:00:51.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:51.036+0000 7f54ae3cc640 1 -- 172.21.15.55:0/2528779606 shutdown_connections 2024-03-19T02:00:51.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:51.036+0000 7f54ae3cc640 1 --2- 172.21.15.55:0/2528779606 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f54940676b0 0x7f5494069b70 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:51.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:51.036+0000 7f54ae3cc640 1 --2- 172.21.15.55:0/2528779606 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54a815b540 0x7f54a8139600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:51.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:51.036+0000 7f54ae3cc640 1 --2- 172.21.15.55:0/2528779606 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f54a80c39c0 0x7f54a81f1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:51.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:51.036+0000 7f54ae3cc640 1 --2- 172.21.15.55:0/2528779606 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f54a8072c60 0x7f54a8136560 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:51.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:51.036+0000 7f54ae3cc640 1 -- 172.21.15.55:0/2528779606 >> 172.21.15.55:0/2528779606 conn(0x7f54a80cc9c0 msgr2=0x7f54a80cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:51.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:51.036+0000 7f54ae3cc640 1 -- 172.21.15.55:0/2528779606 shutdown_connections 2024-03-19T02:00:51.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:51.036+0000 7f54ae3cc640 1 -- 172.21.15.55:0/2528779606 wait complete. 2024-03-19T02:00:51.659 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:51.659 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:52.660 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:52.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.776+0000 7f09dca4e640 1 Processor -- start 2024-03-19T02:00:52.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.776+0000 7f09dca4e640 1 -- start start 2024-03-19T02:00:52.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.780+0000 7f09dca4e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f09d81542f0 0x7f09d8067000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:52.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.780+0000 7f09dca4e640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f09d80c6ea0 0x7f09d806b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:52.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.780+0000 7f09dca4e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09d80c7310 con 0x7f09d806e730 2024-03-19T02:00:52.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.780+0000 7f09dca4e640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f09d81546d0 con 0x7f09d80c6ea0 2024-03-19T02:00:52.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.780+0000 7f09dca4e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f09d8067650 con 0x7f09d81542f0 2024-03-19T02:00:52.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.780+0000 7f09d5d9b640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f09d806e730 0x7f09d815ac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:46842/0 (socket says 172.21.15.55:46842) 2024-03-19T02:00:52.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.780+0000 7f09d5d9b640 1 -- 172.21.15.55:0/2815296081 learned_addr learned my addr 172.21.15.55:0/2815296081 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:00:52.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.780+0000 7f09d6d9d640 1 --2- 172.21.15.55:0/2815296081 >> v2:172.21.15.84:3300/0 conn(0x7f09d80c6ea0 0x7f09d806b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:52.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.780+0000 7f09d659c640 1 --2- 172.21.15.55:0/2815296081 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f09d81542f0 0x7f09d8067000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:52.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.784+0000 7f09d559a640 1 -- 172.21.15.55:0/2815296081 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2440459288 0 0) 0x7f09d80c7310 con 0x7f09d806e730 2024-03-19T02:00:52.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.784+0000 7f09d559a640 1 -- 172.21.15.55:0/2815296081 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09c0003530 con 0x7f09d806e730 2024-03-19T02:00:52.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.784+0000 7f09d659c640 1 -- 172.21.15.55:0/2815296081 >> v2:172.21.15.84:3300/0 conn(0x7f09d80c6ea0 msgr2=0x7f09d806b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:52.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.784+0000 7f09d659c640 1 --2- 172.21.15.55:0/2815296081 >> v2:172.21.15.84:3300/0 conn(0x7f09d80c6ea0 0x7f09d806b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:52.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.784+0000 7f09d659c640 1 -- 172.21.15.55:0/2815296081 >> v1:172.21.15.55:6789/0 conn(0x7f09d806e730 legacy=0x7f09d815ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:52.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.784+0000 7f09d659c640 1 -- 172.21.15.55:0/2815296081 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09d806d970 con 0x7f09d81542f0 2024-03-19T02:00:52.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.788+0000 7f09d659c640 1 --2- 172.21.15.55:0/2815296081 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f09d81542f0 0x7f09d8067000 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f09c80097c0 tx=0x7f09c801ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6164bf8ab557b02 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:52.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.796+0000 7f09d559a640 1 -- 172.21.15.55:0/2815296081 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c80b5070 con 0x7f09d81542f0 2024-03-19T02:00:52.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.796+0000 7f09d559a640 1 -- 172.21.15.55:0/2815296081 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09c80a8990 con 0x7f09d81542f0 2024-03-19T02:00:52.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.796+0000 7f09d559a640 1 -- 172.21.15.55:0/2815296081 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c80a8d60 con 0x7f09d81542f0 2024-03-19T02:00:52.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.796+0000 7f09dca4e640 1 -- 172.21.15.55:0/2815296081 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f09d81542f0 msgr2=0x7f09d8067000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:52.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.796+0000 7f09dca4e640 1 --2- 172.21.15.55:0/2815296081 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f09d81542f0 0x7f09d8067000 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f09c80097c0 tx=0x7f09c801ec90 comp rx=0 tx=0).stop 2024-03-19T02:00:52.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.796+0000 7f09dca4e640 1 -- 172.21.15.55:0/2815296081 shutdown_connections 2024-03-19T02:00:52.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.796+0000 7f09dca4e640 1 --2- 172.21.15.55:0/2815296081 >> v2:172.21.15.84:3300/0 conn(0x7f09d80c6ea0 0x7f09d806b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:52.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.796+0000 7f09dca4e640 1 --2- 172.21.15.55:0/2815296081 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f09d81542f0 0x7f09d8067000 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:52.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.796+0000 7f09dca4e640 1 -- 172.21.15.55:0/2815296081 >> 172.21.15.55:0/2815296081 conn(0x7f09d80cc200 msgr2=0x7f09d80cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:52.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.796+0000 7f09dca4e640 1 -- 172.21.15.55:0/2815296081 shutdown_connections 2024-03-19T02:00:52.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.796+0000 7f09dca4e640 1 -- 172.21.15.55:0/2815296081 wait complete. 2024-03-19T02:00:52.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.800+0000 7f09dca4e640 1 Processor -- start 2024-03-19T02:00:52.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.800+0000 7f09dca4e640 1 -- start start 2024-03-19T02:00:52.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.800+0000 7f09dca4e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09d806e730 0x7f09d8145df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:52.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.800+0000 7f09dca4e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09d80c6ea0 0x7f09d81529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:52.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.800+0000 7f09dca4e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f09d81542f0 0x7f09d8152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:52.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.800+0000 7f09dca4e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f09d806e210 con 0x7f09d80c6ea0 2024-03-19T02:00:52.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.800+0000 7f09dca4e640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f09d806df30 con 0x7f09d806e730 2024-03-19T02:00:52.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.800+0000 7f09dca4e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f09d806bef0 con 0x7f09d81542f0 2024-03-19T02:00:52.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.800+0000 7f09d5d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09d80c6ea0 0x7f09d81529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:52.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.800+0000 7f09d659c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09d806e730 0x7f09d8145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:52.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.800+0000 7f09d5d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09d80c6ea0 0x7f09d81529b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57920/0 (socket says 172.21.15.55:57920) 2024-03-19T02:00:52.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.800+0000 7f09d5d9b640 1 -- 172.21.15.55:0/1771217526 learned_addr learned my addr 172.21.15.55:0/1771217526 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:52.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.800+0000 7f09d6d9d640 1 --2- 172.21.15.55:0/1771217526 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f09d81542f0 0x7f09d8152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:52.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.804+0000 7f09d5d9b640 1 -- 172.21.15.55:0/1771217526 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f09d81542f0 msgr2=0x7f09d8152ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:52.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.804+0000 7f09d5d9b640 1 --2- 172.21.15.55:0/1771217526 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f09d81542f0 0x7f09d8152ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:52.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.804+0000 7f09d5d9b640 1 -- 172.21.15.55:0/1771217526 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09d806e730 msgr2=0x7f09d8145df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:52.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.804+0000 7f09d5d9b640 1 --2- 172.21.15.55:0/1771217526 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09d806e730 0x7f09d8145df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:52.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.804+0000 7f09d5d9b640 1 -- 172.21.15.55:0/1771217526 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09d8153460 con 0x7f09d80c6ea0 2024-03-19T02:00:52.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.808+0000 7f09d5d9b640 1 --2- 172.21.15.55:0/1771217526 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09d80c6ea0 0x7f09d81529b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00c480 tx=0x7f09cc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:52.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.812+0000 7f09c6ffd640 1 -- 172.21.15.55:0/1771217526 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc017070 con 0x7f09d80c6ea0 2024-03-19T02:00:52.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.812+0000 7f09c6ffd640 1 -- 172.21.15.55:0/1771217526 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09cc005cf0 con 0x7f09d80c6ea0 2024-03-19T02:00:52.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.812+0000 7f09dca4e640 1 -- 172.21.15.55:0/1771217526 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09d81f43e0 con 0x7f09d80c6ea0 2024-03-19T02:00:52.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.812+0000 7f09c6ffd640 1 -- 172.21.15.55:0/1771217526 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc007290 con 0x7f09d80c6ea0 2024-03-19T02:00:52.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.812+0000 7f09dca4e640 1 -- 172.21.15.55:0/1771217526 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09d8151a80 con 0x7f09d80c6ea0 2024-03-19T02:00:52.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.812+0000 7f09dca4e640 1 -- 172.21.15.55:0/1771217526 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09d8068a40 con 0x7f09d80c6ea0 2024-03-19T02:00:52.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.836+0000 7f09c6ffd640 1 -- 172.21.15.55:0/1771217526 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f09cc01a070 con 0x7f09d80c6ea0 2024-03-19T02:00:52.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.836+0000 7f09c6ffd640 1 --2- 172.21.15.55:0/1771217526 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f09b80676e0 0x7f09b8069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:52.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.836+0000 7f09d659c640 1 --2- 172.21.15.55:0/1771217526 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f09b80676e0 0x7f09b8069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:52.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:52.836+0000 7f09d659c640 1 --2- 172.21.15.55:0/1771217526 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f09b80676e0 0x7f09b8069ba0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f09c80b2000 tx=0x7f09c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:53.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:53.136+0000 7f09c6ffd640 1 -- 172.21.15.55:0/1771217526 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f09cc0537a0 con 0x7f09d80c6ea0 2024-03-19T02:00:53.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:53.136+0000 7f09c6ffd640 1 -- 172.21.15.55:0/1771217526 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f09cc021020 con 0x7f09d80c6ea0 2024-03-19T02:00:53.388 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:53.388+0000 7f09dca4e640 1 -- 172.21.15.55:0/1771217526 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09d814a320 con 0x7f09d80c6ea0 2024-03-19T02:00:53.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:53.416+0000 7f09c6ffd640 1 -- 172.21.15.55:0/1771217526 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f09cc055700 con 0x7f09d80c6ea0 2024-03-19T02:00:53.416 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:53.416 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:53.417 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:53.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:53.420+0000 7f09dca4e640 1 -- 172.21.15.55:0/1771217526 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f09b80676e0 msgr2=0x7f09b8069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:53.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:53.420+0000 7f09dca4e640 1 --2- 172.21.15.55:0/1771217526 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f09b80676e0 0x7f09b8069ba0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f09c80b2000 tx=0x7f09c8008040 comp rx=0 tx=0).stop 2024-03-19T02:00:53.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:53.420+0000 7f09dca4e640 1 -- 172.21.15.55:0/1771217526 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09d80c6ea0 msgr2=0x7f09d81529b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:53.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:53.420+0000 7f09dca4e640 1 --2- 172.21.15.55:0/1771217526 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09d80c6ea0 0x7f09d81529b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00c480 tx=0x7f09cc00c940 comp rx=0 tx=0).stop 2024-03-19T02:00:53.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:53.420+0000 7f09dca4e640 1 -- 172.21.15.55:0/1771217526 shutdown_connections 2024-03-19T02:00:53.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:53.420+0000 7f09dca4e640 1 --2- 172.21.15.55:0/1771217526 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f09b80676e0 0x7f09b8069ba0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:53.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:53.420+0000 7f09dca4e640 1 --2- 172.21.15.55:0/1771217526 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f09d81542f0 0x7f09d8152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:53.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:53.420+0000 7f09dca4e640 1 --2- 172.21.15.55:0/1771217526 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09d80c6ea0 0x7f09d81529b0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:53.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:53.420+0000 7f09dca4e640 1 --2- 172.21.15.55:0/1771217526 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09d806e730 0x7f09d8145df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:53.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:53.420+0000 7f09dca4e640 1 -- 172.21.15.55:0/1771217526 >> 172.21.15.55:0/1771217526 conn(0x7f09d80cc200 msgr2=0x7f09d81467a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:53.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:53.420+0000 7f09dca4e640 1 -- 172.21.15.55:0/1771217526 shutdown_connections 2024-03-19T02:00:53.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:53.420+0000 7f09dca4e640 1 -- 172.21.15.55:0/1771217526 wait complete. 2024-03-19T02:00:54.040 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:54.040 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:55.041 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:55.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.168+0000 7f2087c1a640 1 Processor -- start 2024-03-19T02:00:55.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.168+0000 7f2087c1a640 1 -- start start 2024-03-19T02:00:55.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.172+0000 7f2087c1a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2080154340 0x7f208015e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:55.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.172+0000 7f2087c1a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f20800c73c0 0x7f208015aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:55.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.172+0000 7f2087c1a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2080071280 con 0x7f2080154720 2024-03-19T02:00:55.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.172+0000 7f2087c1a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f2080073840 con 0x7f2080154340 2024-03-19T02:00:55.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.172+0000 7f2087c1a640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f20800c7850 con 0x7f20800c73c0 2024-03-19T02:00:55.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.172+0000 7f20859b6640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2080154340 0x7f208015e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:55.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.172+0000 7f20861b7640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f20800c73c0 0x7f208015aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:55.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.172+0000 7f20851b5640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f2080154720 0x7f2080072e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:46850/0 (socket says 172.21.15.55:46850) 2024-03-19T02:00:55.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.172+0000 7f20851b5640 1 -- 172.21.15.55:0/3301812928 learned_addr learned my addr 172.21.15.55:0/3301812928 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:00:55.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.176+0000 7f20859b6640 1 -- 172.21.15.55:0/3301812928 >> v2:172.21.15.84:3300/0 conn(0x7f20800c73c0 msgr2=0x7f208015aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:55.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.176+0000 7f20859b6640 1 --2- 172.21.15.55:0/3301812928 >> v2:172.21.15.84:3300/0 conn(0x7f20800c73c0 0x7f208015aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:55.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.176+0000 7f20859b6640 1 -- 172.21.15.55:0/3301812928 >> v1:172.21.15.55:6789/0 conn(0x7f2080154720 legacy=0x7f2080072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:55.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.176+0000 7f20859b6640 1 -- 172.21.15.55:0/3301812928 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f208015d900 con 0x7f2080154340 2024-03-19T02:00:55.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.180+0000 7f20859b6640 1 --2- 172.21.15.55:0/3301812928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2080154340 0x7f208015e3a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f2080154b40 tx=0x7f207c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca5cd479ffca1a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:55.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.184+0000 7f20849b4640 1 -- 172.21.15.55:0/3301812928 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207c0b0070 con 0x7f2080154340 2024-03-19T02:00:55.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.184+0000 7f20849b4640 1 -- 172.21.15.55:0/3301812928 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f207c0ab600 con 0x7f2080154340 2024-03-19T02:00:55.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.184+0000 7f20849b4640 1 -- 172.21.15.55:0/3301812928 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207c0ab9d0 con 0x7f2080154340 2024-03-19T02:00:55.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.184+0000 7f2087c1a640 1 -- 172.21.15.55:0/3301812928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2080154340 msgr2=0x7f208015e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:55.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.184+0000 7f2087c1a640 1 --2- 172.21.15.55:0/3301812928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2080154340 0x7f208015e3a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f2080154b40 tx=0x7f207c01c460 comp rx=0 tx=0).stop 2024-03-19T02:00:55.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.184+0000 7f2087c1a640 1 -- 172.21.15.55:0/3301812928 shutdown_connections 2024-03-19T02:00:55.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.184+0000 7f2087c1a640 1 --2- 172.21.15.55:0/3301812928 >> v2:172.21.15.84:3300/0 conn(0x7f20800c73c0 0x7f208015aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:55.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.184+0000 7f2087c1a640 1 --2- 172.21.15.55:0/3301812928 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2080154340 0x7f208015e3a0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:55.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.184+0000 7f2087c1a640 1 -- 172.21.15.55:0/3301812928 >> 172.21.15.55:0/3301812928 conn(0x7f20800cc9c0 msgr2=0x7f20800ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:55.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.184+0000 7f2087c1a640 1 -- 172.21.15.55:0/3301812928 shutdown_connections 2024-03-19T02:00:55.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.184+0000 7f2087c1a640 1 -- 172.21.15.55:0/3301812928 wait complete. 2024-03-19T02:00:55.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.188+0000 7f2087c1a640 1 Processor -- start 2024-03-19T02:00:55.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.188+0000 7f2087c1a640 1 -- start start 2024-03-19T02:00:55.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.188+0000 7f2087c1a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f20800c73c0 0x7f2080150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:55.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.188+0000 7f2087c1a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2080154340 0x7f208014a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:55.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.188+0000 7f2087c1a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2080154720 0x7f208014a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:55.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.188+0000 7f2087c1a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2080166b70 con 0x7f2080154340 2024-03-19T02:00:55.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.188+0000 7f2087c1a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f208015b1e0 con 0x7f20800c73c0 2024-03-19T02:00:55.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.188+0000 7f2087c1a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2080166cf0 con 0x7f2080154720 2024-03-19T02:00:55.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.188+0000 7f20851b5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2080154340 0x7f208014a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:55.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.188+0000 7f20861b7640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2080154720 0x7f208014a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:55.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.192+0000 7f20859b6640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f20800c73c0 0x7f2080150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:55.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.192+0000 7f20861b7640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2080154720 0x7f208014a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:55988/0 (socket says 172.21.15.55:55988) 2024-03-19T02:00:55.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.192+0000 7f20861b7640 1 -- 172.21.15.55:0/1360284395 learned_addr learned my addr 172.21.15.55:0/1360284395 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:55.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.196+0000 7f20851b5640 1 -- 172.21.15.55:0/1360284395 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2080154720 msgr2=0x7f208014a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:55.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.196+0000 7f20851b5640 1 --2- 172.21.15.55:0/1360284395 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2080154720 0x7f208014a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:55.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.196+0000 7f20851b5640 1 -- 172.21.15.55:0/1360284395 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f20800c73c0 msgr2=0x7f2080150f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:55.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.196+0000 7f20851b5640 1 --2- 172.21.15.55:0/1360284395 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f20800c73c0 0x7f2080150f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:55.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.196+0000 7f20851b5640 1 -- 172.21.15.55:0/1360284395 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2080158960 con 0x7f2080154340 2024-03-19T02:00:55.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.200+0000 7f20851b5640 1 --2- 172.21.15.55:0/1360284395 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2080154340 0x7f208014a2b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f207000c3c0 tx=0x7f207000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:55.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.204+0000 7f20767fc640 1 -- 172.21.15.55:0/1360284395 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2070017070 con 0x7f2080154340 2024-03-19T02:00:55.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.204+0000 7f20767fc640 1 -- 172.21.15.55:0/1360284395 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2070005cf0 con 0x7f2080154340 2024-03-19T02:00:55.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.204+0000 7f20767fc640 1 -- 172.21.15.55:0/1360284395 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20700072f0 con 0x7f2080154340 2024-03-19T02:00:55.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.204+0000 7f2087c1a640 1 -- 172.21.15.55:0/1360284395 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f208014f600 con 0x7f2080154340 2024-03-19T02:00:55.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.204+0000 7f2087c1a640 1 -- 172.21.15.55:0/1360284395 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f208014f8e0 con 0x7f2080154340 2024-03-19T02:00:55.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.204+0000 7f2087c1a640 1 -- 172.21.15.55:0/1360284395 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2080068a40 con 0x7f2080154340 2024-03-19T02:00:55.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.468+0000 7f20767fc640 1 -- 172.21.15.55:0/1360284395 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f207001a070 con 0x7f2080154340 2024-03-19T02:00:55.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.468+0000 7f20767fc640 1 --2- 172.21.15.55:0/1360284395 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2064067660 0x7f2064069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:55.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.468+0000 7f20859b6640 1 --2- 172.21.15.55:0/1360284395 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2064067660 0x7f2064069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:55.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.468+0000 7f20859b6640 1 --2- 172.21.15.55:0/1360284395 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2064067660 0x7f2064069b20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f207c0ad000 tx=0x7f207c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:55.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.480+0000 7f20767fc640 1 -- 172.21.15.55:0/1360284395 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f20700535b0 con 0x7f2080154340 2024-03-19T02:00:55.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.480+0000 7f20767fc640 1 -- 172.21.15.55:0/1360284395 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2070021020 con 0x7f2080154340 2024-03-19T02:00:55.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.692+0000 7f2087c1a640 1 -- 172.21.15.55:0/1360284395 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20800706a0 con 0x7f2080154340 2024-03-19T02:00:55.717 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.716+0000 7f20767fc640 1 -- 172.21.15.55:0/1360284395 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f2070055510 con 0x7f2080154340 2024-03-19T02:00:55.717 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:55.717 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:55.718 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:55.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.720+0000 7f2087c1a640 1 -- 172.21.15.55:0/1360284395 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2064067660 msgr2=0x7f2064069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:55.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.720+0000 7f2087c1a640 1 --2- 172.21.15.55:0/1360284395 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2064067660 0x7f2064069b20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f207c0ad000 tx=0x7f207c008040 comp rx=0 tx=0).stop 2024-03-19T02:00:55.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.720+0000 7f2087c1a640 1 -- 172.21.15.55:0/1360284395 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2080154340 msgr2=0x7f208014a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:55.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.720+0000 7f2087c1a640 1 --2- 172.21.15.55:0/1360284395 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2080154340 0x7f208014a2b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f207000c3c0 tx=0x7f207000c880 comp rx=0 tx=0).stop 2024-03-19T02:00:55.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.720+0000 7f2087c1a640 1 -- 172.21.15.55:0/1360284395 shutdown_connections 2024-03-19T02:00:55.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.720+0000 7f2087c1a640 1 --2- 172.21.15.55:0/1360284395 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2064067660 0x7f2064069b20 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:55.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.720+0000 7f2087c1a640 1 --2- 172.21.15.55:0/1360284395 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2080154720 0x7f208014a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:55.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.720+0000 7f2087c1a640 1 --2- 172.21.15.55:0/1360284395 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2080154340 0x7f208014a2b0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:55.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.720+0000 7f2087c1a640 1 --2- 172.21.15.55:0/1360284395 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f20800c73c0 0x7f2080150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:55.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.720+0000 7f2087c1a640 1 -- 172.21.15.55:0/1360284395 >> 172.21.15.55:0/1360284395 conn(0x7f20800cc9c0 msgr2=0x7f20800cb5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:55.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.720+0000 7f2087c1a640 1 -- 172.21.15.55:0/1360284395 shutdown_connections 2024-03-19T02:00:55.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:55.720+0000 7f2087c1a640 1 -- 172.21.15.55:0/1360284395 wait complete. 2024-03-19T02:00:56.341 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:56.341 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:57.342 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:57.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.468+0000 7f0ea631a640 1 Processor -- start 2024-03-19T02:00:57.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.468+0000 7f0ea631a640 1 -- start start 2024-03-19T02:00:57.471 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.468+0000 7f0ea631a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0ea01438c0 0x7f0ea00c9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:57.471 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.468+0000 7f0ea631a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ea0070bd0 con 0x7f0ea015a920 2024-03-19T02:00:57.471 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.468+0000 7f0ea631a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ea0070e90 con 0x7f0ea015a540 2024-03-19T02:00:57.471 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.472+0000 7f0ea631a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f0ea006d440 con 0x7f0ea01438c0 2024-03-19T02:00:57.472 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.472+0000 7f0e9f7fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f0ea015a920 0x7f0ea01431b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:46852/0 (socket says 172.21.15.55:46852) 2024-03-19T02:00:57.472 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.472+0000 7f0e9f7fe640 1 -- 172.21.15.55:0/1127729511 learned_addr learned my addr 172.21.15.55:0/1127729511 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:00:57.472 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.472+0000 7f0ea48b7640 1 --2- 172.21.15.55:0/1127729511 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0ea01438c0 0x7f0ea00c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:57.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.476+0000 7f0e9effd640 1 -- 172.21.15.55:0/1127729511 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3823427502 0 0) 0x7f0ea0070bd0 con 0x7f0ea015a920 2024-03-19T02:00:57.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.476+0000 7f0e9effd640 1 -- 172.21.15.55:0/1127729511 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e88003530 con 0x7f0ea015a920 2024-03-19T02:00:57.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.476+0000 7f0e9effd640 1 -- 172.21.15.55:0/1127729511 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 520194037 0 0) 0x7f0ea0070e90 con 0x7f0ea015a540 2024-03-19T02:00:57.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.476+0000 7f0e9effd640 1 -- 172.21.15.55:0/1127729511 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ea0070bd0 con 0x7f0ea015a540 2024-03-19T02:00:57.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.480+0000 7f0e9effd640 1 -- 172.21.15.55:0/1127729511 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2366464887 0 0) 0x7f0e88003530 con 0x7f0ea015a920 2024-03-19T02:00:57.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.480+0000 7f0e9effd640 1 -- 172.21.15.55:0/1127729511 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0ea01438c0 msgr2=0x7f0ea00c9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:57.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.480+0000 7f0e9effd640 1 --2- 172.21.15.55:0/1127729511 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0ea01438c0 0x7f0ea00c9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:57.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.480+0000 7f0e9effd640 1 -- 172.21.15.55:0/1127729511 >> v1:172.21.15.84:6789/0 conn(0x7f0ea015a540 legacy=0x7f0ea006cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:57.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.480+0000 7f0e9effd640 1 -- 172.21.15.55:0/1127729511 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ea006f830 con 0x7f0ea015a920 2024-03-19T02:00:57.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.480+0000 7f0e9effd640 1 -- 172.21.15.55:0/1127729511 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f0e94001680 con 0x7f0ea015a920 2024-03-19T02:00:57.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.480+0000 7f0e9effd640 1 -- 172.21.15.55:0/1127729511 >> v1:172.21.15.55:6789/0 conn(0x7f0ea015a920 legacy=0x7f0ea01431b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:57.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.480+0000 7f0e9effd640 1 --2- 172.21.15.55:0/1127729511 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0e88003b80 0x7f0e88003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:57.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.480+0000 7f0e9effd640 1 --2- 172.21.15.55:0/1127729511 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e88004810 0x7f0e88004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:57.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.480+0000 7f0e9effd640 1 --2- 172.21.15.55:0/1127729511 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0ea01438c0 0x7f0e880051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:57.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.480+0000 7f0e9effd640 1 -- 172.21.15.55:0/1127729511 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8800ba80 con 0x7f0e88004810 2024-03-19T02:00:57.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.480+0000 7f0e9effd640 1 -- 172.21.15.55:0/1127729511 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8800bc00 con 0x7f0e88003b80 2024-03-19T02:00:57.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.480+0000 7f0e9effd640 1 -- 172.21.15.55:0/1127729511 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f0e8800bdd0 con 0x7f0ea01438c0 2024-03-19T02:00:57.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.480+0000 7f0e9f7fe640 1 --2- 172.21.15.55:0/1127729511 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e88004810 0x7f0e88004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:57.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.484+0000 7f0e9f7fe640 1 -- 172.21.15.55:0/1127729511 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0ea01438c0 msgr2=0x7f0e880051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:57.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.484+0000 7f0e9f7fe640 1 --2- 172.21.15.55:0/1127729511 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0ea01438c0 0x7f0e880051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:57.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.484+0000 7f0e9f7fe640 1 -- 172.21.15.55:0/1127729511 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0e88003b80 msgr2=0x7f0e88003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:00:57.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.484+0000 7f0e9f7fe640 1 --2- 172.21.15.55:0/1127729511 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0e88003b80 0x7f0e88003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:57.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.484+0000 7f0e9f7fe640 1 -- 172.21.15.55:0/1127729511 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ea006f830 con 0x7f0e88004810 2024-03-19T02:00:57.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.488+0000 7f0e9f7fe640 1 --2- 172.21.15.55:0/1127729511 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e88004810 0x7f0e88004c90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400c290 tx=0x7f0e9401f370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d5835221eb169d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:57.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.496+0000 7f0e9effd640 1 -- 172.21.15.55:0/1127729511 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e94007d40 con 0x7f0e88004810 2024-03-19T02:00:57.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.496+0000 7f0e9effd640 1 -- 172.21.15.55:0/1127729511 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e94005b00 con 0x7f0e88004810 2024-03-19T02:00:57.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.496+0000 7f0e9effd640 1 -- 172.21.15.55:0/1127729511 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e94007070 con 0x7f0e88004810 2024-03-19T02:00:57.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.496+0000 7f0ea631a640 1 -- 172.21.15.55:0/1127729511 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e88004810 msgr2=0x7f0e88004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:57.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.496+0000 7f0ea631a640 1 --2- 172.21.15.55:0/1127729511 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e88004810 0x7f0e88004c90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400c290 tx=0x7f0e9401f370 comp rx=0 tx=0).stop 2024-03-19T02:00:57.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.496+0000 7f0ea48b7640 1 -- 172.21.15.55:0/1127729511 reap_dead start 2024-03-19T02:00:57.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.496+0000 7f0ea631a640 1 -- 172.21.15.55:0/1127729511 shutdown_connections 2024-03-19T02:00:57.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.496+0000 7f0ea631a640 1 -- 172.21.15.55:0/1127729511 >> 172.21.15.55:0/1127729511 conn(0x7f0ea0067600 msgr2=0x7f0ea0071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:57.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.496+0000 7f0ea631a640 1 -- 172.21.15.55:0/1127729511 shutdown_connections 2024-03-19T02:00:57.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.496+0000 7f0ea631a640 1 -- 172.21.15.55:0/1127729511 wait complete. 2024-03-19T02:00:57.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.496+0000 7f0ea631a640 1 Processor -- start 2024-03-19T02:00:57.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.496+0000 7f0ea631a640 1 -- start start 2024-03-19T02:00:57.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.500+0000 7f0ea631a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0e88004810 0x7f0ea00c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:57.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.500+0000 7f0ea631a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ea015c250 0x7f0ea00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:57.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.500+0000 7f0ea631a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0ea00cc4d0 0x7f0ea00cc950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:57.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.500+0000 7f0ea631a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea006fdf0 con 0x7f0ea015c250 2024-03-19T02:00:57.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.500+0000 7f0ea631a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea006f5e0 con 0x7f0ea00cc4d0 2024-03-19T02:00:57.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.500+0000 7f0ea631a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f0ea0070860 con 0x7f0e88004810 2024-03-19T02:00:57.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.500+0000 7f0e9f7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ea015c250 0x7f0ea00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:57.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.500+0000 7f0e9f7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ea015c250 0x7f0ea00ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57954/0 (socket says 172.21.15.55:57954) 2024-03-19T02:00:57.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.500+0000 7f0e9f7fe640 1 -- 172.21.15.55:0/1220412703 learned_addr learned my addr 172.21.15.55:0/1220412703 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:57.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.500+0000 7f0ea48b7640 1 --2- 172.21.15.55:0/1220412703 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0ea00cc4d0 0x7f0ea00cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:57.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.500+0000 7f0e9ffff640 1 --2- 172.21.15.55:0/1220412703 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0e88004810 0x7f0ea00c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:57.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.504+0000 7f0e9f7fe640 1 -- 172.21.15.55:0/1220412703 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0e88004810 msgr2=0x7f0ea00c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:57.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.504+0000 7f0e9f7fe640 1 --2- 172.21.15.55:0/1220412703 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0e88004810 0x7f0ea00c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:57.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.504+0000 7f0e9f7fe640 1 -- 172.21.15.55:0/1220412703 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0ea00cc4d0 msgr2=0x7f0ea00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:57.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.504+0000 7f0e9f7fe640 1 --2- 172.21.15.55:0/1220412703 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0ea00cc4d0 0x7f0ea00cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:57.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.504+0000 7f0e9f7fe640 1 -- 172.21.15.55:0/1220412703 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ea00c5090 con 0x7f0ea015c250 2024-03-19T02:00:57.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.508+0000 7f0e9f7fe640 1 --2- 172.21.15.55:0/1220412703 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ea015c250 0x7f0ea00ce360 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f0e94025fe0 tx=0x7f0e88005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:57.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.512+0000 7f0e9cff9640 1 -- 172.21.15.55:0/1220412703 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e94007d40 con 0x7f0ea015c250 2024-03-19T02:00:57.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.512+0000 7f0e9cff9640 1 -- 172.21.15.55:0/1220412703 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e94005cb0 con 0x7f0ea015c250 2024-03-19T02:00:57.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.512+0000 7f0e9cff9640 1 -- 172.21.15.55:0/1220412703 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e94005e60 con 0x7f0ea015c250 2024-03-19T02:00:57.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.512+0000 7f0ea631a640 1 -- 172.21.15.55:0/1220412703 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ea00c2eb0 con 0x7f0ea015c250 2024-03-19T02:00:57.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.512+0000 7f0ea631a640 1 -- 172.21.15.55:0/1220412703 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ea00cd2c0 con 0x7f0ea015c250 2024-03-19T02:00:57.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.516+0000 7f0ea631a640 1 -- 172.21.15.55:0/1220412703 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ea0154630 con 0x7f0ea015c250 2024-03-19T02:00:57.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.792+0000 7f0e9cff9640 1 -- 172.21.15.55:0/1220412703 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f0e940047f0 con 0x7f0ea015c250 2024-03-19T02:00:57.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.792+0000 7f0e9cff9640 1 --2- 172.21.15.55:0/1220412703 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0e8c0676b0 0x7f0e8c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:57.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.792+0000 7f0e9ffff640 1 --2- 172.21.15.55:0/1220412703 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0e8c0676b0 0x7f0e8c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:57.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.792+0000 7f0e9ffff640 1 --2- 172.21.15.55:0/1220412703 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0e8c0676b0 0x7f0e8c069b70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0e90000f70 tx=0x7f0e90007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:57.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.832+0000 7f0e9cff9640 1 -- 172.21.15.55:0/1220412703 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f0e940f17d0 con 0x7f0ea015c250 2024-03-19T02:00:57.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:57.832+0000 7f0e9cff9640 1 -- 172.21.15.55:0/1220412703 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f0e940b7070 con 0x7f0ea015c250 2024-03-19T02:00:58.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:58.052+0000 7f0ea631a640 1 -- 172.21.15.55:0/1220412703 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ea00ce8a0 con 0x7f0ea015c250 2024-03-19T02:00:58.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:58.072+0000 7f0e9cff9640 1 -- 172.21.15.55:0/1220412703 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f0e94019080 con 0x7f0ea015c250 2024-03-19T02:00:58.074 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:00:58.074 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:00:58.074 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:00:58.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:58.076+0000 7f0ea631a640 1 -- 172.21.15.55:0/1220412703 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0e8c0676b0 msgr2=0x7f0e8c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:58.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:58.076+0000 7f0ea631a640 1 --2- 172.21.15.55:0/1220412703 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0e8c0676b0 0x7f0e8c069b70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0e90000f70 tx=0x7f0e90007d60 comp rx=0 tx=0).stop 2024-03-19T02:00:58.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:58.076+0000 7f0ea631a640 1 -- 172.21.15.55:0/1220412703 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ea015c250 msgr2=0x7f0ea00ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:58.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:58.076+0000 7f0ea631a640 1 --2- 172.21.15.55:0/1220412703 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ea015c250 0x7f0ea00ce360 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f0e94025fe0 tx=0x7f0e88005840 comp rx=0 tx=0).stop 2024-03-19T02:00:58.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:58.076+0000 7f0ea631a640 1 -- 172.21.15.55:0/1220412703 shutdown_connections 2024-03-19T02:00:58.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:58.076+0000 7f0ea631a640 1 --2- 172.21.15.55:0/1220412703 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0e8c0676b0 0x7f0e8c069b70 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:58.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:58.076+0000 7f0ea631a640 1 --2- 172.21.15.55:0/1220412703 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0ea00cc4d0 0x7f0ea00cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:58.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:58.076+0000 7f0ea631a640 1 --2- 172.21.15.55:0/1220412703 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0ea015c250 0x7f0ea00ce360 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:58.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:58.076+0000 7f0ea631a640 1 --2- 172.21.15.55:0/1220412703 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0e88004810 0x7f0ea00c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:58.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:58.076+0000 7f0ea631a640 1 -- 172.21.15.55:0/1220412703 >> 172.21.15.55:0/1220412703 conn(0x7f0ea0067600 msgr2=0x7f0ea00c8be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:58.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:58.076+0000 7f0ea631a640 1 -- 172.21.15.55:0/1220412703 shutdown_connections 2024-03-19T02:00:58.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:58.076+0000 7f0ea631a640 1 -- 172.21.15.55:0/1220412703 wait complete. 2024-03-19T02:00:58.697 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:00:58.698 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:00:59.699 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:00:59.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.816+0000 7f191ea88640 1 Processor -- start 2024-03-19T02:00:59.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.816+0000 7f191ea88640 1 -- start start 2024-03-19T02:00:59.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.820+0000 7f191ea88640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f191815be50 0x7f19180c87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:59.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.820+0000 7f191ea88640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1918157fa0 con 0x7f191813df90 2024-03-19T02:00:59.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.820+0000 7f191ea88640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1918158260 con 0x7f19180c7530 2024-03-19T02:00:59.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.820+0000 7f191ea88640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f1918158540 con 0x7f191815be50 2024-03-19T02:00:59.820 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.820+0000 7f191d025640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f191815be50 0x7f19180c87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:59.820 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.820+0000 7f1917fff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f19180c7530 0x7f191815b740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:41918/0 (socket says 172.21.15.55:41918) 2024-03-19T02:00:59.820 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.820+0000 7f1917fff640 1 -- 172.21.15.55:0/2943177208 learned_addr learned my addr 172.21.15.55:0/2943177208 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:00:59.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.824+0000 7f191d025640 1 -- 172.21.15.55:0/2943177208 >> v1:172.21.15.84:6789/0 conn(0x7f19180c7530 legacy=0x7f191815b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:59.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.824+0000 7f191d025640 1 -- 172.21.15.55:0/2943177208 >> v1:172.21.15.55:6789/0 conn(0x7f191813df90 legacy=0x7f19180c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:59.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.824+0000 7f191d025640 1 -- 172.21.15.55:0/2943177208 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1918158820 con 0x7f191815be50 2024-03-19T02:00:59.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.828+0000 7f191d025640 1 --2- 172.21.15.55:0/2943177208 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f191815be50 0x7f19180c87b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1910009d90 tx=0x7f191001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66ea701fb726396a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:59.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.832+0000 7f19177fe640 1 -- 172.21.15.55:0/2943177208 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19100b2070 con 0x7f191815be50 2024-03-19T02:00:59.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.832+0000 7f19177fe640 1 -- 172.21.15.55:0/2943177208 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19100a58b0 con 0x7f191815be50 2024-03-19T02:00:59.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.832+0000 7f19177fe640 1 -- 172.21.15.55:0/2943177208 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19100a5c80 con 0x7f191815be50 2024-03-19T02:00:59.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.832+0000 7f191ea88640 1 -- 172.21.15.55:0/2943177208 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f191815be50 msgr2=0x7f19180c87b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:59.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.832+0000 7f191ea88640 1 --2- 172.21.15.55:0/2943177208 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f191815be50 0x7f19180c87b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1910009d90 tx=0x7f191001c8e0 comp rx=0 tx=0).stop 2024-03-19T02:00:59.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.832+0000 7f191ea88640 1 -- 172.21.15.55:0/2943177208 shutdown_connections 2024-03-19T02:00:59.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.832+0000 7f191ea88640 1 --2- 172.21.15.55:0/2943177208 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f191815be50 0x7f19180c87b0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:59.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.832+0000 7f191ea88640 1 -- 172.21.15.55:0/2943177208 >> 172.21.15.55:0/2943177208 conn(0x7f191806e3c0 msgr2=0x7f1918072030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:00:59.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.832+0000 7f191ea88640 1 -- 172.21.15.55:0/2943177208 shutdown_connections 2024-03-19T02:00:59.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.832+0000 7f191ea88640 1 -- 172.21.15.55:0/2943177208 wait complete. 2024-03-19T02:00:59.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.836+0000 7f191ea88640 1 Processor -- start 2024-03-19T02:00:59.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.836+0000 7f191ea88640 1 -- start start 2024-03-19T02:00:59.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.836+0000 7f191ea88640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f19180c7530 0x7f191815acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:59.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.836+0000 7f191ea88640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f191813df90 0x7f1918150c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:59.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.836+0000 7f191ea88640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f191815be50 0x7f19181511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:00:59.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.836+0000 7f191ea88640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f19181666e0 con 0x7f191815be50 2024-03-19T02:00:59.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.836+0000 7f191ea88640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f1918166560 con 0x7f19180c7530 2024-03-19T02:00:59.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.836+0000 7f191ea88640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1918166860 con 0x7f191813df90 2024-03-19T02:00:59.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.836+0000 7f191d025640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f191815be50 0x7f19181511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:59.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.836+0000 7f191d025640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f191815be50 0x7f19181511b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40612/0 (socket says 172.21.15.55:40612) 2024-03-19T02:00:59.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.836+0000 7f191d025640 1 -- 172.21.15.55:0/2611898006 learned_addr learned my addr 172.21.15.55:0/2611898006 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:00:59.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.836+0000 7f1917fff640 1 --2- 172.21.15.55:0/2611898006 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f191813df90 0x7f1918150c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:00:59.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.840+0000 7f191d025640 1 -- 172.21.15.55:0/2611898006 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f191813df90 msgr2=0x7f1918150c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:59.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.840+0000 7f191d025640 1 --2- 172.21.15.55:0/2611898006 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f191813df90 0x7f1918150c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:59.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.840+0000 7f191d025640 1 -- 172.21.15.55:0/2611898006 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f19180c7530 msgr2=0x7f191815acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:00:59.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.840+0000 7f191d025640 1 --2- 172.21.15.55:0/2611898006 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f19180c7530 0x7f191815acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:00:59.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.840+0000 7f191d025640 1 -- 172.21.15.55:0/2611898006 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1918156dc0 con 0x7f191815be50 2024-03-19T02:00:59.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.844+0000 7f191d025640 1 --2- 172.21.15.55:0/2611898006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f191815be50 0x7f19181511b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f191001a510 tx=0x7f1910019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:00:59.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.848+0000 7f19157fa640 1 -- 172.21.15.55:0/2611898006 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19100bd020 con 0x7f191815be50 2024-03-19T02:00:59.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.848+0000 7f19157fa640 1 -- 172.21.15.55:0/2611898006 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19100ae6d0 con 0x7f191815be50 2024-03-19T02:00:59.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.848+0000 7f19157fa640 1 -- 172.21.15.55:0/2611898006 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19100b2040 con 0x7f191815be50 2024-03-19T02:00:59.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.848+0000 7f191ea88640 1 -- 172.21.15.55:0/2611898006 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1918162c10 con 0x7f191815be50 2024-03-19T02:00:59.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.848+0000 7f191ea88640 1 -- 172.21.15.55:0/2611898006 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19181631d0 con 0x7f191815be50 2024-03-19T02:00:59.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:00:59.848+0000 7f191ea88640 1 -- 172.21.15.55:0/2611898006 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1918068a40 con 0x7f191815be50 2024-03-19T02:01:00.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:00.124+0000 7f19157fa640 1 -- 172.21.15.55:0/2611898006 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f19100c0020 con 0x7f191815be50 2024-03-19T02:01:00.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:00.124+0000 7f19157fa640 1 --2- 172.21.15.55:0/2611898006 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f18fc067660 0x7f18fc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:00.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:00.124+0000 7f191c824640 1 --2- 172.21.15.55:0/2611898006 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f18fc067660 0x7f18fc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:00.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:00.128+0000 7f191c824640 1 --2- 172.21.15.55:0/2611898006 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f18fc067660 0x7f18fc069b20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f190c000f70 tx=0x7f190c0079e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:00.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:00.152+0000 7f19157fa640 1 -- 172.21.15.55:0/2611898006 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f19100ef5b0 con 0x7f191815be50 2024-03-19T02:01:00.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:00.152+0000 7f19157fa640 1 -- 172.21.15.55:0/2611898006 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f19100c7030 con 0x7f191815be50 2024-03-19T02:01:00.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:00.356+0000 7f191ea88640 1 -- 172.21.15.55:0/2611898006 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19180c95f0 con 0x7f191815be50 2024-03-19T02:01:00.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:00.372+0000 7f19157fa640 1 -- 172.21.15.55:0/2611898006 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f1910016560 con 0x7f191815be50 2024-03-19T02:01:00.375 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:00.375 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:00.375 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:00.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:00.376+0000 7f191ea88640 1 -- 172.21.15.55:0/2611898006 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f18fc067660 msgr2=0x7f18fc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:00.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:00.376+0000 7f191ea88640 1 --2- 172.21.15.55:0/2611898006 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f18fc067660 0x7f18fc069b20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f190c000f70 tx=0x7f190c0079e0 comp rx=0 tx=0).stop 2024-03-19T02:01:00.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:00.376+0000 7f191ea88640 1 -- 172.21.15.55:0/2611898006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f191815be50 msgr2=0x7f19181511b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:00.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:00.376+0000 7f191ea88640 1 --2- 172.21.15.55:0/2611898006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f191815be50 0x7f19181511b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f191001a510 tx=0x7f1910019680 comp rx=0 tx=0).stop 2024-03-19T02:01:00.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:00.376+0000 7f191ea88640 1 -- 172.21.15.55:0/2611898006 shutdown_connections 2024-03-19T02:01:00.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:00.376+0000 7f191ea88640 1 --2- 172.21.15.55:0/2611898006 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f18fc067660 0x7f18fc069b20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:00.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:00.376+0000 7f191ea88640 1 --2- 172.21.15.55:0/2611898006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f191815be50 0x7f19181511b0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:00.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:00.376+0000 7f191ea88640 1 --2- 172.21.15.55:0/2611898006 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f191813df90 0x7f1918150c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:00.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:00.376+0000 7f191ea88640 1 --2- 172.21.15.55:0/2611898006 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f19180c7530 0x7f191815acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:00.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:00.376+0000 7f191ea88640 1 -- 172.21.15.55:0/2611898006 >> 172.21.15.55:0/2611898006 conn(0x7f191806e3c0 msgr2=0x7f191806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:00.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:00.376+0000 7f191ea88640 1 -- 172.21.15.55:0/2611898006 shutdown_connections 2024-03-19T02:01:00.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:00.376+0000 7f191ea88640 1 -- 172.21.15.55:0/2611898006 wait complete. 2024-03-19T02:01:00.998 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:00.998 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:01.999 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:02.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.124+0000 7fae0745a640 1 Processor -- start 2024-03-19T02:01:02.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.124+0000 7fae0745a640 1 -- start start 2024-03-19T02:01:02.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.124+0000 7fae0745a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fae000c3320 0x7fae0015b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:02.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.124+0000 7fae0745a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae00151330 con 0x7fae000ca350 2024-03-19T02:01:02.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.124+0000 7fae0745a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae000c3840 con 0x7fae00150f50 2024-03-19T02:01:02.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.124+0000 7fae0745a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fae000c3e00 con 0x7fae000c3320 2024-03-19T02:01:02.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.128+0000 7fae049f5640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fae000ca350 0x7fae000ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:54918/0 (socket says 172.21.15.55:54918) 2024-03-19T02:01:02.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.128+0000 7fae049f5640 1 -- 172.21.15.55:0/3263674695 learned_addr learned my addr 172.21.15.55:0/3263674695 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:01:02.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.128+0000 7fae059f7640 1 --2- 172.21.15.55:0/3263674695 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fae000c3320 0x7fae0015b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:02.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.128+0000 7fadf7fff640 1 -- 172.21.15.55:0/3263674695 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3359880201 0 0) 0x7fae00151330 con 0x7fae000ca350 2024-03-19T02:01:02.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.128+0000 7fadf7fff640 1 -- 172.21.15.55:0/3263674695 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fadec005740 con 0x7fae000ca350 2024-03-19T02:01:02.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.132+0000 7fae059f7640 1 -- 172.21.15.55:0/3263674695 >> v1:172.21.15.84:6789/0 conn(0x7fae00150f50 legacy=0x7fae00157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:02.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.132+0000 7fae059f7640 1 -- 172.21.15.55:0/3263674695 >> v1:172.21.15.55:6789/0 conn(0x7fae000ca350 legacy=0x7fae000ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:02.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.132+0000 7fae059f7640 1 -- 172.21.15.55:0/3263674695 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae000c3b20 con 0x7fae000c3320 2024-03-19T02:01:02.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.136+0000 7fae059f7640 1 --2- 172.21.15.55:0/3263674695 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fae000c3320 0x7fae0015b0a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fadfc009dc0 tx=0x7fadfc01c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2c512e665a5dca2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:02.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.144+0000 7fadf7fff640 1 -- 172.21.15.55:0/3263674695 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadfc0b2070 con 0x7fae000c3320 2024-03-19T02:01:02.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.144+0000 7fadf7fff640 1 -- 172.21.15.55:0/3263674695 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadfc0a5840 con 0x7fae000c3320 2024-03-19T02:01:02.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.144+0000 7fadf7fff640 1 -- 172.21.15.55:0/3263674695 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadfc0a5c10 con 0x7fae000c3320 2024-03-19T02:01:02.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.144+0000 7fae0745a640 1 -- 172.21.15.55:0/3263674695 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fae000c3320 msgr2=0x7fae0015b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:02.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.144+0000 7fae0745a640 1 --2- 172.21.15.55:0/3263674695 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fae000c3320 0x7fae0015b0a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fadfc009dc0 tx=0x7fadfc01c990 comp rx=0 tx=0).stop 2024-03-19T02:01:02.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.144+0000 7fae0745a640 1 -- 172.21.15.55:0/3263674695 shutdown_connections 2024-03-19T02:01:02.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.144+0000 7fae0745a640 1 --2- 172.21.15.55:0/3263674695 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fae000c3320 0x7fae0015b0a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:02.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.144+0000 7fae0745a640 1 -- 172.21.15.55:0/3263674695 >> 172.21.15.55:0/3263674695 conn(0x7fae0006e670 msgr2=0x7fae0006ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:02.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.144+0000 7fae0745a640 1 -- 172.21.15.55:0/3263674695 shutdown_connections 2024-03-19T02:01:02.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.144+0000 7fae0745a640 1 -- 172.21.15.55:0/3263674695 wait complete. 2024-03-19T02:01:02.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.144+0000 7fae0745a640 1 Processor -- start 2024-03-19T02:01:02.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.144+0000 7fae0745a640 1 -- start start 2024-03-19T02:01:02.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.144+0000 7fae0745a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fae000c3320 0x7fae0013c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:02.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.144+0000 7fae0745a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae000ca350 0x7fae0013a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:02.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.144+0000 7fae0745a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae00150f50 0x7fae00139390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:02.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.144+0000 7fae0745a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fae000cd7f0 con 0x7fae000ca350 2024-03-19T02:01:02.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.144+0000 7fae0745a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fae000cd670 con 0x7fae00150f50 2024-03-19T02:01:02.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.144+0000 7fae0745a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fae0006bab0 con 0x7fae000c3320 2024-03-19T02:01:02.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.144+0000 7fae049f5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae000ca350 0x7fae0013a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:02.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.148+0000 7fae049f5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae000ca350 0x7fae0013a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40628/0 (socket says 172.21.15.55:40628) 2024-03-19T02:01:02.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.148+0000 7fae049f5640 1 -- 172.21.15.55:0/4078497893 learned_addr learned my addr 172.21.15.55:0/4078497893 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:02.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.148+0000 7fae059f7640 1 --2- 172.21.15.55:0/4078497893 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae00150f50 0x7fae00139390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:02.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.148+0000 7fae051f6640 1 --2- 172.21.15.55:0/4078497893 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fae000c3320 0x7fae0013c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:02.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.152+0000 7fae049f5640 1 -- 172.21.15.55:0/4078497893 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fae000c3320 msgr2=0x7fae0013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:02.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.152+0000 7fae049f5640 1 --2- 172.21.15.55:0/4078497893 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fae000c3320 0x7fae0013c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:02.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.152+0000 7fae049f5640 1 -- 172.21.15.55:0/4078497893 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae00150f50 msgr2=0x7fae00139390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:02.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.152+0000 7fae049f5640 1 --2- 172.21.15.55:0/4078497893 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae00150f50 0x7fae00139390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:02.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.152+0000 7fae049f5640 1 -- 172.21.15.55:0/4078497893 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae0013cdd0 con 0x7fae000ca350 2024-03-19T02:01:02.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.152+0000 7fae049f5640 1 --2- 172.21.15.55:0/4078497893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae000ca350 0x7fae0013a330 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fadf000c500 tx=0x7fadf000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:02.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.156+0000 7fadf5ffb640 1 -- 172.21.15.55:0/4078497893 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf0017070 con 0x7fae000ca350 2024-03-19T02:01:02.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.156+0000 7fae0745a640 1 -- 172.21.15.55:0/4078497893 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae001f6af0 con 0x7fae000ca350 2024-03-19T02:01:02.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.156+0000 7fae0745a640 1 -- 172.21.15.55:0/4078497893 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae001f70b0 con 0x7fae000ca350 2024-03-19T02:01:02.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.156+0000 7fae0745a640 1 -- 172.21.15.55:0/4078497893 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae00068a40 con 0x7fae000ca350 2024-03-19T02:01:02.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.432+0000 7fadf5ffb640 1 -- 172.21.15.55:0/4078497893 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadf0005cf0 con 0x7fae000ca350 2024-03-19T02:01:02.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.432+0000 7fadf5ffb640 1 -- 172.21.15.55:0/4078497893 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf0021060 con 0x7fae000ca350 2024-03-19T02:01:02.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.436+0000 7fadf5ffb640 1 -- 172.21.15.55:0/4078497893 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fadf001a070 con 0x7fae000ca350 2024-03-19T02:01:02.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.436+0000 7fadf5ffb640 1 --2- 172.21.15.55:0/4078497893 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fade4067690 0x7fade4069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:02.437 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.436+0000 7fae051f6640 1 --2- 172.21.15.55:0/4078497893 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fade4067690 0x7fade4069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:02.437 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.436+0000 7fae051f6640 1 --2- 172.21.15.55:0/4078497893 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fade4067690 0x7fade4069b50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fadf8007fd0 tx=0x7fadf8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:02.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.452+0000 7fadf5ffb640 1 -- 172.21.15.55:0/4078497893 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fadf0053b40 con 0x7fae000ca350 2024-03-19T02:01:02.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.452+0000 7fadf5ffb640 1 -- 172.21.15.55:0/4078497893 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fadf0055aa0 con 0x7fae000ca350 2024-03-19T02:01:02.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.704+0000 7fae0745a640 1 -- 172.21.15.55:0/4078497893 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae00144180 con 0x7fae000ca350 2024-03-19T02:01:02.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.720+0000 7fadf5ffb640 1 -- 172.21.15.55:0/4078497893 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fadf0059b40 con 0x7fae000ca350 2024-03-19T02:01:02.720 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:02.720 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:02.721 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:02.722 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.720+0000 7fae0745a640 1 -- 172.21.15.55:0/4078497893 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fade4067690 msgr2=0x7fade4069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:02.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.720+0000 7fae0745a640 1 --2- 172.21.15.55:0/4078497893 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fade4067690 0x7fade4069b50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fadf8007fd0 tx=0x7fadf8009040 comp rx=0 tx=0).stop 2024-03-19T02:01:02.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.720+0000 7fae0745a640 1 -- 172.21.15.55:0/4078497893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae000ca350 msgr2=0x7fae0013a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:02.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.720+0000 7fae0745a640 1 --2- 172.21.15.55:0/4078497893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae000ca350 0x7fae0013a330 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fadf000c500 tx=0x7fadf000c9c0 comp rx=0 tx=0).stop 2024-03-19T02:01:02.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.720+0000 7fae0745a640 1 -- 172.21.15.55:0/4078497893 shutdown_connections 2024-03-19T02:01:02.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.720+0000 7fae0745a640 1 --2- 172.21.15.55:0/4078497893 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fade4067690 0x7fade4069b50 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:02.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.720+0000 7fae0745a640 1 --2- 172.21.15.55:0/4078497893 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae00150f50 0x7fae00139390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:02.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.720+0000 7fae0745a640 1 --2- 172.21.15.55:0/4078497893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae000ca350 0x7fae0013a330 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:02.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.720+0000 7fae0745a640 1 --2- 172.21.15.55:0/4078497893 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fae000c3320 0x7fae0013c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:02.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.720+0000 7fae0745a640 1 -- 172.21.15.55:0/4078497893 >> 172.21.15.55:0/4078497893 conn(0x7fae0006e670 msgr2=0x7fae000ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:02.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.720+0000 7fae0745a640 1 -- 172.21.15.55:0/4078497893 shutdown_connections 2024-03-19T02:01:02.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:02.720+0000 7fae0745a640 1 -- 172.21.15.55:0/4078497893 wait complete. 2024-03-19T02:01:03.348 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:03.348 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:04.349 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:04.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.464+0000 7fde1bbcd640 1 Processor -- start 2024-03-19T02:01:04.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.464+0000 7fde1bbcd640 1 -- start start 2024-03-19T02:01:04.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.464+0000 7fde1bbcd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fde140ca0c0 0x7fde1406bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:04.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.464+0000 7fde1bbcd640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde1415b0c0 con 0x7fde14154060 2024-03-19T02:01:04.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.464+0000 7fde1bbcd640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde140ccd80 con 0x7fde14070410 2024-03-19T02:01:04.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.464+0000 7fde1bbcd640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fde140ccb50 con 0x7fde140ca0c0 2024-03-19T02:01:04.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.464+0000 7fde19168640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fde140ca0c0 0x7fde1406bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:04.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.464+0000 7fde1a16a640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fde14070410 0x7fde14164ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:41934/0 (socket says 172.21.15.55:41934) 2024-03-19T02:01:04.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.464+0000 7fde1a16a640 1 -- 172.21.15.55:0/2371265686 learned_addr learned my addr 172.21.15.55:0/2371265686 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:01:04.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.468+0000 7fde18967640 1 -- 172.21.15.55:0/2371265686 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3790882375 0 0) 0x7fde140ccd80 con 0x7fde14070410 2024-03-19T02:01:04.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.468+0000 7fde18967640 1 -- 172.21.15.55:0/2371265686 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde00003530 con 0x7fde14070410 2024-03-19T02:01:04.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.468+0000 7fde19168640 1 -- 172.21.15.55:0/2371265686 >> v1:172.21.15.84:6789/0 conn(0x7fde14070410 legacy=0x7fde14164ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:04.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.468+0000 7fde19168640 1 -- 172.21.15.55:0/2371265686 >> v1:172.21.15.55:6789/0 conn(0x7fde14154060 legacy=0x7fde1415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:04.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.468+0000 7fde19168640 1 -- 172.21.15.55:0/2371265686 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde140cd040 con 0x7fde140ca0c0 2024-03-19T02:01:04.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.472+0000 7fde19168640 1 --2- 172.21.15.55:0/2371265686 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde140ca0c0 0x7fde1406bcb0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fde040097c0 tx=0x7fde0401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6fcd373a54cf8f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:04.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.480+0000 7fde18967640 1 -- 172.21.15.55:0/2371265686 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde040b5070 con 0x7fde140ca0c0 2024-03-19T02:01:04.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.480+0000 7fde18967640 1 -- 172.21.15.55:0/2371265686 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde040a8990 con 0x7fde140ca0c0 2024-03-19T02:01:04.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.480+0000 7fde18967640 1 -- 172.21.15.55:0/2371265686 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde040a8d60 con 0x7fde140ca0c0 2024-03-19T02:01:04.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.480+0000 7fde1bbcd640 1 -- 172.21.15.55:0/2371265686 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde140ca0c0 msgr2=0x7fde1406bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:04.480 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.480+0000 7fde1bbcd640 1 --2- 172.21.15.55:0/2371265686 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde140ca0c0 0x7fde1406bcb0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fde040097c0 tx=0x7fde0401ec90 comp rx=0 tx=0).stop 2024-03-19T02:01:04.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.480+0000 7fde1bbcd640 1 -- 172.21.15.55:0/2371265686 shutdown_connections 2024-03-19T02:01:04.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.480+0000 7fde1bbcd640 1 --2- 172.21.15.55:0/2371265686 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde140ca0c0 0x7fde1406bcb0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:04.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.480+0000 7fde1bbcd640 1 -- 172.21.15.55:0/2371265686 >> 172.21.15.55:0/2371265686 conn(0x7fde1406c3b0 msgr2=0x7fde1406c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:04.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.480+0000 7fde1bbcd640 1 -- 172.21.15.55:0/2371265686 shutdown_connections 2024-03-19T02:01:04.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.480+0000 7fde1bbcd640 1 -- 172.21.15.55:0/2371265686 wait complete. 2024-03-19T02:01:04.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.480+0000 7fde1bbcd640 1 Processor -- start 2024-03-19T02:01:04.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.480+0000 7fde1bbcd640 1 -- start start 2024-03-19T02:01:04.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.480+0000 7fde1bbcd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fde14070410 0x7fde14159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:04.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.480+0000 7fde1bbcd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde140ca0c0 0x7fde1415b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:04.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.480+0000 7fde1bbcd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde14154060 0x7fde1415bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:04.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.480+0000 7fde1bbcd640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fde140cd8e0 con 0x7fde14154060 2024-03-19T02:01:04.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.480+0000 7fde1bbcd640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fde140cd600 con 0x7fde140ca0c0 2024-03-19T02:01:04.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.480+0000 7fde1bbcd640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fde1415a490 con 0x7fde14070410 2024-03-19T02:01:04.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.480+0000 7fde1a16a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde14154060 0x7fde1415bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:04.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.484+0000 7fde19168640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde140ca0c0 0x7fde1415b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:04.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.484+0000 7fde19969640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fde14070410 0x7fde14159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:04.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.484+0000 7fde1a16a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde14154060 0x7fde1415bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40650/0 (socket says 172.21.15.55:40650) 2024-03-19T02:01:04.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.484+0000 7fde1a16a640 1 -- 172.21.15.55:0/3034910931 learned_addr learned my addr 172.21.15.55:0/3034910931 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:04.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.484+0000 7fde1a16a640 1 -- 172.21.15.55:0/3034910931 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fde14070410 msgr2=0x7fde14159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:04.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.484+0000 7fde1a16a640 1 --2- 172.21.15.55:0/3034910931 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fde14070410 0x7fde14159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:04.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.484+0000 7fde1a16a640 1 -- 172.21.15.55:0/3034910931 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde140ca0c0 msgr2=0x7fde1415b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:04.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.484+0000 7fde1a16a640 1 --2- 172.21.15.55:0/3034910931 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde140ca0c0 0x7fde1415b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:04.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.484+0000 7fde1a16a640 1 -- 172.21.15.55:0/3034910931 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde141efca0 con 0x7fde14154060 2024-03-19T02:01:04.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.488+0000 7fde1a16a640 1 --2- 172.21.15.55:0/3034910931 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde14154060 0x7fde1415bcf0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fde0c00cad0 tx=0x7fde0c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:04.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.492+0000 7fde0a7fc640 1 -- 172.21.15.55:0/3034910931 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0c017070 con 0x7fde14154060 2024-03-19T02:01:04.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.492+0000 7fde0a7fc640 1 -- 172.21.15.55:0/3034910931 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde0c00ac30 con 0x7fde14154060 2024-03-19T02:01:04.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.492+0000 7fde1bbcd640 1 -- 172.21.15.55:0/3034910931 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde141eff80 con 0x7fde14154060 2024-03-19T02:01:04.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.492+0000 7fde0a7fc640 1 -- 172.21.15.55:0/3034910931 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0c012400 con 0x7fde14154060 2024-03-19T02:01:04.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.492+0000 7fde1bbcd640 1 -- 172.21.15.55:0/3034910931 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde141f0540 con 0x7fde14154060 2024-03-19T02:01:04.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.496+0000 7fde1bbcd640 1 -- 172.21.15.55:0/3034910931 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde14068a40 con 0x7fde14154060 2024-03-19T02:01:04.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.532+0000 7fde0a7fc640 1 -- 172.21.15.55:0/3034910931 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fde0c01a070 con 0x7fde14154060 2024-03-19T02:01:04.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.532+0000 7fde0a7fc640 1 --2- 172.21.15.55:0/3034910931 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fddf80676e0 0x7fddf8069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:04.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.532+0000 7fde19969640 1 --2- 172.21.15.55:0/3034910931 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fddf80676e0 0x7fddf8069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:04.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.536+0000 7fde19969640 1 --2- 172.21.15.55:0/3034910931 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fddf80676e0 0x7fddf8069ba0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fde10001810 tx=0x7fde10009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:04.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.836+0000 7fde0a7fc640 1 -- 172.21.15.55:0/3034910931 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fde0c0540b0 con 0x7fde14154060 2024-03-19T02:01:04.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:04.836+0000 7fde0a7fc640 1 -- 172.21.15.55:0/3034910931 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fde0c021020 con 0x7fde14154060 2024-03-19T02:01:05.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:05.044+0000 7fde1bbcd640 1 -- 172.21.15.55:0/3034910931 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde140c9570 con 0x7fde14154060 2024-03-19T02:01:05.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:05.068+0000 7fde0a7fc640 1 -- 172.21.15.55:0/3034910931 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fde0c056010 con 0x7fde14154060 2024-03-19T02:01:05.069 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:05.069 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:05.069 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:05.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:05.068+0000 7fde1bbcd640 1 -- 172.21.15.55:0/3034910931 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fddf80676e0 msgr2=0x7fddf8069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:05.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:05.068+0000 7fde1bbcd640 1 --2- 172.21.15.55:0/3034910931 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fddf80676e0 0x7fddf8069ba0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fde10001810 tx=0x7fde10009040 comp rx=0 tx=0).stop 2024-03-19T02:01:05.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:05.068+0000 7fde1bbcd640 1 -- 172.21.15.55:0/3034910931 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde14154060 msgr2=0x7fde1415bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:05.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:05.068+0000 7fde1bbcd640 1 --2- 172.21.15.55:0/3034910931 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde14154060 0x7fde1415bcf0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fde0c00cad0 tx=0x7fde0c00cf90 comp rx=0 tx=0).stop 2024-03-19T02:01:05.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:05.068+0000 7fde1bbcd640 1 -- 172.21.15.55:0/3034910931 shutdown_connections 2024-03-19T02:01:05.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:05.068+0000 7fde1bbcd640 1 --2- 172.21.15.55:0/3034910931 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fddf80676e0 0x7fddf8069ba0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:05.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:05.068+0000 7fde1bbcd640 1 --2- 172.21.15.55:0/3034910931 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde14154060 0x7fde1415bcf0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:05.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:05.068+0000 7fde1bbcd640 1 --2- 172.21.15.55:0/3034910931 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde140ca0c0 0x7fde1415b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:05.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:05.068+0000 7fde1bbcd640 1 --2- 172.21.15.55:0/3034910931 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fde14070410 0x7fde14159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:05.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:05.068+0000 7fde1bbcd640 1 -- 172.21.15.55:0/3034910931 >> 172.21.15.55:0/3034910931 conn(0x7fde1406c3b0 msgr2=0x7fde1406bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:05.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:05.068+0000 7fde1bbcd640 1 -- 172.21.15.55:0/3034910931 shutdown_connections 2024-03-19T02:01:05.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:05.068+0000 7fde1bbcd640 1 -- 172.21.15.55:0/3034910931 wait complete. 2024-03-19T02:01:05.691 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:05.692 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:06.693 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:06.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.808+0000 7f72c1e8d640 1 Processor -- start 2024-03-19T02:01:06.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.808+0000 7f72c1e8d640 1 -- start start 2024-03-19T02:01:06.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.808+0000 7f72c1e8d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f72bc14e930 0x7f72bc06b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:06.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.808+0000 7f72c1e8d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f72bc0c60d0 0x7f72bc0c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:06.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.808+0000 7f72c1e8d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72bc0c7540 con 0x7f72bc06be50 2024-03-19T02:01:06.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.808+0000 7f72c1e8d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f72bc0c80a0 con 0x7f72bc14e930 2024-03-19T02:01:06.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.808+0000 7f72c1e8d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f72bc0c7800 con 0x7f72bc0c60d0 2024-03-19T02:01:06.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.808+0000 7f72bb7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f72bc14e930 0x7f72bc06b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:06.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.808+0000 7f72baffd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f72bc06be50 0x7f72bc1585a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:41948/0 (socket says 172.21.15.55:41948) 2024-03-19T02:01:06.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.808+0000 7f72baffd640 1 -- 172.21.15.55:0/3468844635 learned_addr learned my addr 172.21.15.55:0/3468844635 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:01:06.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.808+0000 7f72bbfff640 1 --2- 172.21.15.55:0/3468844635 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f72bc0c60d0 0x7f72bc0c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:06.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.812+0000 7f72bb7fe640 1 -- 172.21.15.55:0/3468844635 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f72bc0c60d0 msgr2=0x7f72bc0c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:06.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.812+0000 7f72bb7fe640 1 --2- 172.21.15.55:0/3468844635 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f72bc0c60d0 0x7f72bc0c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:06.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.812+0000 7f72bb7fe640 1 -- 172.21.15.55:0/3468844635 >> v1:172.21.15.84:6789/0 conn(0x7f72bc06be50 legacy=0x7f72bc1585a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:06.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.812+0000 7f72bb7fe640 1 -- 172.21.15.55:0/3468844635 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72bc0c7ae0 con 0x7f72bc14e930 2024-03-19T02:01:06.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.816+0000 7f72bb7fe640 1 --2- 172.21.15.55:0/3468844635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72bc14e930 0x7f72bc06b910 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f72ac0097c0 tx=0x7f72ac01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e39d7f3f63c01d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:06.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.820+0000 7f72ba7fc640 1 -- 172.21.15.55:0/3468844635 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ac0b5070 con 0x7f72bc14e930 2024-03-19T02:01:06.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.820+0000 7f72ba7fc640 1 -- 172.21.15.55:0/3468844635 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72ac0a8990 con 0x7f72bc14e930 2024-03-19T02:01:06.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.820+0000 7f72ba7fc640 1 -- 172.21.15.55:0/3468844635 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ac0a8d60 con 0x7f72bc14e930 2024-03-19T02:01:06.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.820+0000 7f72c1e8d640 1 -- 172.21.15.55:0/3468844635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72bc14e930 msgr2=0x7f72bc06b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:06.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.820+0000 7f72c1e8d640 1 --2- 172.21.15.55:0/3468844635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72bc14e930 0x7f72bc06b910 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f72ac0097c0 tx=0x7f72ac01ec90 comp rx=0 tx=0).stop 2024-03-19T02:01:06.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.820+0000 7f72c1e8d640 1 -- 172.21.15.55:0/3468844635 shutdown_connections 2024-03-19T02:01:06.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.820+0000 7f72c1e8d640 1 --2- 172.21.15.55:0/3468844635 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f72bc0c60d0 0x7f72bc0c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:06.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.820+0000 7f72c1e8d640 1 --2- 172.21.15.55:0/3468844635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72bc14e930 0x7f72bc06b910 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:06.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.820+0000 7f72c1e8d640 1 -- 172.21.15.55:0/3468844635 >> 172.21.15.55:0/3468844635 conn(0x7f72bc0cc9c0 msgr2=0x7f72bc0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:06.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.820+0000 7f72c1e8d640 1 -- 172.21.15.55:0/3468844635 shutdown_connections 2024-03-19T02:01:06.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.820+0000 7f72c1e8d640 1 -- 172.21.15.55:0/3468844635 wait complete. 2024-03-19T02:01:06.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.820+0000 7f72c1e8d640 1 Processor -- start 2024-03-19T02:01:06.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.824+0000 7f72c1e8d640 1 -- start start 2024-03-19T02:01:06.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.824+0000 7f72c1e8d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f72bc06be50 0x7f72bc06c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:06.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.824+0000 7f72c1e8d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f72bc0c60d0 0x7f72bc1e2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:06.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.824+0000 7f72c1e8d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72bc14e930 0x7f72bc1e3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:06.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.824+0000 7f72c1e8d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f72bc14ed10 con 0x7f72bc14e930 2024-03-19T02:01:06.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.824+0000 7f72c1e8d640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f72bc158080 con 0x7f72bc0c60d0 2024-03-19T02:01:06.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.824+0000 7f72c1e8d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f72bc073600 con 0x7f72bc06be50 2024-03-19T02:01:06.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.824+0000 7f72bbfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72bc14e930 0x7f72bc1e3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:06.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.824+0000 7f72baffd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f72bc0c60d0 0x7f72bc1e2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:06.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.824+0000 7f72bbfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72bc14e930 0x7f72bc1e3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40660/0 (socket says 172.21.15.55:40660) 2024-03-19T02:01:06.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.824+0000 7f72bbfff640 1 -- 172.21.15.55:0/2597308303 learned_addr learned my addr 172.21.15.55:0/2597308303 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:06.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.824+0000 7f72bb7fe640 1 --2- 172.21.15.55:0/2597308303 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f72bc06be50 0x7f72bc06c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:06.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.828+0000 7f72bbfff640 1 -- 172.21.15.55:0/2597308303 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f72bc06be50 msgr2=0x7f72bc06c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:06.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.828+0000 7f72bbfff640 1 --2- 172.21.15.55:0/2597308303 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f72bc06be50 0x7f72bc06c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:06.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.828+0000 7f72bbfff640 1 -- 172.21.15.55:0/2597308303 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f72bc0c60d0 msgr2=0x7f72bc1e2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:06.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.828+0000 7f72bbfff640 1 --2- 172.21.15.55:0/2597308303 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f72bc0c60d0 0x7f72bc1e2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:06.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.828+0000 7f72bbfff640 1 -- 172.21.15.55:0/2597308303 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72bc1eec40 con 0x7f72bc14e930 2024-03-19T02:01:06.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.832+0000 7f72bbfff640 1 --2- 172.21.15.55:0/2597308303 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72bc14e930 0x7f72bc1e3230 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f72b400efc0 tx=0x7f72b400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:06.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.836+0000 7f72a3fff640 1 -- 172.21.15.55:0/2597308303 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b4018020 con 0x7f72bc14e930 2024-03-19T02:01:06.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.836+0000 7f72a3fff640 1 -- 172.21.15.55:0/2597308303 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72b4014480 con 0x7f72bc14e930 2024-03-19T02:01:06.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.836+0000 7f72c1e8d640 1 -- 172.21.15.55:0/2597308303 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72bc1576e0 con 0x7f72bc14e930 2024-03-19T02:01:06.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.836+0000 7f72a3fff640 1 -- 172.21.15.55:0/2597308303 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b4014870 con 0x7f72bc14e930 2024-03-19T02:01:06.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.836+0000 7f72c1e8d640 1 -- 172.21.15.55:0/2597308303 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72bc1579a0 con 0x7f72bc14e930 2024-03-19T02:01:06.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:06.836+0000 7f72c1e8d640 1 -- 172.21.15.55:0/2597308303 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72bc068a40 con 0x7f72bc14e930 2024-03-19T02:01:07.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:07.128+0000 7f72a3fff640 1 -- 172.21.15.55:0/2597308303 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f72b401b020 con 0x7f72bc14e930 2024-03-19T02:01:07.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:07.128+0000 7f72a3fff640 1 --2- 172.21.15.55:0/2597308303 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f72a8067660 0x7f72a8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:07.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:07.128+0000 7f72bb7fe640 1 --2- 172.21.15.55:0/2597308303 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f72a8067660 0x7f72a8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:07.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:07.128+0000 7f72bb7fe640 1 --2- 172.21.15.55:0/2597308303 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f72a8067660 0x7f72a8069b20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f72ac0b2000 tx=0x7f72ac0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:07.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:07.164+0000 7f72a3fff640 1 -- 172.21.15.55:0/2597308303 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f72b4055120 con 0x7f72bc14e930 2024-03-19T02:01:07.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:07.164+0000 7f72a3fff640 1 -- 172.21.15.55:0/2597308303 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f72b4022070 con 0x7f72bc14e930 2024-03-19T02:01:07.369 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:07.372+0000 7f72c1e8d640 1 -- 172.21.15.55:0/2597308303 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f72bc152310 con 0x7f72bc14e930 2024-03-19T02:01:07.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:07.396+0000 7f72a3fff640 1 -- 172.21.15.55:0/2597308303 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f72b4010040 con 0x7f72bc14e930 2024-03-19T02:01:07.393 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:07.393 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:07.394 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:07.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:07.400+0000 7f72c1e8d640 1 -- 172.21.15.55:0/2597308303 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f72a8067660 msgr2=0x7f72a8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:07.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:07.400+0000 7f72c1e8d640 1 --2- 172.21.15.55:0/2597308303 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f72a8067660 0x7f72a8069b20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f72ac0b2000 tx=0x7f72ac0aa070 comp rx=0 tx=0).stop 2024-03-19T02:01:07.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:07.400+0000 7f72c1e8d640 1 -- 172.21.15.55:0/2597308303 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72bc14e930 msgr2=0x7f72bc1e3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:07.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:07.400+0000 7f72c1e8d640 1 --2- 172.21.15.55:0/2597308303 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72bc14e930 0x7f72bc1e3230 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f72b400efc0 tx=0x7f72b400c490 comp rx=0 tx=0).stop 2024-03-19T02:01:07.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:07.400+0000 7f72c1e8d640 1 -- 172.21.15.55:0/2597308303 shutdown_connections 2024-03-19T02:01:07.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:07.400+0000 7f72c1e8d640 1 --2- 172.21.15.55:0/2597308303 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f72a8067660 0x7f72a8069b20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:07.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:07.400+0000 7f72c1e8d640 1 --2- 172.21.15.55:0/2597308303 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72bc14e930 0x7f72bc1e3230 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:07.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:07.400+0000 7f72c1e8d640 1 --2- 172.21.15.55:0/2597308303 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f72bc0c60d0 0x7f72bc1e2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:07.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:07.400+0000 7f72c1e8d640 1 --2- 172.21.15.55:0/2597308303 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f72bc06be50 0x7f72bc06c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:07.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:07.400+0000 7f72c1e8d640 1 -- 172.21.15.55:0/2597308303 >> 172.21.15.55:0/2597308303 conn(0x7f72bc0cc9c0 msgr2=0x7f72bc14bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:07.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:07.400+0000 7f72c1e8d640 1 -- 172.21.15.55:0/2597308303 shutdown_connections 2024-03-19T02:01:07.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:07.400+0000 7f72c1e8d640 1 -- 172.21.15.55:0/2597308303 wait complete. 2024-03-19T02:01:08.016 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:08.016 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:09.018 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:09.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.136+0000 7fe7a3706640 1 Processor -- start 2024-03-19T02:01:09.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.136+0000 7fe7a3706640 1 -- start start 2024-03-19T02:01:09.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.136+0000 7fe7a3706640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe79c13dc50 0x7fe79c0c5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:09.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.136+0000 7fe7a3706640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe79c15c090 0x7fe79c0c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:09.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.136+0000 7fe7a3706640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe79c1581c0 con 0x7fe79c13e030 2024-03-19T02:01:09.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.136+0000 7fe7a3706640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fe79c158460 con 0x7fe79c15c090 2024-03-19T02:01:09.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.136+0000 7fe7a3706640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fe79c158740 con 0x7fe79c13dc50 2024-03-19T02:01:09.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.136+0000 7fe7a1ca3640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe79c15c090 0x7fe79c0c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:09.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.136+0000 7fe7a14a2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe79c13dc50 0x7fe79c0c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:09.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.136+0000 7fe7a1ca3640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe79c15c090 0x7fe79c0c5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57304/0 (socket says 172.21.15.55:57304) 2024-03-19T02:01:09.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.136+0000 7fe7a1ca3640 1 -- 172.21.15.55:0/3105140557 learned_addr learned my addr 172.21.15.55:0/3105140557 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:09.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.140+0000 7fe7a14a2640 1 -- 172.21.15.55:0/3105140557 >> v2:172.21.15.55:3300/0 conn(0x7fe79c15c090 msgr2=0x7fe79c0c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:09.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.140+0000 7fe7a14a2640 1 --2- 172.21.15.55:0/3105140557 >> v2:172.21.15.55:3300/0 conn(0x7fe79c15c090 0x7fe79c0c5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:09.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.140+0000 7fe7a14a2640 1 -- 172.21.15.55:0/3105140557 >> v1:172.21.15.55:6789/0 conn(0x7fe79c13e030 legacy=0x7fe79c15b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:09.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.140+0000 7fe7a14a2640 1 -- 172.21.15.55:0/3105140557 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe79c158a20 con 0x7fe79c13dc50 2024-03-19T02:01:09.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.144+0000 7fe7a14a2640 1 --2- 172.21.15.55:0/3105140557 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe79c13dc50 0x7fe79c0c5810 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe78c009d90 tx=0x7fe78c01f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32509b474ac33014 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:09.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.148+0000 7fe79bfff640 1 -- 172.21.15.55:0/3105140557 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78c0b4070 con 0x7fe79c13dc50 2024-03-19T02:01:09.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.152+0000 7fe79bfff640 1 -- 172.21.15.55:0/3105140557 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe78c0af600 con 0x7fe79c13dc50 2024-03-19T02:01:09.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.152+0000 7fe79bfff640 1 -- 172.21.15.55:0/3105140557 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78c0af9d0 con 0x7fe79c13dc50 2024-03-19T02:01:09.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.152+0000 7fe7a3706640 1 -- 172.21.15.55:0/3105140557 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe79c13dc50 msgr2=0x7fe79c0c5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:09.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.152+0000 7fe7a3706640 1 --2- 172.21.15.55:0/3105140557 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe79c13dc50 0x7fe79c0c5810 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe78c009d90 tx=0x7fe78c01f150 comp rx=0 tx=0).stop 2024-03-19T02:01:09.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.156+0000 7fe7a3706640 1 -- 172.21.15.55:0/3105140557 shutdown_connections 2024-03-19T02:01:09.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.156+0000 7fe7a3706640 1 --2- 172.21.15.55:0/3105140557 >> v2:172.21.15.55:3300/0 conn(0x7fe79c15c090 0x7fe79c0c5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:09.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.156+0000 7fe7a3706640 1 --2- 172.21.15.55:0/3105140557 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe79c13dc50 0x7fe79c0c5810 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:09.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.156+0000 7fe7a3706640 1 -- 172.21.15.55:0/3105140557 >> 172.21.15.55:0/3105140557 conn(0x7fe79c06fb40 msgr2=0x7fe79c06ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:09.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.156+0000 7fe7a3706640 1 -- 172.21.15.55:0/3105140557 shutdown_connections 2024-03-19T02:01:09.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.156+0000 7fe7a3706640 1 -- 172.21.15.55:0/3105140557 wait complete. 2024-03-19T02:01:09.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.156+0000 7fe7a3706640 1 Processor -- start 2024-03-19T02:01:09.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.156+0000 7fe7a3706640 1 -- start start 2024-03-19T02:01:09.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.156+0000 7fe7a3706640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe79c13dc50 0x7fe79c0cc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:09.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.156+0000 7fe7a3706640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe79c13e030 0x7fe79c138080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:09.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.156+0000 7fe7a3706640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe79c15c090 0x7fe79c0c3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:09.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.156+0000 7fe7a3706640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe79c0c6c80 con 0x7fe79c15c090 2024-03-19T02:01:09.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.156+0000 7fe7a3706640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fe79c0c6b00 con 0x7fe79c13e030 2024-03-19T02:01:09.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.156+0000 7fe7a3706640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fe79c0c6e00 con 0x7fe79c13dc50 2024-03-19T02:01:09.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.156+0000 7fe7a1ca3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe79c15c090 0x7fe79c0c3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:09.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.156+0000 7fe7a0ca1640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe79c13e030 0x7fe79c138080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:09.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.156+0000 7fe7a1ca3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe79c15c090 0x7fe79c0c3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57306/0 (socket says 172.21.15.55:57306) 2024-03-19T02:01:09.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.156+0000 7fe7a1ca3640 1 -- 172.21.15.55:0/3696230374 learned_addr learned my addr 172.21.15.55:0/3696230374 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:09.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.156+0000 7fe7a14a2640 1 --2- 172.21.15.55:0/3696230374 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe79c13dc50 0x7fe79c0cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:09.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.160+0000 7fe7a1ca3640 1 -- 172.21.15.55:0/3696230374 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe79c13dc50 msgr2=0x7fe79c0cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:09.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.160+0000 7fe7a1ca3640 1 --2- 172.21.15.55:0/3696230374 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe79c13dc50 0x7fe79c0cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:09.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.160+0000 7fe7a1ca3640 1 -- 172.21.15.55:0/3696230374 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe79c13e030 msgr2=0x7fe79c138080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:09.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.160+0000 7fe7a1ca3640 1 --2- 172.21.15.55:0/3696230374 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe79c13e030 0x7fe79c138080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:09.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.160+0000 7fe7a1ca3640 1 -- 172.21.15.55:0/3696230374 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe79c1f4650 con 0x7fe79c15c090 2024-03-19T02:01:09.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.164+0000 7fe7a1ca3640 1 --2- 172.21.15.55:0/3696230374 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe79c15c090 0x7fe79c0c3bb0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe79400e7f0 tx=0x7fe79400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:09.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.168+0000 7fe799ffb640 1 -- 172.21.15.55:0/3696230374 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe794017070 con 0x7fe79c15c090 2024-03-19T02:01:09.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.168+0000 7fe7a3706640 1 -- 172.21.15.55:0/3696230374 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe79c0ca8f0 con 0x7fe79c15c090 2024-03-19T02:01:09.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.168+0000 7fe7a3706640 1 -- 172.21.15.55:0/3696230374 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe79c0cae50 con 0x7fe79c15c090 2024-03-19T02:01:09.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.168+0000 7fe799ffb640 1 -- 172.21.15.55:0/3696230374 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7940059a0 con 0x7fe79c15c090 2024-03-19T02:01:09.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.168+0000 7fe799ffb640 1 -- 172.21.15.55:0/3696230374 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe794021060 con 0x7fe79c15c090 2024-03-19T02:01:09.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.168+0000 7fe7a3706640 1 -- 172.21.15.55:0/3696230374 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe77c0052a0 con 0x7fe79c15c090 2024-03-19T02:01:09.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.192+0000 7fe799ffb640 1 -- 172.21.15.55:0/3696230374 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fe79401a070 con 0x7fe79c15c090 2024-03-19T02:01:09.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.192+0000 7fe799ffb640 1 --2- 172.21.15.55:0/3696230374 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe788067660 0x7fe788069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:09.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.192+0000 7fe7a14a2640 1 --2- 172.21.15.55:0/3696230374 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe788067660 0x7fe788069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:09.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.192+0000 7fe7a14a2640 1 --2- 172.21.15.55:0/3696230374 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe788067660 0x7fe788069b20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe78c0b1000 tx=0x7fe78c0aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:09.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.196+0000 7fe799ffb640 1 -- 172.21.15.55:0/3696230374 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fe794054140 con 0x7fe79c15c090 2024-03-19T02:01:09.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.468+0000 7fe799ffb640 1 -- 172.21.15.55:0/3696230374 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fe7940560a0 con 0x7fe79c15c090 2024-03-19T02:01:09.712 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.716+0000 7fe7a3706640 1 -- 172.21.15.55:0/3696230374 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe77c005590 con 0x7fe79c15c090 2024-03-19T02:01:09.738 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.740+0000 7fe799ffb640 1 -- 172.21.15.55:0/3696230374 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fe79405a140 con 0x7fe79c15c090 2024-03-19T02:01:09.738 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:09.738 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:09.739 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:09.739 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.740+0000 7fe7a3706640 1 -- 172.21.15.55:0/3696230374 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe788067660 msgr2=0x7fe788069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:09.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.740+0000 7fe7a3706640 1 --2- 172.21.15.55:0/3696230374 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe788067660 0x7fe788069b20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe78c0b1000 tx=0x7fe78c0aa000 comp rx=0 tx=0).stop 2024-03-19T02:01:09.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.740+0000 7fe7a3706640 1 -- 172.21.15.55:0/3696230374 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe79c15c090 msgr2=0x7fe79c0c3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:09.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.740+0000 7fe7a3706640 1 --2- 172.21.15.55:0/3696230374 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe79c15c090 0x7fe79c0c3bb0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe79400e7f0 tx=0x7fe79400ecb0 comp rx=0 tx=0).stop 2024-03-19T02:01:09.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.740+0000 7fe7a3706640 1 -- 172.21.15.55:0/3696230374 shutdown_connections 2024-03-19T02:01:09.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.740+0000 7fe7a3706640 1 --2- 172.21.15.55:0/3696230374 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe788067660 0x7fe788069b20 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:09.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.740+0000 7fe7a3706640 1 --2- 172.21.15.55:0/3696230374 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe79c15c090 0x7fe79c0c3bb0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:09.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.740+0000 7fe7a3706640 1 --2- 172.21.15.55:0/3696230374 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe79c13e030 0x7fe79c138080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:09.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.740+0000 7fe7a3706640 1 --2- 172.21.15.55:0/3696230374 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe79c13dc50 0x7fe79c0cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:09.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.740+0000 7fe7a3706640 1 -- 172.21.15.55:0/3696230374 >> 172.21.15.55:0/3696230374 conn(0x7fe79c06fb40 msgr2=0x7fe79c155e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:09.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.740+0000 7fe7a3706640 1 -- 172.21.15.55:0/3696230374 shutdown_connections 2024-03-19T02:01:09.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:09.740+0000 7fe7a3706640 1 -- 172.21.15.55:0/3696230374 wait complete. 2024-03-19T02:01:10.203 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:01:10.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:01:10.358 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:10.359 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:11.359 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:11.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.476+0000 7f8eca826640 1 Processor -- start 2024-03-19T02:01:11.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.476+0000 7f8eca826640 1 -- start start 2024-03-19T02:01:11.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.480+0000 7f8eca826640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f8ec415a920 0x7f8ec40c9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:11.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.480+0000 7f8eca826640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f8ec415ade0 0x7f8ec41431b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:11.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.480+0000 7f8eca826640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ec4070bd0 con 0x7f8ec415a540 2024-03-19T02:01:11.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.480+0000 7f8eca826640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f8ec406d440 con 0x7f8ec415ade0 2024-03-19T02:01:11.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.480+0000 7f8eca826640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f8ec406f5e0 con 0x7f8ec415a920 2024-03-19T02:01:11.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.480+0000 7f8ec8dc3640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f8ec415ade0 0x7f8ec41431b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:11.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.480+0000 7f8ec37fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f8ec415a920 0x7f8ec40c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:11.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.480+0000 7f8ec8dc3640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f8ec415ade0 0x7f8ec41431b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57322/0 (socket says 172.21.15.55:57322) 2024-03-19T02:01:11.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.480+0000 7f8ec8dc3640 1 -- 172.21.15.55:0/1085296478 learned_addr learned my addr 172.21.15.55:0/1085296478 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:11.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.484+0000 7f8ec8dc3640 1 -- 172.21.15.55:0/1085296478 >> v2:172.21.15.84:3300/0 conn(0x7f8ec415a920 msgr2=0x7f8ec40c9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:11.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.484+0000 7f8ec8dc3640 1 --2- 172.21.15.55:0/1085296478 >> v2:172.21.15.84:3300/0 conn(0x7f8ec415a920 0x7f8ec40c9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:11.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.484+0000 7f8ec8dc3640 1 -- 172.21.15.55:0/1085296478 >> v1:172.21.15.84:6789/0 conn(0x7f8ec415a540 legacy=0x7f8ec406cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:11.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.484+0000 7f8ec8dc3640 1 -- 172.21.15.55:0/1085296478 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ec4070e90 con 0x7f8ec415ade0 2024-03-19T02:01:11.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.484+0000 7f8ec8dc3640 1 --2- 172.21.15.55:0/1085296478 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ec415ade0 0x7f8ec41431b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8ec415b3a0 tx=0x7f8ebc01f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc500ebe06cd072 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:11.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.492+0000 7f8ec2ffd640 1 -- 172.21.15.55:0/1085296478 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc0b4070 con 0x7f8ec415ade0 2024-03-19T02:01:11.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.492+0000 7f8ec2ffd640 1 -- 172.21.15.55:0/1085296478 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ebc0af400 con 0x7f8ec415ade0 2024-03-19T02:01:11.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.492+0000 7f8ec2ffd640 1 -- 172.21.15.55:0/1085296478 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc0af7d0 con 0x7f8ec415ade0 2024-03-19T02:01:11.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.492+0000 7f8eca826640 1 -- 172.21.15.55:0/1085296478 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ec415ade0 msgr2=0x7f8ec41431b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:11.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.492+0000 7f8eca826640 1 --2- 172.21.15.55:0/1085296478 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ec415ade0 0x7f8ec41431b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8ec415b3a0 tx=0x7f8ebc01f1a0 comp rx=0 tx=0).stop 2024-03-19T02:01:11.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.492+0000 7f8eca826640 1 -- 172.21.15.55:0/1085296478 shutdown_connections 2024-03-19T02:01:11.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.492+0000 7f8eca826640 1 --2- 172.21.15.55:0/1085296478 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ec415ade0 0x7f8ec41431b0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:11.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.492+0000 7f8eca826640 1 --2- 172.21.15.55:0/1085296478 >> v2:172.21.15.84:3300/0 conn(0x7f8ec415a920 0x7f8ec40c9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:11.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.492+0000 7f8eca826640 1 -- 172.21.15.55:0/1085296478 >> 172.21.15.55:0/1085296478 conn(0x7f8ec4067600 msgr2=0x7f8ec4071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:11.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.492+0000 7f8eca826640 1 -- 172.21.15.55:0/1085296478 shutdown_connections 2024-03-19T02:01:11.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.492+0000 7f8eca826640 1 -- 172.21.15.55:0/1085296478 wait complete. 2024-03-19T02:01:11.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.492+0000 7f8eca826640 1 Processor -- start 2024-03-19T02:01:11.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.492+0000 7f8eca826640 1 -- start start 2024-03-19T02:01:11.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.492+0000 7f8eca826640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ec415a540 0x7f8ec41f3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:11.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.492+0000 7f8eca826640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8ec415a920 0x7f8ec41f0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:11.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.492+0000 7f8eca826640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ec415ade0 0x7f8ec41e7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:11.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.492+0000 7f8eca826640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec406fdf0 con 0x7f8ec415a540 2024-03-19T02:01:11.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.492+0000 7f8eca826640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec406fb10 con 0x7f8ec415ade0 2024-03-19T02:01:11.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.492+0000 7f8eca826640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f8ec4070860 con 0x7f8ec415a920 2024-03-19T02:01:11.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.496+0000 7f8ec3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ec415a540 0x7f8ec41f3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:11.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.496+0000 7f8ec8dc3640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ec415ade0 0x7f8ec41e7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:11.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.496+0000 7f8ec3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ec415a540 0x7f8ec41f3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57332/0 (socket says 172.21.15.55:57332) 2024-03-19T02:01:11.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.496+0000 7f8ec3fff640 1 -- 172.21.15.55:0/2915961279 learned_addr learned my addr 172.21.15.55:0/2915961279 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:11.492 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.496+0000 7f8ec37fe640 1 --2- 172.21.15.55:0/2915961279 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8ec415a920 0x7f8ec41f0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:11.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.500+0000 7f8ec3fff640 1 -- 172.21.15.55:0/2915961279 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8ec415a920 msgr2=0x7f8ec41f0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:11.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.500+0000 7f8ec3fff640 1 --2- 172.21.15.55:0/2915961279 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8ec415a920 0x7f8ec41f0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:11.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.500+0000 7f8ec3fff640 1 -- 172.21.15.55:0/2915961279 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ec415ade0 msgr2=0x7f8ec41e7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:11.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.500+0000 7f8ec3fff640 1 --2- 172.21.15.55:0/2915961279 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ec415ade0 0x7f8ec41e7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:11.496 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.500+0000 7f8ec3fff640 1 -- 172.21.15.55:0/2915961279 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ec413e340 con 0x7f8ec415a540 2024-03-19T02:01:11.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.500+0000 7f8ec3fff640 1 --2- 172.21.15.55:0/2915961279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ec415a540 0x7f8ec41f3b40 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f8eb400b440 tx=0x7f8eb400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:11.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.504+0000 7f8ec0ff9640 1 -- 172.21.15.55:0/2915961279 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb4015020 con 0x7f8ec415a540 2024-03-19T02:01:11.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.504+0000 7f8ec0ff9640 1 -- 172.21.15.55:0/2915961279 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8eb4011890 con 0x7f8ec415a540 2024-03-19T02:01:11.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.504+0000 7f8ec0ff9640 1 -- 172.21.15.55:0/2915961279 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb4011c80 con 0x7f8ec415a540 2024-03-19T02:01:11.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.504+0000 7f8eca826640 1 -- 172.21.15.55:0/2915961279 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ec413e620 con 0x7f8ec415a540 2024-03-19T02:01:11.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.504+0000 7f8eca826640 1 -- 172.21.15.55:0/2915961279 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ec41f3060 con 0x7f8ec415a540 2024-03-19T02:01:11.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.504+0000 7f8eca826640 1 -- 172.21.15.55:0/2915961279 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ec4068a40 con 0x7f8ec415a540 2024-03-19T02:01:11.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.532+0000 7f8ec0ff9640 1 -- 172.21.15.55:0/2915961279 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f8eb4018020 con 0x7f8ec415a540 2024-03-19T02:01:11.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.532+0000 7f8ec0ff9640 1 --2- 172.21.15.55:0/2915961279 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8eb00676b0 0x7f8eb0069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:11.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.532+0000 7f8ec37fe640 1 --2- 172.21.15.55:0/2915961279 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8eb00676b0 0x7f8eb0069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:11.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.532+0000 7f8ec37fe640 1 --2- 172.21.15.55:0/2915961279 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8eb00676b0 0x7f8eb0069b70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8ec415dd90 tx=0x7f8eb8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:11.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.532+0000 7f8ec0ff9640 1 -- 172.21.15.55:0/2915961279 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f8eb4054170 con 0x7f8ec415a540 2024-03-19T02:01:11.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:11.824+0000 7f8ec0ff9640 1 -- 172.21.15.55:0/2915961279 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f8eb401f030 con 0x7f8ec415a540 2024-03-19T02:01:12.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:12.036+0000 7f8eca826640 1 -- 172.21.15.55:0/2915961279 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ec40b2ee0 con 0x7f8ec415a540 2024-03-19T02:01:12.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:12.060+0000 7f8ec0ff9640 1 -- 172.21.15.55:0/2915961279 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f8eb400d040 con 0x7f8ec415a540 2024-03-19T02:01:12.059 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:12.059 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:12.060 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:12.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:12.064+0000 7f8eca826640 1 -- 172.21.15.55:0/2915961279 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8eb00676b0 msgr2=0x7f8eb0069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:12.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:12.064+0000 7f8eca826640 1 --2- 172.21.15.55:0/2915961279 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8eb00676b0 0x7f8eb0069b70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8ec415dd90 tx=0x7f8eb8009190 comp rx=0 tx=0).stop 2024-03-19T02:01:12.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:12.064+0000 7f8eca826640 1 -- 172.21.15.55:0/2915961279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ec415a540 msgr2=0x7f8ec41f3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:12.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:12.064+0000 7f8eca826640 1 --2- 172.21.15.55:0/2915961279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ec415a540 0x7f8ec41f3b40 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f8eb400b440 tx=0x7f8eb400b900 comp rx=0 tx=0).stop 2024-03-19T02:01:12.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:12.064+0000 7f8eca826640 1 -- 172.21.15.55:0/2915961279 shutdown_connections 2024-03-19T02:01:12.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:12.064+0000 7f8eca826640 1 --2- 172.21.15.55:0/2915961279 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8eb00676b0 0x7f8eb0069b70 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:12.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:12.064+0000 7f8eca826640 1 --2- 172.21.15.55:0/2915961279 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ec415ade0 0x7f8ec41e7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:12.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:12.064+0000 7f8eca826640 1 --2- 172.21.15.55:0/2915961279 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8ec415a920 0x7f8ec41f0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:12.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:12.064+0000 7f8eca826640 1 --2- 172.21.15.55:0/2915961279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8ec415a540 0x7f8ec41f3b40 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:12.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:12.064+0000 7f8eca826640 1 -- 172.21.15.55:0/2915961279 >> 172.21.15.55:0/2915961279 conn(0x7f8ec4067600 msgr2=0x7f8ec406cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:12.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:12.064+0000 7f8eca826640 1 -- 172.21.15.55:0/2915961279 shutdown_connections 2024-03-19T02:01:12.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:12.064+0000 7f8eca826640 1 -- 172.21.15.55:0/2915961279 wait complete. 2024-03-19T02:01:12.682 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:12.682 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:13.684 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:13.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.812+0000 7f3b433fb640 1 Processor -- start 2024-03-19T02:01:13.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.812+0000 7f3b433fb640 1 -- start start 2024-03-19T02:01:13.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.816+0000 7f3b433fb640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3b3c0c2e40 0x7f3b3c0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:13.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.816+0000 7f3b433fb640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f3b3c0c8260 0x7f3b3c156f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:13.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.816+0000 7f3b433fb640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b3c0c3330 con 0x7f3b3c1511f0 2024-03-19T02:01:13.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.816+0000 7f3b433fb640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f3b3c157490 con 0x7f3b3c0c8260 2024-03-19T02:01:13.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.816+0000 7f3b433fb640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3b3c0c5040 con 0x7f3b3c0c2e40 2024-03-19T02:01:13.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.816+0000 7f3b41197640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f3b3c1511f0 0x7f3b3c0c7450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:43498/0 (socket says 172.21.15.55:43498) 2024-03-19T02:01:13.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.816+0000 7f3b41197640 1 -- 172.21.15.55:0/3254602958 learned_addr learned my addr 172.21.15.55:0/3254602958 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:01:13.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.816+0000 7f3b40996640 1 --2- 172.21.15.55:0/3254602958 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3b3c0c2e40 0x7f3b3c0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:13.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.816+0000 7f3b41998640 1 --2- 172.21.15.55:0/3254602958 >> v2:172.21.15.84:3300/0 conn(0x7f3b3c0c8260 0x7f3b3c156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:13.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.820+0000 7f3b33fff640 1 -- 172.21.15.55:0/3254602958 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 260903364 0 0) 0x7f3b3c0c3330 con 0x7f3b3c1511f0 2024-03-19T02:01:13.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.820+0000 7f3b33fff640 1 -- 172.21.15.55:0/3254602958 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b28003530 con 0x7f3b3c1511f0 2024-03-19T02:01:13.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.820+0000 7f3b41998640 1 -- 172.21.15.55:0/3254602958 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3b3c0c2e40 msgr2=0x7f3b3c0c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:13.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.820+0000 7f3b41998640 1 --2- 172.21.15.55:0/3254602958 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3b3c0c2e40 0x7f3b3c0c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:13.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.820+0000 7f3b41998640 1 -- 172.21.15.55:0/3254602958 >> v1:172.21.15.55:6789/0 conn(0x7f3b3c1511f0 legacy=0x7f3b3c0c7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:13.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.820+0000 7f3b41998640 1 -- 172.21.15.55:0/3254602958 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b3c0c3610 con 0x7f3b3c0c8260 2024-03-19T02:01:13.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.824+0000 7f3b41998640 1 --2- 172.21.15.55:0/3254602958 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b3c0c8260 0x7f3b3c156f50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3b34009d90 tx=0x7f3b3401f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f219be79283765 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:13.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.832+0000 7f3b33fff640 1 -- 172.21.15.55:0/3254602958 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b340b4070 con 0x7f3b3c0c8260 2024-03-19T02:01:13.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.832+0000 7f3b33fff640 1 -- 172.21.15.55:0/3254602958 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b340af600 con 0x7f3b3c0c8260 2024-03-19T02:01:13.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.832+0000 7f3b33fff640 1 -- 172.21.15.55:0/3254602958 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b340af9d0 con 0x7f3b3c0c8260 2024-03-19T02:01:13.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.832+0000 7f3b433fb640 1 -- 172.21.15.55:0/3254602958 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b3c0c8260 msgr2=0x7f3b3c156f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:13.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.832+0000 7f3b433fb640 1 --2- 172.21.15.55:0/3254602958 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b3c0c8260 0x7f3b3c156f50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3b34009d90 tx=0x7f3b3401f150 comp rx=0 tx=0).stop 2024-03-19T02:01:13.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.832+0000 7f3b433fb640 1 -- 172.21.15.55:0/3254602958 shutdown_connections 2024-03-19T02:01:13.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.832+0000 7f3b433fb640 1 --2- 172.21.15.55:0/3254602958 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3b3c0c2e40 0x7f3b3c0c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:13.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.832+0000 7f3b433fb640 1 --2- 172.21.15.55:0/3254602958 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b3c0c8260 0x7f3b3c156f50 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:13.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.832+0000 7f3b433fb640 1 -- 172.21.15.55:0/3254602958 >> 172.21.15.55:0/3254602958 conn(0x7f3b3c06f3b0 msgr2=0x7f3b3c06f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:13.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.832+0000 7f3b433fb640 1 -- 172.21.15.55:0/3254602958 shutdown_connections 2024-03-19T02:01:13.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.832+0000 7f3b433fb640 1 -- 172.21.15.55:0/3254602958 wait complete. 2024-03-19T02:01:13.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.832+0000 7f3b433fb640 1 Processor -- start 2024-03-19T02:01:13.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.832+0000 7f3b433fb640 1 -- start start 2024-03-19T02:01:13.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.836+0000 7f3b433fb640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b3c0c2e40 0x7f3b3c156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:13.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.836+0000 7f3b433fb640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3b3c0c8260 0x7f3b3c14fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:13.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.836+0000 7f3b433fb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b3c1511f0 0x7f3b3c1459e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:13.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.836+0000 7f3b433fb640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c1515d0 con 0x7f3b3c1511f0 2024-03-19T02:01:13.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.836+0000 7f3b433fb640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c0c5290 con 0x7f3b3c0c2e40 2024-03-19T02:01:13.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.836+0000 7f3b433fb640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3b3c0c29c0 con 0x7f3b3c0c8260 2024-03-19T02:01:13.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.836+0000 7f3b41197640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b3c0c2e40 0x7f3b3c156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:13.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.836+0000 7f3b41998640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b3c1511f0 0x7f3b3c1459e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:13.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.836+0000 7f3b41998640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b3c1511f0 0x7f3b3c1459e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57338/0 (socket says 172.21.15.55:57338) 2024-03-19T02:01:13.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.836+0000 7f3b41998640 1 -- 172.21.15.55:0/2400937591 learned_addr learned my addr 172.21.15.55:0/2400937591 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:13.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.840+0000 7f3b41998640 1 -- 172.21.15.55:0/2400937591 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3b3c0c8260 msgr2=0x7f3b3c14fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:13.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.840+0000 7f3b41998640 1 --2- 172.21.15.55:0/2400937591 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3b3c0c8260 0x7f3b3c14fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:13.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.840+0000 7f3b41998640 1 -- 172.21.15.55:0/2400937591 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b3c0c2e40 msgr2=0x7f3b3c156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:13.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.840+0000 7f3b41998640 1 --2- 172.21.15.55:0/2400937591 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b3c0c2e40 0x7f3b3c156990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:13.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.840+0000 7f3b41998640 1 -- 172.21.15.55:0/2400937591 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b3c155ab0 con 0x7f3b3c1511f0 2024-03-19T02:01:13.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.840+0000 7f3b41998640 1 --2- 172.21.15.55:0/2400937591 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b3c1511f0 0x7f3b3c1459e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3b340b2040 tx=0x7f3b3401bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:13.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.844+0000 7f3b31ffb640 1 -- 172.21.15.55:0/2400937591 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b340b4040 con 0x7f3b3c1511f0 2024-03-19T02:01:13.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.844+0000 7f3b31ffb640 1 -- 172.21.15.55:0/2400937591 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b340c1070 con 0x7f3b3c1511f0 2024-03-19T02:01:13.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.844+0000 7f3b433fb640 1 -- 172.21.15.55:0/2400937591 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b3c155d50 con 0x7f3b3c1511f0 2024-03-19T02:01:13.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.844+0000 7f3b31ffb640 1 -- 172.21.15.55:0/2400937591 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b34009450 con 0x7f3b3c1511f0 2024-03-19T02:01:13.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.844+0000 7f3b433fb640 1 -- 172.21.15.55:0/2400937591 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b3c144f00 con 0x7f3b3c1511f0 2024-03-19T02:01:13.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:13.848+0000 7f3b433fb640 1 -- 172.21.15.55:0/2400937591 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b3c068330 con 0x7f3b3c1511f0 2024-03-19T02:01:14.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:14.036+0000 7f3b31ffb640 1 -- 172.21.15.55:0/2400937591 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f3b340ba6b0 con 0x7f3b3c1511f0 2024-03-19T02:01:14.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:14.036+0000 7f3b31ffb640 1 --2- 172.21.15.55:0/2400937591 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3b20067690 0x7f3b20069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:14.035 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:14.040+0000 7f3b41197640 1 --2- 172.21.15.55:0/2400937591 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3b20067690 0x7f3b20069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:14.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:14.040+0000 7f3b41197640 1 --2- 172.21.15.55:0/2400937591 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3b20067690 0x7f3b20069b50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3b38007fd0 tx=0x7f3b38009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:14.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:14.044+0000 7f3b31ffb640 1 -- 172.21.15.55:0/2400937591 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f3b340f31b0 con 0x7f3b3c1511f0 2024-03-19T02:01:14.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:14.164+0000 7f3b31ffb640 1 -- 172.21.15.55:0/2400937591 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f3b34018cb0 con 0x7f3b3c1511f0 2024-03-19T02:01:14.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:14.388+0000 7f3b433fb640 1 -- 172.21.15.55:0/2400937591 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b3c14f980 con 0x7f3b3c1511f0 2024-03-19T02:01:14.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:14.412+0000 7f3b31ffb640 1 -- 172.21.15.55:0/2400937591 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f3b340f5110 con 0x7f3b3c1511f0 2024-03-19T02:01:14.409 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:14.409 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:14.410 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:14.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:14.412+0000 7f3b433fb640 1 -- 172.21.15.55:0/2400937591 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3b20067690 msgr2=0x7f3b20069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:14.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:14.412+0000 7f3b433fb640 1 --2- 172.21.15.55:0/2400937591 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3b20067690 0x7f3b20069b50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3b38007fd0 tx=0x7f3b38009040 comp rx=0 tx=0).stop 2024-03-19T02:01:14.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:14.412+0000 7f3b433fb640 1 -- 172.21.15.55:0/2400937591 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b3c1511f0 msgr2=0x7f3b3c1459e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:14.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:14.412+0000 7f3b433fb640 1 --2- 172.21.15.55:0/2400937591 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b3c1511f0 0x7f3b3c1459e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3b340b2040 tx=0x7f3b3401bef0 comp rx=0 tx=0).stop 2024-03-19T02:01:14.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:14.412+0000 7f3b433fb640 1 -- 172.21.15.55:0/2400937591 shutdown_connections 2024-03-19T02:01:14.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:14.412+0000 7f3b433fb640 1 --2- 172.21.15.55:0/2400937591 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3b20067690 0x7f3b20069b50 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:14.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:14.412+0000 7f3b433fb640 1 --2- 172.21.15.55:0/2400937591 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3b3c1511f0 0x7f3b3c1459e0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:14.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:14.412+0000 7f3b433fb640 1 --2- 172.21.15.55:0/2400937591 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3b3c0c8260 0x7f3b3c14fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:14.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:14.412+0000 7f3b433fb640 1 --2- 172.21.15.55:0/2400937591 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b3c0c2e40 0x7f3b3c156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:14.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:14.412+0000 7f3b433fb640 1 -- 172.21.15.55:0/2400937591 >> 172.21.15.55:0/2400937591 conn(0x7f3b3c06f3b0 msgr2=0x7f3b3c0c9820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:14.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:14.412+0000 7f3b433fb640 1 -- 172.21.15.55:0/2400937591 shutdown_connections 2024-03-19T02:01:14.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:14.412+0000 7f3b433fb640 1 -- 172.21.15.55:0/2400937591 wait complete. 2024-03-19T02:01:15.032 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:15.032 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:16.033 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:16.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.151+0000 7eff0ca85640 1 Processor -- start 2024-03-19T02:01:16.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.151+0000 7eff0ca85640 1 -- start start 2024-03-19T02:01:16.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.151+0000 7eff0ca85640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7eff08154340 0x7eff0815e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:16.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.151+0000 7eff0ca85640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7eff08154720 0x7eff080c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:16.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.151+0000 7eff0ca85640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff08071280 con 0x7eff0815aca0 2024-03-19T02:01:16.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.151+0000 7eff0ca85640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7eff08073840 con 0x7eff08154720 2024-03-19T02:01:16.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.151+0000 7eff0ca85640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7eff0815df10 con 0x7eff08154340 2024-03-19T02:01:16.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.151+0000 7eff05d9b640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7eff08154720 0x7eff080c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:16.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.151+0000 7eff06d9d640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7eff0815aca0 0x7eff08072e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:43506/0 (socket says 172.21.15.55:43506) 2024-03-19T02:01:16.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.151+0000 7eff06d9d640 1 -- 172.21.15.55:0/1704302508 learned_addr learned my addr 172.21.15.55:0/1704302508 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:01:16.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.151+0000 7eff0659c640 1 --2- 172.21.15.55:0/1704302508 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7eff08154340 0x7eff0815e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:16.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.155+0000 7eff0559a640 1 -- 172.21.15.55:0/1704302508 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 852222303 0 0) 0x7eff08071280 con 0x7eff0815aca0 2024-03-19T02:01:16.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.155+0000 7eff0559a640 1 -- 172.21.15.55:0/1704302508 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efef0003530 con 0x7eff0815aca0 2024-03-19T02:01:16.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.155+0000 7eff05d9b640 1 -- 172.21.15.55:0/1704302508 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7eff08154340 msgr2=0x7eff0815e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:16.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.155+0000 7eff05d9b640 1 --2- 172.21.15.55:0/1704302508 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7eff08154340 0x7eff0815e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:16.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.155+0000 7eff05d9b640 1 -- 172.21.15.55:0/1704302508 >> v1:172.21.15.55:6789/0 conn(0x7eff0815aca0 legacy=0x7eff08072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:16.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.155+0000 7eff05d9b640 1 -- 172.21.15.55:0/1704302508 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff0815d900 con 0x7eff08154720 2024-03-19T02:01:16.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.159+0000 7eff05d9b640 1 --2- 172.21.15.55:0/1704302508 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff08154720 0x7eff080c73c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efefc0097c0 tx=0x7efefc01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf16944cca6d1675 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:16.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.167+0000 7eff0559a640 1 -- 172.21.15.55:0/1704302508 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efefc0b2070 con 0x7eff08154720 2024-03-19T02:01:16.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.167+0000 7eff0559a640 1 -- 172.21.15.55:0/1704302508 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efefc0ad640 con 0x7eff08154720 2024-03-19T02:01:16.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.167+0000 7eff0559a640 1 -- 172.21.15.55:0/1704302508 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efefc0ada10 con 0x7eff08154720 2024-03-19T02:01:16.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.167+0000 7eff0ca85640 1 -- 172.21.15.55:0/1704302508 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff08154720 msgr2=0x7eff080c73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:16.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.167+0000 7eff0ca85640 1 --2- 172.21.15.55:0/1704302508 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff08154720 0x7eff080c73c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efefc0097c0 tx=0x7efefc01c4a0 comp rx=0 tx=0).stop 2024-03-19T02:01:16.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.167+0000 7eff0ca85640 1 -- 172.21.15.55:0/1704302508 shutdown_connections 2024-03-19T02:01:16.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.167+0000 7eff0ca85640 1 --2- 172.21.15.55:0/1704302508 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff08154720 0x7eff080c73c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:16.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.167+0000 7eff0ca85640 1 --2- 172.21.15.55:0/1704302508 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7eff08154340 0x7eff0815e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:16.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.167+0000 7eff0ca85640 1 -- 172.21.15.55:0/1704302508 >> 172.21.15.55:0/1704302508 conn(0x7eff080cc9c0 msgr2=0x7eff080ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:16.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.167+0000 7eff0ca85640 1 -- 172.21.15.55:0/1704302508 shutdown_connections 2024-03-19T02:01:16.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.167+0000 7eff0ca85640 1 -- 172.21.15.55:0/1704302508 wait complete. 2024-03-19T02:01:16.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.167+0000 7eff0ca85640 1 Processor -- start 2024-03-19T02:01:16.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.167+0000 7eff0ca85640 1 -- start start 2024-03-19T02:01:16.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.171+0000 7eff0ca85640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff08154340 0x7eff0813b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:16.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.171+0000 7eff0ca85640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff08154720 0x7eff0815cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:16.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.171+0000 7eff0ca85640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7eff0815aca0 0x7eff0815d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:16.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.171+0000 7eff0ca85640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7eff0815b200 con 0x7eff08154720 2024-03-19T02:01:16.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.171+0000 7eff0ca85640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7eff0815b080 con 0x7eff08154340 2024-03-19T02:01:16.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.171+0000 7eff0ca85640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7eff08166b70 con 0x7eff0815aca0 2024-03-19T02:01:16.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.171+0000 7eff05d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff08154720 0x7eff0815cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:16.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.171+0000 7eff05d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff08154720 0x7eff0815cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57340/0 (socket says 172.21.15.55:57340) 2024-03-19T02:01:16.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.171+0000 7eff05d9b640 1 -- 172.21.15.55:0/780577355 learned_addr learned my addr 172.21.15.55:0/780577355 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:16.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.171+0000 7eff0659c640 1 --2- 172.21.15.55:0/780577355 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff08154340 0x7eff0813b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:16.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.171+0000 7eff06d9d640 1 --2- 172.21.15.55:0/780577355 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7eff0815aca0 0x7eff0815d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:16.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.175+0000 7eff05d9b640 1 -- 172.21.15.55:0/780577355 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7eff0815aca0 msgr2=0x7eff0815d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:16.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.175+0000 7eff05d9b640 1 --2- 172.21.15.55:0/780577355 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7eff0815aca0 0x7eff0815d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:16.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.175+0000 7eff05d9b640 1 -- 172.21.15.55:0/780577355 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff08154340 msgr2=0x7eff0813b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:16.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.175+0000 7eff05d9b640 1 --2- 172.21.15.55:0/780577355 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff08154340 0x7eff0813b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:16.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.175+0000 7eff05d9b640 1 -- 172.21.15.55:0/780577355 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff08154d30 con 0x7eff08154720 2024-03-19T02:01:16.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.175+0000 7eff05d9b640 1 --2- 172.21.15.55:0/780577355 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff08154720 0x7eff0815cc40 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7efefc0097c0 tx=0x7efefc019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:16.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.183+0000 7efef6ffd640 1 -- 172.21.15.55:0/780577355 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efefc0b2040 con 0x7eff08154720 2024-03-19T02:01:16.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.183+0000 7efef6ffd640 1 -- 172.21.15.55:0/780577355 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efefc0bb070 con 0x7eff08154720 2024-03-19T02:01:16.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.183+0000 7eff0ca85640 1 -- 172.21.15.55:0/780577355 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff08154fd0 con 0x7eff08154720 2024-03-19T02:01:16.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.183+0000 7efef6ffd640 1 -- 172.21.15.55:0/780577355 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efefc0b6630 con 0x7eff08154720 2024-03-19T02:01:16.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.183+0000 7eff0ca85640 1 -- 172.21.15.55:0/780577355 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff08155590 con 0x7eff08154720 2024-03-19T02:01:16.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.183+0000 7eff0ca85640 1 -- 172.21.15.55:0/780577355 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff08068a40 con 0x7eff08154720 2024-03-19T02:01:16.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.207+0000 7efef6ffd640 1 -- 172.21.15.55:0/780577355 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7efefc0b67d0 con 0x7eff08154720 2024-03-19T02:01:16.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.207+0000 7efef6ffd640 1 --2- 172.21.15.55:0/780577355 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7efee80676e0 0x7efee8069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:16.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.207+0000 7eff0659c640 1 --2- 172.21.15.55:0/780577355 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7efee80676e0 0x7efee8069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:16.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.207+0000 7eff0659c640 1 --2- 172.21.15.55:0/780577355 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7efee80676e0 0x7efee8069ba0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7efef8009b60 tx=0x7efef8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:16.209 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.211+0000 7efef6ffd640 1 -- 172.21.15.55:0/780577355 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7efefc0eea40 con 0x7eff08154720 2024-03-19T02:01:16.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.503+0000 7efef6ffd640 1 -- 172.21.15.55:0/780577355 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7efefc0c5020 con 0x7eff08154720 2024-03-19T02:01:16.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.715+0000 7eff0ca85640 1 -- 172.21.15.55:0/780577355 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff08151f70 con 0x7eff08154720 2024-03-19T02:01:16.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.787+0000 7efef6ffd640 1 -- 172.21.15.55:0/780577355 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7efefc016090 con 0x7eff08154720 2024-03-19T02:01:16.786 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:16.786 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:16.786 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:16.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.791+0000 7eff0ca85640 1 -- 172.21.15.55:0/780577355 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7efee80676e0 msgr2=0x7efee8069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:16.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.791+0000 7eff0ca85640 1 --2- 172.21.15.55:0/780577355 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7efee80676e0 0x7efee8069ba0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7efef8009b60 tx=0x7efef8009190 comp rx=0 tx=0).stop 2024-03-19T02:01:16.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.791+0000 7eff0ca85640 1 -- 172.21.15.55:0/780577355 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff08154720 msgr2=0x7eff0815cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:16.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.791+0000 7eff0ca85640 1 --2- 172.21.15.55:0/780577355 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff08154720 0x7eff0815cc40 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7efefc0097c0 tx=0x7efefc019240 comp rx=0 tx=0).stop 2024-03-19T02:01:16.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.791+0000 7eff0ca85640 1 -- 172.21.15.55:0/780577355 shutdown_connections 2024-03-19T02:01:16.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.791+0000 7eff0ca85640 1 --2- 172.21.15.55:0/780577355 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7efee80676e0 0x7efee8069ba0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:16.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.791+0000 7eff0ca85640 1 --2- 172.21.15.55:0/780577355 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7eff0815aca0 0x7eff0815d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:16.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.791+0000 7eff0ca85640 1 --2- 172.21.15.55:0/780577355 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7eff08154720 0x7eff0815cc40 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:16.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.791+0000 7eff0ca85640 1 --2- 172.21.15.55:0/780577355 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff08154340 0x7eff0813b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:16.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.791+0000 7eff0ca85640 1 -- 172.21.15.55:0/780577355 >> 172.21.15.55:0/780577355 conn(0x7eff080cc9c0 msgr2=0x7eff0814cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:16.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.791+0000 7eff0ca85640 1 -- 172.21.15.55:0/780577355 shutdown_connections 2024-03-19T02:01:16.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:16.791+0000 7eff0ca85640 1 -- 172.21.15.55:0/780577355 wait complete. 2024-03-19T02:01:17.408 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:17.409 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:18.410 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:18.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.527+0000 7fb243604640 1 Processor -- start 2024-03-19T02:01:18.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.527+0000 7fb243604640 1 -- start start 2024-03-19T02:01:18.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.531+0000 7fb243604640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb23c0c7100 0x7fb23c150610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:18.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.531+0000 7fb243604640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb23c0c5940 0x7fb23c150b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:18.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.531+0000 7fb243604640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb23c0c3810 con 0x7fb23c0c5560 2024-03-19T02:01:18.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.531+0000 7fb243604640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fb23c0c8bf0 con 0x7fb23c0c5940 2024-03-19T02:01:18.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.531+0000 7fb243604640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fb23c0c6c70 con 0x7fb23c0c7100 2024-03-19T02:01:18.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.531+0000 7fb240b9f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb23c0c7100 0x7fb23c150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:18.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.531+0000 7fb240b9f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb23c0c7100 0x7fb23c150610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:55182/0 (socket says 172.21.15.55:55182) 2024-03-19T02:01:18.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.531+0000 7fb240b9f640 1 -- 172.21.15.55:0/1342108045 learned_addr learned my addr 172.21.15.55:0/1342108045 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:18.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.539+0000 7fb240b9f640 1 -- 172.21.15.55:0/1342108045 >> v2:172.21.15.55:3300/0 conn(0x7fb23c0c5940 msgr2=0x7fb23c150b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:18.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.539+0000 7fb240b9f640 1 --2- 172.21.15.55:0/1342108045 >> v2:172.21.15.55:3300/0 conn(0x7fb23c0c5940 0x7fb23c150b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:18.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.539+0000 7fb240b9f640 1 -- 172.21.15.55:0/1342108045 >> v1:172.21.15.55:6789/0 conn(0x7fb23c0c5560 legacy=0x7fb23c156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:18.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.539+0000 7fb240b9f640 1 -- 172.21.15.55:0/1342108045 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb23c0c3ad0 con 0x7fb23c0c7100 2024-03-19T02:01:18.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.543+0000 7fb240b9f640 1 --2- 172.21.15.55:0/1342108045 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb23c0c7100 0x7fb23c150610 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb238009870 tx=0x7fb23801ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d00b91200ea7713 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:18.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.551+0000 7fb233fff640 1 -- 172.21.15.55:0/1342108045 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2380b5070 con 0x7fb23c0c7100 2024-03-19T02:01:18.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.551+0000 7fb233fff640 1 -- 172.21.15.55:0/1342108045 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2380a89d0 con 0x7fb23c0c7100 2024-03-19T02:01:18.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.551+0000 7fb233fff640 1 -- 172.21.15.55:0/1342108045 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2380a8da0 con 0x7fb23c0c7100 2024-03-19T02:01:18.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.551+0000 7fb243604640 1 -- 172.21.15.55:0/1342108045 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb23c0c7100 msgr2=0x7fb23c150610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:18.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.551+0000 7fb243604640 1 --2- 172.21.15.55:0/1342108045 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb23c0c7100 0x7fb23c150610 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb238009870 tx=0x7fb23801ed10 comp rx=0 tx=0).stop 2024-03-19T02:01:18.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.551+0000 7fb243604640 1 -- 172.21.15.55:0/1342108045 shutdown_connections 2024-03-19T02:01:18.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.551+0000 7fb243604640 1 --2- 172.21.15.55:0/1342108045 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb23c0c7100 0x7fb23c150610 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:18.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.551+0000 7fb243604640 1 --2- 172.21.15.55:0/1342108045 >> v2:172.21.15.55:3300/0 conn(0x7fb23c0c5940 0x7fb23c150b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:18.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.551+0000 7fb243604640 1 -- 172.21.15.55:0/1342108045 >> 172.21.15.55:0/1342108045 conn(0x7fb23c0cc270 msgr2=0x7fb23c0cc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:18.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.551+0000 7fb243604640 1 -- 172.21.15.55:0/1342108045 shutdown_connections 2024-03-19T02:01:18.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.551+0000 7fb243604640 1 -- 172.21.15.55:0/1342108045 wait complete. 2024-03-19T02:01:18.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.551+0000 7fb243604640 1 Processor -- start 2024-03-19T02:01:18.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.551+0000 7fb243604640 1 -- start start 2024-03-19T02:01:18.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.555+0000 7fb243604640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb23c0c5560 0x7fb23c135e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:18.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.555+0000 7fb243604640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23c0c5940 0x7fb23c1524c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:18.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.555+0000 7fb243604640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb23c152a00 0x7fb23c0cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:18.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.555+0000 7fb243604640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c0c6470 con 0x7fb23c0c5940 2024-03-19T02:01:18.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.555+0000 7fb243604640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c06c9c0 con 0x7fb23c0c5560 2024-03-19T02:01:18.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.555+0000 7fb243604640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fb23c0c65f0 con 0x7fb23c152a00 2024-03-19T02:01:18.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.555+0000 7fb240b9f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23c0c5940 0x7fb23c1524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:18.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.555+0000 7fb240b9f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23c0c5940 0x7fb23c1524c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57358/0 (socket says 172.21.15.55:57358) 2024-03-19T02:01:18.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.555+0000 7fb240b9f640 1 -- 172.21.15.55:0/492072741 learned_addr learned my addr 172.21.15.55:0/492072741 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:18.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.555+0000 7fb2413a0640 1 --2- 172.21.15.55:0/492072741 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb23c0c5560 0x7fb23c135e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:18.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.555+0000 7fb241ba1640 1 --2- 172.21.15.55:0/492072741 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb23c152a00 0x7fb23c0cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:18.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.559+0000 7fb240b9f640 1 -- 172.21.15.55:0/492072741 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb23c152a00 msgr2=0x7fb23c0cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:18.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.559+0000 7fb240b9f640 1 --2- 172.21.15.55:0/492072741 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb23c152a00 0x7fb23c0cb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:18.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.559+0000 7fb240b9f640 1 -- 172.21.15.55:0/492072741 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb23c0c5560 msgr2=0x7fb23c135e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:18.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.559+0000 7fb240b9f640 1 --2- 172.21.15.55:0/492072741 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb23c0c5560 0x7fb23c135e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:18.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.559+0000 7fb240b9f640 1 -- 172.21.15.55:0/492072741 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb23c151a50 con 0x7fb23c0c5940 2024-03-19T02:01:18.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.563+0000 7fb240b9f640 1 --2- 172.21.15.55:0/492072741 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23c0c5940 0x7fb23c1524c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb23801d3c0 tx=0x7fb23801bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:18.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.567+0000 7fb231ffb640 1 -- 172.21.15.55:0/492072741 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2380b5040 con 0x7fb23c0c5940 2024-03-19T02:01:18.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.567+0000 7fb231ffb640 1 -- 172.21.15.55:0/492072741 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2380c2070 con 0x7fb23c0c5940 2024-03-19T02:01:18.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.567+0000 7fb231ffb640 1 -- 172.21.15.55:0/492072741 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2380b0ae0 con 0x7fb23c0c5940 2024-03-19T02:01:18.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.567+0000 7fb243604640 1 -- 172.21.15.55:0/492072741 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb23c145420 con 0x7fb23c0c5940 2024-03-19T02:01:18.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.567+0000 7fb243604640 1 -- 172.21.15.55:0/492072741 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb23c151490 con 0x7fb23c0c5940 2024-03-19T02:01:18.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.567+0000 7fb243604640 1 -- 172.21.15.55:0/492072741 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb23c068a40 con 0x7fb23c0c5940 2024-03-19T02:01:18.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.835+0000 7fb231ffb640 1 -- 172.21.15.55:0/492072741 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb2380b0c80 con 0x7fb23c0c5940 2024-03-19T02:01:18.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.835+0000 7fb231ffb640 1 --2- 172.21.15.55:0/492072741 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb2200676e0 0x7fb220069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:18.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.835+0000 7fb2413a0640 1 --2- 172.21.15.55:0/492072741 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb2200676e0 0x7fb220069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:18.833 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.835+0000 7fb2413a0640 1 --2- 172.21.15.55:0/492072741 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb2200676e0 0x7fb220069ba0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb22c007fd0 tx=0x7fb22c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:18.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.851+0000 7fb231ffb640 1 -- 172.21.15.55:0/492072741 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fb2380f26d0 con 0x7fb23c0c5940 2024-03-19T02:01:18.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:18.851+0000 7fb231ffb640 1 -- 172.21.15.55:0/492072741 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fb238018870 con 0x7fb23c0c5940 2024-03-19T02:01:19.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:19.063+0000 7fb243604640 1 -- 172.21.15.55:0/492072741 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb23c14f500 con 0x7fb23c0c5940 2024-03-19T02:01:19.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:19.095+0000 7fb231ffb640 1 -- 172.21.15.55:0/492072741 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fb2380f4630 con 0x7fb23c0c5940 2024-03-19T02:01:19.093 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:19.093 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:19.094 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:19.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:19.099+0000 7fb243604640 1 -- 172.21.15.55:0/492072741 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb2200676e0 msgr2=0x7fb220069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:19.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:19.099+0000 7fb243604640 1 --2- 172.21.15.55:0/492072741 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb2200676e0 0x7fb220069ba0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb22c007fd0 tx=0x7fb22c009040 comp rx=0 tx=0).stop 2024-03-19T02:01:19.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:19.099+0000 7fb243604640 1 -- 172.21.15.55:0/492072741 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23c0c5940 msgr2=0x7fb23c1524c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:19.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:19.099+0000 7fb243604640 1 --2- 172.21.15.55:0/492072741 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23c0c5940 0x7fb23c1524c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb23801d3c0 tx=0x7fb23801bab0 comp rx=0 tx=0).stop 2024-03-19T02:01:19.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:19.099+0000 7fb243604640 1 -- 172.21.15.55:0/492072741 shutdown_connections 2024-03-19T02:01:19.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:19.099+0000 7fb243604640 1 --2- 172.21.15.55:0/492072741 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb2200676e0 0x7fb220069ba0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:19.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:19.099+0000 7fb243604640 1 --2- 172.21.15.55:0/492072741 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb23c152a00 0x7fb23c0cb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:19.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:19.099+0000 7fb243604640 1 --2- 172.21.15.55:0/492072741 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23c0c5940 0x7fb23c1524c0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:19.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:19.099+0000 7fb243604640 1 --2- 172.21.15.55:0/492072741 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb23c0c5560 0x7fb23c135e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:19.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:19.099+0000 7fb243604640 1 -- 172.21.15.55:0/492072741 >> 172.21.15.55:0/492072741 conn(0x7fb23c0cc270 msgr2=0x7fb23c06ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:19.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:19.099+0000 7fb243604640 1 -- 172.21.15.55:0/492072741 shutdown_connections 2024-03-19T02:01:19.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:19.099+0000 7fb243604640 1 -- 172.21.15.55:0/492072741 wait complete. 2024-03-19T02:01:19.717 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:19.717 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:20.718 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:20.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.839+0000 7fec5dd3f640 1 Processor -- start 2024-03-19T02:01:20.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.839+0000 7fec5dd3f640 1 -- start start 2024-03-19T02:01:20.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.839+0000 7fec5dd3f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fec580c9780 0x7fec580c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:20.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.839+0000 7fec5dd3f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fec5815b550 0x7fec580c5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:20.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.839+0000 7fec5dd3f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec580c8f70 con 0x7fec5813dd00 2024-03-19T02:01:20.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.839+0000 7fec5dd3f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fec580c9250 con 0x7fec580c9780 2024-03-19T02:01:20.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.839+0000 7fec5dd3f640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fec581582b0 con 0x7fec5815b550 2024-03-19T02:01:20.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.839+0000 7fec56ffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fec580c9780 0x7fec580c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:20.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.839+0000 7fec56ffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fec580c9780 0x7fec580c7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39346/0 (socket says 172.21.15.55:39346) 2024-03-19T02:01:20.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.839+0000 7fec56ffd640 1 -- 172.21.15.55:0/3905496923 learned_addr learned my addr 172.21.15.55:0/3905496923 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:20.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.839+0000 7fec57fff640 1 --2- 172.21.15.55:0/3905496923 >> v2:172.21.15.84:3300/0 conn(0x7fec5815b550 0x7fec580c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:20.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.843+0000 7fec56ffd640 1 -- 172.21.15.55:0/3905496923 >> v2:172.21.15.84:3300/0 conn(0x7fec5815b550 msgr2=0x7fec580c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:20.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.843+0000 7fec56ffd640 1 --2- 172.21.15.55:0/3905496923 >> v2:172.21.15.84:3300/0 conn(0x7fec5815b550 0x7fec580c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:20.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.843+0000 7fec56ffd640 1 -- 172.21.15.55:0/3905496923 >> v1:172.21.15.84:6789/0 conn(0x7fec5813dd00 legacy=0x7fec580c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:20.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.843+0000 7fec56ffd640 1 -- 172.21.15.55:0/3905496923 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec58158590 con 0x7fec580c9780 2024-03-19T02:01:20.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.847+0000 7fec56ffd640 1 --2- 172.21.15.55:0/3905496923 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fec580c9780 0x7fec580c7430 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fec58155b30 tx=0x7fec4c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=925d316b7f3cad40 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:20.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.851+0000 7fec567fc640 1 -- 172.21.15.55:0/3905496923 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4c0ab430 con 0x7fec580c9780 2024-03-19T02:01:20.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.851+0000 7fec567fc640 1 -- 172.21.15.55:0/3905496923 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec4c0b0070 con 0x7fec580c9780 2024-03-19T02:01:20.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.851+0000 7fec567fc640 1 -- 172.21.15.55:0/3905496923 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4c0ab7f0 con 0x7fec580c9780 2024-03-19T02:01:20.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.851+0000 7fec5dd3f640 1 -- 172.21.15.55:0/3905496923 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fec580c9780 msgr2=0x7fec580c7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:20.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.851+0000 7fec5dd3f640 1 --2- 172.21.15.55:0/3905496923 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fec580c9780 0x7fec580c7430 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fec58155b30 tx=0x7fec4c01c520 comp rx=0 tx=0).stop 2024-03-19T02:01:20.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.851+0000 7fec5dd3f640 1 -- 172.21.15.55:0/3905496923 shutdown_connections 2024-03-19T02:01:20.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.851+0000 7fec5dd3f640 1 --2- 172.21.15.55:0/3905496923 >> v2:172.21.15.84:3300/0 conn(0x7fec5815b550 0x7fec580c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:20.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.851+0000 7fec5dd3f640 1 --2- 172.21.15.55:0/3905496923 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fec580c9780 0x7fec580c7430 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:20.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.851+0000 7fec5dd3f640 1 -- 172.21.15.55:0/3905496923 >> 172.21.15.55:0/3905496923 conn(0x7fec5806e5a0 msgr2=0x7fec58072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:20.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.851+0000 7fec5dd3f640 1 -- 172.21.15.55:0/3905496923 shutdown_connections 2024-03-19T02:01:20.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.851+0000 7fec5dd3f640 1 -- 172.21.15.55:0/3905496923 wait complete. 2024-03-19T02:01:20.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.851+0000 7fec5dd3f640 1 Processor -- start 2024-03-19T02:01:20.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.851+0000 7fec5dd3f640 1 -- start start 2024-03-19T02:01:20.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.855+0000 7fec5dd3f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fec580c9780 0x7fec5813a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:20.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.855+0000 7fec5dd3f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec5813dd00 0x7fec58137890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:20.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.855+0000 7fec5dd3f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fec5815b550 0x7fec581f4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:20.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.855+0000 7fec5dd3f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fec58165280 con 0x7fec580c9780 2024-03-19T02:01:20.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.855+0000 7fec5dd3f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fec58165100 con 0x7fec5813dd00 2024-03-19T02:01:20.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.855+0000 7fec5dd3f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fec58165400 con 0x7fec5815b550 2024-03-19T02:01:20.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.855+0000 7fec56ffd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec5813dd00 0x7fec58137890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:20.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.855+0000 7fec57fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fec5815b550 0x7fec581f4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:20.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.855+0000 7fec577fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fec580c9780 0x7fec5813a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:20.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.855+0000 7fec577fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fec580c9780 0x7fec5813a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39352/0 (socket says 172.21.15.55:39352) 2024-03-19T02:01:20.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.855+0000 7fec56ffd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec5813dd00 0x7fec58137890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:60474/0 (socket says 172.21.15.55:60474) 2024-03-19T02:01:20.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.855+0000 7fec577fe640 1 -- 172.21.15.55:0/3844604118 learned_addr learned my addr 172.21.15.55:0/3844604118 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:20.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.859+0000 7fec577fe640 1 -- 172.21.15.55:0/3844604118 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fec5815b550 msgr2=0x7fec581f4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:20.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.859+0000 7fec577fe640 1 --2- 172.21.15.55:0/3844604118 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fec5815b550 0x7fec581f4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:20.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.859+0000 7fec577fe640 1 -- 172.21.15.55:0/3844604118 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec5813dd00 msgr2=0x7fec58137890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:20.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.859+0000 7fec577fe640 1 --2- 172.21.15.55:0/3844604118 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec5813dd00 0x7fec58137890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:20.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.859+0000 7fec577fe640 1 -- 172.21.15.55:0/3844604118 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec581f2690 con 0x7fec580c9780 2024-03-19T02:01:20.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.859+0000 7fec577fe640 1 --2- 172.21.15.55:0/3844604118 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fec580c9780 0x7fec5813a6b0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fec4800c370 tx=0x7fec4800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:20.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.867+0000 7fec3ffff640 1 -- 172.21.15.55:0/3844604118 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec480052d0 con 0x7fec580c9780 2024-03-19T02:01:20.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.867+0000 7fec3ffff640 1 -- 172.21.15.55:0/3844604118 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec48005470 con 0x7fec580c9780 2024-03-19T02:01:20.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.867+0000 7fec5dd3f640 1 -- 172.21.15.55:0/3844604118 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec5815b330 con 0x7fec580c9780 2024-03-19T02:01:20.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.867+0000 7fec3ffff640 1 -- 172.21.15.55:0/3844604118 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec48007260 con 0x7fec580c9780 2024-03-19T02:01:20.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.867+0000 7fec5dd3f640 1 -- 172.21.15.55:0/3844604118 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec58138ce0 con 0x7fec580c9780 2024-03-19T02:01:20.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.867+0000 7fec5dd3f640 1 -- 172.21.15.55:0/3844604118 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec58138eb0 con 0x7fec580c9780 2024-03-19T02:01:20.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.891+0000 7fec3ffff640 1 -- 172.21.15.55:0/3844604118 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fec48007400 con 0x7fec580c9780 2024-03-19T02:01:20.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.891+0000 7fec3ffff640 1 --2- 172.21.15.55:0/3844604118 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fec44067660 0x7fec44069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:20.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.891+0000 7fec56ffd640 1 --2- 172.21.15.55:0/3844604118 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fec44067660 0x7fec44069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:20.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:20.891+0000 7fec56ffd640 1 --2- 172.21.15.55:0/3844604118 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fec44067660 0x7fec44069b20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fec4c0af000 tx=0x7fec4c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:21.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:21.199+0000 7fec3ffff640 1 -- 172.21.15.55:0/3844604118 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fec48053840 con 0x7fec580c9780 2024-03-19T02:01:21.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:21.199+0000 7fec3ffff640 1 -- 172.21.15.55:0/3844604118 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fec480557a0 con 0x7fec580c9780 2024-03-19T02:01:21.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:21.411+0000 7fec5dd3f640 1 -- 172.21.15.55:0/3844604118 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec58068a40 con 0x7fec580c9780 2024-03-19T02:01:21.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:21.427+0000 7fec3ffff640 1 -- 172.21.15.55:0/3844604118 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fec48059840 con 0x7fec580c9780 2024-03-19T02:01:21.427 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:21.427 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:21.427 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:21.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:21.431+0000 7fec5dd3f640 1 -- 172.21.15.55:0/3844604118 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fec44067660 msgr2=0x7fec44069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:21.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:21.431+0000 7fec5dd3f640 1 --2- 172.21.15.55:0/3844604118 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fec44067660 0x7fec44069b20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fec4c0af000 tx=0x7fec4c0a7000 comp rx=0 tx=0).stop 2024-03-19T02:01:21.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:21.431+0000 7fec5dd3f640 1 -- 172.21.15.55:0/3844604118 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fec580c9780 msgr2=0x7fec5813a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:21.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:21.431+0000 7fec5dd3f640 1 --2- 172.21.15.55:0/3844604118 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fec580c9780 0x7fec5813a6b0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fec4800c370 tx=0x7fec4800c830 comp rx=0 tx=0).stop 2024-03-19T02:01:21.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:21.431+0000 7fec5dd3f640 1 -- 172.21.15.55:0/3844604118 shutdown_connections 2024-03-19T02:01:21.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:21.431+0000 7fec5dd3f640 1 --2- 172.21.15.55:0/3844604118 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fec44067660 0x7fec44069b20 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:21.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:21.431+0000 7fec5dd3f640 1 --2- 172.21.15.55:0/3844604118 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fec5815b550 0x7fec581f4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:21.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:21.431+0000 7fec5dd3f640 1 --2- 172.21.15.55:0/3844604118 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec5813dd00 0x7fec58137890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:21.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:21.431+0000 7fec5dd3f640 1 --2- 172.21.15.55:0/3844604118 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fec580c9780 0x7fec5813a6b0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:21.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:21.431+0000 7fec5dd3f640 1 -- 172.21.15.55:0/3844604118 >> 172.21.15.55:0/3844604118 conn(0x7fec5806e5a0 msgr2=0x7fec580cb7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:21.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:21.431+0000 7fec5dd3f640 1 -- 172.21.15.55:0/3844604118 shutdown_connections 2024-03-19T02:01:21.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:21.431+0000 7fec5dd3f640 1 -- 172.21.15.55:0/3844604118 wait complete. 2024-03-19T02:01:22.049 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:22.049 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:23.050 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:23.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.163+0000 7f1b254b8640 1 Processor -- start 2024-03-19T02:01:23.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.163+0000 7f1b254b8640 1 -- start start 2024-03-19T02:01:23.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.167+0000 7f1b254b8640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f1b201544f0 0x7f1b20073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:23.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.167+0000 7f1b254b8640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b2015acb0 0x7f1b200cce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:23.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.167+0000 7f1b254b8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b200c7fa0 con 0x7f1b2006b2a0 2024-03-19T02:01:23.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.167+0000 7f1b254b8640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f1b2015b210 con 0x7f1b201544f0 2024-03-19T02:01:23.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.167+0000 7f1b254b8640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1b200c8240 con 0x7f1b2015acb0 2024-03-19T02:01:23.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.167+0000 7f1b1effd640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f1b201544f0 0x7f1b20073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:23.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.167+0000 7f1b17fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f1b2006b2a0 0x7f1b2015e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:47590/0 (socket says 172.21.15.55:47590) 2024-03-19T02:01:23.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.167+0000 7f1b17fff640 1 -- 172.21.15.55:0/3883079891 learned_addr learned my addr 172.21.15.55:0/3883079891 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:01:23.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.167+0000 7f1b1f7fe640 1 --2- 172.21.15.55:0/3883079891 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b2015acb0 0x7f1b200cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:23.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.171+0000 7f1b1e7fc640 1 -- 172.21.15.55:0/3883079891 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3226968131 0 0) 0x7f1b200c7fa0 con 0x7f1b2006b2a0 2024-03-19T02:01:23.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.171+0000 7f1b1e7fc640 1 -- 172.21.15.55:0/3883079891 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b0c003530 con 0x7f1b2006b2a0 2024-03-19T02:01:23.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.171+0000 7f1b1effd640 1 -- 172.21.15.55:0/3883079891 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b2015acb0 msgr2=0x7f1b200cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:23.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.171+0000 7f1b1effd640 1 --2- 172.21.15.55:0/3883079891 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b2015acb0 0x7f1b200cce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:23.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.171+0000 7f1b1effd640 1 -- 172.21.15.55:0/3883079891 >> v1:172.21.15.55:6789/0 conn(0x7f1b2006b2a0 legacy=0x7f1b2015e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:23.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.171+0000 7f1b1effd640 1 -- 172.21.15.55:0/3883079891 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b2015d920 con 0x7f1b201544f0 2024-03-19T02:01:23.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.175+0000 7f1b1effd640 1 --2- 172.21.15.55:0/3883079891 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b201544f0 0x7f1b20073540 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1b100097c0 tx=0x7f1b1001ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c72c27862b005b86 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:23.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.183+0000 7f1b1e7fc640 1 -- 172.21.15.55:0/3883079891 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b100b5070 con 0x7f1b201544f0 2024-03-19T02:01:23.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.183+0000 7f1b1e7fc640 1 -- 172.21.15.55:0/3883079891 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b100a8990 con 0x7f1b201544f0 2024-03-19T02:01:23.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.183+0000 7f1b1e7fc640 1 -- 172.21.15.55:0/3883079891 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b100a8d60 con 0x7f1b201544f0 2024-03-19T02:01:23.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.183+0000 7f1b254b8640 1 -- 172.21.15.55:0/3883079891 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b201544f0 msgr2=0x7f1b20073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:23.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.183+0000 7f1b254b8640 1 --2- 172.21.15.55:0/3883079891 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b201544f0 0x7f1b20073540 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1b100097c0 tx=0x7f1b1001ec90 comp rx=0 tx=0).stop 2024-03-19T02:01:23.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.183+0000 7f1b254b8640 1 -- 172.21.15.55:0/3883079891 shutdown_connections 2024-03-19T02:01:23.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.183+0000 7f1b254b8640 1 --2- 172.21.15.55:0/3883079891 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b2015acb0 0x7f1b200cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:23.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.183+0000 7f1b254b8640 1 --2- 172.21.15.55:0/3883079891 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b201544f0 0x7f1b20073540 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:23.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.183+0000 7f1b254b8640 1 -- 172.21.15.55:0/3883079891 >> 172.21.15.55:0/3883079891 conn(0x7f1b2006e670 msgr2=0x7f1b2006ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:23.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.183+0000 7f1b254b8640 1 -- 172.21.15.55:0/3883079891 shutdown_connections 2024-03-19T02:01:23.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.183+0000 7f1b254b8640 1 -- 172.21.15.55:0/3883079891 wait complete. 2024-03-19T02:01:23.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.183+0000 7f1b254b8640 1 Processor -- start 2024-03-19T02:01:23.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.183+0000 7f1b254b8640 1 -- start start 2024-03-19T02:01:23.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.187+0000 7f1b254b8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b2006b2a0 0x7f1b201f1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:23.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.187+0000 7f1b254b8640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b201544f0 0x7f1b2015c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:23.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.187+0000 7f1b254b8640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b2015acb0 0x7f1b201f06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:23.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.187+0000 7f1b254b8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1b200ca120 con 0x7f1b2006b2a0 2024-03-19T02:01:23.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.187+0000 7f1b254b8640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f1b201548d0 con 0x7f1b2015acb0 2024-03-19T02:01:23.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.187+0000 7f1b254b8640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1b200ca2a0 con 0x7f1b201544f0 2024-03-19T02:01:23.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.187+0000 7f1b1effd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b2006b2a0 0x7f1b201f1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:23.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.187+0000 7f1b1effd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b2006b2a0 0x7f1b201f1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39360/0 (socket says 172.21.15.55:39360) 2024-03-19T02:01:23.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.187+0000 7f1b1effd640 1 -- 172.21.15.55:0/1796264773 learned_addr learned my addr 172.21.15.55:0/1796264773 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:23.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.187+0000 7f1b17fff640 1 --2- 172.21.15.55:0/1796264773 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b201544f0 0x7f1b2015c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:23.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.187+0000 7f1b1f7fe640 1 --2- 172.21.15.55:0/1796264773 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b2015acb0 0x7f1b201f06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:23.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.191+0000 7f1b1effd640 1 -- 172.21.15.55:0/1796264773 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b201544f0 msgr2=0x7f1b2015c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:23.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.191+0000 7f1b1effd640 1 --2- 172.21.15.55:0/1796264773 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b201544f0 0x7f1b2015c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:23.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.191+0000 7f1b1effd640 1 -- 172.21.15.55:0/1796264773 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b2015acb0 msgr2=0x7f1b201f06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:23.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.191+0000 7f1b1effd640 1 --2- 172.21.15.55:0/1796264773 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b2015acb0 0x7f1b201f06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:23.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.191+0000 7f1b1effd640 1 -- 172.21.15.55:0/1796264773 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b201f0c10 con 0x7f1b2006b2a0 2024-03-19T02:01:23.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.195+0000 7f1b1effd640 1 --2- 172.21.15.55:0/1796264773 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b2006b2a0 0x7f1b201f1110 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1b1001c8c0 tx=0x7f1b1001ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:23.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.199+0000 7f1b177fe640 1 -- 172.21.15.55:0/1796264773 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b100bf070 con 0x7f1b2006b2a0 2024-03-19T02:01:23.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.199+0000 7f1b177fe640 1 -- 172.21.15.55:0/1796264773 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b100b0700 con 0x7f1b2006b2a0 2024-03-19T02:01:23.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.199+0000 7f1b177fe640 1 -- 172.21.15.55:0/1796264773 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b100b5040 con 0x7f1b2006b2a0 2024-03-19T02:01:23.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.199+0000 7f1b254b8640 1 -- 172.21.15.55:0/1796264773 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b2015d420 con 0x7f1b2006b2a0 2024-03-19T02:01:23.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.199+0000 7f1b254b8640 1 -- 172.21.15.55:0/1796264773 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b2015b830 con 0x7f1b2006b2a0 2024-03-19T02:01:23.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.199+0000 7f1b254b8640 1 -- 172.21.15.55:0/1796264773 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b20068a40 con 0x7f1b2006b2a0 2024-03-19T02:01:23.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.231+0000 7f1b177fe640 1 -- 172.21.15.55:0/1796264773 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1b100c2070 con 0x7f1b2006b2a0 2024-03-19T02:01:23.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.235+0000 7f1b177fe640 1 --2- 172.21.15.55:0/1796264773 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1b000676b0 0x7f1b00069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:23.232 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.235+0000 7f1b17fff640 1 --2- 172.21.15.55:0/1796264773 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1b000676b0 0x7f1b00069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:23.232 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.235+0000 7f1b17fff640 1 --2- 172.21.15.55:0/1796264773 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1b000676b0 0x7f1b00069b70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1b201f3170 tx=0x7f1b08000bd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:23.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.543+0000 7f1b177fe640 1 -- 172.21.15.55:0/1796264773 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f1b100a8910 con 0x7f1b2006b2a0 2024-03-19T02:01:23.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.543+0000 7f1b177fe640 1 -- 172.21.15.55:0/1796264773 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1b100c9080 con 0x7f1b2006b2a0 2024-03-19T02:01:23.753 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.755+0000 7f1b254b8640 1 -- 172.21.15.55:0/1796264773 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b20153eb0 con 0x7f1b2006b2a0 2024-03-19T02:01:23.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.771+0000 7f1b177fe640 1 -- 172.21.15.55:0/1796264773 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f1b100187f0 con 0x7f1b2006b2a0 2024-03-19T02:01:23.770 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:23.770 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:23.770 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:23.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.775+0000 7f1b254b8640 1 -- 172.21.15.55:0/1796264773 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1b000676b0 msgr2=0x7f1b00069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:23.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.775+0000 7f1b254b8640 1 --2- 172.21.15.55:0/1796264773 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1b000676b0 0x7f1b00069b70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1b201f3170 tx=0x7f1b08000bd0 comp rx=0 tx=0).stop 2024-03-19T02:01:23.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.775+0000 7f1b254b8640 1 -- 172.21.15.55:0/1796264773 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b2006b2a0 msgr2=0x7f1b201f1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:23.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.775+0000 7f1b254b8640 1 --2- 172.21.15.55:0/1796264773 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b2006b2a0 0x7f1b201f1110 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1b1001c8c0 tx=0x7f1b1001ba30 comp rx=0 tx=0).stop 2024-03-19T02:01:23.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.775+0000 7f1b254b8640 1 -- 172.21.15.55:0/1796264773 shutdown_connections 2024-03-19T02:01:23.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.775+0000 7f1b254b8640 1 --2- 172.21.15.55:0/1796264773 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1b000676b0 0x7f1b00069b70 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:23.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.775+0000 7f1b254b8640 1 --2- 172.21.15.55:0/1796264773 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b2015acb0 0x7f1b201f06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:23.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.775+0000 7f1b254b8640 1 --2- 172.21.15.55:0/1796264773 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b201544f0 0x7f1b2015c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:23.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.775+0000 7f1b254b8640 1 --2- 172.21.15.55:0/1796264773 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b2006b2a0 0x7f1b201f1110 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:23.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.775+0000 7f1b254b8640 1 -- 172.21.15.55:0/1796264773 >> 172.21.15.55:0/1796264773 conn(0x7f1b2006e670 msgr2=0x7f1b200707d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:23.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.775+0000 7f1b254b8640 1 -- 172.21.15.55:0/1796264773 shutdown_connections 2024-03-19T02:01:23.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:23.775+0000 7f1b254b8640 1 -- 172.21.15.55:0/1796264773 wait complete. 2024-03-19T02:01:24.394 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:24.394 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:25.395 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:25.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.527+0000 7efe5d68e640 1 Processor -- start 2024-03-19T02:01:25.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.527+0000 7efe5d68e640 1 -- start start 2024-03-19T02:01:25.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.527+0000 7efe5d68e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7efe581544f0 0x7efe58073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:25.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.527+0000 7efe5d68e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7efe5806b2a0 0x7efe5815e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:25.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.527+0000 7efe5d68e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe580c7fa0 con 0x7efe5815acb0 2024-03-19T02:01:25.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.527+0000 7efe5d68e640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7efe5815b210 con 0x7efe5806b2a0 2024-03-19T02:01:25.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.527+0000 7efe5d68e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7efe580c8240 con 0x7efe581544f0 2024-03-19T02:01:25.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.527+0000 7efe567fc640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7efe5806b2a0 0x7efe5815e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:25.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.527+0000 7efe567fc640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7efe5806b2a0 0x7efe5815e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39362/0 (socket says 172.21.15.55:39362) 2024-03-19T02:01:25.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.527+0000 7efe567fc640 1 -- 172.21.15.55:0/2382633354 learned_addr learned my addr 172.21.15.55:0/2382633354 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:25.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.527+0000 7efe56ffd640 1 --2- 172.21.15.55:0/2382633354 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7efe581544f0 0x7efe58073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:25.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.531+0000 7efe567fc640 1 -- 172.21.15.55:0/2382633354 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7efe581544f0 msgr2=0x7efe58073540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:25.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.531+0000 7efe567fc640 1 --2- 172.21.15.55:0/2382633354 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7efe581544f0 0x7efe58073540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:25.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.531+0000 7efe567fc640 1 -- 172.21.15.55:0/2382633354 >> v1:172.21.15.55:6789/0 conn(0x7efe5815acb0 legacy=0x7efe580cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:25.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.531+0000 7efe567fc640 1 -- 172.21.15.55:0/2382633354 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe5815d920 con 0x7efe5806b2a0 2024-03-19T02:01:25.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.539+0000 7efe567fc640 1 --2- 172.21.15.55:0/2382633354 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efe5806b2a0 0x7efe5815e120 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7efe4c009870 tx=0x7efe4c01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27cebbac4bcbefd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:25.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.543+0000 7efe55ffb640 1 -- 172.21.15.55:0/2382633354 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4c0b5070 con 0x7efe5806b2a0 2024-03-19T02:01:25.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.543+0000 7efe55ffb640 1 -- 172.21.15.55:0/2382633354 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe4c0a8a50 con 0x7efe5806b2a0 2024-03-19T02:01:25.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.543+0000 7efe55ffb640 1 -- 172.21.15.55:0/2382633354 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4c0a8e20 con 0x7efe5806b2a0 2024-03-19T02:01:25.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.543+0000 7efe5d68e640 1 -- 172.21.15.55:0/2382633354 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efe5806b2a0 msgr2=0x7efe5815e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:25.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.543+0000 7efe5d68e640 1 --2- 172.21.15.55:0/2382633354 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efe5806b2a0 0x7efe5815e120 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7efe4c009870 tx=0x7efe4c01ed50 comp rx=0 tx=0).stop 2024-03-19T02:01:25.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.543+0000 7efe5d68e640 1 -- 172.21.15.55:0/2382633354 shutdown_connections 2024-03-19T02:01:25.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.543+0000 7efe5d68e640 1 --2- 172.21.15.55:0/2382633354 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efe5806b2a0 0x7efe5815e120 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:25.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.543+0000 7efe5d68e640 1 --2- 172.21.15.55:0/2382633354 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7efe581544f0 0x7efe58073540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:25.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.543+0000 7efe5d68e640 1 -- 172.21.15.55:0/2382633354 >> 172.21.15.55:0/2382633354 conn(0x7efe5806e670 msgr2=0x7efe5806ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:25.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.543+0000 7efe5d68e640 1 -- 172.21.15.55:0/2382633354 shutdown_connections 2024-03-19T02:01:25.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.543+0000 7efe5d68e640 1 -- 172.21.15.55:0/2382633354 wait complete. 2024-03-19T02:01:25.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.543+0000 7efe5d68e640 1 Processor -- start 2024-03-19T02:01:25.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.543+0000 7efe5d68e640 1 -- start start 2024-03-19T02:01:25.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.547+0000 7efe5d68e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efe5806b2a0 0x7efe581e3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:25.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.547+0000 7efe5d68e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efe581544f0 0x7efe580cbe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:25.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.547+0000 7efe5d68e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7efe5815acb0 0x7efe581f04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:25.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.547+0000 7efe5d68e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7efe581548d0 con 0x7efe5806b2a0 2024-03-19T02:01:25.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.547+0000 7efe5d68e640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7efe5815e660 con 0x7efe581544f0 2024-03-19T02:01:25.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.547+0000 7efe5d68e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7efe580cd540 con 0x7efe5815acb0 2024-03-19T02:01:25.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.547+0000 7efe56ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efe5806b2a0 0x7efe581e3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:25.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.547+0000 7efe577fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7efe5815acb0 0x7efe581f04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:25.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.547+0000 7efe567fc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efe581544f0 0x7efe580cbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:25.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.547+0000 7efe56ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efe5806b2a0 0x7efe581e3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39364/0 (socket says 172.21.15.55:39364) 2024-03-19T02:01:25.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.547+0000 7efe56ffd640 1 -- 172.21.15.55:0/1691621905 learned_addr learned my addr 172.21.15.55:0/1691621905 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:25.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.551+0000 7efe56ffd640 1 -- 172.21.15.55:0/1691621905 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7efe5815acb0 msgr2=0x7efe581f04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:25.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.551+0000 7efe56ffd640 1 --2- 172.21.15.55:0/1691621905 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7efe5815acb0 0x7efe581f04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:25.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.551+0000 7efe56ffd640 1 -- 172.21.15.55:0/1691621905 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efe581544f0 msgr2=0x7efe580cbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:25.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.551+0000 7efe56ffd640 1 --2- 172.21.15.55:0/1691621905 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efe581544f0 0x7efe580cbe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:25.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.551+0000 7efe56ffd640 1 -- 172.21.15.55:0/1691621905 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe581e3ac0 con 0x7efe5806b2a0 2024-03-19T02:01:25.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.555+0000 7efe56ffd640 1 --2- 172.21.15.55:0/1691621905 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efe5806b2a0 0x7efe581e3580 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7efe48009db0 tx=0x7efe4800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:25.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.559+0000 7efe3f7fe640 1 -- 172.21.15.55:0/1691621905 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe48018020 con 0x7efe5806b2a0 2024-03-19T02:01:25.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.559+0000 7efe3f7fe640 1 -- 172.21.15.55:0/1691621905 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe48014480 con 0x7efe5806b2a0 2024-03-19T02:01:25.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.559+0000 7efe3f7fe640 1 -- 172.21.15.55:0/1691621905 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe48014850 con 0x7efe5806b2a0 2024-03-19T02:01:25.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.559+0000 7efe5d68e640 1 -- 172.21.15.55:0/1691621905 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe581effe0 con 0x7efe5806b2a0 2024-03-19T02:01:25.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.559+0000 7efe5d68e640 1 -- 172.21.15.55:0/1691621905 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe5815a520 con 0x7efe5806b2a0 2024-03-19T02:01:25.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.563+0000 7efe5d68e640 1 -- 172.21.15.55:0/1691621905 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe58068a40 con 0x7efe5806b2a0 2024-03-19T02:01:25.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.591+0000 7efe3f7fe640 1 -- 172.21.15.55:0/1691621905 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7efe4801b020 con 0x7efe5806b2a0 2024-03-19T02:01:25.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.591+0000 7efe3f7fe640 1 --2- 172.21.15.55:0/1691621905 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7efe440676e0 0x7efe44069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:25.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.591+0000 7efe567fc640 1 --2- 172.21.15.55:0/1691621905 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7efe440676e0 0x7efe44069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:25.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.591+0000 7efe567fc640 1 --2- 172.21.15.55:0/1691621905 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7efe440676e0 0x7efe44069ba0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7efe4c0b2000 tx=0x7efe4c0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:25.591 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.595+0000 7efe3f7fe640 1 -- 172.21.15.55:0/1691621905 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7efe480551c0 con 0x7efe5806b2a0 2024-03-19T02:01:25.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:25.899+0000 7efe3f7fe640 1 -- 172.21.15.55:0/1691621905 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7efe48022030 con 0x7efe5806b2a0 2024-03-19T02:01:26.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:26.135+0000 7efe5d68e640 1 -- 172.21.15.55:0/1691621905 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe580c7be0 con 0x7efe5806b2a0 2024-03-19T02:01:26.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:26.179+0000 7efe3f7fe640 1 -- 172.21.15.55:0/1691621905 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7efe48010040 con 0x7efe5806b2a0 2024-03-19T02:01:26.179 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:26.179 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:26.179 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:26.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:26.183+0000 7efe5d68e640 1 -- 172.21.15.55:0/1691621905 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7efe440676e0 msgr2=0x7efe44069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:26.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:26.183+0000 7efe5d68e640 1 --2- 172.21.15.55:0/1691621905 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7efe440676e0 0x7efe44069ba0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7efe4c0b2000 tx=0x7efe4c0aa070 comp rx=0 tx=0).stop 2024-03-19T02:01:26.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:26.183+0000 7efe5d68e640 1 -- 172.21.15.55:0/1691621905 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efe5806b2a0 msgr2=0x7efe581e3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:26.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:26.183+0000 7efe5d68e640 1 --2- 172.21.15.55:0/1691621905 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efe5806b2a0 0x7efe581e3580 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7efe48009db0 tx=0x7efe4800c490 comp rx=0 tx=0).stop 2024-03-19T02:01:26.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:26.183+0000 7efe5d68e640 1 -- 172.21.15.55:0/1691621905 shutdown_connections 2024-03-19T02:01:26.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:26.183+0000 7efe5d68e640 1 --2- 172.21.15.55:0/1691621905 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7efe440676e0 0x7efe44069ba0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:26.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:26.183+0000 7efe5d68e640 1 --2- 172.21.15.55:0/1691621905 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7efe5815acb0 0x7efe581f04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:26.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:26.183+0000 7efe5d68e640 1 --2- 172.21.15.55:0/1691621905 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efe581544f0 0x7efe580cbe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:26.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:26.183+0000 7efe5d68e640 1 --2- 172.21.15.55:0/1691621905 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efe5806b2a0 0x7efe581e3580 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:26.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:26.183+0000 7efe5d68e640 1 -- 172.21.15.55:0/1691621905 >> 172.21.15.55:0/1691621905 conn(0x7efe5806e670 msgr2=0x7efe580caf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:26.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:26.183+0000 7efe5d68e640 1 -- 172.21.15.55:0/1691621905 shutdown_connections 2024-03-19T02:01:26.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:26.183+0000 7efe5d68e640 1 -- 172.21.15.55:0/1691621905 wait complete. 2024-03-19T02:01:26.800 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:26.801 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:27.801 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:27.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.923+0000 7fded64f8640 1 Processor -- start 2024-03-19T02:01:27.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.923+0000 7fded64f8640 1 -- start start 2024-03-19T02:01:27.922 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.923+0000 7fded64f8640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fded0150f50 0x7fded015b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:27.922 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.923+0000 7fded64f8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fded00c3320 0x7fded00728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:27.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.923+0000 7fded64f8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fded0151330 con 0x7fded00ca350 2024-03-19T02:01:27.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.923+0000 7fded64f8640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fded00c3e00 con 0x7fded00c3320 2024-03-19T02:01:27.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.923+0000 7fded64f8640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fded015ac10 con 0x7fded0150f50 2024-03-19T02:01:27.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.927+0000 7fded4a95640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fded00c3320 0x7fded00728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:27.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.927+0000 7fdecffff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fded0150f50 0x7fded015b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:27.924 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.927+0000 7fded4a95640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fded00c3320 0x7fded00728b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39378/0 (socket says 172.21.15.55:39378) 2024-03-19T02:01:27.924 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.927+0000 7fded4a95640 1 -- 172.21.15.55:0/3919547133 learned_addr learned my addr 172.21.15.55:0/3919547133 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:27.927 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.927+0000 7fdecffff640 1 -- 172.21.15.55:0/3919547133 >> v2:172.21.15.55:3300/0 conn(0x7fded00c3320 msgr2=0x7fded00728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:27.927 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.927+0000 7fdecffff640 1 --2- 172.21.15.55:0/3919547133 >> v2:172.21.15.55:3300/0 conn(0x7fded00c3320 0x7fded00728b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:27.927 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.927+0000 7fdecffff640 1 -- 172.21.15.55:0/3919547133 >> v1:172.21.15.55:6789/0 conn(0x7fded00ca350 legacy=0x7fded0157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:27.927 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.927+0000 7fdecffff640 1 -- 172.21.15.55:0/3919547133 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded00c3840 con 0x7fded0150f50 2024-03-19T02:01:27.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.935+0000 7fdecffff640 1 --2- 172.21.15.55:0/3919547133 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fded0150f50 0x7fded015b0a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fded00cdbf0 tx=0x7fdec001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=164fac64b83fdcbc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:27.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.943+0000 7fdeceffd640 1 -- 172.21.15.55:0/3919547133 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec00ab680 con 0x7fded0150f50 2024-03-19T02:01:27.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.943+0000 7fdeceffd640 1 -- 172.21.15.55:0/3919547133 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdec00b0070 con 0x7fded0150f50 2024-03-19T02:01:27.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.943+0000 7fdeceffd640 1 -- 172.21.15.55:0/3919547133 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec00aba40 con 0x7fded0150f50 2024-03-19T02:01:27.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.943+0000 7fded64f8640 1 -- 172.21.15.55:0/3919547133 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fded0150f50 msgr2=0x7fded015b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:27.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.943+0000 7fded64f8640 1 --2- 172.21.15.55:0/3919547133 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fded0150f50 0x7fded015b0a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fded00cdbf0 tx=0x7fdec001c480 comp rx=0 tx=0).stop 2024-03-19T02:01:27.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.943+0000 7fded64f8640 1 -- 172.21.15.55:0/3919547133 shutdown_connections 2024-03-19T02:01:27.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.943+0000 7fded64f8640 1 --2- 172.21.15.55:0/3919547133 >> v2:172.21.15.55:3300/0 conn(0x7fded00c3320 0x7fded00728b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:27.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.943+0000 7fded64f8640 1 --2- 172.21.15.55:0/3919547133 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fded0150f50 0x7fded015b0a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:27.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.943+0000 7fded64f8640 1 -- 172.21.15.55:0/3919547133 >> 172.21.15.55:0/3919547133 conn(0x7fded006e670 msgr2=0x7fded006ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:27.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.943+0000 7fded64f8640 1 -- 172.21.15.55:0/3919547133 shutdown_connections 2024-03-19T02:01:27.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.943+0000 7fded64f8640 1 -- 172.21.15.55:0/3919547133 wait complete. 2024-03-19T02:01:27.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.943+0000 7fded64f8640 1 Processor -- start 2024-03-19T02:01:27.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.943+0000 7fded64f8640 1 -- start start 2024-03-19T02:01:27.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.947+0000 7fded64f8640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fded00c3320 0x7fded006b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:27.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.947+0000 7fded64f8640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fded00ca350 0x7fded006b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:27.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.947+0000 7fded64f8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fded0150f50 0x7fded0135e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:27.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.947+0000 7fded64f8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fded01639b0 con 0x7fded0150f50 2024-03-19T02:01:27.945 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.947+0000 7fded64f8640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fded0163830 con 0x7fded00ca350 2024-03-19T02:01:27.945 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.947+0000 7fded64f8640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fded0163b30 con 0x7fded00c3320 2024-03-19T02:01:27.945 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.947+0000 7fded4a95640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fded0150f50 0x7fded0135e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:27.945 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.947+0000 7fdecf7fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fded00ca350 0x7fded006b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:27.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.947+0000 7fded4a95640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fded0150f50 0x7fded0135e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39388/0 (socket says 172.21.15.55:39388) 2024-03-19T02:01:27.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.947+0000 7fded4a95640 1 -- 172.21.15.55:0/2671987587 learned_addr learned my addr 172.21.15.55:0/2671987587 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:27.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.947+0000 7fdecffff640 1 --2- 172.21.15.55:0/2671987587 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fded00c3320 0x7fded006b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:27.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.951+0000 7fded4a95640 1 -- 172.21.15.55:0/2671987587 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fded00c3320 msgr2=0x7fded006b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:27.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.951+0000 7fded4a95640 1 --2- 172.21.15.55:0/2671987587 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fded00c3320 0x7fded006b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:27.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.951+0000 7fded4a95640 1 -- 172.21.15.55:0/2671987587 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fded00ca350 msgr2=0x7fded006b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:27.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.951+0000 7fded4a95640 1 --2- 172.21.15.55:0/2671987587 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fded00ca350 0x7fded006b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:27.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.951+0000 7fded4a95640 1 -- 172.21.15.55:0/2671987587 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded0143b70 con 0x7fded0150f50 2024-03-19T02:01:27.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.955+0000 7fded4a95640 1 --2- 172.21.15.55:0/2671987587 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fded0150f50 0x7fded0135e80 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fdec800efc0 tx=0x7fdec800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:27.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.959+0000 7fdeccff9640 1 -- 172.21.15.55:0/2671987587 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec800cd20 con 0x7fded0150f50 2024-03-19T02:01:27.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.959+0000 7fdeccff9640 1 -- 172.21.15.55:0/2671987587 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdec8009e50 con 0x7fded0150f50 2024-03-19T02:01:27.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.959+0000 7fded64f8640 1 -- 172.21.15.55:0/2671987587 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fded0143e50 con 0x7fded0150f50 2024-03-19T02:01:27.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.959+0000 7fdeccff9640 1 -- 172.21.15.55:0/2671987587 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec8003be0 con 0x7fded0150f50 2024-03-19T02:01:27.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.959+0000 7fded64f8640 1 -- 172.21.15.55:0/2671987587 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fded01f38f0 con 0x7fded0150f50 2024-03-19T02:01:27.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:27.959+0000 7fded64f8640 1 -- 172.21.15.55:0/2671987587 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fded01f3400 con 0x7fded0150f50 2024-03-19T02:01:28.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:28.275+0000 7fdeccff9640 1 -- 172.21.15.55:0/2671987587 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fdec8012050 con 0x7fded0150f50 2024-03-19T02:01:28.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:28.275+0000 7fdeccff9640 1 --2- 172.21.15.55:0/2671987587 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdebc0676b0 0x7fdebc069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:28.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:28.275+0000 7fdecffff640 1 --2- 172.21.15.55:0/2671987587 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdebc0676b0 0x7fdebc069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:28.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:28.275+0000 7fdecffff640 1 --2- 172.21.15.55:0/2671987587 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdebc0676b0 0x7fdebc069b70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdec00af000 tx=0x7fdec00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:28.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:28.315+0000 7fdeccff9640 1 -- 172.21.15.55:0/2671987587 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fdec8055010 con 0x7fded0150f50 2024-03-19T02:01:28.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:28.315+0000 7fdeccff9640 1 -- 172.21.15.55:0/2671987587 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fdec8010040 con 0x7fded0150f50 2024-03-19T02:01:28.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:28.555+0000 7fded64f8640 1 -- 172.21.15.55:0/2671987587 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fded0068a40 con 0x7fded0150f50 2024-03-19T02:01:28.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:28.567+0000 7fdeccff9640 1 -- 172.21.15.55:0/2671987587 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fdec8056f70 con 0x7fded0150f50 2024-03-19T02:01:28.569 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:28.569 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:28.569 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:28.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:28.571+0000 7fded64f8640 1 -- 172.21.15.55:0/2671987587 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdebc0676b0 msgr2=0x7fdebc069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:28.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:28.571+0000 7fded64f8640 1 --2- 172.21.15.55:0/2671987587 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdebc0676b0 0x7fdebc069b70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdec00af000 tx=0x7fdec00a7000 comp rx=0 tx=0).stop 2024-03-19T02:01:28.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:28.571+0000 7fded64f8640 1 -- 172.21.15.55:0/2671987587 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fded0150f50 msgr2=0x7fded0135e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:28.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:28.571+0000 7fded64f8640 1 --2- 172.21.15.55:0/2671987587 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fded0150f50 0x7fded0135e80 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fdec800efc0 tx=0x7fdec800c490 comp rx=0 tx=0).stop 2024-03-19T02:01:28.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:28.571+0000 7fded64f8640 1 -- 172.21.15.55:0/2671987587 shutdown_connections 2024-03-19T02:01:28.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:28.571+0000 7fded64f8640 1 --2- 172.21.15.55:0/2671987587 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdebc0676b0 0x7fdebc069b70 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:28.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:28.571+0000 7fded64f8640 1 --2- 172.21.15.55:0/2671987587 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fded0150f50 0x7fded0135e80 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:28.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:28.571+0000 7fded64f8640 1 --2- 172.21.15.55:0/2671987587 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fded00ca350 0x7fded006b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:28.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:28.571+0000 7fded64f8640 1 --2- 172.21.15.55:0/2671987587 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fded00c3320 0x7fded006b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:28.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:28.571+0000 7fded64f8640 1 -- 172.21.15.55:0/2671987587 >> 172.21.15.55:0/2671987587 conn(0x7fded006e670 msgr2=0x7fded0144c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:28.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:28.571+0000 7fded64f8640 1 -- 172.21.15.55:0/2671987587 shutdown_connections 2024-03-19T02:01:28.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:28.571+0000 7fded64f8640 1 -- 172.21.15.55:0/2671987587 wait complete. 2024-03-19T02:01:29.191 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:29.192 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:30.193 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:30.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.307+0000 7f202311f640 1 Processor -- start 2024-03-19T02:01:30.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.307+0000 7f202311f640 1 -- start start 2024-03-19T02:01:30.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.311+0000 7f202311f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f201c154410 0x7f201c0cae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:30.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.311+0000 7f202311f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f201c06f5e0 0x7f201c06fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:30.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.311+0000 7f202311f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f201c0c9460 con 0x7f201c15d8b0 2024-03-19T02:01:30.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.311+0000 7f202311f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f201c15a2c0 con 0x7f201c154410 2024-03-19T02:01:30.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.311+0000 7f202311f640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f201c15dda0 con 0x7f201c06f5e0 2024-03-19T02:01:30.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.311+0000 7f2020ebb640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f201c154410 0x7f201c0cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:30.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.311+0000 7f2020ebb640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f201c154410 0x7f201c0cae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34364/0 (socket says 172.21.15.55:34364) 2024-03-19T02:01:30.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.311+0000 7f2020ebb640 1 -- 172.21.15.55:0/1828699802 learned_addr learned my addr 172.21.15.55:0/1828699802 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:30.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.311+0000 7f20216bc640 1 --2- 172.21.15.55:0/1828699802 >> v2:172.21.15.84:3300/0 conn(0x7f201c06f5e0 0x7f201c06fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:30.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.315+0000 7f2020ebb640 1 -- 172.21.15.55:0/1828699802 >> v2:172.21.15.84:3300/0 conn(0x7f201c06f5e0 msgr2=0x7f201c06fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:30.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.315+0000 7f2020ebb640 1 --2- 172.21.15.55:0/1828699802 >> v2:172.21.15.84:3300/0 conn(0x7f201c06f5e0 0x7f201c06fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:30.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.315+0000 7f2020ebb640 1 -- 172.21.15.55:0/1828699802 >> v1:172.21.15.55:6789/0 conn(0x7f201c15d8b0 legacy=0x7f201c15aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:30.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.315+0000 7f2020ebb640 1 -- 172.21.15.55:0/1828699802 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201c15e080 con 0x7f201c154410 2024-03-19T02:01:30.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.323+0000 7f2020ebb640 1 --2- 172.21.15.55:0/1828699802 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f201c154410 0x7f201c0cae30 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f201c0c2d80 tx=0x7f201001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5156952ad15d7976 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:30.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.327+0000 7f201b7fe640 1 -- 172.21.15.55:0/1828699802 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20100ab660 con 0x7f201c154410 2024-03-19T02:01:30.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.327+0000 7f201b7fe640 1 -- 172.21.15.55:0/1828699802 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20100b0070 con 0x7f201c154410 2024-03-19T02:01:30.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.327+0000 7f201b7fe640 1 -- 172.21.15.55:0/1828699802 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20100aba20 con 0x7f201c154410 2024-03-19T02:01:30.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.327+0000 7f202311f640 1 -- 172.21.15.55:0/1828699802 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f201c154410 msgr2=0x7f201c0cae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:30.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.327+0000 7f202311f640 1 --2- 172.21.15.55:0/1828699802 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f201c154410 0x7f201c0cae30 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f201c0c2d80 tx=0x7f201001c460 comp rx=0 tx=0).stop 2024-03-19T02:01:30.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.327+0000 7f202311f640 1 -- 172.21.15.55:0/1828699802 shutdown_connections 2024-03-19T02:01:30.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.327+0000 7f202311f640 1 --2- 172.21.15.55:0/1828699802 >> v2:172.21.15.84:3300/0 conn(0x7f201c06f5e0 0x7f201c06fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:30.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.327+0000 7f202311f640 1 --2- 172.21.15.55:0/1828699802 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f201c154410 0x7f201c0cae30 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:30.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.327+0000 7f202311f640 1 -- 172.21.15.55:0/1828699802 >> 172.21.15.55:0/1828699802 conn(0x7f201c070cc0 msgr2=0x7f201c067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:30.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.327+0000 7f202311f640 1 -- 172.21.15.55:0/1828699802 shutdown_connections 2024-03-19T02:01:30.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.327+0000 7f202311f640 1 -- 172.21.15.55:0/1828699802 wait complete. 2024-03-19T02:01:30.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.331+0000 7f202311f640 1 Processor -- start 2024-03-19T02:01:30.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.331+0000 7f202311f640 1 -- start start 2024-03-19T02:01:30.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.331+0000 7f202311f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f201c06f5e0 0x7f201c157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:30.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.331+0000 7f202311f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f201c154410 0x7f201c155a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:30.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.331+0000 7f202311f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f201c15d8b0 0x7f201c155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:30.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.331+0000 7f202311f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f201c0ce8a0 con 0x7f201c06f5e0 2024-03-19T02:01:30.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.331+0000 7f202311f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f201c15b1b0 con 0x7f201c15d8b0 2024-03-19T02:01:30.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.331+0000 7f202311f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f201c0cea20 con 0x7f201c154410 2024-03-19T02:01:30.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.331+0000 7f2020ebb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f201c06f5e0 0x7f201c157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:30.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.331+0000 7f20216bc640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f201c15d8b0 0x7f201c155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:30.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.331+0000 7f2020ebb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f201c06f5e0 0x7f201c157cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34368/0 (socket says 172.21.15.55:34368) 2024-03-19T02:01:30.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.331+0000 7f2020ebb640 1 -- 172.21.15.55:0/1715766883 learned_addr learned my addr 172.21.15.55:0/1715766883 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:30.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.331+0000 7f201bfff640 1 --2- 172.21.15.55:0/1715766883 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f201c154410 0x7f201c155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:30.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.335+0000 7f2020ebb640 1 -- 172.21.15.55:0/1715766883 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f201c154410 msgr2=0x7f201c155a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:30.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.335+0000 7f2020ebb640 1 --2- 172.21.15.55:0/1715766883 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f201c154410 0x7f201c155a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:30.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.335+0000 7f2020ebb640 1 -- 172.21.15.55:0/1715766883 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f201c15d8b0 msgr2=0x7f201c155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:30.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.335+0000 7f2020ebb640 1 --2- 172.21.15.55:0/1715766883 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f201c15d8b0 0x7f201c155f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:30.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.335+0000 7f2020ebb640 1 -- 172.21.15.55:0/1715766883 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201c14e040 con 0x7f201c06f5e0 2024-03-19T02:01:30.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.339+0000 7f2020ebb640 1 --2- 172.21.15.55:0/1715766883 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f201c06f5e0 0x7f201c157cb0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f20100ad040 tx=0x7f201001aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:30.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.343+0000 7f20197fa640 1 -- 172.21.15.55:0/1715766883 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20100be5f0 con 0x7f201c06f5e0 2024-03-19T02:01:30.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.343+0000 7f20197fa640 1 -- 172.21.15.55:0/1715766883 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20100be7b0 con 0x7f201c06f5e0 2024-03-19T02:01:30.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.343+0000 7f202311f640 1 -- 172.21.15.55:0/1715766883 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f201c14e320 con 0x7f201c06f5e0 2024-03-19T02:01:30.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.343+0000 7f20197fa640 1 -- 172.21.15.55:0/1715766883 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20100beba0 con 0x7f201c06f5e0 2024-03-19T02:01:30.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.343+0000 7f202311f640 1 -- 172.21.15.55:0/1715766883 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f201c154f30 con 0x7f201c06f5e0 2024-03-19T02:01:30.342 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.343+0000 7f202311f640 1 -- 172.21.15.55:0/1715766883 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f201c068330 con 0x7f201c06f5e0 2024-03-19T02:01:30.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.375+0000 7f20197fa640 1 -- 172.21.15.55:0/1715766883 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f20100b43f0 con 0x7f201c06f5e0 2024-03-19T02:01:30.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.379+0000 7f20197fa640 1 --2- 172.21.15.55:0/1715766883 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f20000676b0 0x7f2000069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:30.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.379+0000 7f201bfff640 1 --2- 172.21.15.55:0/1715766883 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f20000676b0 0x7f2000069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:30.376 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.379+0000 7f201bfff640 1 --2- 172.21.15.55:0/1715766883 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f20000676b0 0x7f2000069b70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f201c0b2cb0 tx=0x7f2008009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:30.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.383+0000 7f20197fa640 1 -- 172.21.15.55:0/1715766883 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f20100ee720 con 0x7f201c06f5e0 2024-03-19T02:01:30.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.651+0000 7f20197fa640 1 -- 172.21.15.55:0/1715766883 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f20100c4060 con 0x7f201c06f5e0 2024-03-19T02:01:30.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.855+0000 7f202311f640 1 -- 172.21.15.55:0/1715766883 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f201c0ca880 con 0x7f201c06f5e0 2024-03-19T02:01:30.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.875+0000 7f20197fa640 1 -- 172.21.15.55:0/1715766883 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f2010016110 con 0x7f201c06f5e0 2024-03-19T02:01:30.875 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:30.875 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:30.876 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:30.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.879+0000 7f202311f640 1 -- 172.21.15.55:0/1715766883 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f20000676b0 msgr2=0x7f2000069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:30.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.879+0000 7f202311f640 1 --2- 172.21.15.55:0/1715766883 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f20000676b0 0x7f2000069b70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f201c0b2cb0 tx=0x7f2008009040 comp rx=0 tx=0).stop 2024-03-19T02:01:30.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.879+0000 7f202311f640 1 -- 172.21.15.55:0/1715766883 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f201c06f5e0 msgr2=0x7f201c157cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:30.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.879+0000 7f202311f640 1 --2- 172.21.15.55:0/1715766883 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f201c06f5e0 0x7f201c157cb0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f20100ad040 tx=0x7f201001aca0 comp rx=0 tx=0).stop 2024-03-19T02:01:30.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.879+0000 7f202311f640 1 -- 172.21.15.55:0/1715766883 shutdown_connections 2024-03-19T02:01:30.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.879+0000 7f202311f640 1 --2- 172.21.15.55:0/1715766883 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f20000676b0 0x7f2000069b70 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:30.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.879+0000 7f202311f640 1 --2- 172.21.15.55:0/1715766883 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f201c15d8b0 0x7f201c155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:30.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.879+0000 7f202311f640 1 --2- 172.21.15.55:0/1715766883 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f201c154410 0x7f201c155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:30.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.879+0000 7f202311f640 1 --2- 172.21.15.55:0/1715766883 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f201c06f5e0 0x7f201c157cb0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:30.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.879+0000 7f202311f640 1 -- 172.21.15.55:0/1715766883 >> 172.21.15.55:0/1715766883 conn(0x7f201c070cc0 msgr2=0x7f201c159370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:30.880 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.879+0000 7f202311f640 1 -- 172.21.15.55:0/1715766883 shutdown_connections 2024-03-19T02:01:30.880 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:30.879+0000 7f202311f640 1 -- 172.21.15.55:0/1715766883 wait complete. 2024-03-19T02:01:31.503 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:31.503 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:32.504 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:32.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.627+0000 7f408dc61640 1 Processor -- start 2024-03-19T02:01:32.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.627+0000 7f408dc61640 1 -- start start 2024-03-19T02:01:32.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.627+0000 7f408dc61640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f4088154420 0x7f408815dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:32.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.627+0000 7f408dc61640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f408815e320 0x7f40880728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:32.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.627+0000 7f408dc61640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f408815a660 con 0x7f408815a1b0 2024-03-19T02:01:32.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.627+0000 7f408dc61640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f408815d950 con 0x7f408815e320 2024-03-19T02:01:32.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.627+0000 7f408dc61640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f408815a940 con 0x7f4088154420 2024-03-19T02:01:32.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.631+0000 7f40877fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f4088154420 0x7f408815dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:32.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.631+0000 7f40877fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f4088154420 0x7f408815dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:40132/0 (socket says 172.21.15.55:40132) 2024-03-19T02:01:32.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.631+0000 7f40877fe640 1 -- 172.21.15.55:0/3424335226 learned_addr learned my addr 172.21.15.55:0/3424335226 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:32.631 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.631+0000 7f40877fe640 1 -- 172.21.15.55:0/3424335226 >> v2:172.21.15.55:3300/0 conn(0x7f408815e320 msgr2=0x7f40880728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:32.631 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.631+0000 7f40877fe640 1 --2- 172.21.15.55:0/3424335226 >> v2:172.21.15.55:3300/0 conn(0x7f408815e320 0x7f40880728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:32.631 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.631+0000 7f40877fe640 1 -- 172.21.15.55:0/3424335226 >> v1:172.21.15.55:6789/0 conn(0x7f408815a1b0 legacy=0x7f40880708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:32.631 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.631+0000 7f40877fe640 1 -- 172.21.15.55:0/3424335226 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408815ac20 con 0x7f4088154420 2024-03-19T02:01:32.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.635+0000 7f40877fe640 1 --2- 172.21.15.55:0/3424335226 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4088154420 0x7f408815dde0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f40880705c0 tx=0x7f407801ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94d6ff6f0ade074d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:32.639 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.639+0000 7f40867fc640 1 -- 172.21.15.55:0/3424335226 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40780b5070 con 0x7f4088154420 2024-03-19T02:01:32.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.647+0000 7f40867fc640 1 -- 172.21.15.55:0/3424335226 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40780a89d0 con 0x7f4088154420 2024-03-19T02:01:32.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.647+0000 7f40867fc640 1 -- 172.21.15.55:0/3424335226 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40780a8da0 con 0x7f4088154420 2024-03-19T02:01:32.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.647+0000 7f408dc61640 1 -- 172.21.15.55:0/3424335226 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4088154420 msgr2=0x7f408815dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:32.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.647+0000 7f408dc61640 1 --2- 172.21.15.55:0/3424335226 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4088154420 0x7f408815dde0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f40880705c0 tx=0x7f407801ecd0 comp rx=0 tx=0).stop 2024-03-19T02:01:32.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.647+0000 7f408dc61640 1 -- 172.21.15.55:0/3424335226 shutdown_connections 2024-03-19T02:01:32.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.647+0000 7f408dc61640 1 --2- 172.21.15.55:0/3424335226 >> v2:172.21.15.55:3300/0 conn(0x7f408815e320 0x7f40880728b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:32.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.647+0000 7f408dc61640 1 --2- 172.21.15.55:0/3424335226 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4088154420 0x7f408815dde0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:32.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.647+0000 7f408dc61640 1 -- 172.21.15.55:0/3424335226 >> 172.21.15.55:0/3424335226 conn(0x7f408806c3b0 msgr2=0x7f408806c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:32.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.647+0000 7f408dc61640 1 -- 172.21.15.55:0/3424335226 shutdown_connections 2024-03-19T02:01:32.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.647+0000 7f408dc61640 1 -- 172.21.15.55:0/3424335226 wait complete. 2024-03-19T02:01:32.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.647+0000 7f408dc61640 1 Processor -- start 2024-03-19T02:01:32.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.647+0000 7f408dc61640 1 -- start start 2024-03-19T02:01:32.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.647+0000 7f408dc61640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4088154420 0x7f4088159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:32.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.647+0000 7f408dc61640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f408815a1b0 0x7f408814ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:32.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.647+0000 7f408dc61640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f408815e320 0x7f408814a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:32.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.647+0000 7f408dc61640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4088154980 con 0x7f408815a1b0 2024-03-19T02:01:32.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.647+0000 7f408dc61640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4088154800 con 0x7f408815e320 2024-03-19T02:01:32.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.647+0000 7f408dc61640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4088070fb0 con 0x7f4088154420 2024-03-19T02:01:32.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.651+0000 7f40877fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4088154420 0x7f4088159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:32.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.651+0000 7f4087fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f408815e320 0x7f408814a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:32.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.651+0000 7f4087fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f408815e320 0x7f408814a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:40142/0 (socket says 172.21.15.55:40142) 2024-03-19T02:01:32.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.651+0000 7f4087fff640 1 -- 172.21.15.55:0/406902419 learned_addr learned my addr 172.21.15.55:0/406902419 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:32.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.655+0000 7f4087fff640 1 -- 172.21.15.55:0/406902419 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4088154420 msgr2=0x7f4088159400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:32.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.655+0000 7f4087fff640 1 --2- 172.21.15.55:0/406902419 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4088154420 0x7f4088159400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:32.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.655+0000 7f4087fff640 1 -- 172.21.15.55:0/406902419 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f408815a1b0 msgr2=0x7f408814ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:32.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.655+0000 7f4087fff640 1 --2- 172.21.15.55:0/406902419 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f408815a1b0 0x7f408814ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:32.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.655+0000 7f4087fff640 1 -- 172.21.15.55:0/406902419 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408814b7c0 con 0x7f408815e320 2024-03-19T02:01:32.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.663+0000 7f4087fff640 1 --2- 172.21.15.55:0/406902419 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f408815e320 0x7f408814a1d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f408000b970 tx=0x7f408000be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:32.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.671+0000 7f406ffff640 1 -- 172.21.15.55:0/406902419 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4080019070 con 0x7f408815e320 2024-03-19T02:01:32.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.671+0000 7f406ffff640 1 -- 172.21.15.55:0/406902419 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f408000cb60 con 0x7f408815e320 2024-03-19T02:01:32.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.671+0000 7f408dc61640 1 -- 172.21.15.55:0/406902419 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f408814c710 con 0x7f408815e320 2024-03-19T02:01:32.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.671+0000 7f406ffff640 1 -- 172.21.15.55:0/406902419 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4080014440 con 0x7f408815e320 2024-03-19T02:01:32.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.671+0000 7f408dc61640 1 -- 172.21.15.55:0/406902419 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f408814df00 con 0x7f408815e320 2024-03-19T02:01:32.670 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:32.671+0000 7f408dc61640 1 -- 172.21.15.55:0/406902419 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4088068a40 con 0x7f408815e320 2024-03-19T02:01:33.015 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:33.015+0000 7f406ffff640 1 -- 172.21.15.55:0/406902419 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f408001c070 con 0x7f408815e320 2024-03-19T02:01:33.016 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:33.019+0000 7f406ffff640 1 --2- 172.21.15.55:0/406902419 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4074067660 0x7f4074069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:33.016 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:33.019+0000 7f40877fe640 1 --2- 172.21.15.55:0/406902419 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4074067660 0x7f4074069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:33.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:33.019+0000 7f40877fe640 1 --2- 172.21.15.55:0/406902419 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4074067660 0x7f4074069b20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f40780b2000 tx=0x7f4078008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:33.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:33.023+0000 7f406ffff640 1 -- 172.21.15.55:0/406902419 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f4080054140 con 0x7f408815e320 2024-03-19T02:01:33.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:33.031+0000 7f406ffff640 1 -- 172.21.15.55:0/406902419 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4080023020 con 0x7f408815e320 2024-03-19T02:01:33.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:33.263+0000 7f408dc61640 1 -- 172.21.15.55:0/406902419 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f408815c0b0 con 0x7f408815e320 2024-03-19T02:01:33.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:33.295+0000 7f406ffff640 1 -- 172.21.15.55:0/406902419 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f40800560a0 con 0x7f408815e320 2024-03-19T02:01:33.292 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:33.292 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:33.293 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:33.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:33.295+0000 7f408dc61640 1 -- 172.21.15.55:0/406902419 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4074067660 msgr2=0x7f4074069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:33.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:33.295+0000 7f408dc61640 1 --2- 172.21.15.55:0/406902419 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4074067660 0x7f4074069b20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f40780b2000 tx=0x7f4078008040 comp rx=0 tx=0).stop 2024-03-19T02:01:33.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:33.295+0000 7f408dc61640 1 -- 172.21.15.55:0/406902419 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f408815e320 msgr2=0x7f408814a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:33.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:33.295+0000 7f408dc61640 1 --2- 172.21.15.55:0/406902419 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f408815e320 0x7f408814a1d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f408000b970 tx=0x7f408000be30 comp rx=0 tx=0).stop 2024-03-19T02:01:33.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:33.295+0000 7f408dc61640 1 -- 172.21.15.55:0/406902419 shutdown_connections 2024-03-19T02:01:33.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:33.295+0000 7f408dc61640 1 --2- 172.21.15.55:0/406902419 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4074067660 0x7f4074069b20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:33.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:33.295+0000 7f408dc61640 1 --2- 172.21.15.55:0/406902419 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f408815e320 0x7f408814a1d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:33.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:33.295+0000 7f408dc61640 1 --2- 172.21.15.55:0/406902419 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f408815a1b0 0x7f408814ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:33.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:33.295+0000 7f408dc61640 1 --2- 172.21.15.55:0/406902419 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4088154420 0x7f4088159400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:33.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:33.295+0000 7f408dc61640 1 -- 172.21.15.55:0/406902419 >> 172.21.15.55:0/406902419 conn(0x7f408806c3b0 msgr2=0x7f408815b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:33.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:33.295+0000 7f408dc61640 1 -- 172.21.15.55:0/406902419 shutdown_connections 2024-03-19T02:01:33.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:33.295+0000 7f408dc61640 1 -- 172.21.15.55:0/406902419 wait complete. 2024-03-19T02:01:33.915 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:33.915 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:34.917 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:35.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.047+0000 7fa443291640 1 Processor -- start 2024-03-19T02:01:35.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.047+0000 7fa443291640 1 -- start start 2024-03-19T02:01:35.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.051+0000 7fa443291640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa43c154340 0x7fa43c15e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:35.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.051+0000 7fa443291640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa43c0c73c0 0x7fa43c15aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:35.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.051+0000 7fa443291640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa43c071280 con 0x7fa43c154720 2024-03-19T02:01:35.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.051+0000 7fa443291640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fa43c073840 con 0x7fa43c154340 2024-03-19T02:01:35.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.051+0000 7fa443291640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa43c0c7850 con 0x7fa43c0c73c0 2024-03-19T02:01:35.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.051+0000 7fa44102d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa43c154340 0x7fa43c15e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:35.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.051+0000 7fa44082c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fa43c154720 0x7fa43c072e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:57706/0 (socket says 172.21.15.55:57706) 2024-03-19T02:01:35.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.051+0000 7fa44082c640 1 -- 172.21.15.55:0/288528669 learned_addr learned my addr 172.21.15.55:0/288528669 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:01:35.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.051+0000 7fa44182e640 1 --2- 172.21.15.55:0/288528669 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa43c0c73c0 0x7fa43c15aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:35.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.055+0000 7fa433fff640 1 -- 172.21.15.55:0/288528669 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2827010445 0 0) 0x7fa43c071280 con 0x7fa43c154720 2024-03-19T02:01:35.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.055+0000 7fa433fff640 1 -- 172.21.15.55:0/288528669 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa428003530 con 0x7fa43c154720 2024-03-19T02:01:35.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.055+0000 7fa44102d640 1 -- 172.21.15.55:0/288528669 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa43c0c73c0 msgr2=0x7fa43c15aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:35.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.055+0000 7fa44102d640 1 --2- 172.21.15.55:0/288528669 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa43c0c73c0 0x7fa43c15aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:35.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.055+0000 7fa44102d640 1 -- 172.21.15.55:0/288528669 >> v1:172.21.15.55:6789/0 conn(0x7fa43c154720 legacy=0x7fa43c072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:35.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.055+0000 7fa44102d640 1 -- 172.21.15.55:0/288528669 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa43c15d900 con 0x7fa43c154340 2024-03-19T02:01:35.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.059+0000 7fa44102d640 1 --2- 172.21.15.55:0/288528669 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa43c154340 0x7fa43c15e3a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa4380097f0 tx=0x7fa43801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7868c92f6d3f39fc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:35.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.063+0000 7fa433fff640 1 -- 172.21.15.55:0/288528669 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4380b0070 con 0x7fa43c154340 2024-03-19T02:01:35.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.063+0000 7fa433fff640 1 -- 172.21.15.55:0/288528669 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4380ab600 con 0x7fa43c154340 2024-03-19T02:01:35.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.063+0000 7fa433fff640 1 -- 172.21.15.55:0/288528669 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4380ab9d0 con 0x7fa43c154340 2024-03-19T02:01:35.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.063+0000 7fa443291640 1 -- 172.21.15.55:0/288528669 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa43c154340 msgr2=0x7fa43c15e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:35.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.063+0000 7fa443291640 1 --2- 172.21.15.55:0/288528669 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa43c154340 0x7fa43c15e3a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa4380097f0 tx=0x7fa43801c4a0 comp rx=0 tx=0).stop 2024-03-19T02:01:35.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.063+0000 7fa443291640 1 -- 172.21.15.55:0/288528669 shutdown_connections 2024-03-19T02:01:35.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.063+0000 7fa443291640 1 --2- 172.21.15.55:0/288528669 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa43c0c73c0 0x7fa43c15aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:35.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.063+0000 7fa443291640 1 --2- 172.21.15.55:0/288528669 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa43c154340 0x7fa43c15e3a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:35.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.063+0000 7fa443291640 1 -- 172.21.15.55:0/288528669 >> 172.21.15.55:0/288528669 conn(0x7fa43c0cc9c0 msgr2=0x7fa43c0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:35.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.063+0000 7fa443291640 1 -- 172.21.15.55:0/288528669 shutdown_connections 2024-03-19T02:01:35.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.063+0000 7fa443291640 1 -- 172.21.15.55:0/288528669 wait complete. 2024-03-19T02:01:35.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.067+0000 7fa443291640 1 Processor -- start 2024-03-19T02:01:35.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.067+0000 7fa443291640 1 -- start start 2024-03-19T02:01:35.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.067+0000 7fa443291640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa43c0c73c0 0x7fa43c152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:35.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.067+0000 7fa443291640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa43c154340 0x7fa43c154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:35.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.067+0000 7fa443291640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa43c154720 0x7fa43c1554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:35.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.067+0000 7fa443291640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa43c166b70 con 0x7fa43c154720 2024-03-19T02:01:35.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.067+0000 7fa443291640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa43c15b1e0 con 0x7fa43c0c73c0 2024-03-19T02:01:35.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.067+0000 7fa443291640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa43c166cf0 con 0x7fa43c154340 2024-03-19T02:01:35.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.067+0000 7fa44182e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa43c154720 0x7fa43c1554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:35.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.067+0000 7fa44102d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa43c0c73c0 0x7fa43c152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:35.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.067+0000 7fa44082c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa43c154340 0x7fa43c154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:35.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.067+0000 7fa44102d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa43c0c73c0 0x7fa43c152920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:40152/0 (socket says 172.21.15.55:40152) 2024-03-19T02:01:35.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.067+0000 7fa44102d640 1 -- 172.21.15.55:0/2930398106 learned_addr learned my addr 172.21.15.55:0/2930398106 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:35.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.071+0000 7fa44082c640 1 -- 172.21.15.55:0/2930398106 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa43c0c73c0 msgr2=0x7fa43c152920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:35.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.071+0000 7fa44082c640 1 --2- 172.21.15.55:0/2930398106 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa43c0c73c0 0x7fa43c152920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:35.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.071+0000 7fa44082c640 1 -- 172.21.15.55:0/2930398106 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa43c154720 msgr2=0x7fa43c1554b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:35.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.071+0000 7fa44082c640 1 --2- 172.21.15.55:0/2930398106 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa43c154720 0x7fa43c1554b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:35.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.071+0000 7fa44082c640 1 -- 172.21.15.55:0/2930398106 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa43c14f490 con 0x7fa43c154340 2024-03-19T02:01:35.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.075+0000 7fa44082c640 1 --2- 172.21.15.55:0/2930398106 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa43c154340 0x7fa43c154f70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa42c00c400 tx=0x7fa42c00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:35.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.079+0000 7fa431ffb640 1 -- 172.21.15.55:0/2930398106 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42c017070 con 0x7fa43c154340 2024-03-19T02:01:35.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.079+0000 7fa431ffb640 1 -- 172.21.15.55:0/2930398106 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa42c005cf0 con 0x7fa43c154340 2024-03-19T02:01:35.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.079+0000 7fa443291640 1 -- 172.21.15.55:0/2930398106 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa43c1533f0 con 0x7fa43c154340 2024-03-19T02:01:35.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.079+0000 7fa431ffb640 1 -- 172.21.15.55:0/2930398106 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42c0072f0 con 0x7fa43c154340 2024-03-19T02:01:35.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.079+0000 7fa443291640 1 -- 172.21.15.55:0/2930398106 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa43c1f4040 con 0x7fa43c154340 2024-03-19T02:01:35.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.079+0000 7fa443291640 1 -- 172.21.15.55:0/2930398106 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa43c068a40 con 0x7fa43c154340 2024-03-19T02:01:35.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.355+0000 7fa431ffb640 1 -- 172.21.15.55:0/2930398106 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa42c01a070 con 0x7fa43c154340 2024-03-19T02:01:35.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.355+0000 7fa431ffb640 1 --2- 172.21.15.55:0/2930398106 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa4200676b0 0x7fa420069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:35.354 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.355+0000 7fa44102d640 1 --2- 172.21.15.55:0/2930398106 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa4200676b0 0x7fa420069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:35.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.355+0000 7fa44102d640 1 --2- 172.21.15.55:0/2930398106 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa4200676b0 0x7fa420069b70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa43801ab50 tx=0x7fa4380a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:35.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.359+0000 7fa431ffb640 1 -- 172.21.15.55:0/2930398106 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fa42c0536d0 con 0x7fa43c154340 2024-03-19T02:01:35.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.375+0000 7fa431ffb640 1 -- 172.21.15.55:0/2930398106 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa42c021020 con 0x7fa43c154340 2024-03-19T02:01:35.591 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.591+0000 7fa443291640 1 -- 172.21.15.55:0/2930398106 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa43c0ca2a0 con 0x7fa43c154340 2024-03-19T02:01:35.623 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.623+0000 7fa431ffb640 1 -- 172.21.15.55:0/2930398106 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fa42c055630 con 0x7fa43c154340 2024-03-19T02:01:35.623 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:35.624 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:35.624 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:35.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.627+0000 7fa443291640 1 -- 172.21.15.55:0/2930398106 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa4200676b0 msgr2=0x7fa420069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:35.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.627+0000 7fa443291640 1 --2- 172.21.15.55:0/2930398106 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa4200676b0 0x7fa420069b70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa43801ab50 tx=0x7fa4380a6000 comp rx=0 tx=0).stop 2024-03-19T02:01:35.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.627+0000 7fa443291640 1 -- 172.21.15.55:0/2930398106 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa43c154340 msgr2=0x7fa43c154f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:35.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.627+0000 7fa443291640 1 --2- 172.21.15.55:0/2930398106 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa43c154340 0x7fa43c154f70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa42c00c400 tx=0x7fa42c00c8c0 comp rx=0 tx=0).stop 2024-03-19T02:01:35.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.627+0000 7fa443291640 1 -- 172.21.15.55:0/2930398106 shutdown_connections 2024-03-19T02:01:35.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.627+0000 7fa443291640 1 --2- 172.21.15.55:0/2930398106 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa4200676b0 0x7fa420069b70 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:35.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.627+0000 7fa443291640 1 --2- 172.21.15.55:0/2930398106 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa43c154720 0x7fa43c1554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:35.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.627+0000 7fa443291640 1 --2- 172.21.15.55:0/2930398106 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa43c154340 0x7fa43c154f70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:35.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.627+0000 7fa443291640 1 --2- 172.21.15.55:0/2930398106 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa43c0c73c0 0x7fa43c152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:35.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.627+0000 7fa443291640 1 -- 172.21.15.55:0/2930398106 >> 172.21.15.55:0/2930398106 conn(0x7fa43c0cc9c0 msgr2=0x7fa43c0cbbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:35.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.627+0000 7fa443291640 1 -- 172.21.15.55:0/2930398106 shutdown_connections 2024-03-19T02:01:35.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:35.627+0000 7fa443291640 1 -- 172.21.15.55:0/2930398106 wait complete. 2024-03-19T02:01:36.248 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:36.248 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:37.250 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:37.365 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.367+0000 7f4d05231640 1 Processor -- start 2024-03-19T02:01:37.365 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.367+0000 7f4d05231640 1 -- start start 2024-03-19T02:01:37.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.367+0000 7f4d05231640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4d000c96b0 0x7f4d00070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:37.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.367+0000 7f4d05231640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f4d001540c0 0x7f4d00154570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:37.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.367+0000 7f4d05231640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d0015e3a0 con 0x7f4d000c9a90 2024-03-19T02:01:37.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.367+0000 7f4d05231640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f4d000c7ad0 con 0x7f4d001540c0 2024-03-19T02:01:37.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.367+0000 7f4d05231640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4d0015e680 con 0x7f4d000c96b0 2024-03-19T02:01:37.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.367+0000 7f4cfed9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4d000c96b0 0x7f4d00070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:37.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.371+0000 7f4cfed9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4d000c96b0 0x7f4d00070f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:39140/0 (socket says 172.21.15.55:39140) 2024-03-19T02:01:37.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.371+0000 7f4cfed9d640 1 -- 172.21.15.55:0/2426889338 learned_addr learned my addr 172.21.15.55:0/2426889338 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:37.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.371+0000 7f4cff59e640 1 --2- 172.21.15.55:0/2426889338 >> v2:172.21.15.84:3300/0 conn(0x7f4d001540c0 0x7f4d00154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:37.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.371+0000 7f4cfed9d640 1 -- 172.21.15.55:0/2426889338 >> v2:172.21.15.84:3300/0 conn(0x7f4d001540c0 msgr2=0x7f4d00154570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:37.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.371+0000 7f4cfed9d640 1 --2- 172.21.15.55:0/2426889338 >> v2:172.21.15.84:3300/0 conn(0x7f4d001540c0 0x7f4d00154570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:37.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.371+0000 7f4cfed9d640 1 -- 172.21.15.55:0/2426889338 >> v1:172.21.15.84:6789/0 conn(0x7f4d000c9a90 legacy=0x7f4d000c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:37.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.371+0000 7f4cfed9d640 1 -- 172.21.15.55:0/2426889338 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d0015a330 con 0x7f4d000c96b0 2024-03-19T02:01:37.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.375+0000 7f4cfed9d640 1 --2- 172.21.15.55:0/2426889338 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4d000c96b0 0x7f4d00070f20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0009870 tx=0x7f4cf001c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12469dd66b2eb7e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:37.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.379+0000 7f4cfdd9b640 1 -- 172.21.15.55:0/2426889338 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf00b1070 con 0x7f4d000c96b0 2024-03-19T02:01:37.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.383+0000 7f4cfdd9b640 1 -- 172.21.15.55:0/2426889338 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cf00ac620 con 0x7f4d000c96b0 2024-03-19T02:01:37.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.383+0000 7f4cfdd9b640 1 -- 172.21.15.55:0/2426889338 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf00ac9f0 con 0x7f4d000c96b0 2024-03-19T02:01:37.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.383+0000 7f4d05231640 1 -- 172.21.15.55:0/2426889338 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4d000c96b0 msgr2=0x7f4d00070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:37.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.383+0000 7f4d05231640 1 --2- 172.21.15.55:0/2426889338 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4d000c96b0 0x7f4d00070f20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0009870 tx=0x7f4cf001c4d0 comp rx=0 tx=0).stop 2024-03-19T02:01:37.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.383+0000 7f4d05231640 1 -- 172.21.15.55:0/2426889338 shutdown_connections 2024-03-19T02:01:37.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.383+0000 7f4d05231640 1 --2- 172.21.15.55:0/2426889338 >> v2:172.21.15.84:3300/0 conn(0x7f4d001540c0 0x7f4d00154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:37.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.383+0000 7f4d05231640 1 --2- 172.21.15.55:0/2426889338 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4d000c96b0 0x7f4d00070f20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:37.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.383+0000 7f4d05231640 1 -- 172.21.15.55:0/2426889338 >> 172.21.15.55:0/2426889338 conn(0x7f4d000cc9c0 msgr2=0x7f4d000ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:37.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.383+0000 7f4d05231640 1 -- 172.21.15.55:0/2426889338 shutdown_connections 2024-03-19T02:01:37.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.383+0000 7f4d05231640 1 -- 172.21.15.55:0/2426889338 wait complete. 2024-03-19T02:01:37.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.383+0000 7f4d05231640 1 Processor -- start 2024-03-19T02:01:37.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.383+0000 7f4d05231640 1 -- start start 2024-03-19T02:01:37.384 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.387+0000 7f4d05231640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4d000c96b0 0x7f4d00154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:37.384 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.387+0000 7f4d05231640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4d000c9a90 0x7f4d001456a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:37.384 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.387+0000 7f4d05231640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4d001540c0 0x7f4d00143180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:37.384 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.387+0000 7f4d05231640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4d00166b90 con 0x7f4d001540c0 2024-03-19T02:01:37.384 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.387+0000 7f4d05231640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4d0015b0f0 con 0x7f4d000c9a90 2024-03-19T02:01:37.384 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.387+0000 7f4d05231640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4d00166d10 con 0x7f4d000c96b0 2024-03-19T02:01:37.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.387+0000 7f4cfed9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4d000c96b0 0x7f4d00154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:37.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.387+0000 7f4cff59e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4d001540c0 0x7f4d00143180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:37.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.387+0000 7f4cfed9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4d000c96b0 0x7f4d00154b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:39146/0 (socket says 172.21.15.55:39146) 2024-03-19T02:01:37.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.387+0000 7f4cfed9d640 1 -- 172.21.15.55:0/3516609625 learned_addr learned my addr 172.21.15.55:0/3516609625 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:37.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.387+0000 7f4cfe59c640 1 --2- 172.21.15.55:0/3516609625 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4d000c9a90 0x7f4d001456a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:37.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.391+0000 7f4cff59e640 1 -- 172.21.15.55:0/3516609625 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4d000c96b0 msgr2=0x7f4d00154b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:37.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.391+0000 7f4cff59e640 1 --2- 172.21.15.55:0/3516609625 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4d000c96b0 0x7f4d00154b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:37.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.391+0000 7f4cff59e640 1 -- 172.21.15.55:0/3516609625 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4d000c9a90 msgr2=0x7f4d001456a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:37.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.391+0000 7f4cff59e640 1 --2- 172.21.15.55:0/3516609625 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4d000c9a90 0x7f4d001456a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:37.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.391+0000 7f4cff59e640 1 -- 172.21.15.55:0/3516609625 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d001450f0 con 0x7f4d001540c0 2024-03-19T02:01:37.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.391+0000 7f4cff59e640 1 --2- 172.21.15.55:0/3516609625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4d001540c0 0x7f4d00143180 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8007d20 tx=0x7f4cf800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:37.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.399+0000 7f4ce77fe640 1 -- 172.21.15.55:0/3516609625 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf800edd0 con 0x7f4d001540c0 2024-03-19T02:01:37.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.399+0000 7f4ce77fe640 1 -- 172.21.15.55:0/3516609625 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cf80039a0 con 0x7f4d001540c0 2024-03-19T02:01:37.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.399+0000 7f4ce77fe640 1 -- 172.21.15.55:0/3516609625 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf8003d90 con 0x7f4d001540c0 2024-03-19T02:01:37.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.399+0000 7f4d05231640 1 -- 172.21.15.55:0/3516609625 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d00145390 con 0x7f4d001540c0 2024-03-19T02:01:37.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.399+0000 7f4d05231640 1 -- 172.21.15.55:0/3516609625 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d001f6f60 con 0x7f4d001540c0 2024-03-19T02:01:37.397 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.399+0000 7f4d05231640 1 -- 172.21.15.55:0/3516609625 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d00068a40 con 0x7f4d001540c0 2024-03-19T02:01:37.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.431+0000 7f4ce77fe640 1 -- 172.21.15.55:0/3516609625 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f4cf801b020 con 0x7f4d001540c0 2024-03-19T02:01:37.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.435+0000 7f4ce77fe640 1 --2- 172.21.15.55:0/3516609625 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4cec0676b0 0x7f4cec069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:37.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.435+0000 7f4cfed9d640 1 --2- 172.21.15.55:0/3516609625 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4cec0676b0 0x7f4cec069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:37.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.435+0000 7f4cfed9d640 1 --2- 172.21.15.55:0/3516609625 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4cec0676b0 0x7f4cec069b70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0009870 tx=0x7f4cf00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:37.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.711+0000 7f4ce77fe640 1 -- 172.21.15.55:0/3516609625 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f4cf8054f30 con 0x7f4d001540c0 2024-03-19T02:01:37.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.711+0000 7f4ce77fe640 1 -- 172.21.15.55:0/3516609625 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4cf8056e90 con 0x7f4d001540c0 2024-03-19T02:01:37.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.907+0000 7f4d05231640 1 -- 172.21.15.55:0/3516609625 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d00144ca0 con 0x7f4d001540c0 2024-03-19T02:01:37.928 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.931+0000 7f4ce77fe640 1 -- 172.21.15.55:0/3516609625 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f4cf805af30 con 0x7f4d001540c0 2024-03-19T02:01:37.928 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:37.928 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:37.929 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:37.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.931+0000 7f4d05231640 1 -- 172.21.15.55:0/3516609625 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4cec0676b0 msgr2=0x7f4cec069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:37.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.931+0000 7f4d05231640 1 --2- 172.21.15.55:0/3516609625 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4cec0676b0 0x7f4cec069b70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0009870 tx=0x7f4cf00a7000 comp rx=0 tx=0).stop 2024-03-19T02:01:37.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.931+0000 7f4d05231640 1 -- 172.21.15.55:0/3516609625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4d001540c0 msgr2=0x7f4d00143180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:37.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.931+0000 7f4d05231640 1 --2- 172.21.15.55:0/3516609625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4d001540c0 0x7f4d00143180 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8007d20 tx=0x7f4cf800e4b0 comp rx=0 tx=0).stop 2024-03-19T02:01:37.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.931+0000 7f4d05231640 1 -- 172.21.15.55:0/3516609625 shutdown_connections 2024-03-19T02:01:37.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.931+0000 7f4d05231640 1 --2- 172.21.15.55:0/3516609625 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4cec0676b0 0x7f4cec069b70 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:37.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.931+0000 7f4d05231640 1 --2- 172.21.15.55:0/3516609625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4d001540c0 0x7f4d00143180 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:37.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.931+0000 7f4d05231640 1 --2- 172.21.15.55:0/3516609625 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4d000c9a90 0x7f4d001456a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:37.931 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.931+0000 7f4d05231640 1 --2- 172.21.15.55:0/3516609625 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4d000c96b0 0x7f4d00154b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:37.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.931+0000 7f4d05231640 1 -- 172.21.15.55:0/3516609625 >> 172.21.15.55:0/3516609625 conn(0x7f4d000cc9c0 msgr2=0x7f4d00070a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:37.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.931+0000 7f4d05231640 1 -- 172.21.15.55:0/3516609625 shutdown_connections 2024-03-19T02:01:37.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:37.931+0000 7f4d05231640 1 -- 172.21.15.55:0/3516609625 wait complete. 2024-03-19T02:01:38.552 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:38.552 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:39.553 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:39.681 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.683+0000 7f17d7ee4640 1 Processor -- start 2024-03-19T02:01:39.681 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.683+0000 7f17d7ee4640 1 -- start start 2024-03-19T02:01:39.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.683+0000 7f17d7ee4640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f17d0154340 0x7f17d015e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:39.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.683+0000 7f17d7ee4640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17d00c73c0 0x7f17d015aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:39.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.683+0000 7f17d7ee4640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17d0071280 con 0x7f17d0154720 2024-03-19T02:01:39.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.683+0000 7f17d7ee4640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f17d0073840 con 0x7f17d0154340 2024-03-19T02:01:39.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.683+0000 7f17d7ee4640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f17d00c7850 con 0x7f17d00c73c0 2024-03-19T02:01:39.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.687+0000 7f17d547f640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f17d0154720 0x7f17d0072e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:32792/0 (socket says 172.21.15.55:32792) 2024-03-19T02:01:39.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.687+0000 7f17d547f640 1 -- 172.21.15.55:0/1633432370 learned_addr learned my addr 172.21.15.55:0/1633432370 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:01:39.684 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.687+0000 7f17d6481640 1 --2- 172.21.15.55:0/1633432370 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17d00c73c0 0x7f17d015aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:39.684 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.687+0000 7f17d5c80640 1 --2- 172.21.15.55:0/1633432370 >> v2:172.21.15.84:3300/0 conn(0x7f17d0154340 0x7f17d015e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:39.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.687+0000 7f17d4c7e640 1 -- 172.21.15.55:0/1633432370 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1691172122 0 0) 0x7f17d0071280 con 0x7f17d0154720 2024-03-19T02:01:39.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.687+0000 7f17d4c7e640 1 -- 172.21.15.55:0/1633432370 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17bc003530 con 0x7f17d0154720 2024-03-19T02:01:39.691 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.691+0000 7f17d4c7e640 1 -- 172.21.15.55:0/1633432370 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1958114774 0 0) 0x7f17bc003530 con 0x7f17d0154720 2024-03-19T02:01:39.692 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.691+0000 7f17d4c7e640 1 -- 172.21.15.55:0/1633432370 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17d00c73c0 msgr2=0x7f17d015aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:39.692 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.691+0000 7f17d4c7e640 1 --2- 172.21.15.55:0/1633432370 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17d00c73c0 0x7f17d015aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:39.692 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.691+0000 7f17d4c7e640 1 -- 172.21.15.55:0/1633432370 >> v2:172.21.15.84:3300/0 conn(0x7f17d0154340 msgr2=0x7f17d015e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:39.692 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.691+0000 7f17d4c7e640 1 --2- 172.21.15.55:0/1633432370 >> v2:172.21.15.84:3300/0 conn(0x7f17d0154340 0x7f17d015e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:39.692 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.691+0000 7f17d4c7e640 1 -- 172.21.15.55:0/1633432370 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17d015d900 con 0x7f17d0154720 2024-03-19T02:01:39.692 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.691+0000 7f17d4c7e640 1 -- 172.21.15.55:0/1633432370 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f17c0001680 con 0x7f17d0154720 2024-03-19T02:01:39.692 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.691+0000 7f17d4c7e640 1 -- 172.21.15.55:0/1633432370 >> v1:172.21.15.55:6789/0 conn(0x7f17d0154720 legacy=0x7f17d0072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:39.692 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.691+0000 7f17d4c7e640 1 --2- 172.21.15.55:0/1633432370 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17d00c73c0 0x7f17bc0039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:39.692 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.691+0000 7f17d4c7e640 1 --2- 172.21.15.55:0/1633432370 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f17bc003fc0 0x7f17bc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:39.692 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.691+0000 7f17d4c7e640 1 --2- 172.21.15.55:0/1633432370 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17bc004d50 0x7f17bc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:39.692 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.691+0000 7f17d4c7e640 1 -- 172.21.15.55:0/1633432370 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc00b9d0 con 0x7f17bc004d50 2024-03-19T02:01:39.692 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.691+0000 7f17d4c7e640 1 -- 172.21.15.55:0/1633432370 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc00bb50 con 0x7f17bc003fc0 2024-03-19T02:01:39.692 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.691+0000 7f17d4c7e640 1 -- 172.21.15.55:0/1633432370 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f17bc00bd00 con 0x7f17d00c73c0 2024-03-19T02:01:39.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.695+0000 7f17d6481640 1 --2- 172.21.15.55:0/1633432370 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17bc004d50 0x7f17bc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:39.696 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.699+0000 7f17d5c80640 1 --2- 172.21.15.55:0/1633432370 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17d00c73c0 0x7f17bc0039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:39.697 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.699+0000 7f17d6481640 1 -- 172.21.15.55:0/1633432370 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17d00c73c0 msgr2=0x7f17bc0039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:39.697 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.699+0000 7f17d6481640 1 --2- 172.21.15.55:0/1633432370 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17d00c73c0 0x7f17bc0039f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:39.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.699+0000 7f17d6481640 1 -- 172.21.15.55:0/1633432370 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f17bc003fc0 msgr2=0x7f17bc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:39.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.699+0000 7f17d6481640 1 --2- 172.21.15.55:0/1633432370 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f17bc003fc0 0x7f17bc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:39.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.699+0000 7f17d6481640 1 -- 172.21.15.55:0/1633432370 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17bc003530 con 0x7f17bc004d50 2024-03-19T02:01:39.701 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.703+0000 7f17d6481640 1 --2- 172.21.15.55:0/1633432370 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17bc004d50 0x7f17bc0051d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f17c8009fc0 tx=0x7f17c801cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=273ec16e752020ff server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:39.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.707+0000 7f17d4c7e640 1 -- 172.21.15.55:0/1633432370 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c80b2070 con 0x7f17bc004d50 2024-03-19T02:01:39.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.707+0000 7f17d4c7e640 1 -- 172.21.15.55:0/1633432370 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17c80a5c90 con 0x7f17bc004d50 2024-03-19T02:01:39.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.707+0000 7f17d4c7e640 1 -- 172.21.15.55:0/1633432370 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c8009c10 con 0x7f17bc004d50 2024-03-19T02:01:39.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.707+0000 7f17d7ee4640 1 -- 172.21.15.55:0/1633432370 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17bc004d50 msgr2=0x7f17bc0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:39.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.707+0000 7f17d7ee4640 1 --2- 172.21.15.55:0/1633432370 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17bc004d50 0x7f17bc0051d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f17c8009fc0 tx=0x7f17c801cc20 comp rx=0 tx=0).stop 2024-03-19T02:01:39.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.707+0000 7f17d6481640 1 -- 172.21.15.55:0/1633432370 reap_dead start 2024-03-19T02:01:39.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.707+0000 7f17d7ee4640 1 -- 172.21.15.55:0/1633432370 shutdown_connections 2024-03-19T02:01:39.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.707+0000 7f17d7ee4640 1 -- 172.21.15.55:0/1633432370 >> 172.21.15.55:0/1633432370 conn(0x7f17d00cc9c0 msgr2=0x7f17d00ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:39.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.707+0000 7f17d7ee4640 1 -- 172.21.15.55:0/1633432370 shutdown_connections 2024-03-19T02:01:39.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.707+0000 7f17d7ee4640 1 -- 172.21.15.55:0/1633432370 wait complete. 2024-03-19T02:01:39.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.711+0000 7f17d7ee4640 1 Processor -- start 2024-03-19T02:01:39.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.711+0000 7f17d7ee4640 1 -- start start 2024-03-19T02:01:39.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.711+0000 7f17d7ee4640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17d006c610 0x7f17d01efbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:39.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.711+0000 7f17d7ee4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17d0157970 0x7f17d00c8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:39.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.711+0000 7f17d7ee4640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f17d006c9f0 0x7f17d01f3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:39.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.711+0000 7f17d7ee4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f17d0070fa0 con 0x7f17d0157970 2024-03-19T02:01:39.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.711+0000 7f17d7ee4640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f17d015df10 con 0x7f17d006c9f0 2024-03-19T02:01:39.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.711+0000 7f17d7ee4640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f17d015dba0 con 0x7f17d006c610 2024-03-19T02:01:39.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.711+0000 7f17d547f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17d0157970 0x7f17d00c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:39.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.711+0000 7f17d547f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17d0157970 0x7f17d00c8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:46060/0 (socket says 172.21.15.55:46060) 2024-03-19T02:01:39.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.711+0000 7f17d547f640 1 -- 172.21.15.55:0/3568244881 learned_addr learned my addr 172.21.15.55:0/3568244881 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:39.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.711+0000 7f17d5c80640 1 --2- 172.21.15.55:0/3568244881 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17d006c610 0x7f17d01efbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:39.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.711+0000 7f17d6481640 1 --2- 172.21.15.55:0/3568244881 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f17d006c9f0 0x7f17d01f3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:39.714 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.715+0000 7f17d547f640 1 -- 172.21.15.55:0/3568244881 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17d006c610 msgr2=0x7f17d01efbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:39.714 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.715+0000 7f17d547f640 1 --2- 172.21.15.55:0/3568244881 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17d006c610 0x7f17d01efbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:39.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.715+0000 7f17d547f640 1 -- 172.21.15.55:0/3568244881 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f17d006c9f0 msgr2=0x7f17d01f3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:39.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.715+0000 7f17d547f640 1 --2- 172.21.15.55:0/3568244881 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f17d006c9f0 0x7f17d01f3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:39.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.715+0000 7f17d547f640 1 -- 172.21.15.55:0/3568244881 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17d01f0400 con 0x7f17d0157970 2024-03-19T02:01:39.717 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.719+0000 7f17d547f640 1 --2- 172.21.15.55:0/3568244881 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17d0157970 0x7f17d00c8220 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f17c000c6e0 tx=0x7f17c000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:39.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.723+0000 7f17c67fc640 1 -- 172.21.15.55:0/3568244881 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c0018070 con 0x7f17d0157970 2024-03-19T02:01:39.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.723+0000 7f17c67fc640 1 -- 172.21.15.55:0/3568244881 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17c0004ad0 con 0x7f17d0157970 2024-03-19T02:01:39.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.727+0000 7f17d7ee4640 1 -- 172.21.15.55:0/3568244881 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17d01f06e0 con 0x7f17d0157970 2024-03-19T02:01:39.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.727+0000 7f17c67fc640 1 -- 172.21.15.55:0/3568244881 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c00070d0 con 0x7f17d0157970 2024-03-19T02:01:39.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.727+0000 7f17d7ee4640 1 -- 172.21.15.55:0/3568244881 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17d00c7a20 con 0x7f17d0157970 2024-03-19T02:01:39.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.727+0000 7f17d7ee4640 1 -- 172.21.15.55:0/3568244881 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17d0068a40 con 0x7f17d0157970 2024-03-19T02:01:39.760 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.763+0000 7f17c67fc640 1 -- 172.21.15.55:0/3568244881 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f17c001b070 con 0x7f17d0157970 2024-03-19T02:01:39.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.763+0000 7f17c67fc640 1 --2- 172.21.15.55:0/3568244881 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f17b40676e0 0x7f17b4069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:39.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.763+0000 7f17d5c80640 1 --2- 172.21.15.55:0/3568244881 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f17b40676e0 0x7f17b4069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:39.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.763+0000 7f17d5c80640 1 --2- 172.21.15.55:0/3568244881 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f17b40676e0 0x7f17b4069ba0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009b60 tx=0x7f17cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:39.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:39.767+0000 7f17c67fc640 1 -- 172.21.15.55:0/3568244881 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f17c0053960 con 0x7f17d0157970 2024-03-19T02:01:40.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:40.055+0000 7f17c67fc640 1 -- 172.21.15.55:0/3568244881 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f17c0022020 con 0x7f17d0157970 2024-03-19T02:01:40.262 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:40.263+0000 7f17d7ee4640 1 -- 172.21.15.55:0/3568244881 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17d00cc300 con 0x7f17d0157970 2024-03-19T02:01:40.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:40.307+0000 7f17c67fc640 1 -- 172.21.15.55:0/3568244881 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f17c00558c0 con 0x7f17d0157970 2024-03-19T02:01:40.307 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:40.307 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:40.307 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:40.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:40.311+0000 7f17d7ee4640 1 -- 172.21.15.55:0/3568244881 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f17b40676e0 msgr2=0x7f17b4069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:40.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:40.311+0000 7f17d7ee4640 1 --2- 172.21.15.55:0/3568244881 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f17b40676e0 0x7f17b4069ba0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009b60 tx=0x7f17cc008040 comp rx=0 tx=0).stop 2024-03-19T02:01:40.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:40.311+0000 7f17d7ee4640 1 -- 172.21.15.55:0/3568244881 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17d0157970 msgr2=0x7f17d00c8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:40.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:40.311+0000 7f17d7ee4640 1 --2- 172.21.15.55:0/3568244881 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17d0157970 0x7f17d00c8220 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f17c000c6e0 tx=0x7f17c000cba0 comp rx=0 tx=0).stop 2024-03-19T02:01:40.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:40.311+0000 7f17d7ee4640 1 -- 172.21.15.55:0/3568244881 shutdown_connections 2024-03-19T02:01:40.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:40.311+0000 7f17d7ee4640 1 --2- 172.21.15.55:0/3568244881 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f17b40676e0 0x7f17b4069ba0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:40.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:40.311+0000 7f17d7ee4640 1 --2- 172.21.15.55:0/3568244881 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f17d006c9f0 0x7f17d01f3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:40.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:40.311+0000 7f17d7ee4640 1 --2- 172.21.15.55:0/3568244881 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17d0157970 0x7f17d00c8220 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:40.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:40.311+0000 7f17d7ee4640 1 --2- 172.21.15.55:0/3568244881 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17d006c610 0x7f17d01efbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:40.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:40.311+0000 7f17d7ee4640 1 -- 172.21.15.55:0/3568244881 >> 172.21.15.55:0/3568244881 conn(0x7f17d00cc9c0 msgr2=0x7f17d00ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:40.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:40.311+0000 7f17d7ee4640 1 -- 172.21.15.55:0/3568244881 shutdown_connections 2024-03-19T02:01:40.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:40.311+0000 7f17d7ee4640 1 -- 172.21.15.55:0/3568244881 wait complete. 2024-03-19T02:01:40.340 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:01:40.342 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:01:40.937 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:40.937 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:41.938 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:42.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.055+0000 7f530376f640 1 Processor -- start 2024-03-19T02:01:42.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.055+0000 7f530376f640 1 -- start start 2024-03-19T02:01:42.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.055+0000 7f530376f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f52fc14eaa0 0x7f52fc06b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:42.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.055+0000 7f530376f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f52fc06bec0 0x7f52fc0c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:42.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.055+0000 7f530376f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52fc14ee80 con 0x7f52fc0c6610 2024-03-19T02:01:42.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.055+0000 7f530376f640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f52fc0c7540 con 0x7f52fc14eaa0 2024-03-19T02:01:42.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.055+0000 7f530376f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f52fc0c77c0 con 0x7f52fc06bec0 2024-03-19T02:01:42.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.055+0000 7f5301d0c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f52fc0c6610 0x7f52fc1585f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:32794/0 (socket says 172.21.15.55:32794) 2024-03-19T02:01:42.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.055+0000 7f5301d0c640 1 -- 172.21.15.55:0/883881552 learned_addr learned my addr 172.21.15.55:0/883881552 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:01:42.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.055+0000 7f530150b640 1 --2- 172.21.15.55:0/883881552 >> v2:172.21.15.84:3300/0 conn(0x7f52fc14eaa0 0x7f52fc06b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:42.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.059+0000 7f5300d0a640 1 --2- 172.21.15.55:0/883881552 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f52fc06bec0 0x7f52fc0c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:42.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.059+0000 7f530150b640 1 -- 172.21.15.55:0/883881552 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f52fc06bec0 msgr2=0x7f52fc0c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:42.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.059+0000 7f530150b640 1 --2- 172.21.15.55:0/883881552 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f52fc06bec0 0x7f52fc0c60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:42.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.059+0000 7f530150b640 1 -- 172.21.15.55:0/883881552 >> v1:172.21.15.55:6789/0 conn(0x7f52fc0c6610 legacy=0x7f52fc1585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:42.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.059+0000 7f530150b640 1 -- 172.21.15.55:0/883881552 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52fc0c7aa0 con 0x7f52fc14eaa0 2024-03-19T02:01:42.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.067+0000 7f530150b640 1 --2- 172.21.15.55:0/883881552 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f52fc14eaa0 0x7f52fc06b980 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f52f80097c0 tx=0x7f52f801c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e02f3cb579bbe369 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:42.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.071+0000 7f52f3fff640 1 -- 172.21.15.55:0/883881552 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f80b0070 con 0x7f52fc14eaa0 2024-03-19T02:01:42.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.075+0000 7f52f3fff640 1 -- 172.21.15.55:0/883881552 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52f80ab600 con 0x7f52fc14eaa0 2024-03-19T02:01:42.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.075+0000 7f52f3fff640 1 -- 172.21.15.55:0/883881552 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f80ab9d0 con 0x7f52fc14eaa0 2024-03-19T02:01:42.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.075+0000 7f530376f640 1 -- 172.21.15.55:0/883881552 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f52fc14eaa0 msgr2=0x7f52fc06b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:42.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.075+0000 7f530376f640 1 --2- 172.21.15.55:0/883881552 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f52fc14eaa0 0x7f52fc06b980 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f52f80097c0 tx=0x7f52f801c410 comp rx=0 tx=0).stop 2024-03-19T02:01:42.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.075+0000 7f530376f640 1 -- 172.21.15.55:0/883881552 shutdown_connections 2024-03-19T02:01:42.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.075+0000 7f530376f640 1 --2- 172.21.15.55:0/883881552 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f52fc06bec0 0x7f52fc0c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:42.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.075+0000 7f530376f640 1 --2- 172.21.15.55:0/883881552 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f52fc14eaa0 0x7f52fc06b980 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:42.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.075+0000 7f530376f640 1 -- 172.21.15.55:0/883881552 >> 172.21.15.55:0/883881552 conn(0x7f52fc0cc9c0 msgr2=0x7f52fc0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:42.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.075+0000 7f530376f640 1 -- 172.21.15.55:0/883881552 shutdown_connections 2024-03-19T02:01:42.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.075+0000 7f530376f640 1 -- 172.21.15.55:0/883881552 wait complete. 2024-03-19T02:01:42.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.075+0000 7f530376f640 1 Processor -- start 2024-03-19T02:01:42.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.075+0000 7f530376f640 1 -- start start 2024-03-19T02:01:42.075 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.079+0000 7f530376f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52fc06bec0 0x7f52fc14be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:42.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.079+0000 7f530376f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f52fc0c6610 0x7f52fc14daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:42.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.079+0000 7f530376f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f52fc14eaa0 0x7f52fc1f6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:42.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.079+0000 7f530376f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc0c6d70 con 0x7f52fc06bec0 2024-03-19T02:01:42.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.079+0000 7f530376f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc0c6bf0 con 0x7f52fc0c6610 2024-03-19T02:01:42.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.079+0000 7f530376f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f52fc158d00 con 0x7f52fc14eaa0 2024-03-19T02:01:42.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.079+0000 7f530150b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52fc06bec0 0x7f52fc14be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:42.076 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.079+0000 7f5300d0a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f52fc0c6610 0x7f52fc14daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:42.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.079+0000 7f5301d0c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f52fc14eaa0 0x7f52fc1f6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:42.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.079+0000 7f530150b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52fc06bec0 0x7f52fc14be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:46096/0 (socket says 172.21.15.55:46096) 2024-03-19T02:01:42.077 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.079+0000 7f530150b640 1 -- 172.21.15.55:0/3418726887 learned_addr learned my addr 172.21.15.55:0/3418726887 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:42.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.083+0000 7f530150b640 1 -- 172.21.15.55:0/3418726887 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f52fc14eaa0 msgr2=0x7f52fc1f6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:42.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.083+0000 7f530150b640 1 --2- 172.21.15.55:0/3418726887 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f52fc14eaa0 0x7f52fc1f6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:42.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.083+0000 7f530150b640 1 -- 172.21.15.55:0/3418726887 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f52fc0c6610 msgr2=0x7f52fc14daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:42.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.083+0000 7f530150b640 1 --2- 172.21.15.55:0/3418726887 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f52fc0c6610 0x7f52fc14daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:42.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.083+0000 7f530150b640 1 -- 172.21.15.55:0/3418726887 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52fc1f3350 con 0x7f52fc06bec0 2024-03-19T02:01:42.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.083+0000 7f530150b640 1 --2- 172.21.15.55:0/3418726887 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52fc06bec0 0x7f52fc14be20 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f52f801a040 tx=0x7f52f80191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:42.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.087+0000 7f52f1ffb640 1 -- 172.21.15.55:0/3418726887 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f80b8070 con 0x7f52fc06bec0 2024-03-19T02:01:42.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.087+0000 7f52f1ffb640 1 -- 172.21.15.55:0/3418726887 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52f80b4410 con 0x7f52fc06bec0 2024-03-19T02:01:42.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.087+0000 7f52f1ffb640 1 -- 172.21.15.55:0/3418726887 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f80b0040 con 0x7f52fc06bec0 2024-03-19T02:01:42.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.087+0000 7f530376f640 1 -- 172.21.15.55:0/3418726887 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52fc1f3630 con 0x7f52fc06bec0 2024-03-19T02:01:42.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.091+0000 7f530376f640 1 -- 172.21.15.55:0/3418726887 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52fc1f3bf0 con 0x7f52fc06bec0 2024-03-19T02:01:42.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.091+0000 7f530376f640 1 -- 172.21.15.55:0/3418726887 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52fc068a40 con 0x7f52fc06bec0 2024-03-19T02:01:42.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.115+0000 7f52f1ffb640 1 -- 172.21.15.55:0/3418726887 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f52f80bb070 con 0x7f52fc06bec0 2024-03-19T02:01:42.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.115+0000 7f52f1ffb640 1 --2- 172.21.15.55:0/3418726887 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f52e0067690 0x7f52e0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:42.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.115+0000 7f5300d0a640 1 --2- 172.21.15.55:0/3418726887 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f52e0067690 0x7f52e0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:42.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.115+0000 7f5300d0a640 1 --2- 172.21.15.55:0/3418726887 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f52e0067690 0x7f52e0069b50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f52fc15c5e0 tx=0x7f52ec009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:42.440 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.443+0000 7f52f1ffb640 1 -- 172.21.15.55:0/3418726887 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f52f80ede70 con 0x7f52fc06bec0 2024-03-19T02:01:42.441 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.443+0000 7f52f1ffb640 1 -- 172.21.15.55:0/3418726887 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f52f8016090 con 0x7f52fc06bec0 2024-03-19T02:01:42.669 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.671+0000 7f530376f640 1 -- 172.21.15.55:0/3418726887 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52fc071240 con 0x7f52fc06bec0 2024-03-19T02:01:42.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.759+0000 7f52f1ffb640 1 -- 172.21.15.55:0/3418726887 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f52f80efdd0 con 0x7f52fc06bec0 2024-03-19T02:01:42.759 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:42.759 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:42.760 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:42.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.763+0000 7f530376f640 1 -- 172.21.15.55:0/3418726887 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f52e0067690 msgr2=0x7f52e0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:42.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.763+0000 7f530376f640 1 --2- 172.21.15.55:0/3418726887 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f52e0067690 0x7f52e0069b50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f52fc15c5e0 tx=0x7f52ec009190 comp rx=0 tx=0).stop 2024-03-19T02:01:42.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.763+0000 7f530376f640 1 -- 172.21.15.55:0/3418726887 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52fc06bec0 msgr2=0x7f52fc14be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:42.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.763+0000 7f530376f640 1 --2- 172.21.15.55:0/3418726887 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52fc06bec0 0x7f52fc14be20 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f52f801a040 tx=0x7f52f80191b0 comp rx=0 tx=0).stop 2024-03-19T02:01:42.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.763+0000 7f530376f640 1 -- 172.21.15.55:0/3418726887 shutdown_connections 2024-03-19T02:01:42.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.763+0000 7f530376f640 1 --2- 172.21.15.55:0/3418726887 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f52e0067690 0x7f52e0069b50 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:42.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.763+0000 7f530376f640 1 --2- 172.21.15.55:0/3418726887 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f52fc14eaa0 0x7f52fc1f6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:42.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.763+0000 7f530376f640 1 --2- 172.21.15.55:0/3418726887 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f52fc0c6610 0x7f52fc14daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:42.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.763+0000 7f530376f640 1 --2- 172.21.15.55:0/3418726887 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f52fc06bec0 0x7f52fc14be20 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:42.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.763+0000 7f530376f640 1 -- 172.21.15.55:0/3418726887 >> 172.21.15.55:0/3418726887 conn(0x7f52fc0cc9c0 msgr2=0x7f52fc148620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:42.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.763+0000 7f530376f640 1 -- 172.21.15.55:0/3418726887 shutdown_connections 2024-03-19T02:01:42.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:42.763+0000 7f530376f640 1 -- 172.21.15.55:0/3418726887 wait complete. 2024-03-19T02:01:43.387 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:43.388 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:44.389 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:44.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.519+0000 7fac34f5a640 1 Processor -- start 2024-03-19T02:01:44.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.519+0000 7fac34f5a640 1 -- start start 2024-03-19T02:01:44.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.519+0000 7fac34f5a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fac300ca310 0x7fac3015e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:44.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.519+0000 7fac34f5a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fac3015ad40 0x7fac30154670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:44.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.519+0000 7fac34f5a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac3006bc90 con 0x7fac30154290 2024-03-19T02:01:44.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.519+0000 7fac34f5a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fac3015b2c0 con 0x7fac300ca310 2024-03-19T02:01:44.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.519+0000 7fac34f5a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fac3006c1d0 con 0x7fac3015ad40 2024-03-19T02:01:44.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.519+0000 7fac2dd9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fac300ca310 0x7fac3015e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:44.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.523+0000 7fac2dd9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fac300ca310 0x7fac3015e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:46106/0 (socket says 172.21.15.55:46106) 2024-03-19T02:01:44.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.523+0000 7fac2dd9b640 1 -- 172.21.15.55:0/1436147642 learned_addr learned my addr 172.21.15.55:0/1436147642 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:44.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.527+0000 7fac2dd9b640 1 -- 172.21.15.55:0/1436147642 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fac3015ad40 msgr2=0x7fac30154670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:44.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.527+0000 7fac2dd9b640 1 --2- 172.21.15.55:0/1436147642 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fac3015ad40 0x7fac30154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:44.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.527+0000 7fac2dd9b640 1 -- 172.21.15.55:0/1436147642 >> v1:172.21.15.55:6789/0 conn(0x7fac30154290 legacy=0x7fac30070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:44.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.527+0000 7fac2dd9b640 1 -- 172.21.15.55:0/1436147642 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac3006bf30 con 0x7fac300ca310 2024-03-19T02:01:44.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.531+0000 7fac2dd9b640 1 --2- 172.21.15.55:0/1436147642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac300ca310 0x7fac3015e4c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fac24009870 tx=0x7fac2401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8563cbbaa0530be server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:44.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.555+0000 7fac2d59a640 1 -- 172.21.15.55:0/1436147642 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac240ab5f0 con 0x7fac300ca310 2024-03-19T02:01:44.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.555+0000 7fac2d59a640 1 -- 172.21.15.55:0/1436147642 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac240b0070 con 0x7fac300ca310 2024-03-19T02:01:44.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.555+0000 7fac2d59a640 1 -- 172.21.15.55:0/1436147642 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac240ab9b0 con 0x7fac300ca310 2024-03-19T02:01:44.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.559+0000 7fac34f5a640 1 -- 172.21.15.55:0/1436147642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac300ca310 msgr2=0x7fac3015e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:44.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.559+0000 7fac34f5a640 1 --2- 172.21.15.55:0/1436147642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac300ca310 0x7fac3015e4c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fac24009870 tx=0x7fac2401c520 comp rx=0 tx=0).stop 2024-03-19T02:01:44.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.559+0000 7fac34f5a640 1 -- 172.21.15.55:0/1436147642 shutdown_connections 2024-03-19T02:01:44.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.559+0000 7fac34f5a640 1 --2- 172.21.15.55:0/1436147642 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fac3015ad40 0x7fac30154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:44.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.559+0000 7fac34f5a640 1 --2- 172.21.15.55:0/1436147642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac300ca310 0x7fac3015e4c0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:44.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.559+0000 7fac34f5a640 1 -- 172.21.15.55:0/1436147642 >> 172.21.15.55:0/1436147642 conn(0x7fac3006c580 msgr2=0x7fac3006c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:44.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.559+0000 7fac34f5a640 1 -- 172.21.15.55:0/1436147642 shutdown_connections 2024-03-19T02:01:44.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.559+0000 7fac34f5a640 1 -- 172.21.15.55:0/1436147642 wait complete. 2024-03-19T02:01:44.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.559+0000 7fac34f5a640 1 Processor -- start 2024-03-19T02:01:44.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.559+0000 7fac34f5a640 1 -- start start 2024-03-19T02:01:44.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.559+0000 7fac34f5a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fac300ca310 0x7fac301f6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:44.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.559+0000 7fac34f5a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac30154290 0x7fac3014d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:44.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.559+0000 7fac34f5a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac3015ad40 0x7fac301f47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:44.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.559+0000 7fac34f5a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fac300b2460 con 0x7fac3015ad40 2024-03-19T02:01:44.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.559+0000 7fac34f5a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fac300b22e0 con 0x7fac30154290 2024-03-19T02:01:44.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.559+0000 7fac34f5a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fac30070d50 con 0x7fac300ca310 2024-03-19T02:01:44.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.559+0000 7fac2ed9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac3015ad40 0x7fac301f47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:44.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.559+0000 7fac2e59c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fac300ca310 0x7fac301f6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:44.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.559+0000 7fac2ed9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac3015ad40 0x7fac301f47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:46118/0 (socket says 172.21.15.55:46118) 2024-03-19T02:01:44.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.559+0000 7fac2ed9d640 1 -- 172.21.15.55:0/947874995 learned_addr learned my addr 172.21.15.55:0/947874995 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:44.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.559+0000 7fac2dd9b640 1 --2- 172.21.15.55:0/947874995 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac30154290 0x7fac3014d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:44.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.567+0000 7fac2ed9d640 1 -- 172.21.15.55:0/947874995 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fac300ca310 msgr2=0x7fac301f6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:44.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.567+0000 7fac2ed9d640 1 --2- 172.21.15.55:0/947874995 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fac300ca310 0x7fac301f6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:44.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.567+0000 7fac2ed9d640 1 -- 172.21.15.55:0/947874995 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac30154290 msgr2=0x7fac3014d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:44.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.567+0000 7fac2ed9d640 1 --2- 172.21.15.55:0/947874995 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac30154290 0x7fac3014d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:44.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.567+0000 7fac2ed9d640 1 -- 172.21.15.55:0/947874995 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac301f74f0 con 0x7fac3015ad40 2024-03-19T02:01:44.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.567+0000 7fac2ed9d640 1 --2- 172.21.15.55:0/947874995 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac3015ad40 0x7fac301f47c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fac2800b9a0 tx=0x7fac2800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:44.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.571+0000 7fac1effd640 1 -- 172.21.15.55:0/947874995 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac28019070 con 0x7fac3015ad40 2024-03-19T02:01:44.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.575+0000 7fac1effd640 1 -- 172.21.15.55:0/947874995 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac2800cbc0 con 0x7fac3015ad40 2024-03-19T02:01:44.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.575+0000 7fac34f5a640 1 -- 172.21.15.55:0/947874995 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac301f7790 con 0x7fac3015ad40 2024-03-19T02:01:44.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.575+0000 7fac1effd640 1 -- 172.21.15.55:0/947874995 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac28014440 con 0x7fac3015ad40 2024-03-19T02:01:44.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.575+0000 7fac34f5a640 1 -- 172.21.15.55:0/947874995 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac3014c5f0 con 0x7fac3015ad40 2024-03-19T02:01:44.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.575+0000 7fac34f5a640 1 -- 172.21.15.55:0/947874995 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac30068a40 con 0x7fac3015ad40 2024-03-19T02:01:44.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.851+0000 7fac1effd640 1 -- 172.21.15.55:0/947874995 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fac2801c070 con 0x7fac3015ad40 2024-03-19T02:01:44.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.851+0000 7fac1effd640 1 --2- 172.21.15.55:0/947874995 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fac100676b0 0x7fac10069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:44.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.851+0000 7fac2e59c640 1 --2- 172.21.15.55:0/947874995 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fac100676b0 0x7fac10069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:44.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.851+0000 7fac2e59c640 1 --2- 172.21.15.55:0/947874995 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fac100676b0 0x7fac10069b70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fac20007fd0 tx=0x7fac20007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:44.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.879+0000 7fac1effd640 1 -- 172.21.15.55:0/947874995 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fac28054040 con 0x7fac3015ad40 2024-03-19T02:01:44.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:44.879+0000 7fac1effd640 1 -- 172.21.15.55:0/947874995 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fac28023020 con 0x7fac3015ad40 2024-03-19T02:01:45.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:45.107+0000 7fac34f5a640 1 -- 172.21.15.55:0/947874995 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac3006b960 con 0x7fac3015ad40 2024-03-19T02:01:45.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:45.187+0000 7fac1effd640 1 -- 172.21.15.55:0/947874995 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fac28055fa0 con 0x7fac3015ad40 2024-03-19T02:01:45.185 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:45.185 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:45.186 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:45.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:45.187+0000 7fac34f5a640 1 -- 172.21.15.55:0/947874995 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fac100676b0 msgr2=0x7fac10069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:45.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:45.187+0000 7fac34f5a640 1 --2- 172.21.15.55:0/947874995 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fac100676b0 0x7fac10069b70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fac20007fd0 tx=0x7fac20007e40 comp rx=0 tx=0).stop 2024-03-19T02:01:45.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:45.187+0000 7fac34f5a640 1 -- 172.21.15.55:0/947874995 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac3015ad40 msgr2=0x7fac301f47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:45.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:45.187+0000 7fac34f5a640 1 --2- 172.21.15.55:0/947874995 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac3015ad40 0x7fac301f47c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fac2800b9a0 tx=0x7fac2800be60 comp rx=0 tx=0).stop 2024-03-19T02:01:45.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:45.187+0000 7fac34f5a640 1 -- 172.21.15.55:0/947874995 shutdown_connections 2024-03-19T02:01:45.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:45.187+0000 7fac34f5a640 1 --2- 172.21.15.55:0/947874995 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fac100676b0 0x7fac10069b70 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:45.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:45.187+0000 7fac34f5a640 1 --2- 172.21.15.55:0/947874995 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac3015ad40 0x7fac301f47c0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:45.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:45.187+0000 7fac34f5a640 1 --2- 172.21.15.55:0/947874995 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac30154290 0x7fac3014d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:45.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:45.187+0000 7fac34f5a640 1 --2- 172.21.15.55:0/947874995 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fac300ca310 0x7fac301f6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:45.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:45.187+0000 7fac34f5a640 1 -- 172.21.15.55:0/947874995 >> 172.21.15.55:0/947874995 conn(0x7fac3006c580 msgr2=0x7fac3015c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:45.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:45.187+0000 7fac34f5a640 1 -- 172.21.15.55:0/947874995 shutdown_connections 2024-03-19T02:01:45.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:45.187+0000 7fac34f5a640 1 -- 172.21.15.55:0/947874995 wait complete. 2024-03-19T02:01:45.807 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:45.807 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:46.808 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:46.922 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.923+0000 7fb1cbab0640 1 Processor -- start 2024-03-19T02:01:46.922 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.923+0000 7fb1cbab0640 1 -- start start 2024-03-19T02:01:46.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.923+0000 7fb1cbab0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb1c41585a0 0x7fb1c4158a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:46.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.923+0000 7fb1cbab0640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fb1c40c60d0 0x7fb1c40c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:46.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.923+0000 7fb1cbab0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1c40c7540 con 0x7fb1c414e930 2024-03-19T02:01:46.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.923+0000 7fb1cbab0640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fb1c40c80a0 con 0x7fb1c40c60d0 2024-03-19T02:01:46.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.923+0000 7fb1cbab0640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fb1c40c7800 con 0x7fb1c41585a0 2024-03-19T02:01:46.924 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.927+0000 7fb1ca04d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fb1c40c60d0 0x7fb1c40c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:46.924 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.927+0000 7fb1c904b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb1c41585a0 0x7fb1c4158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:46.924 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.927+0000 7fb1c984c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fb1c414e930 0x7fb1c406b910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:57578/0 (socket says 172.21.15.55:57578) 2024-03-19T02:01:46.924 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.927+0000 7fb1c984c640 1 -- 172.21.15.55:0/249290843 learned_addr learned my addr 172.21.15.55:0/249290843 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:01:46.928 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.931+0000 7fb1c904b640 1 -- 172.21.15.55:0/249290843 >> v2:172.21.15.84:3300/0 conn(0x7fb1c40c60d0 msgr2=0x7fb1c40c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:46.928 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.931+0000 7fb1c904b640 1 --2- 172.21.15.55:0/249290843 >> v2:172.21.15.84:3300/0 conn(0x7fb1c40c60d0 0x7fb1c40c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:46.928 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.931+0000 7fb1c904b640 1 -- 172.21.15.55:0/249290843 >> v1:172.21.15.84:6789/0 conn(0x7fb1c414e930 legacy=0x7fb1c406b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:46.928 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.931+0000 7fb1c904b640 1 -- 172.21.15.55:0/249290843 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c40c7ae0 con 0x7fb1c41585a0 2024-03-19T02:01:46.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.935+0000 7fb1c904b640 1 --2- 172.21.15.55:0/249290843 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb1c41585a0 0x7fb1c4158a00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb1b40097c0 tx=0x7fb1b401ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1d049861bd82a07 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:46.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.943+0000 7fb1c884a640 1 -- 172.21.15.55:0/249290843 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b40b5070 con 0x7fb1c41585a0 2024-03-19T02:01:46.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.943+0000 7fb1c884a640 1 -- 172.21.15.55:0/249290843 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1b40a8990 con 0x7fb1c41585a0 2024-03-19T02:01:46.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.943+0000 7fb1c884a640 1 -- 172.21.15.55:0/249290843 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b40a8d60 con 0x7fb1c41585a0 2024-03-19T02:01:46.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.943+0000 7fb1cbab0640 1 -- 172.21.15.55:0/249290843 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb1c41585a0 msgr2=0x7fb1c4158a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:46.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.943+0000 7fb1cbab0640 1 --2- 172.21.15.55:0/249290843 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb1c41585a0 0x7fb1c4158a00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb1b40097c0 tx=0x7fb1b401ec90 comp rx=0 tx=0).stop 2024-03-19T02:01:46.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.943+0000 7fb1cbab0640 1 -- 172.21.15.55:0/249290843 shutdown_connections 2024-03-19T02:01:46.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.943+0000 7fb1cbab0640 1 --2- 172.21.15.55:0/249290843 >> v2:172.21.15.84:3300/0 conn(0x7fb1c40c60d0 0x7fb1c40c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:46.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.943+0000 7fb1cbab0640 1 --2- 172.21.15.55:0/249290843 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb1c41585a0 0x7fb1c4158a00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:46.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.943+0000 7fb1cbab0640 1 -- 172.21.15.55:0/249290843 >> 172.21.15.55:0/249290843 conn(0x7fb1c40cc9c0 msgr2=0x7fb1c40ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:46.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.943+0000 7fb1cbab0640 1 -- 172.21.15.55:0/249290843 shutdown_connections 2024-03-19T02:01:46.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.943+0000 7fb1cbab0640 1 -- 172.21.15.55:0/249290843 wait complete. 2024-03-19T02:01:46.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.947+0000 7fb1cbab0640 1 Processor -- start 2024-03-19T02:01:46.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.947+0000 7fb1cbab0640 1 -- start start 2024-03-19T02:01:46.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.947+0000 7fb1cbab0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb1c40c60d0 0x7fb1c414adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:46.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.947+0000 7fb1cbab0640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1c414e930 0x7fb1c4157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:46.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.947+0000 7fb1cbab0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1c41585a0 0x7fb1c4154a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:46.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.947+0000 7fb1cbab0640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c406c020 con 0x7fb1c41585a0 2024-03-19T02:01:46.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.947+0000 7fb1cbab0640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c4158080 con 0x7fb1c414e930 2024-03-19T02:01:46.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.947+0000 7fb1cbab0640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fb1c414ed10 con 0x7fb1c40c60d0 2024-03-19T02:01:46.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.947+0000 7fb1c904b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1c414e930 0x7fb1c4157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:46.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.947+0000 7fb1ca04d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1c41585a0 0x7fb1c4154a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:46.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.947+0000 7fb1c984c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb1c40c60d0 0x7fb1c414adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:46.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.947+0000 7fb1ca04d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1c41585a0 0x7fb1c4154a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:46122/0 (socket says 172.21.15.55:46122) 2024-03-19T02:01:46.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.947+0000 7fb1ca04d640 1 -- 172.21.15.55:0/2029025336 learned_addr learned my addr 172.21.15.55:0/2029025336 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:46.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.951+0000 7fb1ca04d640 1 -- 172.21.15.55:0/2029025336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb1c40c60d0 msgr2=0x7fb1c414adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:46.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.951+0000 7fb1ca04d640 1 --2- 172.21.15.55:0/2029025336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb1c40c60d0 0x7fb1c414adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:46.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.951+0000 7fb1ca04d640 1 -- 172.21.15.55:0/2029025336 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1c414e930 msgr2=0x7fb1c4157770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:46.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.951+0000 7fb1ca04d640 1 --2- 172.21.15.55:0/2029025336 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1c414e930 0x7fb1c4157770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:46.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.951+0000 7fb1ca04d640 1 -- 172.21.15.55:0/2029025336 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c4155b30 con 0x7fb1c41585a0 2024-03-19T02:01:46.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.955+0000 7fb1ca04d640 1 --2- 172.21.15.55:0/2029025336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1c41585a0 0x7fb1c4154a20 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc00efc0 tx=0x7fb1bc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:46.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.959+0000 7fb1ba7fc640 1 -- 172.21.15.55:0/2029025336 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc018020 con 0x7fb1c41585a0 2024-03-19T02:01:46.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.959+0000 7fb1ba7fc640 1 -- 172.21.15.55:0/2029025336 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1bc014480 con 0x7fb1c41585a0 2024-03-19T02:01:46.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.959+0000 7fb1ba7fc640 1 -- 172.21.15.55:0/2029025336 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc014870 con 0x7fb1c41585a0 2024-03-19T02:01:46.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.959+0000 7fb1cbab0640 1 -- 172.21.15.55:0/2029025336 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1c40cc030 con 0x7fb1c41585a0 2024-03-19T02:01:46.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.959+0000 7fb1cbab0640 1 -- 172.21.15.55:0/2029025336 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1c4155dd0 con 0x7fb1c41585a0 2024-03-19T02:01:46.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.959+0000 7fb1cbab0640 1 -- 172.21.15.55:0/2029025336 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1c4068a40 con 0x7fb1c41585a0 2024-03-19T02:01:46.991 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.995+0000 7fb1ba7fc640 1 -- 172.21.15.55:0/2029025336 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb1bc01b020 con 0x7fb1c41585a0 2024-03-19T02:01:46.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.995+0000 7fb1ba7fc640 1 --2- 172.21.15.55:0/2029025336 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb1a80676b0 0x7fb1a8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:46.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.995+0000 7fb1c984c640 1 --2- 172.21.15.55:0/2029025336 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb1a80676b0 0x7fb1a8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:46.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:46.995+0000 7fb1c984c640 1 --2- 172.21.15.55:0/2029025336 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb1a80676b0 0x7fb1a8069b70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0000f70 tx=0x7fb1c0007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:47.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:47.275+0000 7fb1ba7fc640 1 -- 172.21.15.55:0/2029025336 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fb1bc055100 con 0x7fb1c41585a0 2024-03-19T02:01:47.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:47.275+0000 7fb1ba7fc640 1 -- 172.21.15.55:0/2029025336 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fb1bc022070 con 0x7fb1c41585a0 2024-03-19T02:01:47.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:47.491+0000 7fb1cbab0640 1 -- 172.21.15.55:0/2029025336 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1c40729d0 con 0x7fb1c41585a0 2024-03-19T02:01:47.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:47.563+0000 7fb1ba7fc640 1 -- 172.21.15.55:0/2029025336 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fb1bc010040 con 0x7fb1c41585a0 2024-03-19T02:01:47.562 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:47.562 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:47.563 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:47.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:47.567+0000 7fb1cbab0640 1 -- 172.21.15.55:0/2029025336 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb1a80676b0 msgr2=0x7fb1a8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:47.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:47.567+0000 7fb1cbab0640 1 --2- 172.21.15.55:0/2029025336 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb1a80676b0 0x7fb1a8069b70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0000f70 tx=0x7fb1c0007e90 comp rx=0 tx=0).stop 2024-03-19T02:01:47.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:47.567+0000 7fb1cbab0640 1 -- 172.21.15.55:0/2029025336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1c41585a0 msgr2=0x7fb1c4154a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:47.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:47.567+0000 7fb1cbab0640 1 --2- 172.21.15.55:0/2029025336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1c41585a0 0x7fb1c4154a20 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc00efc0 tx=0x7fb1bc00c490 comp rx=0 tx=0).stop 2024-03-19T02:01:47.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:47.567+0000 7fb1cbab0640 1 -- 172.21.15.55:0/2029025336 shutdown_connections 2024-03-19T02:01:47.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:47.567+0000 7fb1cbab0640 1 --2- 172.21.15.55:0/2029025336 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb1a80676b0 0x7fb1a8069b70 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:47.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:47.567+0000 7fb1cbab0640 1 --2- 172.21.15.55:0/2029025336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1c41585a0 0x7fb1c4154a20 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:47.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:47.567+0000 7fb1cbab0640 1 --2- 172.21.15.55:0/2029025336 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1c414e930 0x7fb1c4157770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:47.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:47.567+0000 7fb1cbab0640 1 --2- 172.21.15.55:0/2029025336 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb1c40c60d0 0x7fb1c414adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:47.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:47.567+0000 7fb1cbab0640 1 -- 172.21.15.55:0/2029025336 >> 172.21.15.55:0/2029025336 conn(0x7fb1c40cc9c0 msgr2=0x7fb1c40c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:47.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:47.567+0000 7fb1cbab0640 1 -- 172.21.15.55:0/2029025336 shutdown_connections 2024-03-19T02:01:47.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:47.567+0000 7fb1cbab0640 1 -- 172.21.15.55:0/2029025336 wait complete. 2024-03-19T02:01:48.187 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:48.187 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:49.188 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:49.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.307+0000 7fefb871f640 1 Processor -- start 2024-03-19T02:01:49.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.307+0000 7fefb871f640 1 -- start start 2024-03-19T02:01:49.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.307+0000 7fefb871f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fefb015bcd0 0x7fefb015c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:49.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.307+0000 7fefb871f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefb006fff0 con 0x7fefb0070780 2024-03-19T02:01:49.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.307+0000 7fefb871f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefb00c39c0 con 0x7fefb0152120 2024-03-19T02:01:49.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.307+0000 7fefb871f640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fefb0152500 con 0x7fefb015bcd0 2024-03-19T02:01:49.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.307+0000 7fefb5cba640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fefb0070780 0x7fefb015b5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:48200/0 (socket says 172.21.15.55:48200) 2024-03-19T02:01:49.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.307+0000 7fefb5cba640 1 -- 172.21.15.55:0/4102925743 learned_addr learned my addr 172.21.15.55:0/4102925743 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:01:49.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.307+0000 7fefb6cbc640 1 --2- 172.21.15.55:0/4102925743 >> v2:172.21.15.84:3300/0 conn(0x7fefb015bcd0 0x7fefb015c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:49.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.311+0000 7fefb54b9640 1 -- 172.21.15.55:0/4102925743 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2241371238 0 0) 0x7fefb006fff0 con 0x7fefb0070780 2024-03-19T02:01:49.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.311+0000 7fefb54b9640 1 -- 172.21.15.55:0/4102925743 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef9c003530 con 0x7fefb0070780 2024-03-19T02:01:49.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.311+0000 7fefb6cbc640 1 -- 172.21.15.55:0/4102925743 >> v1:172.21.15.84:6789/0 conn(0x7fefb0152120 legacy=0x7fefb00c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:49.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.311+0000 7fefb6cbc640 1 -- 172.21.15.55:0/4102925743 >> v1:172.21.15.55:6789/0 conn(0x7fefb0070780 legacy=0x7fefb015b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:49.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.311+0000 7fefb6cbc640 1 -- 172.21.15.55:0/4102925743 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefb00702b0 con 0x7fefb015bcd0 2024-03-19T02:01:49.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.323+0000 7fefb6cbc640 1 --2- 172.21.15.55:0/4102925743 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fefb015bcd0 0x7fefb015c160 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fefa8009d90 tx=0x7fefa801c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e070bbb890a96530 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:49.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.327+0000 7fefb54b9640 1 -- 172.21.15.55:0/4102925743 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa80b2070 con 0x7fefb015bcd0 2024-03-19T02:01:49.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.327+0000 7fefb54b9640 1 -- 172.21.15.55:0/4102925743 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefa80a5830 con 0x7fefb015bcd0 2024-03-19T02:01:49.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.327+0000 7fefb54b9640 1 -- 172.21.15.55:0/4102925743 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa80a5c00 con 0x7fefb015bcd0 2024-03-19T02:01:49.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.331+0000 7fefb871f640 1 -- 172.21.15.55:0/4102925743 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fefb015bcd0 msgr2=0x7fefb015c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:49.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.331+0000 7fefb871f640 1 --2- 172.21.15.55:0/4102925743 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fefb015bcd0 0x7fefb015c160 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fefa8009d90 tx=0x7fefa801c970 comp rx=0 tx=0).stop 2024-03-19T02:01:49.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.331+0000 7fefb871f640 1 -- 172.21.15.55:0/4102925743 shutdown_connections 2024-03-19T02:01:49.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.331+0000 7fefb871f640 1 --2- 172.21.15.55:0/4102925743 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fefb015bcd0 0x7fefb015c160 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:49.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.331+0000 7fefb871f640 1 -- 172.21.15.55:0/4102925743 >> 172.21.15.55:0/4102925743 conn(0x7fefb0067600 msgr2=0x7fefb0071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:49.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.331+0000 7fefb871f640 1 -- 172.21.15.55:0/4102925743 shutdown_connections 2024-03-19T02:01:49.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.331+0000 7fefb871f640 1 -- 172.21.15.55:0/4102925743 wait complete. 2024-03-19T02:01:49.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.331+0000 7fefb871f640 1 Processor -- start 2024-03-19T02:01:49.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.331+0000 7fefb871f640 1 -- start start 2024-03-19T02:01:49.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.331+0000 7fefb871f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefb0070780 0x7fefb01434e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:49.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.331+0000 7fefb871f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fefb0152120 0x7fefb0143a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:49.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.331+0000 7fefb871f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fefb015bcd0 0x7fefb0155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:49.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.331+0000 7fefb871f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fefb00c97e0 con 0x7fefb0070780 2024-03-19T02:01:49.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.331+0000 7fefb871f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fefb00c3530 con 0x7fefb0152120 2024-03-19T02:01:49.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.331+0000 7fefb871f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fefb00c86d0 con 0x7fefb015bcd0 2024-03-19T02:01:49.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.335+0000 7fefb64bb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefb0070780 0x7fefb01434e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:49.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.335+0000 7fefb5cba640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fefb0152120 0x7fefb0143a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:49.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.335+0000 7fefb6cbc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fefb015bcd0 0x7fefb0155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:49.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.335+0000 7fefb64bb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefb0070780 0x7fefb01434e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49108/0 (socket says 172.21.15.55:49108) 2024-03-19T02:01:49.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.335+0000 7fefb64bb640 1 -- 172.21.15.55:0/4111458871 learned_addr learned my addr 172.21.15.55:0/4111458871 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:49.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.339+0000 7fefb64bb640 1 -- 172.21.15.55:0/4111458871 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fefb015bcd0 msgr2=0x7fefb0155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:49.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.339+0000 7fefb64bb640 1 --2- 172.21.15.55:0/4111458871 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fefb015bcd0 0x7fefb0155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:49.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.339+0000 7fefb64bb640 1 -- 172.21.15.55:0/4111458871 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fefb0152120 msgr2=0x7fefb0143a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:49.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.339+0000 7fefb64bb640 1 --2- 172.21.15.55:0/4111458871 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fefb0152120 0x7fefb0143a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:49.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.339+0000 7fefb64bb640 1 -- 172.21.15.55:0/4111458871 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefb0158ec0 con 0x7fefb0070780 2024-03-19T02:01:49.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.339+0000 7fefb64bb640 1 --2- 172.21.15.55:0/4111458871 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefb0070780 0x7fefb01434e0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fefac00c450 tx=0x7fefac00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:49.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.343+0000 7fefa6ffd640 1 -- 172.21.15.55:0/4111458871 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefac016020 con 0x7fefb0070780 2024-03-19T02:01:49.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.343+0000 7fefa6ffd640 1 -- 172.21.15.55:0/4111458871 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefac0052d0 con 0x7fefb0070780 2024-03-19T02:01:49.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.343+0000 7fefb871f640 1 -- 172.21.15.55:0/4111458871 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefb0147b00 con 0x7fefb0070780 2024-03-19T02:01:49.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.343+0000 7fefa6ffd640 1 -- 172.21.15.55:0/4111458871 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefac007260 con 0x7fefb0070780 2024-03-19T02:01:49.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.347+0000 7fefb871f640 1 -- 172.21.15.55:0/4111458871 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefb01585f0 con 0x7fefb0070780 2024-03-19T02:01:49.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.347+0000 7fefb871f640 1 -- 172.21.15.55:0/4111458871 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefb0068a40 con 0x7fefb0070780 2024-03-19T02:01:49.376 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.379+0000 7fefa6ffd640 1 -- 172.21.15.55:0/4111458871 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fefac007400 con 0x7fefb0070780 2024-03-19T02:01:49.377 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.379+0000 7fefa6ffd640 1 --2- 172.21.15.55:0/4111458871 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fef940676b0 0x7fef94069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:49.377 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.379+0000 7fefb5cba640 1 --2- 172.21.15.55:0/4111458871 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fef940676b0 0x7fef94069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:49.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.379+0000 7fefb5cba640 1 --2- 172.21.15.55:0/4111458871 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fef940676b0 0x7fef94069b70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fefa0007fd0 tx=0x7fefa0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:49.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.383+0000 7fefa6ffd640 1 -- 172.21.15.55:0/4111458871 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fefac054640 con 0x7fefb0070780 2024-03-19T02:01:49.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.707+0000 7fefa6ffd640 1 -- 172.21.15.55:0/4111458871 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fefac020020 con 0x7fefb0070780 2024-03-19T02:01:49.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:49.995+0000 7fefb871f640 1 -- 172.21.15.55:0/4111458871 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fefb00c84b0 con 0x7fefb0070780 2024-03-19T02:01:50.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:50.011+0000 7fefa6ffd640 1 -- 172.21.15.55:0/4111458871 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fefac005480 con 0x7fefb0070780 2024-03-19T02:01:50.010 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:50.010 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:50.010 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:50.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:50.015+0000 7fefb871f640 1 -- 172.21.15.55:0/4111458871 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fef940676b0 msgr2=0x7fef94069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:50.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:50.015+0000 7fefb871f640 1 --2- 172.21.15.55:0/4111458871 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fef940676b0 0x7fef94069b70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fefa0007fd0 tx=0x7fefa0009040 comp rx=0 tx=0).stop 2024-03-19T02:01:50.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:50.015+0000 7fefb871f640 1 -- 172.21.15.55:0/4111458871 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefb0070780 msgr2=0x7fefb01434e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:50.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:50.015+0000 7fefb871f640 1 --2- 172.21.15.55:0/4111458871 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefb0070780 0x7fefb01434e0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fefac00c450 tx=0x7fefac00c910 comp rx=0 tx=0).stop 2024-03-19T02:01:50.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:50.015+0000 7fefb871f640 1 -- 172.21.15.55:0/4111458871 shutdown_connections 2024-03-19T02:01:50.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:50.015+0000 7fefb871f640 1 --2- 172.21.15.55:0/4111458871 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fef940676b0 0x7fef94069b70 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:50.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:50.015+0000 7fefb871f640 1 --2- 172.21.15.55:0/4111458871 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fefb015bcd0 0x7fefb0155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:50.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:50.015+0000 7fefb871f640 1 --2- 172.21.15.55:0/4111458871 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fefb0152120 0x7fefb0143a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:50.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:50.015+0000 7fefb871f640 1 --2- 172.21.15.55:0/4111458871 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefb0070780 0x7fefb01434e0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:50.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:50.015+0000 7fefb871f640 1 -- 172.21.15.55:0/4111458871 >> 172.21.15.55:0/4111458871 conn(0x7fefb0067600 msgr2=0x7fefb00cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:50.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:50.015+0000 7fefb871f640 1 -- 172.21.15.55:0/4111458871 shutdown_connections 2024-03-19T02:01:50.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:50.015+0000 7fefb871f640 1 -- 172.21.15.55:0/4111458871 wait complete. 2024-03-19T02:01:50.631 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:50.631 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:51.632 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:51.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.747+0000 7fef9e4f3640 1 Processor -- start 2024-03-19T02:01:51.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.747+0000 7fef9e4f3640 1 -- start start 2024-03-19T02:01:51.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.747+0000 7fef9e4f3640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fef981510f0 0x7fef980c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:51.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.747+0000 7fef9e4f3640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fef9815a410 0x7fef980708a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:51.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.747+0000 7fef9e4f3640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fef980c73c0 0x7fef980c7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:51.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.747+0000 7fef9e4f3640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fef980c3f60 con 0x7fef980c73c0 2024-03-19T02:01:51.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.747+0000 7fef9e4f3640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fef981514d0 con 0x7fef9815a410 2024-03-19T02:01:51.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.747+0000 7fef9e4f3640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fef980c6870 con 0x7fef981510f0 2024-03-19T02:01:51.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.751+0000 7fef9ca90640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fef980c73c0 0x7fef980c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:51.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.751+0000 7fef977fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fef9815a410 0x7fef980708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:51.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.751+0000 7fef9ca90640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fef980c73c0 0x7fef980c7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49114/0 (socket says 172.21.15.55:49114) 2024-03-19T02:01:51.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.751+0000 7fef9ca90640 1 -- 172.21.15.55:0/492360523 learned_addr learned my addr 172.21.15.55:0/492360523 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:51.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.751+0000 7fef97fff640 1 --2- 172.21.15.55:0/492360523 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fef981510f0 0x7fef980c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:51.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.755+0000 7fef9ca90640 1 -- 172.21.15.55:0/492360523 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fef981510f0 msgr2=0x7fef980c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:51.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.755+0000 7fef9ca90640 1 --2- 172.21.15.55:0/492360523 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fef981510f0 0x7fef980c61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:51.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.755+0000 7fef9ca90640 1 -- 172.21.15.55:0/492360523 >> v2:172.21.15.84:3300/0 conn(0x7fef9815a410 msgr2=0x7fef980708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:51.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.755+0000 7fef9ca90640 1 --2- 172.21.15.55:0/492360523 >> v2:172.21.15.84:3300/0 conn(0x7fef9815a410 0x7fef980708a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:51.752 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.755+0000 7fef9ca90640 1 -- 172.21.15.55:0/492360523 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef980c6b50 con 0x7fef980c73c0 2024-03-19T02:01:51.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.759+0000 7fef9ca90640 1 --2- 172.21.15.55:0/492360523 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef980c73c0 0x7fef980c7830 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fef980ca580 tx=0x7fef9001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5315e8723ba6ac server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:51.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.763+0000 7fef96ffd640 1 -- 172.21.15.55:0/492360523 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef900b4070 con 0x7fef980c73c0 2024-03-19T02:01:51.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.763+0000 7fef96ffd640 1 -- 172.21.15.55:0/492360523 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef900af400 con 0x7fef980c73c0 2024-03-19T02:01:51.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.763+0000 7fef96ffd640 1 -- 172.21.15.55:0/492360523 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef900af7d0 con 0x7fef980c73c0 2024-03-19T02:01:51.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.763+0000 7fef9e4f3640 1 -- 172.21.15.55:0/492360523 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef980c73c0 msgr2=0x7fef980c7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:51.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.763+0000 7fef9e4f3640 1 --2- 172.21.15.55:0/492360523 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef980c73c0 0x7fef980c7830 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fef980ca580 tx=0x7fef9001f1e0 comp rx=0 tx=0).stop 2024-03-19T02:01:51.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.763+0000 7fef9e4f3640 1 -- 172.21.15.55:0/492360523 shutdown_connections 2024-03-19T02:01:51.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.763+0000 7fef9e4f3640 1 --2- 172.21.15.55:0/492360523 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef980c73c0 0x7fef980c7830 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:51.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.763+0000 7fef9e4f3640 1 --2- 172.21.15.55:0/492360523 >> v2:172.21.15.84:3300/0 conn(0x7fef9815a410 0x7fef980708a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:51.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.763+0000 7fef9e4f3640 1 --2- 172.21.15.55:0/492360523 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fef981510f0 0x7fef980c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:51.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.763+0000 7fef9e4f3640 1 -- 172.21.15.55:0/492360523 >> 172.21.15.55:0/492360523 conn(0x7fef980cc9c0 msgr2=0x7fef980ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:51.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.763+0000 7fef9e4f3640 1 -- 172.21.15.55:0/492360523 shutdown_connections 2024-03-19T02:01:51.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.763+0000 7fef9e4f3640 1 -- 172.21.15.55:0/492360523 wait complete. 2024-03-19T02:01:51.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.763+0000 7fef9e4f3640 1 Processor -- start 2024-03-19T02:01:51.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.763+0000 7fef9e4f3640 1 -- start start 2024-03-19T02:01:51.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.767+0000 7fef9e4f3640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fef980c73c0 0x7fef981efed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:51.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.767+0000 7fef9e4f3640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef981510f0 0x7fef98158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:51.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.767+0000 7fef9e4f3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef9815a410 0x7fef98158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:51.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.767+0000 7fef9e4f3640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fef980c6e30 con 0x7fef9815a410 2024-03-19T02:01:51.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.767+0000 7fef9e4f3640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fef980c44e0 con 0x7fef981510f0 2024-03-19T02:01:51.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.767+0000 7fef9e4f3640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fef980c4a00 con 0x7fef980c73c0 2024-03-19T02:01:51.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.767+0000 7fef9ca90640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef9815a410 0x7fef98158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:51.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.767+0000 7fef9ca90640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef9815a410 0x7fef98158240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49116/0 (socket says 172.21.15.55:49116) 2024-03-19T02:01:51.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.767+0000 7fef9ca90640 1 -- 172.21.15.55:0/2852487103 learned_addr learned my addr 172.21.15.55:0/2852487103 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:51.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.767+0000 7fef977fe640 1 --2- 172.21.15.55:0/2852487103 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef981510f0 0x7fef98158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:51.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.767+0000 7fef97fff640 1 --2- 172.21.15.55:0/2852487103 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fef980c73c0 0x7fef981efed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:51.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.771+0000 7fef9ca90640 1 -- 172.21.15.55:0/2852487103 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fef980c73c0 msgr2=0x7fef981efed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:51.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.771+0000 7fef9ca90640 1 --2- 172.21.15.55:0/2852487103 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fef980c73c0 0x7fef981efed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:51.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.771+0000 7fef9ca90640 1 -- 172.21.15.55:0/2852487103 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef981510f0 msgr2=0x7fef98158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:51.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.771+0000 7fef9ca90640 1 --2- 172.21.15.55:0/2852487103 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef981510f0 0x7fef98158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:51.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.771+0000 7fef9ca90640 1 -- 172.21.15.55:0/2852487103 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef98159610 con 0x7fef9815a410 2024-03-19T02:01:51.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.775+0000 7fef9ca90640 1 --2- 172.21.15.55:0/2852487103 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef9815a410 0x7fef98158240 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fef900b2040 tx=0x7fef9001bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:51.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.779+0000 7fef94ff9640 1 -- 172.21.15.55:0/2852487103 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef900b4040 con 0x7fef9815a410 2024-03-19T02:01:51.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.779+0000 7fef94ff9640 1 -- 172.21.15.55:0/2852487103 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef900c0070 con 0x7fef9815a410 2024-03-19T02:01:51.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.779+0000 7fef94ff9640 1 -- 172.21.15.55:0/2852487103 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef90009510 con 0x7fef9815a410 2024-03-19T02:01:51.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.779+0000 7fef9e4f3640 1 -- 172.21.15.55:0/2852487103 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef981598f0 con 0x7fef9815a410 2024-03-19T02:01:51.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.779+0000 7fef9e4f3640 1 -- 172.21.15.55:0/2852487103 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef980cb1a0 con 0x7fef9815a410 2024-03-19T02:01:51.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:51.779+0000 7fef9e4f3640 1 -- 172.21.15.55:0/2852487103 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef98068a40 con 0x7fef9815a410 2024-03-19T02:01:52.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:52.095+0000 7fef94ff9640 1 -- 172.21.15.55:0/2852487103 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fef900c5460 con 0x7fef9815a410 2024-03-19T02:01:52.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:52.095+0000 7fef94ff9640 1 --2- 172.21.15.55:0/2852487103 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fef84067660 0x7fef84069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:52.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:52.095+0000 7fef97fff640 1 --2- 172.21.15.55:0/2852487103 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fef84067660 0x7fef84069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:52.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:52.095+0000 7fef97fff640 1 --2- 172.21.15.55:0/2852487103 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fef84067660 0x7fef84069b20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fef88005bd0 tx=0x7fef8800f040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:52.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:52.119+0000 7fef94ff9640 1 -- 172.21.15.55:0/2852487103 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fef900f28c0 con 0x7fef9815a410 2024-03-19T02:01:52.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:52.119+0000 7fef94ff9640 1 -- 172.21.15.55:0/2852487103 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fef900ca020 con 0x7fef9815a410 2024-03-19T02:01:52.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:52.343+0000 7fef9e4f3640 1 -- 172.21.15.55:0/2852487103 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef9814ce50 con 0x7fef9815a410 2024-03-19T02:01:52.365 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:52.367+0000 7fef94ff9640 1 -- 172.21.15.55:0/2852487103 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fef90018d40 con 0x7fef9815a410 2024-03-19T02:01:52.365 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:52.365 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:52.366 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:52.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:52.371+0000 7fef9e4f3640 1 -- 172.21.15.55:0/2852487103 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fef84067660 msgr2=0x7fef84069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:52.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:52.371+0000 7fef9e4f3640 1 --2- 172.21.15.55:0/2852487103 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fef84067660 0x7fef84069b20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fef88005bd0 tx=0x7fef8800f040 comp rx=0 tx=0).stop 2024-03-19T02:01:52.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:52.371+0000 7fef9e4f3640 1 -- 172.21.15.55:0/2852487103 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef9815a410 msgr2=0x7fef98158240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:52.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:52.371+0000 7fef9e4f3640 1 --2- 172.21.15.55:0/2852487103 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef9815a410 0x7fef98158240 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fef900b2040 tx=0x7fef9001bf80 comp rx=0 tx=0).stop 2024-03-19T02:01:52.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:52.371+0000 7fef9e4f3640 1 -- 172.21.15.55:0/2852487103 shutdown_connections 2024-03-19T02:01:52.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:52.371+0000 7fef9e4f3640 1 --2- 172.21.15.55:0/2852487103 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fef84067660 0x7fef84069b20 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:52.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:52.371+0000 7fef9e4f3640 1 --2- 172.21.15.55:0/2852487103 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fef9815a410 0x7fef98158240 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:52.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:52.371+0000 7fef9e4f3640 1 --2- 172.21.15.55:0/2852487103 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef981510f0 0x7fef98158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:52.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:52.371+0000 7fef9e4f3640 1 --2- 172.21.15.55:0/2852487103 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fef980c73c0 0x7fef981efed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:52.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:52.371+0000 7fef9e4f3640 1 -- 172.21.15.55:0/2852487103 >> 172.21.15.55:0/2852487103 conn(0x7fef980cc9c0 msgr2=0x7fef9814c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:52.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:52.371+0000 7fef9e4f3640 1 -- 172.21.15.55:0/2852487103 shutdown_connections 2024-03-19T02:01:52.369 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:52.371+0000 7fef9e4f3640 1 -- 172.21.15.55:0/2852487103 wait complete. 2024-03-19T02:01:52.988 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:52.989 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:53.990 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:54.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.103+0000 7f07dfdcf640 1 Processor -- start 2024-03-19T02:01:54.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.103+0000 7f07dfdcf640 1 -- start start 2024-03-19T02:01:54.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.103+0000 7f07dfdcf640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f07d8154290 0x7f07d815e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:54.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.107+0000 7f07dfdcf640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f07d80ca310 0x7f07d8154670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:54.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.107+0000 7f07dfdcf640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07d815ad40 0x7f07d8072900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:54.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.107+0000 7f07dfdcf640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f07d815b2c0 con 0x7f07d80ca310 2024-03-19T02:01:54.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.107+0000 7f07dfdcf640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f07d806c1d0 con 0x7f07d8154290 2024-03-19T02:01:54.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.107+0000 7f07dfdcf640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f07d815e030 con 0x7f07d815ad40 2024-03-19T02:01:54.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.107+0000 7f07dd36a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f07d80ca310 0x7f07d8154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:54.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.107+0000 7f07dd36a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f07d80ca310 0x7f07d8154670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49130/0 (socket says 172.21.15.55:49130) 2024-03-19T02:01:54.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.107+0000 7f07dd36a640 1 -- 172.21.15.55:0/2622766717 learned_addr learned my addr 172.21.15.55:0/2622766717 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:54.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.107+0000 7f07ddb6b640 1 --2- 172.21.15.55:0/2622766717 >> v2:172.21.15.84:3300/0 conn(0x7f07d8154290 0x7f07d815e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:54.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.107+0000 7f07de36c640 1 --2- 172.21.15.55:0/2622766717 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07d815ad40 0x7f07d8072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:54.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.107+0000 7f07dd36a640 1 -- 172.21.15.55:0/2622766717 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07d815ad40 msgr2=0x7f07d8072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:54.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.107+0000 7f07dd36a640 1 --2- 172.21.15.55:0/2622766717 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07d815ad40 0x7f07d8072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:54.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.107+0000 7f07dd36a640 1 -- 172.21.15.55:0/2622766717 >> v2:172.21.15.84:3300/0 conn(0x7f07d8154290 msgr2=0x7f07d815e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:54.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.107+0000 7f07dd36a640 1 --2- 172.21.15.55:0/2622766717 >> v2:172.21.15.84:3300/0 conn(0x7f07d8154290 0x7f07d815e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:54.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.107+0000 7f07dd36a640 1 -- 172.21.15.55:0/2622766717 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d806bc90 con 0x7f07d80ca310 2024-03-19T02:01:54.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.111+0000 7f07dd36a640 1 --2- 172.21.15.55:0/2622766717 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07d80ca310 0x7f07d8154670 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f07d8071360 tx=0x7f07c801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc76e7bb531df97 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:54.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.119+0000 7f07dcb69640 1 -- 172.21.15.55:0/2622766717 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c80aa660 con 0x7f07d80ca310 2024-03-19T02:01:54.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.119+0000 7f07dcb69640 1 -- 172.21.15.55:0/2622766717 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07c80af070 con 0x7f07d80ca310 2024-03-19T02:01:54.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.119+0000 7f07dcb69640 1 -- 172.21.15.55:0/2622766717 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c80aaa20 con 0x7f07d80ca310 2024-03-19T02:01:54.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.119+0000 7f07dfdcf640 1 -- 172.21.15.55:0/2622766717 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07d80ca310 msgr2=0x7f07d8154670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:54.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.119+0000 7f07dfdcf640 1 --2- 172.21.15.55:0/2622766717 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07d80ca310 0x7f07d8154670 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f07d8071360 tx=0x7f07c801c520 comp rx=0 tx=0).stop 2024-03-19T02:01:54.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.119+0000 7f07dfdcf640 1 -- 172.21.15.55:0/2622766717 shutdown_connections 2024-03-19T02:01:54.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.119+0000 7f07dfdcf640 1 --2- 172.21.15.55:0/2622766717 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07d815ad40 0x7f07d8072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:54.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.119+0000 7f07dfdcf640 1 --2- 172.21.15.55:0/2622766717 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07d80ca310 0x7f07d8154670 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:54.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.119+0000 7f07dfdcf640 1 --2- 172.21.15.55:0/2622766717 >> v2:172.21.15.84:3300/0 conn(0x7f07d8154290 0x7f07d815e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:54.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.119+0000 7f07dfdcf640 1 -- 172.21.15.55:0/2622766717 >> 172.21.15.55:0/2622766717 conn(0x7f07d806c580 msgr2=0x7f07d806c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:54.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.119+0000 7f07dfdcf640 1 -- 172.21.15.55:0/2622766717 shutdown_connections 2024-03-19T02:01:54.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.119+0000 7f07dfdcf640 1 -- 172.21.15.55:0/2622766717 wait complete. 2024-03-19T02:01:54.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.119+0000 7f07dfdcf640 1 Processor -- start 2024-03-19T02:01:54.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.119+0000 7f07dfdcf640 1 -- start start 2024-03-19T02:01:54.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.119+0000 7f07dfdcf640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07d80ca310 0x7f07d8157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:54.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.119+0000 7f07dfdcf640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07d8154290 0x7f07d8153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:54.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.119+0000 7f07dfdcf640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07d815ad40 0x7f07d8153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:54.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.119+0000 7f07dfdcf640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f07d80ccb50 con 0x7f07d8154290 2024-03-19T02:01:54.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.119+0000 7f07dfdcf640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f07d806bf30 con 0x7f07d80ca310 2024-03-19T02:01:54.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.119+0000 7f07dfdcf640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f07d80cccd0 con 0x7f07d815ad40 2024-03-19T02:01:54.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.123+0000 7f07dd36a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07d8154290 0x7f07d8153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:54.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.123+0000 7f07ddb6b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07d80ca310 0x7f07d8157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:54.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.123+0000 7f07de36c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07d815ad40 0x7f07d8153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:54.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.123+0000 7f07dd36a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07d8154290 0x7f07d8153700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49132/0 (socket says 172.21.15.55:49132) 2024-03-19T02:01:54.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.123+0000 7f07dd36a640 1 -- 172.21.15.55:0/4239945691 learned_addr learned my addr 172.21.15.55:0/4239945691 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:54.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.127+0000 7f07ddb6b640 1 -- 172.21.15.55:0/4239945691 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07d815ad40 msgr2=0x7f07d8153c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:54.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.127+0000 7f07ddb6b640 1 --2- 172.21.15.55:0/4239945691 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07d815ad40 0x7f07d8153c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:54.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.127+0000 7f07ddb6b640 1 -- 172.21.15.55:0/4239945691 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07d8154290 msgr2=0x7f07d8153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:54.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.127+0000 7f07ddb6b640 1 --2- 172.21.15.55:0/4239945691 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07d8154290 0x7f07d8153700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:54.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.127+0000 7f07ddb6b640 1 -- 172.21.15.55:0/4239945691 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d8164a30 con 0x7f07d80ca310 2024-03-19T02:01:54.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.127+0000 7f07ddb6b640 1 --2- 172.21.15.55:0/4239945691 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07d80ca310 0x7f07d8157120 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f07d400e7f0 tx=0x7f07d400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:54.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.131+0000 7f07ce7fc640 1 -- 172.21.15.55:0/4239945691 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d4017070 con 0x7f07d80ca310 2024-03-19T02:01:54.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.131+0000 7f07ce7fc640 1 -- 172.21.15.55:0/4239945691 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07d400c940 con 0x7f07d80ca310 2024-03-19T02:01:54.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.131+0000 7f07ce7fc640 1 -- 172.21.15.55:0/4239945691 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d400ccf0 con 0x7f07d80ca310 2024-03-19T02:01:54.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.131+0000 7f07dfdcf640 1 -- 172.21.15.55:0/4239945691 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07d81f2f10 con 0x7f07d80ca310 2024-03-19T02:01:54.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.131+0000 7f07dfdcf640 1 -- 172.21.15.55:0/4239945691 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07d81f34d0 con 0x7f07d80ca310 2024-03-19T02:01:54.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.135+0000 7f07dfdcf640 1 -- 172.21.15.55:0/4239945691 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07d8068a40 con 0x7f07d80ca310 2024-03-19T02:01:54.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.159+0000 7f07ce7fc640 1 -- 172.21.15.55:0/4239945691 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f07d401a070 con 0x7f07d80ca310 2024-03-19T02:01:54.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.159+0000 7f07ce7fc640 1 --2- 172.21.15.55:0/4239945691 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f07bc0676b0 0x7f07bc069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:54.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.159+0000 7f07dd36a640 1 --2- 172.21.15.55:0/4239945691 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f07bc0676b0 0x7f07bc069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:54.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.159+0000 7f07dd36a640 1 --2- 172.21.15.55:0/4239945691 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f07bc0676b0 0x7f07bc069b70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f07c801c560 tx=0x7f07c80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:54.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.435+0000 7f07ce7fc640 1 -- 172.21.15.55:0/4239945691 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f07d4054620 con 0x7f07d80ca310 2024-03-19T02:01:54.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.455+0000 7f07ce7fc640 1 -- 172.21.15.55:0/4239945691 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f07d4056580 con 0x7f07d80ca310 2024-03-19T02:01:54.657 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.659+0000 7f07dfdcf640 1 -- 172.21.15.55:0/4239945691 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07d80caca0 con 0x7f07d80ca310 2024-03-19T02:01:54.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.691+0000 7f07ce7fc640 1 -- 172.21.15.55:0/4239945691 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f07d405a620 con 0x7f07d80ca310 2024-03-19T02:01:54.688 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:54.688 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:54.688 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:54.692 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.695+0000 7f07dfdcf640 1 -- 172.21.15.55:0/4239945691 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f07bc0676b0 msgr2=0x7f07bc069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:54.692 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.695+0000 7f07dfdcf640 1 --2- 172.21.15.55:0/4239945691 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f07bc0676b0 0x7f07bc069b70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f07c801c560 tx=0x7f07c80a7000 comp rx=0 tx=0).stop 2024-03-19T02:01:54.692 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.695+0000 7f07dfdcf640 1 -- 172.21.15.55:0/4239945691 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07d80ca310 msgr2=0x7f07d8157120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:54.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.695+0000 7f07dfdcf640 1 --2- 172.21.15.55:0/4239945691 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07d80ca310 0x7f07d8157120 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f07d400e7f0 tx=0x7f07d400ecb0 comp rx=0 tx=0).stop 2024-03-19T02:01:54.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.695+0000 7f07dfdcf640 1 -- 172.21.15.55:0/4239945691 shutdown_connections 2024-03-19T02:01:54.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.695+0000 7f07dfdcf640 1 --2- 172.21.15.55:0/4239945691 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f07bc0676b0 0x7f07bc069b70 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:54.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.695+0000 7f07dfdcf640 1 --2- 172.21.15.55:0/4239945691 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f07d815ad40 0x7f07d8153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:54.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.695+0000 7f07dfdcf640 1 --2- 172.21.15.55:0/4239945691 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f07d8154290 0x7f07d8153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:54.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.695+0000 7f07dfdcf640 1 --2- 172.21.15.55:0/4239945691 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07d80ca310 0x7f07d8157120 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:54.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.695+0000 7f07dfdcf640 1 -- 172.21.15.55:0/4239945691 >> 172.21.15.55:0/4239945691 conn(0x7f07d806c580 msgr2=0x7f07d806c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:54.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.695+0000 7f07dfdcf640 1 -- 172.21.15.55:0/4239945691 shutdown_connections 2024-03-19T02:01:54.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:54.695+0000 7f07dfdcf640 1 -- 172.21.15.55:0/4239945691 wait complete. 2024-03-19T02:01:55.313 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:55.313 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:56.315 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:56.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.427+0000 7ffaef7a6640 1 Processor -- start 2024-03-19T02:01:56.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.427+0000 7ffaef7a6640 1 -- start start 2024-03-19T02:01:56.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.431+0000 7ffaef7a6640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ffae806c000 0x7ffae814e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:56.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.431+0000 7ffaef7a6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffae8158330 con 0x7ffae80cd470 2024-03-19T02:01:56.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.431+0000 7ffaef7a6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffae81585f0 con 0x7ffae80cd090 2024-03-19T02:01:56.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.431+0000 7ffaef7a6640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7ffae8158e90 con 0x7ffae806c000 2024-03-19T02:01:56.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.431+0000 7ffaedd43640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ffae806c000 0x7ffae814e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:56.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.435+0000 7ffaedd43640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ffae806c000 0x7ffae814e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:51562/0 (socket says 172.21.15.55:51562) 2024-03-19T02:01:56.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.435+0000 7ffaedd43640 1 -- 172.21.15.55:0/4214746541 learned_addr learned my addr 172.21.15.55:0/4214746541 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:56.437 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.439+0000 7ffaedd43640 1 -- 172.21.15.55:0/4214746541 >> v1:172.21.15.84:6789/0 conn(0x7ffae80cd090 legacy=0x7ffae80732b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:56.437 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.439+0000 7ffaedd43640 1 -- 172.21.15.55:0/4214746541 >> v1:172.21.15.55:6789/0 conn(0x7ffae80cd470 legacy=0x7ffae806b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:56.437 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.439+0000 7ffaedd43640 1 -- 172.21.15.55:0/4214746541 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffae81588d0 con 0x7ffae806c000 2024-03-19T02:01:56.445 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.447+0000 7ffaedd43640 1 --2- 172.21.15.55:0/4214746541 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffae806c000 0x7ffae814e4e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ffae0009e50 tx=0x7ffae001c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a61f69461813b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:56.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.455+0000 7ffadffff640 1 -- 172.21.15.55:0/4214746541 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae00a56c0 con 0x7ffae806c000 2024-03-19T02:01:56.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.455+0000 7ffadffff640 1 -- 172.21.15.55:0/4214746541 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffae00a5860 con 0x7ffae806c000 2024-03-19T02:01:56.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.455+0000 7ffadffff640 1 -- 172.21.15.55:0/4214746541 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae00a5c30 con 0x7ffae806c000 2024-03-19T02:01:56.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.455+0000 7ffaef7a6640 1 -- 172.21.15.55:0/4214746541 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffae806c000 msgr2=0x7ffae814e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:56.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.455+0000 7ffaef7a6640 1 --2- 172.21.15.55:0/4214746541 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffae806c000 0x7ffae814e4e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ffae0009e50 tx=0x7ffae001c9f0 comp rx=0 tx=0).stop 2024-03-19T02:01:56.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.455+0000 7ffaef7a6640 1 -- 172.21.15.55:0/4214746541 shutdown_connections 2024-03-19T02:01:56.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.455+0000 7ffaef7a6640 1 --2- 172.21.15.55:0/4214746541 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffae806c000 0x7ffae814e4e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:56.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.455+0000 7ffaef7a6640 1 -- 172.21.15.55:0/4214746541 >> 172.21.15.55:0/4214746541 conn(0x7ffae806e670 msgr2=0x7ffae806ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:56.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.455+0000 7ffaef7a6640 1 -- 172.21.15.55:0/4214746541 shutdown_connections 2024-03-19T02:01:56.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.455+0000 7ffaef7a6640 1 -- 172.21.15.55:0/4214746541 wait complete. 2024-03-19T02:01:56.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.455+0000 7ffaef7a6640 1 Processor -- start 2024-03-19T02:01:56.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.455+0000 7ffaef7a6640 1 -- start start 2024-03-19T02:01:56.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.459+0000 7ffaef7a6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffae806c000 0x7ffae813d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:56.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.459+0000 7ffaef7a6640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffae80cd090 0x7ffae80c35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:56.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.459+0000 7ffaef7a6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffae80cd470 0x7ffae813f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:56.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.459+0000 7ffaef7a6640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ffae81662b0 con 0x7ffae80cd470 2024-03-19T02:01:56.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.459+0000 7ffaef7a6640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ffae8166130 con 0x7ffae80cd090 2024-03-19T02:01:56.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.459+0000 7ffaef7a6640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ffae8166430 con 0x7ffae806c000 2024-03-19T02:01:56.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.459+0000 7ffaedd43640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffae80cd470 0x7ffae813f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:56.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.459+0000 7ffaed542640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffae806c000 0x7ffae813d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:56.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.459+0000 7ffaedd43640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffae80cd470 0x7ffae813f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49134/0 (socket says 172.21.15.55:49134) 2024-03-19T02:01:56.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.459+0000 7ffaedd43640 1 -- 172.21.15.55:0/106423828 learned_addr learned my addr 172.21.15.55:0/106423828 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:56.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.463+0000 7ffaedd43640 1 -- 172.21.15.55:0/106423828 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffae806c000 msgr2=0x7ffae813d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:56.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.463+0000 7ffaedd43640 1 --2- 172.21.15.55:0/106423828 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffae806c000 0x7ffae813d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:56.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.463+0000 7ffaedd43640 1 -- 172.21.15.55:0/106423828 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffae80cd090 msgr2=0x7ffae80c35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:56.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.463+0000 7ffaedd43640 1 --2- 172.21.15.55:0/106423828 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffae80cd090 0x7ffae80c35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:56.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.463+0000 7ffaedd43640 1 -- 172.21.15.55:0/106423828 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffae80c63f0 con 0x7ffae80cd470 2024-03-19T02:01:56.462 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.463+0000 7ffaedd43640 1 --2- 172.21.15.55:0/106423828 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffae80cd470 0x7ffae813f280 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ffae0009e50 tx=0x7ffae0019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:56.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.467+0000 7ffaddffb640 1 -- 172.21.15.55:0/106423828 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae00b8670 con 0x7ffae80cd470 2024-03-19T02:01:56.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.467+0000 7ffaddffb640 1 -- 172.21.15.55:0/106423828 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffae00be070 con 0x7ffae80cd470 2024-03-19T02:01:56.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.471+0000 7ffaef7a6640 1 -- 172.21.15.55:0/106423828 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffae80c46f0 con 0x7ffae80cd470 2024-03-19T02:01:56.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.471+0000 7ffaddffb640 1 -- 172.21.15.55:0/106423828 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae00b8a70 con 0x7ffae80cd470 2024-03-19T02:01:56.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.471+0000 7ffaef7a6640 1 -- 172.21.15.55:0/106423828 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffae80c4990 con 0x7ffae80cd470 2024-03-19T02:01:56.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.471+0000 7ffaef7a6640 1 -- 172.21.15.55:0/106423828 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffae81f3970 con 0x7ffae80cd470 2024-03-19T02:01:56.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.491+0000 7ffaddffb640 1 -- 172.21.15.55:0/106423828 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ffae00b8c10 con 0x7ffae80cd470 2024-03-19T02:01:56.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.491+0000 7ffaddffb640 1 --2- 172.21.15.55:0/106423828 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffacc0676b0 0x7ffacc069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:56.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.491+0000 7ffaed542640 1 --2- 172.21.15.55:0/106423828 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffacc0676b0 0x7ffacc069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:56.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.491+0000 7ffaed542640 1 --2- 172.21.15.55:0/106423828 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffacc0676b0 0x7ffacc069b70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ffae4001770 tx=0x7ffae4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:56.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.495+0000 7ffaddffb640 1 -- 172.21.15.55:0/106423828 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7ffae00efa00 con 0x7ffae80cd470 2024-03-19T02:01:56.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.775+0000 7ffaddffb640 1 -- 172.21.15.55:0/106423828 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ffae80c63f0 con 0x7ffae80cd470 2024-03-19T02:01:56.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:56.975+0000 7ffaef7a6640 1 -- 172.21.15.55:0/106423828 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffae8068a40 con 0x7ffae80cd470 2024-03-19T02:01:57.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:57.003+0000 7ffaddffb640 1 -- 172.21.15.55:0/106423828 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7ffae00165e0 con 0x7ffae80cd470 2024-03-19T02:01:57.001 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:57.001 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:57.001 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:57.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:57.007+0000 7ffaef7a6640 1 -- 172.21.15.55:0/106423828 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffacc0676b0 msgr2=0x7ffacc069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:57.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:57.007+0000 7ffaef7a6640 1 --2- 172.21.15.55:0/106423828 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffacc0676b0 0x7ffacc069b70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ffae4001770 tx=0x7ffae4009040 comp rx=0 tx=0).stop 2024-03-19T02:01:57.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:57.007+0000 7ffaef7a6640 1 -- 172.21.15.55:0/106423828 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffae80cd470 msgr2=0x7ffae813f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:57.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:57.007+0000 7ffaef7a6640 1 --2- 172.21.15.55:0/106423828 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffae80cd470 0x7ffae813f280 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ffae0009e50 tx=0x7ffae0019790 comp rx=0 tx=0).stop 2024-03-19T02:01:57.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:57.007+0000 7ffaef7a6640 1 -- 172.21.15.55:0/106423828 shutdown_connections 2024-03-19T02:01:57.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:57.007+0000 7ffaef7a6640 1 --2- 172.21.15.55:0/106423828 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ffacc0676b0 0x7ffacc069b70 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:57.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:57.007+0000 7ffaef7a6640 1 --2- 172.21.15.55:0/106423828 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffae80cd470 0x7ffae813f280 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:57.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:57.007+0000 7ffaef7a6640 1 --2- 172.21.15.55:0/106423828 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffae80cd090 0x7ffae80c35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:57.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:57.007+0000 7ffaef7a6640 1 --2- 172.21.15.55:0/106423828 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ffae806c000 0x7ffae813d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:57.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:57.007+0000 7ffaef7a6640 1 -- 172.21.15.55:0/106423828 >> 172.21.15.55:0/106423828 conn(0x7ffae806e670 msgr2=0x7ffae814d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:57.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:57.007+0000 7ffaef7a6640 1 -- 172.21.15.55:0/106423828 shutdown_connections 2024-03-19T02:01:57.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:57.007+0000 7ffaef7a6640 1 -- 172.21.15.55:0/106423828 wait complete. 2024-03-19T02:01:57.629 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:01:57.629 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:01:58.630 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:01:58.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.751+0000 7f4bdfccf640 1 Processor -- start 2024-03-19T02:01:58.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.751+0000 7f4bdfccf640 1 -- start start 2024-03-19T02:01:58.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.755+0000 7f4bdfccf640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4bd81510f0 0x7f4bd80c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:58.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.755+0000 7f4bdfccf640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4bd815a410 0x7f4bd80708a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:58.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.755+0000 7f4bdfccf640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bd80c3f60 con 0x7f4bd80c73c0 2024-03-19T02:01:58.752 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.755+0000 7f4bdfccf640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f4bd81514d0 con 0x7f4bd815a410 2024-03-19T02:01:58.752 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.755+0000 7f4bdfccf640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4bd80c6870 con 0x7f4bd81510f0 2024-03-19T02:01:58.752 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.755+0000 7f4bdd26a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4bd815a410 0x7f4bd80708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:58.752 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.755+0000 7f4bdda6b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4bd81510f0 0x7f4bd80c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:58.753 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.755+0000 7f4bdd26a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4bd815a410 0x7f4bd80708a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41088/0 (socket says 172.21.15.55:41088) 2024-03-19T02:01:58.753 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.755+0000 7f4bdd26a640 1 -- 172.21.15.55:0/1990071381 learned_addr learned my addr 172.21.15.55:0/1990071381 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:58.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.759+0000 7f4bdd26a640 1 -- 172.21.15.55:0/1990071381 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4bd81510f0 msgr2=0x7f4bd80c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:01:58.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.759+0000 7f4bdd26a640 1 --2- 172.21.15.55:0/1990071381 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4bd81510f0 0x7f4bd80c61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:58.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.759+0000 7f4bdd26a640 1 -- 172.21.15.55:0/1990071381 >> v1:172.21.15.84:6789/0 conn(0x7f4bd80c73c0 legacy=0x7f4bd8072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:58.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.759+0000 7f4bdd26a640 1 -- 172.21.15.55:0/1990071381 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd80c6b50 con 0x7f4bd815a410 2024-03-19T02:01:58.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.763+0000 7f4bdd26a640 1 --2- 172.21.15.55:0/1990071381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bd815a410 0x7f4bd80708a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8009870 tx=0x7f4bc801ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ad3081fbe5d26 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:58.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.767+0000 7f4bdca69640 1 -- 172.21.15.55:0/1990071381 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc80b5070 con 0x7f4bd815a410 2024-03-19T02:01:58.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.767+0000 7f4bdca69640 1 -- 172.21.15.55:0/1990071381 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bc80a8a50 con 0x7f4bd815a410 2024-03-19T02:01:58.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.767+0000 7f4bdca69640 1 -- 172.21.15.55:0/1990071381 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc80a8e20 con 0x7f4bd815a410 2024-03-19T02:01:58.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.767+0000 7f4bdfccf640 1 -- 172.21.15.55:0/1990071381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bd815a410 msgr2=0x7f4bd80708a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:58.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.767+0000 7f4bdfccf640 1 --2- 172.21.15.55:0/1990071381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bd815a410 0x7f4bd80708a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8009870 tx=0x7f4bc801ed50 comp rx=0 tx=0).stop 2024-03-19T02:01:58.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.767+0000 7f4bdfccf640 1 -- 172.21.15.55:0/1990071381 shutdown_connections 2024-03-19T02:01:58.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.767+0000 7f4bdfccf640 1 --2- 172.21.15.55:0/1990071381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bd815a410 0x7f4bd80708a0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:58.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.767+0000 7f4bdfccf640 1 --2- 172.21.15.55:0/1990071381 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4bd81510f0 0x7f4bd80c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:58.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.767+0000 7f4bdfccf640 1 -- 172.21.15.55:0/1990071381 >> 172.21.15.55:0/1990071381 conn(0x7f4bd80cc9c0 msgr2=0x7f4bd80ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:58.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.767+0000 7f4bdfccf640 1 -- 172.21.15.55:0/1990071381 shutdown_connections 2024-03-19T02:01:58.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.767+0000 7f4bdfccf640 1 -- 172.21.15.55:0/1990071381 wait complete. 2024-03-19T02:01:58.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.771+0000 7f4bdfccf640 1 Processor -- start 2024-03-19T02:01:58.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.771+0000 7f4bdfccf640 1 -- start start 2024-03-19T02:01:58.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.771+0000 7f4bdfccf640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4bd80c73c0 0x7f4bd80c37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:58.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.771+0000 7f4bdfccf640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4bd81510f0 0x7f4bd8148640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:58.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.771+0000 7f4bdfccf640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bd815a410 0x7f4bd814bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:58.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.771+0000 7f4bdfccf640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd80c6e30 con 0x7f4bd815a410 2024-03-19T02:01:58.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.771+0000 7f4bdfccf640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd80c44e0 con 0x7f4bd81510f0 2024-03-19T02:01:58.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.771+0000 7f4bdfccf640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4bd80c4a00 con 0x7f4bd80c73c0 2024-03-19T02:01:58.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.771+0000 7f4bde26c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bd815a410 0x7f4bd814bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:58.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.771+0000 7f4bde26c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bd815a410 0x7f4bd814bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41092/0 (socket says 172.21.15.55:41092) 2024-03-19T02:01:58.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.771+0000 7f4bde26c640 1 -- 172.21.15.55:0/3741737869 learned_addr learned my addr 172.21.15.55:0/3741737869 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:01:58.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.771+0000 7f4bdda6b640 1 --2- 172.21.15.55:0/3741737869 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4bd80c73c0 0x7f4bd80c37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:58.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.771+0000 7f4bdd26a640 1 --2- 172.21.15.55:0/3741737869 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4bd81510f0 0x7f4bd8148640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:58.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.775+0000 7f4bde26c640 1 -- 172.21.15.55:0/3741737869 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4bd80c73c0 msgr2=0x7f4bd80c37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:58.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.775+0000 7f4bde26c640 1 --2- 172.21.15.55:0/3741737869 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4bd80c73c0 0x7f4bd80c37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:58.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.775+0000 7f4bde26c640 1 -- 172.21.15.55:0/3741737869 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4bd81510f0 msgr2=0x7f4bd8148640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:58.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.775+0000 7f4bde26c640 1 --2- 172.21.15.55:0/3741737869 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4bd81510f0 0x7f4bd8148640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:58.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.775+0000 7f4bde26c640 1 -- 172.21.15.55:0/3741737869 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd814deb0 con 0x7f4bd815a410 2024-03-19T02:01:58.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.779+0000 7f4bde26c640 1 --2- 172.21.15.55:0/3741737869 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bd815a410 0x7f4bd814bdb0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4bd000c770 tx=0x7f4bd000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:58.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.783+0000 7f4bce7fc640 1 -- 172.21.15.55:0/3741737869 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0003b80 con 0x7f4bd815a410 2024-03-19T02:01:58.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.783+0000 7f4bdfccf640 1 -- 172.21.15.55:0/3741737869 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bd81f3320 con 0x7f4bd815a410 2024-03-19T02:01:58.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.783+0000 7f4bdfccf640 1 -- 172.21.15.55:0/3741737869 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bd814e150 con 0x7f4bd815a410 2024-03-19T02:01:58.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.783+0000 7f4bdfccf640 1 -- 172.21.15.55:0/3741737869 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bd81f34f0 con 0x7f4bd815a410 2024-03-19T02:01:58.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.783+0000 7f4bce7fc640 1 -- 172.21.15.55:0/3741737869 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bd0003d20 con 0x7f4bd815a410 2024-03-19T02:01:58.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:58.783+0000 7f4bce7fc640 1 -- 172.21.15.55:0/3741737869 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd000aa50 con 0x7f4bd815a410 2024-03-19T02:01:59.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:59.071+0000 7f4bce7fc640 1 -- 172.21.15.55:0/3741737869 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f4bd000abf0 con 0x7f4bd815a410 2024-03-19T02:01:59.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:59.071+0000 7f4bce7fc640 1 --2- 172.21.15.55:0/3741737869 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4bbc067660 0x7f4bbc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:01:59.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:59.071+0000 7f4bdda6b640 1 --2- 172.21.15.55:0/3741737869 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4bbc067660 0x7f4bbc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:01:59.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:59.071+0000 7f4bdda6b640 1 --2- 172.21.15.55:0/3741737869 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4bbc067660 0x7f4bbc069b20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009b60 tx=0x7f4bd4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:01:59.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:59.111+0000 7f4bce7fc640 1 -- 172.21.15.55:0/3741737869 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f4bd0054810 con 0x7f4bd815a410 2024-03-19T02:01:59.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:59.111+0000 7f4bce7fc640 1 -- 172.21.15.55:0/3741737869 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4bd0056770 con 0x7f4bd815a410 2024-03-19T02:01:59.349 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:59.351+0000 7f4bdfccf640 1 -- 172.21.15.55:0/3741737869 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4bd8068a40 con 0x7f4bd815a410 2024-03-19T02:01:59.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:59.435+0000 7f4bce7fc640 1 -- 172.21.15.55:0/3741737869 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f4bd005a810 con 0x7f4bd815a410 2024-03-19T02:01:59.432 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:01:59.433 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:01:59.433 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:01:59.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:59.435+0000 7f4bdfccf640 1 -- 172.21.15.55:0/3741737869 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4bbc067660 msgr2=0x7f4bbc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:59.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:59.435+0000 7f4bdfccf640 1 --2- 172.21.15.55:0/3741737869 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4bbc067660 0x7f4bbc069b20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009b60 tx=0x7f4bd4009190 comp rx=0 tx=0).stop 2024-03-19T02:01:59.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:59.435+0000 7f4bdfccf640 1 -- 172.21.15.55:0/3741737869 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bd815a410 msgr2=0x7f4bd814bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:01:59.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:59.435+0000 7f4bdfccf640 1 --2- 172.21.15.55:0/3741737869 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bd815a410 0x7f4bd814bdb0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4bd000c770 tx=0x7f4bd000cc30 comp rx=0 tx=0).stop 2024-03-19T02:01:59.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:59.435+0000 7f4bdfccf640 1 -- 172.21.15.55:0/3741737869 shutdown_connections 2024-03-19T02:01:59.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:59.435+0000 7f4bdfccf640 1 --2- 172.21.15.55:0/3741737869 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4bbc067660 0x7f4bbc069b20 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:59.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:59.435+0000 7f4bdfccf640 1 --2- 172.21.15.55:0/3741737869 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bd815a410 0x7f4bd814bdb0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:59.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:59.435+0000 7f4bdfccf640 1 --2- 172.21.15.55:0/3741737869 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4bd81510f0 0x7f4bd8148640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:59.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:59.435+0000 7f4bdfccf640 1 --2- 172.21.15.55:0/3741737869 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4bd80c73c0 0x7f4bd80c37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:01:59.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:59.435+0000 7f4bdfccf640 1 -- 172.21.15.55:0/3741737869 >> 172.21.15.55:0/3741737869 conn(0x7f4bd80cc9c0 msgr2=0x7f4bd814a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:01:59.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:59.435+0000 7f4bdfccf640 1 -- 172.21.15.55:0/3741737869 shutdown_connections 2024-03-19T02:01:59.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:01:59.435+0000 7f4bdfccf640 1 -- 172.21.15.55:0/3741737869 wait complete. 2024-03-19T02:02:00.060 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:00.060 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:01.061 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:01.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.183+0000 7fdb62c2d640 1 Processor -- start 2024-03-19T02:02:01.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.183+0000 7fdb62c2d640 1 -- start start 2024-03-19T02:02:01.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.183+0000 7fdb62c2d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fdb5c14eaa0 0x7fdb5c06b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:01.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.183+0000 7fdb62c2d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb5c14ee80 con 0x7fdb5c06bec0 2024-03-19T02:02:01.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.183+0000 7fdb62c2d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb5c0c7540 con 0x7fdb5c1585f0 2024-03-19T02:02:01.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.183+0000 7fdb62c2d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fdb5c0c77c0 con 0x7fdb5c14eaa0 2024-03-19T02:02:01.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.183+0000 7fdb609c9640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fdb5c14eaa0 0x7fdb5c06b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:01.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.183+0000 7fdb611ca640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fdb5c1585f0 0x7fdb5c162940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:37378/0 (socket says 172.21.15.55:37378) 2024-03-19T02:02:01.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.183+0000 7fdb611ca640 1 -- 172.21.15.55:0/2834647447 learned_addr learned my addr 172.21.15.55:0/2834647447 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:02:01.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.191+0000 7fdb609c9640 1 -- 172.21.15.55:0/2834647447 >> v1:172.21.15.84:6789/0 conn(0x7fdb5c1585f0 legacy=0x7fdb5c162940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:01.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.191+0000 7fdb609c9640 1 -- 172.21.15.55:0/2834647447 >> v1:172.21.15.55:6789/0 conn(0x7fdb5c06bec0 legacy=0x7fdb5c0c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:01.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.191+0000 7fdb609c9640 1 -- 172.21.15.55:0/2834647447 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb5c0c7aa0 con 0x7fdb5c14eaa0 2024-03-19T02:02:01.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.195+0000 7fdb609c9640 1 --2- 172.21.15.55:0/2834647447 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb5c14eaa0 0x7fdb5c06b980 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdb500097c0 tx=0x7fdb5001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4322f94e4db0461 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:01.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.199+0000 7fdb5b7fe640 1 -- 172.21.15.55:0/2834647447 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb500ab610 con 0x7fdb5c14eaa0 2024-03-19T02:02:01.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.199+0000 7fdb5b7fe640 1 -- 172.21.15.55:0/2834647447 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb500b0070 con 0x7fdb5c14eaa0 2024-03-19T02:02:01.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.199+0000 7fdb5b7fe640 1 -- 172.21.15.55:0/2834647447 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb500ab9d0 con 0x7fdb5c14eaa0 2024-03-19T02:02:01.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.199+0000 7fdb62c2d640 1 -- 172.21.15.55:0/2834647447 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb5c14eaa0 msgr2=0x7fdb5c06b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:01.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.199+0000 7fdb62c2d640 1 --2- 172.21.15.55:0/2834647447 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb5c14eaa0 0x7fdb5c06b980 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdb500097c0 tx=0x7fdb5001c410 comp rx=0 tx=0).stop 2024-03-19T02:02:01.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.199+0000 7fdb62c2d640 1 -- 172.21.15.55:0/2834647447 shutdown_connections 2024-03-19T02:02:01.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.199+0000 7fdb62c2d640 1 --2- 172.21.15.55:0/2834647447 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb5c14eaa0 0x7fdb5c06b980 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:01.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.199+0000 7fdb62c2d640 1 -- 172.21.15.55:0/2834647447 >> 172.21.15.55:0/2834647447 conn(0x7fdb5c0cc9c0 msgr2=0x7fdb5c0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:01.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.199+0000 7fdb62c2d640 1 -- 172.21.15.55:0/2834647447 shutdown_connections 2024-03-19T02:02:01.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.199+0000 7fdb62c2d640 1 -- 172.21.15.55:0/2834647447 wait complete. 2024-03-19T02:02:01.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.203+0000 7fdb62c2d640 1 Processor -- start 2024-03-19T02:02:01.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.203+0000 7fdb62c2d640 1 -- start start 2024-03-19T02:02:01.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.203+0000 7fdb62c2d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdb5c06bec0 0x7fdb5c1578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:01.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.203+0000 7fdb62c2d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdb5c14eaa0 0x7fdb5c156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:01.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.203+0000 7fdb62c2d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb5c1585f0 0x7fdb5c0c5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:01.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.203+0000 7fdb62c2d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5c0c67e0 con 0x7fdb5c1585f0 2024-03-19T02:02:01.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.203+0000 7fdb62c2d640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5c158e10 con 0x7fdb5c06bec0 2024-03-19T02:02:01.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.203+0000 7fdb62c2d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fdb5c0c6960 con 0x7fdb5c14eaa0 2024-03-19T02:02:01.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.203+0000 7fdb611ca640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb5c1585f0 0x7fdb5c0c5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:01.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.203+0000 7fdb611ca640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb5c1585f0 0x7fdb5c0c5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41100/0 (socket says 172.21.15.55:41100) 2024-03-19T02:02:01.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.203+0000 7fdb611ca640 1 -- 172.21.15.55:0/2647734401 learned_addr learned my addr 172.21.15.55:0/2647734401 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:01.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.203+0000 7fdb5bfff640 1 --2- 172.21.15.55:0/2647734401 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdb5c14eaa0 0x7fdb5c156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:01.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.203+0000 7fdb609c9640 1 --2- 172.21.15.55:0/2647734401 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdb5c06bec0 0x7fdb5c1578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:01.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.207+0000 7fdb611ca640 1 -- 172.21.15.55:0/2647734401 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdb5c14eaa0 msgr2=0x7fdb5c156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:01.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.207+0000 7fdb611ca640 1 --2- 172.21.15.55:0/2647734401 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdb5c14eaa0 0x7fdb5c156060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:01.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.207+0000 7fdb611ca640 1 -- 172.21.15.55:0/2647734401 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdb5c06bec0 msgr2=0x7fdb5c1578c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:01.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.207+0000 7fdb611ca640 1 --2- 172.21.15.55:0/2647734401 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdb5c06bec0 0x7fdb5c1578c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:01.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.207+0000 7fdb611ca640 1 -- 172.21.15.55:0/2647734401 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb5c14dee0 con 0x7fdb5c1585f0 2024-03-19T02:02:01.209 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.211+0000 7fdb611ca640 1 --2- 172.21.15.55:0/2647734401 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb5c1585f0 0x7fdb5c0c5470 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fdb5400cab0 tx=0x7fdb5400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:01.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.215+0000 7fdb597fa640 1 -- 172.21.15.55:0/2647734401 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54017070 con 0x7fdb5c1585f0 2024-03-19T02:02:01.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.215+0000 7fdb597fa640 1 -- 172.21.15.55:0/2647734401 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb5400ac30 con 0x7fdb5c1585f0 2024-03-19T02:02:01.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.215+0000 7fdb62c2d640 1 -- 172.21.15.55:0/2647734401 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb5c14e1a0 con 0x7fdb5c1585f0 2024-03-19T02:02:01.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.215+0000 7fdb597fa640 1 -- 172.21.15.55:0/2647734401 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54012400 con 0x7fdb5c1585f0 2024-03-19T02:02:01.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.215+0000 7fdb62c2d640 1 -- 172.21.15.55:0/2647734401 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb5c0cbd40 con 0x7fdb5c1585f0 2024-03-19T02:02:01.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.215+0000 7fdb62c2d640 1 -- 172.21.15.55:0/2647734401 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb5c068a40 con 0x7fdb5c1585f0 2024-03-19T02:02:01.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.235+0000 7fdb597fa640 1 -- 172.21.15.55:0/2647734401 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fdb5401a070 con 0x7fdb5c1585f0 2024-03-19T02:02:01.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.235+0000 7fdb597fa640 1 --2- 172.21.15.55:0/2647734401 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdb400676b0 0x7fdb40069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:01.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.239+0000 7fdb609c9640 1 --2- 172.21.15.55:0/2647734401 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdb400676b0 0x7fdb40069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:01.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.239+0000 7fdb609c9640 1 --2- 172.21.15.55:0/2647734401 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdb400676b0 0x7fdb40069b70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fdb500af000 tx=0x7fdb500a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:01.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.495+0000 7fdb597fa640 1 -- 172.21.15.55:0/2647734401 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fdb540543e0 con 0x7fdb5c1585f0 2024-03-19T02:02:01.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.495+0000 7fdb597fa640 1 -- 172.21.15.55:0/2647734401 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fdb5c0c7540 con 0x7fdb5c1585f0 2024-03-19T02:02:01.688 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.691+0000 7fdb62c2d640 1 -- 172.21.15.55:0/2647734401 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb5c1558e0 con 0x7fdb5c1585f0 2024-03-19T02:02:01.703 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.703+0000 7fdb597fa640 1 -- 172.21.15.55:0/2647734401 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fdb54056340 con 0x7fdb5c1585f0 2024-03-19T02:02:01.703 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:01.703 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:01.703 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:01.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.707+0000 7fdb62c2d640 1 -- 172.21.15.55:0/2647734401 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdb400676b0 msgr2=0x7fdb40069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:01.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.707+0000 7fdb62c2d640 1 --2- 172.21.15.55:0/2647734401 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdb400676b0 0x7fdb40069b70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fdb500af000 tx=0x7fdb500a7000 comp rx=0 tx=0).stop 2024-03-19T02:02:01.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.707+0000 7fdb62c2d640 1 -- 172.21.15.55:0/2647734401 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb5c1585f0 msgr2=0x7fdb5c0c5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:01.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.707+0000 7fdb62c2d640 1 --2- 172.21.15.55:0/2647734401 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb5c1585f0 0x7fdb5c0c5470 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fdb5400cab0 tx=0x7fdb5400cf70 comp rx=0 tx=0).stop 2024-03-19T02:02:01.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.707+0000 7fdb62c2d640 1 -- 172.21.15.55:0/2647734401 shutdown_connections 2024-03-19T02:02:01.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.707+0000 7fdb62c2d640 1 --2- 172.21.15.55:0/2647734401 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdb400676b0 0x7fdb40069b70 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:01.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.707+0000 7fdb62c2d640 1 --2- 172.21.15.55:0/2647734401 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb5c1585f0 0x7fdb5c0c5470 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:01.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.707+0000 7fdb62c2d640 1 --2- 172.21.15.55:0/2647734401 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdb5c14eaa0 0x7fdb5c156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:01.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.707+0000 7fdb62c2d640 1 --2- 172.21.15.55:0/2647734401 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdb5c06bec0 0x7fdb5c1578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:01.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.707+0000 7fdb62c2d640 1 -- 172.21.15.55:0/2647734401 >> 172.21.15.55:0/2647734401 conn(0x7fdb5c0cc9c0 msgr2=0x7fdb5c0ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:01.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.707+0000 7fdb62c2d640 1 -- 172.21.15.55:0/2647734401 shutdown_connections 2024-03-19T02:02:01.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:01.707+0000 7fdb62c2d640 1 -- 172.21.15.55:0/2647734401 wait complete. 2024-03-19T02:02:02.326 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:02.326 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:03.327 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:03.441 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.443+0000 7fc624710640 1 Processor -- start 2024-03-19T02:02:03.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.443+0000 7fc624710640 1 -- start start 2024-03-19T02:02:03.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.443+0000 7fc624710640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc61c0ca0c0 0x7fc61c06bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:03.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.443+0000 7fc624710640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc61c070410 0x7fc61c070880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:03.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.443+0000 7fc624710640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc61c15b0c0 con 0x7fc61c154060 2024-03-19T02:02:03.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.443+0000 7fc624710640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fc61c0ccb50 con 0x7fc61c0ca0c0 2024-03-19T02:02:03.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.443+0000 7fc624710640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc61c0ccd80 con 0x7fc61c070410 2024-03-19T02:02:03.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.443+0000 7fc621cab640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc61c0ca0c0 0x7fc61c06bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:03.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.447+0000 7fc622cad640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc61c070410 0x7fc61c070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:03.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.447+0000 7fc6224ac640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fc61c154060 0x7fc61c15a9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:48618/0 (socket says 172.21.15.55:48618) 2024-03-19T02:02:03.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.447+0000 7fc6224ac640 1 -- 172.21.15.55:0/850490722 learned_addr learned my addr 172.21.15.55:0/850490722 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:02:03.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.451+0000 7fc621cab640 1 -- 172.21.15.55:0/850490722 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc61c070410 msgr2=0x7fc61c070880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:03.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.451+0000 7fc621cab640 1 --2- 172.21.15.55:0/850490722 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc61c070410 0x7fc61c070880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:03.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.451+0000 7fc621cab640 1 -- 172.21.15.55:0/850490722 >> v1:172.21.15.55:6789/0 conn(0x7fc61c154060 legacy=0x7fc61c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:03.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.451+0000 7fc621cab640 1 -- 172.21.15.55:0/850490722 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61c0cd040 con 0x7fc61c0ca0c0 2024-03-19T02:02:03.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.455+0000 7fc621cab640 1 --2- 172.21.15.55:0/850490722 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc61c0ca0c0 0x7fc61c06bcb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fc60c0097c0 tx=0x7fc60c01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64c1ab93bac9c2bb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:03.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.459+0000 7fc6214aa640 1 -- 172.21.15.55:0/850490722 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60c0b5070 con 0x7fc61c0ca0c0 2024-03-19T02:02:03.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.459+0000 7fc6214aa640 1 -- 172.21.15.55:0/850490722 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc60c0a8990 con 0x7fc61c0ca0c0 2024-03-19T02:02:03.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.459+0000 7fc6214aa640 1 -- 172.21.15.55:0/850490722 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60c0a8d60 con 0x7fc61c0ca0c0 2024-03-19T02:02:03.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.459+0000 7fc624710640 1 -- 172.21.15.55:0/850490722 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc61c0ca0c0 msgr2=0x7fc61c06bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:03.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.459+0000 7fc624710640 1 --2- 172.21.15.55:0/850490722 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc61c0ca0c0 0x7fc61c06bcb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fc60c0097c0 tx=0x7fc60c01ec90 comp rx=0 tx=0).stop 2024-03-19T02:02:03.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.459+0000 7fc624710640 1 -- 172.21.15.55:0/850490722 shutdown_connections 2024-03-19T02:02:03.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.459+0000 7fc624710640 1 --2- 172.21.15.55:0/850490722 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc61c070410 0x7fc61c070880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:03.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.459+0000 7fc624710640 1 --2- 172.21.15.55:0/850490722 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc61c0ca0c0 0x7fc61c06bcb0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:03.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.459+0000 7fc624710640 1 -- 172.21.15.55:0/850490722 >> 172.21.15.55:0/850490722 conn(0x7fc61c06c3b0 msgr2=0x7fc61c06c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:03.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.459+0000 7fc624710640 1 -- 172.21.15.55:0/850490722 shutdown_connections 2024-03-19T02:02:03.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.459+0000 7fc624710640 1 -- 172.21.15.55:0/850490722 wait complete. 2024-03-19T02:02:03.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.459+0000 7fc624710640 1 Processor -- start 2024-03-19T02:02:03.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.459+0000 7fc624710640 1 -- start start 2024-03-19T02:02:03.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.459+0000 7fc624710640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc61c070410 0x7fc61c1e5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:03.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.459+0000 7fc624710640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc61c0ca0c0 0x7fc61c157170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:03.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.459+0000 7fc624710640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc61c154060 0x7fc61c158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:03.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.459+0000 7fc624710640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c0cd8e0 con 0x7fc61c0ca0c0 2024-03-19T02:02:03.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.459+0000 7fc624710640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c0cd600 con 0x7fc61c070410 2024-03-19T02:02:03.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.459+0000 7fc624710640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc61c15a490 con 0x7fc61c154060 2024-03-19T02:02:03.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.463+0000 7fc621cab640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc61c0ca0c0 0x7fc61c157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:03.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.463+0000 7fc622cad640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc61c154060 0x7fc61c158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:03.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.463+0000 7fc6224ac640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc61c070410 0x7fc61c1e5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:03.461 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.463+0000 7fc621cab640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc61c0ca0c0 0x7fc61c157170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41124/0 (socket says 172.21.15.55:41124) 2024-03-19T02:02:03.461 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.463+0000 7fc621cab640 1 -- 172.21.15.55:0/4090531082 learned_addr learned my addr 172.21.15.55:0/4090531082 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:03.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.467+0000 7fc622cad640 1 -- 172.21.15.55:0/4090531082 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc61c070410 msgr2=0x7fc61c1e5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:03.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.467+0000 7fc622cad640 1 --2- 172.21.15.55:0/4090531082 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc61c070410 0x7fc61c1e5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:03.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.467+0000 7fc622cad640 1 -- 172.21.15.55:0/4090531082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc61c0ca0c0 msgr2=0x7fc61c157170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:03.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.467+0000 7fc622cad640 1 --2- 172.21.15.55:0/4090531082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc61c0ca0c0 0x7fc61c157170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:03.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.467+0000 7fc622cad640 1 -- 172.21.15.55:0/4090531082 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61c1efdd0 con 0x7fc61c154060 2024-03-19T02:02:03.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.471+0000 7fc622cad640 1 --2- 172.21.15.55:0/4090531082 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc61c154060 0x7fc61c158960 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc61400ec00 tx=0x7fc61400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:03.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.475+0000 7fc612ffd640 1 -- 172.21.15.55:0/4090531082 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc614018020 con 0x7fc61c154060 2024-03-19T02:02:03.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.475+0000 7fc612ffd640 1 -- 172.21.15.55:0/4090531082 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc61400cd40 con 0x7fc61c154060 2024-03-19T02:02:03.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.475+0000 7fc612ffd640 1 -- 172.21.15.55:0/4090531082 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6140146c0 con 0x7fc61c154060 2024-03-19T02:02:03.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.475+0000 7fc624710640 1 -- 172.21.15.55:0/4090531082 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc61c1e4760 con 0x7fc61c154060 2024-03-19T02:02:03.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.475+0000 7fc624710640 1 -- 172.21.15.55:0/4090531082 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc61c1e4d20 con 0x7fc61c154060 2024-03-19T02:02:03.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.475+0000 7fc624710640 1 -- 172.21.15.55:0/4090531082 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc61c068a40 con 0x7fc61c154060 2024-03-19T02:02:03.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.774+0000 7fc612ffd640 1 -- 172.21.15.55:0/4090531082 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc61401b020 con 0x7fc61c154060 2024-03-19T02:02:03.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.774+0000 7fc612ffd640 1 --2- 172.21.15.55:0/4090531082 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc600067690 0x7fc600069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:03.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.778+0000 7fc6224ac640 1 --2- 172.21.15.55:0/4090531082 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc600067690 0x7fc600069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:03.776 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.778+0000 7fc6224ac640 1 --2- 172.21.15.55:0/4090531082 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc600067690 0x7fc600069b50 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc618000f70 tx=0x7fc618007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:03.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.778+0000 7fc612ffd640 1 -- 172.21.15.55:0/4090531082 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fc614055030 con 0x7fc61c154060 2024-03-19T02:02:03.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.798+0000 7fc612ffd640 1 -- 172.21.15.55:0/4090531082 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc614022030 con 0x7fc61c154060 2024-03-19T02:02:03.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:03.998+0000 7fc624710640 1 -- 172.21.15.55:0/4090531082 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc61c1e63a0 con 0x7fc61c154060 2024-03-19T02:02:04.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:04.022+0000 7fc612ffd640 1 -- 172.21.15.55:0/4090531082 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fc614010040 con 0x7fc61c154060 2024-03-19T02:02:04.020 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:04.020 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:04.021 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:04.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:04.022+0000 7fc624710640 1 -- 172.21.15.55:0/4090531082 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc600067690 msgr2=0x7fc600069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:04.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:04.022+0000 7fc624710640 1 --2- 172.21.15.55:0/4090531082 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc600067690 0x7fc600069b50 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc618000f70 tx=0x7fc618007e90 comp rx=0 tx=0).stop 2024-03-19T02:02:04.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:04.022+0000 7fc624710640 1 -- 172.21.15.55:0/4090531082 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc61c154060 msgr2=0x7fc61c158960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:04.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:04.022+0000 7fc624710640 1 --2- 172.21.15.55:0/4090531082 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc61c154060 0x7fc61c158960 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc61400ec00 tx=0x7fc61400c490 comp rx=0 tx=0).stop 2024-03-19T02:02:04.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:04.022+0000 7fc624710640 1 -- 172.21.15.55:0/4090531082 shutdown_connections 2024-03-19T02:02:04.025 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:04.022+0000 7fc624710640 1 --2- 172.21.15.55:0/4090531082 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc600067690 0x7fc600069b50 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:04.025 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:04.022+0000 7fc624710640 1 --2- 172.21.15.55:0/4090531082 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc61c154060 0x7fc61c158960 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:04.025 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:04.022+0000 7fc624710640 1 --2- 172.21.15.55:0/4090531082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc61c0ca0c0 0x7fc61c157170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:04.025 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:04.022+0000 7fc624710640 1 --2- 172.21.15.55:0/4090531082 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc61c070410 0x7fc61c1e5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:04.025 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:04.022+0000 7fc624710640 1 -- 172.21.15.55:0/4090531082 >> 172.21.15.55:0/4090531082 conn(0x7fc61c06c3b0 msgr2=0x7fc61c06c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:04.026 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:04.022+0000 7fc624710640 1 -- 172.21.15.55:0/4090531082 shutdown_connections 2024-03-19T02:02:04.026 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:04.022+0000 7fc624710640 1 -- 172.21.15.55:0/4090531082 wait complete. 2024-03-19T02:02:04.642 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:04.642 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:05.643 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:05.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.770+0000 7fbf0951c640 1 Processor -- start 2024-03-19T02:02:05.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.770+0000 7fbf0951c640 1 -- start start 2024-03-19T02:02:05.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.774+0000 7fbf0951c640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fbf040c96b0 0x7fbf04070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:05.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.774+0000 7fbf0951c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fbf040c9a90 0x7fbf040c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:05.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.774+0000 7fbf0951c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbf041540c0 0x7fbf04154570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:05.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.774+0000 7fbf0951c640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fbf0415e3a0 con 0x7fbf040c9a90 2024-03-19T02:02:05.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.774+0000 7fbf0951c640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fbf0415e680 con 0x7fbf040c96b0 2024-03-19T02:02:05.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.774+0000 7fbf0951c640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbf0415a330 con 0x7fbf041540c0 2024-03-19T02:02:05.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.774+0000 7fbf027fc640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fbf040c9a90 0x7fbf040c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:05.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.774+0000 7fbf02ffd640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fbf040c96b0 0x7fbf04070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:05.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.774+0000 7fbf027fc640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fbf040c9a90 0x7fbf040c73c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41140/0 (socket says 172.21.15.55:41140) 2024-03-19T02:02:05.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.774+0000 7fbf027fc640 1 -- 172.21.15.55:0/3368894314 learned_addr learned my addr 172.21.15.55:0/3368894314 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:05.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.774+0000 7fbf037fe640 1 --2- 172.21.15.55:0/3368894314 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbf041540c0 0x7fbf04154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:05.776 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.774+0000 7fbf027fc640 1 -- 172.21.15.55:0/3368894314 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbf041540c0 msgr2=0x7fbf04154570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:05.776 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.774+0000 7fbf027fc640 1 --2- 172.21.15.55:0/3368894314 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbf041540c0 0x7fbf04154570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:05.776 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.778+0000 7fbf027fc640 1 -- 172.21.15.55:0/3368894314 >> v2:172.21.15.84:3300/0 conn(0x7fbf040c96b0 msgr2=0x7fbf04070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:05.776 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.778+0000 7fbf027fc640 1 --2- 172.21.15.55:0/3368894314 >> v2:172.21.15.84:3300/0 conn(0x7fbf040c96b0 0x7fbf04070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:05.776 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.778+0000 7fbf027fc640 1 -- 172.21.15.55:0/3368894314 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0415a610 con 0x7fbf040c9a90 2024-03-19T02:02:05.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.778+0000 7fbf027fc640 1 --2- 172.21.15.55:0/3368894314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbf040c9a90 0x7fbf040c73c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fbf04072b40 tx=0x7fbef801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a7940bd132fc737 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:05.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.786+0000 7fbf01ffb640 1 -- 172.21.15.55:0/3368894314 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef80b0070 con 0x7fbf040c9a90 2024-03-19T02:02:05.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.786+0000 7fbf01ffb640 1 -- 172.21.15.55:0/3368894314 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbef80ab600 con 0x7fbf040c9a90 2024-03-19T02:02:05.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.786+0000 7fbf01ffb640 1 -- 172.21.15.55:0/3368894314 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef80ab9d0 con 0x7fbf040c9a90 2024-03-19T02:02:05.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.786+0000 7fbf0951c640 1 -- 172.21.15.55:0/3368894314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbf040c9a90 msgr2=0x7fbf040c73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:05.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.786+0000 7fbf0951c640 1 --2- 172.21.15.55:0/3368894314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbf040c9a90 0x7fbf040c73c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fbf04072b40 tx=0x7fbef801c490 comp rx=0 tx=0).stop 2024-03-19T02:02:05.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.786+0000 7fbf0951c640 1 -- 172.21.15.55:0/3368894314 shutdown_connections 2024-03-19T02:02:05.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.786+0000 7fbf0951c640 1 --2- 172.21.15.55:0/3368894314 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbf041540c0 0x7fbf04154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:05.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.786+0000 7fbf0951c640 1 --2- 172.21.15.55:0/3368894314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbf040c9a90 0x7fbf040c73c0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:05.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.786+0000 7fbf0951c640 1 --2- 172.21.15.55:0/3368894314 >> v2:172.21.15.84:3300/0 conn(0x7fbf040c96b0 0x7fbf04070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:05.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.786+0000 7fbf0951c640 1 -- 172.21.15.55:0/3368894314 >> 172.21.15.55:0/3368894314 conn(0x7fbf040cc9c0 msgr2=0x7fbf040ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:05.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.786+0000 7fbf0951c640 1 -- 172.21.15.55:0/3368894314 shutdown_connections 2024-03-19T02:02:05.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.786+0000 7fbf0951c640 1 -- 172.21.15.55:0/3368894314 wait complete. 2024-03-19T02:02:05.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.786+0000 7fbf0951c640 1 Processor -- start 2024-03-19T02:02:05.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.786+0000 7fbf0951c640 1 -- start start 2024-03-19T02:02:05.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.786+0000 7fbf0951c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbf040c96b0 0x7fbf041f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:05.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.786+0000 7fbf0951c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf040c9a90 0x7fbf04158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:05.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.786+0000 7fbf0951c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbf041540c0 0x7fbf041f70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:05.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.786+0000 7fbf0951c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbf040c7a80 con 0x7fbf041540c0 2024-03-19T02:02:05.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.786+0000 7fbf0951c640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fbf040c7900 con 0x7fbf040c9a90 2024-03-19T02:02:05.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.786+0000 7fbf0951c640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbf0406b2a0 con 0x7fbf040c96b0 2024-03-19T02:02:05.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.790+0000 7fbf037fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbf041540c0 0x7fbf041f70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:05.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.790+0000 7fbf037fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbf041540c0 0x7fbf041f70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41150/0 (socket says 172.21.15.55:41150) 2024-03-19T02:02:05.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.790+0000 7fbf037fe640 1 -- 172.21.15.55:0/508750923 learned_addr learned my addr 172.21.15.55:0/508750923 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:05.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.790+0000 7fbf02ffd640 1 --2- 172.21.15.55:0/508750923 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbf040c96b0 0x7fbf041f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:05.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.790+0000 7fbf027fc640 1 --2- 172.21.15.55:0/508750923 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf040c9a90 0x7fbf04158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:05.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.794+0000 7fbf037fe640 1 -- 172.21.15.55:0/508750923 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbf040c96b0 msgr2=0x7fbf041f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:05.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.794+0000 7fbf037fe640 1 --2- 172.21.15.55:0/508750923 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbf040c96b0 0x7fbf041f3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:05.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.794+0000 7fbf037fe640 1 -- 172.21.15.55:0/508750923 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf040c9a90 msgr2=0x7fbf04158920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:05.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.794+0000 7fbf037fe640 1 --2- 172.21.15.55:0/508750923 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf040c9a90 0x7fbf04158920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:05.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.794+0000 7fbf037fe640 1 -- 172.21.15.55:0/508750923 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0415c7e0 con 0x7fbf041540c0 2024-03-19T02:02:05.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.798+0000 7fbf037fe640 1 --2- 172.21.15.55:0/508750923 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbf041540c0 0x7fbf041f70e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fbefc009d30 tx=0x7fbefc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:05.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.802+0000 7fbeeb7fe640 1 -- 172.21.15.55:0/508750923 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc018020 con 0x7fbf041540c0 2024-03-19T02:02:05.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.802+0000 7fbeeb7fe640 1 -- 172.21.15.55:0/508750923 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbefc014480 con 0x7fbf041540c0 2024-03-19T02:02:05.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.802+0000 7fbf0951c640 1 -- 172.21.15.55:0/508750923 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf0415c220 con 0x7fbf041540c0 2024-03-19T02:02:05.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.802+0000 7fbeeb7fe640 1 -- 172.21.15.55:0/508750923 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc014870 con 0x7fbf041540c0 2024-03-19T02:02:05.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.802+0000 7fbf0951c640 1 -- 172.21.15.55:0/508750923 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf041f3270 con 0x7fbf041540c0 2024-03-19T02:02:05.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.802+0000 7fbf0951c640 1 -- 172.21.15.55:0/508750923 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf04068a40 con 0x7fbf041540c0 2024-03-19T02:02:05.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.834+0000 7fbeeb7fe640 1 -- 172.21.15.55:0/508750923 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fbefc01b020 con 0x7fbf041540c0 2024-03-19T02:02:05.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.838+0000 7fbeeb7fe640 1 --2- 172.21.15.55:0/508750923 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbef0067690 0x7fbef0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:05.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.838+0000 7fbf02ffd640 1 --2- 172.21.15.55:0/508750923 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbef0067690 0x7fbef0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:05.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:05.838+0000 7fbf02ffd640 1 --2- 172.21.15.55:0/508750923 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbef0067690 0x7fbef0069b50 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009b60 tx=0x7fbef4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:06.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:06.130+0000 7fbeeb7fe640 1 -- 172.21.15.55:0/508750923 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fbefc054ff0 con 0x7fbf041540c0 2024-03-19T02:02:06.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:06.130+0000 7fbeeb7fe640 1 -- 172.21.15.55:0/508750923 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fbefc022030 con 0x7fbf041540c0 2024-03-19T02:02:06.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:06.358+0000 7fbf0951c640 1 -- 172.21.15.55:0/508750923 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf040c4140 con 0x7fbf041540c0 2024-03-19T02:02:06.376 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:06.378+0000 7fbeeb7fe640 1 -- 172.21.15.55:0/508750923 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fbefc010040 con 0x7fbf041540c0 2024-03-19T02:02:06.377 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:06.377 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:06.377 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:06.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:06.382+0000 7fbf0951c640 1 -- 172.21.15.55:0/508750923 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbef0067690 msgr2=0x7fbef0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:06.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:06.382+0000 7fbf0951c640 1 --2- 172.21.15.55:0/508750923 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbef0067690 0x7fbef0069b50 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009b60 tx=0x7fbef4009190 comp rx=0 tx=0).stop 2024-03-19T02:02:06.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:06.382+0000 7fbf0951c640 1 -- 172.21.15.55:0/508750923 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbf041540c0 msgr2=0x7fbf041f70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:06.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:06.382+0000 7fbf0951c640 1 --2- 172.21.15.55:0/508750923 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbf041540c0 0x7fbf041f70e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fbefc009d30 tx=0x7fbefc00c490 comp rx=0 tx=0).stop 2024-03-19T02:02:06.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:06.382+0000 7fbf0951c640 1 -- 172.21.15.55:0/508750923 shutdown_connections 2024-03-19T02:02:06.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:06.382+0000 7fbf0951c640 1 --2- 172.21.15.55:0/508750923 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbef0067690 0x7fbef0069b50 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:06.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:06.382+0000 7fbf0951c640 1 --2- 172.21.15.55:0/508750923 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbf041540c0 0x7fbf041f70e0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:06.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:06.382+0000 7fbf0951c640 1 --2- 172.21.15.55:0/508750923 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf040c9a90 0x7fbf04158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:06.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:06.382+0000 7fbf0951c640 1 --2- 172.21.15.55:0/508750923 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbf040c96b0 0x7fbf041f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:06.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:06.382+0000 7fbf0951c640 1 -- 172.21.15.55:0/508750923 >> 172.21.15.55:0/508750923 conn(0x7fbf040cc9c0 msgr2=0x7fbf0406b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:06.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:06.382+0000 7fbf0951c640 1 -- 172.21.15.55:0/508750923 shutdown_connections 2024-03-19T02:02:06.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:06.382+0000 7fbf0951c640 1 -- 172.21.15.55:0/508750923 wait complete. 2024-03-19T02:02:07.000 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:07.000 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:08.000 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:08.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.118+0000 7fa675290640 1 Processor -- start 2024-03-19T02:02:08.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.118+0000 7fa675290640 1 -- start start 2024-03-19T02:02:08.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.118+0000 7fa675290640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa6701544f0 0x7fa670073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:08.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.122+0000 7fa675290640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6700c7fa0 con 0x7fa67006b2a0 2024-03-19T02:02:08.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.122+0000 7fa675290640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6700c8240 con 0x7fa67015acb0 2024-03-19T02:02:08.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.122+0000 7fa675290640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa67015b210 con 0x7fa6701544f0 2024-03-19T02:02:08.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.122+0000 7fa66e7fc640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fa67006b2a0 0x7fa67015e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:48628/0 (socket says 172.21.15.55:48628) 2024-03-19T02:02:08.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.122+0000 7fa66e7fc640 1 -- 172.21.15.55:0/1416773114 learned_addr learned my addr 172.21.15.55:0/1416773114 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:02:08.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.122+0000 7fa66effd640 1 --2- 172.21.15.55:0/1416773114 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa6701544f0 0x7fa670073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:08.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.122+0000 7fa66dffb640 1 -- 172.21.15.55:0/1416773114 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 888832857 0 0) 0x7fa6700c7fa0 con 0x7fa67006b2a0 2024-03-19T02:02:08.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.122+0000 7fa66dffb640 1 -- 172.21.15.55:0/1416773114 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa658003530 con 0x7fa67006b2a0 2024-03-19T02:02:08.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.126+0000 7fa66effd640 1 -- 172.21.15.55:0/1416773114 >> v1:172.21.15.84:6789/0 conn(0x7fa67015acb0 legacy=0x7fa6700cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:08.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.126+0000 7fa66effd640 1 -- 172.21.15.55:0/1416773114 >> v1:172.21.15.55:6789/0 conn(0x7fa67006b2a0 legacy=0x7fa67015e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:08.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.126+0000 7fa66effd640 1 -- 172.21.15.55:0/1416773114 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa67015d920 con 0x7fa6701544f0 2024-03-19T02:02:08.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.130+0000 7fa66effd640 1 --2- 172.21.15.55:0/1416773114 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa6701544f0 0x7fa670073540 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa6600097f0 tx=0x7fa66001ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=feaf94aef7da5ef7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:08.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.134+0000 7fa66dffb640 1 -- 172.21.15.55:0/1416773114 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6600b5070 con 0x7fa6701544f0 2024-03-19T02:02:08.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.134+0000 7fa66dffb640 1 -- 172.21.15.55:0/1416773114 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6600a89d0 con 0x7fa6701544f0 2024-03-19T02:02:08.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.134+0000 7fa66dffb640 1 -- 172.21.15.55:0/1416773114 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6600a8da0 con 0x7fa6701544f0 2024-03-19T02:02:08.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.134+0000 7fa675290640 1 -- 172.21.15.55:0/1416773114 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa6701544f0 msgr2=0x7fa670073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:08.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.134+0000 7fa675290640 1 --2- 172.21.15.55:0/1416773114 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa6701544f0 0x7fa670073540 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa6600097f0 tx=0x7fa66001ecd0 comp rx=0 tx=0).stop 2024-03-19T02:02:08.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.134+0000 7fa675290640 1 -- 172.21.15.55:0/1416773114 shutdown_connections 2024-03-19T02:02:08.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.134+0000 7fa675290640 1 --2- 172.21.15.55:0/1416773114 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa6701544f0 0x7fa670073540 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:08.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.134+0000 7fa675290640 1 -- 172.21.15.55:0/1416773114 >> 172.21.15.55:0/1416773114 conn(0x7fa67006e670 msgr2=0x7fa67006ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:08.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.134+0000 7fa675290640 1 -- 172.21.15.55:0/1416773114 shutdown_connections 2024-03-19T02:02:08.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.134+0000 7fa675290640 1 -- 172.21.15.55:0/1416773114 wait complete. 2024-03-19T02:02:08.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.138+0000 7fa675290640 1 Processor -- start 2024-03-19T02:02:08.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.138+0000 7fa675290640 1 -- start start 2024-03-19T02:02:08.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.138+0000 7fa675290640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa67006b2a0 0x7fa6701574e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:08.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.138+0000 7fa675290640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6701544f0 0x7fa670151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:08.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.138+0000 7fa675290640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa67015acb0 0x7fa6701f0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:08.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.138+0000 7fa675290640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa6700cd540 con 0x7fa6701544f0 2024-03-19T02:02:08.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.138+0000 7fa675290640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa6701548d0 con 0x7fa67006b2a0 2024-03-19T02:02:08.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.138+0000 7fa675290640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa6700cd6c0 con 0x7fa67015acb0 2024-03-19T02:02:08.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.138+0000 7fa66e7fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6701544f0 0x7fa670151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:08.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.138+0000 7fa66effd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa67006b2a0 0x7fa6701574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:08.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.138+0000 7fa66f7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa67015acb0 0x7fa6701f0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:08.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.138+0000 7fa66e7fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6701544f0 0x7fa670151fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41152/0 (socket says 172.21.15.55:41152) 2024-03-19T02:02:08.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.138+0000 7fa66e7fc640 1 -- 172.21.15.55:0/30955071 learned_addr learned my addr 172.21.15.55:0/30955071 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:08.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.142+0000 7fa66e7fc640 1 -- 172.21.15.55:0/30955071 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa67015acb0 msgr2=0x7fa6701f0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:08.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.142+0000 7fa66e7fc640 1 --2- 172.21.15.55:0/30955071 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa67015acb0 0x7fa6701f0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:08.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.142+0000 7fa66e7fc640 1 -- 172.21.15.55:0/30955071 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa67006b2a0 msgr2=0x7fa6701574e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:08.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.142+0000 7fa66e7fc640 1 --2- 172.21.15.55:0/30955071 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa67006b2a0 0x7fa6701574e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:08.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.142+0000 7fa66e7fc640 1 -- 172.21.15.55:0/30955071 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa670152c10 con 0x7fa6701544f0 2024-03-19T02:02:08.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.146+0000 7fa66e7fc640 1 --2- 172.21.15.55:0/30955071 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6701544f0 0x7fa670151fd0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa66400c490 tx=0x7fa66400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:08.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.150+0000 7fa6577fe640 1 -- 172.21.15.55:0/30955071 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa664017070 con 0x7fa6701544f0 2024-03-19T02:02:08.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.150+0000 7fa6577fe640 1 -- 172.21.15.55:0/30955071 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa664005cf0 con 0x7fa6701544f0 2024-03-19T02:02:08.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.150+0000 7fa675290640 1 -- 172.21.15.55:0/30955071 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa670152ef0 con 0x7fa6701544f0 2024-03-19T02:02:08.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.150+0000 7fa6577fe640 1 -- 172.21.15.55:0/30955071 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa664007290 con 0x7fa6701544f0 2024-03-19T02:02:08.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.150+0000 7fa675290640 1 -- 172.21.15.55:0/30955071 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6701f3ad0 con 0x7fa6701544f0 2024-03-19T02:02:08.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.150+0000 7fa675290640 1 -- 172.21.15.55:0/30955071 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa670068a40 con 0x7fa6701544f0 2024-03-19T02:02:08.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.178+0000 7fa6577fe640 1 -- 172.21.15.55:0/30955071 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa66401a070 con 0x7fa6701544f0 2024-03-19T02:02:08.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.178+0000 7fa6577fe640 1 --2- 172.21.15.55:0/30955071 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa65c0676b0 0x7fa65c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:08.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.178+0000 7fa66effd640 1 --2- 172.21.15.55:0/30955071 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa65c0676b0 0x7fa65c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:08.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.178+0000 7fa66effd640 1 --2- 172.21.15.55:0/30955071 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa65c0676b0 0x7fa65c069b70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa6600b2000 tx=0x7fa660008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:08.499 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.502+0000 7fa6577fe640 1 -- 172.21.15.55:0/30955071 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fa664053580 con 0x7fa6701544f0 2024-03-19T02:02:08.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.502+0000 7fa6577fe640 1 -- 172.21.15.55:0/30955071 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa664021080 con 0x7fa6701544f0 2024-03-19T02:02:08.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.722+0000 7fa675290640 1 -- 172.21.15.55:0/30955071 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6700c3860 con 0x7fa6701544f0 2024-03-19T02:02:08.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.746+0000 7fa6577fe640 1 -- 172.21.15.55:0/30955071 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fa6640554e0 con 0x7fa6701544f0 2024-03-19T02:02:08.746 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:08.746 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:08.746 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:08.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.750+0000 7fa675290640 1 -- 172.21.15.55:0/30955071 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa65c0676b0 msgr2=0x7fa65c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:08.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.750+0000 7fa675290640 1 --2- 172.21.15.55:0/30955071 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa65c0676b0 0x7fa65c069b70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa6600b2000 tx=0x7fa660008040 comp rx=0 tx=0).stop 2024-03-19T02:02:08.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.750+0000 7fa675290640 1 -- 172.21.15.55:0/30955071 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6701544f0 msgr2=0x7fa670151fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:08.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.750+0000 7fa675290640 1 --2- 172.21.15.55:0/30955071 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6701544f0 0x7fa670151fd0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa66400c490 tx=0x7fa66400c950 comp rx=0 tx=0).stop 2024-03-19T02:02:08.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.750+0000 7fa675290640 1 -- 172.21.15.55:0/30955071 shutdown_connections 2024-03-19T02:02:08.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.750+0000 7fa675290640 1 --2- 172.21.15.55:0/30955071 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa65c0676b0 0x7fa65c069b70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:08.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.750+0000 7fa675290640 1 --2- 172.21.15.55:0/30955071 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa67015acb0 0x7fa6701f0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:08.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.750+0000 7fa675290640 1 --2- 172.21.15.55:0/30955071 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6701544f0 0x7fa670151fd0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:08.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.750+0000 7fa675290640 1 --2- 172.21.15.55:0/30955071 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa67006b2a0 0x7fa6701574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:08.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.750+0000 7fa675290640 1 -- 172.21.15.55:0/30955071 >> 172.21.15.55:0/30955071 conn(0x7fa67006e670 msgr2=0x7fa6700c8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:08.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.750+0000 7fa675290640 1 -- 172.21.15.55:0/30955071 shutdown_connections 2024-03-19T02:02:08.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:08.750+0000 7fa675290640 1 -- 172.21.15.55:0/30955071 wait complete. 2024-03-19T02:02:09.367 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:09.367 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:10.369 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:10.384 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:02:10.417 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:02:10.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.506+0000 7f512cde9640 1 Processor -- start 2024-03-19T02:02:10.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.506+0000 7f512cde9640 1 -- start start 2024-03-19T02:02:10.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.510+0000 7f512cde9640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f51281511f0 0x7f51280c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:10.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.510+0000 7f512cde9640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f51280c2e40 0x7f5128156f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:10.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.510+0000 7f512cde9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51280c8260 0x7f51280c7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:10.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.510+0000 7f512cde9640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f5128157490 con 0x7f51280c2e40 2024-03-19T02:02:10.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.510+0000 7f512cde9640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f51280c5040 con 0x7f51281511f0 2024-03-19T02:02:10.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.510+0000 7f512cde9640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f51280c3330 con 0x7f51280c8260 2024-03-19T02:02:10.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.510+0000 7f5125d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f51280c2e40 0x7f5128156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:10.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.510+0000 7f5126d9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51280c8260 0x7f51280c7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:10.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.510+0000 7f5125d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f51280c2e40 0x7f5128156f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50554/0 (socket says 172.21.15.55:50554) 2024-03-19T02:02:10.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.510+0000 7f5125d9b640 1 -- 172.21.15.55:0/679170144 learned_addr learned my addr 172.21.15.55:0/679170144 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:10.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.510+0000 7f512659c640 1 --2- 172.21.15.55:0/679170144 >> v2:172.21.15.84:3300/0 conn(0x7f51281511f0 0x7f51280c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:10.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.514+0000 7f5125d9b640 1 -- 172.21.15.55:0/679170144 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51280c8260 msgr2=0x7f51280c7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:10.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.514+0000 7f5125d9b640 1 --2- 172.21.15.55:0/679170144 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51280c8260 0x7f51280c7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:10.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.514+0000 7f5125d9b640 1 -- 172.21.15.55:0/679170144 >> v2:172.21.15.84:3300/0 conn(0x7f51281511f0 msgr2=0x7f51280c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:10.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.514+0000 7f5125d9b640 1 --2- 172.21.15.55:0/679170144 >> v2:172.21.15.84:3300/0 conn(0x7f51281511f0 0x7f51280c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:10.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.514+0000 7f5125d9b640 1 -- 172.21.15.55:0/679170144 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51280c3610 con 0x7f51280c2e40 2024-03-19T02:02:10.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.518+0000 7f5125d9b640 1 --2- 172.21.15.55:0/679170144 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51280c2e40 0x7f5128156f50 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f512806c140 tx=0x7f511c0213e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ba99f512bf7c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:10.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.522+0000 7f512559a640 1 -- 172.21.15.55:0/679170144 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c0b6070 con 0x7f51280c2e40 2024-03-19T02:02:10.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.522+0000 7f512559a640 1 -- 172.21.15.55:0/679170144 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f511c0b1600 con 0x7f51280c2e40 2024-03-19T02:02:10.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.522+0000 7f512559a640 1 -- 172.21.15.55:0/679170144 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c0b19d0 con 0x7f51280c2e40 2024-03-19T02:02:10.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.522+0000 7f512cde9640 1 -- 172.21.15.55:0/679170144 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51280c2e40 msgr2=0x7f5128156f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:10.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.522+0000 7f512cde9640 1 --2- 172.21.15.55:0/679170144 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51280c2e40 0x7f5128156f50 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f512806c140 tx=0x7f511c0213e0 comp rx=0 tx=0).stop 2024-03-19T02:02:10.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.522+0000 7f512cde9640 1 -- 172.21.15.55:0/679170144 shutdown_connections 2024-03-19T02:02:10.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.522+0000 7f512cde9640 1 --2- 172.21.15.55:0/679170144 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51280c8260 0x7f51280c7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:10.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.522+0000 7f512cde9640 1 --2- 172.21.15.55:0/679170144 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51280c2e40 0x7f5128156f50 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:10.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.522+0000 7f512cde9640 1 --2- 172.21.15.55:0/679170144 >> v2:172.21.15.84:3300/0 conn(0x7f51281511f0 0x7f51280c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:10.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.522+0000 7f512cde9640 1 -- 172.21.15.55:0/679170144 >> 172.21.15.55:0/679170144 conn(0x7f512806f3b0 msgr2=0x7f512806f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:10.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.522+0000 7f512cde9640 1 -- 172.21.15.55:0/679170144 shutdown_connections 2024-03-19T02:02:10.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.522+0000 7f512cde9640 1 -- 172.21.15.55:0/679170144 wait complete. 2024-03-19T02:02:10.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.526+0000 7f512cde9640 1 Processor -- start 2024-03-19T02:02:10.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.526+0000 7f512cde9640 1 -- start start 2024-03-19T02:02:10.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.526+0000 7f512cde9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51280c2e40 0x7f5128143130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:10.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.526+0000 7f512cde9640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51280c8260 0x7f5128145220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:10.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.526+0000 7f512cde9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51281511f0 0x7f51281ee9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:10.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.526+0000 7f512cde9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f51281515d0 con 0x7f51280c2e40 2024-03-19T02:02:10.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.526+0000 7f512cde9640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f51280c5290 con 0x7f51280c8260 2024-03-19T02:02:10.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.526+0000 7f512cde9640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f51280c29c0 con 0x7f51281511f0 2024-03-19T02:02:10.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.526+0000 7f512659c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51280c2e40 0x7f5128143130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:10.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.526+0000 7f5125d9b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51280c8260 0x7f5128145220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:10.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.526+0000 7f512659c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51280c2e40 0x7f5128143130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50566/0 (socket says 172.21.15.55:50566) 2024-03-19T02:02:10.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.526+0000 7f512659c640 1 -- 172.21.15.55:0/2562226872 learned_addr learned my addr 172.21.15.55:0/2562226872 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:10.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.526+0000 7f5126d9d640 1 --2- 172.21.15.55:0/2562226872 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51281511f0 0x7f51281ee9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:10.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.530+0000 7f5125d9b640 1 -- 172.21.15.55:0/2562226872 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51281511f0 msgr2=0x7f51281ee9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:10.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.530+0000 7f5125d9b640 1 --2- 172.21.15.55:0/2562226872 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51281511f0 0x7f51281ee9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:10.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.530+0000 7f5125d9b640 1 -- 172.21.15.55:0/2562226872 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51280c2e40 msgr2=0x7f5128143130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:10.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.530+0000 7f5125d9b640 1 --2- 172.21.15.55:0/2562226872 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51280c2e40 0x7f5128143130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:10.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.530+0000 7f5125d9b640 1 -- 172.21.15.55:0/2562226872 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5128144180 con 0x7f51280c8260 2024-03-19T02:02:10.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.558+0000 7f5125d9b640 1 --2- 172.21.15.55:0/2562226872 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51280c8260 0x7f5128145220 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f511c0b1000 tx=0x7f511c01e180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:10.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.570+0000 7f5116ffd640 1 -- 172.21.15.55:0/2562226872 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c0c0020 con 0x7f51280c8260 2024-03-19T02:02:10.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.570+0000 7f5116ffd640 1 -- 172.21.15.55:0/2562226872 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f511c0bb460 con 0x7f51280c8260 2024-03-19T02:02:10.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.570+0000 7f512cde9640 1 -- 172.21.15.55:0/2562226872 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5128144460 con 0x7f51280c8260 2024-03-19T02:02:10.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.570+0000 7f5116ffd640 1 -- 172.21.15.55:0/2562226872 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c0b6040 con 0x7f51280c8260 2024-03-19T02:02:10.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.570+0000 7f512cde9640 1 -- 172.21.15.55:0/2562226872 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5128144a20 con 0x7f51280c8260 2024-03-19T02:02:10.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.574+0000 7f512cde9640 1 -- 172.21.15.55:0/2562226872 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5128068330 con 0x7f51280c8260 2024-03-19T02:02:10.605 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.606+0000 7f5116ffd640 1 -- 172.21.15.55:0/2562226872 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f511c0c3020 con 0x7f51280c8260 2024-03-19T02:02:10.605 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.606+0000 7f5116ffd640 1 --2- 172.21.15.55:0/2562226872 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f51080676b0 0x7f5108069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:10.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.606+0000 7f512659c640 1 --2- 172.21.15.55:0/2562226872 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f51080676b0 0x7f5108069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:10.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.606+0000 7f512659c640 1 --2- 172.21.15.55:0/2562226872 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f51080676b0 0x7f5108069b70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5118009b80 tx=0x7f5118009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:10.610 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.610+0000 7f5116ffd640 1 -- 172.21.15.55:0/2562226872 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f511c0f47c0 con 0x7f51280c8260 2024-03-19T02:02:10.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:10.890+0000 7f5116ffd640 1 -- 172.21.15.55:0/2562226872 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f511c0ca030 con 0x7f51280c8260 2024-03-19T02:02:11.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:11.110+0000 7f512cde9640 1 -- 172.21.15.55:0/2562226872 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51281518a0 con 0x7f51280c8260 2024-03-19T02:02:11.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:11.210+0000 7f5116ffd640 1 -- 172.21.15.55:0/2562226872 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f511c01af40 con 0x7f51280c8260 2024-03-19T02:02:11.208 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:11.208 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:11.208 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:11.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:11.210+0000 7f512cde9640 1 -- 172.21.15.55:0/2562226872 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f51080676b0 msgr2=0x7f5108069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:11.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:11.210+0000 7f512cde9640 1 --2- 172.21.15.55:0/2562226872 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f51080676b0 0x7f5108069b70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5118009b80 tx=0x7f5118009190 comp rx=0 tx=0).stop 2024-03-19T02:02:11.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:11.210+0000 7f512cde9640 1 -- 172.21.15.55:0/2562226872 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51280c8260 msgr2=0x7f5128145220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:11.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:11.210+0000 7f512cde9640 1 --2- 172.21.15.55:0/2562226872 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51280c8260 0x7f5128145220 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f511c0b1000 tx=0x7f511c01e180 comp rx=0 tx=0).stop 2024-03-19T02:02:11.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:11.210+0000 7f512cde9640 1 -- 172.21.15.55:0/2562226872 shutdown_connections 2024-03-19T02:02:11.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:11.210+0000 7f512cde9640 1 --2- 172.21.15.55:0/2562226872 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f51080676b0 0x7f5108069b70 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:11.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:11.210+0000 7f512cde9640 1 --2- 172.21.15.55:0/2562226872 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f51281511f0 0x7f51281ee9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:11.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:11.210+0000 7f512cde9640 1 --2- 172.21.15.55:0/2562226872 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51280c8260 0x7f5128145220 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:11.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:11.210+0000 7f512cde9640 1 --2- 172.21.15.55:0/2562226872 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51280c2e40 0x7f5128143130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:11.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:11.210+0000 7f512cde9640 1 -- 172.21.15.55:0/2562226872 >> 172.21.15.55:0/2562226872 conn(0x7f512806f3b0 msgr2=0x7f5128071810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:11.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:11.210+0000 7f512cde9640 1 -- 172.21.15.55:0/2562226872 shutdown_connections 2024-03-19T02:02:11.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:11.210+0000 7f512cde9640 1 -- 172.21.15.55:0/2562226872 wait complete. 2024-03-19T02:02:11.831 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:11.832 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:12.833 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:12.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.950+0000 7f80693ab640 1 Processor -- start 2024-03-19T02:02:12.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.950+0000 7f80693ab640 1 -- start start 2024-03-19T02:02:12.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.950+0000 7f80693ab640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f80641520b0 0x7f80640709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:12.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.950+0000 7f80693ab640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8064158d50 con 0x7f80640c9450 2024-03-19T02:02:12.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.950+0000 7f80693ab640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f806415b720 con 0x7f80640c9000 2024-03-19T02:02:12.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.950+0000 7f80693ab640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f806415c4e0 con 0x7f80641520b0 2024-03-19T02:02:12.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.950+0000 7f80637fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f80640c9450 0x7f80640c31e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:47700/0 (socket says 172.21.15.55:47700) 2024-03-19T02:02:12.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.950+0000 7f80637fe640 1 -- 172.21.15.55:0/4122431501 learned_addr learned my addr 172.21.15.55:0/4122431501 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:02:12.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.950+0000 7f8062ffd640 1 --2- 172.21.15.55:0/4122431501 >> v2:172.21.15.84:3300/0 conn(0x7f80641520b0 0x7f80640709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:12.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.954+0000 7f8061ffb640 1 -- 172.21.15.55:0/4122431501 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1117245780 0 0) 0x7f8064158d50 con 0x7f80640c9450 2024-03-19T02:02:12.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.954+0000 7f8061ffb640 1 -- 172.21.15.55:0/4122431501 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f804c003530 con 0x7f80640c9450 2024-03-19T02:02:12.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.958+0000 7f8061ffb640 1 -- 172.21.15.55:0/4122431501 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 132258757 0 0) 0x7f804c003530 con 0x7f80640c9450 2024-03-19T02:02:12.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.958+0000 7f8061ffb640 1 -- 172.21.15.55:0/4122431501 >> v2:172.21.15.84:3300/0 conn(0x7f80641520b0 msgr2=0x7f80640709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:12.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.958+0000 7f8061ffb640 1 --2- 172.21.15.55:0/4122431501 >> v2:172.21.15.84:3300/0 conn(0x7f80641520b0 0x7f80640709a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:12.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.958+0000 7f8061ffb640 1 -- 172.21.15.55:0/4122431501 >> v1:172.21.15.84:6789/0 conn(0x7f80640c9000 legacy=0x7f80640c2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:12.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.958+0000 7f8061ffb640 1 -- 172.21.15.55:0/4122431501 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f806415ba00 con 0x7f80640c9450 2024-03-19T02:02:12.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.958+0000 7f8061ffb640 1 -- 172.21.15.55:0/4122431501 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f805c001a00 con 0x7f80640c9450 2024-03-19T02:02:12.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.958+0000 7f8061ffb640 1 -- 172.21.15.55:0/4122431501 >> v1:172.21.15.55:6789/0 conn(0x7f80640c9450 legacy=0x7f80640c31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:12.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.958+0000 7f8061ffb640 1 --2- 172.21.15.55:0/4122431501 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f804c003a10 0x7f804c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:12.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.958+0000 7f8061ffb640 1 --2- 172.21.15.55:0/4122431501 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f804c004810 0x7f804c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:12.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.958+0000 7f8061ffb640 1 --2- 172.21.15.55:0/4122431501 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f804c0051d0 0x7f804c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:12.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.958+0000 7f8061ffb640 1 -- 172.21.15.55:0/4122431501 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f806415c4e0 con 0x7f804c004810 2024-03-19T02:02:12.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.958+0000 7f8061ffb640 1 -- 172.21.15.55:0/4122431501 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f804c00c0b0 con 0x7f804c003a10 2024-03-19T02:02:12.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.958+0000 7f8061ffb640 1 -- 172.21.15.55:0/4122431501 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f804c00c2b0 con 0x7f804c0051d0 2024-03-19T02:02:12.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.958+0000 7f80637fe640 1 --2- 172.21.15.55:0/4122431501 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f804c0051d0 0x7f804c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:12.960 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.962+0000 7f80627fc640 1 --2- 172.21.15.55:0/4122431501 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f804c004810 0x7f804c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:12.961 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.962+0000 7f80637fe640 1 -- 172.21.15.55:0/4122431501 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f804c003a10 msgr2=0x7f804c003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:12.961 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.962+0000 7f80637fe640 1 --2- 172.21.15.55:0/4122431501 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f804c003a10 0x7f804c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:12.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.962+0000 7f80637fe640 1 -- 172.21.15.55:0/4122431501 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f804c004810 msgr2=0x7f804c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:12.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.962+0000 7f80637fe640 1 --2- 172.21.15.55:0/4122431501 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f804c004810 0x7f804c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:12.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.962+0000 7f80637fe640 1 -- 172.21.15.55:0/4122431501 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f804c003530 con 0x7f804c0051d0 2024-03-19T02:02:12.963 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.966+0000 7f80637fe640 1 -- 172.21.15.55:0/4122431501 reap_dead start 2024-03-19T02:02:12.965 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.966+0000 7f80637fe640 1 --2- 172.21.15.55:0/4122431501 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f804c0051d0 0x7f804c00b9d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f805c00a770 tx=0x7f805c01d2a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d6ed044169d5c46 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:12.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.970+0000 7f8061ffb640 1 -- 172.21.15.55:0/4122431501 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805c00a390 con 0x7f804c0051d0 2024-03-19T02:02:12.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.970+0000 7f8061ffb640 1 -- 172.21.15.55:0/4122431501 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f805c0ac640 con 0x7f804c0051d0 2024-03-19T02:02:12.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.970+0000 7f8061ffb640 1 -- 172.21.15.55:0/4122431501 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805c0aca30 con 0x7f804c0051d0 2024-03-19T02:02:12.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.974+0000 7f80693ab640 1 -- 172.21.15.55:0/4122431501 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f804c0051d0 msgr2=0x7f804c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:12.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.974+0000 7f80693ab640 1 --2- 172.21.15.55:0/4122431501 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f804c0051d0 0x7f804c00b9d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f805c00a770 tx=0x7f805c01d2a0 comp rx=0 tx=0).stop 2024-03-19T02:02:12.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.974+0000 7f80693ab640 1 -- 172.21.15.55:0/4122431501 shutdown_connections 2024-03-19T02:02:12.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.974+0000 7f80693ab640 1 --2- 172.21.15.55:0/4122431501 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f804c0051d0 0x7f804c00b9d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:12.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.974+0000 7f80693ab640 1 -- 172.21.15.55:0/4122431501 >> 172.21.15.55:0/4122431501 conn(0x7f8064067600 msgr2=0x7f8064071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:12.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.974+0000 7f80693ab640 1 -- 172.21.15.55:0/4122431501 shutdown_connections 2024-03-19T02:02:12.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.974+0000 7f80693ab640 1 -- 172.21.15.55:0/4122431501 wait complete. 2024-03-19T02:02:12.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.974+0000 7f80693ab640 1 Processor -- start 2024-03-19T02:02:12.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.974+0000 7f80693ab640 1 -- start start 2024-03-19T02:02:12.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.974+0000 7f80693ab640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f804c0051d0 0x7f80640c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:12.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.974+0000 7f80693ab640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f806415ad40 0x7f80640ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:12.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.974+0000 7f80693ab640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80640cc250 0x7f8064146200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:12.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.974+0000 7f80693ab640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8064158f20 con 0x7f804c0051d0 2024-03-19T02:02:12.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.974+0000 7f80693ab640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f80640c9880 con 0x7f80640cc250 2024-03-19T02:02:12.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.974+0000 7f80693ab640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f806415c050 con 0x7f806415ad40 2024-03-19T02:02:12.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.978+0000 7f8062ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f804c0051d0 0x7f80640c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:12.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.978+0000 7f80627fc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f806415ad40 0x7f80640ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:12.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.978+0000 7f80637fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80640cc250 0x7f8064146200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:12.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.978+0000 7f8062ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f804c0051d0 0x7f80640c6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50578/0 (socket says 172.21.15.55:50578) 2024-03-19T02:02:12.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.978+0000 7f8062ffd640 1 -- 172.21.15.55:0/967171417 learned_addr learned my addr 172.21.15.55:0/967171417 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:12.980 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.982+0000 7f8062ffd640 1 -- 172.21.15.55:0/967171417 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f806415ad40 msgr2=0x7f80640ca2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:12.980 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.982+0000 7f8062ffd640 1 --2- 172.21.15.55:0/967171417 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f806415ad40 0x7f80640ca2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:12.980 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.982+0000 7f8062ffd640 1 -- 172.21.15.55:0/967171417 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80640cc250 msgr2=0x7f8064146200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:12.980 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.982+0000 7f8062ffd640 1 --2- 172.21.15.55:0/967171417 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80640cc250 0x7f8064146200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:12.980 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.982+0000 7f8062ffd640 1 -- 172.21.15.55:0/967171417 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80640c4eb0 con 0x7f804c0051d0 2024-03-19T02:02:12.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.982+0000 7f8062ffd640 1 --2- 172.21.15.55:0/967171417 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f804c0051d0 0x7f80640c6e40 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f805400e7f0 tx=0x7f805400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:12.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.986+0000 7f804b7fe640 1 -- 172.21.15.55:0/967171417 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8054017070 con 0x7f804c0051d0 2024-03-19T02:02:12.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.986+0000 7f804b7fe640 1 -- 172.21.15.55:0/967171417 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f805400c940 con 0x7f804c0051d0 2024-03-19T02:02:12.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.986+0000 7f80693ab640 1 -- 172.21.15.55:0/967171417 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80640c7460 con 0x7f804c0051d0 2024-03-19T02:02:12.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.986+0000 7f804b7fe640 1 -- 172.21.15.55:0/967171417 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805400eea0 con 0x7f804c0051d0 2024-03-19T02:02:12.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.986+0000 7f80693ab640 1 -- 172.21.15.55:0/967171417 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80640c7a20 con 0x7f804c0051d0 2024-03-19T02:02:12.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:12.986+0000 7f80693ab640 1 -- 172.21.15.55:0/967171417 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8064068a40 con 0x7f804c0051d0 2024-03-19T02:02:13.282 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:13.282+0000 7f804b7fe640 1 -- 172.21.15.55:0/967171417 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f805401a070 con 0x7f804c0051d0 2024-03-19T02:02:13.283 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:13.282+0000 7f804b7fe640 1 --2- 172.21.15.55:0/967171417 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8050067690 0x7f8050069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:13.283 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:13.282+0000 7f80627fc640 1 --2- 172.21.15.55:0/967171417 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8050067690 0x7f8050069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:13.283 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:13.286+0000 7f80627fc640 1 --2- 172.21.15.55:0/967171417 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8050067690 0x7f8050069b50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8064147410 tx=0x7f8058009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:13.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:13.322+0000 7f804b7fe640 1 -- 172.21.15.55:0/967171417 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f8054054570 con 0x7f804c0051d0 2024-03-19T02:02:13.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:13.322+0000 7f804b7fe640 1 -- 172.21.15.55:0/967171417 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f80540564d0 con 0x7f804c0051d0 2024-03-19T02:02:13.594 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:13.594+0000 7f80693ab640 1 -- 172.21.15.55:0/967171417 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8064147160 con 0x7f804c0051d0 2024-03-19T02:02:13.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:13.698+0000 7f804b7fe640 1 -- 172.21.15.55:0/967171417 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f805405a570 con 0x7f804c0051d0 2024-03-19T02:02:13.696 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:13.696 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:13.696 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:13.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:13.698+0000 7f80693ab640 1 -- 172.21.15.55:0/967171417 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8050067690 msgr2=0x7f8050069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:13.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:13.698+0000 7f80693ab640 1 --2- 172.21.15.55:0/967171417 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8050067690 0x7f8050069b50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8064147410 tx=0x7f8058009040 comp rx=0 tx=0).stop 2024-03-19T02:02:13.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:13.698+0000 7f80693ab640 1 -- 172.21.15.55:0/967171417 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f804c0051d0 msgr2=0x7f80640c6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:13.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:13.698+0000 7f80693ab640 1 --2- 172.21.15.55:0/967171417 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f804c0051d0 0x7f80640c6e40 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f805400e7f0 tx=0x7f805400ecb0 comp rx=0 tx=0).stop 2024-03-19T02:02:13.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:13.698+0000 7f80693ab640 1 -- 172.21.15.55:0/967171417 shutdown_connections 2024-03-19T02:02:13.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:13.698+0000 7f80693ab640 1 --2- 172.21.15.55:0/967171417 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8050067690 0x7f8050069b50 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:13.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:13.698+0000 7f80693ab640 1 --2- 172.21.15.55:0/967171417 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80640cc250 0x7f8064146200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:13.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:13.698+0000 7f80693ab640 1 --2- 172.21.15.55:0/967171417 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f806415ad40 0x7f80640ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:13.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:13.698+0000 7f80693ab640 1 --2- 172.21.15.55:0/967171417 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f804c0051d0 0x7f80640c6e40 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:13.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:13.698+0000 7f80693ab640 1 -- 172.21.15.55:0/967171417 >> 172.21.15.55:0/967171417 conn(0x7f8064067600 msgr2=0x7f8064150c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:13.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:13.698+0000 7f80693ab640 1 -- 172.21.15.55:0/967171417 shutdown_connections 2024-03-19T02:02:13.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:13.698+0000 7f80693ab640 1 -- 172.21.15.55:0/967171417 wait complete. 2024-03-19T02:02:14.324 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:14.324 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:15.325 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:15.439 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.438+0000 7ff6f6016640 1 Processor -- start 2024-03-19T02:02:15.439 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.438+0000 7ff6f6016640 1 -- start start 2024-03-19T02:02:15.439 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.442+0000 7ff6f6016640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff6f014e930 0x7ff6f006b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:15.439 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.442+0000 7ff6f6016640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff6f006be50 0x7ff6f01585a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:15.440 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.442+0000 7ff6f6016640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6f00c7540 con 0x7ff6f0158ae0 2024-03-19T02:02:15.440 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.442+0000 7ff6f6016640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7ff6f00c80a0 con 0x7ff6f006be50 2024-03-19T02:02:15.440 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.442+0000 7ff6f6016640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff6f00c7800 con 0x7ff6f014e930 2024-03-19T02:02:15.440 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.442+0000 7ff6effff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7ff6f0158ae0 0x7ff6f00c60d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:47710/0 (socket says 172.21.15.55:47710) 2024-03-19T02:02:15.440 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.442+0000 7ff6effff640 1 -- 172.21.15.55:0/3479476316 learned_addr learned my addr 172.21.15.55:0/3479476316 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:02:15.441 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.442+0000 7ff6ef7fe640 1 --2- 172.21.15.55:0/3479476316 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff6f014e930 0x7ff6f006b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:15.441 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.442+0000 7ff6eeffd640 1 --2- 172.21.15.55:0/3479476316 >> v2:172.21.15.84:3300/0 conn(0x7ff6f006be50 0x7ff6f01585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:15.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.446+0000 7ff6ee7fc640 1 -- 172.21.15.55:0/3479476316 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1173351412 0 0) 0x7ff6f00c7540 con 0x7ff6f0158ae0 2024-03-19T02:02:15.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.446+0000 7ff6ee7fc640 1 -- 172.21.15.55:0/3479476316 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6d8003530 con 0x7ff6f0158ae0 2024-03-19T02:02:15.445 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.446+0000 7ff6ef7fe640 1 -- 172.21.15.55:0/3479476316 >> v2:172.21.15.84:3300/0 conn(0x7ff6f006be50 msgr2=0x7ff6f01585a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:15.445 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.446+0000 7ff6ef7fe640 1 --2- 172.21.15.55:0/3479476316 >> v2:172.21.15.84:3300/0 conn(0x7ff6f006be50 0x7ff6f01585a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:15.445 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.446+0000 7ff6ef7fe640 1 -- 172.21.15.55:0/3479476316 >> v1:172.21.15.55:6789/0 conn(0x7ff6f0158ae0 legacy=0x7ff6f00c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:15.445 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.446+0000 7ff6ef7fe640 1 -- 172.21.15.55:0/3479476316 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6f00c7ae0 con 0x7ff6f014e930 2024-03-19T02:02:15.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.450+0000 7ff6ef7fe640 1 --2- 172.21.15.55:0/3479476316 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff6f014e930 0x7ff6f006b910 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff6f00c8220 tx=0x7ff6e001ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3fa77c411b3a6a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:15.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.454+0000 7ff6ee7fc640 1 -- 172.21.15.55:0/3479476316 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e00b5070 con 0x7ff6f014e930 2024-03-19T02:02:15.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.458+0000 7ff6ee7fc640 1 -- 172.21.15.55:0/3479476316 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6e00a8990 con 0x7ff6f014e930 2024-03-19T02:02:15.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.458+0000 7ff6ee7fc640 1 -- 172.21.15.55:0/3479476316 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e00a8d60 con 0x7ff6f014e930 2024-03-19T02:02:15.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.458+0000 7ff6f6016640 1 -- 172.21.15.55:0/3479476316 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff6f014e930 msgr2=0x7ff6f006b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:15.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.458+0000 7ff6f6016640 1 --2- 172.21.15.55:0/3479476316 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff6f014e930 0x7ff6f006b910 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff6f00c8220 tx=0x7ff6e001ec90 comp rx=0 tx=0).stop 2024-03-19T02:02:15.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.458+0000 7ff6f6016640 1 -- 172.21.15.55:0/3479476316 shutdown_connections 2024-03-19T02:02:15.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.458+0000 7ff6f6016640 1 --2- 172.21.15.55:0/3479476316 >> v2:172.21.15.84:3300/0 conn(0x7ff6f006be50 0x7ff6f01585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:15.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.458+0000 7ff6f6016640 1 --2- 172.21.15.55:0/3479476316 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff6f014e930 0x7ff6f006b910 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:15.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.458+0000 7ff6f6016640 1 -- 172.21.15.55:0/3479476316 >> 172.21.15.55:0/3479476316 conn(0x7ff6f00cc9c0 msgr2=0x7ff6f00ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:15.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.458+0000 7ff6f6016640 1 -- 172.21.15.55:0/3479476316 shutdown_connections 2024-03-19T02:02:15.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.458+0000 7ff6f6016640 1 -- 172.21.15.55:0/3479476316 wait complete. 2024-03-19T02:02:15.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.458+0000 7ff6f6016640 1 Processor -- start 2024-03-19T02:02:15.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.458+0000 7ff6f6016640 1 -- start start 2024-03-19T02:02:15.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.458+0000 7ff6f6016640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff6f006be50 0x7ff6f006c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:15.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.458+0000 7ff6f6016640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff6f014e930 0x7ff6f0155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:15.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.458+0000 7ff6f6016640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6f0158ae0 0x7ff6f01efda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:15.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.458+0000 7ff6f6016640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f014ed10 con 0x7ff6f0158ae0 2024-03-19T02:02:15.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.458+0000 7ff6f6016640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f0158080 con 0x7ff6f006be50 2024-03-19T02:02:15.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.458+0000 7ff6f6016640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff6f0073600 con 0x7ff6f014e930 2024-03-19T02:02:15.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.458+0000 7ff6effff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6f0158ae0 0x7ff6f01efda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:15.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.458+0000 7ff6eeffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff6f014e930 0x7ff6f0155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:15.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.458+0000 7ff6ef7fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff6f006be50 0x7ff6f006c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:15.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.462+0000 7ff6effff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6f0158ae0 0x7ff6f01efda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50588/0 (socket says 172.21.15.55:50588) 2024-03-19T02:02:15.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.462+0000 7ff6effff640 1 -- 172.21.15.55:0/2668662340 learned_addr learned my addr 172.21.15.55:0/2668662340 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:15.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.462+0000 7ff6effff640 1 -- 172.21.15.55:0/2668662340 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff6f014e930 msgr2=0x7ff6f0155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:15.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.462+0000 7ff6effff640 1 --2- 172.21.15.55:0/2668662340 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff6f014e930 0x7ff6f0155d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:15.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.462+0000 7ff6effff640 1 -- 172.21.15.55:0/2668662340 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff6f006be50 msgr2=0x7ff6f006c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:15.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.462+0000 7ff6effff640 1 --2- 172.21.15.55:0/2668662340 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff6f006be50 0x7ff6f006c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:15.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.462+0000 7ff6effff640 1 -- 172.21.15.55:0/2668662340 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6f01f3310 con 0x7ff6f0158ae0 2024-03-19T02:02:15.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.466+0000 7ff6effff640 1 --2- 172.21.15.55:0/2668662340 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6f0158ae0 0x7ff6f01efda0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff6e800cb70 tx=0x7ff6e800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:15.471 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.470+0000 7ff6d7fff640 1 -- 172.21.15.55:0/2668662340 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8017070 con 0x7ff6f0158ae0 2024-03-19T02:02:15.471 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.470+0000 7ff6d7fff640 1 -- 172.21.15.55:0/2668662340 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6e800ad40 con 0x7ff6f0158ae0 2024-03-19T02:02:15.471 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.470+0000 7ff6d7fff640 1 -- 172.21.15.55:0/2668662340 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8012620 con 0x7ff6f0158ae0 2024-03-19T02:02:15.471 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.470+0000 7ff6f6016640 1 -- 172.21.15.55:0/2668662340 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6f01f35f0 con 0x7ff6f0158ae0 2024-03-19T02:02:15.471 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.470+0000 7ff6f6016640 1 -- 172.21.15.55:0/2668662340 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6f01f3bb0 con 0x7ff6f0158ae0 2024-03-19T02:02:15.472 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.474+0000 7ff6f6016640 1 -- 172.21.15.55:0/2668662340 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6f0068a40 con 0x7ff6f0158ae0 2024-03-19T02:02:15.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.502+0000 7ff6d7fff640 1 -- 172.21.15.55:0/2668662340 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ff6e801a070 con 0x7ff6f0158ae0 2024-03-19T02:02:15.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.502+0000 7ff6d7fff640 1 --2- 172.21.15.55:0/2668662340 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff6dc0676b0 0x7ff6dc069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:15.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.502+0000 7ff6ef7fe640 1 --2- 172.21.15.55:0/2668662340 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff6dc0676b0 0x7ff6dc069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:15.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.502+0000 7ff6ef7fe640 1 --2- 172.21.15.55:0/2668662340 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff6dc0676b0 0x7ff6dc069b70 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff6e00b2000 tx=0x7ff6e00aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:15.506 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.506+0000 7ff6d7fff640 1 -- 172.21.15.55:0/2668662340 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7ff6e8054180 con 0x7ff6f0158ae0 2024-03-19T02:02:15.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:15.790+0000 7ff6d7fff640 1 -- 172.21.15.55:0/2668662340 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ff6e8021020 con 0x7ff6f0158ae0 2024-03-19T02:02:16.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:16.002+0000 7ff6f6016640 1 -- 172.21.15.55:0/2668662340 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6f00729d0 con 0x7ff6f0158ae0 2024-03-19T02:02:16.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:16.026+0000 7ff6d7fff640 1 -- 172.21.15.55:0/2668662340 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7ff6e80560e0 con 0x7ff6f0158ae0 2024-03-19T02:02:16.025 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:16.025 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:16.025 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:16.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:16.026+0000 7ff6f6016640 1 -- 172.21.15.55:0/2668662340 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff6dc0676b0 msgr2=0x7ff6dc069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:16.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:16.026+0000 7ff6f6016640 1 --2- 172.21.15.55:0/2668662340 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff6dc0676b0 0x7ff6dc069b70 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff6e00b2000 tx=0x7ff6e00aa070 comp rx=0 tx=0).stop 2024-03-19T02:02:16.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:16.026+0000 7ff6f6016640 1 -- 172.21.15.55:0/2668662340 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6f0158ae0 msgr2=0x7ff6f01efda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:16.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:16.026+0000 7ff6f6016640 1 --2- 172.21.15.55:0/2668662340 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6f0158ae0 0x7ff6f01efda0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff6e800cb70 tx=0x7ff6e800a430 comp rx=0 tx=0).stop 2024-03-19T02:02:16.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:16.026+0000 7ff6f6016640 1 -- 172.21.15.55:0/2668662340 shutdown_connections 2024-03-19T02:02:16.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:16.026+0000 7ff6f6016640 1 --2- 172.21.15.55:0/2668662340 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff6dc0676b0 0x7ff6dc069b70 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:16.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:16.026+0000 7ff6f6016640 1 --2- 172.21.15.55:0/2668662340 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6f0158ae0 0x7ff6f01efda0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:16.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:16.026+0000 7ff6f6016640 1 --2- 172.21.15.55:0/2668662340 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff6f014e930 0x7ff6f0155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:16.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:16.026+0000 7ff6f6016640 1 --2- 172.21.15.55:0/2668662340 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff6f006be50 0x7ff6f006c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:16.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:16.026+0000 7ff6f6016640 1 -- 172.21.15.55:0/2668662340 >> 172.21.15.55:0/2668662340 conn(0x7ff6f00cc9c0 msgr2=0x7ff6f0070100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:16.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:16.026+0000 7ff6f6016640 1 -- 172.21.15.55:0/2668662340 shutdown_connections 2024-03-19T02:02:16.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:16.026+0000 7ff6f6016640 1 -- 172.21.15.55:0/2668662340 wait complete. 2024-03-19T02:02:16.647 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:16.647 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:17.648 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:17.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.762+0000 7fe611b41640 1 Processor -- start 2024-03-19T02:02:17.761 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.762+0000 7fe611b41640 1 -- start start 2024-03-19T02:02:17.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.762+0000 7fe611b41640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe60c06b2a0 0x7fe60c073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:17.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.762+0000 7fe611b41640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe60c15acb0 0x7fe60c0cce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:17.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.762+0000 7fe611b41640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe60c0c7fa0 con 0x7fe60c1544f0 2024-03-19T02:02:17.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.762+0000 7fe611b41640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fe60c15b210 con 0x7fe60c15acb0 2024-03-19T02:02:17.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.762+0000 7fe611b41640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fe60c0c8240 con 0x7fe60c06b2a0 2024-03-19T02:02:17.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.762+0000 7fe60bfff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe60c15acb0 0x7fe60c0cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:17.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.762+0000 7fe60b7fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fe60c1544f0 0x7fe60c15e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:51678/0 (socket says 172.21.15.55:51678) 2024-03-19T02:02:17.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.762+0000 7fe60b7fe640 1 -- 172.21.15.55:0/3466513013 learned_addr learned my addr 172.21.15.55:0/3466513013 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:02:17.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.762+0000 7fe60affd640 1 --2- 172.21.15.55:0/3466513013 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe60c06b2a0 0x7fe60c073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:17.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.766+0000 7fe60bfff640 1 -- 172.21.15.55:0/3466513013 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe60c06b2a0 msgr2=0x7fe60c073540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:17.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.766+0000 7fe60bfff640 1 --2- 172.21.15.55:0/3466513013 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe60c06b2a0 0x7fe60c073540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:17.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.766+0000 7fe60bfff640 1 -- 172.21.15.55:0/3466513013 >> v1:172.21.15.84:6789/0 conn(0x7fe60c1544f0 legacy=0x7fe60c15e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:17.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.766+0000 7fe60bfff640 1 -- 172.21.15.55:0/3466513013 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe60c15d920 con 0x7fe60c15acb0 2024-03-19T02:02:17.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.770+0000 7fe60bfff640 1 --2- 172.21.15.55:0/3466513013 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe60c15acb0 0x7fe60c0cce30 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe604009d90 tx=0x7fe60401f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd4b1c40a73529b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:17.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.774+0000 7fe60a7fc640 1 -- 172.21.15.55:0/3466513013 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6040b4070 con 0x7fe60c15acb0 2024-03-19T02:02:17.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.774+0000 7fe60a7fc640 1 -- 172.21.15.55:0/3466513013 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6040af600 con 0x7fe60c15acb0 2024-03-19T02:02:17.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.774+0000 7fe60a7fc640 1 -- 172.21.15.55:0/3466513013 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6040af9d0 con 0x7fe60c15acb0 2024-03-19T02:02:17.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.774+0000 7fe611b41640 1 -- 172.21.15.55:0/3466513013 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe60c15acb0 msgr2=0x7fe60c0cce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:17.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.774+0000 7fe611b41640 1 --2- 172.21.15.55:0/3466513013 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe60c15acb0 0x7fe60c0cce30 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe604009d90 tx=0x7fe60401f150 comp rx=0 tx=0).stop 2024-03-19T02:02:17.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.774+0000 7fe611b41640 1 -- 172.21.15.55:0/3466513013 shutdown_connections 2024-03-19T02:02:17.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.774+0000 7fe611b41640 1 --2- 172.21.15.55:0/3466513013 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe60c15acb0 0x7fe60c0cce30 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:17.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.774+0000 7fe611b41640 1 --2- 172.21.15.55:0/3466513013 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe60c06b2a0 0x7fe60c073540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:17.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.774+0000 7fe611b41640 1 -- 172.21.15.55:0/3466513013 >> 172.21.15.55:0/3466513013 conn(0x7fe60c06e670 msgr2=0x7fe60c06ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:17.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.774+0000 7fe611b41640 1 -- 172.21.15.55:0/3466513013 shutdown_connections 2024-03-19T02:02:17.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.774+0000 7fe611b41640 1 -- 172.21.15.55:0/3466513013 wait complete. 2024-03-19T02:02:17.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.778+0000 7fe611b41640 1 Processor -- start 2024-03-19T02:02:17.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.778+0000 7fe611b41640 1 -- start start 2024-03-19T02:02:17.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.778+0000 7fe611b41640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe60c06b2a0 0x7fe60c06c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:17.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.778+0000 7fe611b41640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe60c1544f0 0x7fe60c06b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:17.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.778+0000 7fe611b41640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe60c15acb0 0x7fe60c141480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:17.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.778+0000 7fe611b41640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c0ca120 con 0x7fe60c1544f0 2024-03-19T02:02:17.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.778+0000 7fe611b41640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c1548d0 con 0x7fe60c06b2a0 2024-03-19T02:02:17.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.778+0000 7fe611b41640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fe60c0ca2a0 con 0x7fe60c15acb0 2024-03-19T02:02:17.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.778+0000 7fe60affd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe60c1544f0 0x7fe60c06b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:17.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.778+0000 7fe60b7fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe60c06b2a0 0x7fe60c06c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:17.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.778+0000 7fe60bfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe60c15acb0 0x7fe60c141480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:17.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.778+0000 7fe60affd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe60c1544f0 0x7fe60c06b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50600/0 (socket says 172.21.15.55:50600) 2024-03-19T02:02:17.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.778+0000 7fe60affd640 1 -- 172.21.15.55:0/2177283295 learned_addr learned my addr 172.21.15.55:0/2177283295 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:17.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.786+0000 7fe60affd640 1 -- 172.21.15.55:0/2177283295 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe60c15acb0 msgr2=0x7fe60c141480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:17.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.786+0000 7fe60affd640 1 --2- 172.21.15.55:0/2177283295 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe60c15acb0 0x7fe60c141480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:17.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.786+0000 7fe60affd640 1 -- 172.21.15.55:0/2177283295 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe60c06b2a0 msgr2=0x7fe60c06c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:17.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.786+0000 7fe60affd640 1 --2- 172.21.15.55:0/2177283295 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe60c06b2a0 0x7fe60c06c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:17.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.786+0000 7fe60affd640 1 -- 172.21.15.55:0/2177283295 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe60c1f3920 con 0x7fe60c1544f0 2024-03-19T02:02:17.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.786+0000 7fe60affd640 1 --2- 172.21.15.55:0/2177283295 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe60c1544f0 0x7fe60c06b910 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fe60000e840 tx=0x7fe60000ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:17.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.790+0000 7fe5f3fff640 1 -- 172.21.15.55:0/2177283295 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe600017070 con 0x7fe60c1544f0 2024-03-19T02:02:17.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.794+0000 7fe5f3fff640 1 -- 172.21.15.55:0/2177283295 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe60000c6d0 con 0x7fe60c1544f0 2024-03-19T02:02:17.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.794+0000 7fe5f3fff640 1 -- 172.21.15.55:0/2177283295 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe60000cac0 con 0x7fe60c1544f0 2024-03-19T02:02:17.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.794+0000 7fe611b41640 1 -- 172.21.15.55:0/2177283295 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe60c1f3c00 con 0x7fe60c1544f0 2024-03-19T02:02:17.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.794+0000 7fe611b41640 1 -- 172.21.15.55:0/2177283295 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe60c1f41c0 con 0x7fe60c1544f0 2024-03-19T02:02:17.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:17.794+0000 7fe611b41640 1 -- 172.21.15.55:0/2177283295 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe60c068a40 con 0x7fe60c1544f0 2024-03-19T02:02:18.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:18.086+0000 7fe5f3fff640 1 -- 172.21.15.55:0/2177283295 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fe60001a070 con 0x7fe60c1544f0 2024-03-19T02:02:18.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:18.090+0000 7fe5f3fff640 1 --2- 172.21.15.55:0/2177283295 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe5f80676b0 0x7fe5f8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:18.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:18.090+0000 7fe60b7fe640 1 --2- 172.21.15.55:0/2177283295 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe5f80676b0 0x7fe5f8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:18.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:18.090+0000 7fe60b7fe640 1 --2- 172.21.15.55:0/2177283295 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe5f80676b0 0x7fe5f8069b70 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc000f70 tx=0x7fe5fc007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:18.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:18.130+0000 7fe5f3fff640 1 -- 172.21.15.55:0/2177283295 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fe6000544d0 con 0x7fe60c1544f0 2024-03-19T02:02:18.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:18.134+0000 7fe5f3fff640 1 -- 172.21.15.55:0/2177283295 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fe600056430 con 0x7fe60c1544f0 2024-03-19T02:02:18.369 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:18.370+0000 7fe611b41640 1 -- 172.21.15.55:0/2177283295 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe60c0cae10 con 0x7fe60c1544f0 2024-03-19T02:02:18.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:18.390+0000 7fe5f3fff640 1 -- 172.21.15.55:0/2177283295 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fe60005a4d0 con 0x7fe60c1544f0 2024-03-19T02:02:18.391 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:18.391 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:18.392 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:18.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:18.394+0000 7fe611b41640 1 -- 172.21.15.55:0/2177283295 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe5f80676b0 msgr2=0x7fe5f8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:18.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:18.394+0000 7fe611b41640 1 --2- 172.21.15.55:0/2177283295 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe5f80676b0 0x7fe5f8069b70 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc000f70 tx=0x7fe5fc007e90 comp rx=0 tx=0).stop 2024-03-19T02:02:18.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:18.394+0000 7fe611b41640 1 -- 172.21.15.55:0/2177283295 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe60c1544f0 msgr2=0x7fe60c06b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:18.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:18.394+0000 7fe611b41640 1 --2- 172.21.15.55:0/2177283295 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe60c1544f0 0x7fe60c06b910 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fe60000e840 tx=0x7fe60000ec40 comp rx=0 tx=0).stop 2024-03-19T02:02:18.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:18.394+0000 7fe611b41640 1 -- 172.21.15.55:0/2177283295 shutdown_connections 2024-03-19T02:02:18.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:18.394+0000 7fe611b41640 1 --2- 172.21.15.55:0/2177283295 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe5f80676b0 0x7fe5f8069b70 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:18.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:18.394+0000 7fe611b41640 1 --2- 172.21.15.55:0/2177283295 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe60c15acb0 0x7fe60c141480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:18.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:18.394+0000 7fe611b41640 1 --2- 172.21.15.55:0/2177283295 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe60c1544f0 0x7fe60c06b910 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:18.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:18.394+0000 7fe611b41640 1 --2- 172.21.15.55:0/2177283295 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe60c06b2a0 0x7fe60c06c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:18.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:18.394+0000 7fe611b41640 1 -- 172.21.15.55:0/2177283295 >> 172.21.15.55:0/2177283295 conn(0x7fe60c06e670 msgr2=0x7fe60c0cb030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:18.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:18.394+0000 7fe611b41640 1 -- 172.21.15.55:0/2177283295 shutdown_connections 2024-03-19T02:02:18.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:18.394+0000 7fe611b41640 1 -- 172.21.15.55:0/2177283295 wait complete. 2024-03-19T02:02:19.014 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:19.014 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:20.015 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:20.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.126+0000 7f763b54b640 1 Processor -- start 2024-03-19T02:02:20.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.126+0000 7f763b54b640 1 -- start start 2024-03-19T02:02:20.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.130+0000 7f763b54b640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f763414eaa0 0x7f763406b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:20.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.130+0000 7f763b54b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f76341585f0 0x7f7634158a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:20.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.130+0000 7f763b54b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f763414ee80 con 0x7f763406bec0 2024-03-19T02:02:20.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.130+0000 7f763b54b640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f76340c7540 con 0x7f76341585f0 2024-03-19T02:02:20.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.130+0000 7f763b54b640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f76340c77c0 con 0x7f763414eaa0 2024-03-19T02:02:20.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.130+0000 7f7639ae8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f76341585f0 0x7f7634158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:20.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.130+0000 7f7639ae8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f76341585f0 0x7f7634158a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59090/0 (socket says 172.21.15.55:59090) 2024-03-19T02:02:20.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.130+0000 7f7639ae8640 1 -- 172.21.15.55:0/4092010297 learned_addr learned my addr 172.21.15.55:0/4092010297 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:20.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.130+0000 7f76392e7640 1 --2- 172.21.15.55:0/4092010297 >> v2:172.21.15.84:3300/0 conn(0x7f763414eaa0 0x7f763406b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:20.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.134+0000 7f7639ae8640 1 -- 172.21.15.55:0/4092010297 >> v2:172.21.15.84:3300/0 conn(0x7f763414eaa0 msgr2=0x7f763406b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:20.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.134+0000 7f7639ae8640 1 --2- 172.21.15.55:0/4092010297 >> v2:172.21.15.84:3300/0 conn(0x7f763414eaa0 0x7f763406b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:20.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.134+0000 7f7639ae8640 1 -- 172.21.15.55:0/4092010297 >> v1:172.21.15.84:6789/0 conn(0x7f763406bec0 legacy=0x7f76340c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:20.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.134+0000 7f7639ae8640 1 -- 172.21.15.55:0/4092010297 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76340c7aa0 con 0x7f76341585f0 2024-03-19T02:02:20.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.138+0000 7f7639ae8640 1 --2- 172.21.15.55:0/4092010297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f76341585f0 0x7f7634158a40 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f763406d470 tx=0x7f762c01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0a1b8cd30530437 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:20.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.142+0000 7f762bfff640 1 -- 172.21.15.55:0/4092010297 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762c0a5730 con 0x7f76341585f0 2024-03-19T02:02:20.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.142+0000 7f762bfff640 1 -- 172.21.15.55:0/4092010297 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f762c0a58d0 con 0x7f76341585f0 2024-03-19T02:02:20.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.142+0000 7f762bfff640 1 -- 172.21.15.55:0/4092010297 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762c0a5ca0 con 0x7f76341585f0 2024-03-19T02:02:20.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.142+0000 7f763b54b640 1 -- 172.21.15.55:0/4092010297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f76341585f0 msgr2=0x7f7634158a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:20.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.142+0000 7f763b54b640 1 --2- 172.21.15.55:0/4092010297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f76341585f0 0x7f7634158a40 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f763406d470 tx=0x7f762c01c960 comp rx=0 tx=0).stop 2024-03-19T02:02:20.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.142+0000 7f763b54b640 1 -- 172.21.15.55:0/4092010297 shutdown_connections 2024-03-19T02:02:20.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.142+0000 7f763b54b640 1 --2- 172.21.15.55:0/4092010297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f76341585f0 0x7f7634158a40 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:20.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.142+0000 7f763b54b640 1 --2- 172.21.15.55:0/4092010297 >> v2:172.21.15.84:3300/0 conn(0x7f763414eaa0 0x7f763406b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:20.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.142+0000 7f763b54b640 1 -- 172.21.15.55:0/4092010297 >> 172.21.15.55:0/4092010297 conn(0x7f76340cc9c0 msgr2=0x7f76340ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:20.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.142+0000 7f763b54b640 1 -- 172.21.15.55:0/4092010297 shutdown_connections 2024-03-19T02:02:20.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.142+0000 7f763b54b640 1 -- 172.21.15.55:0/4092010297 wait complete. 2024-03-19T02:02:20.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.146+0000 7f763b54b640 1 Processor -- start 2024-03-19T02:02:20.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.146+0000 7f763b54b640 1 -- start start 2024-03-19T02:02:20.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.146+0000 7f763b54b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f763406bec0 0x7f7634157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:20.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.146+0000 7f763b54b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f763414eaa0 0x7f76341409c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:20.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.146+0000 7f763b54b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f76341585f0 0x7f7634140f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:20.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.146+0000 7f763b54b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f76340c6960 con 0x7f76341585f0 2024-03-19T02:02:20.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.146+0000 7f763b54b640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f76340c67e0 con 0x7f763406bec0 2024-03-19T02:02:20.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.146+0000 7f763b54b640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f76340c6bf0 con 0x7f763414eaa0 2024-03-19T02:02:20.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.146+0000 7f7639ae8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f76341585f0 0x7f7634140f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:20.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.146+0000 7f7639ae8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f76341585f0 0x7f7634140f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59094/0 (socket says 172.21.15.55:59094) 2024-03-19T02:02:20.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.146+0000 7f7639ae8640 1 -- 172.21.15.55:0/2150991006 learned_addr learned my addr 172.21.15.55:0/2150991006 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:20.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.146+0000 7f76392e7640 1 --2- 172.21.15.55:0/2150991006 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f763406bec0 0x7f7634157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:20.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.146+0000 7f7638ae6640 1 --2- 172.21.15.55:0/2150991006 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f763414eaa0 0x7f76341409c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:20.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.150+0000 7f7639ae8640 1 -- 172.21.15.55:0/2150991006 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f763414eaa0 msgr2=0x7f76341409c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:20.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.150+0000 7f7639ae8640 1 --2- 172.21.15.55:0/2150991006 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f763414eaa0 0x7f76341409c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:20.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.150+0000 7f7639ae8640 1 -- 172.21.15.55:0/2150991006 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f763406bec0 msgr2=0x7f7634157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:20.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.150+0000 7f7639ae8640 1 --2- 172.21.15.55:0/2150991006 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f763406bec0 0x7f7634157720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:20.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.150+0000 7f7639ae8640 1 -- 172.21.15.55:0/2150991006 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76340cb1d0 con 0x7f76341585f0 2024-03-19T02:02:20.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.154+0000 7f7639ae8640 1 --2- 172.21.15.55:0/2150991006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f76341585f0 0x7f7634140f00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f762c0af040 tx=0x7f762c019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:20.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.158+0000 7f7629ffb640 1 -- 172.21.15.55:0/2150991006 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762c0b8470 con 0x7f76341585f0 2024-03-19T02:02:20.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.158+0000 7f7629ffb640 1 -- 172.21.15.55:0/2150991006 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f762c0b8670 con 0x7f76341585f0 2024-03-19T02:02:20.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.158+0000 7f763b54b640 1 -- 172.21.15.55:0/2150991006 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76341f6ee0 con 0x7f76341585f0 2024-03-19T02:02:20.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.158+0000 7f7629ffb640 1 -- 172.21.15.55:0/2150991006 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762c0b8a60 con 0x7f76341585f0 2024-03-19T02:02:20.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.158+0000 7f763b54b640 1 -- 172.21.15.55:0/2150991006 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76341401c0 con 0x7f76341585f0 2024-03-19T02:02:20.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.158+0000 7f763b54b640 1 -- 172.21.15.55:0/2150991006 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76340cb6b0 con 0x7f76341585f0 2024-03-19T02:02:20.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.182+0000 7f7629ffb640 1 -- 172.21.15.55:0/2150991006 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f762c0c0030 con 0x7f76341585f0 2024-03-19T02:02:20.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.182+0000 7f7629ffb640 1 --2- 172.21.15.55:0/2150991006 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f76180676b0 0x7f7618069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:20.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.182+0000 7f76392e7640 1 --2- 172.21.15.55:0/2150991006 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f76180676b0 0x7f7618069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:20.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.186+0000 7f76392e7640 1 --2- 172.21.15.55:0/2150991006 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f76180676b0 0x7f7618069b70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7630007a60 tx=0x7f7630008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:20.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.454+0000 7f7629ffb640 1 -- 172.21.15.55:0/2150991006 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f762c0ef6d0 con 0x7f76341585f0 2024-03-19T02:02:20.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.454+0000 7f7629ffb640 1 -- 172.21.15.55:0/2150991006 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f76340cb6b0 con 0x7f76341585f0 2024-03-19T02:02:20.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.666+0000 7f763b54b640 1 -- 172.21.15.55:0/2150991006 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7634068a40 con 0x7f76341585f0 2024-03-19T02:02:20.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.774+0000 7f7629ffb640 1 -- 172.21.15.55:0/2150991006 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f762c0c7020 con 0x7f76341585f0 2024-03-19T02:02:20.773 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:20.773 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:20.773 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:20.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.778+0000 7f763b54b640 1 -- 172.21.15.55:0/2150991006 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f76180676b0 msgr2=0x7f7618069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:20.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.778+0000 7f763b54b640 1 --2- 172.21.15.55:0/2150991006 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f76180676b0 0x7f7618069b70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7630007a60 tx=0x7f7630008040 comp rx=0 tx=0).stop 2024-03-19T02:02:20.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.778+0000 7f763b54b640 1 -- 172.21.15.55:0/2150991006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f76341585f0 msgr2=0x7f7634140f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:20.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.778+0000 7f763b54b640 1 --2- 172.21.15.55:0/2150991006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f76341585f0 0x7f7634140f00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f762c0af040 tx=0x7f762c019700 comp rx=0 tx=0).stop 2024-03-19T02:02:20.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.778+0000 7f763b54b640 1 -- 172.21.15.55:0/2150991006 shutdown_connections 2024-03-19T02:02:20.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.778+0000 7f763b54b640 1 --2- 172.21.15.55:0/2150991006 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f76180676b0 0x7f7618069b70 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:20.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.778+0000 7f763b54b640 1 --2- 172.21.15.55:0/2150991006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f76341585f0 0x7f7634140f00 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:20.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.778+0000 7f763b54b640 1 --2- 172.21.15.55:0/2150991006 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f763414eaa0 0x7f76341409c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:20.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.778+0000 7f763b54b640 1 --2- 172.21.15.55:0/2150991006 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f763406bec0 0x7f7634157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:20.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.778+0000 7f763b54b640 1 -- 172.21.15.55:0/2150991006 >> 172.21.15.55:0/2150991006 conn(0x7f76340cc9c0 msgr2=0x7f7634145310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:20.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.778+0000 7f763b54b640 1 -- 172.21.15.55:0/2150991006 shutdown_connections 2024-03-19T02:02:20.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:20.778+0000 7f763b54b640 1 -- 172.21.15.55:0/2150991006 wait complete. 2024-03-19T02:02:21.398 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:21.398 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:22.399 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:22.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.514+0000 7fe98bafe640 1 Processor -- start 2024-03-19T02:02:22.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.514+0000 7fe98bafe640 1 -- start start 2024-03-19T02:02:22.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.514+0000 7fe98bafe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe984158420 0x7fe984158800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:22.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.514+0000 7fe98bafe640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe98415b740 con 0x7fe98406f1b0 2024-03-19T02:02:22.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.514+0000 7fe98bafe640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe98415ba20 con 0x7fe98413de70 2024-03-19T02:02:22.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.514+0000 7fe98bafe640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fe98413e250 con 0x7fe984158420 2024-03-19T02:02:22.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.518+0000 7fe98a09b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe984158420 0x7fe984158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:22.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.518+0000 7fe98989a640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fe98413de70 0x7fe98406e630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:44816/0 (socket says 172.21.15.55:44816) 2024-03-19T02:02:22.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.518+0000 7fe98989a640 1 -- 172.21.15.55:0/1502756112 learned_addr learned my addr 172.21.15.55:0/1502756112 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:02:22.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.518+0000 7fe98a09b640 1 -- 172.21.15.55:0/1502756112 >> v1:172.21.15.84:6789/0 conn(0x7fe98413de70 legacy=0x7fe98406e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:22.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.518+0000 7fe98a09b640 1 -- 172.21.15.55:0/1502756112 >> v1:172.21.15.55:6789/0 conn(0x7fe98406f1b0 legacy=0x7fe9840c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:22.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.518+0000 7fe98a09b640 1 -- 172.21.15.55:0/1502756112 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe98415bd00 con 0x7fe984158420 2024-03-19T02:02:22.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.526+0000 7fe98a09b640 1 --2- 172.21.15.55:0/1502756112 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe984158420 0x7fe984158800 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe97c009d90 tx=0x7fe97c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a352f476a9160279 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:22.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.530+0000 7fe988898640 1 -- 172.21.15.55:0/1502756112 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c0b2070 con 0x7fe984158420 2024-03-19T02:02:22.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.530+0000 7fe988898640 1 -- 172.21.15.55:0/1502756112 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe97c0a5830 con 0x7fe984158420 2024-03-19T02:02:22.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.530+0000 7fe988898640 1 -- 172.21.15.55:0/1502756112 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c0a5c00 con 0x7fe984158420 2024-03-19T02:02:22.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.530+0000 7fe98bafe640 1 -- 172.21.15.55:0/1502756112 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe984158420 msgr2=0x7fe984158800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:22.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.530+0000 7fe98bafe640 1 --2- 172.21.15.55:0/1502756112 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe984158420 0x7fe984158800 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe97c009d90 tx=0x7fe97c01c970 comp rx=0 tx=0).stop 2024-03-19T02:02:22.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.530+0000 7fe98bafe640 1 -- 172.21.15.55:0/1502756112 shutdown_connections 2024-03-19T02:02:22.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.530+0000 7fe98bafe640 1 --2- 172.21.15.55:0/1502756112 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe984158420 0x7fe984158800 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:22.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.530+0000 7fe98bafe640 1 -- 172.21.15.55:0/1502756112 >> 172.21.15.55:0/1502756112 conn(0x7fe98406fb40 msgr2=0x7fe98406ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:22.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.530+0000 7fe98bafe640 1 -- 172.21.15.55:0/1502756112 shutdown_connections 2024-03-19T02:02:22.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.530+0000 7fe98bafe640 1 -- 172.21.15.55:0/1502756112 wait complete. 2024-03-19T02:02:22.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.534+0000 7fe98bafe640 1 Processor -- start 2024-03-19T02:02:22.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.534+0000 7fe98bafe640 1 -- start start 2024-03-19T02:02:22.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.534+0000 7fe98bafe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe98406f1b0 0x7fe98415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:22.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.534+0000 7fe98bafe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe98413de70 0x7fe9841f4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:22.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.534+0000 7fe98bafe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe984158420 0x7fe9841f5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:22.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.534+0000 7fe98bafe640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe9840c5f20 con 0x7fe98406f1b0 2024-03-19T02:02:22.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.534+0000 7fe98bafe640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fe984158d40 con 0x7fe98413de70 2024-03-19T02:02:22.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.534+0000 7fe98bafe640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fe9840c60a0 con 0x7fe984158420 2024-03-19T02:02:22.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.534+0000 7fe989099640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe98413de70 0x7fe9841f4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:22.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.538+0000 7fe989099640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe98413de70 0x7fe9841f4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:41762/0 (socket says 172.21.15.55:41762) 2024-03-19T02:02:22.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.538+0000 7fe989099640 1 -- 172.21.15.55:0/1405922126 learned_addr learned my addr 172.21.15.55:0/1405922126 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:22.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.538+0000 7fe98a09b640 1 --2- 172.21.15.55:0/1405922126 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe984158420 0x7fe9841f5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:22.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.542+0000 7fe98a09b640 1 -- 172.21.15.55:0/1405922126 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe98413de70 msgr2=0x7fe9841f4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:22.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.542+0000 7fe98a09b640 1 --2- 172.21.15.55:0/1405922126 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe98413de70 0x7fe9841f4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:22.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.542+0000 7fe98a09b640 1 -- 172.21.15.55:0/1405922126 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe98406f1b0 msgr2=0x7fe98415ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:22.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.542+0000 7fe98a09b640 1 --2- 172.21.15.55:0/1405922126 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe98406f1b0 0x7fe98415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:22.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.542+0000 7fe98a09b640 1 -- 172.21.15.55:0/1405922126 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe98415a000 con 0x7fe984158420 2024-03-19T02:02:22.543 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.546+0000 7fe98a09b640 1 --2- 172.21.15.55:0/1405922126 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe984158420 0x7fe9841f5220 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe97c0ac000 tx=0x7fe97c019710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:22.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.550+0000 7fe97a7fc640 1 -- 172.21.15.55:0/1405922126 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c0b2050 con 0x7fe984158420 2024-03-19T02:02:22.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.550+0000 7fe97a7fc640 1 -- 172.21.15.55:0/1405922126 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe97c0b3040 con 0x7fe984158420 2024-03-19T02:02:22.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.550+0000 7fe97a7fc640 1 -- 172.21.15.55:0/1405922126 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c0ad850 con 0x7fe984158420 2024-03-19T02:02:22.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.550+0000 7fe98bafe640 1 -- 172.21.15.55:0/1405922126 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9840cae50 con 0x7fe984158420 2024-03-19T02:02:22.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.550+0000 7fe98bafe640 1 -- 172.21.15.55:0/1405922126 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe98415a270 con 0x7fe984158420 2024-03-19T02:02:22.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.550+0000 7fe98bafe640 1 -- 172.21.15.55:0/1405922126 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe984068a40 con 0x7fe984158420 2024-03-19T02:02:22.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.818+0000 7fe97a7fc640 1 -- 172.21.15.55:0/1405922126 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fe97c0c0020 con 0x7fe984158420 2024-03-19T02:02:22.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.818+0000 7fe97a7fc640 1 --2- 172.21.15.55:0/1405922126 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe9680676e0 0x7fe968069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:22.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.818+0000 7fe98989a640 1 --2- 172.21.15.55:0/1405922126 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe9680676e0 0x7fe968069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:22.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.818+0000 7fe98989a640 1 --2- 172.21.15.55:0/1405922126 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe9680676e0 0x7fe968069ba0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe980001560 tx=0x7fe980007a50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:22.820 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.822+0000 7fe97a7fc640 1 -- 172.21.15.55:0/1405922126 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fe97c0ef2a0 con 0x7fe984158420 2024-03-19T02:02:22.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:22.846+0000 7fe97a7fc640 1 -- 172.21.15.55:0/1405922126 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fe97c016560 con 0x7fe984158420 2024-03-19T02:02:23.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:23.058+0000 7fe98bafe640 1 -- 172.21.15.55:0/1405922126 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9841526e0 con 0x7fe984158420 2024-03-19T02:02:23.229 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:23.230+0000 7fe97a7fc640 1 -- 172.21.15.55:0/1405922126 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fe97c0f1900 con 0x7fe984158420 2024-03-19T02:02:23.229 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:23.229 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:23.229 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:23.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:23.234+0000 7fe98bafe640 1 -- 172.21.15.55:0/1405922126 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe9680676e0 msgr2=0x7fe968069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:23.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:23.234+0000 7fe98bafe640 1 --2- 172.21.15.55:0/1405922126 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe9680676e0 0x7fe968069ba0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe980001560 tx=0x7fe980007a50 comp rx=0 tx=0).stop 2024-03-19T02:02:23.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:23.234+0000 7fe98bafe640 1 -- 172.21.15.55:0/1405922126 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe984158420 msgr2=0x7fe9841f5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:23.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:23.234+0000 7fe98bafe640 1 --2- 172.21.15.55:0/1405922126 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe984158420 0x7fe9841f5220 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe97c0ac000 tx=0x7fe97c019710 comp rx=0 tx=0).stop 2024-03-19T02:02:23.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:23.234+0000 7fe98bafe640 1 -- 172.21.15.55:0/1405922126 shutdown_connections 2024-03-19T02:02:23.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:23.234+0000 7fe98bafe640 1 --2- 172.21.15.55:0/1405922126 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe9680676e0 0x7fe968069ba0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:23.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:23.234+0000 7fe98bafe640 1 --2- 172.21.15.55:0/1405922126 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe984158420 0x7fe9841f5220 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:23.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:23.234+0000 7fe98bafe640 1 --2- 172.21.15.55:0/1405922126 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe98413de70 0x7fe9841f4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:23.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:23.234+0000 7fe98bafe640 1 --2- 172.21.15.55:0/1405922126 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe98406f1b0 0x7fe98415ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:23.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:23.234+0000 7fe98bafe640 1 -- 172.21.15.55:0/1405922126 >> 172.21.15.55:0/1405922126 conn(0x7fe98406fb40 msgr2=0x7fe98413bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:23.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:23.234+0000 7fe98bafe640 1 -- 172.21.15.55:0/1405922126 shutdown_connections 2024-03-19T02:02:23.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:23.234+0000 7fe98bafe640 1 -- 172.21.15.55:0/1405922126 wait complete. 2024-03-19T02:02:23.853 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:23.853 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:24.854 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:24.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.970+0000 7fa750aa1640 1 Processor -- start 2024-03-19T02:02:24.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.970+0000 7fa750aa1640 1 -- start start 2024-03-19T02:02:24.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.970+0000 7fa750aa1640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa74c070780 0x7fa74c15b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:24.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.970+0000 7fa750aa1640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa74c15bb00 0x7fa74c15bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:24.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.970+0000 7fa750aa1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa74c15c4d0 con 0x7fa74c152120 2024-03-19T02:02:24.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.970+0000 7fa750aa1640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fa74c152500 con 0x7fa74c070780 2024-03-19T02:02:24.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.970+0000 7fa750aa1640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa74c070590 con 0x7fa74c15bb00 2024-03-19T02:02:24.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.970+0000 7fa749d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa74c070780 0x7fa74c15b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:24.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.970+0000 7fa74ad9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa74c15bb00 0x7fa74c15bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:24.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.970+0000 7fa74a59c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fa74c152120 0x7fa74c0c90d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:44828/0 (socket says 172.21.15.55:44828) 2024-03-19T02:02:24.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.970+0000 7fa74a59c640 1 -- 172.21.15.55:0/1450311862 learned_addr learned my addr 172.21.15.55:0/1450311862 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:02:24.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.974+0000 7fa749d9b640 1 -- 172.21.15.55:0/1450311862 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa74c15bb00 msgr2=0x7fa74c15bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:24.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.974+0000 7fa749d9b640 1 --2- 172.21.15.55:0/1450311862 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa74c15bb00 0x7fa74c15bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:24.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.974+0000 7fa749d9b640 1 -- 172.21.15.55:0/1450311862 >> v1:172.21.15.84:6789/0 conn(0x7fa74c152120 legacy=0x7fa74c0c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:24.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.974+0000 7fa749d9b640 1 -- 172.21.15.55:0/1450311862 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa74c06fff0 con 0x7fa74c070780 2024-03-19T02:02:24.979 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.978+0000 7fa749d9b640 1 --2- 172.21.15.55:0/1450311862 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa74c070780 0x7fa74c15b5c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa7400097c0 tx=0x7fa74001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc6a13f93f2fa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:24.984 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.986+0000 7fa74959a640 1 -- 172.21.15.55:0/1450311862 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7400b0070 con 0x7fa74c070780 2024-03-19T02:02:24.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.986+0000 7fa74959a640 1 -- 172.21.15.55:0/1450311862 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7400ab600 con 0x7fa74c070780 2024-03-19T02:02:24.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.986+0000 7fa74959a640 1 -- 172.21.15.55:0/1450311862 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7400ab9d0 con 0x7fa74c070780 2024-03-19T02:02:24.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.986+0000 7fa750aa1640 1 -- 172.21.15.55:0/1450311862 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa74c070780 msgr2=0x7fa74c15b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:24.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.986+0000 7fa750aa1640 1 --2- 172.21.15.55:0/1450311862 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa74c070780 0x7fa74c15b5c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa7400097c0 tx=0x7fa74001c4a0 comp rx=0 tx=0).stop 2024-03-19T02:02:24.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.986+0000 7fa750aa1640 1 -- 172.21.15.55:0/1450311862 shutdown_connections 2024-03-19T02:02:24.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.986+0000 7fa750aa1640 1 --2- 172.21.15.55:0/1450311862 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa74c15bb00 0x7fa74c15bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:24.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.986+0000 7fa750aa1640 1 --2- 172.21.15.55:0/1450311862 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa74c070780 0x7fa74c15b5c0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:24.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.986+0000 7fa750aa1640 1 -- 172.21.15.55:0/1450311862 >> 172.21.15.55:0/1450311862 conn(0x7fa74c067600 msgr2=0x7fa74c071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:24.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.986+0000 7fa750aa1640 1 -- 172.21.15.55:0/1450311862 shutdown_connections 2024-03-19T02:02:24.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.986+0000 7fa750aa1640 1 -- 172.21.15.55:0/1450311862 wait complete. 2024-03-19T02:02:24.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.986+0000 7fa750aa1640 1 Processor -- start 2024-03-19T02:02:24.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.986+0000 7fa750aa1640 1 -- start start 2024-03-19T02:02:24.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.990+0000 7fa750aa1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa74c070780 0x7fa74c14bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:24.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.990+0000 7fa750aa1640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa74c152120 0x7fa74c0b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:24.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.990+0000 7fa750aa1640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa74c15bb00 0x7fa74c0b24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:24.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.990+0000 7fa750aa1640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa74c0c31c0 con 0x7fa74c070780 2024-03-19T02:02:24.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.990+0000 7fa750aa1640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa74c0c2ee0 con 0x7fa74c15bb00 2024-03-19T02:02:24.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.990+0000 7fa750aa1640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa74c0c97e0 con 0x7fa74c152120 2024-03-19T02:02:24.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.990+0000 7fa74a59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa74c070780 0x7fa74c14bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:24.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.990+0000 7fa749d9b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa74c152120 0x7fa74c0b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:24.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.990+0000 7fa74a59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa74c070780 0x7fa74c14bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59128/0 (socket says 172.21.15.55:59128) 2024-03-19T02:02:24.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.990+0000 7fa74a59c640 1 -- 172.21.15.55:0/345563313 learned_addr learned my addr 172.21.15.55:0/345563313 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:24.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.990+0000 7fa74ad9d640 1 --2- 172.21.15.55:0/345563313 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa74c15bb00 0x7fa74c0b24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:24.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.994+0000 7fa74a59c640 1 -- 172.21.15.55:0/345563313 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa74c152120 msgr2=0x7fa74c0b1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:24.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.994+0000 7fa74a59c640 1 --2- 172.21.15.55:0/345563313 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa74c152120 0x7fa74c0b1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:24.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.994+0000 7fa74a59c640 1 -- 172.21.15.55:0/345563313 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa74c15bb00 msgr2=0x7fa74c0b24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:24.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.994+0000 7fa74a59c640 1 --2- 172.21.15.55:0/345563313 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa74c15bb00 0x7fa74c0b24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:24.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.994+0000 7fa74a59c640 1 -- 172.21.15.55:0/345563313 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa74c0c5050 con 0x7fa74c070780 2024-03-19T02:02:24.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:24.994+0000 7fa74a59c640 1 --2- 172.21.15.55:0/345563313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa74c070780 0x7fa74c14bb10 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa73c00c450 tx=0x7fa73c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:25.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.002+0000 7fa73affd640 1 -- 172.21.15.55:0/345563313 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73c017070 con 0x7fa74c070780 2024-03-19T02:02:25.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.002+0000 7fa73affd640 1 -- 172.21.15.55:0/345563313 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa73c005cf0 con 0x7fa74c070780 2024-03-19T02:02:25.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.002+0000 7fa750aa1640 1 -- 172.21.15.55:0/345563313 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa74c0cc460 con 0x7fa74c070780 2024-03-19T02:02:25.003 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.002+0000 7fa73affd640 1 -- 172.21.15.55:0/345563313 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73c007290 con 0x7fa74c070780 2024-03-19T02:02:25.003 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.002+0000 7fa750aa1640 1 -- 172.21.15.55:0/345563313 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa74c0c52f0 con 0x7fa74c070780 2024-03-19T02:02:25.003 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.002+0000 7fa750aa1640 1 -- 172.21.15.55:0/345563313 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa74c068a40 con 0x7fa74c070780 2024-03-19T02:02:25.270 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.270+0000 7fa73affd640 1 -- 172.21.15.55:0/345563313 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa73c01a070 con 0x7fa74c070780 2024-03-19T02:02:25.270 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.270+0000 7fa73affd640 1 --2- 172.21.15.55:0/345563313 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa72c0676e0 0x7fa72c069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:25.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.270+0000 7fa749d9b640 1 --2- 172.21.15.55:0/345563313 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa72c0676e0 0x7fa72c069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:25.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.270+0000 7fa749d9b640 1 --2- 172.21.15.55:0/345563313 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa72c0676e0 0x7fa72c069ba0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa7400ad000 tx=0x7fa7400a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:25.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.302+0000 7fa73affd640 1 -- 172.21.15.55:0/345563313 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fa73c053660 con 0x7fa74c070780 2024-03-19T02:02:25.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.302+0000 7fa73affd640 1 -- 172.21.15.55:0/345563313 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa73c021020 con 0x7fa74c070780 2024-03-19T02:02:25.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.534+0000 7fa750aa1640 1 -- 172.21.15.55:0/345563313 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa74c158ec0 con 0x7fa74c070780 2024-03-19T02:02:25.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.554+0000 7fa73affd640 1 -- 172.21.15.55:0/345563313 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fa74c15c4d0 con 0x7fa74c070780 2024-03-19T02:02:25.555 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:25.555 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:25.555 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:25.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.558+0000 7fa750aa1640 1 -- 172.21.15.55:0/345563313 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa72c0676e0 msgr2=0x7fa72c069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:25.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.558+0000 7fa750aa1640 1 --2- 172.21.15.55:0/345563313 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa72c0676e0 0x7fa72c069ba0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa7400ad000 tx=0x7fa7400a6000 comp rx=0 tx=0).stop 2024-03-19T02:02:25.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.558+0000 7fa750aa1640 1 -- 172.21.15.55:0/345563313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa74c070780 msgr2=0x7fa74c14bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:25.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.558+0000 7fa750aa1640 1 --2- 172.21.15.55:0/345563313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa74c070780 0x7fa74c14bb10 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa73c00c450 tx=0x7fa73c00c910 comp rx=0 tx=0).stop 2024-03-19T02:02:25.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.558+0000 7fa750aa1640 1 -- 172.21.15.55:0/345563313 shutdown_connections 2024-03-19T02:02:25.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.558+0000 7fa750aa1640 1 --2- 172.21.15.55:0/345563313 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa72c0676e0 0x7fa72c069ba0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:25.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.558+0000 7fa750aa1640 1 --2- 172.21.15.55:0/345563313 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa74c15bb00 0x7fa74c0b24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:25.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.558+0000 7fa750aa1640 1 --2- 172.21.15.55:0/345563313 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa74c152120 0x7fa74c0b1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:25.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.558+0000 7fa750aa1640 1 --2- 172.21.15.55:0/345563313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa74c070780 0x7fa74c14bb10 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:25.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.558+0000 7fa750aa1640 1 -- 172.21.15.55:0/345563313 >> 172.21.15.55:0/345563313 conn(0x7fa74c067600 msgr2=0x7fa74c150f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:25.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.558+0000 7fa750aa1640 1 -- 172.21.15.55:0/345563313 shutdown_connections 2024-03-19T02:02:25.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:25.558+0000 7fa750aa1640 1 -- 172.21.15.55:0/345563313 wait complete. 2024-03-19T02:02:26.178 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:26.178 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:27.179 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:27.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.306+0000 7f74b6d10640 1 Processor -- start 2024-03-19T02:02:27.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.306+0000 7f74b6d10640 1 -- start start 2024-03-19T02:02:27.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.306+0000 7f74b6d10640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f74b0154290 0x7f74b015e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:27.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.306+0000 7f74b6d10640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f74b00ca310 0x7f74b0154670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:27.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.306+0000 7f74b6d10640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f74b015ad40 0x7f74b0072900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:27.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.306+0000 7f74b6d10640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f74b015b2c0 con 0x7f74b0154290 2024-03-19T02:02:27.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.306+0000 7f74b6d10640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f74b006c1d0 con 0x7f74b00ca310 2024-03-19T02:02:27.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.306+0000 7f74b6d10640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f74b015e030 con 0x7f74b015ad40 2024-03-19T02:02:27.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.310+0000 7f74b4aac640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f74b0154290 0x7f74b015e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:27.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.310+0000 7f74b52ad640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f74b015ad40 0x7f74b0072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:27.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.310+0000 7f74affff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f74b00ca310 0x7f74b0154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:27.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.310+0000 7f74b4aac640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f74b0154290 0x7f74b015e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59134/0 (socket says 172.21.15.55:59134) 2024-03-19T02:02:27.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.310+0000 7f74b4aac640 1 -- 172.21.15.55:0/2855673192 learned_addr learned my addr 172.21.15.55:0/2855673192 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:27.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.310+0000 7f74b4aac640 1 -- 172.21.15.55:0/2855673192 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f74b015ad40 msgr2=0x7f74b0072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:27.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.310+0000 7f74b4aac640 1 --2- 172.21.15.55:0/2855673192 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f74b015ad40 0x7f74b0072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:27.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.310+0000 7f74b4aac640 1 -- 172.21.15.55:0/2855673192 >> v2:172.21.15.84:3300/0 conn(0x7f74b00ca310 msgr2=0x7f74b0154670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:27.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.310+0000 7f74b4aac640 1 --2- 172.21.15.55:0/2855673192 >> v2:172.21.15.84:3300/0 conn(0x7f74b00ca310 0x7f74b0154670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:27.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.310+0000 7f74b4aac640 1 -- 172.21.15.55:0/2855673192 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74b006bc90 con 0x7f74b0154290 2024-03-19T02:02:27.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.314+0000 7f74b4aac640 1 --2- 172.21.15.55:0/2855673192 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74b0154290 0x7f74b015e4c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f74b00c72c0 tx=0x7f74a401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff6e39ad3d8ede7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:27.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.318+0000 7f74af7fe640 1 -- 172.21.15.55:0/2855673192 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a40aa660 con 0x7f74b0154290 2024-03-19T02:02:27.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.318+0000 7f74af7fe640 1 -- 172.21.15.55:0/2855673192 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74a40af070 con 0x7f74b0154290 2024-03-19T02:02:27.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.318+0000 7f74af7fe640 1 -- 172.21.15.55:0/2855673192 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a40aaa20 con 0x7f74b0154290 2024-03-19T02:02:27.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.322+0000 7f74b6d10640 1 -- 172.21.15.55:0/2855673192 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74b0154290 msgr2=0x7f74b015e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:27.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.322+0000 7f74b6d10640 1 --2- 172.21.15.55:0/2855673192 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74b0154290 0x7f74b015e4c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f74b00c72c0 tx=0x7f74a401c520 comp rx=0 tx=0).stop 2024-03-19T02:02:27.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.322+0000 7f74b6d10640 1 -- 172.21.15.55:0/2855673192 shutdown_connections 2024-03-19T02:02:27.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.322+0000 7f74b6d10640 1 --2- 172.21.15.55:0/2855673192 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f74b015ad40 0x7f74b0072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:27.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.322+0000 7f74b6d10640 1 --2- 172.21.15.55:0/2855673192 >> v2:172.21.15.84:3300/0 conn(0x7f74b00ca310 0x7f74b0154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:27.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.322+0000 7f74b6d10640 1 --2- 172.21.15.55:0/2855673192 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74b0154290 0x7f74b015e4c0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:27.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.322+0000 7f74b6d10640 1 -- 172.21.15.55:0/2855673192 >> 172.21.15.55:0/2855673192 conn(0x7f74b006c580 msgr2=0x7f74b006c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:27.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.322+0000 7f74b6d10640 1 -- 172.21.15.55:0/2855673192 shutdown_connections 2024-03-19T02:02:27.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.322+0000 7f74b6d10640 1 -- 172.21.15.55:0/2855673192 wait complete. 2024-03-19T02:02:27.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.322+0000 7f74b6d10640 1 Processor -- start 2024-03-19T02:02:27.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.322+0000 7f74b6d10640 1 -- start start 2024-03-19T02:02:27.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.326+0000 7f74b6d10640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74b00ca310 0x7f74b0154ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:27.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.326+0000 7f74b6d10640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74b0154290 0x7f74b014f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:27.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.326+0000 7f74b6d10640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f74b015ad40 0x7f74b00c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:27.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.326+0000 7f74b6d10640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f74b00ccb50 con 0x7f74b00ca310 2024-03-19T02:02:27.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.326+0000 7f74b6d10640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f74b006bf30 con 0x7f74b0154290 2024-03-19T02:02:27.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.326+0000 7f74b6d10640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f74b00cccd0 con 0x7f74b015ad40 2024-03-19T02:02:27.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.326+0000 7f74b4aac640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74b00ca310 0x7f74b0154ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:27.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.326+0000 7f74b52ad640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f74b015ad40 0x7f74b00c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:27.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.326+0000 7f74b4aac640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74b00ca310 0x7f74b0154ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59144/0 (socket says 172.21.15.55:59144) 2024-03-19T02:02:27.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.326+0000 7f74b4aac640 1 -- 172.21.15.55:0/2272173039 learned_addr learned my addr 172.21.15.55:0/2272173039 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:27.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.326+0000 7f74affff640 1 --2- 172.21.15.55:0/2272173039 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74b0154290 0x7f74b014f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:27.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.330+0000 7f74b4aac640 1 -- 172.21.15.55:0/2272173039 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f74b015ad40 msgr2=0x7f74b00c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:27.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.330+0000 7f74b4aac640 1 --2- 172.21.15.55:0/2272173039 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f74b015ad40 0x7f74b00c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:27.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.330+0000 7f74b4aac640 1 -- 172.21.15.55:0/2272173039 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74b0154290 msgr2=0x7f74b014f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:27.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.330+0000 7f74b4aac640 1 --2- 172.21.15.55:0/2272173039 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74b0154290 0x7f74b014f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:27.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.330+0000 7f74b4aac640 1 -- 172.21.15.55:0/2272173039 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74b01f3e40 con 0x7f74b00ca310 2024-03-19T02:02:27.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.330+0000 7f74b4aac640 1 --2- 172.21.15.55:0/2272173039 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74b00ca310 0x7f74b0154ef0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f74a401abd0 tx=0x7f74a401ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:27.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.334+0000 7f74ad7fa640 1 -- 172.21.15.55:0/2272173039 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a40bd3f0 con 0x7f74b00ca310 2024-03-19T02:02:27.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.334+0000 7f74ad7fa640 1 -- 172.21.15.55:0/2272173039 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74a40bd590 con 0x7f74b00ca310 2024-03-19T02:02:27.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.334+0000 7f74b6d10640 1 -- 172.21.15.55:0/2272173039 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74b014d400 con 0x7f74b00ca310 2024-03-19T02:02:27.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.334+0000 7f74ad7fa640 1 -- 172.21.15.55:0/2272173039 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a40bd960 con 0x7f74b00ca310 2024-03-19T02:02:27.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.334+0000 7f74b6d10640 1 -- 172.21.15.55:0/2272173039 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74b014dd20 con 0x7f74b00ca310 2024-03-19T02:02:27.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.338+0000 7f74b6d10640 1 -- 172.21.15.55:0/2272173039 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74b0068a40 con 0x7f74b00ca310 2024-03-19T02:02:27.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.362+0000 7f74ad7fa640 1 -- 172.21.15.55:0/2272173039 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f74a40c7420 con 0x7f74b00ca310 2024-03-19T02:02:27.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.362+0000 7f74ad7fa640 1 --2- 172.21.15.55:0/2272173039 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f74940676e0 0x7f7494069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:27.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.362+0000 7f74affff640 1 --2- 172.21.15.55:0/2272173039 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f74940676e0 0x7f7494069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:27.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.362+0000 7f74affff640 1 --2- 172.21.15.55:0/2272173039 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f74940676e0 0x7f7494069ba0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f74b01f76e0 tx=0x7f749c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:27.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.366+0000 7f74ad7fa640 1 -- 172.21.15.55:0/2272173039 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f74a40ee780 con 0x7f74b00ca310 2024-03-19T02:02:27.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.646+0000 7f74ad7fa640 1 -- 172.21.15.55:0/2272173039 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f74a40c3060 con 0x7f74b00ca310 2024-03-19T02:02:27.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.862+0000 7f74b6d10640 1 -- 172.21.15.55:0/2272173039 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74b0073650 con 0x7f74b00ca310 2024-03-19T02:02:27.891 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.890+0000 7f74ad7fa640 1 -- 172.21.15.55:0/2272173039 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f74a4016110 con 0x7f74b00ca310 2024-03-19T02:02:27.891 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:27.891 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:27.891 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:27.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.894+0000 7f74b6d10640 1 -- 172.21.15.55:0/2272173039 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f74940676e0 msgr2=0x7f7494069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:27.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.894+0000 7f74b6d10640 1 --2- 172.21.15.55:0/2272173039 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f74940676e0 0x7f7494069ba0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f74b01f76e0 tx=0x7f749c009190 comp rx=0 tx=0).stop 2024-03-19T02:02:27.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.894+0000 7f74b6d10640 1 -- 172.21.15.55:0/2272173039 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74b00ca310 msgr2=0x7f74b0154ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:27.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.894+0000 7f74b6d10640 1 --2- 172.21.15.55:0/2272173039 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74b00ca310 0x7f74b0154ef0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f74a401abd0 tx=0x7f74a401ad60 comp rx=0 tx=0).stop 2024-03-19T02:02:27.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.894+0000 7f74b6d10640 1 -- 172.21.15.55:0/2272173039 shutdown_connections 2024-03-19T02:02:27.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.894+0000 7f74b6d10640 1 --2- 172.21.15.55:0/2272173039 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f74940676e0 0x7f7494069ba0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:27.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.894+0000 7f74b6d10640 1 --2- 172.21.15.55:0/2272173039 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f74b015ad40 0x7f74b00c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:27.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.894+0000 7f74b6d10640 1 --2- 172.21.15.55:0/2272173039 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74b0154290 0x7f74b014f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:27.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.894+0000 7f74b6d10640 1 --2- 172.21.15.55:0/2272173039 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74b00ca310 0x7f74b0154ef0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:27.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.894+0000 7f74b6d10640 1 -- 172.21.15.55:0/2272173039 >> 172.21.15.55:0/2272173039 conn(0x7f74b006c580 msgr2=0x7f74b006c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:27.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.894+0000 7f74b6d10640 1 -- 172.21.15.55:0/2272173039 shutdown_connections 2024-03-19T02:02:27.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:27.894+0000 7f74b6d10640 1 -- 172.21.15.55:0/2272173039 wait complete. 2024-03-19T02:02:28.513 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:28.513 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:29.514 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:29.632 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.630+0000 7f3fe6efb640 1 Processor -- start 2024-03-19T02:02:29.632 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.630+0000 7f3fe6efb640 1 -- start start 2024-03-19T02:02:29.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.634+0000 7f3fe6efb640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fe01544b0 0x7f3fe006b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:29.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.634+0000 7f3fe6efb640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3fe015a0e0 0x7f3fe00cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:29.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.634+0000 7f3fe6efb640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fe015a6b0 con 0x7f3fe00cc4f0 2024-03-19T02:02:29.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.634+0000 7f3fe6efb640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f3fe015a990 con 0x7f3fe015a0e0 2024-03-19T02:02:29.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.634+0000 7f3fe6efb640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3fe015ac70 con 0x7f3fe01544b0 2024-03-19T02:02:29.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.634+0000 7f3fdffff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3fe015a0e0 0x7f3fe00cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:29.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.634+0000 7f3fe4c97640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fe01544b0 0x7f3fe006b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:29.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.634+0000 7f3fdffff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3fe015a0e0 0x7f3fe00cbfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57872/0 (socket says 172.21.15.55:57872) 2024-03-19T02:02:29.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.634+0000 7f3fdffff640 1 -- 172.21.15.55:0/4211682569 learned_addr learned my addr 172.21.15.55:0/4211682569 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:29.636 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.638+0000 7f3fdffff640 1 -- 172.21.15.55:0/4211682569 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fe01544b0 msgr2=0x7f3fe006b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:29.636 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.638+0000 7f3fdffff640 1 --2- 172.21.15.55:0/4211682569 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fe01544b0 0x7f3fe006b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:29.636 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.638+0000 7f3fdffff640 1 -- 172.21.15.55:0/4211682569 >> v1:172.21.15.84:6789/0 conn(0x7f3fe00cc4f0 legacy=0x7f3fe00c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:29.636 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.638+0000 7f3fdffff640 1 -- 172.21.15.55:0/4211682569 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fe015af50 con 0x7f3fe015a0e0 2024-03-19T02:02:29.639 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.642+0000 7f3fdffff640 1 --2- 172.21.15.55:0/4211682569 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fe015a0e0 0x7f3fe00cbfb0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc009870 tx=0x7f3fcc01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bec1dc1c57aec50 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:29.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.646+0000 7f3fdf7fe640 1 -- 172.21.15.55:0/4211682569 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc0b5070 con 0x7f3fe015a0e0 2024-03-19T02:02:29.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.646+0000 7f3fdf7fe640 1 -- 172.21.15.55:0/4211682569 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fcc0a8a50 con 0x7f3fe015a0e0 2024-03-19T02:02:29.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.646+0000 7f3fdf7fe640 1 -- 172.21.15.55:0/4211682569 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc0a8e20 con 0x7f3fe015a0e0 2024-03-19T02:02:29.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.646+0000 7f3fe6efb640 1 -- 172.21.15.55:0/4211682569 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fe015a0e0 msgr2=0x7f3fe00cbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:29.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.646+0000 7f3fe6efb640 1 --2- 172.21.15.55:0/4211682569 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fe015a0e0 0x7f3fe00cbfb0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc009870 tx=0x7f3fcc01ed50 comp rx=0 tx=0).stop 2024-03-19T02:02:29.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.646+0000 7f3fe6efb640 1 -- 172.21.15.55:0/4211682569 shutdown_connections 2024-03-19T02:02:29.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.646+0000 7f3fe6efb640 1 --2- 172.21.15.55:0/4211682569 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fe015a0e0 0x7f3fe00cbfb0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:29.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.646+0000 7f3fe6efb640 1 --2- 172.21.15.55:0/4211682569 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fe01544b0 0x7f3fe006b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:29.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.646+0000 7f3fe6efb640 1 -- 172.21.15.55:0/4211682569 >> 172.21.15.55:0/4211682569 conn(0x7f3fe006e5a0 msgr2=0x7f3fe0072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:29.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.646+0000 7f3fe6efb640 1 -- 172.21.15.55:0/4211682569 shutdown_connections 2024-03-19T02:02:29.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.646+0000 7f3fe6efb640 1 -- 172.21.15.55:0/4211682569 wait complete. 2024-03-19T02:02:29.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.650+0000 7f3fe6efb640 1 Processor -- start 2024-03-19T02:02:29.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.650+0000 7f3fe6efb640 1 -- start start 2024-03-19T02:02:29.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.650+0000 7f3fe6efb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fe00cc4f0 0x7f3fe01511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:29.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.650+0000 7f3fe6efb640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fe01544b0 0x7f3fe01e6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:29.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.650+0000 7f3fe6efb640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3fe015a0e0 0x7f3fe0150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:29.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.650+0000 7f3fe6efb640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe006b7e0 con 0x7f3fe00cc4f0 2024-03-19T02:02:29.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.650+0000 7f3fe6efb640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe00c6be0 con 0x7f3fe015a0e0 2024-03-19T02:02:29.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.650+0000 7f3fe6efb640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3fe00c8760 con 0x7f3fe01544b0 2024-03-19T02:02:29.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.650+0000 7f3fe4c97640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fe00cc4f0 0x7f3fe01511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:29.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.650+0000 7f3fdffff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fe01544b0 0x7f3fe01e6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:29.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.650+0000 7f3fe4c97640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fe00cc4f0 0x7f3fe01511e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57878/0 (socket says 172.21.15.55:57878) 2024-03-19T02:02:29.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.650+0000 7f3fe4c97640 1 -- 172.21.15.55:0/3146463837 learned_addr learned my addr 172.21.15.55:0/3146463837 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:29.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.650+0000 7f3fe5498640 1 --2- 172.21.15.55:0/3146463837 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3fe015a0e0 0x7f3fe0150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:29.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.654+0000 7f3fe4c97640 1 -- 172.21.15.55:0/3146463837 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fe01544b0 msgr2=0x7f3fe01e6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:29.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.654+0000 7f3fe4c97640 1 --2- 172.21.15.55:0/3146463837 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fe01544b0 0x7f3fe01e6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:29.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.654+0000 7f3fe4c97640 1 -- 172.21.15.55:0/3146463837 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3fe015a0e0 msgr2=0x7f3fe0150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:29.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.654+0000 7f3fe4c97640 1 --2- 172.21.15.55:0/3146463837 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3fe015a0e0 0x7f3fe0150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:29.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.654+0000 7f3fe4c97640 1 -- 172.21.15.55:0/3146463837 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fe01f0ce0 con 0x7f3fe00cc4f0 2024-03-19T02:02:29.657 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.658+0000 7f3fe4c97640 1 --2- 172.21.15.55:0/3146463837 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fe00cc4f0 0x7f3fe01511e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f3fd400e930 tx=0x7f3fd400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:29.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.662+0000 7f3fdd7fa640 1 -- 172.21.15.55:0/3146463837 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd4017070 con 0x7f3fe00cc4f0 2024-03-19T02:02:29.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.662+0000 7f3fdd7fa640 1 -- 172.21.15.55:0/3146463837 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fd400ca00 con 0x7f3fe00cc4f0 2024-03-19T02:02:29.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.662+0000 7f3fe6efb640 1 -- 172.21.15.55:0/3146463837 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fe01f0a00 con 0x7f3fe00cc4f0 2024-03-19T02:02:29.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.662+0000 7f3fdd7fa640 1 -- 172.21.15.55:0/3146463837 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd400cdb0 con 0x7f3fe00cc4f0 2024-03-19T02:02:29.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.662+0000 7f3fe6efb640 1 -- 172.21.15.55:0/3146463837 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fe014f010 con 0x7f3fe00cc4f0 2024-03-19T02:02:29.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.666+0000 7f3fe6efb640 1 -- 172.21.15.55:0/3146463837 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fe006bb10 con 0x7f3fe00cc4f0 2024-03-19T02:02:29.688 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.690+0000 7f3fdd7fa640 1 -- 172.21.15.55:0/3146463837 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f3fd401a070 con 0x7f3fe00cc4f0 2024-03-19T02:02:29.689 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.690+0000 7f3fdd7fa640 1 --2- 172.21.15.55:0/3146463837 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3fc40676b0 0x7f3fc4069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:29.689 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.690+0000 7f3fdffff640 1 --2- 172.21.15.55:0/3146463837 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3fc40676b0 0x7f3fc4069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:29.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.690+0000 7f3fdffff640 1 --2- 172.21.15.55:0/3146463837 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3fc40676b0 0x7f3fc4069b70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc0b2000 tx=0x7f3fcc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:29.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.998+0000 7f3fdd7fa640 1 -- 172.21.15.55:0/3146463837 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f3fd40547b0 con 0x7f3fe00cc4f0 2024-03-19T02:02:29.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:29.998+0000 7f3fdd7fa640 1 -- 172.21.15.55:0/3146463837 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f3fd4056710 con 0x7f3fe00cc4f0 2024-03-19T02:02:30.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:30.202+0000 7f3fe6efb640 1 -- 172.21.15.55:0/3146463837 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3fe0151720 con 0x7f3fe00cc4f0 2024-03-19T02:02:30.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:30.222+0000 7f3fdd7fa640 1 -- 172.21.15.55:0/3146463837 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f3fd405a7b0 con 0x7f3fe00cc4f0 2024-03-19T02:02:30.223 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:30.224 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:30.224 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:30.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:30.226+0000 7f3fe6efb640 1 -- 172.21.15.55:0/3146463837 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3fc40676b0 msgr2=0x7f3fc4069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:30.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:30.226+0000 7f3fe6efb640 1 --2- 172.21.15.55:0/3146463837 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3fc40676b0 0x7f3fc4069b70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc0b2000 tx=0x7f3fcc009040 comp rx=0 tx=0).stop 2024-03-19T02:02:30.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:30.226+0000 7f3fe6efb640 1 -- 172.21.15.55:0/3146463837 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fe00cc4f0 msgr2=0x7f3fe01511e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:30.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:30.226+0000 7f3fe6efb640 1 --2- 172.21.15.55:0/3146463837 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fe00cc4f0 0x7f3fe01511e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f3fd400e930 tx=0x7f3fd400edf0 comp rx=0 tx=0).stop 2024-03-19T02:02:30.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:30.226+0000 7f3fe6efb640 1 -- 172.21.15.55:0/3146463837 shutdown_connections 2024-03-19T02:02:30.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:30.226+0000 7f3fe6efb640 1 --2- 172.21.15.55:0/3146463837 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3fc40676b0 0x7f3fc4069b70 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:30.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:30.226+0000 7f3fe6efb640 1 --2- 172.21.15.55:0/3146463837 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3fe015a0e0 0x7f3fe0150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:30.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:30.226+0000 7f3fe6efb640 1 --2- 172.21.15.55:0/3146463837 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3fe01544b0 0x7f3fe01e6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:30.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:30.226+0000 7f3fe6efb640 1 --2- 172.21.15.55:0/3146463837 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3fe00cc4f0 0x7f3fe01511e0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:30.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:30.226+0000 7f3fe6efb640 1 -- 172.21.15.55:0/3146463837 >> 172.21.15.55:0/3146463837 conn(0x7f3fe006e5a0 msgr2=0x7f3fe00cb2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:30.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:30.226+0000 7f3fe6efb640 1 -- 172.21.15.55:0/3146463837 shutdown_connections 2024-03-19T02:02:30.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:30.226+0000 7f3fe6efb640 1 -- 172.21.15.55:0/3146463837 wait complete. 2024-03-19T02:02:30.847 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:30.847 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:31.848 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:31.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.970+0000 7f11e2572640 1 Processor -- start 2024-03-19T02:02:31.970 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.970+0000 7f11e2572640 1 -- start start 2024-03-19T02:02:31.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.970+0000 7f11e2572640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f11dc14eaa0 0x7f11dc06b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:31.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.970+0000 7f11e2572640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f11dc1585f0 0x7f11dc158a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:31.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.970+0000 7f11e2572640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11dc14ee80 con 0x7f11dc06bec0 2024-03-19T02:02:31.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.970+0000 7f11e2572640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f11dc158f50 con 0x7f11dc14eaa0 2024-03-19T02:02:31.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.970+0000 7f11e2572640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f11dc0c7540 con 0x7f11dc1585f0 2024-03-19T02:02:31.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.974+0000 7f11dbfff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f11dc14eaa0 0x7f11dc06b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:31.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.974+0000 7f11e0b0f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f11dc1585f0 0x7f11dc158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:31.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.974+0000 7f11dbfff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f11dc14eaa0 0x7f11dc06b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:51488/0 (socket says 172.21.15.55:51488) 2024-03-19T02:02:31.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.974+0000 7f11dbfff640 1 -- 172.21.15.55:0/2190222165 learned_addr learned my addr 172.21.15.55:0/2190222165 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:31.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.978+0000 7f11dbfff640 1 -- 172.21.15.55:0/2190222165 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f11dc1585f0 msgr2=0x7f11dc158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:31.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.978+0000 7f11dbfff640 1 --2- 172.21.15.55:0/2190222165 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f11dc1585f0 0x7f11dc158a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:31.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.978+0000 7f11dbfff640 1 -- 172.21.15.55:0/2190222165 >> v1:172.21.15.84:6789/0 conn(0x7f11dc06bec0 legacy=0x7f11dc0c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:31.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.978+0000 7f11dbfff640 1 -- 172.21.15.55:0/2190222165 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11dc0c77c0 con 0x7f11dc14eaa0 2024-03-19T02:02:31.981 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.982+0000 7f11dbfff640 1 --2- 172.21.15.55:0/2190222165 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f11dc14eaa0 0x7f11dc06b980 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f11cc009870 tx=0x7f11cc01c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=704c07075d949a1a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:31.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.986+0000 7f11daffd640 1 -- 172.21.15.55:0/2190222165 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11cc0ab650 con 0x7f11dc14eaa0 2024-03-19T02:02:31.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.986+0000 7f11daffd640 1 -- 172.21.15.55:0/2190222165 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11cc0b0070 con 0x7f11dc14eaa0 2024-03-19T02:02:31.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.986+0000 7f11daffd640 1 -- 172.21.15.55:0/2190222165 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11cc0aba10 con 0x7f11dc14eaa0 2024-03-19T02:02:31.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.986+0000 7f11e2572640 1 -- 172.21.15.55:0/2190222165 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f11dc14eaa0 msgr2=0x7f11dc06b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:31.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.986+0000 7f11e2572640 1 --2- 172.21.15.55:0/2190222165 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f11dc14eaa0 0x7f11dc06b980 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f11cc009870 tx=0x7f11cc01c490 comp rx=0 tx=0).stop 2024-03-19T02:02:31.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.986+0000 7f11e2572640 1 -- 172.21.15.55:0/2190222165 shutdown_connections 2024-03-19T02:02:31.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.986+0000 7f11e2572640 1 --2- 172.21.15.55:0/2190222165 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f11dc1585f0 0x7f11dc158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:31.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.986+0000 7f11e2572640 1 --2- 172.21.15.55:0/2190222165 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f11dc14eaa0 0x7f11dc06b980 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:31.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.986+0000 7f11e2572640 1 -- 172.21.15.55:0/2190222165 >> 172.21.15.55:0/2190222165 conn(0x7f11dc0cc9c0 msgr2=0x7f11dc0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:31.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.986+0000 7f11e2572640 1 -- 172.21.15.55:0/2190222165 shutdown_connections 2024-03-19T02:02:31.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.986+0000 7f11e2572640 1 -- 172.21.15.55:0/2190222165 wait complete. 2024-03-19T02:02:31.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.990+0000 7f11e2572640 1 Processor -- start 2024-03-19T02:02:31.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.990+0000 7f11e2572640 1 -- start start 2024-03-19T02:02:31.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.990+0000 7f11e2572640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f11dc06bec0 0x7f11dc14ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:31.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.990+0000 7f11e2572640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11dc14eaa0 0x7f11dc0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:31.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.990+0000 7f11e2572640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f11dc1585f0 0x7f11dc1f4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:31.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.990+0000 7f11e2572640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc0c67e0 con 0x7f11dc14eaa0 2024-03-19T02:02:31.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.990+0000 7f11e2572640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc0c8060 con 0x7f11dc1585f0 2024-03-19T02:02:31.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.990+0000 7f11e2572640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f11dc0c6960 con 0x7f11dc06bec0 2024-03-19T02:02:31.991 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.990+0000 7f11db7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11dc14eaa0 0x7f11dc0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:31.991 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.990+0000 7f11dbfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f11dc06bec0 0x7f11dc14ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:31.991 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.990+0000 7f11e0b0f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f11dc1585f0 0x7f11dc1f4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:31.991 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.990+0000 7f11db7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11dc14eaa0 0x7f11dc0cbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57882/0 (socket says 172.21.15.55:57882) 2024-03-19T02:02:31.991 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.990+0000 7f11db7fe640 1 -- 172.21.15.55:0/3947671726 learned_addr learned my addr 172.21.15.55:0/3947671726 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:31.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.994+0000 7f11db7fe640 1 -- 172.21.15.55:0/3947671726 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f11dc06bec0 msgr2=0x7f11dc14ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:31.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.994+0000 7f11db7fe640 1 --2- 172.21.15.55:0/3947671726 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f11dc06bec0 0x7f11dc14ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:31.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.994+0000 7f11db7fe640 1 -- 172.21.15.55:0/3947671726 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f11dc1585f0 msgr2=0x7f11dc1f4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:31.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.994+0000 7f11db7fe640 1 --2- 172.21.15.55:0/3947671726 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f11dc1585f0 0x7f11dc1f4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:31.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.994+0000 7f11db7fe640 1 -- 172.21.15.55:0/3947671726 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11dc159310 con 0x7f11dc14eaa0 2024-03-19T02:02:31.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:31.998+0000 7f11db7fe640 1 --2- 172.21.15.55:0/3947671726 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11dc14eaa0 0x7f11dc0cbba0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f11d000b410 tx=0x7f11d000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:32.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.002+0000 7f11d8ff9640 1 -- 172.21.15.55:0/3947671726 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0015020 con 0x7f11dc14eaa0 2024-03-19T02:02:32.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.002+0000 7f11d8ff9640 1 -- 172.21.15.55:0/3947671726 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11d0011830 con 0x7f11dc14eaa0 2024-03-19T02:02:32.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.002+0000 7f11d8ff9640 1 -- 172.21.15.55:0/3947671726 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0011c20 con 0x7f11dc14eaa0 2024-03-19T02:02:32.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.002+0000 7f11e2572640 1 -- 172.21.15.55:0/3947671726 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11dc0cc5e0 con 0x7f11dc14eaa0 2024-03-19T02:02:32.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.002+0000 7f11e2572640 1 -- 172.21.15.55:0/3947671726 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11dc0c3520 con 0x7f11dc14eaa0 2024-03-19T02:02:32.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.002+0000 7f11e2572640 1 -- 172.21.15.55:0/3947671726 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11dc154580 con 0x7f11dc14eaa0 2024-03-19T02:02:32.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.022+0000 7f11d8ff9640 1 -- 172.21.15.55:0/3947671726 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f11d0018020 con 0x7f11dc14eaa0 2024-03-19T02:02:32.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.022+0000 7f11d8ff9640 1 --2- 172.21.15.55:0/3947671726 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f11c8067690 0x7f11c8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:32.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.022+0000 7f11dbfff640 1 --2- 172.21.15.55:0/3947671726 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f11c8067690 0x7f11c8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:32.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.026+0000 7f11dbfff640 1 --2- 172.21.15.55:0/3947671726 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f11c8067690 0x7f11c8069b50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f11cc0af000 tx=0x7f11cc0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:32.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.314+0000 7f11d8ff9640 1 -- 172.21.15.55:0/3947671726 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f11d0054330 con 0x7f11dc14eaa0 2024-03-19T02:02:32.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.314+0000 7f11d8ff9640 1 -- 172.21.15.55:0/3947671726 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f11d001f020 con 0x7f11dc14eaa0 2024-03-19T02:02:32.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.518+0000 7f11e2572640 1 -- 172.21.15.55:0/3947671726 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11dc068a40 con 0x7f11dc14eaa0 2024-03-19T02:02:32.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.534+0000 7f11d8ff9640 1 -- 172.21.15.55:0/3947671726 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f11d0056290 con 0x7f11dc14eaa0 2024-03-19T02:02:32.534 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:32.534 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:32.534 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:32.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.538+0000 7f11e2572640 1 -- 172.21.15.55:0/3947671726 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f11c8067690 msgr2=0x7f11c8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:32.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.538+0000 7f11e2572640 1 --2- 172.21.15.55:0/3947671726 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f11c8067690 0x7f11c8069b50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f11cc0af000 tx=0x7f11cc0a7000 comp rx=0 tx=0).stop 2024-03-19T02:02:32.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.538+0000 7f11e2572640 1 -- 172.21.15.55:0/3947671726 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11dc14eaa0 msgr2=0x7f11dc0cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:32.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.538+0000 7f11e2572640 1 --2- 172.21.15.55:0/3947671726 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11dc14eaa0 0x7f11dc0cbba0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f11d000b410 tx=0x7f11d000b8d0 comp rx=0 tx=0).stop 2024-03-19T02:02:32.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.538+0000 7f11e2572640 1 -- 172.21.15.55:0/3947671726 shutdown_connections 2024-03-19T02:02:32.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.538+0000 7f11e2572640 1 --2- 172.21.15.55:0/3947671726 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f11c8067690 0x7f11c8069b50 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:32.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.538+0000 7f11e2572640 1 --2- 172.21.15.55:0/3947671726 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f11dc1585f0 0x7f11dc1f4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:32.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.538+0000 7f11e2572640 1 --2- 172.21.15.55:0/3947671726 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11dc14eaa0 0x7f11dc0cbba0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:32.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.538+0000 7f11e2572640 1 --2- 172.21.15.55:0/3947671726 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f11dc06bec0 0x7f11dc14ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:32.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.538+0000 7f11e2572640 1 -- 172.21.15.55:0/3947671726 >> 172.21.15.55:0/3947671726 conn(0x7f11dc0cc9c0 msgr2=0x7f11dc149b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:32.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.538+0000 7f11e2572640 1 -- 172.21.15.55:0/3947671726 shutdown_connections 2024-03-19T02:02:32.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:32.538+0000 7f11e2572640 1 -- 172.21.15.55:0/3947671726 wait complete. 2024-03-19T02:02:33.158 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:33.158 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:34.160 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:34.288 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.290+0000 7fe7bc4a1640 1 Processor -- start 2024-03-19T02:02:34.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.290+0000 7fe7bc4a1640 1 -- start start 2024-03-19T02:02:34.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.290+0000 7fe7bc4a1640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fe7b415a0e0 0x7fe7b40cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:34.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.290+0000 7fe7bc4a1640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7b415a6b0 con 0x7fe7b40cc4f0 2024-03-19T02:02:34.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.290+0000 7fe7bc4a1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7b415a990 con 0x7fe7b41544b0 2024-03-19T02:02:34.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.290+0000 7fe7bc4a1640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fe7b415ac70 con 0x7fe7b415a0e0 2024-03-19T02:02:34.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.290+0000 7fe7baa3e640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fe7b40cc4f0 0x7fe7b40c9050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:51378/0 (socket says 172.21.15.55:51378) 2024-03-19T02:02:34.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.290+0000 7fe7baa3e640 1 -- 172.21.15.55:0/3677081152 learned_addr learned my addr 172.21.15.55:0/3677081152 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:02:34.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.290+0000 7fe7b9a3c640 1 --2- 172.21.15.55:0/3677081152 >> v2:172.21.15.84:3300/0 conn(0x7fe7b415a0e0 0x7fe7b40cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:34.293 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.294+0000 7fe7b923b640 1 -- 172.21.15.55:0/3677081152 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 611193576 0 0) 0x7fe7b415a6b0 con 0x7fe7b40cc4f0 2024-03-19T02:02:34.293 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.294+0000 7fe7b923b640 1 -- 172.21.15.55:0/3677081152 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7a0003530 con 0x7fe7b40cc4f0 2024-03-19T02:02:34.294 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.294+0000 7fe7b9a3c640 1 -- 172.21.15.55:0/3677081152 >> v1:172.21.15.84:6789/0 conn(0x7fe7b41544b0 legacy=0x7fe7b406b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:34.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.294+0000 7fe7b9a3c640 1 -- 172.21.15.55:0/3677081152 >> v1:172.21.15.55:6789/0 conn(0x7fe7b40cc4f0 legacy=0x7fe7b40c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:34.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.294+0000 7fe7b9a3c640 1 -- 172.21.15.55:0/3677081152 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7b415af50 con 0x7fe7b415a0e0 2024-03-19T02:02:34.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.298+0000 7fe7b9a3c640 1 --2- 172.21.15.55:0/3677081152 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7b415a0e0 0x7fe7b40cbfb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe7a40097c0 tx=0x7fe7a401ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d77889cd775f6b69 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:34.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.306+0000 7fe7b923b640 1 -- 172.21.15.55:0/3677081152 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a40b5070 con 0x7fe7b415a0e0 2024-03-19T02:02:34.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.306+0000 7fe7b923b640 1 -- 172.21.15.55:0/3677081152 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7a40a8950 con 0x7fe7b415a0e0 2024-03-19T02:02:34.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.306+0000 7fe7b923b640 1 -- 172.21.15.55:0/3677081152 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a40a8d20 con 0x7fe7b415a0e0 2024-03-19T02:02:34.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.306+0000 7fe7bc4a1640 1 -- 172.21.15.55:0/3677081152 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7b415a0e0 msgr2=0x7fe7b40cbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:34.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.306+0000 7fe7bc4a1640 1 --2- 172.21.15.55:0/3677081152 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7b415a0e0 0x7fe7b40cbfb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe7a40097c0 tx=0x7fe7a401ec50 comp rx=0 tx=0).stop 2024-03-19T02:02:34.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.306+0000 7fe7bc4a1640 1 -- 172.21.15.55:0/3677081152 shutdown_connections 2024-03-19T02:02:34.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.306+0000 7fe7bc4a1640 1 --2- 172.21.15.55:0/3677081152 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7b415a0e0 0x7fe7b40cbfb0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:34.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.306+0000 7fe7bc4a1640 1 -- 172.21.15.55:0/3677081152 >> 172.21.15.55:0/3677081152 conn(0x7fe7b406e5a0 msgr2=0x7fe7b4072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:34.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.306+0000 7fe7bc4a1640 1 -- 172.21.15.55:0/3677081152 shutdown_connections 2024-03-19T02:02:34.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.306+0000 7fe7bc4a1640 1 -- 172.21.15.55:0/3677081152 wait complete. 2024-03-19T02:02:34.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.306+0000 7fe7bc4a1640 1 Processor -- start 2024-03-19T02:02:34.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.306+0000 7fe7bc4a1640 1 -- start start 2024-03-19T02:02:34.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.310+0000 7fe7bc4a1640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7b40cc4f0 0x7fe7b41e5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:34.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.310+0000 7fe7bc4a1640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7b41544b0 0x7fe7b41e48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:34.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.310+0000 7fe7bc4a1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b415a0e0 0x7fe7b41e4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:34.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.310+0000 7fe7bc4a1640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b40c8760 con 0x7fe7b415a0e0 2024-03-19T02:02:34.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.310+0000 7fe7bc4a1640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b40c6be0 con 0x7fe7b40cc4f0 2024-03-19T02:02:34.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.310+0000 7fe7bc4a1640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fe7b40c88e0 con 0x7fe7b41544b0 2024-03-19T02:02:34.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.310+0000 7fe7ba23d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7b40cc4f0 0x7fe7b41e5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:34.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.310+0000 7fe7baa3e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b415a0e0 0x7fe7b41e4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:34.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.310+0000 7fe7b9a3c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7b41544b0 0x7fe7b41e48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:34.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.310+0000 7fe7baa3e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b415a0e0 0x7fe7b41e4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57886/0 (socket says 172.21.15.55:57886) 2024-03-19T02:02:34.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.310+0000 7fe7baa3e640 1 -- 172.21.15.55:0/2850629897 learned_addr learned my addr 172.21.15.55:0/2850629897 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:34.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.314+0000 7fe7baa3e640 1 -- 172.21.15.55:0/2850629897 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7b41544b0 msgr2=0x7fe7b41e48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:34.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.314+0000 7fe7baa3e640 1 --2- 172.21.15.55:0/2850629897 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7b41544b0 0x7fe7b41e48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:34.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.314+0000 7fe7baa3e640 1 -- 172.21.15.55:0/2850629897 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7b40cc4f0 msgr2=0x7fe7b41e5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:34.313 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.314+0000 7fe7baa3e640 1 --2- 172.21.15.55:0/2850629897 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7b40cc4f0 0x7fe7b41e5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:34.313 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.314+0000 7fe7baa3e640 1 -- 172.21.15.55:0/2850629897 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7b41e3510 con 0x7fe7b415a0e0 2024-03-19T02:02:34.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.314+0000 7fe7baa3e640 1 --2- 172.21.15.55:0/2850629897 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b415a0e0 0x7fe7b41e4e30 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac00cb40 tx=0x7fe7ac00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:34.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.322+0000 7fe7aaffd640 1 -- 172.21.15.55:0/2850629897 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ac017070 con 0x7fe7b415a0e0 2024-03-19T02:02:34.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.322+0000 7fe7aaffd640 1 -- 172.21.15.55:0/2850629897 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7ac00aca0 con 0x7fe7b415a0e0 2024-03-19T02:02:34.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.322+0000 7fe7aaffd640 1 -- 172.21.15.55:0/2850629897 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ac012620 con 0x7fe7b415a0e0 2024-03-19T02:02:34.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.322+0000 7fe7bc4a1640 1 -- 172.21.15.55:0/2850629897 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7b41e6180 con 0x7fe7b415a0e0 2024-03-19T02:02:34.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.322+0000 7fe7bc4a1640 1 -- 172.21.15.55:0/2850629897 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7b41f0c00 con 0x7fe7b415a0e0 2024-03-19T02:02:34.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.322+0000 7fe7bc4a1640 1 -- 172.21.15.55:0/2850629897 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7b4068a40 con 0x7fe7b415a0e0 2024-03-19T02:02:34.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.358+0000 7fe7aaffd640 1 -- 172.21.15.55:0/2850629897 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fe7ac01a070 con 0x7fe7b415a0e0 2024-03-19T02:02:34.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.358+0000 7fe7aaffd640 1 --2- 172.21.15.55:0/2850629897 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe7980676b0 0x7fe798069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:34.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.358+0000 7fe7ba23d640 1 --2- 172.21.15.55:0/2850629897 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe7980676b0 0x7fe798069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:34.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.358+0000 7fe7ba23d640 1 --2- 172.21.15.55:0/2850629897 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe7980676b0 0x7fe798069b70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0007fd0 tx=0x7fe7b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:34.737 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.738+0000 7fe7aaffd640 1 -- 172.21.15.55:0/2850629897 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fe7ac0541c0 con 0x7fe7b415a0e0 2024-03-19T02:02:34.737 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.738+0000 7fe7aaffd640 1 -- 172.21.15.55:0/2850629897 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fe7ac021020 con 0x7fe7b415a0e0 2024-03-19T02:02:34.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:34.954+0000 7fe7bc4a1640 1 -- 172.21.15.55:0/2850629897 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7b40cb160 con 0x7fe7b415a0e0 2024-03-19T02:02:35.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:35.018+0000 7fe7aaffd640 1 -- 172.21.15.55:0/2850629897 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fe7ac056120 con 0x7fe7b415a0e0 2024-03-19T02:02:35.018 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:35.019 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:35.019 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:35.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:35.022+0000 7fe7bc4a1640 1 -- 172.21.15.55:0/2850629897 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe7980676b0 msgr2=0x7fe798069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:35.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:35.022+0000 7fe7bc4a1640 1 --2- 172.21.15.55:0/2850629897 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe7980676b0 0x7fe798069b70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0007fd0 tx=0x7fe7b0009040 comp rx=0 tx=0).stop 2024-03-19T02:02:35.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:35.022+0000 7fe7bc4a1640 1 -- 172.21.15.55:0/2850629897 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b415a0e0 msgr2=0x7fe7b41e4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:35.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:35.022+0000 7fe7bc4a1640 1 --2- 172.21.15.55:0/2850629897 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b415a0e0 0x7fe7b41e4e30 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac00cb40 tx=0x7fe7ac00a060 comp rx=0 tx=0).stop 2024-03-19T02:02:35.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:35.022+0000 7fe7bc4a1640 1 -- 172.21.15.55:0/2850629897 shutdown_connections 2024-03-19T02:02:35.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:35.022+0000 7fe7bc4a1640 1 --2- 172.21.15.55:0/2850629897 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe7980676b0 0x7fe798069b70 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:35.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:35.022+0000 7fe7bc4a1640 1 --2- 172.21.15.55:0/2850629897 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b415a0e0 0x7fe7b41e4e30 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:35.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:35.022+0000 7fe7bc4a1640 1 --2- 172.21.15.55:0/2850629897 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe7b41544b0 0x7fe7b41e48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:35.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:35.022+0000 7fe7bc4a1640 1 --2- 172.21.15.55:0/2850629897 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7b40cc4f0 0x7fe7b41e5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:35.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:35.022+0000 7fe7bc4a1640 1 -- 172.21.15.55:0/2850629897 >> 172.21.15.55:0/2850629897 conn(0x7fe7b406e5a0 msgr2=0x7fe7b40c7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:35.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:35.022+0000 7fe7bc4a1640 1 -- 172.21.15.55:0/2850629897 shutdown_connections 2024-03-19T02:02:35.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:35.022+0000 7fe7bc4a1640 1 -- 172.21.15.55:0/2850629897 wait complete. 2024-03-19T02:02:35.640 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:35.640 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:36.641 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:36.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.758+0000 7f3c10ea7640 1 Processor -- start 2024-03-19T02:02:36.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.758+0000 7f3c10ea7640 1 -- start start 2024-03-19T02:02:36.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.758+0000 7f3c10ea7640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3c0c1544f0 0x7f3c0c073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:36.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.758+0000 7f3c10ea7640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3c0c15acb0 0x7f3c0c0cce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:36.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.758+0000 7f3c10ea7640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c0c0c7fa0 con 0x7f3c0c06b2a0 2024-03-19T02:02:36.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.758+0000 7f3c10ea7640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f3c0c15b210 con 0x7f3c0c1544f0 2024-03-19T02:02:36.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.758+0000 7f3c10ea7640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3c0c0c8240 con 0x7f3c0c15acb0 2024-03-19T02:02:36.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.758+0000 7f3c0ad9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3c0c15acb0 0x7f3c0c0cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:36.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.758+0000 7f3c0a59c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3c0c1544f0 0x7f3c0c073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:36.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.762+0000 7f3c0a59c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3c0c1544f0 0x7f3c0c073540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57900/0 (socket says 172.21.15.55:57900) 2024-03-19T02:02:36.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.762+0000 7f3c0a59c640 1 -- 172.21.15.55:0/2179175363 learned_addr learned my addr 172.21.15.55:0/2179175363 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:36.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.762+0000 7f3c0a59c640 1 -- 172.21.15.55:0/2179175363 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3c0c15acb0 msgr2=0x7f3c0c0cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:36.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.762+0000 7f3c0a59c640 1 --2- 172.21.15.55:0/2179175363 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3c0c15acb0 0x7f3c0c0cce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:36.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.762+0000 7f3c0a59c640 1 -- 172.21.15.55:0/2179175363 >> v1:172.21.15.55:6789/0 conn(0x7f3c0c06b2a0 legacy=0x7f3c0c15e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:36.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.762+0000 7f3c0a59c640 1 -- 172.21.15.55:0/2179175363 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c0c15d920 con 0x7f3c0c1544f0 2024-03-19T02:02:36.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.770+0000 7f3c0a59c640 1 --2- 172.21.15.55:0/2179175363 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c0c1544f0 0x7f3c0c073540 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc009870 tx=0x7f3bfc01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff119e23bf1dc65 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:36.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.774+0000 7f3c0959a640 1 -- 172.21.15.55:0/2179175363 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc0b5070 con 0x7f3c0c1544f0 2024-03-19T02:02:36.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.774+0000 7f3c0959a640 1 -- 172.21.15.55:0/2179175363 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bfc0a8a50 con 0x7f3c0c1544f0 2024-03-19T02:02:36.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.774+0000 7f3c0959a640 1 -- 172.21.15.55:0/2179175363 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc0a8e20 con 0x7f3c0c1544f0 2024-03-19T02:02:36.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.774+0000 7f3c10ea7640 1 -- 172.21.15.55:0/2179175363 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c0c1544f0 msgr2=0x7f3c0c073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:36.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.774+0000 7f3c10ea7640 1 --2- 172.21.15.55:0/2179175363 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c0c1544f0 0x7f3c0c073540 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc009870 tx=0x7f3bfc01ed50 comp rx=0 tx=0).stop 2024-03-19T02:02:36.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.774+0000 7f3c10ea7640 1 -- 172.21.15.55:0/2179175363 shutdown_connections 2024-03-19T02:02:36.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.774+0000 7f3c10ea7640 1 --2- 172.21.15.55:0/2179175363 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3c0c15acb0 0x7f3c0c0cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:36.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.774+0000 7f3c10ea7640 1 --2- 172.21.15.55:0/2179175363 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c0c1544f0 0x7f3c0c073540 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:36.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.774+0000 7f3c10ea7640 1 -- 172.21.15.55:0/2179175363 >> 172.21.15.55:0/2179175363 conn(0x7f3c0c06e670 msgr2=0x7f3c0c06ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:36.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.774+0000 7f3c10ea7640 1 -- 172.21.15.55:0/2179175363 shutdown_connections 2024-03-19T02:02:36.776 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.774+0000 7f3c10ea7640 1 -- 172.21.15.55:0/2179175363 wait complete. 2024-03-19T02:02:36.776 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.778+0000 7f3c10ea7640 1 Processor -- start 2024-03-19T02:02:36.776 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.778+0000 7f3c10ea7640 1 -- start start 2024-03-19T02:02:36.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.778+0000 7f3c10ea7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c0c06b2a0 0x7f3c0c0caf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:36.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.778+0000 7f3c10ea7640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3c0c1544f0 0x7f3c0c1f0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:36.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.778+0000 7f3c10ea7640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3c0c15acb0 0x7f3c0c153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:36.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.778+0000 7f3c10ea7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0c0ca120 con 0x7f3c0c06b2a0 2024-03-19T02:02:36.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.778+0000 7f3c10ea7640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0c1548d0 con 0x7f3c0c15acb0 2024-03-19T02:02:36.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.778+0000 7f3c10ea7640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3c0c0ca2a0 con 0x7f3c0c1544f0 2024-03-19T02:02:36.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.778+0000 7f3c0a59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c0c06b2a0 0x7f3c0c0caf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:36.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.778+0000 7f3c0ad9d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3c0c15acb0 0x7f3c0c153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:36.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.778+0000 7f3c0a59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c0c06b2a0 0x7f3c0c0caf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57908/0 (socket says 172.21.15.55:57908) 2024-03-19T02:02:36.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.778+0000 7f3c0a59c640 1 -- 172.21.15.55:0/1600866512 learned_addr learned my addr 172.21.15.55:0/1600866512 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:36.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.778+0000 7f3c09d9b640 1 --2- 172.21.15.55:0/1600866512 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3c0c1544f0 0x7f3c0c1f0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:36.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.782+0000 7f3c0a59c640 1 -- 172.21.15.55:0/1600866512 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3c0c1544f0 msgr2=0x7f3c0c1f0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:36.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.782+0000 7f3c0a59c640 1 --2- 172.21.15.55:0/1600866512 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3c0c1544f0 0x7f3c0c1f0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:36.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.782+0000 7f3c0a59c640 1 -- 172.21.15.55:0/1600866512 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3c0c15acb0 msgr2=0x7f3c0c153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:36.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.782+0000 7f3c0a59c640 1 --2- 172.21.15.55:0/1600866512 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3c0c15acb0 0x7f3c0c153150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:36.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.782+0000 7f3c0a59c640 1 -- 172.21.15.55:0/1600866512 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c0c0c9a00 con 0x7f3c0c06b2a0 2024-03-19T02:02:36.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.786+0000 7f3c0a59c640 1 --2- 172.21.15.55:0/1600866512 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c0c06b2a0 0x7f3c0c0caf20 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc0b3040 tx=0x7f3bfc01baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:36.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.790+0000 7f3bfaffd640 1 -- 172.21.15.55:0/1600866512 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc0b5070 con 0x7f3c0c06b2a0 2024-03-19T02:02:36.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.790+0000 7f3bfaffd640 1 -- 172.21.15.55:0/1600866512 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bfc0b0de0 con 0x7f3c0c06b2a0 2024-03-19T02:02:36.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.790+0000 7f3bfaffd640 1 -- 172.21.15.55:0/1600866512 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc0bb670 con 0x7f3c0c06b2a0 2024-03-19T02:02:36.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.790+0000 7f3c10ea7640 1 -- 172.21.15.55:0/1600866512 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c0c1f06f0 con 0x7f3c0c06b2a0 2024-03-19T02:02:36.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.790+0000 7f3c10ea7640 1 -- 172.21.15.55:0/1600866512 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c0c15b7f0 con 0x7f3c0c06b2a0 2024-03-19T02:02:36.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.790+0000 7f3c10ea7640 1 -- 172.21.15.55:0/1600866512 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c0c1f0500 con 0x7f3c0c06b2a0 2024-03-19T02:02:36.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.830+0000 7f3bfaffd640 1 -- 172.21.15.55:0/1600866512 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f3bfc0c2060 con 0x7f3c0c06b2a0 2024-03-19T02:02:36.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.830+0000 7f3bfaffd640 1 --2- 172.21.15.55:0/1600866512 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3bec067690 0x7f3bec069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:36.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.834+0000 7f3c09d9b640 1 --2- 172.21.15.55:0/1600866512 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3bec067690 0x7f3bec069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:36.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:36.834+0000 7f3c09d9b640 1 --2- 172.21.15.55:0/1600866512 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3bec067690 0x7f3bec069b50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c153a90 tx=0x7f3c00009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:37.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:37.170+0000 7f3bfaffd640 1 -- 172.21.15.55:0/1600866512 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f3bfc0f15a0 con 0x7f3c0c06b2a0 2024-03-19T02:02:37.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:37.170+0000 7f3bfaffd640 1 -- 172.21.15.55:0/1600866512 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f3bfc0188b0 con 0x7f3c0c06b2a0 2024-03-19T02:02:37.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:37.378+0000 7f3c10ea7640 1 -- 172.21.15.55:0/1600866512 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c0c068a40 con 0x7f3c0c06b2a0 2024-03-19T02:02:37.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:37.406+0000 7f3bfaffd640 1 -- 172.21.15.55:0/1600866512 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f3bfc0f3500 con 0x7f3c0c06b2a0 2024-03-19T02:02:37.404 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:37.404 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:37.404 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:37.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:37.406+0000 7f3c10ea7640 1 -- 172.21.15.55:0/1600866512 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3bec067690 msgr2=0x7f3bec069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:37.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:37.406+0000 7f3c10ea7640 1 --2- 172.21.15.55:0/1600866512 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3bec067690 0x7f3bec069b50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c153a90 tx=0x7f3c00009040 comp rx=0 tx=0).stop 2024-03-19T02:02:37.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:37.406+0000 7f3c10ea7640 1 -- 172.21.15.55:0/1600866512 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c0c06b2a0 msgr2=0x7f3c0c0caf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:37.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:37.406+0000 7f3c10ea7640 1 --2- 172.21.15.55:0/1600866512 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c0c06b2a0 0x7f3c0c0caf20 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc0b3040 tx=0x7f3bfc01baf0 comp rx=0 tx=0).stop 2024-03-19T02:02:37.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:37.406+0000 7f3c10ea7640 1 -- 172.21.15.55:0/1600866512 shutdown_connections 2024-03-19T02:02:37.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:37.406+0000 7f3c10ea7640 1 --2- 172.21.15.55:0/1600866512 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3bec067690 0x7f3bec069b50 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:37.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:37.406+0000 7f3c10ea7640 1 --2- 172.21.15.55:0/1600866512 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3c0c15acb0 0x7f3c0c153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:37.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:37.406+0000 7f3c10ea7640 1 --2- 172.21.15.55:0/1600866512 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3c0c1544f0 0x7f3c0c1f0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:37.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:37.406+0000 7f3c10ea7640 1 --2- 172.21.15.55:0/1600866512 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c0c06b2a0 0x7f3c0c0caf20 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:37.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:37.406+0000 7f3c10ea7640 1 -- 172.21.15.55:0/1600866512 >> 172.21.15.55:0/1600866512 conn(0x7f3c0c06e670 msgr2=0x7f3c0c0cce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:37.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:37.406+0000 7f3c10ea7640 1 -- 172.21.15.55:0/1600866512 shutdown_connections 2024-03-19T02:02:37.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:37.406+0000 7f3c10ea7640 1 -- 172.21.15.55:0/1600866512 wait complete. 2024-03-19T02:02:38.026 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:38.027 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:39.028 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:39.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.158+0000 7f2045185640 1 Processor -- start 2024-03-19T02:02:39.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.158+0000 7f2045185640 1 -- start start 2024-03-19T02:02:39.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.158+0000 7f2045185640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f204006c9f0 0x7f20400c92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:39.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.158+0000 7f2045185640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2040070e80 con 0x7f204006c610 2024-03-19T02:02:39.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.158+0000 7f2045185640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f204006fb00 con 0x7f20400c9800 2024-03-19T02:02:39.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.158+0000 7f2045185640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f204006fde0 con 0x7f204006c9f0 2024-03-19T02:02:39.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.162+0000 7f203e59c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f204006c9f0 0x7f20400c92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:39.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.162+0000 7f203f59e640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f20400c9800 0x7f20401427f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:51746/0 (socket says 172.21.15.55:51746) 2024-03-19T02:02:39.159 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.162+0000 7f203f59e640 1 -- 172.21.15.55:0/1768131103 learned_addr learned my addr 172.21.15.55:0/1768131103 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:02:39.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.162+0000 7f203dd9b640 1 -- 172.21.15.55:0/1768131103 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2259594060 0 0) 0x7f204006fb00 con 0x7f20400c9800 2024-03-19T02:02:39.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.166+0000 7f203dd9b640 1 -- 172.21.15.55:0/1768131103 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2028003530 con 0x7f20400c9800 2024-03-19T02:02:39.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.166+0000 7f203e59c640 1 -- 172.21.15.55:0/1768131103 >> v1:172.21.15.84:6789/0 conn(0x7f20400c9800 legacy=0x7f20401427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:39.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.166+0000 7f203e59c640 1 -- 172.21.15.55:0/1768131103 >> v1:172.21.15.55:6789/0 conn(0x7f204006c610 legacy=0x7f204015ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:39.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.166+0000 7f203e59c640 1 -- 172.21.15.55:0/1768131103 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20400700c0 con 0x7f204006c9f0 2024-03-19T02:02:39.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.170+0000 7f203e59c640 1 --2- 172.21.15.55:0/1768131103 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f204006c9f0 0x7f20400c92c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f20340097c0 tx=0x7f203401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fe4c770ea3234af server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:39.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.174+0000 7f203dd9b640 1 -- 172.21.15.55:0/1768131103 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20340b5070 con 0x7f204006c9f0 2024-03-19T02:02:39.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.174+0000 7f203dd9b640 1 -- 172.21.15.55:0/1768131103 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20340a8990 con 0x7f204006c9f0 2024-03-19T02:02:39.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.174+0000 7f203dd9b640 1 -- 172.21.15.55:0/1768131103 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20340a8d60 con 0x7f204006c9f0 2024-03-19T02:02:39.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.178+0000 7f2045185640 1 -- 172.21.15.55:0/1768131103 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f204006c9f0 msgr2=0x7f20400c92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:39.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.178+0000 7f2045185640 1 --2- 172.21.15.55:0/1768131103 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f204006c9f0 0x7f20400c92c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f20340097c0 tx=0x7f203401ec90 comp rx=0 tx=0).stop 2024-03-19T02:02:39.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.178+0000 7f2045185640 1 -- 172.21.15.55:0/1768131103 shutdown_connections 2024-03-19T02:02:39.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.178+0000 7f2045185640 1 --2- 172.21.15.55:0/1768131103 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f204006c9f0 0x7f20400c92c0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:39.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.178+0000 7f2045185640 1 -- 172.21.15.55:0/1768131103 >> 172.21.15.55:0/1768131103 conn(0x7f2040067600 msgr2=0x7f2040071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:39.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.178+0000 7f2045185640 1 -- 172.21.15.55:0/1768131103 shutdown_connections 2024-03-19T02:02:39.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.178+0000 7f2045185640 1 -- 172.21.15.55:0/1768131103 wait complete. 2024-03-19T02:02:39.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.178+0000 7f2045185640 1 Processor -- start 2024-03-19T02:02:39.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.178+0000 7f2045185640 1 -- start start 2024-03-19T02:02:39.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.178+0000 7f2045185640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f204006c610 0x7f20401f38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:39.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.178+0000 7f2045185640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f204006c9f0 0x7f20400c4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:39.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.178+0000 7f2045185640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20400c9800 0x7f20401f0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:39.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.178+0000 7f2045185640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2040070960 con 0x7f20400c9800 2024-03-19T02:02:39.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.178+0000 7f2045185640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2040070680 con 0x7f204006c610 2024-03-19T02:02:39.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.178+0000 7f2045185640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2040142f00 con 0x7f204006c9f0 2024-03-19T02:02:39.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.178+0000 7f203ed9d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f204006c610 0x7f20401f38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:39.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.178+0000 7f203f59e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20400c9800 0x7f20401f0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:39.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.178+0000 7f203ed9d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f204006c610 0x7f20401f38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:55512/0 (socket says 172.21.15.55:55512) 2024-03-19T02:02:39.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.178+0000 7f203ed9d640 1 -- 172.21.15.55:0/2548848280 learned_addr learned my addr 172.21.15.55:0/2548848280 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:39.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.182+0000 7f203e59c640 1 --2- 172.21.15.55:0/2548848280 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f204006c9f0 0x7f20400c4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:39.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.182+0000 7f203f59e640 1 -- 172.21.15.55:0/2548848280 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f204006c9f0 msgr2=0x7f20400c4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:39.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.182+0000 7f203f59e640 1 --2- 172.21.15.55:0/2548848280 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f204006c9f0 0x7f20400c4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:39.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.182+0000 7f203f59e640 1 -- 172.21.15.55:0/2548848280 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f204006c610 msgr2=0x7f20401f38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:39.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.182+0000 7f203f59e640 1 --2- 172.21.15.55:0/2548848280 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f204006c610 0x7f20401f38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:39.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.182+0000 7f203f59e640 1 -- 172.21.15.55:0/2548848280 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20400ce920 con 0x7f20400c9800 2024-03-19T02:02:39.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.190+0000 7f203f59e640 1 --2- 172.21.15.55:0/2548848280 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20400c9800 0x7f20401f0100 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f203800ca80 tx=0x7f203800cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:39.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.194+0000 7f20277fe640 1 -- 172.21.15.55:0/2548848280 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2038017070 con 0x7f20400c9800 2024-03-19T02:02:39.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.194+0000 7f20277fe640 1 -- 172.21.15.55:0/2548848280 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f203800ac30 con 0x7f20400c9800 2024-03-19T02:02:39.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.194+0000 7f2045185640 1 -- 172.21.15.55:0/2548848280 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20400cd2e0 con 0x7f20400c9800 2024-03-19T02:02:39.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.194+0000 7f20277fe640 1 -- 172.21.15.55:0/2548848280 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2038012600 con 0x7f20400c9800 2024-03-19T02:02:39.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.194+0000 7f2045185640 1 -- 172.21.15.55:0/2548848280 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20401598d0 con 0x7f20400c9800 2024-03-19T02:02:39.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.194+0000 7f2045185640 1 -- 172.21.15.55:0/2548848280 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2040068a40 con 0x7f20400c9800 2024-03-19T02:02:39.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.486+0000 7f20277fe640 1 -- 172.21.15.55:0/2548848280 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f203801a070 con 0x7f20400c9800 2024-03-19T02:02:39.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.486+0000 7f20277fe640 1 --2- 172.21.15.55:0/2548848280 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f202c0676b0 0x7f202c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:39.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.486+0000 7f203ed9d640 1 --2- 172.21.15.55:0/2548848280 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f202c0676b0 0x7f202c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:39.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.490+0000 7f203ed9d640 1 --2- 172.21.15.55:0/2548848280 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f202c0676b0 0x7f202c069b70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2030007fd0 tx=0x7f2030009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:39.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.510+0000 7f20277fe640 1 -- 172.21.15.55:0/2548848280 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f2038054170 con 0x7f20400c9800 2024-03-19T02:02:39.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.510+0000 7f20277fe640 1 -- 172.21.15.55:0/2548848280 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f203800ae20 con 0x7f20400c9800 2024-03-19T02:02:39.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.730+0000 7f2045185640 1 -- 172.21.15.55:0/2548848280 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2040159bb0 con 0x7f20400c9800 2024-03-19T02:02:39.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.750+0000 7f20277fe640 1 -- 172.21.15.55:0/2548848280 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f20380560d0 con 0x7f20400c9800 2024-03-19T02:02:39.751 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:39.752 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:39.752 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:39.754 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.754+0000 7f2045185640 1 -- 172.21.15.55:0/2548848280 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f202c0676b0 msgr2=0x7f202c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:39.754 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.754+0000 7f2045185640 1 --2- 172.21.15.55:0/2548848280 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f202c0676b0 0x7f202c069b70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2030007fd0 tx=0x7f2030009040 comp rx=0 tx=0).stop 2024-03-19T02:02:39.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.754+0000 7f2045185640 1 -- 172.21.15.55:0/2548848280 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20400c9800 msgr2=0x7f20401f0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:39.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.754+0000 7f2045185640 1 --2- 172.21.15.55:0/2548848280 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20400c9800 0x7f20401f0100 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f203800ca80 tx=0x7f203800cf40 comp rx=0 tx=0).stop 2024-03-19T02:02:39.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.754+0000 7f2045185640 1 -- 172.21.15.55:0/2548848280 shutdown_connections 2024-03-19T02:02:39.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.754+0000 7f2045185640 1 --2- 172.21.15.55:0/2548848280 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f202c0676b0 0x7f202c069b70 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:39.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.754+0000 7f2045185640 1 --2- 172.21.15.55:0/2548848280 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20400c9800 0x7f20401f0100 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:39.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.754+0000 7f2045185640 1 --2- 172.21.15.55:0/2548848280 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f204006c9f0 0x7f20400c4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:39.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.754+0000 7f2045185640 1 --2- 172.21.15.55:0/2548848280 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f204006c610 0x7f20401f38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:39.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.754+0000 7f2045185640 1 -- 172.21.15.55:0/2548848280 >> 172.21.15.55:0/2548848280 conn(0x7f2040067600 msgr2=0x7f2040074290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:39.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.754+0000 7f2045185640 1 -- 172.21.15.55:0/2548848280 shutdown_connections 2024-03-19T02:02:39.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:39.754+0000 7f2045185640 1 -- 172.21.15.55:0/2548848280 wait complete. 2024-03-19T02:02:40.376 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:40.376 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:40.522 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:02:40.525 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:02:41.377 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:41.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.490+0000 7f70d7344640 1 Processor -- start 2024-03-19T02:02:41.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.490+0000 7f70d7344640 1 -- start start 2024-03-19T02:02:41.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.490+0000 7f70d7344640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f70d006b910 0x7f70d006bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:41.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.490+0000 7f70d7344640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70d00c7720 con 0x7f70d00c60d0 2024-03-19T02:02:41.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.490+0000 7f70d7344640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70d00c7a00 con 0x7f70d00c2ce0 2024-03-19T02:02:41.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.490+0000 7f70d7344640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f70d0158460 con 0x7f70d006b910 2024-03-19T02:02:41.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.494+0000 7f70d58e1640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f70d00c60d0 0x7f70d0162960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:52554/0 (socket says 172.21.15.55:52554) 2024-03-19T02:02:41.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.494+0000 7f70d58e1640 1 -- 172.21.15.55:0/3672763482 learned_addr learned my addr 172.21.15.55:0/3672763482 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:02:41.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.494+0000 7f70d48df640 1 --2- 172.21.15.55:0/3672763482 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f70d006b910 0x7f70d006bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:41.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.494+0000 7f70c7fff640 1 -- 172.21.15.55:0/3672763482 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3258037305 0 0) 0x7f70d00c7720 con 0x7f70d00c60d0 2024-03-19T02:02:41.494 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.494+0000 7f70c7fff640 1 -- 172.21.15.55:0/3672763482 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70bc003530 con 0x7f70d00c60d0 2024-03-19T02:02:41.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.498+0000 7f70d48df640 1 -- 172.21.15.55:0/3672763482 >> v1:172.21.15.84:6789/0 conn(0x7f70d00c2ce0 legacy=0x7f70d01588f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:41.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.498+0000 7f70d48df640 1 -- 172.21.15.55:0/3672763482 >> v1:172.21.15.55:6789/0 conn(0x7f70d00c60d0 legacy=0x7f70d0162960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:41.495 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.498+0000 7f70d48df640 1 -- 172.21.15.55:0/3672763482 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d00c7ce0 con 0x7f70d006b910 2024-03-19T02:02:41.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.502+0000 7f70d48df640 1 --2- 172.21.15.55:0/3672763482 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f70d006b910 0x7f70d006bd50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f70c00097f0 tx=0x7f70c001ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cf327cc2ba83a7f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:41.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.506+0000 7f70c7fff640 1 -- 172.21.15.55:0/3672763482 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c00b5070 con 0x7f70d006b910 2024-03-19T02:02:41.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.506+0000 7f70c7fff640 1 -- 172.21.15.55:0/3672763482 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70c00a89d0 con 0x7f70d006b910 2024-03-19T02:02:41.507 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.506+0000 7f70c7fff640 1 -- 172.21.15.55:0/3672763482 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c00a8da0 con 0x7f70d006b910 2024-03-19T02:02:41.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.510+0000 7f70d7344640 1 -- 172.21.15.55:0/3672763482 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f70d006b910 msgr2=0x7f70d006bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:41.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.510+0000 7f70d7344640 1 --2- 172.21.15.55:0/3672763482 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f70d006b910 0x7f70d006bd50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f70c00097f0 tx=0x7f70c001ecd0 comp rx=0 tx=0).stop 2024-03-19T02:02:41.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.510+0000 7f70d7344640 1 -- 172.21.15.55:0/3672763482 shutdown_connections 2024-03-19T02:02:41.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.510+0000 7f70d7344640 1 --2- 172.21.15.55:0/3672763482 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f70d006b910 0x7f70d006bd50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:41.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.510+0000 7f70d7344640 1 -- 172.21.15.55:0/3672763482 >> 172.21.15.55:0/3672763482 conn(0x7f70d00cc9c0 msgr2=0x7f70d00ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:41.508 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.510+0000 7f70d7344640 1 -- 172.21.15.55:0/3672763482 shutdown_connections 2024-03-19T02:02:41.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.510+0000 7f70d7344640 1 -- 172.21.15.55:0/3672763482 wait complete. 2024-03-19T02:02:41.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.510+0000 7f70d7344640 1 Processor -- start 2024-03-19T02:02:41.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.510+0000 7f70d7344640 1 -- start start 2024-03-19T02:02:41.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.510+0000 7f70d7344640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f70d006b910 0x7f70d01546d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:41.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.510+0000 7f70d7344640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70d00c2ce0 0x7f70d0153050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:41.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.510+0000 7f70d7344640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70d00c60d0 0x7f70d0153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:41.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.510+0000 7f70d7344640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f70d0070310 con 0x7f70d00c2ce0 2024-03-19T02:02:41.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.510+0000 7f70d7344640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f70d00c73c0 con 0x7f70d00c60d0 2024-03-19T02:02:41.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.510+0000 7f70d7344640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f70d0070490 con 0x7f70d006b910 2024-03-19T02:02:41.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.510+0000 7f70d48df640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70d00c2ce0 0x7f70d0153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:41.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.514+0000 7f70d58e1640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70d00c60d0 0x7f70d0153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:41.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.514+0000 7f70d50e0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f70d006b910 0x7f70d01546d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:41.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.514+0000 7f70d48df640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70d00c2ce0 0x7f70d0153050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:46804/0 (socket says 172.21.15.55:46804) 2024-03-19T02:02:41.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.514+0000 7f70d48df640 1 -- 172.21.15.55:0/3975214663 learned_addr learned my addr 172.21.15.55:0/3975214663 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:41.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.514+0000 7f70d48df640 1 -- 172.21.15.55:0/3975214663 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f70d006b910 msgr2=0x7f70d01546d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:41.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.518+0000 7f70d48df640 1 --2- 172.21.15.55:0/3975214663 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f70d006b910 0x7f70d01546d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:41.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.518+0000 7f70d48df640 1 -- 172.21.15.55:0/3975214663 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70d00c60d0 msgr2=0x7f70d0153590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:41.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.518+0000 7f70d48df640 1 --2- 172.21.15.55:0/3975214663 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70d00c60d0 0x7f70d0153590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:41.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.518+0000 7f70d48df640 1 -- 172.21.15.55:0/3975214663 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d0155590 con 0x7f70d00c2ce0 2024-03-19T02:02:41.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.522+0000 7f70d48df640 1 --2- 172.21.15.55:0/3975214663 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70d00c2ce0 0x7f70d0153050 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f70c001b8e0 tx=0x7f70c001ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:41.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.526+0000 7f70c5ffb640 1 -- 172.21.15.55:0/3975214663 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c00b5040 con 0x7f70d00c2ce0 2024-03-19T02:02:41.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.526+0000 7f70c5ffb640 1 -- 172.21.15.55:0/3975214663 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70c00c0070 con 0x7f70d00c2ce0 2024-03-19T02:02:41.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.526+0000 7f70c5ffb640 1 -- 172.21.15.55:0/3975214663 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c00b09c0 con 0x7f70d00c2ce0 2024-03-19T02:02:41.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.526+0000 7f70d7344640 1 -- 172.21.15.55:0/3975214663 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70d014d260 con 0x7f70d00c2ce0 2024-03-19T02:02:41.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.526+0000 7f70d7344640 1 -- 172.21.15.55:0/3975214663 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70d00c51b0 con 0x7f70d00c2ce0 2024-03-19T02:02:41.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.526+0000 7f70d7344640 1 -- 172.21.15.55:0/3975214663 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70d0068a40 con 0x7f70d00c2ce0 2024-03-19T02:02:41.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.830+0000 7f70c5ffb640 1 -- 172.21.15.55:0/3975214663 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f70c00b8070 con 0x7f70d00c2ce0 2024-03-19T02:02:41.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.830+0000 7f70c5ffb640 1 --2- 172.21.15.55:0/3975214663 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f70b4067660 0x7f70b4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:41.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.830+0000 7f70d50e0640 1 --2- 172.21.15.55:0/3975214663 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f70b4067660 0x7f70b4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:41.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.830+0000 7f70d50e0640 1 --2- 172.21.15.55:0/3975214663 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f70b4067660 0x7f70b4069b20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f70cc001810 tx=0x7f70cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:41.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.866+0000 7f70c5ffb640 1 -- 172.21.15.55:0/3975214663 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f70c00f1df0 con 0x7f70d00c2ce0 2024-03-19T02:02:41.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:41.866+0000 7f70c5ffb640 1 -- 172.21.15.55:0/3975214663 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f70c0018830 con 0x7f70d00c2ce0 2024-03-19T02:02:42.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:42.086+0000 7f70d7344640 1 -- 172.21.15.55:0/3975214663 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70d0155900 con 0x7f70d00c2ce0 2024-03-19T02:02:42.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:42.110+0000 7f70c5ffb640 1 -- 172.21.15.55:0/3975214663 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f70c00a89a0 con 0x7f70d00c2ce0 2024-03-19T02:02:42.109 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:42.109 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:42.109 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:42.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:42.114+0000 7f70d7344640 1 -- 172.21.15.55:0/3975214663 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f70b4067660 msgr2=0x7f70b4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:42.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:42.114+0000 7f70d7344640 1 --2- 172.21.15.55:0/3975214663 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f70b4067660 0x7f70b4069b20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f70cc001810 tx=0x7f70cc009040 comp rx=0 tx=0).stop 2024-03-19T02:02:42.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:42.114+0000 7f70d7344640 1 -- 172.21.15.55:0/3975214663 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70d00c2ce0 msgr2=0x7f70d0153050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:42.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:42.114+0000 7f70d7344640 1 --2- 172.21.15.55:0/3975214663 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70d00c2ce0 0x7f70d0153050 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f70c001b8e0 tx=0x7f70c001ba70 comp rx=0 tx=0).stop 2024-03-19T02:02:42.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:42.114+0000 7f70d7344640 1 -- 172.21.15.55:0/3975214663 shutdown_connections 2024-03-19T02:02:42.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:42.114+0000 7f70d7344640 1 --2- 172.21.15.55:0/3975214663 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f70b4067660 0x7f70b4069b20 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:42.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:42.114+0000 7f70d7344640 1 --2- 172.21.15.55:0/3975214663 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70d00c60d0 0x7f70d0153590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:42.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:42.114+0000 7f70d7344640 1 --2- 172.21.15.55:0/3975214663 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70d00c2ce0 0x7f70d0153050 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:42.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:42.114+0000 7f70d7344640 1 --2- 172.21.15.55:0/3975214663 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f70d006b910 0x7f70d01546d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:42.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:42.114+0000 7f70d7344640 1 -- 172.21.15.55:0/3975214663 >> 172.21.15.55:0/3975214663 conn(0x7f70d00cc9c0 msgr2=0x7f70d0155e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:42.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:42.114+0000 7f70d7344640 1 -- 172.21.15.55:0/3975214663 shutdown_connections 2024-03-19T02:02:42.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:42.114+0000 7f70d7344640 1 -- 172.21.15.55:0/3975214663 wait complete. 2024-03-19T02:02:42.737 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:42.737 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:43.738 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:43.850 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.850+0000 7f5fb1cb2640 1 Processor -- start 2024-03-19T02:02:43.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.850+0000 7f5fb1cb2640 1 -- start start 2024-03-19T02:02:43.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.854+0000 7f5fb1cb2640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f5fac152430 0x7f5fac158070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:43.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.854+0000 7f5fb1cb2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5fac0c89f0 0x7f5fac1585b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:43.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.854+0000 7f5fb1cb2640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fac0724c0 con 0x7f5fac0c52b0 2024-03-19T02:02:43.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.854+0000 7f5fb1cb2640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f5fac13d290 con 0x7f5fac152430 2024-03-19T02:02:43.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.854+0000 7f5fb1cb2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f5fac13d570 con 0x7f5fac0c89f0 2024-03-19T02:02:43.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.854+0000 7f5fabfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5fac0c89f0 0x7f5fac1585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:43.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.854+0000 7f5faaffd640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f5fac152430 0x7f5fac158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:43.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.854+0000 7f5fabfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5fac0c89f0 0x7f5fac1585b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:58654/0 (socket says 172.21.15.55:58654) 2024-03-19T02:02:43.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.854+0000 7f5fabfff640 1 -- 172.21.15.55:0/3873718000 learned_addr learned my addr 172.21.15.55:0/3873718000 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:43.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.858+0000 7f5fabfff640 1 -- 172.21.15.55:0/3873718000 >> v2:172.21.15.84:3300/0 conn(0x7f5fac152430 msgr2=0x7f5fac158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:43.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.858+0000 7f5fabfff640 1 --2- 172.21.15.55:0/3873718000 >> v2:172.21.15.84:3300/0 conn(0x7f5fac152430 0x7f5fac158070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:43.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.858+0000 7f5fabfff640 1 -- 172.21.15.55:0/3873718000 >> v1:172.21.15.84:6789/0 conn(0x7f5fac0c52b0 legacy=0x7f5fac151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:43.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.858+0000 7f5fabfff640 1 -- 172.21.15.55:0/3873718000 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fac13d850 con 0x7f5fac0c89f0 2024-03-19T02:02:43.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.862+0000 7f5fabfff640 1 --2- 172.21.15.55:0/3873718000 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5fac0c89f0 0x7f5fac1585b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4009e50 tx=0x7f5fa401f1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5a6fa5cf465c3ff server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:43.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.866+0000 7f5faa7fc640 1 -- 172.21.15.55:0/3873718000 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa40b4070 con 0x7f5fac0c89f0 2024-03-19T02:02:43.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.866+0000 7f5faa7fc640 1 -- 172.21.15.55:0/3873718000 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa40af400 con 0x7f5fac0c89f0 2024-03-19T02:02:43.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.866+0000 7f5faa7fc640 1 -- 172.21.15.55:0/3873718000 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa40af7d0 con 0x7f5fac0c89f0 2024-03-19T02:02:43.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.870+0000 7f5fb1cb2640 1 -- 172.21.15.55:0/3873718000 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5fac0c89f0 msgr2=0x7f5fac1585b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:43.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.870+0000 7f5fb1cb2640 1 --2- 172.21.15.55:0/3873718000 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5fac0c89f0 0x7f5fac1585b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4009e50 tx=0x7f5fa401f1e0 comp rx=0 tx=0).stop 2024-03-19T02:02:43.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.870+0000 7f5fb1cb2640 1 -- 172.21.15.55:0/3873718000 shutdown_connections 2024-03-19T02:02:43.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.870+0000 7f5fb1cb2640 1 --2- 172.21.15.55:0/3873718000 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5fac0c89f0 0x7f5fac1585b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:43.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.870+0000 7f5fb1cb2640 1 --2- 172.21.15.55:0/3873718000 >> v2:172.21.15.84:3300/0 conn(0x7f5fac152430 0x7f5fac158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:43.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.870+0000 7f5fb1cb2640 1 -- 172.21.15.55:0/3873718000 >> 172.21.15.55:0/3873718000 conn(0x7f5fac06f3b0 msgr2=0x7f5fac06f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:43.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.870+0000 7f5fb1cb2640 1 -- 172.21.15.55:0/3873718000 shutdown_connections 2024-03-19T02:02:43.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.870+0000 7f5fb1cb2640 1 -- 172.21.15.55:0/3873718000 wait complete. 2024-03-19T02:02:43.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.870+0000 7f5fb1cb2640 1 Processor -- start 2024-03-19T02:02:43.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.870+0000 7f5fb1cb2640 1 -- start start 2024-03-19T02:02:43.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.870+0000 7f5fb1cb2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5fac0c52b0 0x7f5fac0c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:43.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.870+0000 7f5fb1cb2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fac0c89f0 0x7f5fac1f4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:43.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.870+0000 7f5fb1cb2640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5fac152430 0x7f5fac136540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:43.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.870+0000 7f5fb1cb2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5fac0c5690 con 0x7f5fac0c89f0 2024-03-19T02:02:43.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.870+0000 7f5fb1cb2640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f5fac13de10 con 0x7f5fac152430 2024-03-19T02:02:43.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.870+0000 7f5fb1cb2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f5fac0c5810 con 0x7f5fac0c52b0 2024-03-19T02:02:43.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.874+0000 7f5fabfff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5fac152430 0x7f5fac136540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:43.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.874+0000 7f5faaffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fac0c89f0 0x7f5fac1f4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:43.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.874+0000 7f5fab7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5fac0c52b0 0x7f5fac0c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:43.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.874+0000 7f5fabfff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5fac152430 0x7f5fac136540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:55542/0 (socket says 172.21.15.55:55542) 2024-03-19T02:02:43.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.874+0000 7f5fabfff640 1 -- 172.21.15.55:0/319158788 learned_addr learned my addr 172.21.15.55:0/319158788 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:43.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.878+0000 7f5fab7fe640 1 -- 172.21.15.55:0/319158788 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5fac152430 msgr2=0x7f5fac136540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:43.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.878+0000 7f5fab7fe640 1 --2- 172.21.15.55:0/319158788 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5fac152430 0x7f5fac136540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:43.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.878+0000 7f5fab7fe640 1 -- 172.21.15.55:0/319158788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fac0c89f0 msgr2=0x7f5fac1f4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:43.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.878+0000 7f5fab7fe640 1 --2- 172.21.15.55:0/319158788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fac0c89f0 0x7f5fac1f4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:43.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.878+0000 7f5fab7fe640 1 -- 172.21.15.55:0/319158788 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fac0c6f40 con 0x7f5fac0c52b0 2024-03-19T02:02:43.881 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.882+0000 7f5fab7fe640 1 --2- 172.21.15.55:0/319158788 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5fac0c52b0 0x7f5fac0c63a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c00c390 tx=0x7f5f9c00c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:43.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.890+0000 7f5f93fff640 1 -- 172.21.15.55:0/319158788 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c017070 con 0x7f5fac0c52b0 2024-03-19T02:02:43.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.890+0000 7f5f93fff640 1 -- 172.21.15.55:0/319158788 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f9c003600 con 0x7f5fac0c52b0 2024-03-19T02:02:43.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.890+0000 7f5f93fff640 1 -- 172.21.15.55:0/319158788 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c0039f0 con 0x7f5fac0c52b0 2024-03-19T02:02:43.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.890+0000 7f5fb1cb2640 1 -- 172.21.15.55:0/319158788 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fac0c77a0 con 0x7f5fac0c52b0 2024-03-19T02:02:43.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.890+0000 7f5fb1cb2640 1 -- 172.21.15.55:0/319158788 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fac0c71e0 con 0x7f5fac0c52b0 2024-03-19T02:02:43.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:43.890+0000 7f5fb1cb2640 1 -- 172.21.15.55:0/319158788 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fac068330 con 0x7f5fac0c52b0 2024-03-19T02:02:44.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:44.174+0000 7f5f93fff640 1 -- 172.21.15.55:0/319158788 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f5f9c003b90 con 0x7f5fac0c52b0 2024-03-19T02:02:44.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:44.174+0000 7f5f93fff640 1 --2- 172.21.15.55:0/319158788 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5f98067660 0x7f5f98069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:44.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:44.174+0000 7f5faaffd640 1 --2- 172.21.15.55:0/319158788 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5f98067660 0x7f5f98069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:44.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:44.178+0000 7f5faaffd640 1 --2- 172.21.15.55:0/319158788 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5f98067660 0x7f5f98069b20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5fac0cc030 tx=0x7f5fa0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:44.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:44.178+0000 7f5f93fff640 1 -- 172.21.15.55:0/319158788 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f5f9c053310 con 0x7f5fac0c52b0 2024-03-19T02:02:44.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:44.198+0000 7f5f93fff640 1 -- 172.21.15.55:0/319158788 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f5f9c055270 con 0x7f5fac0c52b0 2024-03-19T02:02:44.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:44.402+0000 7f5fb1cb2640 1 -- 172.21.15.55:0/319158788 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5fac0c4d30 con 0x7f5fac0c52b0 2024-03-19T02:02:44.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:44.426+0000 7f5f93fff640 1 -- 172.21.15.55:0/319158788 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f5f9c059310 con 0x7f5fac0c52b0 2024-03-19T02:02:44.425 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:44.425 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:44.426 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:44.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:44.430+0000 7f5fb1cb2640 1 -- 172.21.15.55:0/319158788 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5f98067660 msgr2=0x7f5f98069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:44.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:44.430+0000 7f5fb1cb2640 1 --2- 172.21.15.55:0/319158788 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5f98067660 0x7f5f98069b20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5fac0cc030 tx=0x7f5fa0009190 comp rx=0 tx=0).stop 2024-03-19T02:02:44.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:44.430+0000 7f5fb1cb2640 1 -- 172.21.15.55:0/319158788 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5fac0c52b0 msgr2=0x7f5fac0c63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:44.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:44.430+0000 7f5fb1cb2640 1 --2- 172.21.15.55:0/319158788 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5fac0c52b0 0x7f5fac0c63a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c00c390 tx=0x7f5f9c00c850 comp rx=0 tx=0).stop 2024-03-19T02:02:44.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:44.430+0000 7f5fb1cb2640 1 -- 172.21.15.55:0/319158788 shutdown_connections 2024-03-19T02:02:44.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:44.430+0000 7f5fb1cb2640 1 --2- 172.21.15.55:0/319158788 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5f98067660 0x7f5f98069b20 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:44.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:44.430+0000 7f5fb1cb2640 1 --2- 172.21.15.55:0/319158788 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5fac152430 0x7f5fac136540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:44.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:44.430+0000 7f5fb1cb2640 1 --2- 172.21.15.55:0/319158788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fac0c89f0 0x7f5fac1f4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:44.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:44.430+0000 7f5fb1cb2640 1 --2- 172.21.15.55:0/319158788 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5fac0c52b0 0x7f5fac0c63a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:44.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:44.430+0000 7f5fb1cb2640 1 -- 172.21.15.55:0/319158788 >> 172.21.15.55:0/319158788 conn(0x7f5fac06f3b0 msgr2=0x7f5fac0c9740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:44.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:44.430+0000 7f5fb1cb2640 1 -- 172.21.15.55:0/319158788 shutdown_connections 2024-03-19T02:02:44.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:44.430+0000 7f5fb1cb2640 1 -- 172.21.15.55:0/319158788 wait complete. 2024-03-19T02:02:45.049 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:45.049 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:46.050 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:46.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.170+0000 7f03b369d640 1 Processor -- start 2024-03-19T02:02:46.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.170+0000 7f03b369d640 1 -- start start 2024-03-19T02:02:46.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.174+0000 7f03b369d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f03ac152120 0x7f03ac0c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:46.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.174+0000 7f03b369d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f03ac070780 0x7f03ac1586e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:46.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.174+0000 7f03b369d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03ac06fff0 con 0x7f03ac0c9610 2024-03-19T02:02:46.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.174+0000 7f03b369d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f03ac152500 con 0x7f03ac070780 2024-03-19T02:02:46.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.174+0000 7f03b369d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f03ac070590 con 0x7f03ac152120 2024-03-19T02:02:46.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.174+0000 7f03b1c3a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f03ac070780 0x7f03ac1586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:46.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.174+0000 7f03b1439640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f03ac152120 0x7f03ac0c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:46.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.174+0000 7f03b0c38640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f03ac0c9610 0x7f03ac157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:52564/0 (socket says 172.21.15.55:52564) 2024-03-19T02:02:46.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.174+0000 7f03b0c38640 1 -- 172.21.15.55:0/2990385510 learned_addr learned my addr 172.21.15.55:0/2990385510 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:02:46.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.178+0000 7f03b1c3a640 1 -- 172.21.15.55:0/2990385510 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f03ac152120 msgr2=0x7f03ac0c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:46.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.178+0000 7f03b1c3a640 1 --2- 172.21.15.55:0/2990385510 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f03ac152120 0x7f03ac0c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:46.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.178+0000 7f03b1c3a640 1 -- 172.21.15.55:0/2990385510 >> v1:172.21.15.55:6789/0 conn(0x7f03ac0c9610 legacy=0x7f03ac157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:46.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.178+0000 7f03b1c3a640 1 -- 172.21.15.55:0/2990385510 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03ac0c39c0 con 0x7f03ac070780 2024-03-19T02:02:46.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.182+0000 7f03b1c3a640 1 --2- 172.21.15.55:0/2990385510 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03ac070780 0x7f03ac1586e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009d90 tx=0x7f03a401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e13e96e723538cf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:46.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.186+0000 7f03a3fff640 1 -- 172.21.15.55:0/2990385510 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a40b2070 con 0x7f03ac070780 2024-03-19T02:02:46.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.186+0000 7f03a3fff640 1 -- 172.21.15.55:0/2990385510 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03a40a5830 con 0x7f03ac070780 2024-03-19T02:02:46.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.186+0000 7f03a3fff640 1 -- 172.21.15.55:0/2990385510 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a40a5c00 con 0x7f03ac070780 2024-03-19T02:02:46.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.190+0000 7f03b369d640 1 -- 172.21.15.55:0/2990385510 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03ac070780 msgr2=0x7f03ac1586e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:46.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.190+0000 7f03b369d640 1 --2- 172.21.15.55:0/2990385510 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03ac070780 0x7f03ac1586e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009d90 tx=0x7f03a401c970 comp rx=0 tx=0).stop 2024-03-19T02:02:46.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.190+0000 7f03b369d640 1 -- 172.21.15.55:0/2990385510 shutdown_connections 2024-03-19T02:02:46.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.190+0000 7f03b369d640 1 --2- 172.21.15.55:0/2990385510 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03ac070780 0x7f03ac1586e0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:46.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.190+0000 7f03b369d640 1 --2- 172.21.15.55:0/2990385510 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f03ac152120 0x7f03ac0c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:46.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.190+0000 7f03b369d640 1 -- 172.21.15.55:0/2990385510 >> 172.21.15.55:0/2990385510 conn(0x7f03ac067600 msgr2=0x7f03ac071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:46.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.190+0000 7f03b369d640 1 -- 172.21.15.55:0/2990385510 shutdown_connections 2024-03-19T02:02:46.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.190+0000 7f03b369d640 1 -- 172.21.15.55:0/2990385510 wait complete. 2024-03-19T02:02:46.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.190+0000 7f03b369d640 1 Processor -- start 2024-03-19T02:02:46.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.190+0000 7f03b369d640 1 -- start start 2024-03-19T02:02:46.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.190+0000 7f03b369d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03ac070780 0x7f03ac156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:46.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.190+0000 7f03b369d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f03ac0c9610 0x7f03ac1f4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:46.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.190+0000 7f03b369d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f03ac152120 0x7f03ac0c4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:46.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.190+0000 7f03b369d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac0c86d0 con 0x7f03ac070780 2024-03-19T02:02:46.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.190+0000 7f03b369d640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac0c31c0 con 0x7f03ac152120 2024-03-19T02:02:46.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.190+0000 7f03b369d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f03ac165170 con 0x7f03ac0c9610 2024-03-19T02:02:46.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.190+0000 7f03b1439640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03ac070780 0x7f03ac156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:46.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.190+0000 7f03b0c38640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f03ac0c9610 0x7f03ac1f4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:46.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.190+0000 7f03b1439640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03ac070780 0x7f03ac156820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:46834/0 (socket says 172.21.15.55:46834) 2024-03-19T02:02:46.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.190+0000 7f03b1439640 1 -- 172.21.15.55:0/3658549554 learned_addr learned my addr 172.21.15.55:0/3658549554 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:46.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.190+0000 7f03b1c3a640 1 --2- 172.21.15.55:0/3658549554 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f03ac152120 0x7f03ac0c4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:46.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.194+0000 7f03b1439640 1 -- 172.21.15.55:0/3658549554 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f03ac0c9610 msgr2=0x7f03ac1f4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:46.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.194+0000 7f03b1439640 1 --2- 172.21.15.55:0/3658549554 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f03ac0c9610 0x7f03ac1f4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:46.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.194+0000 7f03b1439640 1 -- 172.21.15.55:0/3658549554 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f03ac152120 msgr2=0x7f03ac0c4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:46.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.194+0000 7f03b1439640 1 --2- 172.21.15.55:0/3658549554 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f03ac152120 0x7f03ac0c4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:46.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.194+0000 7f03b1439640 1 -- 172.21.15.55:0/3658549554 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03ac142b70 con 0x7f03ac070780 2024-03-19T02:02:46.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.198+0000 7f03b1439640 1 --2- 172.21.15.55:0/3658549554 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03ac070780 0x7f03ac156820 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f03a800e8a0 tx=0x7f03a800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:46.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.202+0000 7f03a1ffb640 1 -- 172.21.15.55:0/3658549554 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a8017070 con 0x7f03ac070780 2024-03-19T02:02:46.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.202+0000 7f03a1ffb640 1 -- 172.21.15.55:0/3658549554 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03a800c940 con 0x7f03ac070780 2024-03-19T02:02:46.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.202+0000 7f03b369d640 1 -- 172.21.15.55:0/3658549554 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03ac142e30 con 0x7f03ac070780 2024-03-19T02:02:46.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.202+0000 7f03a1ffb640 1 -- 172.21.15.55:0/3658549554 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a800ccf0 con 0x7f03ac070780 2024-03-19T02:02:46.203 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.202+0000 7f03b369d640 1 -- 172.21.15.55:0/3658549554 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03ac156020 con 0x7f03ac070780 2024-03-19T02:02:46.203 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.202+0000 7f03b369d640 1 -- 172.21.15.55:0/3658549554 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03ac068a40 con 0x7f03ac070780 2024-03-19T02:02:46.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.478+0000 7f03a1ffb640 1 -- 172.21.15.55:0/3658549554 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f03a801a070 con 0x7f03ac070780 2024-03-19T02:02:46.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.478+0000 7f03a1ffb640 1 --2- 172.21.15.55:0/3658549554 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0390067660 0x7f0390069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:46.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.478+0000 7f03b0c38640 1 --2- 172.21.15.55:0/3658549554 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0390067660 0x7f0390069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:46.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.482+0000 7f03b0c38640 1 --2- 172.21.15.55:0/3658549554 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0390067660 0x7f0390069b20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f03ac0b1910 tx=0x7f039c007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:46.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.502+0000 7f03a1ffb640 1 -- 172.21.15.55:0/3658549554 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f03a80545b0 con 0x7f03ac070780 2024-03-19T02:02:46.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.502+0000 7f03a1ffb640 1 -- 172.21.15.55:0/3658549554 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f03a8056510 con 0x7f03ac070780 2024-03-19T02:02:46.738 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.738+0000 7f03b369d640 1 -- 172.21.15.55:0/3658549554 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03ac0c84b0 con 0x7f03ac070780 2024-03-19T02:02:46.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.834+0000 7f03a1ffb640 1 -- 172.21.15.55:0/3658549554 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f03a805a5b0 con 0x7f03ac070780 2024-03-19T02:02:46.835 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:46.835 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:46.835 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:46.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.838+0000 7f03b369d640 1 -- 172.21.15.55:0/3658549554 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0390067660 msgr2=0x7f0390069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:46.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.838+0000 7f03b369d640 1 --2- 172.21.15.55:0/3658549554 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0390067660 0x7f0390069b20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f03ac0b1910 tx=0x7f039c007e90 comp rx=0 tx=0).stop 2024-03-19T02:02:46.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.838+0000 7f03b369d640 1 -- 172.21.15.55:0/3658549554 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03ac070780 msgr2=0x7f03ac156820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:46.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.838+0000 7f03b369d640 1 --2- 172.21.15.55:0/3658549554 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03ac070780 0x7f03ac156820 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f03a800e8a0 tx=0x7f03a800eca0 comp rx=0 tx=0).stop 2024-03-19T02:02:46.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.838+0000 7f03b369d640 1 -- 172.21.15.55:0/3658549554 shutdown_connections 2024-03-19T02:02:46.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.838+0000 7f03b369d640 1 --2- 172.21.15.55:0/3658549554 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0390067660 0x7f0390069b20 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:46.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.838+0000 7f03b369d640 1 --2- 172.21.15.55:0/3658549554 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f03ac152120 0x7f03ac0c4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:46.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.838+0000 7f03b369d640 1 --2- 172.21.15.55:0/3658549554 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f03ac0c9610 0x7f03ac1f4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:46.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.838+0000 7f03b369d640 1 --2- 172.21.15.55:0/3658549554 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03ac070780 0x7f03ac156820 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:46.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.838+0000 7f03b369d640 1 -- 172.21.15.55:0/3658549554 >> 172.21.15.55:0/3658549554 conn(0x7f03ac067600 msgr2=0x7f03ac0c9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:46.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.838+0000 7f03b369d640 1 -- 172.21.15.55:0/3658549554 shutdown_connections 2024-03-19T02:02:46.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:46.838+0000 7f03b369d640 1 -- 172.21.15.55:0/3658549554 wait complete. 2024-03-19T02:02:47.466 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:47.466 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:48.467 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:48.612 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.610+0000 7f3a4be5d640 1 Processor -- start 2024-03-19T02:02:48.612 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.614+0000 7f3a4be5d640 1 -- start start 2024-03-19T02:02:48.613 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.614+0000 7f3a4be5d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3a4406c9f0 0x7f3a440c92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:48.614 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.614+0000 7f3a4be5d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f3a440c9800 0x7f3a441427f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:48.614 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.614+0000 7f3a4be5d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a44070e80 con 0x7f3a4406c610 2024-03-19T02:02:48.614 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.614+0000 7f3a4be5d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f3a4406fb00 con 0x7f3a4406c9f0 2024-03-19T02:02:48.614 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.614+0000 7f3a4be5d640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f3a4406fde0 con 0x7f3a440c9800 2024-03-19T02:02:48.614 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.614+0000 7f3a493f8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3a4406c9f0 0x7f3a440c92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:48.614 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.614+0000 7f3a4a3fa640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f3a440c9800 0x7f3a441427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:48.615 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.614+0000 7f3a493f8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3a4406c9f0 0x7f3a440c92c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:46836/0 (socket says 172.21.15.55:46836) 2024-03-19T02:02:48.615 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.614+0000 7f3a493f8640 1 -- 172.21.15.55:0/3432579242 learned_addr learned my addr 172.21.15.55:0/3432579242 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:48.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.618+0000 7f3a493f8640 1 -- 172.21.15.55:0/3432579242 >> v2:172.21.15.84:3300/0 conn(0x7f3a440c9800 msgr2=0x7f3a441427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:48.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.618+0000 7f3a493f8640 1 --2- 172.21.15.55:0/3432579242 >> v2:172.21.15.84:3300/0 conn(0x7f3a440c9800 0x7f3a441427f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:48.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.618+0000 7f3a493f8640 1 -- 172.21.15.55:0/3432579242 >> v1:172.21.15.55:6789/0 conn(0x7f3a4406c610 legacy=0x7f3a4415ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:48.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.618+0000 7f3a493f8640 1 -- 172.21.15.55:0/3432579242 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a440700c0 con 0x7f3a4406c9f0 2024-03-19T02:02:48.621 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.622+0000 7f3a493f8640 1 --2- 172.21.15.55:0/3432579242 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a4406c9f0 0x7f3a440c92c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f3a440b3090 tx=0x7f3a3401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=332d7bb191724d9a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:48.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.626+0000 7f3a48bf7640 1 -- 172.21.15.55:0/3432579242 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a340b5070 con 0x7f3a4406c9f0 2024-03-19T02:02:48.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.626+0000 7f3a48bf7640 1 -- 172.21.15.55:0/3432579242 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a340a8a10 con 0x7f3a4406c9f0 2024-03-19T02:02:48.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.626+0000 7f3a48bf7640 1 -- 172.21.15.55:0/3432579242 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a340a8de0 con 0x7f3a4406c9f0 2024-03-19T02:02:48.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.626+0000 7f3a4be5d640 1 -- 172.21.15.55:0/3432579242 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a4406c9f0 msgr2=0x7f3a440c92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:48.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.626+0000 7f3a4be5d640 1 --2- 172.21.15.55:0/3432579242 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a4406c9f0 0x7f3a440c92c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f3a440b3090 tx=0x7f3a3401ed10 comp rx=0 tx=0).stop 2024-03-19T02:02:48.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.626+0000 7f3a4be5d640 1 -- 172.21.15.55:0/3432579242 shutdown_connections 2024-03-19T02:02:48.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.626+0000 7f3a4be5d640 1 --2- 172.21.15.55:0/3432579242 >> v2:172.21.15.84:3300/0 conn(0x7f3a440c9800 0x7f3a441427f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:48.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.626+0000 7f3a4be5d640 1 --2- 172.21.15.55:0/3432579242 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a4406c9f0 0x7f3a440c92c0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:48.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.626+0000 7f3a4be5d640 1 -- 172.21.15.55:0/3432579242 >> 172.21.15.55:0/3432579242 conn(0x7f3a44067600 msgr2=0x7f3a44071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:48.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.626+0000 7f3a4be5d640 1 -- 172.21.15.55:0/3432579242 shutdown_connections 2024-03-19T02:02:48.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.626+0000 7f3a4be5d640 1 -- 172.21.15.55:0/3432579242 wait complete. 2024-03-19T02:02:48.631 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.630+0000 7f3a4be5d640 1 Processor -- start 2024-03-19T02:02:48.632 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.630+0000 7f3a4be5d640 1 -- start start 2024-03-19T02:02:48.632 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.630+0000 7f3a4be5d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3a4406c610 0x7f3a441f1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:48.632 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.630+0000 7f3a4be5d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a4406c9f0 0x7f3a440c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:48.632 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.630+0000 7f3a4be5d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3a440c9800 0x7f3a440cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:48.632 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.630+0000 7f3a4be5d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3a44070960 con 0x7f3a4406c9f0 2024-03-19T02:02:48.632 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.630+0000 7f3a4be5d640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3a44070680 con 0x7f3a440c9800 2024-03-19T02:02:48.632 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.630+0000 7f3a4be5d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3a4406ceb0 con 0x7f3a4406c610 2024-03-19T02:02:48.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.630+0000 7f3a493f8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a4406c9f0 0x7f3a440c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:48.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.630+0000 7f3a4a3fa640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3a440c9800 0x7f3a440cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:48.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.630+0000 7f3a493f8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a4406c9f0 0x7f3a440c9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:46838/0 (socket says 172.21.15.55:46838) 2024-03-19T02:02:48.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.630+0000 7f3a493f8640 1 -- 172.21.15.55:0/1669267176 learned_addr learned my addr 172.21.15.55:0/1669267176 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:48.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.634+0000 7f3a49bf9640 1 --2- 172.21.15.55:0/1669267176 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3a4406c610 0x7f3a441f1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:48.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.634+0000 7f3a493f8640 1 -- 172.21.15.55:0/1669267176 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3a4406c610 msgr2=0x7f3a441f1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:48.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.634+0000 7f3a493f8640 1 --2- 172.21.15.55:0/1669267176 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3a4406c610 0x7f3a441f1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:48.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.634+0000 7f3a493f8640 1 -- 172.21.15.55:0/1669267176 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3a440c9800 msgr2=0x7f3a440cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:48.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.634+0000 7f3a493f8640 1 --2- 172.21.15.55:0/1669267176 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3a440c9800 0x7f3a440cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:48.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.634+0000 7f3a493f8640 1 -- 172.21.15.55:0/1669267176 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a441f22f0 con 0x7f3a4406c9f0 2024-03-19T02:02:48.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.638+0000 7f3a493f8640 1 --2- 172.21.15.55:0/1669267176 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a4406c9f0 0x7f3a440c9e40 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f3a3401b920 tx=0x7f3a3401bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:48.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.646+0000 7f3a3a7fc640 1 -- 172.21.15.55:0/1669267176 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a340b5070 con 0x7f3a4406c9f0 2024-03-19T02:02:48.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.646+0000 7f3a3a7fc640 1 -- 172.21.15.55:0/1669267176 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a340b0a50 con 0x7f3a4406c9f0 2024-03-19T02:02:48.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.646+0000 7f3a3a7fc640 1 -- 172.21.15.55:0/1669267176 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a340b0e40 con 0x7f3a4406c9f0 2024-03-19T02:02:48.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.646+0000 7f3a4be5d640 1 -- 172.21.15.55:0/1669267176 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a441f25d0 con 0x7f3a4406c9f0 2024-03-19T02:02:48.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.646+0000 7f3a4be5d640 1 -- 172.21.15.55:0/1669267176 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a441f2b90 con 0x7f3a4406c9f0 2024-03-19T02:02:48.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.650+0000 7f3a4be5d640 1 -- 172.21.15.55:0/1669267176 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a44068a40 con 0x7f3a4406c9f0 2024-03-19T02:02:48.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.934+0000 7f3a3a7fc640 1 -- 172.21.15.55:0/1669267176 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f3a340c2060 con 0x7f3a4406c9f0 2024-03-19T02:02:48.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.934+0000 7f3a3a7fc640 1 --2- 172.21.15.55:0/1669267176 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3a280676b0 0x7f3a28069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:48.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.934+0000 7f3a49bf9640 1 --2- 172.21.15.55:0/1669267176 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3a280676b0 0x7f3a28069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:48.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.934+0000 7f3a49bf9640 1 --2- 172.21.15.55:0/1669267176 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3a280676b0 0x7f3a28069b70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3a40001650 tx=0x7f3a40009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:48.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.974+0000 7f3a3a7fc640 1 -- 172.21.15.55:0/1669267176 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f3a340a89c0 con 0x7f3a4406c9f0 2024-03-19T02:02:48.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:48.978+0000 7f3a3a7fc640 1 -- 172.21.15.55:0/1669267176 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f3a34018870 con 0x7f3a4406c9f0 2024-03-19T02:02:49.257 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:49.258+0000 7f3a4be5d640 1 -- 172.21.15.55:0/1669267176 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a44142280 con 0x7f3a4406c9f0 2024-03-19T02:02:49.295 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:49.298+0000 7f3a3a7fc640 1 -- 172.21.15.55:0/1669267176 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f3a340f3700 con 0x7f3a4406c9f0 2024-03-19T02:02:49.296 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:49.296 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:49.296 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:49.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:49.298+0000 7f3a4be5d640 1 -- 172.21.15.55:0/1669267176 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3a280676b0 msgr2=0x7f3a28069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:49.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:49.298+0000 7f3a4be5d640 1 --2- 172.21.15.55:0/1669267176 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3a280676b0 0x7f3a28069b70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3a40001650 tx=0x7f3a40009040 comp rx=0 tx=0).stop 2024-03-19T02:02:49.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:49.298+0000 7f3a4be5d640 1 -- 172.21.15.55:0/1669267176 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a4406c9f0 msgr2=0x7f3a440c9e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:49.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:49.298+0000 7f3a4be5d640 1 --2- 172.21.15.55:0/1669267176 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a4406c9f0 0x7f3a440c9e40 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f3a3401b920 tx=0x7f3a3401bab0 comp rx=0 tx=0).stop 2024-03-19T02:02:49.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:49.298+0000 7f3a4be5d640 1 -- 172.21.15.55:0/1669267176 shutdown_connections 2024-03-19T02:02:49.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:49.298+0000 7f3a4be5d640 1 --2- 172.21.15.55:0/1669267176 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3a280676b0 0x7f3a28069b70 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:49.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:49.298+0000 7f3a4be5d640 1 --2- 172.21.15.55:0/1669267176 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3a440c9800 0x7f3a440cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:49.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:49.298+0000 7f3a4be5d640 1 --2- 172.21.15.55:0/1669267176 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a4406c9f0 0x7f3a440c9e40 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:49.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:49.298+0000 7f3a4be5d640 1 --2- 172.21.15.55:0/1669267176 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3a4406c610 0x7f3a441f1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:49.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:49.298+0000 7f3a4be5d640 1 -- 172.21.15.55:0/1669267176 >> 172.21.15.55:0/1669267176 conn(0x7f3a44067600 msgr2=0x7f3a44140230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:49.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:49.298+0000 7f3a4be5d640 1 -- 172.21.15.55:0/1669267176 shutdown_connections 2024-03-19T02:02:49.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:49.298+0000 7f3a4be5d640 1 -- 172.21.15.55:0/1669267176 wait complete. 2024-03-19T02:02:49.923 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:49.923 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:50.925 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:51.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.038+0000 7fc3c1080640 1 Processor -- start 2024-03-19T02:02:51.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.038+0000 7fc3c1080640 1 -- start start 2024-03-19T02:02:51.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.038+0000 7fc3c1080640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fc3bc15acb0 0x7fc3bc073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:51.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.038+0000 7fc3c1080640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3bc0c7fa0 con 0x7fc3bc06b2a0 2024-03-19T02:02:51.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.038+0000 7fc3c1080640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3bc0c8240 con 0x7fc3bc1544f0 2024-03-19T02:02:51.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.038+0000 7fc3c1080640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fc3bc15b210 con 0x7fc3bc15acb0 2024-03-19T02:02:51.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.042+0000 7fc3ba59c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fc3bc06b2a0 0x7fc3bc0cce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:43056/0 (socket says 172.21.15.55:43056) 2024-03-19T02:02:51.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.042+0000 7fc3ba59c640 1 -- 172.21.15.55:0/1975377187 learned_addr learned my addr 172.21.15.55:0/1975377187 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:02:51.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.042+0000 7fc3bb59e640 1 --2- 172.21.15.55:0/1975377187 >> v2:172.21.15.84:3300/0 conn(0x7fc3bc15acb0 0x7fc3bc073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:51.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.042+0000 7fc3b9d9b640 1 -- 172.21.15.55:0/1975377187 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2450309375 0 0) 0x7fc3bc0c7fa0 con 0x7fc3bc06b2a0 2024-03-19T02:02:51.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.042+0000 7fc3b9d9b640 1 -- 172.21.15.55:0/1975377187 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3a4003530 con 0x7fc3bc06b2a0 2024-03-19T02:02:51.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.046+0000 7fc3bb59e640 1 -- 172.21.15.55:0/1975377187 >> v1:172.21.15.84:6789/0 conn(0x7fc3bc1544f0 legacy=0x7fc3bc15e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:51.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.046+0000 7fc3bb59e640 1 -- 172.21.15.55:0/1975377187 >> v1:172.21.15.55:6789/0 conn(0x7fc3bc06b2a0 legacy=0x7fc3bc0cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:51.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.046+0000 7fc3bb59e640 1 -- 172.21.15.55:0/1975377187 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3bc15d920 con 0x7fc3bc15acb0 2024-03-19T02:02:51.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.050+0000 7fc3bb59e640 1 --2- 172.21.15.55:0/1975377187 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3bc15acb0 0x7fc3bc073540 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4009d90 tx=0x7fc3b401f120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae8cab4df0552ec8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:51.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.054+0000 7fc3b9d9b640 1 -- 172.21.15.55:0/1975377187 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b40b4070 con 0x7fc3bc15acb0 2024-03-19T02:02:51.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.058+0000 7fc3b9d9b640 1 -- 172.21.15.55:0/1975377187 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3b40af600 con 0x7fc3bc15acb0 2024-03-19T02:02:51.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.058+0000 7fc3b9d9b640 1 -- 172.21.15.55:0/1975377187 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b40af9b0 con 0x7fc3bc15acb0 2024-03-19T02:02:51.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.058+0000 7fc3c1080640 1 -- 172.21.15.55:0/1975377187 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3bc15acb0 msgr2=0x7fc3bc073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:51.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.058+0000 7fc3c1080640 1 --2- 172.21.15.55:0/1975377187 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3bc15acb0 0x7fc3bc073540 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4009d90 tx=0x7fc3b401f120 comp rx=0 tx=0).stop 2024-03-19T02:02:51.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.058+0000 7fc3c1080640 1 -- 172.21.15.55:0/1975377187 shutdown_connections 2024-03-19T02:02:51.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.058+0000 7fc3c1080640 1 --2- 172.21.15.55:0/1975377187 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3bc15acb0 0x7fc3bc073540 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:51.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.058+0000 7fc3c1080640 1 -- 172.21.15.55:0/1975377187 >> 172.21.15.55:0/1975377187 conn(0x7fc3bc06e670 msgr2=0x7fc3bc06ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:51.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.058+0000 7fc3c1080640 1 -- 172.21.15.55:0/1975377187 shutdown_connections 2024-03-19T02:02:51.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.058+0000 7fc3c1080640 1 -- 172.21.15.55:0/1975377187 wait complete. 2024-03-19T02:02:51.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.062+0000 7fc3c1080640 1 Processor -- start 2024-03-19T02:02:51.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.062+0000 7fc3c1080640 1 -- start start 2024-03-19T02:02:51.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.062+0000 7fc3c1080640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3bc06b2a0 0x7fc3bc1eff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:51.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.062+0000 7fc3c1080640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3bc1544f0 0x7fc3bc158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:51.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.062+0000 7fc3c1080640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3bc15acb0 0x7fc3bc1f2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:51.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.062+0000 7fc3c1080640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc3bc0cd540 con 0x7fc3bc06b2a0 2024-03-19T02:02:51.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.062+0000 7fc3c1080640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc3bc1548d0 con 0x7fc3bc15acb0 2024-03-19T02:02:51.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.062+0000 7fc3c1080640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc3bc0cd6c0 con 0x7fc3bc1544f0 2024-03-19T02:02:51.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.062+0000 7fc3bad9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3bc06b2a0 0x7fc3bc1eff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:51.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.062+0000 7fc3bb59e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3bc15acb0 0x7fc3bc1f2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:51.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.062+0000 7fc3bad9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3bc06b2a0 0x7fc3bc1eff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39418/0 (socket says 172.21.15.55:39418) 2024-03-19T02:02:51.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.062+0000 7fc3bad9d640 1 -- 172.21.15.55:0/2708790548 learned_addr learned my addr 172.21.15.55:0/2708790548 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:51.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.062+0000 7fc3ba59c640 1 --2- 172.21.15.55:0/2708790548 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3bc1544f0 0x7fc3bc158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:51.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.066+0000 7fc3bad9d640 1 -- 172.21.15.55:0/2708790548 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3bc1544f0 msgr2=0x7fc3bc158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:51.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.066+0000 7fc3bad9d640 1 --2- 172.21.15.55:0/2708790548 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3bc1544f0 0x7fc3bc158b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:51.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.066+0000 7fc3bad9d640 1 -- 172.21.15.55:0/2708790548 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3bc15acb0 msgr2=0x7fc3bc1f2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:51.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.066+0000 7fc3bad9d640 1 --2- 172.21.15.55:0/2708790548 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3bc15acb0 0x7fc3bc1f2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:51.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.066+0000 7fc3bad9d640 1 -- 172.21.15.55:0/2708790548 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3bc1590c0 con 0x7fc3bc06b2a0 2024-03-19T02:02:51.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.066+0000 7fc3bad9d640 1 --2- 172.21.15.55:0/2708790548 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3bc06b2a0 0x7fc3bc1eff90 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac00c4c0 tx=0x7fc3ac001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:51.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.074+0000 7fc3a37fe640 1 -- 172.21.15.55:0/2708790548 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac017070 con 0x7fc3bc06b2a0 2024-03-19T02:02:51.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.074+0000 7fc3a37fe640 1 -- 172.21.15.55:0/2708790548 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3ac003600 con 0x7fc3bc06b2a0 2024-03-19T02:02:51.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.074+0000 7fc3c1080640 1 -- 172.21.15.55:0/2708790548 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3bc1f0580 con 0x7fc3bc06b2a0 2024-03-19T02:02:51.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.074+0000 7fc3a37fe640 1 -- 172.21.15.55:0/2708790548 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac0039f0 con 0x7fc3bc06b2a0 2024-03-19T02:02:51.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.074+0000 7fc3c1080640 1 -- 172.21.15.55:0/2708790548 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3bc1f0b20 con 0x7fc3bc06b2a0 2024-03-19T02:02:51.073 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.074+0000 7fc3c1080640 1 -- 172.21.15.55:0/2708790548 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3bc068a40 con 0x7fc3bc06b2a0 2024-03-19T02:02:51.092 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.094+0000 7fc3a37fe640 1 -- 172.21.15.55:0/2708790548 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc3ac01a070 con 0x7fc3bc06b2a0 2024-03-19T02:02:51.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.094+0000 7fc3a37fe640 1 --2- 172.21.15.55:0/2708790548 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3a80676b0 0x7fc3a8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:51.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.094+0000 7fc3ba59c640 1 --2- 172.21.15.55:0/2708790548 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3a80676b0 0x7fc3a8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:51.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.094+0000 7fc3ba59c640 1 --2- 172.21.15.55:0/2708790548 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3a80676b0 0x7fc3a8069b70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc1edbc0 tx=0x7fc3b0000bd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:51.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.390+0000 7fc3a37fe640 1 -- 172.21.15.55:0/2708790548 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fc3ac053690 con 0x7fc3bc06b2a0 2024-03-19T02:02:51.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.390+0000 7fc3a37fe640 1 -- 172.21.15.55:0/2708790548 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc3ac021020 con 0x7fc3bc06b2a0 2024-03-19T02:02:51.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.602+0000 7fc3c1080640 1 -- 172.21.15.55:0/2708790548 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3bc0ccfa0 con 0x7fc3bc06b2a0 2024-03-19T02:02:51.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.618+0000 7fc3a37fe640 1 -- 172.21.15.55:0/2708790548 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fc3ac0555f0 con 0x7fc3bc06b2a0 2024-03-19T02:02:51.617 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:51.617 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:51.618 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:51.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.618+0000 7fc3c1080640 1 -- 172.21.15.55:0/2708790548 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3a80676b0 msgr2=0x7fc3a8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:51.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.618+0000 7fc3c1080640 1 --2- 172.21.15.55:0/2708790548 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3a80676b0 0x7fc3a8069b70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc1edbc0 tx=0x7fc3b0000bd0 comp rx=0 tx=0).stop 2024-03-19T02:02:51.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.618+0000 7fc3c1080640 1 -- 172.21.15.55:0/2708790548 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3bc06b2a0 msgr2=0x7fc3bc1eff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:51.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.618+0000 7fc3c1080640 1 --2- 172.21.15.55:0/2708790548 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3bc06b2a0 0x7fc3bc1eff90 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac00c4c0 tx=0x7fc3ac001e80 comp rx=0 tx=0).stop 2024-03-19T02:02:51.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.618+0000 7fc3c1080640 1 -- 172.21.15.55:0/2708790548 shutdown_connections 2024-03-19T02:02:51.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.618+0000 7fc3c1080640 1 --2- 172.21.15.55:0/2708790548 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3a80676b0 0x7fc3a8069b70 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:51.619 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.618+0000 7fc3c1080640 1 --2- 172.21.15.55:0/2708790548 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3bc15acb0 0x7fc3bc1f2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:51.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.618+0000 7fc3c1080640 1 --2- 172.21.15.55:0/2708790548 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3bc1544f0 0x7fc3bc158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:51.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.618+0000 7fc3c1080640 1 --2- 172.21.15.55:0/2708790548 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3bc06b2a0 0x7fc3bc1eff90 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:51.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.618+0000 7fc3c1080640 1 -- 172.21.15.55:0/2708790548 >> 172.21.15.55:0/2708790548 conn(0x7fc3bc06e670 msgr2=0x7fc3bc1e35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:51.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.618+0000 7fc3c1080640 1 -- 172.21.15.55:0/2708790548 shutdown_connections 2024-03-19T02:02:51.620 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:51.618+0000 7fc3c1080640 1 -- 172.21.15.55:0/2708790548 wait complete. 2024-03-19T02:02:52.242 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:52.242 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:53.243 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:53.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.374+0000 7f616f828640 1 Processor -- start 2024-03-19T02:02:53.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.374+0000 7f616f828640 1 -- start start 2024-03-19T02:02:53.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.374+0000 7f616f828640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f61681544f0 0x7f6168073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:53.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.374+0000 7f616f828640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f616806b2a0 0x7f616815e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:53.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.374+0000 7f616f828640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61680c7fa0 con 0x7f616815acb0 2024-03-19T02:02:53.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.374+0000 7f616f828640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f616815b210 con 0x7f61681544f0 2024-03-19T02:02:53.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.374+0000 7f616f828640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f61680c8240 con 0x7f616806b2a0 2024-03-19T02:02:53.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.378+0000 7f616d5c4640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f61681544f0 0x7f6168073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:53.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.378+0000 7f616d5c4640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f61681544f0 0x7f6168073540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39420/0 (socket says 172.21.15.55:39420) 2024-03-19T02:02:53.376 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.378+0000 7f616d5c4640 1 -- 172.21.15.55:0/3796357199 learned_addr learned my addr 172.21.15.55:0/3796357199 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:53.376 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.378+0000 7f616cdc3640 1 --2- 172.21.15.55:0/3796357199 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f616806b2a0 0x7f616815e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:53.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.378+0000 7f616d5c4640 1 -- 172.21.15.55:0/3796357199 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f616806b2a0 msgr2=0x7f616815e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:53.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.378+0000 7f616d5c4640 1 --2- 172.21.15.55:0/3796357199 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f616806b2a0 0x7f616815e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:53.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.378+0000 7f616d5c4640 1 -- 172.21.15.55:0/3796357199 >> v1:172.21.15.84:6789/0 conn(0x7f616815acb0 legacy=0x7f61680cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:53.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.378+0000 7f616d5c4640 1 -- 172.21.15.55:0/3796357199 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f616815d920 con 0x7f61681544f0 2024-03-19T02:02:53.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.382+0000 7f616d5c4640 1 --2- 172.21.15.55:0/3796357199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61681544f0 0x7f6168073540 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f6164009870 tx=0x7f616401ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f4252051925fed0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:53.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.386+0000 7f615ffff640 1 -- 172.21.15.55:0/3796357199 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61640b5070 con 0x7f61681544f0 2024-03-19T02:02:53.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.386+0000 7f615ffff640 1 -- 172.21.15.55:0/3796357199 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61640a8a50 con 0x7f61681544f0 2024-03-19T02:02:53.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.386+0000 7f615ffff640 1 -- 172.21.15.55:0/3796357199 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61640a8e20 con 0x7f61681544f0 2024-03-19T02:02:53.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.386+0000 7f616f828640 1 -- 172.21.15.55:0/3796357199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61681544f0 msgr2=0x7f6168073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:53.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.386+0000 7f616f828640 1 --2- 172.21.15.55:0/3796357199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61681544f0 0x7f6168073540 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f6164009870 tx=0x7f616401ed50 comp rx=0 tx=0).stop 2024-03-19T02:02:53.387 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.390+0000 7f616f828640 1 -- 172.21.15.55:0/3796357199 shutdown_connections 2024-03-19T02:02:53.388 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.390+0000 7f616f828640 1 --2- 172.21.15.55:0/3796357199 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f616806b2a0 0x7f616815e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:53.388 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.390+0000 7f616f828640 1 --2- 172.21.15.55:0/3796357199 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61681544f0 0x7f6168073540 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:53.388 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.390+0000 7f616f828640 1 -- 172.21.15.55:0/3796357199 >> 172.21.15.55:0/3796357199 conn(0x7f616806e670 msgr2=0x7f616806ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:53.388 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.390+0000 7f616f828640 1 -- 172.21.15.55:0/3796357199 shutdown_connections 2024-03-19T02:02:53.388 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.390+0000 7f616f828640 1 -- 172.21.15.55:0/3796357199 wait complete. 2024-03-19T02:02:53.388 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.390+0000 7f616f828640 1 Processor -- start 2024-03-19T02:02:53.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.390+0000 7f616f828640 1 -- start start 2024-03-19T02:02:53.389 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.390+0000 7f616f828640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f616806b2a0 0x7f61680c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:53.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.390+0000 7f616f828640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61681544f0 0x7f61681f0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:53.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.390+0000 7f616f828640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f616815acb0 0x7f61681f07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:53.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.390+0000 7f616f828640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f61681548d0 con 0x7f616815acb0 2024-03-19T02:02:53.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.390+0000 7f616f828640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f616815e660 con 0x7f61681544f0 2024-03-19T02:02:53.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.390+0000 7f616f828640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f61680cd540 con 0x7f616806b2a0 2024-03-19T02:02:53.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.390+0000 7f616ddc5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f616815acb0 0x7f61681f07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:53.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.390+0000 7f616cdc3640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61681544f0 0x7f61681f0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:53.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.390+0000 7f616ddc5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f616815acb0 0x7f61681f07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39434/0 (socket says 172.21.15.55:39434) 2024-03-19T02:02:53.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.390+0000 7f616ddc5640 1 -- 172.21.15.55:0/3163213991 learned_addr learned my addr 172.21.15.55:0/3163213991 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:53.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.390+0000 7f616d5c4640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f616806b2a0 0x7f61680c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:53.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.398+0000 7f616ddc5640 1 -- 172.21.15.55:0/3163213991 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f616806b2a0 msgr2=0x7f61680c9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:53.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.398+0000 7f616ddc5640 1 --2- 172.21.15.55:0/3163213991 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f616806b2a0 0x7f61680c9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:53.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.398+0000 7f616ddc5640 1 -- 172.21.15.55:0/3163213991 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61681544f0 msgr2=0x7f61681f0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:53.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.398+0000 7f616ddc5640 1 --2- 172.21.15.55:0/3163213991 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61681544f0 0x7f61681f0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:53.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.398+0000 7f616ddc5640 1 -- 172.21.15.55:0/3163213991 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f616815cf90 con 0x7f616815acb0 2024-03-19T02:02:53.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.398+0000 7f616ddc5640 1 --2- 172.21.15.55:0/3163213991 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f616815acb0 0x7f61681f07c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f616000c760 tx=0x7f616000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:53.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.402+0000 7f615dffb640 1 -- 172.21.15.55:0/3163213991 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6160003ba0 con 0x7f616815acb0 2024-03-19T02:02:53.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.402+0000 7f615dffb640 1 -- 172.21.15.55:0/3163213991 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6160003d40 con 0x7f616815acb0 2024-03-19T02:02:53.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.402+0000 7f615dffb640 1 -- 172.21.15.55:0/3163213991 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616000aa90 con 0x7f616815acb0 2024-03-19T02:02:53.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.402+0000 7f616f828640 1 -- 172.21.15.55:0/3163213991 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f616815d230 con 0x7f616815acb0 2024-03-19T02:02:53.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.406+0000 7f616f828640 1 -- 172.21.15.55:0/3163213991 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61681f3c70 con 0x7f616815acb0 2024-03-19T02:02:53.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.406+0000 7f616f828640 1 -- 172.21.15.55:0/3163213991 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6168068a40 con 0x7f616815acb0 2024-03-19T02:02:53.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.682+0000 7f615dffb640 1 -- 172.21.15.55:0/3163213991 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f616000ac30 con 0x7f616815acb0 2024-03-19T02:02:53.683 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.686+0000 7f615dffb640 1 --2- 172.21.15.55:0/3163213991 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f614c0676e0 0x7f614c069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:53.684 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.686+0000 7f616d5c4640 1 --2- 172.21.15.55:0/3163213991 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f614c0676e0 0x7f614c069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:53.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.686+0000 7f616d5c4640 1 --2- 172.21.15.55:0/3163213991 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f614c0676e0 0x7f614c069ba0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f61640b2000 tx=0x7f6164009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:53.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.706+0000 7f615dffb640 1 -- 172.21.15.55:0/3163213991 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f6160054780 con 0x7f616815acb0 2024-03-19T02:02:53.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.706+0000 7f615dffb640 1 -- 172.21.15.55:0/3163213991 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f61600566e0 con 0x7f616815acb0 2024-03-19T02:02:53.926 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.926+0000 7f616f828640 1 -- 172.21.15.55:0/3163213991 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61680c7be0 con 0x7f616815acb0 2024-03-19T02:02:53.945 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.946+0000 7f615dffb640 1 -- 172.21.15.55:0/3163213991 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f616005a780 con 0x7f616815acb0 2024-03-19T02:02:53.945 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:53.945 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:53.946 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:53.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.950+0000 7f616f828640 1 -- 172.21.15.55:0/3163213991 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f614c0676e0 msgr2=0x7f614c069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:53.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.950+0000 7f616f828640 1 --2- 172.21.15.55:0/3163213991 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f614c0676e0 0x7f614c069ba0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f61640b2000 tx=0x7f6164009040 comp rx=0 tx=0).stop 2024-03-19T02:02:53.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.950+0000 7f616f828640 1 -- 172.21.15.55:0/3163213991 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f616815acb0 msgr2=0x7f61681f07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:53.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.950+0000 7f616f828640 1 --2- 172.21.15.55:0/3163213991 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f616815acb0 0x7f61681f07c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f616000c760 tx=0x7f616000cc20 comp rx=0 tx=0).stop 2024-03-19T02:02:53.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.950+0000 7f616f828640 1 -- 172.21.15.55:0/3163213991 shutdown_connections 2024-03-19T02:02:53.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.950+0000 7f616f828640 1 --2- 172.21.15.55:0/3163213991 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f614c0676e0 0x7f614c069ba0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:53.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.950+0000 7f616f828640 1 --2- 172.21.15.55:0/3163213991 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f616815acb0 0x7f61681f07c0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:53.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.950+0000 7f616f828640 1 --2- 172.21.15.55:0/3163213991 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61681544f0 0x7f61681f0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:53.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.950+0000 7f616f828640 1 --2- 172.21.15.55:0/3163213991 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f616806b2a0 0x7f61680c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:53.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.950+0000 7f616f828640 1 -- 172.21.15.55:0/3163213991 >> 172.21.15.55:0/3163213991 conn(0x7f616806e670 msgr2=0x7f61680cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:53.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.950+0000 7f616f828640 1 -- 172.21.15.55:0/3163213991 shutdown_connections 2024-03-19T02:02:53.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:53.950+0000 7f616f828640 1 -- 172.21.15.55:0/3163213991 wait complete. 2024-03-19T02:02:54.568 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:54.569 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:55.569 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:55.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.697+0000 7f0048770640 1 Processor -- start 2024-03-19T02:02:55.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.697+0000 7f0048770640 1 -- start start 2024-03-19T02:02:55.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.697+0000 7f0048770640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f00400cbf30 0x7f00400cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:55.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.697+0000 7f0048770640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f004006c6f0 0x7f00400cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:55.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.697+0000 7f0048770640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f004006cb90 con 0x7f004015a3b0 2024-03-19T02:02:55.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.697+0000 7f0048770640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f00400ce640 con 0x7f004006c6f0 2024-03-19T02:02:55.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.697+0000 7f0048770640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f00400cda30 con 0x7f00400cbf30 2024-03-19T02:02:55.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.701+0000 7f0046d0d640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f004015a3b0 0x7f004015a830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:43066/0 (socket says 172.21.15.55:43066) 2024-03-19T02:02:55.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.701+0000 7f0046d0d640 1 -- 172.21.15.55:0/3154271431 learned_addr learned my addr 172.21.15.55:0/3154271431 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:02:55.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.701+0000 7f004650c640 1 --2- 172.21.15.55:0/3154271431 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f00400cbf30 0x7f00400cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:55.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.701+0000 7f0045d0b640 1 --2- 172.21.15.55:0/3154271431 >> v2:172.21.15.84:3300/0 conn(0x7f004006c6f0 0x7f00400cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:55.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.701+0000 7f004550a640 1 -- 172.21.15.55:0/3154271431 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 372953269 0 0) 0x7f004006cb90 con 0x7f004015a3b0 2024-03-19T02:02:55.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.701+0000 7f004550a640 1 -- 172.21.15.55:0/3154271431 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f002c003530 con 0x7f004015a3b0 2024-03-19T02:02:55.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.705+0000 7f004650c640 1 -- 172.21.15.55:0/3154271431 >> v2:172.21.15.84:3300/0 conn(0x7f004006c6f0 msgr2=0x7f00400cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:55.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.705+0000 7f004650c640 1 --2- 172.21.15.55:0/3154271431 >> v2:172.21.15.84:3300/0 conn(0x7f004006c6f0 0x7f00400cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:55.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.705+0000 7f004650c640 1 -- 172.21.15.55:0/3154271431 >> v1:172.21.15.55:6789/0 conn(0x7f004015a3b0 legacy=0x7f004015a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:55.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.705+0000 7f004650c640 1 -- 172.21.15.55:0/3154271431 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00400cdd10 con 0x7f00400cbf30 2024-03-19T02:02:55.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.709+0000 7f004650c640 1 --2- 172.21.15.55:0/3154271431 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f00400cbf30 0x7f00400cead0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f003c0097c0 tx=0x7f003c01c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f5a26e631f9a121 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:55.714 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.713+0000 7f004550a640 1 -- 172.21.15.55:0/3154271431 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c0b0070 con 0x7f00400cbf30 2024-03-19T02:02:55.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.721+0000 7f004550a640 1 -- 172.21.15.55:0/3154271431 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f003c0ab600 con 0x7f00400cbf30 2024-03-19T02:02:55.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.721+0000 7f0048770640 1 -- 172.21.15.55:0/3154271431 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f00400cbf30 msgr2=0x7f00400cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:55.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.721+0000 7f0048770640 1 --2- 172.21.15.55:0/3154271431 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f00400cbf30 0x7f00400cead0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f003c0097c0 tx=0x7f003c01c410 comp rx=0 tx=0).stop 2024-03-19T02:02:55.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.721+0000 7f0048770640 1 -- 172.21.15.55:0/3154271431 shutdown_connections 2024-03-19T02:02:55.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.721+0000 7f0048770640 1 --2- 172.21.15.55:0/3154271431 >> v2:172.21.15.84:3300/0 conn(0x7f004006c6f0 0x7f00400cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:55.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.721+0000 7f0048770640 1 --2- 172.21.15.55:0/3154271431 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f00400cbf30 0x7f00400cead0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:55.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.721+0000 7f0048770640 1 -- 172.21.15.55:0/3154271431 >> 172.21.15.55:0/3154271431 conn(0x7f00400b16a0 msgr2=0x7f00400b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:55.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.721+0000 7f0048770640 1 -- 172.21.15.55:0/3154271431 shutdown_connections 2024-03-19T02:02:55.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.721+0000 7f0048770640 1 -- 172.21.15.55:0/3154271431 wait complete. 2024-03-19T02:02:55.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.721+0000 7f0048770640 1 Processor -- start 2024-03-19T02:02:55.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.721+0000 7f0048770640 1 -- start start 2024-03-19T02:02:55.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.725+0000 7f0048770640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f004006c6f0 0x7f0040151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:55.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.725+0000 7f0048770640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f00400cbf30 0x7f0040154f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:55.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.725+0000 7f0048770640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f004015a3b0 0x7f004015d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:55.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.725+0000 7f0048770640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f00400cc310 con 0x7f004006c6f0 2024-03-19T02:02:55.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.725+0000 7f0048770640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f00400b23b0 con 0x7f00400cbf30 2024-03-19T02:02:55.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.725+0000 7f0048770640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f00400cc490 con 0x7f004015a3b0 2024-03-19T02:02:55.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.725+0000 7f004650c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f004006c6f0 0x7f0040151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:55.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.725+0000 7f0046d0d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f004015a3b0 0x7f004015d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:55.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.725+0000 7f0045d0b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f00400cbf30 0x7f0040154f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:55.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.725+0000 7f004650c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f004006c6f0 0x7f0040151680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39448/0 (socket says 172.21.15.55:39448) 2024-03-19T02:02:55.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.725+0000 7f004650c640 1 -- 172.21.15.55:0/1860796611 learned_addr learned my addr 172.21.15.55:0/1860796611 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:55.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.729+0000 7f004650c640 1 -- 172.21.15.55:0/1860796611 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f004015a3b0 msgr2=0x7f004015d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:55.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.729+0000 7f004650c640 1 --2- 172.21.15.55:0/1860796611 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f004015a3b0 0x7f004015d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:55.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.729+0000 7f004650c640 1 -- 172.21.15.55:0/1860796611 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f00400cbf30 msgr2=0x7f0040154f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:55.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.729+0000 7f004650c640 1 --2- 172.21.15.55:0/1860796611 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f00400cbf30 0x7f0040154f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:55.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.729+0000 7f004650c640 1 -- 172.21.15.55:0/1860796611 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0040157840 con 0x7f004006c6f0 2024-03-19T02:02:55.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.733+0000 7f004650c640 1 --2- 172.21.15.55:0/1860796611 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f004006c6f0 0x7f0040151680 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f003c0ae040 tx=0x7f003c0191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:55.738 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.737+0000 7f0036ffd640 1 -- 172.21.15.55:0/1860796611 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c0b0040 con 0x7f004006c6f0 2024-03-19T02:02:55.738 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.737+0000 7f0036ffd640 1 -- 172.21.15.55:0/1860796611 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f003c0b9070 con 0x7f004006c6f0 2024-03-19T02:02:55.738 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.737+0000 7f0048770640 1 -- 172.21.15.55:0/1860796611 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0040157b20 con 0x7f004006c6f0 2024-03-19T02:02:55.738 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.737+0000 7f0036ffd640 1 -- 172.21.15.55:0/1860796611 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c0abbf0 con 0x7f004006c6f0 2024-03-19T02:02:55.739 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.737+0000 7f0048770640 1 -- 172.21.15.55:0/1860796611 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00400c8cd0 con 0x7f004006c6f0 2024-03-19T02:02:55.739 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:55.737+0000 7f0048770640 1 -- 172.21.15.55:0/1860796611 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0040068330 con 0x7f004006c6f0 2024-03-19T02:02:56.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:56.033+0000 7f0036ffd640 1 -- 172.21.15.55:0/1860796611 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f003c0ab4c0 con 0x7f004006c6f0 2024-03-19T02:02:56.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:56.033+0000 7f0036ffd640 1 --2- 172.21.15.55:0/1860796611 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f00240676b0 0x7f0024069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:56.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:56.033+0000 7f0045d0b640 1 --2- 172.21.15.55:0/1860796611 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f00240676b0 0x7f0024069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:56.035 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:56.033+0000 7f0045d0b640 1 --2- 172.21.15.55:0/1860796611 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f00240676b0 0x7f0024069b70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f00400b59d0 tx=0x7f0030009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:56.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:56.073+0000 7f0036ffd640 1 -- 172.21.15.55:0/1860796611 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f003c0ee360 con 0x7f004006c6f0 2024-03-19T02:02:56.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:56.073+0000 7f0036ffd640 1 -- 172.21.15.55:0/1860796611 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f003c016090 con 0x7f004006c6f0 2024-03-19T02:02:56.277 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:56.277+0000 7f0048770640 1 -- 172.21.15.55:0/1860796611 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00400b2190 con 0x7f004006c6f0 2024-03-19T02:02:56.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:56.297+0000 7f0036ffd640 1 -- 172.21.15.55:0/1860796611 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f003c0f02c0 con 0x7f004006c6f0 2024-03-19T02:02:56.300 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:56.300 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:56.301 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:56.303 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:56.301+0000 7f0048770640 1 -- 172.21.15.55:0/1860796611 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f00240676b0 msgr2=0x7f0024069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:56.303 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:56.301+0000 7f0048770640 1 --2- 172.21.15.55:0/1860796611 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f00240676b0 0x7f0024069b70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f00400b59d0 tx=0x7f0030009190 comp rx=0 tx=0).stop 2024-03-19T02:02:56.303 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:56.301+0000 7f0048770640 1 -- 172.21.15.55:0/1860796611 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f004006c6f0 msgr2=0x7f0040151680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:56.303 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:56.301+0000 7f0048770640 1 --2- 172.21.15.55:0/1860796611 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f004006c6f0 0x7f0040151680 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f003c0ae040 tx=0x7f003c0191b0 comp rx=0 tx=0).stop 2024-03-19T02:02:56.303 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:56.301+0000 7f0048770640 1 -- 172.21.15.55:0/1860796611 shutdown_connections 2024-03-19T02:02:56.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:56.301+0000 7f0048770640 1 --2- 172.21.15.55:0/1860796611 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f00240676b0 0x7f0024069b70 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:56.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:56.301+0000 7f0048770640 1 --2- 172.21.15.55:0/1860796611 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f004015a3b0 0x7f004015d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:56.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:56.301+0000 7f0048770640 1 --2- 172.21.15.55:0/1860796611 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f00400cbf30 0x7f0040154f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:56.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:56.301+0000 7f0048770640 1 --2- 172.21.15.55:0/1860796611 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f004006c6f0 0x7f0040151680 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:56.304 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:56.301+0000 7f0048770640 1 -- 172.21.15.55:0/1860796611 >> 172.21.15.55:0/1860796611 conn(0x7f00400b16a0 msgr2=0x7f00400af610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:56.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:56.301+0000 7f0048770640 1 -- 172.21.15.55:0/1860796611 shutdown_connections 2024-03-19T02:02:56.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:56.301+0000 7f0048770640 1 -- 172.21.15.55:0/1860796611 wait complete. 2024-03-19T02:02:56.924 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:56.924 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:02:57.925 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:02:58.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.037+0000 7fd50ff2a640 1 Processor -- start 2024-03-19T02:02:58.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.037+0000 7fd50ff2a640 1 -- start start 2024-03-19T02:02:58.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.041+0000 7fd50ff2a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd50806f1b0 0x7fd5080c5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:58.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.041+0000 7fd50ff2a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fd5080c5d50 0x7fd508158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:58.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.041+0000 7fd50ff2a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd50815b740 con 0x7fd50813de70 2024-03-19T02:02:58.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.041+0000 7fd50ff2a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fd50813e250 con 0x7fd50806f1b0 2024-03-19T02:02:58.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.041+0000 7fd50ff2a640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fd50815ba20 con 0x7fd5080c5d50 2024-03-19T02:02:58.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.041+0000 7fd50d4c5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd50806f1b0 0x7fd5080c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:58.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.041+0000 7fd50e4c7640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fd5080c5d50 0x7fd508158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:58.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.041+0000 7fd50d4c5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd50806f1b0 0x7fd5080c5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39464/0 (socket says 172.21.15.55:39464) 2024-03-19T02:02:58.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.041+0000 7fd50d4c5640 1 -- 172.21.15.55:0/661081544 learned_addr learned my addr 172.21.15.55:0/661081544 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:58.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.041+0000 7fd50d4c5640 1 -- 172.21.15.55:0/661081544 >> v2:172.21.15.84:3300/0 conn(0x7fd5080c5d50 msgr2=0x7fd508158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:58.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.041+0000 7fd50d4c5640 1 --2- 172.21.15.55:0/661081544 >> v2:172.21.15.84:3300/0 conn(0x7fd5080c5d50 0x7fd508158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:58.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.041+0000 7fd50d4c5640 1 -- 172.21.15.55:0/661081544 >> v1:172.21.15.84:6789/0 conn(0x7fd50813de70 legacy=0x7fd50806e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:02:58.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.045+0000 7fd50d4c5640 1 -- 172.21.15.55:0/661081544 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd50815bd00 con 0x7fd50806f1b0 2024-03-19T02:02:58.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.045+0000 7fd50d4c5640 1 --2- 172.21.15.55:0/661081544 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd50806f1b0 0x7fd5080c5810 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd508141840 tx=0x7fd4f801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a83bd58141f55f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:58.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.049+0000 7fd50ccc4640 1 -- 172.21.15.55:0/661081544 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f80b0070 con 0x7fd50806f1b0 2024-03-19T02:02:58.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.053+0000 7fd50ccc4640 1 -- 172.21.15.55:0/661081544 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4f80ab600 con 0x7fd50806f1b0 2024-03-19T02:02:58.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.053+0000 7fd50ccc4640 1 -- 172.21.15.55:0/661081544 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f80ab9d0 con 0x7fd50806f1b0 2024-03-19T02:02:58.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.053+0000 7fd50ff2a640 1 -- 172.21.15.55:0/661081544 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd50806f1b0 msgr2=0x7fd5080c5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:58.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.053+0000 7fd50ff2a640 1 --2- 172.21.15.55:0/661081544 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd50806f1b0 0x7fd5080c5810 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd508141840 tx=0x7fd4f801c4e0 comp rx=0 tx=0).stop 2024-03-19T02:02:58.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.053+0000 7fd50ff2a640 1 -- 172.21.15.55:0/661081544 shutdown_connections 2024-03-19T02:02:58.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.053+0000 7fd50ff2a640 1 --2- 172.21.15.55:0/661081544 >> v2:172.21.15.84:3300/0 conn(0x7fd5080c5d50 0x7fd508158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:58.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.053+0000 7fd50ff2a640 1 --2- 172.21.15.55:0/661081544 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd50806f1b0 0x7fd5080c5810 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:58.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.053+0000 7fd50ff2a640 1 -- 172.21.15.55:0/661081544 >> 172.21.15.55:0/661081544 conn(0x7fd50806fb40 msgr2=0x7fd50806ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:58.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.053+0000 7fd50ff2a640 1 -- 172.21.15.55:0/661081544 shutdown_connections 2024-03-19T02:02:58.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.053+0000 7fd50ff2a640 1 -- 172.21.15.55:0/661081544 wait complete. 2024-03-19T02:02:58.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.053+0000 7fd50ff2a640 1 Processor -- start 2024-03-19T02:02:58.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.053+0000 7fd50ff2a640 1 -- start start 2024-03-19T02:02:58.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.053+0000 7fd50ff2a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd50806f1b0 0x7fd5081363a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:58.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.053+0000 7fd50ff2a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5080c5d50 0x7fd5080cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:58.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.053+0000 7fd50ff2a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd50813de70 0x7fd508159190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:58.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.053+0000 7fd50ff2a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd508165170 con 0x7fd5080c5d50 2024-03-19T02:02:58.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.053+0000 7fd50ff2a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd508158d70 con 0x7fd50806f1b0 2024-03-19T02:02:58.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.053+0000 7fd50ff2a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd5081652f0 con 0x7fd50813de70 2024-03-19T02:02:58.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.057+0000 7fd50d4c5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5080c5d50 0x7fd5080cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:58.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.057+0000 7fd50dcc6640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd50806f1b0 0x7fd5081363a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:58.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.057+0000 7fd50d4c5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5080c5d50 0x7fd5080cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39472/0 (socket says 172.21.15.55:39472) 2024-03-19T02:02:58.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.057+0000 7fd50d4c5640 1 -- 172.21.15.55:0/2638056632 learned_addr learned my addr 172.21.15.55:0/2638056632 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:02:58.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.057+0000 7fd50e4c7640 1 --2- 172.21.15.55:0/2638056632 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd50813de70 0x7fd508159190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:58.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.061+0000 7fd50d4c5640 1 -- 172.21.15.55:0/2638056632 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd50813de70 msgr2=0x7fd508159190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:58.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.061+0000 7fd50d4c5640 1 --2- 172.21.15.55:0/2638056632 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd50813de70 0x7fd508159190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:58.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.061+0000 7fd50d4c5640 1 -- 172.21.15.55:0/2638056632 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd50806f1b0 msgr2=0x7fd5081363a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:58.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.061+0000 7fd50d4c5640 1 --2- 172.21.15.55:0/2638056632 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd50806f1b0 0x7fd5081363a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:58.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.061+0000 7fd50d4c5640 1 -- 172.21.15.55:0/2638056632 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd50815a9c0 con 0x7fd5080c5d50 2024-03-19T02:02:58.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.061+0000 7fd50d4c5640 1 --2- 172.21.15.55:0/2638056632 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5080c5d50 0x7fd5080cb1b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd4f80ae040 tx=0x7fd4f801ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:58.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.065+0000 7fd4fe7fc640 1 -- 172.21.15.55:0/2638056632 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f80ba020 con 0x7fd5080c5d50 2024-03-19T02:02:58.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.065+0000 7fd4fe7fc640 1 -- 172.21.15.55:0/2638056632 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4f80b6420 con 0x7fd5080c5d50 2024-03-19T02:02:58.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.065+0000 7fd4fe7fc640 1 -- 172.21.15.55:0/2638056632 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f80b0040 con 0x7fd5080c5d50 2024-03-19T02:02:58.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.065+0000 7fd50ff2a640 1 -- 172.21.15.55:0/2638056632 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd50815a6f0 con 0x7fd5080c5d50 2024-03-19T02:02:58.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.065+0000 7fd50ff2a640 1 -- 172.21.15.55:0/2638056632 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5081f2180 con 0x7fd5080c5d50 2024-03-19T02:02:58.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.069+0000 7fd50ff2a640 1 -- 172.21.15.55:0/2638056632 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd508068a40 con 0x7fd5080c5d50 2024-03-19T02:02:58.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.089+0000 7fd4fe7fc640 1 -- 172.21.15.55:0/2638056632 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fd4f80bd020 con 0x7fd5080c5d50 2024-03-19T02:02:58.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.089+0000 7fd4fe7fc640 1 --2- 172.21.15.55:0/2638056632 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd4ec0676e0 0x7fd4ec069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:02:58.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.089+0000 7fd50dcc6640 1 --2- 172.21.15.55:0/2638056632 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd4ec0676e0 0x7fd4ec069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:02:58.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.089+0000 7fd50dcc6640 1 --2- 172.21.15.55:0/2638056632 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd4ec0676e0 0x7fd4ec069ba0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd5040098e0 tx=0x7fd504009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:02:58.356 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.357+0000 7fd4fe7fc640 1 -- 172.21.15.55:0/2638056632 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fd4f80ee340 con 0x7fd5080c5d50 2024-03-19T02:02:58.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.357+0000 7fd4fe7fc640 1 -- 172.21.15.55:0/2638056632 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fd4f80c4070 con 0x7fd5080c5d50 2024-03-19T02:02:58.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.605+0000 7fd50ff2a640 1 -- 172.21.15.55:0/2638056632 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd50806ef90 con 0x7fd5080c5d50 2024-03-19T02:02:58.641 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.641+0000 7fd4fe7fc640 1 -- 172.21.15.55:0/2638056632 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fd4f80160d0 con 0x7fd5080c5d50 2024-03-19T02:02:58.642 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:02:58.642 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:02:58.642 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:02:58.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.645+0000 7fd50ff2a640 1 -- 172.21.15.55:0/2638056632 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd4ec0676e0 msgr2=0x7fd4ec069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:58.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.645+0000 7fd50ff2a640 1 --2- 172.21.15.55:0/2638056632 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd4ec0676e0 0x7fd4ec069ba0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd5040098e0 tx=0x7fd504009060 comp rx=0 tx=0).stop 2024-03-19T02:02:58.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.645+0000 7fd50ff2a640 1 -- 172.21.15.55:0/2638056632 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5080c5d50 msgr2=0x7fd5080cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:02:58.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.645+0000 7fd50ff2a640 1 --2- 172.21.15.55:0/2638056632 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5080c5d50 0x7fd5080cb1b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd4f80ae040 tx=0x7fd4f801ad20 comp rx=0 tx=0).stop 2024-03-19T02:02:58.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.645+0000 7fd50ff2a640 1 -- 172.21.15.55:0/2638056632 shutdown_connections 2024-03-19T02:02:58.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.645+0000 7fd50ff2a640 1 --2- 172.21.15.55:0/2638056632 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd4ec0676e0 0x7fd4ec069ba0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:58.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.645+0000 7fd50ff2a640 1 --2- 172.21.15.55:0/2638056632 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd50813de70 0x7fd508159190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:58.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.645+0000 7fd50ff2a640 1 --2- 172.21.15.55:0/2638056632 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5080c5d50 0x7fd5080cb1b0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:58.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.645+0000 7fd50ff2a640 1 --2- 172.21.15.55:0/2638056632 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd50806f1b0 0x7fd5081363a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:02:58.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.645+0000 7fd50ff2a640 1 -- 172.21.15.55:0/2638056632 >> 172.21.15.55:0/2638056632 conn(0x7fd50806fb40 msgr2=0x7fd508156870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:02:58.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.645+0000 7fd50ff2a640 1 -- 172.21.15.55:0/2638056632 shutdown_connections 2024-03-19T02:02:58.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:02:58.645+0000 7fd50ff2a640 1 -- 172.21.15.55:0/2638056632 wait complete. 2024-03-19T02:02:59.269 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:02:59.269 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:00.270 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:00.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.389+0000 7ff7f55e5640 1 Processor -- start 2024-03-19T02:03:00.390 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.389+0000 7ff7f55e5640 1 -- start start 2024-03-19T02:03:00.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.389+0000 7ff7f55e5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff7f013dec0 0x7ff7f00c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:00.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.389+0000 7ff7f55e5640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff7f0067440 0x7ff7f006be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:00.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.389+0000 7ff7f55e5640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7f0158220 con 0x7ff7f0067000 2024-03-19T02:03:00.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.389+0000 7ff7f55e5640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7ff7f0067890 con 0x7ff7f013dec0 2024-03-19T02:03:00.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.389+0000 7ff7f55e5640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff7f0158500 con 0x7ff7f0067440 2024-03-19T02:03:00.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.393+0000 7ff7eeffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff7f013dec0 0x7ff7f00c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:00.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.393+0000 7ff7ef7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff7f0067440 0x7ff7f006be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:00.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.393+0000 7ff7ef7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff7f0067440 0x7ff7f006be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:34552/0 (socket says 172.21.15.55:34552) 2024-03-19T02:03:00.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.393+0000 7ff7ef7fe640 1 -- 172.21.15.55:0/2118521199 learned_addr learned my addr 172.21.15.55:0/2118521199 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:00.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.397+0000 7ff7ef7fe640 1 -- 172.21.15.55:0/2118521199 >> v2:172.21.15.55:3300/0 conn(0x7ff7f013dec0 msgr2=0x7ff7f00c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:00.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.397+0000 7ff7ef7fe640 1 --2- 172.21.15.55:0/2118521199 >> v2:172.21.15.55:3300/0 conn(0x7ff7f013dec0 0x7ff7f00c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:00.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.397+0000 7ff7ef7fe640 1 -- 172.21.15.55:0/2118521199 >> v1:172.21.15.55:6789/0 conn(0x7ff7f0067000 legacy=0x7ff7f00c5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:00.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.397+0000 7ff7ef7fe640 1 -- 172.21.15.55:0/2118521199 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f01587e0 con 0x7ff7f0067440 2024-03-19T02:03:00.400 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.401+0000 7ff7ef7fe640 1 --2- 172.21.15.55:0/2118521199 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff7f0067440 0x7ff7f006be30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8009e50 tx=0x7ff7e801f230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcbd8020075b3ea2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:00.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.409+0000 7ff7ee7fc640 1 -- 172.21.15.55:0/2118521199 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e80b4070 con 0x7ff7f0067440 2024-03-19T02:03:00.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.409+0000 7ff7ee7fc640 1 -- 172.21.15.55:0/2118521199 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7e80af400 con 0x7ff7f0067440 2024-03-19T02:03:00.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.409+0000 7ff7ee7fc640 1 -- 172.21.15.55:0/2118521199 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e80af810 con 0x7ff7f0067440 2024-03-19T02:03:00.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.409+0000 7ff7f55e5640 1 -- 172.21.15.55:0/2118521199 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff7f0067440 msgr2=0x7ff7f006be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:00.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.409+0000 7ff7f55e5640 1 --2- 172.21.15.55:0/2118521199 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff7f0067440 0x7ff7f006be30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8009e50 tx=0x7ff7e801f230 comp rx=0 tx=0).stop 2024-03-19T02:03:00.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.409+0000 7ff7f55e5640 1 -- 172.21.15.55:0/2118521199 shutdown_connections 2024-03-19T02:03:00.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.409+0000 7ff7f55e5640 1 --2- 172.21.15.55:0/2118521199 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff7f0067440 0x7ff7f006be30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:00.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.409+0000 7ff7f55e5640 1 --2- 172.21.15.55:0/2118521199 >> v2:172.21.15.55:3300/0 conn(0x7ff7f013dec0 0x7ff7f00c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:00.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.409+0000 7ff7f55e5640 1 -- 172.21.15.55:0/2118521199 >> 172.21.15.55:0/2118521199 conn(0x7ff7f00cc270 msgr2=0x7ff7f00cc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:00.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.409+0000 7ff7f55e5640 1 -- 172.21.15.55:0/2118521199 shutdown_connections 2024-03-19T02:03:00.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.409+0000 7ff7f55e5640 1 -- 172.21.15.55:0/2118521199 wait complete. 2024-03-19T02:03:00.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.409+0000 7ff7f55e5640 1 Processor -- start 2024-03-19T02:03:00.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.409+0000 7ff7f55e5640 1 -- start start 2024-03-19T02:03:00.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.413+0000 7ff7f55e5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7f0067000 0x7ff7f00c44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:00.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.413+0000 7ff7f55e5640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff7f0067440 0x7ff7f013c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:00.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.413+0000 7ff7f55e5640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7f013dec0 0x7ff7f015a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:00.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.413+0000 7ff7f55e5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f013e2a0 con 0x7ff7f0067000 2024-03-19T02:03:00.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.413+0000 7ff7f55e5640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f0158da0 con 0x7ff7f013dec0 2024-03-19T02:03:00.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.413+0000 7ff7f55e5640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff7f00c6090 con 0x7ff7f0067440 2024-03-19T02:03:00.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.413+0000 7ff7eeffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7f0067000 0x7ff7f00c44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:00.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.413+0000 7ff7ef7fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7f013dec0 0x7ff7f015a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:00.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.413+0000 7ff7eeffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7f0067000 0x7ff7f00c44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:44480/0 (socket says 172.21.15.55:44480) 2024-03-19T02:03:00.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.413+0000 7ff7eeffd640 1 -- 172.21.15.55:0/1366212911 learned_addr learned my addr 172.21.15.55:0/1366212911 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:00.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.413+0000 7ff7e7fff640 1 --2- 172.21.15.55:0/1366212911 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff7f0067440 0x7ff7f013c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:00.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.417+0000 7ff7eeffd640 1 -- 172.21.15.55:0/1366212911 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff7f0067440 msgr2=0x7ff7f013c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:00.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.417+0000 7ff7eeffd640 1 --2- 172.21.15.55:0/1366212911 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff7f0067440 0x7ff7f013c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:00.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.417+0000 7ff7eeffd640 1 -- 172.21.15.55:0/1366212911 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7f013dec0 msgr2=0x7ff7f015a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:00.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.417+0000 7ff7eeffd640 1 --2- 172.21.15.55:0/1366212911 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7f013dec0 0x7ff7f015a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:00.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.417+0000 7ff7eeffd640 1 -- 172.21.15.55:0/1366212911 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f00c4ff0 con 0x7ff7f0067000 2024-03-19T02:03:00.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.421+0000 7ff7eeffd640 1 --2- 172.21.15.55:0/1366212911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7f0067000 0x7ff7f00c44a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000e8a0 tx=0x7ff7e000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:00.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.425+0000 7ff7e77fe640 1 -- 172.21.15.55:0/1366212911 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e0017070 con 0x7ff7f0067000 2024-03-19T02:03:00.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.425+0000 7ff7e77fe640 1 -- 172.21.15.55:0/1366212911 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7e000c940 con 0x7ff7f0067000 2024-03-19T02:03:00.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.425+0000 7ff7f55e5640 1 -- 172.21.15.55:0/1366212911 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7f00c4d10 con 0x7ff7f0067000 2024-03-19T02:03:00.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.425+0000 7ff7e77fe640 1 -- 172.21.15.55:0/1366212911 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e000ccf0 con 0x7ff7f0067000 2024-03-19T02:03:00.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.425+0000 7ff7f55e5640 1 -- 172.21.15.55:0/1366212911 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7f01f3c40 con 0x7ff7f0067000 2024-03-19T02:03:00.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.425+0000 7ff7f55e5640 1 -- 172.21.15.55:0/1366212911 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7f0068a40 con 0x7ff7f0067000 2024-03-19T02:03:00.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.445+0000 7ff7e77fe640 1 -- 172.21.15.55:0/1366212911 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ff7e001a070 con 0x7ff7f0067000 2024-03-19T02:03:00.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.445+0000 7ff7e77fe640 1 --2- 172.21.15.55:0/1366212911 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff7d00676b0 0x7ff7d0069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:00.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.445+0000 7ff7e7fff640 1 --2- 172.21.15.55:0/1366212911 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff7d00676b0 0x7ff7d0069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:00.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.449+0000 7ff7e7fff640 1 --2- 172.21.15.55:0/1366212911 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff7d00676b0 0x7ff7d0069b70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0157ec0 tx=0x7ff7d8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:00.735 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.733+0000 7ff7e77fe640 1 -- 172.21.15.55:0/1366212911 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7ff7e0054150 con 0x7ff7f0067000 2024-03-19T02:03:00.735 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.733+0000 7ff7e77fe640 1 -- 172.21.15.55:0/1366212911 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ff7e00560b0 con 0x7ff7f0067000 2024-03-19T02:03:00.984 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:00.985+0000 7ff7f55e5640 1 -- 172.21.15.55:0/1366212911 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff7f00c8b50 con 0x7ff7f0067000 2024-03-19T02:03:01.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:01.005+0000 7ff7e77fe640 1 -- 172.21.15.55:0/1366212911 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7ff7e005a150 con 0x7ff7f0067000 2024-03-19T02:03:01.004 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:01.005 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:01.005 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:01.008 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:01.009+0000 7ff7f55e5640 1 -- 172.21.15.55:0/1366212911 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff7d00676b0 msgr2=0x7ff7d0069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:01.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:01.009+0000 7ff7f55e5640 1 --2- 172.21.15.55:0/1366212911 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff7d00676b0 0x7ff7d0069b70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0157ec0 tx=0x7ff7d8009040 comp rx=0 tx=0).stop 2024-03-19T02:03:01.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:01.009+0000 7ff7f55e5640 1 -- 172.21.15.55:0/1366212911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7f0067000 msgr2=0x7ff7f00c44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:01.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:01.009+0000 7ff7f55e5640 1 --2- 172.21.15.55:0/1366212911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7f0067000 0x7ff7f00c44a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000e8a0 tx=0x7ff7e000eca0 comp rx=0 tx=0).stop 2024-03-19T02:03:01.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:01.009+0000 7ff7f55e5640 1 -- 172.21.15.55:0/1366212911 shutdown_connections 2024-03-19T02:03:01.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:01.009+0000 7ff7f55e5640 1 --2- 172.21.15.55:0/1366212911 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff7d00676b0 0x7ff7d0069b70 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:01.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:01.009+0000 7ff7f55e5640 1 --2- 172.21.15.55:0/1366212911 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7f013dec0 0x7ff7f015a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:01.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:01.009+0000 7ff7f55e5640 1 --2- 172.21.15.55:0/1366212911 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff7f0067440 0x7ff7f013c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:01.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:01.009+0000 7ff7f55e5640 1 --2- 172.21.15.55:0/1366212911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7f0067000 0x7ff7f00c44a0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:01.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:01.009+0000 7ff7f55e5640 1 -- 172.21.15.55:0/1366212911 >> 172.21.15.55:0/1366212911 conn(0x7ff7f00cc270 msgr2=0x7ff7f013cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:01.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:01.009+0000 7ff7f55e5640 1 -- 172.21.15.55:0/1366212911 shutdown_connections 2024-03-19T02:03:01.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:01.009+0000 7ff7f55e5640 1 -- 172.21.15.55:0/1366212911 wait complete. 2024-03-19T02:03:01.368 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:13:13.527 27555== Warning: unimplemented fcntl command: 1036 2024-03-19T02:03:01.377 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:13:13.537 27555== Warning: unimplemented fcntl command: 1036 2024-03-19T02:03:01.479 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:13:13.575 27627== Warning: unimplemented fcntl command: 1036 2024-03-19T02:03:01.483 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:13:13.579 27627== Warning: unimplemented fcntl command: 1036 2024-03-19T02:03:01.484 INFO:tasks.ceph.mon.b.smithi084.stderr:==00:00:13:13.579 27626== Warning: unimplemented fcntl command: 1036 2024-03-19T02:03:01.486 INFO:tasks.ceph.mon.b.smithi084.stderr:==00:00:13:13.582 27626== Warning: unimplemented fcntl command: 1036 2024-03-19T02:03:01.542 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:13:13.702 27555== Warning: unimplemented fcntl command: 1036 2024-03-19T02:03:01.629 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:01.629 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:01.638 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:13:13.734 27627== Warning: unimplemented fcntl command: 1036 2024-03-19T02:03:01.641 INFO:tasks.ceph.mon.b.smithi084.stderr:==00:00:13:13.737 27626== Warning: unimplemented fcntl command: 1036 2024-03-19T02:03:02.630 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:02.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.749+0000 7f40ec96f640 1 Processor -- start 2024-03-19T02:03:02.749 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.749+0000 7f40ec96f640 1 -- start start 2024-03-19T02:03:02.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.749+0000 7f40ec96f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f40e80c52b0 0x7f40e8151d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:02.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.749+0000 7f40ec96f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f40e80c89f0 0x7f40e8152260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:02.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.749+0000 7f40ec96f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f40e80c5690 0x7f40e8158070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:02.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.749+0000 7f40ec96f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f40e80724c0 con 0x7f40e80c5690 2024-03-19T02:03:02.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.749+0000 7f40ec96f640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f40e813d290 con 0x7f40e80c52b0 2024-03-19T02:03:02.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.749+0000 7f40ec96f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f40e813d570 con 0x7f40e80c89f0 2024-03-19T02:03:02.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.749+0000 7f40e6d9d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f40e80c5690 0x7f40e8158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:02.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.753+0000 7f40e5d9b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f40e80c89f0 0x7f40e8152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:02.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.753+0000 7f40e659c640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f40e80c52b0 0x7f40e8151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:02.751 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.753+0000 7f40e6d9d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f40e80c5690 0x7f40e8158070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:44490/0 (socket says 172.21.15.55:44490) 2024-03-19T02:03:02.752 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.753+0000 7f40e6d9d640 1 -- 172.21.15.55:0/247676824 learned_addr learned my addr 172.21.15.55:0/247676824 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:02.754 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.753+0000 7f40e6d9d640 1 -- 172.21.15.55:0/247676824 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f40e80c89f0 msgr2=0x7f40e8152260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:02.754 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.753+0000 7f40e6d9d640 1 --2- 172.21.15.55:0/247676824 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f40e80c89f0 0x7f40e8152260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:02.754 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.753+0000 7f40e6d9d640 1 -- 172.21.15.55:0/247676824 >> v2:172.21.15.84:3300/0 conn(0x7f40e80c52b0 msgr2=0x7f40e8151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:02.754 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.753+0000 7f40e6d9d640 1 --2- 172.21.15.55:0/247676824 >> v2:172.21.15.84:3300/0 conn(0x7f40e80c52b0 0x7f40e8151d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:02.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.753+0000 7f40e6d9d640 1 -- 172.21.15.55:0/247676824 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e813d850 con 0x7f40e80c5690 2024-03-19T02:03:02.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.757+0000 7f40e6d9d640 1 --2- 172.21.15.55:0/247676824 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e80c5690 0x7f40e8158070 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f40e80cca00 tx=0x7f40e001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b117c18858f2878 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:02.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.761+0000 7f40e559a640 1 -- 172.21.15.55:0/247676824 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e00b4070 con 0x7f40e80c5690 2024-03-19T02:03:02.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.761+0000 7f40e559a640 1 -- 172.21.15.55:0/247676824 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40e00af400 con 0x7f40e80c5690 2024-03-19T02:03:02.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.761+0000 7f40e559a640 1 -- 172.21.15.55:0/247676824 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e00af7d0 con 0x7f40e80c5690 2024-03-19T02:03:02.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40ec96f640 1 -- 172.21.15.55:0/247676824 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e80c5690 msgr2=0x7f40e8158070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:02.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40ec96f640 1 --2- 172.21.15.55:0/247676824 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e80c5690 0x7f40e8158070 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f40e80cca00 tx=0x7f40e001f1e0 comp rx=0 tx=0).stop 2024-03-19T02:03:02.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40ec96f640 1 -- 172.21.15.55:0/247676824 shutdown_connections 2024-03-19T02:03:02.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40ec96f640 1 --2- 172.21.15.55:0/247676824 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f40e80c89f0 0x7f40e8152260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:02.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40ec96f640 1 --2- 172.21.15.55:0/247676824 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e80c5690 0x7f40e8158070 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:02.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40ec96f640 1 --2- 172.21.15.55:0/247676824 >> v2:172.21.15.84:3300/0 conn(0x7f40e80c52b0 0x7f40e8151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:02.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40ec96f640 1 -- 172.21.15.55:0/247676824 >> 172.21.15.55:0/247676824 conn(0x7f40e806f3b0 msgr2=0x7f40e806f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:02.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40ec96f640 1 -- 172.21.15.55:0/247676824 shutdown_connections 2024-03-19T02:03:02.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40ec96f640 1 -- 172.21.15.55:0/247676824 wait complete. 2024-03-19T02:03:02.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40ec96f640 1 Processor -- start 2024-03-19T02:03:02.765 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40ec96f640 1 -- start start 2024-03-19T02:03:02.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40ec96f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f40e80c52b0 0x7f40e81ef020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:02.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40ec96f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f40e80c5690 0x7f40e8159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:02.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40ec96f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e80c89f0 0x7f40e81f1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:02.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40ec96f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f40e81585b0 con 0x7f40e80c89f0 2024-03-19T02:03:02.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40ec96f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f40e813de10 con 0x7f40e80c5690 2024-03-19T02:03:02.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40ec96f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f40e8158730 con 0x7f40e80c52b0 2024-03-19T02:03:02.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40e6d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e80c89f0 0x7f40e81f1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:02.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40e5d9b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f40e80c5690 0x7f40e8159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:02.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40e6d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e80c89f0 0x7f40e81f1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:44494/0 (socket says 172.21.15.55:44494) 2024-03-19T02:03:02.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40e6d9d640 1 -- 172.21.15.55:0/1906003188 learned_addr learned my addr 172.21.15.55:0/1906003188 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:02.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.765+0000 7f40e659c640 1 --2- 172.21.15.55:0/1906003188 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f40e80c52b0 0x7f40e81ef020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:02.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.769+0000 7f40e6d9d640 1 -- 172.21.15.55:0/1906003188 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f40e80c52b0 msgr2=0x7f40e81ef020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:02.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.769+0000 7f40e6d9d640 1 --2- 172.21.15.55:0/1906003188 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f40e80c52b0 0x7f40e81ef020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:02.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.769+0000 7f40e6d9d640 1 -- 172.21.15.55:0/1906003188 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f40e80c5690 msgr2=0x7f40e8159130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:02.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.769+0000 7f40e6d9d640 1 --2- 172.21.15.55:0/1906003188 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f40e80c5690 0x7f40e8159130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:02.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.769+0000 7f40e6d9d640 1 -- 172.21.15.55:0/1906003188 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e81f1120 con 0x7f40e80c89f0 2024-03-19T02:03:02.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.773+0000 7f40e6d9d640 1 --2- 172.21.15.55:0/1906003188 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e80c89f0 0x7f40e81f1bc0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f40e00b2040 tx=0x7f40e001bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:02.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.777+0000 7f40d6ffd640 1 -- 172.21.15.55:0/1906003188 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e00b4040 con 0x7f40e80c89f0 2024-03-19T02:03:02.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.777+0000 7f40d6ffd640 1 -- 172.21.15.55:0/1906003188 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40e00c0070 con 0x7f40e80c89f0 2024-03-19T02:03:02.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.777+0000 7f40ec96f640 1 -- 172.21.15.55:0/1906003188 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40e81597e0 con 0x7f40e80c89f0 2024-03-19T02:03:02.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.777+0000 7f40d6ffd640 1 -- 172.21.15.55:0/1906003188 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e0009510 con 0x7f40e80c89f0 2024-03-19T02:03:02.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.777+0000 7f40ec96f640 1 -- 172.21.15.55:0/1906003188 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40e8159da0 con 0x7f40e80c89f0 2024-03-19T02:03:02.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.781+0000 7f40ec96f640 1 -- 172.21.15.55:0/1906003188 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40e8068330 con 0x7f40e80c89f0 2024-03-19T02:03:02.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.801+0000 7f40d6ffd640 1 -- 172.21.15.55:0/1906003188 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f40e00c5460 con 0x7f40e80c89f0 2024-03-19T02:03:02.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.801+0000 7f40d6ffd640 1 --2- 172.21.15.55:0/1906003188 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f40c80676b0 0x7f40c8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:02.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.801+0000 7f40e659c640 1 --2- 172.21.15.55:0/1906003188 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f40c80676b0 0x7f40c8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:02.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:02.801+0000 7f40e659c640 1 --2- 172.21.15.55:0/1906003188 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f40c80676b0 0x7f40c8069b70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f40e81eb7e0 tx=0x7f40d8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:03.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:03.061+0000 7f40d6ffd640 1 -- 172.21.15.55:0/1906003188 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f40e00f2b10 con 0x7f40e80c89f0 2024-03-19T02:03:03.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:03.061+0000 7f40d6ffd640 1 -- 172.21.15.55:0/1906003188 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f40e0018d40 con 0x7f40e80c89f0 2024-03-19T02:03:03.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:03.285+0000 7f40ec96f640 1 -- 172.21.15.55:0/1906003188 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40e813c830 con 0x7f40e80c89f0 2024-03-19T02:03:03.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:03.369+0000 7f40d6ffd640 1 -- 172.21.15.55:0/1906003188 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f40e00f4a70 con 0x7f40e80c89f0 2024-03-19T02:03:03.368 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:03.368 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:03.368 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:03.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:03.369+0000 7f40ec96f640 1 -- 172.21.15.55:0/1906003188 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f40c80676b0 msgr2=0x7f40c8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:03.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:03.369+0000 7f40ec96f640 1 --2- 172.21.15.55:0/1906003188 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f40c80676b0 0x7f40c8069b70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f40e81eb7e0 tx=0x7f40d8009190 comp rx=0 tx=0).stop 2024-03-19T02:03:03.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:03.369+0000 7f40ec96f640 1 -- 172.21.15.55:0/1906003188 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e80c89f0 msgr2=0x7f40e81f1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:03.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:03.369+0000 7f40ec96f640 1 --2- 172.21.15.55:0/1906003188 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e80c89f0 0x7f40e81f1bc0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f40e00b2040 tx=0x7f40e001bf80 comp rx=0 tx=0).stop 2024-03-19T02:03:03.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:03.369+0000 7f40ec96f640 1 -- 172.21.15.55:0/1906003188 shutdown_connections 2024-03-19T02:03:03.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:03.369+0000 7f40ec96f640 1 --2- 172.21.15.55:0/1906003188 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f40c80676b0 0x7f40c8069b70 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:03.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:03.369+0000 7f40ec96f640 1 --2- 172.21.15.55:0/1906003188 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f40e80c89f0 0x7f40e81f1bc0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:03.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:03.369+0000 7f40ec96f640 1 --2- 172.21.15.55:0/1906003188 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f40e80c5690 0x7f40e8159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:03.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:03.369+0000 7f40ec96f640 1 --2- 172.21.15.55:0/1906003188 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f40e80c52b0 0x7f40e81ef020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:03.371 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:03.369+0000 7f40ec96f640 1 -- 172.21.15.55:0/1906003188 >> 172.21.15.55:0/1906003188 conn(0x7f40e806f3b0 msgr2=0x7f40e813ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:03.372 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:03.369+0000 7f40ec96f640 1 -- 172.21.15.55:0/1906003188 shutdown_connections 2024-03-19T02:03:03.372 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:03.369+0000 7f40ec96f640 1 -- 172.21.15.55:0/1906003188 wait complete. 2024-03-19T02:03:03.990 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:03.991 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:04.991 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:05.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.109+0000 7f8fdd016640 1 Processor -- start 2024-03-19T02:03:05.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.109+0000 7f8fdd016640 1 -- start start 2024-03-19T02:03:05.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.109+0000 7f8fdd016640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f8fd806c6c0 0x7f8fd815ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:05.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.109+0000 7f8fdd016640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fd806b340 con 0x7f8fd80c6e80 2024-03-19T02:03:05.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.109+0000 7f8fdd016640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fd806b620 con 0x7f8fd8150c30 2024-03-19T02:03:05.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.109+0000 7f8fdd016640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f8fd806b900 con 0x7f8fd806c6c0 2024-03-19T02:03:05.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.109+0000 7f8fd6d9d640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f8fd8150c30 0x7f8fd806e210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:49356/0 (socket says 172.21.15.55:49356) 2024-03-19T02:03:05.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.109+0000 7f8fd6d9d640 1 -- 172.21.15.55:0/687389427 learned_addr learned my addr 172.21.15.55:0/687389427 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:03:05.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.117+0000 7f8fd5d9b640 1 -- 172.21.15.55:0/687389427 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3912884882 0 0) 0x7f8fd806b620 con 0x7f8fd8150c30 2024-03-19T02:03:05.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.117+0000 7f8fd5d9b640 1 -- 172.21.15.55:0/687389427 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fc0003530 con 0x7f8fd8150c30 2024-03-19T02:03:05.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.121+0000 7f8fd5d9b640 1 -- 172.21.15.55:0/687389427 <== mon.1 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 979185398 0 0) 0x7f8fc0003530 con 0x7f8fd8150c30 2024-03-19T02:03:05.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.121+0000 7f8fd5d9b640 1 -- 172.21.15.55:0/687389427 >> v2:172.21.15.55:3300/0 conn(0x7f8fd806c6c0 msgr2=0x7f8fd815ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:05.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.121+0000 7f8fd5d9b640 1 --2- 172.21.15.55:0/687389427 >> v2:172.21.15.55:3300/0 conn(0x7f8fd806c6c0 0x7f8fd815ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:05.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.121+0000 7f8fd5d9b640 1 -- 172.21.15.55:0/687389427 >> v1:172.21.15.55:6789/0 conn(0x7f8fd80c6e80 legacy=0x7f8fd8162b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:05.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.121+0000 7f8fd5d9b640 1 -- 172.21.15.55:0/687389427 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd806bbe0 con 0x7f8fd8150c30 2024-03-19T02:03:05.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.121+0000 7f8fd5d9b640 1 -- 172.21.15.55:0/687389427 <== mon.1 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f8fcc001680 con 0x7f8fd8150c30 2024-03-19T02:03:05.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.121+0000 7f8fd5d9b640 1 -- 172.21.15.55:0/687389427 >> v1:172.21.15.84:6789/0 conn(0x7f8fd8150c30 legacy=0x7f8fd806e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:05.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.121+0000 7f8fd5d9b640 1 --2- 172.21.15.55:0/687389427 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fc0003a10 0x7f8fc0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:05.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.121+0000 7f8fd5d9b640 1 --2- 172.21.15.55:0/687389427 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8fc0004810 0x7f8fc0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:05.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.121+0000 7f8fd5d9b640 1 --2- 172.21.15.55:0/687389427 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fc00052f0 0x7f8fc000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:05.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.121+0000 7f8fd5d9b640 1 -- 172.21.15.55:0/687389427 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd806b900 con 0x7f8fc0003a10 2024-03-19T02:03:05.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.121+0000 7f8fd5d9b640 1 -- 172.21.15.55:0/687389427 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc000c230 con 0x7f8fc00052f0 2024-03-19T02:03:05.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.121+0000 7f8fd5d9b640 1 -- 172.21.15.55:0/687389427 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f8fc000c430 con 0x7f8fc0004810 2024-03-19T02:03:05.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.121+0000 7f8fd659c640 1 --2- 172.21.15.55:0/687389427 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8fc0004810 0x7f8fc0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:05.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.125+0000 7f8fd6d9d640 1 --2- 172.21.15.55:0/687389427 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fc0003a10 0x7f8fc0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:05.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.125+0000 7f8fd659c640 1 -- 172.21.15.55:0/687389427 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fc00052f0 msgr2=0x7f8fc000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:05.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.125+0000 7f8fd659c640 1 --2- 172.21.15.55:0/687389427 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fc00052f0 0x7f8fc000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:05.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.125+0000 7f8fd659c640 1 -- 172.21.15.55:0/687389427 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fc0003a10 msgr2=0x7f8fc0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:05.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.125+0000 7f8fd659c640 1 --2- 172.21.15.55:0/687389427 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fc0003a10 0x7f8fc0003e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:05.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.125+0000 7f8fd659c640 1 -- 172.21.15.55:0/687389427 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fc0003530 con 0x7f8fc0004810 2024-03-19T02:03:05.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.125+0000 7f8fd759e640 1 -- 172.21.15.55:0/687389427 reap_dead start 2024-03-19T02:03:05.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.129+0000 7f8fd659c640 1 --2- 172.21.15.55:0/687389427 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8fc0004810 0x7f8fc0004c90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8007fd0 tx=0x7f8fc801c2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8379029ced53293 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:05.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.137+0000 7f8fd5d9b640 1 -- 172.21.15.55:0/687389427 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8001250 con 0x7f8fc0004810 2024-03-19T02:03:05.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.137+0000 7f8fd5d9b640 1 -- 172.21.15.55:0/687389427 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fc8005270 con 0x7f8fc0004810 2024-03-19T02:03:05.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.137+0000 7f8fd5d9b640 1 -- 172.21.15.55:0/687389427 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc80054a0 con 0x7f8fc0004810 2024-03-19T02:03:05.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.137+0000 7f8fdd016640 1 -- 172.21.15.55:0/687389427 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8fc0004810 msgr2=0x7f8fc0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:05.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.137+0000 7f8fdd016640 1 --2- 172.21.15.55:0/687389427 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8fc0004810 0x7f8fc0004c90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8007fd0 tx=0x7f8fc801c2c0 comp rx=0 tx=0).stop 2024-03-19T02:03:05.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.137+0000 7f8fdd016640 1 -- 172.21.15.55:0/687389427 shutdown_connections 2024-03-19T02:03:05.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.137+0000 7f8fdd016640 1 --2- 172.21.15.55:0/687389427 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8fc0004810 0x7f8fc0004c90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:05.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.137+0000 7f8fdd016640 1 -- 172.21.15.55:0/687389427 >> 172.21.15.55:0/687389427 conn(0x7f8fd80cc270 msgr2=0x7f8fd80cc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:05.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.137+0000 7f8fdd016640 1 -- 172.21.15.55:0/687389427 shutdown_connections 2024-03-19T02:03:05.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.137+0000 7f8fdd016640 1 -- 172.21.15.55:0/687389427 wait complete. 2024-03-19T02:03:05.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.137+0000 7f8fdd016640 1 Processor -- start 2024-03-19T02:03:05.138 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.137+0000 7f8fdd016640 1 -- start start 2024-03-19T02:03:05.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.141+0000 7f8fdd016640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fc0004810 0x7f8fd81427b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:05.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.141+0000 7f8fdd016640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fd81457d0 0x7f8fd8145c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:05.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.141+0000 7f8fdd016640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8fd813e010 0x7f8fd813e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:05.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.141+0000 7f8fdd016640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd806cad0 con 0x7f8fc0004810 2024-03-19T02:03:05.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.141+0000 7f8fdd016640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd806c1a0 con 0x7f8fd81457d0 2024-03-19T02:03:05.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.141+0000 7f8fdd016640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f8fd8151010 con 0x7f8fd813e010 2024-03-19T02:03:05.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.141+0000 7f8fd6d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fc0004810 0x7f8fd81427b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:05.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.141+0000 7f8fd659c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fd81457d0 0x7f8fd8145c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:05.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.141+0000 7f8fd659c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fd81457d0 0x7f8fd8145c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:48368/0 (socket says 172.21.15.55:48368) 2024-03-19T02:03:05.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.141+0000 7f8fd659c640 1 -- 172.21.15.55:0/4188846458 learned_addr learned my addr 172.21.15.55:0/4188846458 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:05.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.145+0000 7f8fd6d9d640 1 -- 172.21.15.55:0/4188846458 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8fd813e010 msgr2=0x7f8fd813e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:05.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.145+0000 7f8fd6d9d640 1 --2- 172.21.15.55:0/4188846458 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8fd813e010 0x7f8fd813e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:05.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.145+0000 7f8fd6d9d640 1 -- 172.21.15.55:0/4188846458 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fd81457d0 msgr2=0x7f8fd8145c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:05.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.145+0000 7f8fd6d9d640 1 --2- 172.21.15.55:0/4188846458 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fd81457d0 0x7f8fd8145c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:05.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.145+0000 7f8fd6d9d640 1 -- 172.21.15.55:0/4188846458 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd813e9d0 con 0x7f8fc0004810 2024-03-19T02:03:05.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.145+0000 7f8fd6d9d640 1 --2- 172.21.15.55:0/4188846458 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fc0004810 0x7f8fd81427b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc010c60 tx=0x7f8fcc00e4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:05.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.149+0000 7f8fbf7fe640 1 -- 172.21.15.55:0/4188846458 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc01b070 con 0x7f8fc0004810 2024-03-19T02:03:05.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.149+0000 7f8fbf7fe640 1 -- 172.21.15.55:0/4188846458 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fcc011040 con 0x7f8fc0004810 2024-03-19T02:03:05.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.149+0000 7f8fdd016640 1 -- 172.21.15.55:0/4188846458 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fd8140be0 con 0x7f8fc0004810 2024-03-19T02:03:05.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.149+0000 7f8fbf7fe640 1 -- 172.21.15.55:0/4188846458 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc0145c0 con 0x7f8fc0004810 2024-03-19T02:03:05.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.149+0000 7f8fdd016640 1 -- 172.21.15.55:0/4188846458 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fd8141160 con 0x7f8fc0004810 2024-03-19T02:03:05.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.153+0000 7f8fdd016640 1 -- 172.21.15.55:0/4188846458 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fd8142f50 con 0x7f8fc0004810 2024-03-19T02:03:05.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.185+0000 7f8fbf7fe640 1 -- 172.21.15.55:0/4188846458 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f8fcc01e070 con 0x7f8fc0004810 2024-03-19T02:03:05.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.185+0000 7f8fbf7fe640 1 --2- 172.21.15.55:0/4188846458 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8fc4067690 0x7f8fc4069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:05.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.185+0000 7f8fd659c640 1 --2- 172.21.15.55:0/4188846458 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8fc4067690 0x7f8fc4069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:05.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.185+0000 7f8fd659c640 1 --2- 172.21.15.55:0/4188846458 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8fc4067690 0x7f8fc4069b50 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80014a0 tx=0x7f8fc8001030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:05.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.469+0000 7f8fbf7fe640 1 -- 172.21.15.55:0/4188846458 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f8fcc057020 con 0x7f8fc0004810 2024-03-19T02:03:05.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.469+0000 7f8fbf7fe640 1 -- 172.21.15.55:0/4188846458 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f8fcc025020 con 0x7f8fc0004810 2024-03-19T02:03:05.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.689+0000 7f8fdd016640 1 -- 172.21.15.55:0/4188846458 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8fd8068a40 con 0x7f8fc0004810 2024-03-19T02:03:05.766 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.765+0000 7f8fbf7fe640 1 -- 172.21.15.55:0/4188846458 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f8fd813e9d0 con 0x7f8fc0004810 2024-03-19T02:03:05.766 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:05.767 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:05.767 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:05.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.769+0000 7f8fdd016640 1 -- 172.21.15.55:0/4188846458 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8fc4067690 msgr2=0x7f8fc4069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:05.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.769+0000 7f8fdd016640 1 --2- 172.21.15.55:0/4188846458 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8fc4067690 0x7f8fc4069b50 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80014a0 tx=0x7f8fc8001030 comp rx=0 tx=0).stop 2024-03-19T02:03:05.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.769+0000 7f8fdd016640 1 -- 172.21.15.55:0/4188846458 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fc0004810 msgr2=0x7f8fd81427b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:05.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.769+0000 7f8fdd016640 1 --2- 172.21.15.55:0/4188846458 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fc0004810 0x7f8fd81427b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc010c60 tx=0x7f8fcc00e4d0 comp rx=0 tx=0).stop 2024-03-19T02:03:05.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.769+0000 7f8fdd016640 1 -- 172.21.15.55:0/4188846458 shutdown_connections 2024-03-19T02:03:05.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.769+0000 7f8fdd016640 1 --2- 172.21.15.55:0/4188846458 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8fc4067690 0x7f8fc4069b50 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:05.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.769+0000 7f8fdd016640 1 --2- 172.21.15.55:0/4188846458 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8fd813e010 0x7f8fd813e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:05.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.769+0000 7f8fdd016640 1 --2- 172.21.15.55:0/4188846458 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fd81457d0 0x7f8fd8145c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:05.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.769+0000 7f8fdd016640 1 --2- 172.21.15.55:0/4188846458 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8fc0004810 0x7f8fd81427b0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:05.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.769+0000 7f8fdd016640 1 -- 172.21.15.55:0/4188846458 >> 172.21.15.55:0/4188846458 conn(0x7f8fd80cc270 msgr2=0x7f8fd80c85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:05.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.769+0000 7f8fdd016640 1 -- 172.21.15.55:0/4188846458 shutdown_connections 2024-03-19T02:03:05.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:05.769+0000 7f8fdd016640 1 -- 172.21.15.55:0/4188846458 wait complete. 2024-03-19T02:03:06.390 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:06.390 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:07.390 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:07.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.517+0000 7f958b12d640 1 Processor -- start 2024-03-19T02:03:07.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.517+0000 7f958b12d640 1 -- start start 2024-03-19T02:03:07.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.517+0000 7f958b12d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f95840c52b0 0x7f9584151d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:07.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.521+0000 7f958b12d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f9584158780 0x7f9584152260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:07.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.521+0000 7f958b12d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95840724c0 con 0x7f95840c89f0 2024-03-19T02:03:07.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.521+0000 7f958b12d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f958413d290 con 0x7f95840c52b0 2024-03-19T02:03:07.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.521+0000 7f958b12d640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f958413d570 con 0x7f9584158780 2024-03-19T02:03:07.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.521+0000 7f9588ec9640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f95840c52b0 0x7f9584151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:07.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.521+0000 7f9583fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f95840c89f0 0x7f9584158070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:45062/0 (socket says 172.21.15.55:45062) 2024-03-19T02:03:07.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.521+0000 7f9583fff640 1 -- 172.21.15.55:0/938569187 learned_addr learned my addr 172.21.15.55:0/938569187 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:03:07.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.521+0000 7f95896ca640 1 --2- 172.21.15.55:0/938569187 >> v2:172.21.15.84:3300/0 conn(0x7f9584158780 0x7f9584152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:07.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.525+0000 7f9588ec9640 1 -- 172.21.15.55:0/938569187 >> v2:172.21.15.84:3300/0 conn(0x7f9584158780 msgr2=0x7f9584152260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:07.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.525+0000 7f9588ec9640 1 --2- 172.21.15.55:0/938569187 >> v2:172.21.15.84:3300/0 conn(0x7f9584158780 0x7f9584152260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:07.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.525+0000 7f9588ec9640 1 -- 172.21.15.55:0/938569187 >> v1:172.21.15.55:6789/0 conn(0x7f95840c89f0 legacy=0x7f9584158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:07.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.525+0000 7f9588ec9640 1 -- 172.21.15.55:0/938569187 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f958413d850 con 0x7f95840c52b0 2024-03-19T02:03:07.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.529+0000 7f9588ec9640 1 --2- 172.21.15.55:0/938569187 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95840c52b0 0x7f9584151d20 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f95840c94c0 tx=0x7f957801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=315e1242688a04a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:07.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.533+0000 7f95837fe640 1 -- 172.21.15.55:0/938569187 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95780b5070 con 0x7f95840c52b0 2024-03-19T02:03:07.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.533+0000 7f95837fe640 1 -- 172.21.15.55:0/938569187 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95780a8970 con 0x7f95840c52b0 2024-03-19T02:03:07.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.533+0000 7f95837fe640 1 -- 172.21.15.55:0/938569187 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95780a8d40 con 0x7f95840c52b0 2024-03-19T02:03:07.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.537+0000 7f958b12d640 1 -- 172.21.15.55:0/938569187 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95840c52b0 msgr2=0x7f9584151d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:07.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.537+0000 7f958b12d640 1 --2- 172.21.15.55:0/938569187 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95840c52b0 0x7f9584151d20 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f95840c94c0 tx=0x7f957801ec70 comp rx=0 tx=0).stop 2024-03-19T02:03:07.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.537+0000 7f958b12d640 1 -- 172.21.15.55:0/938569187 shutdown_connections 2024-03-19T02:03:07.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.537+0000 7f958b12d640 1 --2- 172.21.15.55:0/938569187 >> v2:172.21.15.84:3300/0 conn(0x7f9584158780 0x7f9584152260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:07.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.537+0000 7f958b12d640 1 --2- 172.21.15.55:0/938569187 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95840c52b0 0x7f9584151d20 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:07.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.537+0000 7f958b12d640 1 -- 172.21.15.55:0/938569187 >> 172.21.15.55:0/938569187 conn(0x7f958406f3b0 msgr2=0x7f958406f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:07.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.537+0000 7f958b12d640 1 -- 172.21.15.55:0/938569187 shutdown_connections 2024-03-19T02:03:07.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.537+0000 7f958b12d640 1 -- 172.21.15.55:0/938569187 wait complete. 2024-03-19T02:03:07.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.537+0000 7f958b12d640 1 Processor -- start 2024-03-19T02:03:07.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.537+0000 7f958b12d640 1 -- start start 2024-03-19T02:03:07.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.537+0000 7f958b12d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95840c52b0 0x7f9584157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:07.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.537+0000 7f958b12d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95840c89f0 0x7f95841f35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:07.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.537+0000 7f958b12d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9584158780 0x7f95841f3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:07.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.537+0000 7f958b12d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f95840c5690 con 0x7f95840c52b0 2024-03-19T02:03:07.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.537+0000 7f958b12d640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f958413de10 con 0x7f9584158780 2024-03-19T02:03:07.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.537+0000 7f958b12d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f95840c5810 con 0x7f95840c89f0 2024-03-19T02:03:07.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.537+0000 7f9588ec9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95840c52b0 0x7f9584157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:07.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.537+0000 7f9588ec9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95840c52b0 0x7f9584157d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:44538/0 (socket says 172.21.15.55:44538) 2024-03-19T02:03:07.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.537+0000 7f9588ec9640 1 -- 172.21.15.55:0/3460424140 learned_addr learned my addr 172.21.15.55:0/3460424140 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:07.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.537+0000 7f9583fff640 1 --2- 172.21.15.55:0/3460424140 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95840c89f0 0x7f95841f35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:07.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.537+0000 7f95896ca640 1 --2- 172.21.15.55:0/3460424140 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9584158780 0x7f95841f3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:07.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.541+0000 7f9588ec9640 1 -- 172.21.15.55:0/3460424140 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95840c89f0 msgr2=0x7f95841f35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:07.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.541+0000 7f9588ec9640 1 --2- 172.21.15.55:0/3460424140 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95840c89f0 0x7f95841f35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:07.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.541+0000 7f9588ec9640 1 -- 172.21.15.55:0/3460424140 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9584158780 msgr2=0x7f95841f3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:07.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.541+0000 7f9588ec9640 1 --2- 172.21.15.55:0/3460424140 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9584158780 0x7f95841f3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:07.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.541+0000 7f9588ec9640 1 -- 172.21.15.55:0/3460424140 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95841f41b0 con 0x7f95840c52b0 2024-03-19T02:03:07.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.545+0000 7f9588ec9640 1 --2- 172.21.15.55:0/3460424140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95840c52b0 0x7f9584157d30 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f95780b3040 tx=0x7f957801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:07.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.549+0000 7f95817fa640 1 -- 172.21.15.55:0/3460424140 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95780bf070 con 0x7f95840c52b0 2024-03-19T02:03:07.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.549+0000 7f95817fa640 1 -- 172.21.15.55:0/3460424140 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95780b0720 con 0x7f95840c52b0 2024-03-19T02:03:07.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.549+0000 7f958b12d640 1 -- 172.21.15.55:0/3460424140 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95840c3980 con 0x7f95840c52b0 2024-03-19T02:03:07.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.549+0000 7f95817fa640 1 -- 172.21.15.55:0/3460424140 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95780b5040 con 0x7f95840c52b0 2024-03-19T02:03:07.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.549+0000 7f958b12d640 1 -- 172.21.15.55:0/3460424140 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95841404a0 con 0x7f95840c52b0 2024-03-19T02:03:07.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.549+0000 7f958b12d640 1 -- 172.21.15.55:0/3460424140 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9584068330 con 0x7f95840c52b0 2024-03-19T02:03:07.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.569+0000 7f95817fa640 1 -- 172.21.15.55:0/3460424140 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f95780c2070 con 0x7f95840c52b0 2024-03-19T02:03:07.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.569+0000 7f95817fa640 1 --2- 172.21.15.55:0/3460424140 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f95680676b0 0x7f9568069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:07.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.569+0000 7f9583fff640 1 --2- 172.21.15.55:0/3460424140 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f95680676b0 0x7f9568069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:07.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.573+0000 7f9583fff640 1 --2- 172.21.15.55:0/3460424140 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f95680676b0 0x7f9568069b70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f95840c1e50 tx=0x7f9570007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:07.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.829+0000 7f95817fa640 1 -- 172.21.15.55:0/3460424140 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f95780f09c0 con 0x7f95840c52b0 2024-03-19T02:03:07.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:07.829+0000 7f95817fa640 1 -- 172.21.15.55:0/3460424140 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f95780c9080 con 0x7f95840c52b0 2024-03-19T02:03:08.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:08.045+0000 7f958b12d640 1 -- 172.21.15.55:0/3460424140 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95840cb3e0 con 0x7f95840c52b0 2024-03-19T02:03:08.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:08.065+0000 7f95817fa640 1 -- 172.21.15.55:0/3460424140 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f95780187d0 con 0x7f95840c52b0 2024-03-19T02:03:08.066 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:08.066 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:08.066 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:08.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:08.069+0000 7f958b12d640 1 -- 172.21.15.55:0/3460424140 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f95680676b0 msgr2=0x7f9568069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:08.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:08.069+0000 7f958b12d640 1 --2- 172.21.15.55:0/3460424140 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f95680676b0 0x7f9568069b70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f95840c1e50 tx=0x7f9570007e90 comp rx=0 tx=0).stop 2024-03-19T02:03:08.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:08.069+0000 7f958b12d640 1 -- 172.21.15.55:0/3460424140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95840c52b0 msgr2=0x7f9584157d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:08.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:08.069+0000 7f958b12d640 1 --2- 172.21.15.55:0/3460424140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95840c52b0 0x7f9584157d30 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f95780b3040 tx=0x7f957801ba10 comp rx=0 tx=0).stop 2024-03-19T02:03:08.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:08.069+0000 7f958b12d640 1 -- 172.21.15.55:0/3460424140 shutdown_connections 2024-03-19T02:03:08.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:08.069+0000 7f958b12d640 1 --2- 172.21.15.55:0/3460424140 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f95680676b0 0x7f9568069b70 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:08.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:08.069+0000 7f958b12d640 1 --2- 172.21.15.55:0/3460424140 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9584158780 0x7f95841f3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:08.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:08.069+0000 7f958b12d640 1 --2- 172.21.15.55:0/3460424140 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f95840c89f0 0x7f95841f35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:08.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:08.069+0000 7f958b12d640 1 --2- 172.21.15.55:0/3460424140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f95840c52b0 0x7f9584157d30 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:08.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:08.069+0000 7f958b12d640 1 -- 172.21.15.55:0/3460424140 >> 172.21.15.55:0/3460424140 conn(0x7f958406f3b0 msgr2=0x7f95840cb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:08.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:08.069+0000 7f958b12d640 1 -- 172.21.15.55:0/3460424140 shutdown_connections 2024-03-19T02:03:08.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:08.069+0000 7f958b12d640 1 -- 172.21.15.55:0/3460424140 wait complete. 2024-03-19T02:03:08.694 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:08.694 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:09.696 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:09.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.809+0000 7f68f730f640 1 Processor -- start 2024-03-19T02:03:09.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.809+0000 7f68f730f640 1 -- start start 2024-03-19T02:03:09.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.809+0000 7f68f730f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f68f00c9000 0x7f68f00c9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:09.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.809+0000 7f68f730f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f68f00c2ad0 0x7f68f00c2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:09.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.809+0000 7f68f730f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f0158d50 con 0x7f68f01520b0 2024-03-19T02:03:09.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.809+0000 7f68f730f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f68f015c4e0 con 0x7f68f00c2ad0 2024-03-19T02:03:09.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.809+0000 7f68f730f640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f68f015c050 con 0x7f68f00c9000 2024-03-19T02:03:09.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.813+0000 7f68f58ac640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f68f00c2ad0 0x7f68f00c2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:09.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.813+0000 7f68f48aa640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f68f00c9000 0x7f68f00c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:09.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.813+0000 7f68f58ac640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f68f00c2ad0 0x7f68f00c2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49200/0 (socket says 172.21.15.55:49200) 2024-03-19T02:03:09.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.813+0000 7f68f58ac640 1 -- 172.21.15.55:0/1760331443 learned_addr learned my addr 172.21.15.55:0/1760331443 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:09.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.813+0000 7f68f58ac640 1 -- 172.21.15.55:0/1760331443 >> v2:172.21.15.84:3300/0 conn(0x7f68f00c9000 msgr2=0x7f68f00c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:09.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.813+0000 7f68f58ac640 1 --2- 172.21.15.55:0/1760331443 >> v2:172.21.15.84:3300/0 conn(0x7f68f00c9000 0x7f68f00c9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:09.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.813+0000 7f68f58ac640 1 -- 172.21.15.55:0/1760331443 >> v1:172.21.15.55:6789/0 conn(0x7f68f01520b0 legacy=0x7f68f00709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:09.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.813+0000 7f68f58ac640 1 -- 172.21.15.55:0/1760331443 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f015b720 con 0x7f68f00c2ad0 2024-03-19T02:03:09.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.817+0000 7f68f58ac640 1 --2- 172.21.15.55:0/1760331443 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f00c2ad0 0x7f68f00c2f00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f68f0072580 tx=0x7f68e801c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20ff204517434817 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:09.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.821+0000 7f68e7fff640 1 -- 172.21.15.55:0/1760331443 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e80a5670 con 0x7f68f00c2ad0 2024-03-19T02:03:09.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.821+0000 7f68e7fff640 1 -- 172.21.15.55:0/1760331443 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68e80a5810 con 0x7f68f00c2ad0 2024-03-19T02:03:09.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.821+0000 7f68e7fff640 1 -- 172.21.15.55:0/1760331443 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e80a5be0 con 0x7f68f00c2ad0 2024-03-19T02:03:09.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.825+0000 7f68f730f640 1 -- 172.21.15.55:0/1760331443 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f00c2ad0 msgr2=0x7f68f00c2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:09.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.825+0000 7f68f730f640 1 --2- 172.21.15.55:0/1760331443 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f00c2ad0 0x7f68f00c2f00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f68f0072580 tx=0x7f68e801c9b0 comp rx=0 tx=0).stop 2024-03-19T02:03:09.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.825+0000 7f68f730f640 1 -- 172.21.15.55:0/1760331443 shutdown_connections 2024-03-19T02:03:09.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.825+0000 7f68f730f640 1 --2- 172.21.15.55:0/1760331443 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f00c2ad0 0x7f68f00c2f00 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:09.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.825+0000 7f68f730f640 1 --2- 172.21.15.55:0/1760331443 >> v2:172.21.15.84:3300/0 conn(0x7f68f00c9000 0x7f68f00c9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:09.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.825+0000 7f68f730f640 1 -- 172.21.15.55:0/1760331443 >> 172.21.15.55:0/1760331443 conn(0x7f68f0067600 msgr2=0x7f68f0071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:09.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.825+0000 7f68f730f640 1 -- 172.21.15.55:0/1760331443 shutdown_connections 2024-03-19T02:03:09.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.825+0000 7f68f730f640 1 -- 172.21.15.55:0/1760331443 wait complete. 2024-03-19T02:03:09.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.825+0000 7f68f730f640 1 Processor -- start 2024-03-19T02:03:09.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.825+0000 7f68f730f640 1 -- start start 2024-03-19T02:03:09.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.825+0000 7f68f730f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68f00c2ad0 0x7f68f015a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:09.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.825+0000 7f68f730f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f00c9000 0x7f68f01511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:09.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.825+0000 7f68f730f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68f01520b0 0x7f68f01f49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:09.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.825+0000 7f68f730f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f68f00c3850 con 0x7f68f00c9000 2024-03-19T02:03:09.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.825+0000 7f68f730f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f68f00710b0 con 0x7f68f01520b0 2024-03-19T02:03:09.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.825+0000 7f68f730f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f68f0152490 con 0x7f68f00c2ad0 2024-03-19T02:03:09.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.829+0000 7f68f48aa640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f00c9000 0x7f68f01511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:09.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.829+0000 7f68f58ac640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68f01520b0 0x7f68f01f49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:09.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.829+0000 7f68f50ab640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68f00c2ad0 0x7f68f015a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:09.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.829+0000 7f68f48aa640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f00c9000 0x7f68f01511c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49210/0 (socket says 172.21.15.55:49210) 2024-03-19T02:03:09.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.829+0000 7f68f48aa640 1 -- 172.21.15.55:0/3488438658 learned_addr learned my addr 172.21.15.55:0/3488438658 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:09.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.833+0000 7f68f48aa640 1 -- 172.21.15.55:0/3488438658 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68f00c2ad0 msgr2=0x7f68f015a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:09.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.833+0000 7f68f48aa640 1 --2- 172.21.15.55:0/3488438658 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68f00c2ad0 0x7f68f015a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:09.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.833+0000 7f68f48aa640 1 -- 172.21.15.55:0/3488438658 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68f01520b0 msgr2=0x7f68f01f49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:09.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.833+0000 7f68f48aa640 1 --2- 172.21.15.55:0/3488438658 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68f01520b0 0x7f68f01f49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:09.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.833+0000 7f68f48aa640 1 -- 172.21.15.55:0/3488438658 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f00c5050 con 0x7f68f00c9000 2024-03-19T02:03:09.834 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.833+0000 7f68f48aa640 1 --2- 172.21.15.55:0/3488438658 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f00c9000 0x7f68f01511c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f68e000e960 tx=0x7f68e000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:09.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.837+0000 7f68e5ffb640 1 -- 172.21.15.55:0/3488438658 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e0017070 con 0x7f68f00c9000 2024-03-19T02:03:09.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.837+0000 7f68e5ffb640 1 -- 172.21.15.55:0/3488438658 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68e000ca00 con 0x7f68f00c9000 2024-03-19T02:03:09.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.837+0000 7f68f730f640 1 -- 172.21.15.55:0/3488438658 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68f00cd760 con 0x7f68f00c9000 2024-03-19T02:03:09.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.837+0000 7f68e5ffb640 1 -- 172.21.15.55:0/3488438658 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e000cdb0 con 0x7f68f00c9000 2024-03-19T02:03:09.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.837+0000 7f68f730f640 1 -- 172.21.15.55:0/3488438658 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68f014c600 con 0x7f68f00c9000 2024-03-19T02:03:09.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.837+0000 7f68f730f640 1 -- 172.21.15.55:0/3488438658 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68f0068a40 con 0x7f68f00c9000 2024-03-19T02:03:09.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.857+0000 7f68e5ffb640 1 -- 172.21.15.55:0/3488438658 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f68e001a070 con 0x7f68f00c9000 2024-03-19T02:03:09.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.857+0000 7f68e5ffb640 1 --2- 172.21.15.55:0/3488438658 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68d40676b0 0x7f68d4069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:09.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.857+0000 7f68f50ab640 1 --2- 172.21.15.55:0/3488438658 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68d40676b0 0x7f68d4069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:09.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.861+0000 7f68f50ab640 1 --2- 172.21.15.55:0/3488438658 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68d40676b0 0x7f68d4069b70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f68ec0098e0 tx=0x7f68ec009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:09.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:09.861+0000 7f68e5ffb640 1 -- 172.21.15.55:0/3488438658 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f68e00546c0 con 0x7f68f00c9000 2024-03-19T02:03:10.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:10.141+0000 7f68e5ffb640 1 -- 172.21.15.55:0/3488438658 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f68e0056620 con 0x7f68f00c9000 2024-03-19T02:03:10.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:10.337+0000 7f68f730f640 1 -- 172.21.15.55:0/3488438658 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68f0074a90 con 0x7f68f00c9000 2024-03-19T02:03:10.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:10.357+0000 7f68e5ffb640 1 -- 172.21.15.55:0/3488438658 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f68e005a6c0 con 0x7f68f00c9000 2024-03-19T02:03:10.362 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:10.362 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:10.363 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:10.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:10.361+0000 7f68f730f640 1 -- 172.21.15.55:0/3488438658 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68d40676b0 msgr2=0x7f68d4069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:10.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:10.361+0000 7f68f730f640 1 --2- 172.21.15.55:0/3488438658 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68d40676b0 0x7f68d4069b70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f68ec0098e0 tx=0x7f68ec009060 comp rx=0 tx=0).stop 2024-03-19T02:03:10.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:10.361+0000 7f68f730f640 1 -- 172.21.15.55:0/3488438658 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f00c9000 msgr2=0x7f68f01511c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:10.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:10.361+0000 7f68f730f640 1 --2- 172.21.15.55:0/3488438658 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f00c9000 0x7f68f01511c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f68e000e960 tx=0x7f68e000ee20 comp rx=0 tx=0).stop 2024-03-19T02:03:10.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:10.361+0000 7f68f730f640 1 -- 172.21.15.55:0/3488438658 shutdown_connections 2024-03-19T02:03:10.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:10.361+0000 7f68f730f640 1 --2- 172.21.15.55:0/3488438658 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68d40676b0 0x7f68d4069b70 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:10.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:10.361+0000 7f68f730f640 1 --2- 172.21.15.55:0/3488438658 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68f01520b0 0x7f68f01f49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:10.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:10.361+0000 7f68f730f640 1 --2- 172.21.15.55:0/3488438658 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f00c9000 0x7f68f01511c0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:10.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:10.361+0000 7f68f730f640 1 --2- 172.21.15.55:0/3488438658 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68f00c2ad0 0x7f68f015a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:10.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:10.361+0000 7f68f730f640 1 -- 172.21.15.55:0/3488438658 >> 172.21.15.55:0/3488438658 conn(0x7f68f0067600 msgr2=0x7f68f0074290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:10.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:10.361+0000 7f68f730f640 1 -- 172.21.15.55:0/3488438658 shutdown_connections 2024-03-19T02:03:10.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:10.361+0000 7f68f730f640 1 -- 172.21.15.55:0/3488438658 wait complete. 2024-03-19T02:03:10.618 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:03:10.621 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:03:10.989 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:10.989 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:11.989 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:12.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.101+0000 7f295d1d5640 1 Processor -- start 2024-03-19T02:03:12.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.101+0000 7f295d1d5640 1 -- start start 2024-03-19T02:03:12.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.101+0000 7f295d1d5640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f29580c9780 0x7f29580c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:12.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.101+0000 7f295d1d5640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29580c8f70 con 0x7f295813dd00 2024-03-19T02:03:12.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.101+0000 7f295d1d5640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29580c9250 con 0x7f295815b550 2024-03-19T02:03:12.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.101+0000 7f295d1d5640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f29581582b0 con 0x7f29580c9780 2024-03-19T02:03:12.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.105+0000 7f2956d9d640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f295813dd00 0x7f29580c8740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:33970/0 (socket says 172.21.15.55:33970) 2024-03-19T02:03:12.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.105+0000 7f2956d9d640 1 -- 172.21.15.55:0/3448254700 learned_addr learned my addr 172.21.15.55:0/3448254700 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:03:12.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.105+0000 7f295659c640 1 --2- 172.21.15.55:0/3448254700 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f29580c9780 0x7f29580c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:12.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.105+0000 7f2955d9b640 1 -- 172.21.15.55:0/3448254700 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4018029645 0 0) 0x7f29580c8f70 con 0x7f295813dd00 2024-03-19T02:03:12.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.105+0000 7f2955d9b640 1 -- 172.21.15.55:0/3448254700 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2940003530 con 0x7f295813dd00 2024-03-19T02:03:12.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.109+0000 7f2955d9b640 1 -- 172.21.15.55:0/3448254700 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1660961546 0 0) 0x7f29580c9250 con 0x7f295815b550 2024-03-19T02:03:12.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.109+0000 7f2955d9b640 1 -- 172.21.15.55:0/3448254700 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29580c8f70 con 0x7f295815b550 2024-03-19T02:03:12.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.109+0000 7f2955d9b640 1 -- 172.21.15.55:0/3448254700 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1490106402 0 0) 0x7f2940003530 con 0x7f295813dd00 2024-03-19T02:03:12.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.109+0000 7f2955d9b640 1 -- 172.21.15.55:0/3448254700 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f29580c9780 msgr2=0x7f29580c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:12.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.109+0000 7f2955d9b640 1 --2- 172.21.15.55:0/3448254700 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f29580c9780 0x7f29580c7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:12.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.109+0000 7f2955d9b640 1 -- 172.21.15.55:0/3448254700 >> v1:172.21.15.84:6789/0 conn(0x7f295815b550 legacy=0x7f29580c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:12.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.109+0000 7f2955d9b640 1 -- 172.21.15.55:0/3448254700 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2958158590 con 0x7f295813dd00 2024-03-19T02:03:12.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.109+0000 7f2955d9b640 1 -- 172.21.15.55:0/3448254700 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7f2948001680 con 0x7f295813dd00 2024-03-19T02:03:12.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.109+0000 7f2955d9b640 1 -- 172.21.15.55:0/3448254700 >> v1:172.21.15.55:6789/0 conn(0x7f295813dd00 legacy=0x7f29580c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:12.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.109+0000 7f2955d9b640 1 --2- 172.21.15.55:0/3448254700 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2940003b80 0x7f2940003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:12.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.109+0000 7f2955d9b640 1 --2- 172.21.15.55:0/3448254700 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f29580c9780 0x7f2940004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:12.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.109+0000 7f2955d9b640 1 --2- 172.21.15.55:0/3448254700 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2940004d50 0x7f29400051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:12.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.109+0000 7f2955d9b640 1 -- 172.21.15.55:0/3448254700 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f29581582b0 con 0x7f2940003b80 2024-03-19T02:03:12.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.109+0000 7f2955d9b640 1 -- 172.21.15.55:0/3448254700 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f294000ba80 con 0x7f2940004d50 2024-03-19T02:03:12.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.109+0000 7f2955d9b640 1 -- 172.21.15.55:0/3448254700 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f294000bc80 con 0x7f29580c9780 2024-03-19T02:03:12.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.113+0000 7f2956d9d640 1 --2- 172.21.15.55:0/3448254700 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2940003b80 0x7f2940003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:12.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.117+0000 7f2956d9d640 1 -- 172.21.15.55:0/3448254700 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f29580c9780 msgr2=0x7f2940004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:12.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.117+0000 7f2956d9d640 1 --2- 172.21.15.55:0/3448254700 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f29580c9780 0x7f2940004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:12.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.117+0000 7f2956d9d640 1 -- 172.21.15.55:0/3448254700 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2940004d50 msgr2=0x7f29400051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:12.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.117+0000 7f2956d9d640 1 --2- 172.21.15.55:0/3448254700 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2940004d50 0x7f29400051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:12.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.117+0000 7f2956d9d640 1 -- 172.21.15.55:0/3448254700 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2958158590 con 0x7f2940003b80 2024-03-19T02:03:12.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.117+0000 7f2956d9d640 1 --2- 172.21.15.55:0/3448254700 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2940003b80 0x7f2940003f80 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f294800c290 tx=0x7f294801cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbbd837d5f149c41 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:12.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.125+0000 7f2955d9b640 1 -- 172.21.15.55:0/3448254700 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2948007d40 con 0x7f2940003b80 2024-03-19T02:03:12.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.125+0000 7f2955d9b640 1 -- 172.21.15.55:0/3448254700 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29480a9070 con 0x7f2940003b80 2024-03-19T02:03:12.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.125+0000 7f2955d9b640 1 -- 172.21.15.55:0/3448254700 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2948005d40 con 0x7f2940003b80 2024-03-19T02:03:12.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.125+0000 7f295d1d5640 1 -- 172.21.15.55:0/3448254700 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2940003b80 msgr2=0x7f2940003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:12.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.125+0000 7f295d1d5640 1 --2- 172.21.15.55:0/3448254700 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2940003b80 0x7f2940003f80 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f294800c290 tx=0x7f294801cc80 comp rx=0 tx=0).stop 2024-03-19T02:03:12.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.125+0000 7f295759e640 1 -- 172.21.15.55:0/3448254700 reap_dead start 2024-03-19T02:03:12.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.125+0000 7f295d1d5640 1 -- 172.21.15.55:0/3448254700 shutdown_connections 2024-03-19T02:03:12.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.125+0000 7f295d1d5640 1 -- 172.21.15.55:0/3448254700 >> 172.21.15.55:0/3448254700 conn(0x7f295806e5a0 msgr2=0x7f2958072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:12.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.125+0000 7f295d1d5640 1 -- 172.21.15.55:0/3448254700 shutdown_connections 2024-03-19T02:03:12.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.125+0000 7f295d1d5640 1 -- 172.21.15.55:0/3448254700 wait complete. 2024-03-19T02:03:12.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.125+0000 7f295d1d5640 1 Processor -- start 2024-03-19T02:03:12.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.125+0000 7f295d1d5640 1 -- start start 2024-03-19T02:03:12.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.129+0000 7f295d1d5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2940003b80 0x7f2958159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:12.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.129+0000 7f295d1d5640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f29581364e0 0x7f295813ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:12.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.129+0000 7f295d1d5640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2958134cc0 0x7f2958135140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:12.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.129+0000 7f295d1d5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f29580c5fc0 con 0x7f2940003b80 2024-03-19T02:03:12.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.129+0000 7f295d1d5640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f29580c5e40 con 0x7f2958134cc0 2024-03-19T02:03:12.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.129+0000 7f295d1d5640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2958164100 con 0x7f29581364e0 2024-03-19T02:03:12.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.129+0000 7f295759e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2958134cc0 0x7f2958135140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:12.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.129+0000 7f2956d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2940003b80 0x7f2958159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:12.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.129+0000 7f2956d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2940003b80 0x7f2958159120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49222/0 (socket says 172.21.15.55:49222) 2024-03-19T02:03:12.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.129+0000 7f2956d9d640 1 -- 172.21.15.55:0/627879342 learned_addr learned my addr 172.21.15.55:0/627879342 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:12.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.129+0000 7f295659c640 1 --2- 172.21.15.55:0/627879342 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f29581364e0 0x7f295813ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:12.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.133+0000 7f2956d9d640 1 -- 172.21.15.55:0/627879342 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f29581364e0 msgr2=0x7f295813ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:12.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.133+0000 7f2956d9d640 1 --2- 172.21.15.55:0/627879342 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f29581364e0 0x7f295813ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:12.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.133+0000 7f2956d9d640 1 -- 172.21.15.55:0/627879342 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2958134cc0 msgr2=0x7f2958135140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:12.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.133+0000 7f2956d9d640 1 --2- 172.21.15.55:0/627879342 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2958134cc0 0x7f2958135140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:12.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.133+0000 7f2956d9d640 1 -- 172.21.15.55:0/627879342 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2958136ab0 con 0x7f2940003b80 2024-03-19T02:03:12.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.137+0000 7f2956d9d640 1 --2- 172.21.15.55:0/627879342 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2940003b80 0x7f2958159120 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f2948098590 tx=0x7f2940005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:12.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.141+0000 7f293f7fe640 1 -- 172.21.15.55:0/627879342 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2948007d40 con 0x7f2940003b80 2024-03-19T02:03:12.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.141+0000 7f293f7fe640 1 -- 172.21.15.55:0/627879342 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2948001480 con 0x7f2940003b80 2024-03-19T02:03:12.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.141+0000 7f295d1d5640 1 -- 172.21.15.55:0/627879342 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29581f45b0 con 0x7f2940003b80 2024-03-19T02:03:12.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.141+0000 7f293f7fe640 1 -- 172.21.15.55:0/627879342 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29480a6e60 con 0x7f2940003b80 2024-03-19T02:03:12.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.141+0000 7f295d1d5640 1 -- 172.21.15.55:0/627879342 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29581f4b70 con 0x7f2940003b80 2024-03-19T02:03:12.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.141+0000 7f295d1d5640 1 -- 172.21.15.55:0/627879342 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2958068a40 con 0x7f2940003b80 2024-03-19T02:03:12.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.173+0000 7f293f7fe640 1 -- 172.21.15.55:0/627879342 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f29480b7030 con 0x7f2940003b80 2024-03-19T02:03:12.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.173+0000 7f293f7fe640 1 --2- 172.21.15.55:0/627879342 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f29440676e0 0x7f2944069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:12.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.173+0000 7f295659c640 1 --2- 172.21.15.55:0/627879342 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f29440676e0 0x7f2944069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:12.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.177+0000 7f295659c640 1 --2- 172.21.15.55:0/627879342 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f29440676e0 0x7f2944069ba0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f2958157f30 tx=0x7f294c0094c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:12.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.481+0000 7f293f7fe640 1 -- 172.21.15.55:0/627879342 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f29480ed660 con 0x7f2940003b80 2024-03-19T02:03:12.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.481+0000 7f293f7fe640 1 -- 172.21.15.55:0/627879342 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2948016990 con 0x7f2940003b80 2024-03-19T02:03:12.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.705+0000 7f295d1d5640 1 -- 172.21.15.55:0/627879342 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29580c9560 con 0x7f2940003b80 2024-03-19T02:03:12.722 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.721+0000 7f293f7fe640 1 -- 172.21.15.55:0/627879342 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f29480ef5c0 con 0x7f2940003b80 2024-03-19T02:03:12.723 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:12.723 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:12.723 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:12.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.725+0000 7f295d1d5640 1 -- 172.21.15.55:0/627879342 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f29440676e0 msgr2=0x7f2944069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:12.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.725+0000 7f295d1d5640 1 --2- 172.21.15.55:0/627879342 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f29440676e0 0x7f2944069ba0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f2958157f30 tx=0x7f294c0094c0 comp rx=0 tx=0).stop 2024-03-19T02:03:12.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.725+0000 7f295d1d5640 1 -- 172.21.15.55:0/627879342 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2940003b80 msgr2=0x7f2958159120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:12.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.725+0000 7f295d1d5640 1 --2- 172.21.15.55:0/627879342 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2940003b80 0x7f2958159120 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f2948098590 tx=0x7f2940005840 comp rx=0 tx=0).stop 2024-03-19T02:03:12.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.725+0000 7f295d1d5640 1 -- 172.21.15.55:0/627879342 shutdown_connections 2024-03-19T02:03:12.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.725+0000 7f295d1d5640 1 --2- 172.21.15.55:0/627879342 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f29440676e0 0x7f2944069ba0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:12.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.725+0000 7f295d1d5640 1 --2- 172.21.15.55:0/627879342 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2958134cc0 0x7f2958135140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:12.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.725+0000 7f295d1d5640 1 --2- 172.21.15.55:0/627879342 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f29581364e0 0x7f295813ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:12.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.725+0000 7f295d1d5640 1 --2- 172.21.15.55:0/627879342 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2940003b80 0x7f2958159120 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:12.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.725+0000 7f295d1d5640 1 -- 172.21.15.55:0/627879342 >> 172.21.15.55:0/627879342 conn(0x7f295806e5a0 msgr2=0x7f295813c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:12.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.725+0000 7f295d1d5640 1 -- 172.21.15.55:0/627879342 shutdown_connections 2024-03-19T02:03:12.726 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:12.725+0000 7f295d1d5640 1 -- 172.21.15.55:0/627879342 wait complete. 2024-03-19T02:03:13.345 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:13.345 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:14.346 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:14.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.465+0000 7f67d49f9640 1 Processor -- start 2024-03-19T02:03:14.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.465+0000 7f67d49f9640 1 -- start start 2024-03-19T02:03:14.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.465+0000 7f67d49f9640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f67d015acb0 0x7f67d0073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:14.466 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.465+0000 7f67d49f9640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67d00c7fa0 con 0x7f67d006b2a0 2024-03-19T02:03:14.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.465+0000 7f67d49f9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67d00c8240 con 0x7f67d01544f0 2024-03-19T02:03:14.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.465+0000 7f67d49f9640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f67d015b210 con 0x7f67d015acb0 2024-03-19T02:03:14.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.469+0000 7f67cdd9b640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f67d006b2a0 0x7f67d00cce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:33984/0 (socket says 172.21.15.55:33984) 2024-03-19T02:03:14.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.469+0000 7f67cdd9b640 1 -- 172.21.15.55:0/446590798 learned_addr learned my addr 172.21.15.55:0/446590798 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:03:14.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.469+0000 7f67ced9d640 1 --2- 172.21.15.55:0/446590798 >> v2:172.21.15.84:3300/0 conn(0x7f67d015acb0 0x7f67d0073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:14.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.469+0000 7f67cd59a640 1 -- 172.21.15.55:0/446590798 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 455713563 0 0) 0x7f67d00c7fa0 con 0x7f67d006b2a0 2024-03-19T02:03:14.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.469+0000 7f67cd59a640 1 -- 172.21.15.55:0/446590798 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67b8003530 con 0x7f67d006b2a0 2024-03-19T02:03:14.472 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.473+0000 7f67ced9d640 1 -- 172.21.15.55:0/446590798 >> v1:172.21.15.84:6789/0 conn(0x7f67d01544f0 legacy=0x7f67d015e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:14.472 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.473+0000 7f67ced9d640 1 -- 172.21.15.55:0/446590798 >> v1:172.21.15.55:6789/0 conn(0x7f67d006b2a0 legacy=0x7f67d00cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:14.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.473+0000 7f67ced9d640 1 -- 172.21.15.55:0/446590798 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d015d920 con 0x7f67d015acb0 2024-03-19T02:03:14.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.477+0000 7f67ced9d640 1 --2- 172.21.15.55:0/446590798 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f67d015acb0 0x7f67d0073540 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f67c8009d90 tx=0x7f67c801f120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cccef7e51ccadea server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:14.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.481+0000 7f67cd59a640 1 -- 172.21.15.55:0/446590798 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c80b4070 con 0x7f67d015acb0 2024-03-19T02:03:14.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.481+0000 7f67cd59a640 1 -- 172.21.15.55:0/446590798 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67c80af600 con 0x7f67d015acb0 2024-03-19T02:03:14.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.481+0000 7f67cd59a640 1 -- 172.21.15.55:0/446590798 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c80af9d0 con 0x7f67d015acb0 2024-03-19T02:03:14.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.481+0000 7f67d49f9640 1 -- 172.21.15.55:0/446590798 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f67d015acb0 msgr2=0x7f67d0073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:14.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.481+0000 7f67d49f9640 1 --2- 172.21.15.55:0/446590798 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f67d015acb0 0x7f67d0073540 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f67c8009d90 tx=0x7f67c801f120 comp rx=0 tx=0).stop 2024-03-19T02:03:14.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.481+0000 7f67d49f9640 1 -- 172.21.15.55:0/446590798 shutdown_connections 2024-03-19T02:03:14.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.481+0000 7f67d49f9640 1 --2- 172.21.15.55:0/446590798 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f67d015acb0 0x7f67d0073540 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:14.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.481+0000 7f67d49f9640 1 -- 172.21.15.55:0/446590798 >> 172.21.15.55:0/446590798 conn(0x7f67d006e670 msgr2=0x7f67d006ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:14.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.481+0000 7f67d49f9640 1 -- 172.21.15.55:0/446590798 shutdown_connections 2024-03-19T02:03:14.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.481+0000 7f67d49f9640 1 -- 172.21.15.55:0/446590798 wait complete. 2024-03-19T02:03:14.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.485+0000 7f67d49f9640 1 Processor -- start 2024-03-19T02:03:14.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.485+0000 7f67d49f9640 1 -- start start 2024-03-19T02:03:14.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.485+0000 7f67d49f9640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f67d006b2a0 0x7f67d014dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:14.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.485+0000 7f67d49f9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67d01544f0 0x7f67d00c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:14.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.485+0000 7f67d49f9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f67d015acb0 0x7f67d00c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:14.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.485+0000 7f67d49f9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f67d00cd540 con 0x7f67d01544f0 2024-03-19T02:03:14.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.485+0000 7f67d49f9640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f67d01548d0 con 0x7f67d006b2a0 2024-03-19T02:03:14.486 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.485+0000 7f67d49f9640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f67d00cd6c0 con 0x7f67d015acb0 2024-03-19T02:03:14.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.485+0000 7f67cdd9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67d01544f0 0x7f67d00c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:14.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.485+0000 7f67cdd9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67d01544f0 0x7f67d00c9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49236/0 (socket says 172.21.15.55:49236) 2024-03-19T02:03:14.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.485+0000 7f67cdd9b640 1 -- 172.21.15.55:0/2745547264 learned_addr learned my addr 172.21.15.55:0/2745547264 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:14.487 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.489+0000 7f67ced9d640 1 --2- 172.21.15.55:0/2745547264 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f67d015acb0 0x7f67d00c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:14.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.489+0000 7f67ce59c640 1 --2- 172.21.15.55:0/2745547264 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f67d006b2a0 0x7f67d014dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:14.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.489+0000 7f67cdd9b640 1 -- 172.21.15.55:0/2745547264 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f67d015acb0 msgr2=0x7f67d00c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:14.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.489+0000 7f67cdd9b640 1 --2- 172.21.15.55:0/2745547264 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f67d015acb0 0x7f67d00c9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:14.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.489+0000 7f67cdd9b640 1 -- 172.21.15.55:0/2745547264 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f67d006b2a0 msgr2=0x7f67d014dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:14.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.489+0000 7f67cdd9b640 1 --2- 172.21.15.55:0/2745547264 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f67d006b2a0 0x7f67d014dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:14.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.489+0000 7f67cdd9b640 1 -- 172.21.15.55:0/2745547264 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d014e2d0 con 0x7f67d01544f0 2024-03-19T02:03:14.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.493+0000 7f67cdd9b640 1 --2- 172.21.15.55:0/2745547264 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67d01544f0 0x7f67d00c9aa0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f67c400c460 tx=0x7f67c400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:14.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.497+0000 7f67beffd640 1 -- 172.21.15.55:0/2745547264 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c4017070 con 0x7f67d01544f0 2024-03-19T02:03:14.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.497+0000 7f67beffd640 1 -- 172.21.15.55:0/2745547264 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67c4005d10 con 0x7f67d01544f0 2024-03-19T02:03:14.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.497+0000 7f67beffd640 1 -- 172.21.15.55:0/2745547264 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c4007310 con 0x7f67d01544f0 2024-03-19T02:03:14.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.497+0000 7f67d49f9640 1 -- 172.21.15.55:0/2745547264 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67d0153380 con 0x7f67d01544f0 2024-03-19T02:03:14.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.497+0000 7f67d49f9640 1 -- 172.21.15.55:0/2745547264 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67d01f3b40 con 0x7f67d01544f0 2024-03-19T02:03:14.499 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.497+0000 7f67d49f9640 1 -- 172.21.15.55:0/2745547264 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67d0068a40 con 0x7f67d01544f0 2024-03-19T02:03:14.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.533+0000 7f67beffd640 1 -- 172.21.15.55:0/2745547264 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f67c401a070 con 0x7f67d01544f0 2024-03-19T02:03:14.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.533+0000 7f67beffd640 1 --2- 172.21.15.55:0/2745547264 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f67b00676b0 0x7f67b0069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:14.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.533+0000 7f67ce59c640 1 --2- 172.21.15.55:0/2745547264 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f67b00676b0 0x7f67b0069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:14.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.533+0000 7f67ce59c640 1 --2- 172.21.15.55:0/2745547264 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f67b00676b0 0x7f67b0069b70 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f67c0007fd0 tx=0x7f67c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:14.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.805+0000 7f67beffd640 1 -- 172.21.15.55:0/2745547264 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f67c40537c0 con 0x7f67d01544f0 2024-03-19T02:03:14.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:14.805+0000 7f67beffd640 1 -- 172.21.15.55:0/2745547264 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f67c4021020 con 0x7f67d01544f0 2024-03-19T02:03:15.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:15.041+0000 7f67d49f9640 1 -- 172.21.15.55:0/2745547264 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67d00c7990 con 0x7f67d01544f0 2024-03-19T02:03:15.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:15.061+0000 7f67beffd640 1 -- 172.21.15.55:0/2745547264 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f67c4055720 con 0x7f67d01544f0 2024-03-19T02:03:15.063 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:15.063 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:15.063 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:15.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:15.065+0000 7f67d49f9640 1 -- 172.21.15.55:0/2745547264 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f67b00676b0 msgr2=0x7f67b0069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:15.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:15.065+0000 7f67d49f9640 1 --2- 172.21.15.55:0/2745547264 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f67b00676b0 0x7f67b0069b70 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f67c0007fd0 tx=0x7f67c0009040 comp rx=0 tx=0).stop 2024-03-19T02:03:15.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:15.065+0000 7f67d49f9640 1 -- 172.21.15.55:0/2745547264 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67d01544f0 msgr2=0x7f67d00c9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:15.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:15.065+0000 7f67d49f9640 1 --2- 172.21.15.55:0/2745547264 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67d01544f0 0x7f67d00c9aa0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f67c400c460 tx=0x7f67c400c920 comp rx=0 tx=0).stop 2024-03-19T02:03:15.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:15.065+0000 7f67d49f9640 1 -- 172.21.15.55:0/2745547264 shutdown_connections 2024-03-19T02:03:15.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:15.065+0000 7f67d49f9640 1 --2- 172.21.15.55:0/2745547264 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f67b00676b0 0x7f67b0069b70 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:15.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:15.065+0000 7f67d49f9640 1 --2- 172.21.15.55:0/2745547264 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f67d015acb0 0x7f67d00c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:15.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:15.065+0000 7f67d49f9640 1 --2- 172.21.15.55:0/2745547264 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f67d01544f0 0x7f67d00c9aa0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:15.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:15.065+0000 7f67d49f9640 1 --2- 172.21.15.55:0/2745547264 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f67d006b2a0 0x7f67d014dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:15.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:15.065+0000 7f67d49f9640 1 -- 172.21.15.55:0/2745547264 >> 172.21.15.55:0/2745547264 conn(0x7f67d006e670 msgr2=0x7f67d00701b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:15.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:15.065+0000 7f67d49f9640 1 -- 172.21.15.55:0/2745547264 shutdown_connections 2024-03-19T02:03:15.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:15.065+0000 7f67d49f9640 1 -- 172.21.15.55:0/2745547264 wait complete. 2024-03-19T02:03:15.685 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:15.685 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:16.686 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:16.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.805+0000 7ff53d89e640 1 Processor -- start 2024-03-19T02:03:16.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.805+0000 7ff53d89e640 1 -- start start 2024-03-19T02:03:16.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.805+0000 7ff53d89e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff538070c80 0x7ff53815b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:16.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.805+0000 7ff53d89e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff53815be10 0x7ff5380ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:16.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.805+0000 7ff53d89e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff53815c2a0 con 0x7ff5380708a0 2024-03-19T02:03:16.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.805+0000 7ff53d89e640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7ff5380ca3f0 con 0x7ff538070c80 2024-03-19T02:03:16.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.805+0000 7ff53d89e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff5380c76a0 con 0x7ff53815be10 2024-03-19T02:03:16.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.809+0000 7ff5377fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff53815be10 0x7ff5380ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:16.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.809+0000 7ff536ffd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7ff5380708a0 0x7ff538151e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:42800/0 (socket says 172.21.15.55:42800) 2024-03-19T02:03:16.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.809+0000 7ff536ffd640 1 -- 172.21.15.55:0/3215757512 learned_addr learned my addr 172.21.15.55:0/3215757512 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:03:16.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.809+0000 7ff52ffff640 1 --2- 172.21.15.55:0/3215757512 >> v2:172.21.15.55:3300/0 conn(0x7ff538070c80 0x7ff53815b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:16.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.813+0000 7ff5367fc640 1 -- 172.21.15.55:0/3215757512 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1492661574 0 0) 0x7ff53815c2a0 con 0x7ff5380708a0 2024-03-19T02:03:16.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.813+0000 7ff5367fc640 1 -- 172.21.15.55:0/3215757512 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff524003530 con 0x7ff5380708a0 2024-03-19T02:03:16.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.813+0000 7ff52ffff640 1 -- 172.21.15.55:0/3215757512 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff53815be10 msgr2=0x7ff5380ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:16.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.813+0000 7ff52ffff640 1 --2- 172.21.15.55:0/3215757512 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff53815be10 0x7ff5380ccc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:16.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.813+0000 7ff52ffff640 1 -- 172.21.15.55:0/3215757512 >> v1:172.21.15.84:6789/0 conn(0x7ff5380708a0 legacy=0x7ff538151e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:16.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.813+0000 7ff52ffff640 1 -- 172.21.15.55:0/3215757512 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5380c7980 con 0x7ff538070c80 2024-03-19T02:03:16.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.817+0000 7ff52ffff640 1 --2- 172.21.15.55:0/3215757512 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff538070c80 0x7ff53815b8d0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ff5200097c0 tx=0x7ff52001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87b45f790700dab8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:16.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.821+0000 7ff5367fc640 1 -- 172.21.15.55:0/3215757512 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5200b5070 con 0x7ff538070c80 2024-03-19T02:03:16.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.821+0000 7ff5367fc640 1 -- 172.21.15.55:0/3215757512 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5200a8990 con 0x7ff538070c80 2024-03-19T02:03:16.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.821+0000 7ff5367fc640 1 -- 172.21.15.55:0/3215757512 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5200a8d60 con 0x7ff538070c80 2024-03-19T02:03:16.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.821+0000 7ff53d89e640 1 -- 172.21.15.55:0/3215757512 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff538070c80 msgr2=0x7ff53815b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:16.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.821+0000 7ff53d89e640 1 --2- 172.21.15.55:0/3215757512 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff538070c80 0x7ff53815b8d0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ff5200097c0 tx=0x7ff52001ec90 comp rx=0 tx=0).stop 2024-03-19T02:03:16.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.821+0000 7ff53d89e640 1 -- 172.21.15.55:0/3215757512 shutdown_connections 2024-03-19T02:03:16.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.821+0000 7ff53d89e640 1 --2- 172.21.15.55:0/3215757512 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff53815be10 0x7ff5380ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:16.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.821+0000 7ff53d89e640 1 --2- 172.21.15.55:0/3215757512 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff538070c80 0x7ff53815b8d0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:16.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.821+0000 7ff53d89e640 1 -- 172.21.15.55:0/3215757512 >> 172.21.15.55:0/3215757512 conn(0x7ff53806c580 msgr2=0x7ff53806c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:16.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.821+0000 7ff53d89e640 1 -- 172.21.15.55:0/3215757512 shutdown_connections 2024-03-19T02:03:16.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.821+0000 7ff53d89e640 1 -- 172.21.15.55:0/3215757512 wait complete. 2024-03-19T02:03:16.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.825+0000 7ff53d89e640 1 Processor -- start 2024-03-19T02:03:16.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.825+0000 7ff53d89e640 1 -- start start 2024-03-19T02:03:16.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.825+0000 7ff53d89e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff5380708a0 0x7ff53806e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:16.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.825+0000 7ff53d89e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff538070c80 0x7ff53815abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:16.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.825+0000 7ff53d89e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff53815be10 0x7ff53815b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:16.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.825+0000 7ff53d89e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff5380c8220 con 0x7ff538070c80 2024-03-19T02:03:16.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.825+0000 7ff53d89e640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff5380c7f40 con 0x7ff5380708a0 2024-03-19T02:03:16.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.825+0000 7ff53d89e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff538152540 con 0x7ff53815be10 2024-03-19T02:03:16.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.825+0000 7ff52ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff538070c80 0x7ff53815abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:16.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.825+0000 7ff5377fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff53815be10 0x7ff53815b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:16.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.825+0000 7ff52ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff538070c80 0x7ff53815abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49254/0 (socket says 172.21.15.55:49254) 2024-03-19T02:03:16.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.825+0000 7ff52ffff640 1 -- 172.21.15.55:0/2004541001 learned_addr learned my addr 172.21.15.55:0/2004541001 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:16.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.825+0000 7ff536ffd640 1 --2- 172.21.15.55:0/2004541001 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff5380708a0 0x7ff53806e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:16.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.829+0000 7ff52ffff640 1 -- 172.21.15.55:0/2004541001 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff53815be10 msgr2=0x7ff53815b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:16.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.829+0000 7ff52ffff640 1 --2- 172.21.15.55:0/2004541001 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff53815be10 0x7ff53815b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:16.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.829+0000 7ff52ffff640 1 -- 172.21.15.55:0/2004541001 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff5380708a0 msgr2=0x7ff53806e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:16.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.829+0000 7ff52ffff640 1 --2- 172.21.15.55:0/2004541001 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff5380708a0 0x7ff53806e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:16.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.829+0000 7ff52ffff640 1 -- 172.21.15.55:0/2004541001 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5380c4600 con 0x7ff538070c80 2024-03-19T02:03:16.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.833+0000 7ff52ffff640 1 --2- 172.21.15.55:0/2004541001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff538070c80 0x7ff53815abe0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7ff52001b8a0 tx=0x7ff52001ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:16.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.837+0000 7ff534ff9640 1 -- 172.21.15.55:0/2004541001 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5200bf070 con 0x7ff538070c80 2024-03-19T02:03:16.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.837+0000 7ff534ff9640 1 -- 172.21.15.55:0/2004541001 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5200b0700 con 0x7ff538070c80 2024-03-19T02:03:16.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.837+0000 7ff53d89e640 1 -- 172.21.15.55:0/2004541001 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff538155600 con 0x7ff538070c80 2024-03-19T02:03:16.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.837+0000 7ff534ff9640 1 -- 172.21.15.55:0/2004541001 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5200b5040 con 0x7ff538070c80 2024-03-19T02:03:16.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.837+0000 7ff53d89e640 1 -- 172.21.15.55:0/2004541001 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff538154d60 con 0x7ff538070c80 2024-03-19T02:03:16.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:16.837+0000 7ff53d89e640 1 -- 172.21.15.55:0/2004541001 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff538068a40 con 0x7ff538070c80 2024-03-19T02:03:17.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:17.097+0000 7ff534ff9640 1 -- 172.21.15.55:0/2004541001 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ff5200c2070 con 0x7ff538070c80 2024-03-19T02:03:17.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:17.097+0000 7ff534ff9640 1 --2- 172.21.15.55:0/2004541001 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff5180676b0 0x7ff518069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:17.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:17.097+0000 7ff536ffd640 1 --2- 172.21.15.55:0/2004541001 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff5180676b0 0x7ff518069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:17.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:17.101+0000 7ff536ffd640 1 --2- 172.21.15.55:0/2004541001 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff5180676b0 0x7ff518069b70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff528007fd0 tx=0x7ff528009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:17.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:17.113+0000 7ff534ff9640 1 -- 172.21.15.55:0/2004541001 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7ff5200f0800 con 0x7ff538070c80 2024-03-19T02:03:17.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:17.113+0000 7ff534ff9640 1 -- 172.21.15.55:0/2004541001 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ff5200187f0 con 0x7ff538070c80 2024-03-19T02:03:17.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:17.329+0000 7ff53d89e640 1 -- 172.21.15.55:0/2004541001 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff53815c4e0 con 0x7ff538070c80 2024-03-19T02:03:17.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:17.409+0000 7ff534ff9640 1 -- 172.21.15.55:0/2004541001 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7ff5200f2760 con 0x7ff538070c80 2024-03-19T02:03:17.409 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:17.410 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:17.410 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:17.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:17.413+0000 7ff53d89e640 1 -- 172.21.15.55:0/2004541001 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff5180676b0 msgr2=0x7ff518069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:17.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:17.413+0000 7ff53d89e640 1 --2- 172.21.15.55:0/2004541001 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff5180676b0 0x7ff518069b70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff528007fd0 tx=0x7ff528009040 comp rx=0 tx=0).stop 2024-03-19T02:03:17.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:17.413+0000 7ff53d89e640 1 -- 172.21.15.55:0/2004541001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff538070c80 msgr2=0x7ff53815abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:17.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:17.413+0000 7ff53d89e640 1 --2- 172.21.15.55:0/2004541001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff538070c80 0x7ff53815abe0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7ff52001b8a0 tx=0x7ff52001ba30 comp rx=0 tx=0).stop 2024-03-19T02:03:17.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:17.413+0000 7ff53d89e640 1 -- 172.21.15.55:0/2004541001 shutdown_connections 2024-03-19T02:03:17.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:17.413+0000 7ff53d89e640 1 --2- 172.21.15.55:0/2004541001 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff5180676b0 0x7ff518069b70 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:17.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:17.413+0000 7ff53d89e640 1 --2- 172.21.15.55:0/2004541001 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff53815be10 0x7ff53815b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:17.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:17.413+0000 7ff53d89e640 1 --2- 172.21.15.55:0/2004541001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff538070c80 0x7ff53815abe0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:17.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:17.413+0000 7ff53d89e640 1 --2- 172.21.15.55:0/2004541001 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff5380708a0 0x7ff53806e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:17.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:17.413+0000 7ff53d89e640 1 -- 172.21.15.55:0/2004541001 >> 172.21.15.55:0/2004541001 conn(0x7ff53806c580 msgr2=0x7ff5381589f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:17.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:17.413+0000 7ff53d89e640 1 -- 172.21.15.55:0/2004541001 shutdown_connections 2024-03-19T02:03:17.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:17.413+0000 7ff53d89e640 1 -- 172.21.15.55:0/2004541001 wait complete. 2024-03-19T02:03:18.036 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:18.036 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:19.037 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:19.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.153+0000 7f2db2c13640 1 Processor -- start 2024-03-19T02:03:19.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.153+0000 7f2db2c13640 1 -- start start 2024-03-19T02:03:19.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.157+0000 7f2db2c13640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dac15a0e0 0x7f2dac0cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:19.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.157+0000 7f2db2c13640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f2dac0cc4f0 0x7f2dac0c9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:19.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.157+0000 7f2db2c13640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dac15a6b0 con 0x7f2dac1544b0 2024-03-19T02:03:19.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.157+0000 7f2db2c13640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f2dac15a990 con 0x7f2dac0cc4f0 2024-03-19T02:03:19.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.157+0000 7f2db2c13640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2dac15ac70 con 0x7f2dac15a0e0 2024-03-19T02:03:19.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.157+0000 7f2dabfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dac15a0e0 0x7f2dac0cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:19.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.157+0000 7f2db11b0640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f2dac0cc4f0 0x7f2dac0c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:19.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.157+0000 7f2dabfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dac15a0e0 0x7f2dac0cbfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:36900/0 (socket says 172.21.15.55:36900) 2024-03-19T02:03:19.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.157+0000 7f2dabfff640 1 -- 172.21.15.55:0/1734500731 learned_addr learned my addr 172.21.15.55:0/1734500731 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:19.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.161+0000 7f2dabfff640 1 -- 172.21.15.55:0/1734500731 >> v2:172.21.15.84:3300/0 conn(0x7f2dac0cc4f0 msgr2=0x7f2dac0c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:19.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.161+0000 7f2dabfff640 1 --2- 172.21.15.55:0/1734500731 >> v2:172.21.15.84:3300/0 conn(0x7f2dac0cc4f0 0x7f2dac0c9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:19.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.161+0000 7f2dabfff640 1 -- 172.21.15.55:0/1734500731 >> v1:172.21.15.84:6789/0 conn(0x7f2dac1544b0 legacy=0x7f2dac06b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:19.160 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.161+0000 7f2dabfff640 1 -- 172.21.15.55:0/1734500731 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dac15af50 con 0x7f2dac15a0e0 2024-03-19T02:03:19.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.165+0000 7f2dabfff640 1 --2- 172.21.15.55:0/1734500731 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dac15a0e0 0x7f2dac0cbfb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2d98009870 tx=0x7f2d9801ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca412892ce9c11cc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:19.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.169+0000 7f2dab7fe640 1 -- 172.21.15.55:0/1734500731 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d980b5070 con 0x7f2dac15a0e0 2024-03-19T02:03:19.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.169+0000 7f2dab7fe640 1 -- 172.21.15.55:0/1734500731 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d980a89d0 con 0x7f2dac15a0e0 2024-03-19T02:03:19.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.169+0000 7f2dab7fe640 1 -- 172.21.15.55:0/1734500731 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d980a8da0 con 0x7f2dac15a0e0 2024-03-19T02:03:19.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.169+0000 7f2db2c13640 1 -- 172.21.15.55:0/1734500731 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dac15a0e0 msgr2=0x7f2dac0cbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:19.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.169+0000 7f2db2c13640 1 --2- 172.21.15.55:0/1734500731 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dac15a0e0 0x7f2dac0cbfb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2d98009870 tx=0x7f2d9801ed10 comp rx=0 tx=0).stop 2024-03-19T02:03:19.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.169+0000 7f2db2c13640 1 -- 172.21.15.55:0/1734500731 shutdown_connections 2024-03-19T02:03:19.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.169+0000 7f2db2c13640 1 --2- 172.21.15.55:0/1734500731 >> v2:172.21.15.84:3300/0 conn(0x7f2dac0cc4f0 0x7f2dac0c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:19.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.169+0000 7f2db2c13640 1 --2- 172.21.15.55:0/1734500731 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dac15a0e0 0x7f2dac0cbfb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:19.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.169+0000 7f2db2c13640 1 -- 172.21.15.55:0/1734500731 >> 172.21.15.55:0/1734500731 conn(0x7f2dac06e5a0 msgr2=0x7f2dac072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:19.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.169+0000 7f2db2c13640 1 -- 172.21.15.55:0/1734500731 shutdown_connections 2024-03-19T02:03:19.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.169+0000 7f2db2c13640 1 -- 172.21.15.55:0/1734500731 wait complete. 2024-03-19T02:03:19.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.173+0000 7f2db2c13640 1 Processor -- start 2024-03-19T02:03:19.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.173+0000 7f2db2c13640 1 -- start start 2024-03-19T02:03:19.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.173+0000 7f2db2c13640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2dac0cc4f0 0x7f2dac145610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:19.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.173+0000 7f2db2c13640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dac1544b0 0x7f2dac1e6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:19.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.173+0000 7f2db2c13640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2dac15a0e0 0x7f2dac152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:19.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.173+0000 7f2db2c13640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2dac0c8760 con 0x7f2dac0cc4f0 2024-03-19T02:03:19.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.173+0000 7f2db2c13640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2dac0c6be0 con 0x7f2dac15a0e0 2024-03-19T02:03:19.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.173+0000 7f2db2c13640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2dac0c88e0 con 0x7f2dac1544b0 2024-03-19T02:03:19.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.173+0000 7f2db09af640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2dac0cc4f0 0x7f2dac145610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:19.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.173+0000 7f2db11b0640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2dac15a0e0 0x7f2dac152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:19.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.173+0000 7f2dabfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dac1544b0 0x7f2dac1e6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:19.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.173+0000 7f2db09af640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2dac0cc4f0 0x7f2dac145610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47740/0 (socket says 172.21.15.55:47740) 2024-03-19T02:03:19.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.173+0000 7f2db09af640 1 -- 172.21.15.55:0/249754812 learned_addr learned my addr 172.21.15.55:0/249754812 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:19.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.177+0000 7f2db09af640 1 -- 172.21.15.55:0/249754812 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dac1544b0 msgr2=0x7f2dac1e6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:19.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.177+0000 7f2db09af640 1 --2- 172.21.15.55:0/249754812 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dac1544b0 0x7f2dac1e6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:19.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.177+0000 7f2db09af640 1 -- 172.21.15.55:0/249754812 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2dac15a0e0 msgr2=0x7f2dac152a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:19.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.177+0000 7f2db09af640 1 --2- 172.21.15.55:0/249754812 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2dac15a0e0 0x7f2dac152a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:19.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.177+0000 7f2db09af640 1 -- 172.21.15.55:0/249754812 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dac142820 con 0x7f2dac0cc4f0 2024-03-19T02:03:19.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.181+0000 7f2db09af640 1 --2- 172.21.15.55:0/249754812 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2dac0cc4f0 0x7f2dac145610 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f2da000c4c0 tx=0x7f2da0001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:19.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.185+0000 7f2da97fa640 1 -- 172.21.15.55:0/249754812 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da0017070 con 0x7f2dac0cc4f0 2024-03-19T02:03:19.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.185+0000 7f2db2c13640 1 -- 172.21.15.55:0/249754812 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dac1f3be0 con 0x7f2dac0cc4f0 2024-03-19T02:03:19.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.185+0000 7f2db2c13640 1 -- 172.21.15.55:0/249754812 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dac145e80 con 0x7f2dac0cc4f0 2024-03-19T02:03:19.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.185+0000 7f2db2c13640 1 -- 172.21.15.55:0/249754812 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dac068a40 con 0x7f2dac0cc4f0 2024-03-19T02:03:19.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.189+0000 7f2da97fa640 1 -- 172.21.15.55:0/249754812 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2da0007030 con 0x7f2dac0cc4f0 2024-03-19T02:03:19.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.189+0000 7f2da97fa640 1 -- 172.21.15.55:0/249754812 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da0021060 con 0x7f2dac0cc4f0 2024-03-19T02:03:19.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.481+0000 7f2da97fa640 1 -- 172.21.15.55:0/249754812 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f2da001a070 con 0x7f2dac0cc4f0 2024-03-19T02:03:19.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.481+0000 7f2da97fa640 1 --2- 172.21.15.55:0/249754812 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2d90067690 0x7f2d90069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:19.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.481+0000 7f2dabfff640 1 --2- 172.21.15.55:0/249754812 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2d90067690 0x7f2d90069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:19.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.481+0000 7f2dabfff640 1 --2- 172.21.15.55:0/249754812 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2d90067690 0x7f2d90069b50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2d980b2000 tx=0x7f2d98009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:19.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.505+0000 7f2da97fa640 1 -- 172.21.15.55:0/249754812 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f2da00010d0 con 0x7f2dac0cc4f0 2024-03-19T02:03:19.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.505+0000 7f2da97fa640 1 -- 172.21.15.55:0/249754812 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2dac142820 con 0x7f2dac0cc4f0 2024-03-19T02:03:19.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.717+0000 7f2db2c13640 1 -- 172.21.15.55:0/249754812 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2dac06bb10 con 0x7f2dac0cc4f0 2024-03-19T02:03:19.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.777+0000 7f2da97fa640 1 -- 172.21.15.55:0/249754812 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f2da00555f0 con 0x7f2dac0cc4f0 2024-03-19T02:03:19.779 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:19.779 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:19.779 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:19.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.781+0000 7f2db2c13640 1 -- 172.21.15.55:0/249754812 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2d90067690 msgr2=0x7f2d90069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:19.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.781+0000 7f2db2c13640 1 --2- 172.21.15.55:0/249754812 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2d90067690 0x7f2d90069b50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2d980b2000 tx=0x7f2d98009040 comp rx=0 tx=0).stop 2024-03-19T02:03:19.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.781+0000 7f2db2c13640 1 -- 172.21.15.55:0/249754812 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2dac0cc4f0 msgr2=0x7f2dac145610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:19.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.781+0000 7f2db2c13640 1 --2- 172.21.15.55:0/249754812 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2dac0cc4f0 0x7f2dac145610 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f2da000c4c0 tx=0x7f2da0001e80 comp rx=0 tx=0).stop 2024-03-19T02:03:19.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.781+0000 7f2db2c13640 1 -- 172.21.15.55:0/249754812 shutdown_connections 2024-03-19T02:03:19.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.781+0000 7f2db2c13640 1 --2- 172.21.15.55:0/249754812 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2d90067690 0x7f2d90069b50 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:19.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.781+0000 7f2db2c13640 1 --2- 172.21.15.55:0/249754812 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2dac15a0e0 0x7f2dac152a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:19.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.781+0000 7f2db2c13640 1 --2- 172.21.15.55:0/249754812 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2dac1544b0 0x7f2dac1e6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:19.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.781+0000 7f2db2c13640 1 --2- 172.21.15.55:0/249754812 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2dac0cc4f0 0x7f2dac145610 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:19.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.781+0000 7f2db2c13640 1 -- 172.21.15.55:0/249754812 >> 172.21.15.55:0/249754812 conn(0x7f2dac06e5a0 msgr2=0x7f2dac06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:19.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.781+0000 7f2db2c13640 1 -- 172.21.15.55:0/249754812 shutdown_connections 2024-03-19T02:03:19.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:19.781+0000 7f2db2c13640 1 -- 172.21.15.55:0/249754812 wait complete. 2024-03-19T02:03:20.403 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:20.403 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:21.404 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:21.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.525+0000 7fb157700640 1 Processor -- start 2024-03-19T02:03:21.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.525+0000 7fb157700640 1 -- start start 2024-03-19T02:03:21.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.525+0000 7fb157700640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb1500c39c0 0x7fb150158a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:21.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.525+0000 7fb157700640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fb150072c60 0x7fb15015b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:21.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.525+0000 7fb157700640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb150152660 con 0x7fb15006b9f0 2024-03-19T02:03:21.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.525+0000 7fb157700640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fb150073130 con 0x7fb1500c39c0 2024-03-19T02:03:21.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.525+0000 7fb157700640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fb1500733f0 con 0x7fb150072c60 2024-03-19T02:03:21.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.525+0000 7fb15549c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb1500c39c0 0x7fb150158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:21.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.525+0000 7fb154c9b640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fb150072c60 0x7fb15015b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:21.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.529+0000 7fb15549c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb1500c39c0 0x7fb150158a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47742/0 (socket says 172.21.15.55:47742) 2024-03-19T02:03:21.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.529+0000 7fb15549c640 1 -- 172.21.15.55:0/2142963877 learned_addr learned my addr 172.21.15.55:0/2142963877 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:21.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.529+0000 7fb15549c640 1 -- 172.21.15.55:0/2142963877 >> v2:172.21.15.84:3300/0 conn(0x7fb150072c60 msgr2=0x7fb15015b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:21.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.529+0000 7fb15549c640 1 --2- 172.21.15.55:0/2142963877 >> v2:172.21.15.84:3300/0 conn(0x7fb150072c60 0x7fb15015b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:21.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.529+0000 7fb15549c640 1 -- 172.21.15.55:0/2142963877 >> v1:172.21.15.84:6789/0 conn(0x7fb15006b9f0 legacy=0x7fb1500c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:21.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.529+0000 7fb15549c640 1 -- 172.21.15.55:0/2142963877 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1500736d0 con 0x7fb1500c39c0 2024-03-19T02:03:21.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.533+0000 7fb15549c640 1 --2- 172.21.15.55:0/2142963877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1500c39c0 0x7fb150158a80 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb150071fa0 tx=0x7fb14001efc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acda38e22232d5d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:21.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.537+0000 7fb14ffff640 1 -- 172.21.15.55:0/2142963877 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1400a8bd0 con 0x7fb1500c39c0 2024-03-19T02:03:21.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.537+0000 7fb14ffff640 1 -- 172.21.15.55:0/2142963877 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1400a8d70 con 0x7fb1500c39c0 2024-03-19T02:03:21.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.537+0000 7fb14ffff640 1 -- 172.21.15.55:0/2142963877 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1400b0610 con 0x7fb1500c39c0 2024-03-19T02:03:21.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.541+0000 7fb157700640 1 -- 172.21.15.55:0/2142963877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1500c39c0 msgr2=0x7fb150158a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:21.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.541+0000 7fb157700640 1 --2- 172.21.15.55:0/2142963877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1500c39c0 0x7fb150158a80 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb150071fa0 tx=0x7fb14001efc0 comp rx=0 tx=0).stop 2024-03-19T02:03:21.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.541+0000 7fb157700640 1 -- 172.21.15.55:0/2142963877 shutdown_connections 2024-03-19T02:03:21.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.541+0000 7fb157700640 1 --2- 172.21.15.55:0/2142963877 >> v2:172.21.15.84:3300/0 conn(0x7fb150072c60 0x7fb15015b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:21.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.541+0000 7fb157700640 1 --2- 172.21.15.55:0/2142963877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1500c39c0 0x7fb150158a80 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:21.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.541+0000 7fb157700640 1 -- 172.21.15.55:0/2142963877 >> 172.21.15.55:0/2142963877 conn(0x7fb1500cc9c0 msgr2=0x7fb1500ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:21.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.541+0000 7fb157700640 1 -- 172.21.15.55:0/2142963877 shutdown_connections 2024-03-19T02:03:21.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.541+0000 7fb157700640 1 -- 172.21.15.55:0/2142963877 wait complete. 2024-03-19T02:03:21.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.541+0000 7fb157700640 1 Processor -- start 2024-03-19T02:03:21.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.541+0000 7fb157700640 1 -- start start 2024-03-19T02:03:21.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.541+0000 7fb157700640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb15006b9f0 0x7fb1501383f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:21.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.541+0000 7fb157700640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb150072c60 0x7fb150139620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:21.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.541+0000 7fb157700640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb1500c39c0 0x7fb150151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:21.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.541+0000 7fb157700640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb150157f50 con 0x7fb150072c60 2024-03-19T02:03:21.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.541+0000 7fb157700640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb150158560 con 0x7fb15006b9f0 2024-03-19T02:03:21.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.541+0000 7fb157700640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fb1500c67e0 con 0x7fb1500c39c0 2024-03-19T02:03:21.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.545+0000 7fb154c9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb150072c60 0x7fb150139620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:21.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.545+0000 7fb15549c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb15006b9f0 0x7fb1501383f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:21.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.545+0000 7fb154c9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb150072c60 0x7fb150139620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47754/0 (socket says 172.21.15.55:47754) 2024-03-19T02:03:21.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.545+0000 7fb154c9b640 1 -- 172.21.15.55:0/1638362496 learned_addr learned my addr 172.21.15.55:0/1638362496 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:21.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.545+0000 7fb155c9d640 1 --2- 172.21.15.55:0/1638362496 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb1500c39c0 0x7fb150151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:21.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.549+0000 7fb154c9b640 1 -- 172.21.15.55:0/1638362496 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb1500c39c0 msgr2=0x7fb150151940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:21.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.549+0000 7fb154c9b640 1 --2- 172.21.15.55:0/1638362496 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb1500c39c0 0x7fb150151940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:21.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.549+0000 7fb154c9b640 1 -- 172.21.15.55:0/1638362496 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb15006b9f0 msgr2=0x7fb1501383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:21.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.549+0000 7fb154c9b640 1 --2- 172.21.15.55:0/1638362496 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb15006b9f0 0x7fb1501383f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:21.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.549+0000 7fb154c9b640 1 -- 172.21.15.55:0/1638362496 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb150137f80 con 0x7fb150072c60 2024-03-19T02:03:21.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.553+0000 7fb154c9b640 1 --2- 172.21.15.55:0/1638362496 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb150072c60 0x7fb150139620 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fb14400e8a0 tx=0x7fb14400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:21.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.557+0000 7fb14dffb640 1 -- 172.21.15.55:0/1638362496 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb144017070 con 0x7fb150072c60 2024-03-19T02:03:21.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.557+0000 7fb14dffb640 1 -- 172.21.15.55:0/1638362496 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb14400c940 con 0x7fb150072c60 2024-03-19T02:03:21.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.557+0000 7fb157700640 1 -- 172.21.15.55:0/1638362496 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1500c4110 con 0x7fb150072c60 2024-03-19T02:03:21.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.557+0000 7fb14dffb640 1 -- 172.21.15.55:0/1638362496 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb14400ccf0 con 0x7fb150072c60 2024-03-19T02:03:21.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.557+0000 7fb157700640 1 -- 172.21.15.55:0/1638362496 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1500c46d0 con 0x7fb150072c60 2024-03-19T02:03:21.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.561+0000 7fb157700640 1 -- 172.21.15.55:0/1638362496 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb150068a40 con 0x7fb150072c60 2024-03-19T02:03:21.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.837+0000 7fb14dffb640 1 -- 172.21.15.55:0/1638362496 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb14401a070 con 0x7fb150072c60 2024-03-19T02:03:21.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.837+0000 7fb14dffb640 1 --2- 172.21.15.55:0/1638362496 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb13c067660 0x7fb13c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:21.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.837+0000 7fb15549c640 1 --2- 172.21.15.55:0/1638362496 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb13c067660 0x7fb13c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:21.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.837+0000 7fb15549c640 1 --2- 172.21.15.55:0/1638362496 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb13c067660 0x7fb13c069b20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb1400b1000 tx=0x7fb1400aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:21.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.869+0000 7fb14dffb640 1 -- 172.21.15.55:0/1638362496 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fb144054600 con 0x7fb150072c60 2024-03-19T02:03:21.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:21.869+0000 7fb14dffb640 1 -- 172.21.15.55:0/1638362496 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fb144056560 con 0x7fb150072c60 2024-03-19T02:03:22.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:22.081+0000 7fb157700640 1 -- 172.21.15.55:0/1638362496 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb150072a40 con 0x7fb150072c60 2024-03-19T02:03:22.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:22.097+0000 7fb14dffb640 1 -- 172.21.15.55:0/1638362496 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fb14405a600 con 0x7fb150072c60 2024-03-19T02:03:22.099 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:22.099 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:22.099 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:22.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:22.101+0000 7fb157700640 1 -- 172.21.15.55:0/1638362496 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb13c067660 msgr2=0x7fb13c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:22.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:22.101+0000 7fb157700640 1 --2- 172.21.15.55:0/1638362496 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb13c067660 0x7fb13c069b20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb1400b1000 tx=0x7fb1400aa070 comp rx=0 tx=0).stop 2024-03-19T02:03:22.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:22.101+0000 7fb157700640 1 -- 172.21.15.55:0/1638362496 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb150072c60 msgr2=0x7fb150139620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:22.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:22.101+0000 7fb157700640 1 --2- 172.21.15.55:0/1638362496 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb150072c60 0x7fb150139620 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fb14400e8a0 tx=0x7fb14400eca0 comp rx=0 tx=0).stop 2024-03-19T02:03:22.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:22.101+0000 7fb157700640 1 -- 172.21.15.55:0/1638362496 shutdown_connections 2024-03-19T02:03:22.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:22.101+0000 7fb157700640 1 --2- 172.21.15.55:0/1638362496 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb13c067660 0x7fb13c069b20 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:22.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:22.101+0000 7fb157700640 1 --2- 172.21.15.55:0/1638362496 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb1500c39c0 0x7fb150151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:22.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:22.101+0000 7fb157700640 1 --2- 172.21.15.55:0/1638362496 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb150072c60 0x7fb150139620 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:22.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:22.101+0000 7fb157700640 1 --2- 172.21.15.55:0/1638362496 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb15006b9f0 0x7fb1501383f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:22.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:22.101+0000 7fb157700640 1 -- 172.21.15.55:0/1638362496 >> 172.21.15.55:0/1638362496 conn(0x7fb1500cc9c0 msgr2=0x7fb1500ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:22.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:22.101+0000 7fb157700640 1 -- 172.21.15.55:0/1638362496 shutdown_connections 2024-03-19T02:03:22.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:22.101+0000 7fb157700640 1 -- 172.21.15.55:0/1638362496 wait complete. 2024-03-19T02:03:22.722 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:22.722 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:23.723 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:23.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.837+0000 7fba55f80640 1 Processor -- start 2024-03-19T02:03:23.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.837+0000 7fba55f80640 1 -- start start 2024-03-19T02:03:23.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.837+0000 7fba55f80640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fba501511f0 0x7fba500c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:23.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.837+0000 7fba55f80640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba500c3330 con 0x7fba500c2e40 2024-03-19T02:03:23.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.837+0000 7fba55f80640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba500c3610 con 0x7fba500c8260 2024-03-19T02:03:23.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.837+0000 7fba55f80640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fba500c5040 con 0x7fba501511f0 2024-03-19T02:03:23.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.837+0000 7fba4effd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fba500c2e40 0x7fba500c7450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:42916/0 (socket says 172.21.15.55:42916) 2024-03-19T02:03:23.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.837+0000 7fba4effd640 1 -- 172.21.15.55:0/3996376330 learned_addr learned my addr 172.21.15.55:0/3996376330 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:03:23.840 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.841+0000 7fba4f7fe640 1 --2- 172.21.15.55:0/3996376330 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fba501511f0 0x7fba500c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:23.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.841+0000 7fba4e7fc640 1 -- 172.21.15.55:0/3996376330 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 46999211 0 0) 0x7fba500c3330 con 0x7fba500c2e40 2024-03-19T02:03:23.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.841+0000 7fba4e7fc640 1 -- 172.21.15.55:0/3996376330 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba38005740 con 0x7fba500c2e40 2024-03-19T02:03:23.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.841+0000 7fba4e7fc640 1 -- 172.21.15.55:0/3996376330 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3318255361 0 0) 0x7fba500c3610 con 0x7fba500c8260 2024-03-19T02:03:23.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.841+0000 7fba4e7fc640 1 -- 172.21.15.55:0/3996376330 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba500c3330 con 0x7fba500c8260 2024-03-19T02:03:23.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.845+0000 7fba4f7fe640 1 -- 172.21.15.55:0/3996376330 >> v1:172.21.15.84:6789/0 conn(0x7fba500c8260 legacy=0x7fba50161810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:23.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.845+0000 7fba4f7fe640 1 -- 172.21.15.55:0/3996376330 >> v1:172.21.15.55:6789/0 conn(0x7fba500c2e40 legacy=0x7fba500c7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:23.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.845+0000 7fba4f7fe640 1 -- 172.21.15.55:0/3996376330 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba500c38f0 con 0x7fba501511f0 2024-03-19T02:03:23.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.849+0000 7fba4f7fe640 1 --2- 172.21.15.55:0/3996376330 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fba501511f0 0x7fba500c5a90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fba400097f0 tx=0x7fba4001ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46871c2486761efe server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:23.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.853+0000 7fba4e7fc640 1 -- 172.21.15.55:0/3996376330 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba400b5070 con 0x7fba501511f0 2024-03-19T02:03:23.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.853+0000 7fba4e7fc640 1 -- 172.21.15.55:0/3996376330 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba400a89d0 con 0x7fba501511f0 2024-03-19T02:03:23.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.853+0000 7fba4e7fc640 1 -- 172.21.15.55:0/3996376330 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba400a8da0 con 0x7fba501511f0 2024-03-19T02:03:23.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.853+0000 7fba55f80640 1 -- 172.21.15.55:0/3996376330 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fba501511f0 msgr2=0x7fba500c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:23.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.853+0000 7fba55f80640 1 --2- 172.21.15.55:0/3996376330 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fba501511f0 0x7fba500c5a90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fba400097f0 tx=0x7fba4001ecd0 comp rx=0 tx=0).stop 2024-03-19T02:03:23.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.853+0000 7fba55f80640 1 -- 172.21.15.55:0/3996376330 shutdown_connections 2024-03-19T02:03:23.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.853+0000 7fba55f80640 1 --2- 172.21.15.55:0/3996376330 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fba501511f0 0x7fba500c5a90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:23.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.853+0000 7fba55f80640 1 -- 172.21.15.55:0/3996376330 >> 172.21.15.55:0/3996376330 conn(0x7fba5006f3b0 msgr2=0x7fba5006f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:23.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.853+0000 7fba55f80640 1 -- 172.21.15.55:0/3996376330 shutdown_connections 2024-03-19T02:03:23.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.853+0000 7fba55f80640 1 -- 172.21.15.55:0/3996376330 wait complete. 2024-03-19T02:03:23.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.853+0000 7fba55f80640 1 Processor -- start 2024-03-19T02:03:23.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.853+0000 7fba55f80640 1 -- start start 2024-03-19T02:03:23.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.857+0000 7fba55f80640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fba500c2e40 0x7fba501f09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:23.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.857+0000 7fba55f80640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fba500c8260 0x7fba5014fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:23.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.857+0000 7fba55f80640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fba501511f0 0x7fba501563f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:23.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.857+0000 7fba55f80640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fba500c29c0 con 0x7fba501511f0 2024-03-19T02:03:23.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.857+0000 7fba55f80640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fba501515d0 con 0x7fba500c2e40 2024-03-19T02:03:23.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.857+0000 7fba55f80640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fba500c2b40 con 0x7fba500c8260 2024-03-19T02:03:23.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.857+0000 7fba4ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fba501511f0 0x7fba501563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:23.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.857+0000 7fba4f7fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fba500c2e40 0x7fba501f09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:23.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.857+0000 7fba4ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fba501511f0 0x7fba501563f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47758/0 (socket says 172.21.15.55:47758) 2024-03-19T02:03:23.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.857+0000 7fba4ffff640 1 -- 172.21.15.55:0/376751247 learned_addr learned my addr 172.21.15.55:0/376751247 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:23.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.857+0000 7fba4effd640 1 --2- 172.21.15.55:0/376751247 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fba500c8260 0x7fba5014fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:23.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.861+0000 7fba4ffff640 1 -- 172.21.15.55:0/376751247 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fba500c8260 msgr2=0x7fba5014fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:23.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.861+0000 7fba4ffff640 1 --2- 172.21.15.55:0/376751247 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fba500c8260 0x7fba5014fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:23.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.861+0000 7fba4ffff640 1 -- 172.21.15.55:0/376751247 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fba500c2e40 msgr2=0x7fba501f09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:23.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.861+0000 7fba4ffff640 1 --2- 172.21.15.55:0/376751247 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fba500c2e40 0x7fba501f09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:23.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.861+0000 7fba4ffff640 1 -- 172.21.15.55:0/376751247 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba501509c0 con 0x7fba501511f0 2024-03-19T02:03:23.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.861+0000 7fba4ffff640 1 --2- 172.21.15.55:0/376751247 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fba501511f0 0x7fba501563f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fba4800c6b0 tx=0x7fba4800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:23.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.865+0000 7fba37fff640 1 -- 172.21.15.55:0/376751247 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba48003b80 con 0x7fba501511f0 2024-03-19T02:03:23.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.865+0000 7fba37fff640 1 -- 172.21.15.55:0/376751247 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba48003d20 con 0x7fba501511f0 2024-03-19T02:03:23.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.865+0000 7fba55f80640 1 -- 172.21.15.55:0/376751247 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba50154ef0 con 0x7fba501511f0 2024-03-19T02:03:23.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.865+0000 7fba37fff640 1 -- 172.21.15.55:0/376751247 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba48005310 con 0x7fba501511f0 2024-03-19T02:03:23.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.865+0000 7fba55f80640 1 -- 172.21.15.55:0/376751247 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba50155470 con 0x7fba501511f0 2024-03-19T02:03:23.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.869+0000 7fba55f80640 1 -- 172.21.15.55:0/376751247 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba50068330 con 0x7fba501511f0 2024-03-19T02:03:23.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.897+0000 7fba37fff640 1 -- 172.21.15.55:0/376751247 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fba480054b0 con 0x7fba501511f0 2024-03-19T02:03:23.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.897+0000 7fba37fff640 1 --2- 172.21.15.55:0/376751247 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fba3c0676e0 0x7fba3c069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:23.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.897+0000 7fba4f7fe640 1 --2- 172.21.15.55:0/376751247 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fba3c0676e0 0x7fba3c069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:23.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.901+0000 7fba4f7fe640 1 --2- 172.21.15.55:0/376751247 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fba3c0676e0 0x7fba3c069ba0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fba400b2000 tx=0x7fba40008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:23.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:23.901+0000 7fba37fff640 1 -- 172.21.15.55:0/376751247 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fba480547a0 con 0x7fba501511f0 2024-03-19T02:03:24.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:24.169+0000 7fba37fff640 1 -- 172.21.15.55:0/376751247 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fba501509c0 con 0x7fba501511f0 2024-03-19T02:03:24.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:24.409+0000 7fba55f80640 1 -- 172.21.15.55:0/376751247 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba500c4860 con 0x7fba501511f0 2024-03-19T02:03:24.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:24.425+0000 7fba37fff640 1 -- 172.21.15.55:0/376751247 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fba48056700 con 0x7fba501511f0 2024-03-19T02:03:24.426 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:24.426 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:24.427 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:24.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:24.429+0000 7fba55f80640 1 -- 172.21.15.55:0/376751247 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fba3c0676e0 msgr2=0x7fba3c069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:24.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:24.429+0000 7fba55f80640 1 --2- 172.21.15.55:0/376751247 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fba3c0676e0 0x7fba3c069ba0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fba400b2000 tx=0x7fba40008040 comp rx=0 tx=0).stop 2024-03-19T02:03:24.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:24.429+0000 7fba55f80640 1 -- 172.21.15.55:0/376751247 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fba501511f0 msgr2=0x7fba501563f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:24.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:24.429+0000 7fba55f80640 1 --2- 172.21.15.55:0/376751247 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fba501511f0 0x7fba501563f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fba4800c6b0 tx=0x7fba4800cb70 comp rx=0 tx=0).stop 2024-03-19T02:03:24.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:24.429+0000 7fba55f80640 1 -- 172.21.15.55:0/376751247 shutdown_connections 2024-03-19T02:03:24.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:24.429+0000 7fba55f80640 1 --2- 172.21.15.55:0/376751247 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fba3c0676e0 0x7fba3c069ba0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:24.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:24.429+0000 7fba55f80640 1 --2- 172.21.15.55:0/376751247 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fba501511f0 0x7fba501563f0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:24.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:24.429+0000 7fba55f80640 1 --2- 172.21.15.55:0/376751247 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fba500c8260 0x7fba5014fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:24.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:24.429+0000 7fba55f80640 1 --2- 172.21.15.55:0/376751247 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fba500c2e40 0x7fba501f09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:24.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:24.429+0000 7fba55f80640 1 -- 172.21.15.55:0/376751247 >> 172.21.15.55:0/376751247 conn(0x7fba5006f3b0 msgr2=0x7fba500c6030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:24.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:24.429+0000 7fba55f80640 1 -- 172.21.15.55:0/376751247 shutdown_connections 2024-03-19T02:03:24.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:24.429+0000 7fba55f80640 1 -- 172.21.15.55:0/376751247 wait complete. 2024-03-19T02:03:25.047 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:25.048 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:26.049 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:26.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.181+0000 7fa9917d5640 1 Processor -- start 2024-03-19T02:03:26.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.181+0000 7fa9917d5640 1 -- start start 2024-03-19T02:03:26.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.181+0000 7fa9917d5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa98c070780 0x7fa98c15b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:26.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.181+0000 7fa9917d5640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa98c15bb00 0x7fa98c15bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:26.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.181+0000 7fa9917d5640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa98c15c4d0 con 0x7fa98c152120 2024-03-19T02:03:26.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.181+0000 7fa9917d5640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fa98c152500 con 0x7fa98c070780 2024-03-19T02:03:26.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.181+0000 7fa9917d5640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa98c070590 con 0x7fa98c15bb00 2024-03-19T02:03:26.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.181+0000 7fa98a7fc640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa98c070780 0x7fa98c15b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:26.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.181+0000 7fa98b7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa98c15bb00 0x7fa98c15bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:26.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.181+0000 7fa98affd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fa98c152120 0x7fa98c0c90d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:42932/0 (socket says 172.21.15.55:42932) 2024-03-19T02:03:26.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.181+0000 7fa98affd640 1 -- 172.21.15.55:0/266468345 learned_addr learned my addr 172.21.15.55:0/266468345 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:03:26.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.185+0000 7fa98b7fe640 1 -- 172.21.15.55:0/266468345 >> v2:172.21.15.55:3300/0 conn(0x7fa98c070780 msgr2=0x7fa98c15b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:26.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.185+0000 7fa98b7fe640 1 --2- 172.21.15.55:0/266468345 >> v2:172.21.15.55:3300/0 conn(0x7fa98c070780 0x7fa98c15b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:26.186 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.185+0000 7fa98b7fe640 1 -- 172.21.15.55:0/266468345 >> v1:172.21.15.55:6789/0 conn(0x7fa98c152120 legacy=0x7fa98c0c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:26.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.185+0000 7fa98b7fe640 1 -- 172.21.15.55:0/266468345 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa98c06fff0 con 0x7fa98c15bb00 2024-03-19T02:03:26.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.193+0000 7fa98b7fe640 1 --2- 172.21.15.55:0/266468345 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa98c15bb00 0x7fa98c15bf90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa984009d90 tx=0x7fa98401c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34cec10d108e2e39 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:26.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.197+0000 7fa989ffb640 1 -- 172.21.15.55:0/266468345 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9840b2070 con 0x7fa98c15bb00 2024-03-19T02:03:26.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.197+0000 7fa989ffb640 1 -- 172.21.15.55:0/266468345 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9840a5830 con 0x7fa98c15bb00 2024-03-19T02:03:26.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.197+0000 7fa989ffb640 1 -- 172.21.15.55:0/266468345 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9840a5c00 con 0x7fa98c15bb00 2024-03-19T02:03:26.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.197+0000 7fa9917d5640 1 -- 172.21.15.55:0/266468345 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa98c15bb00 msgr2=0x7fa98c15bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:26.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.197+0000 7fa9917d5640 1 --2- 172.21.15.55:0/266468345 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa98c15bb00 0x7fa98c15bf90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa984009d90 tx=0x7fa98401c970 comp rx=0 tx=0).stop 2024-03-19T02:03:26.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.197+0000 7fa9917d5640 1 -- 172.21.15.55:0/266468345 shutdown_connections 2024-03-19T02:03:26.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.197+0000 7fa9917d5640 1 --2- 172.21.15.55:0/266468345 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa98c15bb00 0x7fa98c15bf90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:26.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.197+0000 7fa9917d5640 1 --2- 172.21.15.55:0/266468345 >> v2:172.21.15.55:3300/0 conn(0x7fa98c070780 0x7fa98c15b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:26.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.197+0000 7fa9917d5640 1 -- 172.21.15.55:0/266468345 >> 172.21.15.55:0/266468345 conn(0x7fa98c067600 msgr2=0x7fa98c071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:26.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.197+0000 7fa9917d5640 1 -- 172.21.15.55:0/266468345 shutdown_connections 2024-03-19T02:03:26.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.197+0000 7fa9917d5640 1 -- 172.21.15.55:0/266468345 wait complete. 2024-03-19T02:03:26.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.201+0000 7fa9917d5640 1 Processor -- start 2024-03-19T02:03:26.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.201+0000 7fa9917d5640 1 -- start start 2024-03-19T02:03:26.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.201+0000 7fa9917d5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa98c070780 0x7fa98c1576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:26.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.201+0000 7fa9917d5640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa98c152120 0x7fa98c0cd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:26.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.201+0000 7fa9917d5640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa98c15bb00 0x7fa98c1584c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:26.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.201+0000 7fa9917d5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa98c0c31c0 con 0x7fa98c070780 2024-03-19T02:03:26.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.201+0000 7fa9917d5640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa98c0c2ee0 con 0x7fa98c15bb00 2024-03-19T02:03:26.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.201+0000 7fa9917d5640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa98c0c97e0 con 0x7fa98c152120 2024-03-19T02:03:26.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.201+0000 7fa98affd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa98c070780 0x7fa98c1576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:26.203 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.201+0000 7fa98a7fc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa98c152120 0x7fa98c0cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:26.203 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.201+0000 7fa98b7fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa98c15bb00 0x7fa98c1584c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:26.203 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.201+0000 7fa98affd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa98c070780 0x7fa98c1576d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47782/0 (socket says 172.21.15.55:47782) 2024-03-19T02:03:26.203 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.201+0000 7fa98affd640 1 -- 172.21.15.55:0/214807993 learned_addr learned my addr 172.21.15.55:0/214807993 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:26.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.209+0000 7fa98affd640 1 -- 172.21.15.55:0/214807993 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa98c152120 msgr2=0x7fa98c0cd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:26.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.209+0000 7fa98affd640 1 --2- 172.21.15.55:0/214807993 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa98c152120 0x7fa98c0cd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:26.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.209+0000 7fa98affd640 1 -- 172.21.15.55:0/214807993 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa98c15bb00 msgr2=0x7fa98c1584c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:26.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.209+0000 7fa98affd640 1 --2- 172.21.15.55:0/214807993 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa98c15bb00 0x7fa98c1584c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:26.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.209+0000 7fa98affd640 1 -- 172.21.15.55:0/214807993 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa98c158d70 con 0x7fa98c070780 2024-03-19T02:03:26.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.213+0000 7fa98affd640 1 --2- 172.21.15.55:0/214807993 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa98c070780 0x7fa98c1576d0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fa97c00c450 tx=0x7fa97c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:26.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.221+0000 7fa9737fe640 1 -- 172.21.15.55:0/214807993 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c017070 con 0x7fa98c070780 2024-03-19T02:03:26.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.221+0000 7fa9737fe640 1 -- 172.21.15.55:0/214807993 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa97c005cf0 con 0x7fa98c070780 2024-03-19T02:03:26.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.221+0000 7fa9737fe640 1 -- 172.21.15.55:0/214807993 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c007290 con 0x7fa98c070780 2024-03-19T02:03:26.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.221+0000 7fa9917d5640 1 -- 172.21.15.55:0/214807993 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa98c0cbac0 con 0x7fa98c070780 2024-03-19T02:03:26.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.221+0000 7fa9917d5640 1 -- 172.21.15.55:0/214807993 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa98c156ed0 con 0x7fa98c070780 2024-03-19T02:03:26.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.221+0000 7fa9917d5640 1 -- 172.21.15.55:0/214807993 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa98c068a40 con 0x7fa98c070780 2024-03-19T02:03:26.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.489+0000 7fa9737fe640 1 -- 172.21.15.55:0/214807993 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa97c01a070 con 0x7fa98c070780 2024-03-19T02:03:26.488 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.489+0000 7fa9737fe640 1 --2- 172.21.15.55:0/214807993 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa9780676b0 0x7fa978069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:26.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.489+0000 7fa98a7fc640 1 --2- 172.21.15.55:0/214807993 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa9780676b0 0x7fa978069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:26.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.489+0000 7fa98a7fc640 1 --2- 172.21.15.55:0/214807993 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa9780676b0 0x7fa978069b70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa98c14c1f0 tx=0x7fa980009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:26.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.517+0000 7fa9737fe640 1 -- 172.21.15.55:0/214807993 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fa97c0534a0 con 0x7fa98c070780 2024-03-19T02:03:26.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.517+0000 7fa9737fe640 1 -- 172.21.15.55:0/214807993 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa97c021080 con 0x7fa98c070780 2024-03-19T02:03:26.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.745+0000 7fa9917d5640 1 -- 172.21.15.55:0/214807993 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa98c151950 con 0x7fa98c070780 2024-03-19T02:03:26.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.769+0000 7fa9737fe640 1 -- 172.21.15.55:0/214807993 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fa98c15c4d0 con 0x7fa98c070780 2024-03-19T02:03:26.771 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:26.771 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:26.771 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:26.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.773+0000 7fa9917d5640 1 -- 172.21.15.55:0/214807993 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa9780676b0 msgr2=0x7fa978069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:26.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.773+0000 7fa9917d5640 1 --2- 172.21.15.55:0/214807993 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa9780676b0 0x7fa978069b70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa98c14c1f0 tx=0x7fa980009190 comp rx=0 tx=0).stop 2024-03-19T02:03:26.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.773+0000 7fa9917d5640 1 -- 172.21.15.55:0/214807993 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa98c070780 msgr2=0x7fa98c1576d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:26.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.773+0000 7fa9917d5640 1 --2- 172.21.15.55:0/214807993 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa98c070780 0x7fa98c1576d0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fa97c00c450 tx=0x7fa97c00c910 comp rx=0 tx=0).stop 2024-03-19T02:03:26.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.773+0000 7fa9917d5640 1 -- 172.21.15.55:0/214807993 shutdown_connections 2024-03-19T02:03:26.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.773+0000 7fa9917d5640 1 --2- 172.21.15.55:0/214807993 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa9780676b0 0x7fa978069b70 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:26.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.773+0000 7fa9917d5640 1 --2- 172.21.15.55:0/214807993 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa98c15bb00 0x7fa98c1584c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:26.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.773+0000 7fa9917d5640 1 --2- 172.21.15.55:0/214807993 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa98c152120 0x7fa98c0cd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:26.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.773+0000 7fa9917d5640 1 --2- 172.21.15.55:0/214807993 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa98c070780 0x7fa98c1576d0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:26.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.773+0000 7fa9917d5640 1 -- 172.21.15.55:0/214807993 >> 172.21.15.55:0/214807993 conn(0x7fa98c067600 msgr2=0x7fa98c14cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:26.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.773+0000 7fa9917d5640 1 -- 172.21.15.55:0/214807993 shutdown_connections 2024-03-19T02:03:26.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:26.773+0000 7fa9917d5640 1 -- 172.21.15.55:0/214807993 wait complete. 2024-03-19T02:03:27.395 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:27.395 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:28.396 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:28.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.509+0000 7f8603f68640 1 Processor -- start 2024-03-19T02:03:28.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.509+0000 7f8603f68640 1 -- start start 2024-03-19T02:03:28.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.509+0000 7f8603f68640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f85fc152120 0x7f85fc0c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:28.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.509+0000 7f8603f68640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f85fc0c9610 0x7f85fc157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:28.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.509+0000 7f8603f68640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85fc06fff0 con 0x7f85fc070780 2024-03-19T02:03:28.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.509+0000 7f8603f68640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f85fc152500 con 0x7f85fc0c9610 2024-03-19T02:03:28.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.509+0000 7f8603f68640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f85fc070590 con 0x7f85fc152120 2024-03-19T02:03:28.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.513+0000 7f8602505640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f85fc070780 0x7f85fc158510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:42944/0 (socket says 172.21.15.55:42944) 2024-03-19T02:03:28.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.513+0000 7f8602505640 1 -- 172.21.15.55:0/701069618 learned_addr learned my addr 172.21.15.55:0/701069618 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:03:28.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.513+0000 7f8601d04640 1 --2- 172.21.15.55:0/701069618 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f85fc152120 0x7f85fc0c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:28.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.513+0000 7f8601503640 1 --2- 172.21.15.55:0/701069618 >> v2:172.21.15.84:3300/0 conn(0x7f85fc0c9610 0x7f85fc157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:28.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.517+0000 7f8600d02640 1 -- 172.21.15.55:0/701069618 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1660693917 0 0) 0x7f85fc06fff0 con 0x7f85fc070780 2024-03-19T02:03:28.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.517+0000 7f8600d02640 1 -- 172.21.15.55:0/701069618 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85e8003530 con 0x7f85fc070780 2024-03-19T02:03:28.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.517+0000 7f8601503640 1 -- 172.21.15.55:0/701069618 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f85fc152120 msgr2=0x7f85fc0c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:28.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.517+0000 7f8601503640 1 --2- 172.21.15.55:0/701069618 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f85fc152120 0x7f85fc0c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:28.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.517+0000 7f8601503640 1 -- 172.21.15.55:0/701069618 >> v1:172.21.15.55:6789/0 conn(0x7f85fc070780 legacy=0x7f85fc158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:28.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.517+0000 7f8601503640 1 -- 172.21.15.55:0/701069618 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85fc0c39c0 con 0x7f85fc0c9610 2024-03-19T02:03:28.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.521+0000 7f8601503640 1 --2- 172.21.15.55:0/701069618 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85fc0c9610 0x7f85fc157fd0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f85ec0097c0 tx=0x7f85ec01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=837d137b0321e4f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:28.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.525+0000 7f8600d02640 1 -- 172.21.15.55:0/701069618 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ec0b0070 con 0x7f85fc0c9610 2024-03-19T02:03:28.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.529+0000 7f8600d02640 1 -- 172.21.15.55:0/701069618 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85ec0ab600 con 0x7f85fc0c9610 2024-03-19T02:03:28.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.529+0000 7f8600d02640 1 -- 172.21.15.55:0/701069618 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ec0ab9b0 con 0x7f85fc0c9610 2024-03-19T02:03:28.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.529+0000 7f8603f68640 1 -- 172.21.15.55:0/701069618 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85fc0c9610 msgr2=0x7f85fc157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:28.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.529+0000 7f8603f68640 1 --2- 172.21.15.55:0/701069618 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85fc0c9610 0x7f85fc157fd0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f85ec0097c0 tx=0x7f85ec01c4a0 comp rx=0 tx=0).stop 2024-03-19T02:03:28.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.529+0000 7f8603f68640 1 -- 172.21.15.55:0/701069618 shutdown_connections 2024-03-19T02:03:28.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.529+0000 7f8603f68640 1 --2- 172.21.15.55:0/701069618 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85fc0c9610 0x7f85fc157fd0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:28.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.529+0000 7f8603f68640 1 --2- 172.21.15.55:0/701069618 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f85fc152120 0x7f85fc0c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:28.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.529+0000 7f8603f68640 1 -- 172.21.15.55:0/701069618 >> 172.21.15.55:0/701069618 conn(0x7f85fc067600 msgr2=0x7f85fc071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:28.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.529+0000 7f8603f68640 1 -- 172.21.15.55:0/701069618 shutdown_connections 2024-03-19T02:03:28.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.529+0000 7f8603f68640 1 -- 172.21.15.55:0/701069618 wait complete. 2024-03-19T02:03:28.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.529+0000 7f8603f68640 1 Processor -- start 2024-03-19T02:03:28.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.529+0000 7f8603f68640 1 -- start start 2024-03-19T02:03:28.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.533+0000 7f8603f68640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f85fc070780 0x7f85fc1424e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:28.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.533+0000 7f8603f68640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f85fc0c9610 0x7f85fc140020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:28.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.533+0000 7f8603f68640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85fc152120 0x7f85fc156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:28.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.533+0000 7f8603f68640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f85fc0c86d0 con 0x7f85fc0c9610 2024-03-19T02:03:28.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.533+0000 7f8603f68640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f85fc0c31c0 con 0x7f85fc152120 2024-03-19T02:03:28.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.533+0000 7f8603f68640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f85fc165170 con 0x7f85fc070780 2024-03-19T02:03:28.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.533+0000 7f8601503640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f85fc0c9610 0x7f85fc140020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:28.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.533+0000 7f8601d04640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f85fc070780 0x7f85fc1424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:28.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.533+0000 7f8601503640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f85fc0c9610 0x7f85fc140020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47796/0 (socket says 172.21.15.55:47796) 2024-03-19T02:03:28.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.533+0000 7f8601503640 1 -- 172.21.15.55:0/1647484352 learned_addr learned my addr 172.21.15.55:0/1647484352 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:28.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.533+0000 7f8602505640 1 --2- 172.21.15.55:0/1647484352 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85fc152120 0x7f85fc156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:28.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.537+0000 7f8601503640 1 -- 172.21.15.55:0/1647484352 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f85fc070780 msgr2=0x7f85fc1424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:28.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.537+0000 7f8601503640 1 --2- 172.21.15.55:0/1647484352 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f85fc070780 0x7f85fc1424e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:28.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.537+0000 7f8601503640 1 -- 172.21.15.55:0/1647484352 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85fc152120 msgr2=0x7f85fc156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:28.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.537+0000 7f8601503640 1 --2- 172.21.15.55:0/1647484352 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85fc152120 0x7f85fc156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:28.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.537+0000 7f8601503640 1 -- 172.21.15.55:0/1647484352 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85fc142de0 con 0x7f85fc0c9610 2024-03-19T02:03:28.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.541+0000 7f8601503640 1 --2- 172.21.15.55:0/1647484352 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f85fc0c9610 0x7f85fc140020 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f85ec0ae040 tx=0x7f85ec019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:28.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.545+0000 7f85f27fc640 1 -- 172.21.15.55:0/1647484352 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ec0b0040 con 0x7f85fc0c9610 2024-03-19T02:03:28.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.545+0000 7f85f27fc640 1 -- 172.21.15.55:0/1647484352 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85ec0ab9b0 con 0x7f85fc0c9610 2024-03-19T02:03:28.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.545+0000 7f85f27fc640 1 -- 172.21.15.55:0/1647484352 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ec006e20 con 0x7f85fc0c9610 2024-03-19T02:03:28.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.545+0000 7f8603f68640 1 -- 172.21.15.55:0/1647484352 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85fc142b00 con 0x7f85fc0c9610 2024-03-19T02:03:28.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.545+0000 7f8603f68640 1 -- 172.21.15.55:0/1647484352 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85fc0cbcc0 con 0x7f85fc0c9610 2024-03-19T02:03:28.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.545+0000 7f8603f68640 1 -- 172.21.15.55:0/1647484352 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85fc068a40 con 0x7f85fc0c9610 2024-03-19T02:03:28.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.829+0000 7f85f27fc640 1 -- 172.21.15.55:0/1647484352 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f85ec0bb060 con 0x7f85fc0c9610 2024-03-19T02:03:28.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.829+0000 7f85f27fc640 1 --2- 172.21.15.55:0/1647484352 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f85e0067660 0x7f85e0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:28.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.829+0000 7f8601d04640 1 --2- 172.21.15.55:0/1647484352 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f85e0067660 0x7f85e0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:28.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.833+0000 7f8601d04640 1 --2- 172.21.15.55:0/1647484352 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f85e0067660 0x7f85e0069b20 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009ac0 tx=0x7f85f8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:28.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.845+0000 7f85f27fc640 1 -- 172.21.15.55:0/1647484352 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f85ec0ed320 con 0x7f85fc0c9610 2024-03-19T02:03:28.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:28.845+0000 7f85f27fc640 1 -- 172.21.15.55:0/1647484352 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f85ec016090 con 0x7f85fc0c9610 2024-03-19T02:03:29.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:29.101+0000 7f8603f68640 1 -- 172.21.15.55:0/1647484352 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85fc14cf00 con 0x7f85fc0c9610 2024-03-19T02:03:29.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:29.121+0000 7f85f27fc640 1 -- 172.21.15.55:0/1647484352 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f85ec0ef280 con 0x7f85fc0c9610 2024-03-19T02:03:29.124 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:29.124 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:29.124 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:29.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:29.125+0000 7f8603f68640 1 -- 172.21.15.55:0/1647484352 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f85e0067660 msgr2=0x7f85e0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:29.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:29.125+0000 7f8603f68640 1 --2- 172.21.15.55:0/1647484352 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f85e0067660 0x7f85e0069b20 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009ac0 tx=0x7f85f8009190 comp rx=0 tx=0).stop 2024-03-19T02:03:29.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:29.125+0000 7f8603f68640 1 -- 172.21.15.55:0/1647484352 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f85fc0c9610 msgr2=0x7f85fc140020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:29.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:29.125+0000 7f8603f68640 1 --2- 172.21.15.55:0/1647484352 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f85fc0c9610 0x7f85fc140020 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f85ec0ae040 tx=0x7f85ec019240 comp rx=0 tx=0).stop 2024-03-19T02:03:29.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:29.125+0000 7f8603f68640 1 -- 172.21.15.55:0/1647484352 shutdown_connections 2024-03-19T02:03:29.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:29.125+0000 7f8603f68640 1 --2- 172.21.15.55:0/1647484352 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f85e0067660 0x7f85e0069b20 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:29.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:29.125+0000 7f8603f68640 1 --2- 172.21.15.55:0/1647484352 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85fc152120 0x7f85fc156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:29.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:29.125+0000 7f8603f68640 1 --2- 172.21.15.55:0/1647484352 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f85fc0c9610 0x7f85fc140020 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:29.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:29.125+0000 7f8603f68640 1 --2- 172.21.15.55:0/1647484352 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f85fc070780 0x7f85fc1424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:29.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:29.125+0000 7f8603f68640 1 -- 172.21.15.55:0/1647484352 >> 172.21.15.55:0/1647484352 conn(0x7f85fc067600 msgr2=0x7f85fc0ca890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:29.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:29.125+0000 7f8603f68640 1 -- 172.21.15.55:0/1647484352 shutdown_connections 2024-03-19T02:03:29.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:29.125+0000 7f8603f68640 1 -- 172.21.15.55:0/1647484352 wait complete. 2024-03-19T02:03:29.747 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:29.747 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:30.748 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:30.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.877+0000 7f8c91a17640 1 Processor -- start 2024-03-19T02:03:30.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.877+0000 7f8c91a17640 1 -- start start 2024-03-19T02:03:30.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.877+0000 7f8c91a17640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8c8c14eaa0 0x7f8c8c06b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:30.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.877+0000 7f8c91a17640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f8c8c06bec0 0x7f8c8c0c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:30.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.877+0000 7f8c91a17640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c8c14ee80 con 0x7f8c8c0c6610 2024-03-19T02:03:30.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.877+0000 7f8c91a17640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f8c8c0c7540 con 0x7f8c8c06bec0 2024-03-19T02:03:30.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.877+0000 7f8c91a17640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f8c8c0c77c0 con 0x7f8c8c14eaa0 2024-03-19T02:03:30.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.877+0000 7f8c8a7fc640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f8c8c06bec0 0x7f8c8c0c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:30.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.881+0000 7f8c8affd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8c8c14eaa0 0x7f8c8c06b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:30.880 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.881+0000 7f8c8a7fc640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f8c8c06bec0 0x7f8c8c0c60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:48956/0 (socket says 172.21.15.55:48956) 2024-03-19T02:03:30.880 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.881+0000 7f8c8a7fc640 1 -- 172.21.15.55:0/1995825736 learned_addr learned my addr 172.21.15.55:0/1995825736 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:30.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.885+0000 7f8c8affd640 1 -- 172.21.15.55:0/1995825736 >> v2:172.21.15.84:3300/0 conn(0x7f8c8c06bec0 msgr2=0x7f8c8c0c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:30.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.885+0000 7f8c8affd640 1 --2- 172.21.15.55:0/1995825736 >> v2:172.21.15.84:3300/0 conn(0x7f8c8c06bec0 0x7f8c8c0c60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:30.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.885+0000 7f8c8affd640 1 -- 172.21.15.55:0/1995825736 >> v1:172.21.15.84:6789/0 conn(0x7f8c8c0c6610 legacy=0x7f8c8c1585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:30.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.885+0000 7f8c8affd640 1 -- 172.21.15.55:0/1995825736 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c8c0c7aa0 con 0x7f8c8c14eaa0 2024-03-19T02:03:30.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.889+0000 7f8c8affd640 1 --2- 172.21.15.55:0/1995825736 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8c8c14eaa0 0x7f8c8c06b980 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0097c0 tx=0x7f8c7c01c3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16a3b7205208e9c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:30.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.893+0000 7f8c89ffb640 1 -- 172.21.15.55:0/1995825736 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c0b0070 con 0x7f8c8c14eaa0 2024-03-19T02:03:30.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.893+0000 7f8c89ffb640 1 -- 172.21.15.55:0/1995825736 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c7c0ab600 con 0x7f8c8c14eaa0 2024-03-19T02:03:30.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.893+0000 7f8c89ffb640 1 -- 172.21.15.55:0/1995825736 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c0ab9d0 con 0x7f8c8c14eaa0 2024-03-19T02:03:30.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.893+0000 7f8c91a17640 1 -- 172.21.15.55:0/1995825736 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8c8c14eaa0 msgr2=0x7f8c8c06b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:30.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.893+0000 7f8c91a17640 1 --2- 172.21.15.55:0/1995825736 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8c8c14eaa0 0x7f8c8c06b980 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0097c0 tx=0x7f8c7c01c3d0 comp rx=0 tx=0).stop 2024-03-19T02:03:30.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.893+0000 7f8c91a17640 1 -- 172.21.15.55:0/1995825736 shutdown_connections 2024-03-19T02:03:30.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.893+0000 7f8c91a17640 1 --2- 172.21.15.55:0/1995825736 >> v2:172.21.15.84:3300/0 conn(0x7f8c8c06bec0 0x7f8c8c0c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:30.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.893+0000 7f8c91a17640 1 --2- 172.21.15.55:0/1995825736 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8c8c14eaa0 0x7f8c8c06b980 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:30.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.893+0000 7f8c91a17640 1 -- 172.21.15.55:0/1995825736 >> 172.21.15.55:0/1995825736 conn(0x7f8c8c0cc9c0 msgr2=0x7f8c8c0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:30.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.897+0000 7f8c91a17640 1 -- 172.21.15.55:0/1995825736 shutdown_connections 2024-03-19T02:03:30.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.897+0000 7f8c91a17640 1 -- 172.21.15.55:0/1995825736 wait complete. 2024-03-19T02:03:30.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.897+0000 7f8c91a17640 1 Processor -- start 2024-03-19T02:03:30.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.897+0000 7f8c91a17640 1 -- start start 2024-03-19T02:03:30.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.897+0000 7f8c91a17640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c8c06bec0 0x7f8c8c1540a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:30.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.897+0000 7f8c91a17640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8c8c0c6610 0x7f8c8c1f3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:30.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.897+0000 7f8c91a17640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c8c14eaa0 0x7f8c8c1f37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:30.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.897+0000 7f8c91a17640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8c0c6d70 con 0x7f8c8c14eaa0 2024-03-19T02:03:30.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.897+0000 7f8c91a17640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8c0c6bf0 con 0x7f8c8c06bec0 2024-03-19T02:03:30.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.897+0000 7f8c91a17640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f8c8c158d00 con 0x7f8c8c0c6610 2024-03-19T02:03:30.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.897+0000 7f8c8b7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c8c14eaa0 0x7f8c8c1f37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:30.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.897+0000 7f8c8affd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c8c06bec0 0x7f8c8c1540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:30.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.897+0000 7f8c8a7fc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8c8c0c6610 0x7f8c8c1f3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:30.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.897+0000 7f8c8b7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c8c14eaa0 0x7f8c8c1f37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41200/0 (socket says 172.21.15.55:41200) 2024-03-19T02:03:30.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.897+0000 7f8c8b7fe640 1 -- 172.21.15.55:0/3735260281 learned_addr learned my addr 172.21.15.55:0/3735260281 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:30.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.901+0000 7f8c8b7fe640 1 -- 172.21.15.55:0/3735260281 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8c8c0c6610 msgr2=0x7f8c8c1f3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:30.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.901+0000 7f8c8b7fe640 1 --2- 172.21.15.55:0/3735260281 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8c8c0c6610 0x7f8c8c1f3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:30.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.901+0000 7f8c8b7fe640 1 -- 172.21.15.55:0/3735260281 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c8c06bec0 msgr2=0x7f8c8c1540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:30.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.901+0000 7f8c8b7fe640 1 --2- 172.21.15.55:0/3735260281 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c8c06bec0 0x7f8c8c1540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:30.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.901+0000 7f8c8b7fe640 1 -- 172.21.15.55:0/3735260281 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c8c158840 con 0x7f8c8c14eaa0 2024-03-19T02:03:30.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.905+0000 7f8c8b7fe640 1 --2- 172.21.15.55:0/3735260281 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c8c14eaa0 0x7f8c8c1f37c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8c8400b990 tx=0x7f8c8400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:30.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.909+0000 7f8c73fff640 1 -- 172.21.15.55:0/3735260281 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c84019070 con 0x7f8c8c14eaa0 2024-03-19T02:03:30.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.909+0000 7f8c73fff640 1 -- 172.21.15.55:0/3735260281 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c8400cb40 con 0x7f8c8c14eaa0 2024-03-19T02:03:30.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.909+0000 7f8c91a17640 1 -- 172.21.15.55:0/3735260281 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c8c1f67f0 con 0x7f8c8c14eaa0 2024-03-19T02:03:30.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.909+0000 7f8c73fff640 1 -- 172.21.15.55:0/3735260281 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c84014440 con 0x7f8c8c14eaa0 2024-03-19T02:03:30.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.909+0000 7f8c91a17640 1 -- 172.21.15.55:0/3735260281 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c8c1f6db0 con 0x7f8c8c14eaa0 2024-03-19T02:03:30.910 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:30.909+0000 7f8c91a17640 1 -- 172.21.15.55:0/3735260281 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c8c068a40 con 0x7f8c8c14eaa0 2024-03-19T02:03:31.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:31.197+0000 7f8c73fff640 1 -- 172.21.15.55:0/3735260281 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f8c8401c070 con 0x7f8c8c14eaa0 2024-03-19T02:03:31.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:31.201+0000 7f8c73fff640 1 --2- 172.21.15.55:0/3735260281 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8c78067660 0x7f8c78069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:31.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:31.201+0000 7f8c8affd640 1 --2- 172.21.15.55:0/3735260281 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8c78067660 0x7f8c78069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:31.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:31.201+0000 7f8c8affd640 1 --2- 172.21.15.55:0/3735260281 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8c78067660 0x7f8c78069b20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0ad000 tx=0x7f8c7c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:31.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:31.221+0000 7f8c73fff640 1 -- 172.21.15.55:0/3735260281 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f8c84054400 con 0x7f8c8c14eaa0 2024-03-19T02:03:31.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:31.221+0000 7f8c73fff640 1 -- 172.21.15.55:0/3735260281 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f8c84023070 con 0x7f8c8c14eaa0 2024-03-19T02:03:31.482 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:31.481+0000 7f8c91a17640 1 -- 172.21.15.55:0/3735260281 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c8c1558e0 con 0x7f8c8c14eaa0 2024-03-19T02:03:31.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:31.561+0000 7f8c73fff640 1 -- 172.21.15.55:0/3735260281 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f8c84056360 con 0x7f8c8c14eaa0 2024-03-19T02:03:31.560 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:31.560 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:31.561 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:31.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:31.561+0000 7f8c91a17640 1 -- 172.21.15.55:0/3735260281 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8c78067660 msgr2=0x7f8c78069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:31.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:31.561+0000 7f8c91a17640 1 --2- 172.21.15.55:0/3735260281 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8c78067660 0x7f8c78069b20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0ad000 tx=0x7f8c7c0a6000 comp rx=0 tx=0).stop 2024-03-19T02:03:31.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:31.561+0000 7f8c91a17640 1 -- 172.21.15.55:0/3735260281 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c8c14eaa0 msgr2=0x7f8c8c1f37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:31.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:31.561+0000 7f8c91a17640 1 --2- 172.21.15.55:0/3735260281 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c8c14eaa0 0x7f8c8c1f37c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8c8400b990 tx=0x7f8c8400be50 comp rx=0 tx=0).stop 2024-03-19T02:03:31.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:31.561+0000 7f8c91a17640 1 -- 172.21.15.55:0/3735260281 shutdown_connections 2024-03-19T02:03:31.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:31.561+0000 7f8c91a17640 1 --2- 172.21.15.55:0/3735260281 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8c78067660 0x7f8c78069b20 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:31.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:31.561+0000 7f8c91a17640 1 --2- 172.21.15.55:0/3735260281 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c8c14eaa0 0x7f8c8c1f37c0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:31.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:31.561+0000 7f8c91a17640 1 --2- 172.21.15.55:0/3735260281 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8c8c0c6610 0x7f8c8c1f3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:31.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:31.561+0000 7f8c91a17640 1 --2- 172.21.15.55:0/3735260281 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c8c06bec0 0x7f8c8c1540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:31.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:31.561+0000 7f8c91a17640 1 -- 172.21.15.55:0/3735260281 >> 172.21.15.55:0/3735260281 conn(0x7f8c8c0cc9c0 msgr2=0x7f8c8c070100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:31.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:31.561+0000 7f8c91a17640 1 -- 172.21.15.55:0/3735260281 shutdown_connections 2024-03-19T02:03:31.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:31.561+0000 7f8c91a17640 1 -- 172.21.15.55:0/3735260281 wait complete. 2024-03-19T02:03:32.190 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:32.190 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:33.191 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:33.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.305+0000 7f49f3c73640 1 Processor -- start 2024-03-19T02:03:33.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.305+0000 7f49f3c73640 1 -- start start 2024-03-19T02:03:33.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.305+0000 7f49f3c73640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f49ec14e930 0x7f49ec06b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:33.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.305+0000 7f49f3c73640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49ec0c7540 con 0x7f49ec06be50 2024-03-19T02:03:33.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.305+0000 7f49f3c73640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49ec0c7800 con 0x7f49ec0c60d0 2024-03-19T02:03:33.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.305+0000 7f49f3c73640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f49ec0c80a0 con 0x7f49ec14e930 2024-03-19T02:03:33.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.305+0000 7f49f1a0f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f49ec14e930 0x7f49ec06b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:33.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.309+0000 7f49f2210640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f49ec0c60d0 0x7f49ec162860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:34328/0 (socket says 172.21.15.55:34328) 2024-03-19T02:03:33.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.309+0000 7f49f2210640 1 -- 172.21.15.55:0/2772552104 learned_addr learned my addr 172.21.15.55:0/2772552104 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:03:33.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.309+0000 7f49f1a0f640 1 -- 172.21.15.55:0/2772552104 >> v1:172.21.15.84:6789/0 conn(0x7f49ec0c60d0 legacy=0x7f49ec162860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:33.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.309+0000 7f49f1a0f640 1 -- 172.21.15.55:0/2772552104 >> v1:172.21.15.55:6789/0 conn(0x7f49ec06be50 legacy=0x7f49ec1585a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:33.311 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.313+0000 7f49f1a0f640 1 -- 172.21.15.55:0/2772552104 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ec0c7ae0 con 0x7f49ec14e930 2024-03-19T02:03:33.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.317+0000 7f49f1a0f640 1 --2- 172.21.15.55:0/2772552104 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49ec14e930 0x7f49ec06b910 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f49e80097c0 tx=0x7f49e801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e1e9937dd4052bc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:33.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.321+0000 7f49f0a0d640 1 -- 172.21.15.55:0/2772552104 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e80b5070 con 0x7f49ec14e930 2024-03-19T02:03:33.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.321+0000 7f49f0a0d640 1 -- 172.21.15.55:0/2772552104 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49e80a8950 con 0x7f49ec14e930 2024-03-19T02:03:33.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.321+0000 7f49f0a0d640 1 -- 172.21.15.55:0/2772552104 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e80a8d20 con 0x7f49ec14e930 2024-03-19T02:03:33.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.321+0000 7f49f3c73640 1 -- 172.21.15.55:0/2772552104 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49ec14e930 msgr2=0x7f49ec06b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:33.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.321+0000 7f49f3c73640 1 --2- 172.21.15.55:0/2772552104 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49ec14e930 0x7f49ec06b910 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f49e80097c0 tx=0x7f49e801ec50 comp rx=0 tx=0).stop 2024-03-19T02:03:33.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.321+0000 7f49f3c73640 1 -- 172.21.15.55:0/2772552104 shutdown_connections 2024-03-19T02:03:33.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.321+0000 7f49f3c73640 1 --2- 172.21.15.55:0/2772552104 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49ec14e930 0x7f49ec06b910 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:33.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.321+0000 7f49f3c73640 1 -- 172.21.15.55:0/2772552104 >> 172.21.15.55:0/2772552104 conn(0x7f49ec0cc9c0 msgr2=0x7f49ec0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:33.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.321+0000 7f49f3c73640 1 -- 172.21.15.55:0/2772552104 shutdown_connections 2024-03-19T02:03:33.322 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.321+0000 7f49f3c73640 1 -- 172.21.15.55:0/2772552104 wait complete. 2024-03-19T02:03:33.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.325+0000 7f49f3c73640 1 Processor -- start 2024-03-19T02:03:33.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.325+0000 7f49f3c73640 1 -- start start 2024-03-19T02:03:33.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.325+0000 7f49f3c73640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f49ec06be50 0x7f49ec0c5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:33.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.325+0000 7f49f3c73640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49ec0c60d0 0x7f49ec15b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:33.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.325+0000 7f49f3c73640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49ec14e930 0x7f49ec148f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:33.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.325+0000 7f49f3c73640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec14ed10 con 0x7f49ec14e930 2024-03-19T02:03:33.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.325+0000 7f49f3c73640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec158080 con 0x7f49ec06be50 2024-03-19T02:03:33.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.325+0000 7f49f3c73640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f49ec073600 con 0x7f49ec0c60d0 2024-03-19T02:03:33.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.325+0000 7f49f2210640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49ec14e930 0x7f49ec148f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:33.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.325+0000 7f49f1a0f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f49ec06be50 0x7f49ec0c5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:33.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.325+0000 7f49f2210640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49ec14e930 0x7f49ec148f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41220/0 (socket says 172.21.15.55:41220) 2024-03-19T02:03:33.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.325+0000 7f49f2210640 1 -- 172.21.15.55:0/717746232 learned_addr learned my addr 172.21.15.55:0/717746232 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:33.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.325+0000 7f49f120e640 1 --2- 172.21.15.55:0/717746232 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49ec0c60d0 0x7f49ec15b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:33.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.329+0000 7f49f2210640 1 -- 172.21.15.55:0/717746232 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49ec0c60d0 msgr2=0x7f49ec15b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:33.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.329+0000 7f49f2210640 1 --2- 172.21.15.55:0/717746232 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49ec0c60d0 0x7f49ec15b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:33.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.329+0000 7f49f2210640 1 -- 172.21.15.55:0/717746232 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f49ec06be50 msgr2=0x7f49ec0c5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:33.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.329+0000 7f49f2210640 1 --2- 172.21.15.55:0/717746232 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f49ec06be50 0x7f49ec0c5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:33.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.329+0000 7f49f2210640 1 -- 172.21.15.55:0/717746232 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ec0c4300 con 0x7f49ec14e930 2024-03-19T02:03:33.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.333+0000 7f49f2210640 1 --2- 172.21.15.55:0/717746232 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49ec14e930 0x7f49ec148f80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f49e400cbc0 tx=0x7f49e400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:33.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.337+0000 7f49e27fc640 1 -- 172.21.15.55:0/717746232 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e4017070 con 0x7f49ec14e930 2024-03-19T02:03:33.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.337+0000 7f49e27fc640 1 -- 172.21.15.55:0/717746232 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49e400ad40 con 0x7f49ec14e930 2024-03-19T02:03:33.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.337+0000 7f49e27fc640 1 -- 172.21.15.55:0/717746232 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e4012600 con 0x7f49ec14e930 2024-03-19T02:03:33.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.337+0000 7f49f3c73640 1 -- 172.21.15.55:0/717746232 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49ec0c5b40 con 0x7f49ec14e930 2024-03-19T02:03:33.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.337+0000 7f49f3c73640 1 -- 172.21.15.55:0/717746232 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49ec1f1180 con 0x7f49ec14e930 2024-03-19T02:03:33.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.337+0000 7f49f3c73640 1 -- 172.21.15.55:0/717746232 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49ec068a40 con 0x7f49ec14e930 2024-03-19T02:03:33.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.649+0000 7f49e27fc640 1 -- 172.21.15.55:0/717746232 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f49e401a070 con 0x7f49ec14e930 2024-03-19T02:03:33.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.653+0000 7f49e27fc640 1 --2- 172.21.15.55:0/717746232 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f49d0067660 0x7f49d0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:33.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.653+0000 7f49f1a0f640 1 --2- 172.21.15.55:0/717746232 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f49d0067660 0x7f49d0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:33.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.653+0000 7f49f1a0f640 1 --2- 172.21.15.55:0/717746232 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f49d0067660 0x7f49d0069b20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f49e80b2000 tx=0x7f49e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:33.691 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.693+0000 7f49e27fc640 1 -- 172.21.15.55:0/717746232 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f49e4054040 con 0x7f49ec14e930 2024-03-19T02:03:33.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.693+0000 7f49e27fc640 1 -- 172.21.15.55:0/717746232 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f49e4055fa0 con 0x7f49ec14e930 2024-03-19T02:03:33.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.949+0000 7f49f3c73640 1 -- 172.21.15.55:0/717746232 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49ec15c370 con 0x7f49ec14e930 2024-03-19T02:03:33.969 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.969+0000 7f49e27fc640 1 -- 172.21.15.55:0/717746232 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f49e405a040 con 0x7f49ec14e930 2024-03-19T02:03:33.970 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:33.970 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:33.970 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:33.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.973+0000 7f49f3c73640 1 -- 172.21.15.55:0/717746232 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f49d0067660 msgr2=0x7f49d0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:33.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.973+0000 7f49f3c73640 1 --2- 172.21.15.55:0/717746232 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f49d0067660 0x7f49d0069b20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f49e80b2000 tx=0x7f49e8008040 comp rx=0 tx=0).stop 2024-03-19T02:03:33.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.973+0000 7f49f3c73640 1 -- 172.21.15.55:0/717746232 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49ec14e930 msgr2=0x7f49ec148f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:33.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.973+0000 7f49f3c73640 1 --2- 172.21.15.55:0/717746232 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49ec14e930 0x7f49ec148f80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f49e400cbc0 tx=0x7f49e400a430 comp rx=0 tx=0).stop 2024-03-19T02:03:33.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.973+0000 7f49f3c73640 1 -- 172.21.15.55:0/717746232 shutdown_connections 2024-03-19T02:03:33.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.973+0000 7f49f3c73640 1 --2- 172.21.15.55:0/717746232 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f49d0067660 0x7f49d0069b20 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:33.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.973+0000 7f49f3c73640 1 --2- 172.21.15.55:0/717746232 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49ec14e930 0x7f49ec148f80 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:33.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.973+0000 7f49f3c73640 1 --2- 172.21.15.55:0/717746232 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49ec0c60d0 0x7f49ec15b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:33.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.973+0000 7f49f3c73640 1 --2- 172.21.15.55:0/717746232 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f49ec06be50 0x7f49ec0c5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:33.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.973+0000 7f49f3c73640 1 -- 172.21.15.55:0/717746232 >> 172.21.15.55:0/717746232 conn(0x7f49ec0cc9c0 msgr2=0x7f49ec14cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:33.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.973+0000 7f49f3c73640 1 -- 172.21.15.55:0/717746232 shutdown_connections 2024-03-19T02:03:33.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:33.973+0000 7f49f3c73640 1 -- 172.21.15.55:0/717746232 wait complete. 2024-03-19T02:03:34.592 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:34.592 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:35.593 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:35.714 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.713+0000 7f5a1b7f2640 1 Processor -- start 2024-03-19T02:03:35.714 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.713+0000 7f5a1b7f2640 1 -- start start 2024-03-19T02:03:35.714 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.713+0000 7f5a1b7f2640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f5a14154340 0x7f5a1415e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:35.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.713+0000 7f5a1b7f2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a14154720 0x7f5a140c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:35.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.713+0000 7f5a1b7f2640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f5a1415aca0 0x7f5a14072e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:35.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.713+0000 7f5a1b7f2640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f5a14073840 con 0x7f5a1415aca0 2024-03-19T02:03:35.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.713+0000 7f5a1b7f2640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f5a1415df10 con 0x7f5a14154340 2024-03-19T02:03:35.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.713+0000 7f5a1b7f2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f5a14071280 con 0x7f5a14154720 2024-03-19T02:03:35.716 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.717+0000 7f5a19d8f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f5a1415aca0 0x7f5a14072e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:35.716 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.717+0000 7f5a18d8d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a14154720 0x7f5a140c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:35.716 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.717+0000 7f5a1958e640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f5a14154340 0x7f5a1415e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:35.716 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.717+0000 7f5a19d8f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f5a1415aca0 0x7f5a14072e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41226/0 (socket says 172.21.15.55:41226) 2024-03-19T02:03:35.716 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.717+0000 7f5a19d8f640 1 -- 172.21.15.55:0/3643286903 learned_addr learned my addr 172.21.15.55:0/3643286903 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:35.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.717+0000 7f5a19d8f640 1 -- 172.21.15.55:0/3643286903 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a14154720 msgr2=0x7f5a140c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:35.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.717+0000 7f5a19d8f640 1 --2- 172.21.15.55:0/3643286903 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a14154720 0x7f5a140c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:35.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.717+0000 7f5a19d8f640 1 -- 172.21.15.55:0/3643286903 >> v2:172.21.15.84:3300/0 conn(0x7f5a14154340 msgr2=0x7f5a1415e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:35.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.717+0000 7f5a19d8f640 1 --2- 172.21.15.55:0/3643286903 >> v2:172.21.15.84:3300/0 conn(0x7f5a14154340 0x7f5a1415e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:35.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.717+0000 7f5a19d8f640 1 -- 172.21.15.55:0/3643286903 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a1415d900 con 0x7f5a1415aca0 2024-03-19T02:03:35.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.721+0000 7f5a19d8f640 1 --2- 172.21.15.55:0/3643286903 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a1415aca0 0x7f5a14072e90 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f5a140ca170 tx=0x7f5a0c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f3f9af814b0570a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:35.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.725+0000 7f5a0bfff640 1 -- 172.21.15.55:0/3643286903 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c0a5760 con 0x7f5a1415aca0 2024-03-19T02:03:35.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.729+0000 7f5a0bfff640 1 -- 172.21.15.55:0/3643286903 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a0c0a5900 con 0x7f5a1415aca0 2024-03-19T02:03:35.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.729+0000 7f5a0bfff640 1 -- 172.21.15.55:0/3643286903 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c0a5cd0 con 0x7f5a1415aca0 2024-03-19T02:03:35.729 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.729+0000 7f5a1b7f2640 1 -- 172.21.15.55:0/3643286903 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a1415aca0 msgr2=0x7f5a14072e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:35.729 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.729+0000 7f5a1b7f2640 1 --2- 172.21.15.55:0/3643286903 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a1415aca0 0x7f5a14072e90 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f5a140ca170 tx=0x7f5a0c01c9f0 comp rx=0 tx=0).stop 2024-03-19T02:03:35.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.729+0000 7f5a1b7f2640 1 -- 172.21.15.55:0/3643286903 shutdown_connections 2024-03-19T02:03:35.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.729+0000 7f5a1b7f2640 1 --2- 172.21.15.55:0/3643286903 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a14154720 0x7f5a140c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:35.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.729+0000 7f5a1b7f2640 1 --2- 172.21.15.55:0/3643286903 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a1415aca0 0x7f5a14072e90 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:35.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.729+0000 7f5a1b7f2640 1 --2- 172.21.15.55:0/3643286903 >> v2:172.21.15.84:3300/0 conn(0x7f5a14154340 0x7f5a1415e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:35.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.729+0000 7f5a1b7f2640 1 -- 172.21.15.55:0/3643286903 >> 172.21.15.55:0/3643286903 conn(0x7f5a140cc9c0 msgr2=0x7f5a140ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:35.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.729+0000 7f5a1b7f2640 1 -- 172.21.15.55:0/3643286903 shutdown_connections 2024-03-19T02:03:35.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.729+0000 7f5a1b7f2640 1 -- 172.21.15.55:0/3643286903 wait complete. 2024-03-19T02:03:35.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.733+0000 7f5a1b7f2640 1 Processor -- start 2024-03-19T02:03:35.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.733+0000 7f5a1b7f2640 1 -- start start 2024-03-19T02:03:35.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.733+0000 7f5a1b7f2640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a14154340 0x7f5a140c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:35.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.733+0000 7f5a1b7f2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a14154720 0x7f5a1414c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:35.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.733+0000 7f5a1b7f2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a1415aca0 0x7f5a140c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:35.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.733+0000 7f5a1b7f2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5a1415b100 con 0x7f5a1415aca0 2024-03-19T02:03:35.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.733+0000 7f5a1b7f2640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f5a140733d0 con 0x7f5a14154340 2024-03-19T02:03:35.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.733+0000 7f5a1b7f2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f5a1406b2a0 con 0x7f5a14154720 2024-03-19T02:03:35.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.733+0000 7f5a19d8f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a1415aca0 0x7f5a140c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:35.733 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.733+0000 7f5a18d8d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a14154720 0x7f5a1414c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:35.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.733+0000 7f5a18d8d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a14154720 0x7f5a1414c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:45266/0 (socket says 172.21.15.55:45266) 2024-03-19T02:03:35.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.733+0000 7f5a18d8d640 1 -- 172.21.15.55:0/1838632411 learned_addr learned my addr 172.21.15.55:0/1838632411 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:35.734 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.733+0000 7f5a1958e640 1 --2- 172.21.15.55:0/1838632411 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a14154340 0x7f5a140c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:35.739 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.737+0000 7f5a19d8f640 1 -- 172.21.15.55:0/1838632411 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a14154720 msgr2=0x7f5a1414c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:35.739 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.737+0000 7f5a19d8f640 1 --2- 172.21.15.55:0/1838632411 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a14154720 0x7f5a1414c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:35.739 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.737+0000 7f5a19d8f640 1 -- 172.21.15.55:0/1838632411 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a14154340 msgr2=0x7f5a140c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:35.739 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.737+0000 7f5a19d8f640 1 --2- 172.21.15.55:0/1838632411 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a14154340 0x7f5a140c70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:35.739 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.737+0000 7f5a19d8f640 1 -- 172.21.15.55:0/1838632411 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a141f3bb0 con 0x7f5a1415aca0 2024-03-19T02:03:35.742 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.741+0000 7f5a19d8f640 1 --2- 172.21.15.55:0/1838632411 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a1415aca0 0x7f5a140c7b40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c0ab000 tx=0x7f5a0c019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:35.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.745+0000 7f5a09ffb640 1 -- 172.21.15.55:0/1838632411 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c0bc070 con 0x7f5a1415aca0 2024-03-19T02:03:35.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.745+0000 7f5a09ffb640 1 -- 172.21.15.55:0/1838632411 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a0c0b8640 con 0x7f5a1415aca0 2024-03-19T02:03:35.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.745+0000 7f5a09ffb640 1 -- 172.21.15.55:0/1838632411 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c0b8a10 con 0x7f5a1415aca0 2024-03-19T02:03:35.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.745+0000 7f5a1b7f2640 1 -- 172.21.15.55:0/1838632411 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a140c75f0 con 0x7f5a1415aca0 2024-03-19T02:03:35.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.745+0000 7f5a1b7f2640 1 -- 172.21.15.55:0/1838632411 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a1414d250 con 0x7f5a1415aca0 2024-03-19T02:03:35.747 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:35.749+0000 7f5a1b7f2640 1 -- 172.21.15.55:0/1838632411 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a1414d420 con 0x7f5a1415aca0 2024-03-19T02:03:36.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:36.025+0000 7f5a09ffb640 1 -- 172.21.15.55:0/1838632411 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f5a0c0b8bb0 con 0x7f5a1415aca0 2024-03-19T02:03:36.025 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:36.025+0000 7f5a09ffb640 1 --2- 172.21.15.55:0/1838632411 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f59f8067660 0x7f59f8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:36.025 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:36.025+0000 7f5a1958e640 1 --2- 172.21.15.55:0/1838632411 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f59f8067660 0x7f59f8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:36.026 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:36.025+0000 7f5a1958e640 1 --2- 172.21.15.55:0/1838632411 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f59f8067660 0x7f59f8069b20 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5a100045f0 tx=0x7f5a10008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:36.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:36.053+0000 7f5a09ffb640 1 -- 172.21.15.55:0/1838632411 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f5a0c0eedb0 con 0x7f5a1415aca0 2024-03-19T02:03:36.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:36.053+0000 7f5a09ffb640 1 -- 172.21.15.55:0/1838632411 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f5a0c0165e0 con 0x7f5a1415aca0 2024-03-19T02:03:36.269 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:36.269+0000 7f5a1b7f2640 1 -- 172.21.15.55:0/1838632411 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a14068a40 con 0x7f5a1415aca0 2024-03-19T02:03:36.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:36.285+0000 7f5a09ffb640 1 -- 172.21.15.55:0/1838632411 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f5a0c0f0d10 con 0x7f5a1415aca0 2024-03-19T02:03:36.287 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:36.287 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:36.288 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:36.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:36.289+0000 7f5a1b7f2640 1 -- 172.21.15.55:0/1838632411 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f59f8067660 msgr2=0x7f59f8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:36.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:36.289+0000 7f5a1b7f2640 1 --2- 172.21.15.55:0/1838632411 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f59f8067660 0x7f59f8069b20 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5a100045f0 tx=0x7f5a10008040 comp rx=0 tx=0).stop 2024-03-19T02:03:36.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:36.293+0000 7f5a1b7f2640 1 -- 172.21.15.55:0/1838632411 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a1415aca0 msgr2=0x7f5a140c7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:36.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:36.293+0000 7f5a1b7f2640 1 --2- 172.21.15.55:0/1838632411 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a1415aca0 0x7f5a140c7b40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c0ab000 tx=0x7f5a0c019790 comp rx=0 tx=0).stop 2024-03-19T02:03:36.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:36.293+0000 7f5a1b7f2640 1 -- 172.21.15.55:0/1838632411 shutdown_connections 2024-03-19T02:03:36.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:36.293+0000 7f5a1b7f2640 1 --2- 172.21.15.55:0/1838632411 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f59f8067660 0x7f59f8069b20 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:36.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:36.293+0000 7f5a1b7f2640 1 --2- 172.21.15.55:0/1838632411 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5a1415aca0 0x7f5a140c7b40 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:36.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:36.293+0000 7f5a1b7f2640 1 --2- 172.21.15.55:0/1838632411 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5a14154720 0x7f5a1414c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:36.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:36.293+0000 7f5a1b7f2640 1 --2- 172.21.15.55:0/1838632411 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a14154340 0x7f5a140c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:36.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:36.293+0000 7f5a1b7f2640 1 -- 172.21.15.55:0/1838632411 >> 172.21.15.55:0/1838632411 conn(0x7f5a140cc9c0 msgr2=0x7f5a14153000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:36.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:36.293+0000 7f5a1b7f2640 1 -- 172.21.15.55:0/1838632411 shutdown_connections 2024-03-19T02:03:36.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:36.293+0000 7f5a1b7f2640 1 -- 172.21.15.55:0/1838632411 wait complete. 2024-03-19T02:03:36.912 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:36.913 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:37.913 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:38.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.033+0000 7fbfb07e4640 1 Processor -- start 2024-03-19T02:03:38.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.033+0000 7fbfb07e4640 1 -- start start 2024-03-19T02:03:38.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.033+0000 7fbfb07e4640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fbfa80c60d0 0x7fbfa80c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:38.035 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.033+0000 7fbfb07e4640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbfa81585f0 0x7fbfa8158a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:38.035 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.033+0000 7fbfb07e4640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfa814ee80 con 0x7fbfa814eaa0 2024-03-19T02:03:38.035 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.033+0000 7fbfb07e4640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fbfa8158f50 con 0x7fbfa80c60d0 2024-03-19T02:03:38.035 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.033+0000 7fbfb07e4640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbfa80c7540 con 0x7fbfa81585f0 2024-03-19T02:03:38.035 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.037+0000 7fbfae580640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fbfa814eaa0 0x7fbfa806b980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:47662/0 (socket says 172.21.15.55:47662) 2024-03-19T02:03:38.035 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.037+0000 7fbfae580640 1 -- 172.21.15.55:0/2340485126 learned_addr learned my addr 172.21.15.55:0/2340485126 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:03:38.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.037+0000 7fbfadd7f640 1 --2- 172.21.15.55:0/2340485126 >> v2:172.21.15.84:3300/0 conn(0x7fbfa80c60d0 0x7fbfa80c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:38.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.037+0000 7fbfaed81640 1 --2- 172.21.15.55:0/2340485126 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbfa81585f0 0x7fbfa8158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:38.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.037+0000 7fbfad57e640 1 -- 172.21.15.55:0/2340485126 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4204108139 0 0) 0x7fbfa814ee80 con 0x7fbfa814eaa0 2024-03-19T02:03:38.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.037+0000 7fbfad57e640 1 -- 172.21.15.55:0/2340485126 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf94003530 con 0x7fbfa814eaa0 2024-03-19T02:03:38.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.041+0000 7fbfadd7f640 1 -- 172.21.15.55:0/2340485126 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbfa81585f0 msgr2=0x7fbfa8158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:38.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.041+0000 7fbfadd7f640 1 --2- 172.21.15.55:0/2340485126 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbfa81585f0 0x7fbfa8158a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:38.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.041+0000 7fbfadd7f640 1 -- 172.21.15.55:0/2340485126 >> v1:172.21.15.55:6789/0 conn(0x7fbfa814eaa0 legacy=0x7fbfa806b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:38.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.041+0000 7fbfadd7f640 1 -- 172.21.15.55:0/2340485126 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa80c77c0 con 0x7fbfa80c60d0 2024-03-19T02:03:38.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.045+0000 7fbfadd7f640 1 --2- 172.21.15.55:0/2340485126 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbfa80c60d0 0x7fbfa80c6540 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbf980097c0 tx=0x7fbf9801c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23830cfd5387dfdb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:38.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.049+0000 7fbfad57e640 1 -- 172.21.15.55:0/2340485126 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf980ab610 con 0x7fbfa80c60d0 2024-03-19T02:03:38.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.049+0000 7fbfad57e640 1 -- 172.21.15.55:0/2340485126 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf980b0070 con 0x7fbfa80c60d0 2024-03-19T02:03:38.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.049+0000 7fbfad57e640 1 -- 172.21.15.55:0/2340485126 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf980ab9b0 con 0x7fbfa80c60d0 2024-03-19T02:03:38.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.049+0000 7fbfb07e4640 1 -- 172.21.15.55:0/2340485126 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbfa80c60d0 msgr2=0x7fbfa80c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:38.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.049+0000 7fbfb07e4640 1 --2- 172.21.15.55:0/2340485126 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbfa80c60d0 0x7fbfa80c6540 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbf980097c0 tx=0x7fbf9801c410 comp rx=0 tx=0).stop 2024-03-19T02:03:38.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.049+0000 7fbfb07e4640 1 -- 172.21.15.55:0/2340485126 shutdown_connections 2024-03-19T02:03:38.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.049+0000 7fbfb07e4640 1 --2- 172.21.15.55:0/2340485126 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbfa81585f0 0x7fbfa8158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:38.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.049+0000 7fbfb07e4640 1 --2- 172.21.15.55:0/2340485126 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbfa80c60d0 0x7fbfa80c6540 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:38.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.049+0000 7fbfb07e4640 1 -- 172.21.15.55:0/2340485126 >> 172.21.15.55:0/2340485126 conn(0x7fbfa80cc9c0 msgr2=0x7fbfa80ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:38.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.049+0000 7fbfb07e4640 1 -- 172.21.15.55:0/2340485126 shutdown_connections 2024-03-19T02:03:38.051 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.049+0000 7fbfb07e4640 1 -- 172.21.15.55:0/2340485126 wait complete. 2024-03-19T02:03:38.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.053+0000 7fbfb07e4640 1 Processor -- start 2024-03-19T02:03:38.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.053+0000 7fbfb07e4640 1 -- start start 2024-03-19T02:03:38.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.053+0000 7fbfb07e4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfa80c60d0 0x7fbfa8142650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:38.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.053+0000 7fbfb07e4640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbfa814eaa0 0x7fbfa814bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:38.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.053+0000 7fbfb07e4640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbfa81585f0 0x7fbfa81451a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:38.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.053+0000 7fbfb07e4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa806c090 con 0x7fbfa80c60d0 2024-03-19T02:03:38.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.053+0000 7fbfb07e4640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa80c8060 con 0x7fbfa81585f0 2024-03-19T02:03:38.053 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.053+0000 7fbfb07e4640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbfa806c210 con 0x7fbfa814eaa0 2024-03-19T02:03:38.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.053+0000 7fbfae580640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfa80c60d0 0x7fbfa8142650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:38.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.053+0000 7fbfaed81640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbfa81585f0 0x7fbfa81451a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:38.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.053+0000 7fbfadd7f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbfa814eaa0 0x7fbfa814bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:38.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.053+0000 7fbfae580640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfa80c60d0 0x7fbfa8142650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41252/0 (socket says 172.21.15.55:41252) 2024-03-19T02:03:38.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.053+0000 7fbfae580640 1 -- 172.21.15.55:0/1541374263 learned_addr learned my addr 172.21.15.55:0/1541374263 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:38.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.057+0000 7fbfae580640 1 -- 172.21.15.55:0/1541374263 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbfa814eaa0 msgr2=0x7fbfa814bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:38.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.057+0000 7fbfae580640 1 --2- 172.21.15.55:0/1541374263 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbfa814eaa0 0x7fbfa814bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:38.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.057+0000 7fbfae580640 1 -- 172.21.15.55:0/1541374263 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbfa81585f0 msgr2=0x7fbfa81451a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:38.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.057+0000 7fbfae580640 1 --2- 172.21.15.55:0/1541374263 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbfa81585f0 0x7fbfa81451a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:38.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.057+0000 7fbfae580640 1 -- 172.21.15.55:0/1541374263 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa814c730 con 0x7fbfa80c60d0 2024-03-19T02:03:38.060 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.061+0000 7fbfae580640 1 --2- 172.21.15.55:0/1541374263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfa80c60d0 0x7fbfa8142650 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fbfa400c3c0 tx=0x7fbfa400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:38.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.065+0000 7fbf9effd640 1 -- 172.21.15.55:0/1541374263 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4017070 con 0x7fbfa80c60d0 2024-03-19T02:03:38.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.065+0000 7fbf9effd640 1 -- 172.21.15.55:0/1541374263 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfa4005cf0 con 0x7fbfa80c60d0 2024-03-19T02:03:38.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.065+0000 7fbf9effd640 1 -- 172.21.15.55:0/1541374263 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa40072f0 con 0x7fbfa80c60d0 2024-03-19T02:03:38.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.065+0000 7fbfb07e4640 1 -- 172.21.15.55:0/1541374263 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa8161580 con 0x7fbfa80c60d0 2024-03-19T02:03:38.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.065+0000 7fbfb07e4640 1 -- 172.21.15.55:0/1541374263 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa8161b40 con 0x7fbfa80c60d0 2024-03-19T02:03:38.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.065+0000 7fbfb07e4640 1 -- 172.21.15.55:0/1541374263 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfa8068a40 con 0x7fbfa80c60d0 2024-03-19T02:03:38.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.097+0000 7fbf9effd640 1 -- 172.21.15.55:0/1541374263 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fbfa401a070 con 0x7fbfa80c60d0 2024-03-19T02:03:38.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.097+0000 7fbf9effd640 1 --2- 172.21.15.55:0/1541374263 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbf8c0676e0 0x7fbf8c069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:38.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.097+0000 7fbfadd7f640 1 --2- 172.21.15.55:0/1541374263 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbf8c0676e0 0x7fbf8c069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:38.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.097+0000 7fbfadd7f640 1 --2- 172.21.15.55:0/1541374263 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbf8c0676e0 0x7fbf8c069ba0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbf980af000 tx=0x7fbf980b6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:38.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.397+0000 7fbf9effd640 1 -- 172.21.15.55:0/1541374263 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fbfa4053720 con 0x7fbfa80c60d0 2024-03-19T02:03:38.400 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.401+0000 7fbf9effd640 1 -- 172.21.15.55:0/1541374263 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fbfa4021020 con 0x7fbfa80c60d0 2024-03-19T02:03:38.610 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.609+0000 7fbfb07e4640 1 -- 172.21.15.55:0/1541374263 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbfa81558e0 con 0x7fbfa80c60d0 2024-03-19T02:03:38.642 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.641+0000 7fbf9effd640 1 -- 172.21.15.55:0/1541374263 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fbfa4055680 con 0x7fbfa80c60d0 2024-03-19T02:03:38.642 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:38.642 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:38.642 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:38.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.645+0000 7fbfb07e4640 1 -- 172.21.15.55:0/1541374263 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbf8c0676e0 msgr2=0x7fbf8c069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:38.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.645+0000 7fbfb07e4640 1 --2- 172.21.15.55:0/1541374263 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbf8c0676e0 0x7fbf8c069ba0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbf980af000 tx=0x7fbf980b6040 comp rx=0 tx=0).stop 2024-03-19T02:03:38.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.645+0000 7fbfb07e4640 1 -- 172.21.15.55:0/1541374263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfa80c60d0 msgr2=0x7fbfa8142650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:38.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.645+0000 7fbfb07e4640 1 --2- 172.21.15.55:0/1541374263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfa80c60d0 0x7fbfa8142650 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fbfa400c3c0 tx=0x7fbfa400c880 comp rx=0 tx=0).stop 2024-03-19T02:03:38.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.645+0000 7fbfb07e4640 1 -- 172.21.15.55:0/1541374263 shutdown_connections 2024-03-19T02:03:38.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.645+0000 7fbfb07e4640 1 --2- 172.21.15.55:0/1541374263 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbf8c0676e0 0x7fbf8c069ba0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:38.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.645+0000 7fbfb07e4640 1 --2- 172.21.15.55:0/1541374263 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbfa81585f0 0x7fbfa81451a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:38.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.645+0000 7fbfb07e4640 1 --2- 172.21.15.55:0/1541374263 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbfa814eaa0 0x7fbfa814bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:38.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.645+0000 7fbfb07e4640 1 --2- 172.21.15.55:0/1541374263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfa80c60d0 0x7fbfa8142650 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:38.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.645+0000 7fbfb07e4640 1 -- 172.21.15.55:0/1541374263 >> 172.21.15.55:0/1541374263 conn(0x7fbfa80cc9c0 msgr2=0x7fbfa8142e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:38.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.645+0000 7fbfb07e4640 1 -- 172.21.15.55:0/1541374263 shutdown_connections 2024-03-19T02:03:38.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:38.645+0000 7fbfb07e4640 1 -- 172.21.15.55:0/1541374263 wait complete. 2024-03-19T02:03:39.263 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:39.264 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:40.265 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:40.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.377+0000 7fd8fe17f640 1 Processor -- start 2024-03-19T02:03:40.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.377+0000 7fd8fe17f640 1 -- start start 2024-03-19T02:03:40.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.377+0000 7fd8fe17f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd8f8154290 0x7fd8f815e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:40.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.377+0000 7fd8fe17f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fd8f80ca310 0x7fd8f8154670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:40.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.377+0000 7fd8fe17f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8f806bc90 con 0x7fd8f815ad40 2024-03-19T02:03:40.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.377+0000 7fd8fe17f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fd8f815b2c0 con 0x7fd8f8154290 2024-03-19T02:03:40.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.377+0000 7fd8fe17f640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fd8f806c1d0 con 0x7fd8f80ca310 2024-03-19T02:03:40.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.381+0000 7fd8f77fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd8f8154290 0x7fd8f815e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:40.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.381+0000 7fd8f77fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd8f8154290 0x7fd8f815e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56398/0 (socket says 172.21.15.55:56398) 2024-03-19T02:03:40.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.381+0000 7fd8f77fe640 1 -- 172.21.15.55:0/226619833 learned_addr learned my addr 172.21.15.55:0/226619833 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:40.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.381+0000 7fd8f6ffd640 1 --2- 172.21.15.55:0/226619833 >> v2:172.21.15.84:3300/0 conn(0x7fd8f80ca310 0x7fd8f8154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:40.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.381+0000 7fd8f77fe640 1 -- 172.21.15.55:0/226619833 >> v2:172.21.15.84:3300/0 conn(0x7fd8f80ca310 msgr2=0x7fd8f8154670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:40.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.381+0000 7fd8f77fe640 1 --2- 172.21.15.55:0/226619833 >> v2:172.21.15.84:3300/0 conn(0x7fd8f80ca310 0x7fd8f8154670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:40.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.381+0000 7fd8f77fe640 1 -- 172.21.15.55:0/226619833 >> v1:172.21.15.84:6789/0 conn(0x7fd8f815ad40 legacy=0x7fd8f8070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:40.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.381+0000 7fd8f77fe640 1 -- 172.21.15.55:0/226619833 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f806bf30 con 0x7fd8f8154290 2024-03-19T02:03:40.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.385+0000 7fd8f77fe640 1 --2- 172.21.15.55:0/226619833 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8f8154290 0x7fd8f815e4c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fd8f80c72c0 tx=0x7fd8e801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbc60d5e201b0d36 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:40.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.389+0000 7fd8f67fc640 1 -- 172.21.15.55:0/226619833 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e80ab5f0 con 0x7fd8f8154290 2024-03-19T02:03:40.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.389+0000 7fd8f67fc640 1 -- 172.21.15.55:0/226619833 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8e80b0070 con 0x7fd8f8154290 2024-03-19T02:03:40.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.389+0000 7fd8f67fc640 1 -- 172.21.15.55:0/226619833 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e80ab9b0 con 0x7fd8f8154290 2024-03-19T02:03:40.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.393+0000 7fd8fe17f640 1 -- 172.21.15.55:0/226619833 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8f8154290 msgr2=0x7fd8f815e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:40.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.393+0000 7fd8fe17f640 1 --2- 172.21.15.55:0/226619833 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8f8154290 0x7fd8f815e4c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fd8f80c72c0 tx=0x7fd8e801c4e0 comp rx=0 tx=0).stop 2024-03-19T02:03:40.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.393+0000 7fd8fe17f640 1 -- 172.21.15.55:0/226619833 shutdown_connections 2024-03-19T02:03:40.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.393+0000 7fd8fe17f640 1 --2- 172.21.15.55:0/226619833 >> v2:172.21.15.84:3300/0 conn(0x7fd8f80ca310 0x7fd8f8154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:40.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.393+0000 7fd8fe17f640 1 --2- 172.21.15.55:0/226619833 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8f8154290 0x7fd8f815e4c0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:40.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.393+0000 7fd8fe17f640 1 -- 172.21.15.55:0/226619833 >> 172.21.15.55:0/226619833 conn(0x7fd8f806c580 msgr2=0x7fd8f806c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:40.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.393+0000 7fd8fe17f640 1 -- 172.21.15.55:0/226619833 shutdown_connections 2024-03-19T02:03:40.392 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.393+0000 7fd8fe17f640 1 -- 172.21.15.55:0/226619833 wait complete. 2024-03-19T02:03:40.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.393+0000 7fd8fe17f640 1 Processor -- start 2024-03-19T02:03:40.393 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.393+0000 7fd8fe17f640 1 -- start start 2024-03-19T02:03:40.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.393+0000 7fd8fe17f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8f80ca310 0x7fd8f8147510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:40.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.393+0000 7fd8fe17f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd8f8154290 0x7fd8f81f3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:40.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.393+0000 7fd8fe17f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8f815ad40 0x7fd8f8158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:40.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.393+0000 7fd8fe17f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f80b2460 con 0x7fd8f815ad40 2024-03-19T02:03:40.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.393+0000 7fd8fe17f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f80b22e0 con 0x7fd8f80ca310 2024-03-19T02:03:40.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.393+0000 7fd8fe17f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd8f8070d50 con 0x7fd8f8154290 2024-03-19T02:03:40.395 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.393+0000 7fd8f7fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8f815ad40 0x7fd8f8158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:40.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.397+0000 7fd8f7fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8f815ad40 0x7fd8f8158c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56406/0 (socket says 172.21.15.55:56406) 2024-03-19T02:03:40.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.397+0000 7fd8f7fff640 1 -- 172.21.15.55:0/3046289268 learned_addr learned my addr 172.21.15.55:0/3046289268 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:40.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.397+0000 7fd8f77fe640 1 --2- 172.21.15.55:0/3046289268 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8f80ca310 0x7fd8f8147510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:40.396 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.397+0000 7fd8f6ffd640 1 --2- 172.21.15.55:0/3046289268 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd8f8154290 0x7fd8f81f3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:40.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.401+0000 7fd8f7fff640 1 -- 172.21.15.55:0/3046289268 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd8f8154290 msgr2=0x7fd8f81f3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:40.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.401+0000 7fd8f7fff640 1 --2- 172.21.15.55:0/3046289268 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd8f8154290 0x7fd8f81f3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:40.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.401+0000 7fd8f7fff640 1 -- 172.21.15.55:0/3046289268 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8f80ca310 msgr2=0x7fd8f8147510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:40.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.401+0000 7fd8f7fff640 1 --2- 172.21.15.55:0/3046289268 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8f80ca310 0x7fd8f8147510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:40.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.401+0000 7fd8f7fff640 1 -- 172.21.15.55:0/3046289268 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f815a230 con 0x7fd8f815ad40 2024-03-19T02:03:40.401 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.401+0000 7fd8f7fff640 1 --2- 172.21.15.55:0/3046289268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8f815ad40 0x7fd8f8158c50 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fd8f000c7b0 tx=0x7fd8f000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:40.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.405+0000 7fd8dffff640 1 -- 172.21.15.55:0/3046289268 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0017070 con 0x7fd8f815ad40 2024-03-19T02:03:40.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.405+0000 7fd8dffff640 1 -- 172.21.15.55:0/3046289268 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8f0003db0 con 0x7fd8f815ad40 2024-03-19T02:03:40.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.405+0000 7fd8fe17f640 1 -- 172.21.15.55:0/3046289268 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8f815a510 con 0x7fd8f815ad40 2024-03-19T02:03:40.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.405+0000 7fd8dffff640 1 -- 172.21.15.55:0/3046289268 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f00052b0 con 0x7fd8f815ad40 2024-03-19T02:03:40.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.405+0000 7fd8fe17f640 1 -- 172.21.15.55:0/3046289268 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8f8146a30 con 0x7fd8f815ad40 2024-03-19T02:03:40.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.405+0000 7fd8fe17f640 1 -- 172.21.15.55:0/3046289268 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8f8068a40 con 0x7fd8f815ad40 2024-03-19T02:03:40.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.709+0000 7fd8dffff640 1 -- 172.21.15.55:0/3046289268 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fd8f001a070 con 0x7fd8f815ad40 2024-03-19T02:03:40.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.709+0000 7fd8dffff640 1 --2- 172.21.15.55:0/3046289268 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd8e40676b0 0x7fd8e4069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:40.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.709+0000 7fd8f77fe640 1 --2- 172.21.15.55:0/3046289268 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd8e40676b0 0x7fd8e4069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:40.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.709+0000 7fd8f77fe640 1 --2- 172.21.15.55:0/3046289268 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd8e40676b0 0x7fd8e4069b70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd8e80af000 tx=0x7fd8e80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:40.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.725+0000 7fd8dffff640 1 -- 172.21.15.55:0/3046289268 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fd8f0053c80 con 0x7fd8f815ad40 2024-03-19T02:03:40.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.729+0000 7fd8dffff640 1 -- 172.21.15.55:0/3046289268 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fd8f0021070 con 0x7fd8f815ad40 2024-03-19T02:03:40.870 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:03:40.873 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:03:40.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.953+0000 7fd8fe17f640 1 -- 172.21.15.55:0/3046289268 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8f80c1dd0 con 0x7fd8f815ad40 2024-03-19T02:03:40.981 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.981+0000 7fd8dffff640 1 -- 172.21.15.55:0/3046289268 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fd8f0055be0 con 0x7fd8f815ad40 2024-03-19T02:03:40.982 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:40.982 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:40.982 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:40.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.985+0000 7fd8fe17f640 1 -- 172.21.15.55:0/3046289268 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd8e40676b0 msgr2=0x7fd8e4069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:40.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.985+0000 7fd8fe17f640 1 --2- 172.21.15.55:0/3046289268 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd8e40676b0 0x7fd8e4069b70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd8e80af000 tx=0x7fd8e80a7000 comp rx=0 tx=0).stop 2024-03-19T02:03:40.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.985+0000 7fd8fe17f640 1 -- 172.21.15.55:0/3046289268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8f815ad40 msgr2=0x7fd8f8158c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:40.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.985+0000 7fd8fe17f640 1 --2- 172.21.15.55:0/3046289268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8f815ad40 0x7fd8f8158c50 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fd8f000c7b0 tx=0x7fd8f000cc70 comp rx=0 tx=0).stop 2024-03-19T02:03:40.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.985+0000 7fd8fe17f640 1 -- 172.21.15.55:0/3046289268 shutdown_connections 2024-03-19T02:03:40.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.985+0000 7fd8fe17f640 1 --2- 172.21.15.55:0/3046289268 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd8e40676b0 0x7fd8e4069b70 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:40.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.985+0000 7fd8fe17f640 1 --2- 172.21.15.55:0/3046289268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8f815ad40 0x7fd8f8158c50 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:40.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.985+0000 7fd8fe17f640 1 --2- 172.21.15.55:0/3046289268 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd8f8154290 0x7fd8f81f3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:40.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.985+0000 7fd8fe17f640 1 --2- 172.21.15.55:0/3046289268 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8f80ca310 0x7fd8f8147510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:40.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.985+0000 7fd8fe17f640 1 -- 172.21.15.55:0/3046289268 >> 172.21.15.55:0/3046289268 conn(0x7fd8f806c580 msgr2=0x7fd8f80c7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:40.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.985+0000 7fd8fe17f640 1 -- 172.21.15.55:0/3046289268 shutdown_connections 2024-03-19T02:03:40.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:40.985+0000 7fd8fe17f640 1 -- 172.21.15.55:0/3046289268 wait complete. 2024-03-19T02:03:41.604 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:41.605 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:42.606 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:42.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.717+0000 7f2eb262a640 1 Processor -- start 2024-03-19T02:03:42.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.717+0000 7f2eb262a640 1 -- start start 2024-03-19T02:03:42.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.717+0000 7f2eb262a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2eac0708a0 0x7f2eac0c8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:42.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.717+0000 7f2eb262a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f2eac070c80 0x7f2eac06b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:42.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.717+0000 7f2eb262a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2eac06bbd0 con 0x7f2eac157f90 2024-03-19T02:03:42.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.717+0000 7f2eb262a640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f2eac0c7dd0 con 0x7f2eac070c80 2024-03-19T02:03:42.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.717+0000 7f2eb262a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2eac06beb0 con 0x7f2eac0708a0 2024-03-19T02:03:42.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.721+0000 7f2eb0bc7640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f2eac157f90 0x7f2eac0ca010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:60062/0 (socket says 172.21.15.55:60062) 2024-03-19T02:03:42.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.721+0000 7f2eb0bc7640 1 -- 172.21.15.55:0/2385286504 learned_addr learned my addr 172.21.15.55:0/2385286504 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:03:42.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.721+0000 7f2eabfff640 1 --2- 172.21.15.55:0/2385286504 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2eac0708a0 0x7f2eac0c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:42.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.721+0000 7f2eab7fe640 1 --2- 172.21.15.55:0/2385286504 >> v2:172.21.15.84:3300/0 conn(0x7f2eac070c80 0x7f2eac06b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:42.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.721+0000 7f2eaaffd640 1 -- 172.21.15.55:0/2385286504 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2847737130 0 0) 0x7f2eac06bbd0 con 0x7f2eac157f90 2024-03-19T02:03:42.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.721+0000 7f2eaaffd640 1 -- 172.21.15.55:0/2385286504 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e94003530 con 0x7f2eac157f90 2024-03-19T02:03:42.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.725+0000 7f2eabfff640 1 -- 172.21.15.55:0/2385286504 >> v2:172.21.15.84:3300/0 conn(0x7f2eac070c80 msgr2=0x7f2eac06b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:42.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.725+0000 7f2eabfff640 1 --2- 172.21.15.55:0/2385286504 >> v2:172.21.15.84:3300/0 conn(0x7f2eac070c80 0x7f2eac06b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:42.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.725+0000 7f2eabfff640 1 -- 172.21.15.55:0/2385286504 >> v1:172.21.15.55:6789/0 conn(0x7f2eac157f90 legacy=0x7f2eac0ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:42.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.725+0000 7f2eabfff640 1 -- 172.21.15.55:0/2385286504 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eac0c7780 con 0x7f2eac0708a0 2024-03-19T02:03:42.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.729+0000 7f2eabfff640 1 --2- 172.21.15.55:0/2385286504 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2eac0708a0 0x7f2eac0c8260 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c0097c0 tx=0x7f2e9c01c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f42b9108e015c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:42.737 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.737+0000 7f2eaaffd640 1 -- 172.21.15.55:0/2385286504 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c0ab600 con 0x7f2eac0708a0 2024-03-19T02:03:42.738 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.737+0000 7f2eaaffd640 1 -- 172.21.15.55:0/2385286504 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e9c0b0070 con 0x7f2eac0708a0 2024-03-19T02:03:42.738 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.737+0000 7f2eaaffd640 1 -- 172.21.15.55:0/2385286504 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c0ab9c0 con 0x7f2eac0708a0 2024-03-19T02:03:42.738 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.737+0000 7f2eb262a640 1 -- 172.21.15.55:0/2385286504 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2eac0708a0 msgr2=0x7f2eac0c8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:42.739 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.737+0000 7f2eb262a640 1 --2- 172.21.15.55:0/2385286504 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2eac0708a0 0x7f2eac0c8260 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c0097c0 tx=0x7f2e9c01c470 comp rx=0 tx=0).stop 2024-03-19T02:03:42.739 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.737+0000 7f2eb262a640 1 -- 172.21.15.55:0/2385286504 shutdown_connections 2024-03-19T02:03:42.739 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.737+0000 7f2eb262a640 1 --2- 172.21.15.55:0/2385286504 >> v2:172.21.15.84:3300/0 conn(0x7f2eac070c80 0x7f2eac06b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:42.739 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.737+0000 7f2eb262a640 1 --2- 172.21.15.55:0/2385286504 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2eac0708a0 0x7f2eac0c8260 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:42.739 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.737+0000 7f2eb262a640 1 -- 172.21.15.55:0/2385286504 >> 172.21.15.55:0/2385286504 conn(0x7f2eac06c580 msgr2=0x7f2eac06c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:42.739 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.737+0000 7f2eb262a640 1 -- 172.21.15.55:0/2385286504 shutdown_connections 2024-03-19T02:03:42.739 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.737+0000 7f2eb262a640 1 -- 172.21.15.55:0/2385286504 wait complete. 2024-03-19T02:03:42.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.741+0000 7f2eb262a640 1 Processor -- start 2024-03-19T02:03:42.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.741+0000 7f2eb262a640 1 -- start start 2024-03-19T02:03:42.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.741+0000 7f2eb262a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2eac0708a0 0x7f2eac13e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:42.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.741+0000 7f2eb262a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2eac070c80 0x7f2eac13abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:42.740 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.741+0000 7f2eb262a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eac157f90 0x7f2eac1409b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:42.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.741+0000 7f2eb262a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2eac164aa0 con 0x7f2eac157f90 2024-03-19T02:03:42.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.741+0000 7f2eb262a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2eac164920 con 0x7f2eac0708a0 2024-03-19T02:03:42.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.741+0000 7f2eb262a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2eac164c20 con 0x7f2eac070c80 2024-03-19T02:03:42.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.741+0000 7f2eb0bc7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eac157f90 0x7f2eac1409b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:42.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.741+0000 7f2eab7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2eac070c80 0x7f2eac13abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:42.742 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.741+0000 7f2eb0bc7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eac157f90 0x7f2eac1409b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56418/0 (socket says 172.21.15.55:56418) 2024-03-19T02:03:42.742 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.741+0000 7f2eb0bc7640 1 -- 172.21.15.55:0/200031999 learned_addr learned my addr 172.21.15.55:0/200031999 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:42.743 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.745+0000 7f2eabfff640 1 --2- 172.21.15.55:0/200031999 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2eac0708a0 0x7f2eac13e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:42.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.745+0000 7f2eb0bc7640 1 -- 172.21.15.55:0/200031999 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2eac070c80 msgr2=0x7f2eac13abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:42.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.745+0000 7f2eb0bc7640 1 --2- 172.21.15.55:0/200031999 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2eac070c80 0x7f2eac13abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:42.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.745+0000 7f2eb0bc7640 1 -- 172.21.15.55:0/200031999 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2eac0708a0 msgr2=0x7f2eac13e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:42.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.745+0000 7f2eb0bc7640 1 --2- 172.21.15.55:0/200031999 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2eac0708a0 0x7f2eac13e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:42.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.745+0000 7f2eb0bc7640 1 -- 172.21.15.55:0/200031999 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eac137620 con 0x7f2eac157f90 2024-03-19T02:03:42.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.749+0000 7f2eb0bc7640 1 --2- 172.21.15.55:0/200031999 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eac157f90 0x7f2eac1409b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f2ea400cad0 tx=0x7f2ea400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:42.753 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.753+0000 7f2ea8ff9640 1 -- 172.21.15.55:0/200031999 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea4017070 con 0x7f2eac157f90 2024-03-19T02:03:42.753 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.753+0000 7f2ea8ff9640 1 -- 172.21.15.55:0/200031999 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea400ac30 con 0x7f2eac157f90 2024-03-19T02:03:42.753 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.753+0000 7f2eb262a640 1 -- 172.21.15.55:0/200031999 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eac13f6c0 con 0x7f2eac157f90 2024-03-19T02:03:42.753 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.753+0000 7f2ea8ff9640 1 -- 172.21.15.55:0/200031999 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea4012400 con 0x7f2eac157f90 2024-03-19T02:03:42.753 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.753+0000 7f2eb262a640 1 -- 172.21.15.55:0/200031999 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eac137060 con 0x7f2eac157f90 2024-03-19T02:03:42.754 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.753+0000 7f2eb262a640 1 -- 172.21.15.55:0/200031999 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eac068a40 con 0x7f2eac157f90 2024-03-19T02:03:42.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.773+0000 7f2ea8ff9640 1 -- 172.21.15.55:0/200031999 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f2ea401a070 con 0x7f2eac157f90 2024-03-19T02:03:42.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.773+0000 7f2ea8ff9640 1 --2- 172.21.15.55:0/200031999 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2e98067660 0x7f2e98069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:42.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.773+0000 7f2eabfff640 1 --2- 172.21.15.55:0/200031999 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2e98067660 0x7f2e98069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:42.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.777+0000 7f2eabfff640 1 --2- 172.21.15.55:0/200031999 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2e98067660 0x7f2e98069b20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c0af000 tx=0x7f2e9c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:42.776 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:42.777+0000 7f2ea8ff9640 1 -- 172.21.15.55:0/200031999 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f2ea4054110 con 0x7f2eac157f90 2024-03-19T02:03:43.069 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:43.069+0000 7f2ea8ff9640 1 -- 172.21.15.55:0/200031999 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2ea4021020 con 0x7f2eac157f90 2024-03-19T02:03:43.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:43.269+0000 7f2eb262a640 1 -- 172.21.15.55:0/200031999 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2eac137340 con 0x7f2eac157f90 2024-03-19T02:03:43.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:43.361+0000 7f2ea8ff9640 1 -- 172.21.15.55:0/200031999 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f2ea4056070 con 0x7f2eac157f90 2024-03-19T02:03:43.360 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:43.360 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:43.361 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:43.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:43.361+0000 7f2eb262a640 1 -- 172.21.15.55:0/200031999 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2e98067660 msgr2=0x7f2e98069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:43.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:43.361+0000 7f2eb262a640 1 --2- 172.21.15.55:0/200031999 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2e98067660 0x7f2e98069b20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c0af000 tx=0x7f2e9c0a7000 comp rx=0 tx=0).stop 2024-03-19T02:03:43.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:43.361+0000 7f2eb262a640 1 -- 172.21.15.55:0/200031999 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eac157f90 msgr2=0x7f2eac1409b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:43.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:43.361+0000 7f2eb262a640 1 --2- 172.21.15.55:0/200031999 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eac157f90 0x7f2eac1409b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f2ea400cad0 tx=0x7f2ea400cf90 comp rx=0 tx=0).stop 2024-03-19T02:03:43.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:43.361+0000 7f2eb262a640 1 -- 172.21.15.55:0/200031999 shutdown_connections 2024-03-19T02:03:43.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:43.361+0000 7f2eb262a640 1 --2- 172.21.15.55:0/200031999 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2e98067660 0x7f2e98069b20 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:43.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:43.361+0000 7f2eb262a640 1 --2- 172.21.15.55:0/200031999 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eac157f90 0x7f2eac1409b0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:43.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:43.361+0000 7f2eb262a640 1 --2- 172.21.15.55:0/200031999 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2eac070c80 0x7f2eac13abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:43.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:43.361+0000 7f2eb262a640 1 --2- 172.21.15.55:0/200031999 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2eac0708a0 0x7f2eac13e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:43.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:43.361+0000 7f2eb262a640 1 -- 172.21.15.55:0/200031999 >> 172.21.15.55:0/200031999 conn(0x7f2eac06c580 msgr2=0x7f2eac156670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:43.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:43.361+0000 7f2eb262a640 1 -- 172.21.15.55:0/200031999 shutdown_connections 2024-03-19T02:03:43.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:43.361+0000 7f2eb262a640 1 -- 172.21.15.55:0/200031999 wait complete. 2024-03-19T02:03:43.981 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:43.981 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:44.982 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:45.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.096+0000 7fdcab879640 1 Processor -- start 2024-03-19T02:03:45.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.096+0000 7fdcab879640 1 -- start start 2024-03-19T02:03:45.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.096+0000 7fdcab879640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fdca4150f50 0x7fdca415b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:45.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.096+0000 7fdcab879640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdca40ca350 0x7fdca40728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:45.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.096+0000 7fdcab879640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdca4151330 con 0x7fdca40c3320 2024-03-19T02:03:45.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.096+0000 7fdcab879640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fdca40c3e00 con 0x7fdca4150f50 2024-03-19T02:03:45.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.096+0000 7fdcab879640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fdca415ac10 con 0x7fdca40ca350 2024-03-19T02:03:45.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.100+0000 7fdca9615640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fdca4150f50 0x7fdca415b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:45.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.100+0000 7fdca8e14640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdca40ca350 0x7fdca40728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:45.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.100+0000 7fdca9615640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fdca4150f50 0x7fdca415b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:40038/0 (socket says 172.21.15.55:40038) 2024-03-19T02:03:45.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.100+0000 7fdca9615640 1 -- 172.21.15.55:0/491274682 learned_addr learned my addr 172.21.15.55:0/491274682 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:45.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.104+0000 7fdca9615640 1 -- 172.21.15.55:0/491274682 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdca40ca350 msgr2=0x7fdca40728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:45.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.104+0000 7fdca9615640 1 --2- 172.21.15.55:0/491274682 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdca40ca350 0x7fdca40728b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:45.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.104+0000 7fdca9615640 1 -- 172.21.15.55:0/491274682 >> v1:172.21.15.84:6789/0 conn(0x7fdca40c3320 legacy=0x7fdca4157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:45.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.104+0000 7fdca9615640 1 -- 172.21.15.55:0/491274682 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca40c3840 con 0x7fdca4150f50 2024-03-19T02:03:45.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.108+0000 7fdca9615640 1 --2- 172.21.15.55:0/491274682 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdca4150f50 0x7fdca415b0a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c009870 tx=0x7fdc9c01c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f6026ba659d554e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:45.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.116+0000 7fdca3fff640 1 -- 172.21.15.55:0/491274682 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c0a6040 con 0x7fdca4150f50 2024-03-19T02:03:45.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.116+0000 7fdca3fff640 1 -- 172.21.15.55:0/491274682 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc9c0af070 con 0x7fdca4150f50 2024-03-19T02:03:45.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.116+0000 7fdca3fff640 1 -- 172.21.15.55:0/491274682 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c0aa860 con 0x7fdca4150f50 2024-03-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.116+0000 7fdcab879640 1 -- 172.21.15.55:0/491274682 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdca4150f50 msgr2=0x7fdca415b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.116+0000 7fdcab879640 1 --2- 172.21.15.55:0/491274682 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdca4150f50 0x7fdca415b0a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c009870 tx=0x7fdc9c01c520 comp rx=0 tx=0).stop 2024-03-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.116+0000 7fdcab879640 1 -- 172.21.15.55:0/491274682 shutdown_connections 2024-03-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.116+0000 7fdcab879640 1 --2- 172.21.15.55:0/491274682 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdca40ca350 0x7fdca40728b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.116+0000 7fdcab879640 1 --2- 172.21.15.55:0/491274682 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdca4150f50 0x7fdca415b0a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.116+0000 7fdcab879640 1 -- 172.21.15.55:0/491274682 >> 172.21.15.55:0/491274682 conn(0x7fdca406e670 msgr2=0x7fdca406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.116+0000 7fdcab879640 1 -- 172.21.15.55:0/491274682 shutdown_connections 2024-03-19T02:03:45.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.116+0000 7fdcab879640 1 -- 172.21.15.55:0/491274682 wait complete. 2024-03-19T02:03:45.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.116+0000 7fdcab879640 1 Processor -- start 2024-03-19T02:03:45.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.116+0000 7fdcab879640 1 -- start start 2024-03-19T02:03:45.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.120+0000 7fdcab879640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdca40c3320 0x7fdca41f65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:45.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.120+0000 7fdcab879640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdca40ca350 0x7fdca4150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:45.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.120+0000 7fdcab879640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdca4150f50 0x7fdca4149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:45.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.120+0000 7fdcab879640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdca41639b0 con 0x7fdca4150f50 2024-03-19T02:03:45.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.120+0000 7fdcab879640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fdca4163830 con 0x7fdca40c3320 2024-03-19T02:03:45.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.120+0000 7fdcab879640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fdca4163b30 con 0x7fdca40ca350 2024-03-19T02:03:45.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.120+0000 7fdca9615640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdca40c3320 0x7fdca41f65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:45.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.120+0000 7fdca9e16640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdca4150f50 0x7fdca4149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:45.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.120+0000 7fdca9e16640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdca4150f50 0x7fdca4149470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56432/0 (socket says 172.21.15.55:56432) 2024-03-19T02:03:45.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.120+0000 7fdca9e16640 1 -- 172.21.15.55:0/2291824502 learned_addr learned my addr 172.21.15.55:0/2291824502 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:45.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.120+0000 7fdca8e14640 1 --2- 172.21.15.55:0/2291824502 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdca40ca350 0x7fdca4150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:45.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.124+0000 7fdca9e16640 1 -- 172.21.15.55:0/2291824502 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdca40ca350 msgr2=0x7fdca4150040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:45.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.124+0000 7fdca9e16640 1 --2- 172.21.15.55:0/2291824502 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdca40ca350 0x7fdca4150040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:45.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.124+0000 7fdca9e16640 1 -- 172.21.15.55:0/2291824502 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdca40c3320 msgr2=0x7fdca41f65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:45.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.124+0000 7fdca9e16640 1 --2- 172.21.15.55:0/2291824502 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdca40c3320 0x7fdca41f65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:45.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.124+0000 7fdca9e16640 1 -- 172.21.15.55:0/2291824502 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca41f6e50 con 0x7fdca4150f50 2024-03-19T02:03:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.128+0000 7fdca9e16640 1 --2- 172.21.15.55:0/2291824502 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdca4150f50 0x7fdca4149470 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fdc9400c780 tx=0x7fdc9400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:45.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.132+0000 7fdca1ffb640 1 -- 172.21.15.55:0/2291824502 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc94017070 con 0x7fdca4150f50 2024-03-19T02:03:45.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.132+0000 7fdca1ffb640 1 -- 172.21.15.55:0/2291824502 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc94003db0 con 0x7fdca4150f50 2024-03-19T02:03:45.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.132+0000 7fdca1ffb640 1 -- 172.21.15.55:0/2291824502 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc940052b0 con 0x7fdca4150f50 2024-03-19T02:03:45.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.132+0000 7fdcab879640 1 -- 172.21.15.55:0/2291824502 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdca41f7130 con 0x7fdca4150f50 2024-03-19T02:03:45.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.132+0000 7fdcab879640 1 -- 172.21.15.55:0/2291824502 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdca4148990 con 0x7fdca4150f50 2024-03-19T02:03:45.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.132+0000 7fdcab879640 1 -- 172.21.15.55:0/2291824502 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdca4068a40 con 0x7fdca4150f50 2024-03-19T02:03:45.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.412+0000 7fdca1ffb640 1 -- 172.21.15.55:0/2291824502 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fdc94006b20 con 0x7fdca4150f50 2024-03-19T02:03:45.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.412+0000 7fdca1ffb640 1 --2- 172.21.15.55:0/2291824502 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdc90067660 0x7fdc90069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:45.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.412+0000 7fdca9615640 1 --2- 172.21.15.55:0/2291824502 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdc90067660 0x7fdc90069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:45.415 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.412+0000 7fdca9615640 1 --2- 172.21.15.55:0/2291824502 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdc90067660 0x7fdc90069b20 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c0ae000 tx=0x7fdc9c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:45.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.428+0000 7fdca1ffb640 1 -- 172.21.15.55:0/2291824502 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fdc94053660 con 0x7fdca4150f50 2024-03-19T02:03:45.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.428+0000 7fdca1ffb640 1 -- 172.21.15.55:0/2291824502 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fdc94020080 con 0x7fdca4150f50 2024-03-19T02:03:45.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.648+0000 7fdcab879640 1 -- 172.21.15.55:0/2291824502 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdca40c7600 con 0x7fdca4150f50 2024-03-19T02:03:45.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.792+0000 7fdca1ffb640 1 -- 172.21.15.55:0/2291824502 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fdc940555c0 con 0x7fdca4150f50 2024-03-19T02:03:45.793 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:45.793 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:45.793 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:45.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.792+0000 7fdcab879640 1 -- 172.21.15.55:0/2291824502 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdc90067660 msgr2=0x7fdc90069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:45.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.792+0000 7fdcab879640 1 --2- 172.21.15.55:0/2291824502 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdc90067660 0x7fdc90069b20 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c0ae000 tx=0x7fdc9c0a7000 comp rx=0 tx=0).stop 2024-03-19T02:03:45.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.792+0000 7fdcab879640 1 -- 172.21.15.55:0/2291824502 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdca4150f50 msgr2=0x7fdca4149470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:45.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.792+0000 7fdcab879640 1 --2- 172.21.15.55:0/2291824502 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdca4150f50 0x7fdca4149470 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fdc9400c780 tx=0x7fdc9400cc40 comp rx=0 tx=0).stop 2024-03-19T02:03:45.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.792+0000 7fdcab879640 1 -- 172.21.15.55:0/2291824502 shutdown_connections 2024-03-19T02:03:45.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.792+0000 7fdcab879640 1 --2- 172.21.15.55:0/2291824502 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fdc90067660 0x7fdc90069b20 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:45.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.792+0000 7fdcab879640 1 --2- 172.21.15.55:0/2291824502 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdca4150f50 0x7fdca4149470 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:45.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.792+0000 7fdcab879640 1 --2- 172.21.15.55:0/2291824502 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fdca40ca350 0x7fdca4150040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:45.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.792+0000 7fdcab879640 1 --2- 172.21.15.55:0/2291824502 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdca40c3320 0x7fdca41f65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:45.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.792+0000 7fdcab879640 1 -- 172.21.15.55:0/2291824502 >> 172.21.15.55:0/2291824502 conn(0x7fdca406e670 msgr2=0x7fdca4074290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:45.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.792+0000 7fdcab879640 1 -- 172.21.15.55:0/2291824502 shutdown_connections 2024-03-19T02:03:45.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:45.792+0000 7fdcab879640 1 -- 172.21.15.55:0/2291824502 wait complete. 2024-03-19T02:03:46.415 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:46.416 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:47.417 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:47.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.544+0000 7fe3a03e6640 1 Processor -- start 2024-03-19T02:03:47.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.544+0000 7fe3a03e6640 1 -- start start 2024-03-19T02:03:47.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.548+0000 7fe3a03e6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe3980c93f0 0x7fe3980c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:47.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.548+0000 7fe3a03e6640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fe39815a250 0x7fe39815a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:47.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.548+0000 7fe3a03e6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe39815b1c0 con 0x7fe3981544a0 2024-03-19T02:03:47.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.548+0000 7fe3a03e6640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fe39815db90 con 0x7fe39815a250 2024-03-19T02:03:47.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.548+0000 7fe3a03e6640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fe3980c9d30 con 0x7fe3980c93f0 2024-03-19T02:03:47.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.548+0000 7fe39d981640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe3980c93f0 0x7fe3980c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:47.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.548+0000 7fe39e983640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fe39815a250 0x7fe39815a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:47.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.548+0000 7fe39e983640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fe39815a250 0x7fe39815a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:40062/0 (socket says 172.21.15.55:40062) 2024-03-19T02:03:47.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.548+0000 7fe39e983640 1 -- 172.21.15.55:0/1140036711 learned_addr learned my addr 172.21.15.55:0/1140036711 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:47.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.552+0000 7fe39e983640 1 -- 172.21.15.55:0/1140036711 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe3980c93f0 msgr2=0x7fe3980c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:47.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.552+0000 7fe39e983640 1 --2- 172.21.15.55:0/1140036711 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe3980c93f0 0x7fe3980c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:47.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.552+0000 7fe39e983640 1 -- 172.21.15.55:0/1140036711 >> v1:172.21.15.55:6789/0 conn(0x7fe3981544a0 legacy=0x7fe3980706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:47.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.552+0000 7fe39e983640 1 -- 172.21.15.55:0/1140036711 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe39806f5d0 con 0x7fe39815a250 2024-03-19T02:03:47.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.556+0000 7fe39e983640 1 --2- 172.21.15.55:0/1140036711 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe39815a250 0x7fe39815a6b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe390009e50 tx=0x7fe39001f1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ca5b2f4d27b23a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:47.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.564+0000 7fe39d180640 1 -- 172.21.15.55:0/1140036711 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3900b4070 con 0x7fe39815a250 2024-03-19T02:03:47.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.564+0000 7fe39d180640 1 -- 172.21.15.55:0/1140036711 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3900af400 con 0x7fe39815a250 2024-03-19T02:03:47.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.564+0000 7fe39d180640 1 -- 172.21.15.55:0/1140036711 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3900af7d0 con 0x7fe39815a250 2024-03-19T02:03:47.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.564+0000 7fe3a03e6640 1 -- 172.21.15.55:0/1140036711 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe39815a250 msgr2=0x7fe39815a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:47.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.564+0000 7fe3a03e6640 1 --2- 172.21.15.55:0/1140036711 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe39815a250 0x7fe39815a6b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe390009e50 tx=0x7fe39001f1e0 comp rx=0 tx=0).stop 2024-03-19T02:03:47.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.564+0000 7fe3a03e6640 1 -- 172.21.15.55:0/1140036711 shutdown_connections 2024-03-19T02:03:47.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.564+0000 7fe3a03e6640 1 --2- 172.21.15.55:0/1140036711 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe3980c93f0 0x7fe3980c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:47.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.564+0000 7fe3a03e6640 1 --2- 172.21.15.55:0/1140036711 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe39815a250 0x7fe39815a6b0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:47.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.564+0000 7fe3a03e6640 1 -- 172.21.15.55:0/1140036711 >> 172.21.15.55:0/1140036711 conn(0x7fe398067600 msgr2=0x7fe3980714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:47.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.564+0000 7fe3a03e6640 1 -- 172.21.15.55:0/1140036711 shutdown_connections 2024-03-19T02:03:47.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.564+0000 7fe3a03e6640 1 -- 172.21.15.55:0/1140036711 wait complete. 2024-03-19T02:03:47.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.568+0000 7fe3a03e6640 1 Processor -- start 2024-03-19T02:03:47.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.568+0000 7fe3a03e6640 1 -- start start 2024-03-19T02:03:47.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.568+0000 7fe3a03e6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe3980c93f0 0x7fe3981525b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:47.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.568+0000 7fe3a03e6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe3981544a0 0x7fe398155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:47.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.568+0000 7fe3a03e6640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe39815a250 0x7fe3981e68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:47.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.568+0000 7fe3a03e6640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe39806fe70 con 0x7fe3981544a0 2024-03-19T02:03:47.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.568+0000 7fe3a03e6640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fe39806fb90 con 0x7fe39815a250 2024-03-19T02:03:47.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.568+0000 7fe3a03e6640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fe398154880 con 0x7fe3980c93f0 2024-03-19T02:03:47.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.568+0000 7fe39d981640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe3981544a0 0x7fe398155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:47.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.568+0000 7fe39e182640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe3980c93f0 0x7fe3981525b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:47.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.568+0000 7fe39d981640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe3981544a0 0x7fe398155680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56434/0 (socket says 172.21.15.55:56434) 2024-03-19T02:03:47.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.568+0000 7fe39d981640 1 -- 172.21.15.55:0/2137264100 learned_addr learned my addr 172.21.15.55:0/2137264100 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:47.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.568+0000 7fe39e983640 1 --2- 172.21.15.55:0/2137264100 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe39815a250 0x7fe3981e68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:47.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.572+0000 7fe39d981640 1 -- 172.21.15.55:0/2137264100 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe3980c93f0 msgr2=0x7fe3981525b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:47.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.572+0000 7fe39d981640 1 --2- 172.21.15.55:0/2137264100 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe3980c93f0 0x7fe3981525b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:47.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.572+0000 7fe39d981640 1 -- 172.21.15.55:0/2137264100 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe39815a250 msgr2=0x7fe3981e68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:47.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.572+0000 7fe39d981640 1 --2- 172.21.15.55:0/2137264100 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe39815a250 0x7fe3981e68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:47.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.572+0000 7fe39d981640 1 -- 172.21.15.55:0/2137264100 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3981f4160 con 0x7fe3981544a0 2024-03-19T02:03:47.577 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.576+0000 7fe39d981640 1 --2- 172.21.15.55:0/2137264100 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe3981544a0 0x7fe398155680 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fe38800e8a0 tx=0x7fe38800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:47.582 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.580+0000 7fe38effd640 1 -- 172.21.15.55:0/2137264100 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe388017070 con 0x7fe3981544a0 2024-03-19T02:03:47.582 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.580+0000 7fe38effd640 1 -- 172.21.15.55:0/2137264100 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe38800c940 con 0x7fe3981544a0 2024-03-19T02:03:47.582 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.580+0000 7fe3a03e6640 1 -- 172.21.15.55:0/2137264100 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3981576f0 con 0x7fe3981544a0 2024-03-19T02:03:47.582 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.580+0000 7fe38effd640 1 -- 172.21.15.55:0/2137264100 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38800ccf0 con 0x7fe3981544a0 2024-03-19T02:03:47.582 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.580+0000 7fe3a03e6640 1 -- 172.21.15.55:0/2137264100 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3981573f0 con 0x7fe3981544a0 2024-03-19T02:03:47.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.580+0000 7fe3a03e6640 1 -- 172.21.15.55:0/2137264100 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe398068a40 con 0x7fe3981544a0 2024-03-19T02:03:47.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.600+0000 7fe38effd640 1 -- 172.21.15.55:0/2137264100 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fe38801a070 con 0x7fe3981544a0 2024-03-19T02:03:47.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.600+0000 7fe38effd640 1 --2- 172.21.15.55:0/2137264100 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe37c0676e0 0x7fe37c069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:47.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.600+0000 7fe39e182640 1 --2- 172.21.15.55:0/2137264100 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe37c0676e0 0x7fe37c069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:47.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.604+0000 7fe39e182640 1 --2- 172.21.15.55:0/2137264100 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe37c0676e0 0x7fe37c069ba0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe3940098e0 tx=0x7fe394009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:47.605 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.604+0000 7fe38effd640 1 -- 172.21.15.55:0/2137264100 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fe388054590 con 0x7fe3981544a0 2024-03-19T02:03:47.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:47.892+0000 7fe38effd640 1 -- 172.21.15.55:0/2137264100 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fe3880564f0 con 0x7fe3981544a0 2024-03-19T02:03:48.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:48.116+0000 7fe3a03e6640 1 -- 172.21.15.55:0/2137264100 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3980c8690 con 0x7fe3981544a0 2024-03-19T02:03:48.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:48.140+0000 7fe38effd640 1 -- 172.21.15.55:0/2137264100 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fe38805a590 con 0x7fe3981544a0 2024-03-19T02:03:48.140 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:48.141 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:48.141 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:48.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:48.140+0000 7fe3a03e6640 1 -- 172.21.15.55:0/2137264100 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe37c0676e0 msgr2=0x7fe37c069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:48.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:48.140+0000 7fe3a03e6640 1 --2- 172.21.15.55:0/2137264100 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe37c0676e0 0x7fe37c069ba0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe3940098e0 tx=0x7fe394009060 comp rx=0 tx=0).stop 2024-03-19T02:03:48.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:48.140+0000 7fe3a03e6640 1 -- 172.21.15.55:0/2137264100 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe3981544a0 msgr2=0x7fe398155680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:48.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:48.140+0000 7fe3a03e6640 1 --2- 172.21.15.55:0/2137264100 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe3981544a0 0x7fe398155680 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fe38800e8a0 tx=0x7fe38800eca0 comp rx=0 tx=0).stop 2024-03-19T02:03:48.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:48.144+0000 7fe3a03e6640 1 -- 172.21.15.55:0/2137264100 shutdown_connections 2024-03-19T02:03:48.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:48.144+0000 7fe3a03e6640 1 --2- 172.21.15.55:0/2137264100 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe37c0676e0 0x7fe37c069ba0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:48.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:48.144+0000 7fe3a03e6640 1 --2- 172.21.15.55:0/2137264100 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe39815a250 0x7fe3981e68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:48.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:48.144+0000 7fe3a03e6640 1 --2- 172.21.15.55:0/2137264100 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe3981544a0 0x7fe398155680 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:48.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:48.144+0000 7fe3a03e6640 1 --2- 172.21.15.55:0/2137264100 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe3980c93f0 0x7fe3981525b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:48.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:48.144+0000 7fe3a03e6640 1 -- 172.21.15.55:0/2137264100 >> 172.21.15.55:0/2137264100 conn(0x7fe398067600 msgr2=0x7fe3980ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:48.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:48.144+0000 7fe3a03e6640 1 -- 172.21.15.55:0/2137264100 shutdown_connections 2024-03-19T02:03:48.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:48.144+0000 7fe3a03e6640 1 -- 172.21.15.55:0/2137264100 wait complete. 2024-03-19T02:03:48.770 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:48.770 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:49.771 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:49.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.884+0000 7f0cfb967640 1 Processor -- start 2024-03-19T02:03:49.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.884+0000 7f0cfb967640 1 -- start start 2024-03-19T02:03:49.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.884+0000 7f0cfb967640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f0cf41510f0 0x7f0cf40c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:49.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.884+0000 7f0cfb967640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0cf415a410 0x7f0cf40708a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:49.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.884+0000 7f0cfb967640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cf40c3f60 con 0x7f0cf40c73c0 2024-03-19T02:03:49.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.884+0000 7f0cfb967640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f0cf41514d0 con 0x7f0cf415a410 2024-03-19T02:03:49.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.884+0000 7f0cfb967640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f0cf40c6870 con 0x7f0cf41510f0 2024-03-19T02:03:49.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.884+0000 7f0cf8f02640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0cf415a410 0x7f0cf40708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:49.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.884+0000 7f0cf9703640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f0cf41510f0 0x7f0cf40c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:49.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.884+0000 7f0cf8f02640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0cf415a410 0x7f0cf40708a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:35608/0 (socket says 172.21.15.55:35608) 2024-03-19T02:03:49.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.884+0000 7f0cf8f02640 1 -- 172.21.15.55:0/2412024655 learned_addr learned my addr 172.21.15.55:0/2412024655 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:49.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.888+0000 7f0cf8f02640 1 -- 172.21.15.55:0/2412024655 >> v2:172.21.15.84:3300/0 conn(0x7f0cf41510f0 msgr2=0x7f0cf40c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:49.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.888+0000 7f0cf8f02640 1 --2- 172.21.15.55:0/2412024655 >> v2:172.21.15.84:3300/0 conn(0x7f0cf41510f0 0x7f0cf40c61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:49.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.888+0000 7f0cf8f02640 1 -- 172.21.15.55:0/2412024655 >> v1:172.21.15.84:6789/0 conn(0x7f0cf40c73c0 legacy=0x7f0cf4072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:49.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.888+0000 7f0cf8f02640 1 -- 172.21.15.55:0/2412024655 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cf40c6b50 con 0x7f0cf415a410 2024-03-19T02:03:49.891 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.892+0000 7f0cf8f02640 1 --2- 172.21.15.55:0/2412024655 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf415a410 0x7f0cf40708a0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f0cf40cc900 tx=0x7f0ce401ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c97148ad342496 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:49.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.896+0000 7f0cebfff640 1 -- 172.21.15.55:0/2412024655 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce40b5070 con 0x7f0cf415a410 2024-03-19T02:03:49.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.896+0000 7f0cebfff640 1 -- 172.21.15.55:0/2412024655 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ce40a89d0 con 0x7f0cf415a410 2024-03-19T02:03:49.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.896+0000 7f0cebfff640 1 -- 172.21.15.55:0/2412024655 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce40a8da0 con 0x7f0cf415a410 2024-03-19T02:03:49.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.896+0000 7f0cfb967640 1 -- 172.21.15.55:0/2412024655 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf415a410 msgr2=0x7f0cf40708a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:49.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.896+0000 7f0cfb967640 1 --2- 172.21.15.55:0/2412024655 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf415a410 0x7f0cf40708a0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f0cf40cc900 tx=0x7f0ce401ecd0 comp rx=0 tx=0).stop 2024-03-19T02:03:49.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.896+0000 7f0cfb967640 1 -- 172.21.15.55:0/2412024655 shutdown_connections 2024-03-19T02:03:49.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.896+0000 7f0cfb967640 1 --2- 172.21.15.55:0/2412024655 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf415a410 0x7f0cf40708a0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:49.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.896+0000 7f0cfb967640 1 --2- 172.21.15.55:0/2412024655 >> v2:172.21.15.84:3300/0 conn(0x7f0cf41510f0 0x7f0cf40c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:49.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.896+0000 7f0cfb967640 1 -- 172.21.15.55:0/2412024655 >> 172.21.15.55:0/2412024655 conn(0x7f0cf40cc9c0 msgr2=0x7f0cf40ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:49.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.896+0000 7f0cfb967640 1 -- 172.21.15.55:0/2412024655 shutdown_connections 2024-03-19T02:03:49.897 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.896+0000 7f0cfb967640 1 -- 172.21.15.55:0/2412024655 wait complete. 2024-03-19T02:03:49.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.896+0000 7f0cfb967640 1 Processor -- start 2024-03-19T02:03:49.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.896+0000 7f0cfb967640 1 -- start start 2024-03-19T02:03:49.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.900+0000 7f0cfb967640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf40c73c0 0x7f0cf4148c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:49.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.900+0000 7f0cfb967640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0cf41510f0 0x7f0cf41e20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:49.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.900+0000 7f0cfb967640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0cf415a410 0x7f0cf4141270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:49.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.900+0000 7f0cfb967640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf40c6e30 con 0x7f0cf40c73c0 2024-03-19T02:03:49.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.900+0000 7f0cfb967640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf40c44e0 con 0x7f0cf415a410 2024-03-19T02:03:49.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.900+0000 7f0cfb967640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f0cf40c4a00 con 0x7f0cf41510f0 2024-03-19T02:03:49.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.900+0000 7f0cf9703640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf40c73c0 0x7f0cf4148c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:49.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.900+0000 7f0cf9f04640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0cf415a410 0x7f0cf4141270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:49.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.900+0000 7f0cf9703640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf40c73c0 0x7f0cf4148c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:35614/0 (socket says 172.21.15.55:35614) 2024-03-19T02:03:49.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.900+0000 7f0cf9703640 1 -- 172.21.15.55:0/1108906870 learned_addr learned my addr 172.21.15.55:0/1108906870 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:49.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.900+0000 7f0cf8f02640 1 --2- 172.21.15.55:0/1108906870 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0cf41510f0 0x7f0cf41e20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:49.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.904+0000 7f0cf9703640 1 -- 172.21.15.55:0/1108906870 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0cf41510f0 msgr2=0x7f0cf41e20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:49.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.904+0000 7f0cf9703640 1 --2- 172.21.15.55:0/1108906870 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0cf41510f0 0x7f0cf41e20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:49.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.904+0000 7f0cf9703640 1 -- 172.21.15.55:0/1108906870 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0cf415a410 msgr2=0x7f0cf4141270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:49.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.904+0000 7f0cf9703640 1 --2- 172.21.15.55:0/1108906870 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0cf415a410 0x7f0cf4141270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:49.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.904+0000 7f0cf9703640 1 -- 172.21.15.55:0/1108906870 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cf4159560 con 0x7f0cf40c73c0 2024-03-19T02:03:49.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.904+0000 7f0cf9703640 1 --2- 172.21.15.55:0/1108906870 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf40c73c0 0x7f0cf4148c60 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0cf000e960 tx=0x7f0cf000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:49.911 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.908+0000 7f0ce9ffb640 1 -- 172.21.15.55:0/1108906870 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf001e070 con 0x7f0cf40c73c0 2024-03-19T02:03:49.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.912+0000 7f0ce9ffb640 1 -- 172.21.15.55:0/1108906870 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cf000ca00 con 0x7f0cf40c73c0 2024-03-19T02:03:49.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.912+0000 7f0cfb967640 1 -- 172.21.15.55:0/1108906870 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cf4159800 con 0x7f0cf40c73c0 2024-03-19T02:03:49.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.912+0000 7f0ce9ffb640 1 -- 172.21.15.55:0/1108906870 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf000cdb0 con 0x7f0cf40c73c0 2024-03-19T02:03:49.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.912+0000 7f0cfb967640 1 -- 172.21.15.55:0/1108906870 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cf4159dc0 con 0x7f0cf40c73c0 2024-03-19T02:03:49.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:49.912+0000 7f0cfb967640 1 -- 172.21.15.55:0/1108906870 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cf4068a40 con 0x7f0cf40c73c0 2024-03-19T02:03:50.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:50.176+0000 7f0ce9ffb640 1 -- 172.21.15.55:0/1108906870 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f0cf0021070 con 0x7f0cf40c73c0 2024-03-19T02:03:50.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:50.176+0000 7f0ce9ffb640 1 --2- 172.21.15.55:0/1108906870 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0cd80676b0 0x7f0cd8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:50.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:50.176+0000 7f0cf8f02640 1 --2- 172.21.15.55:0/1108906870 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0cd80676b0 0x7f0cd8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:50.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:50.180+0000 7f0cf8f02640 1 --2- 172.21.15.55:0/1108906870 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0cd80676b0 0x7f0cd8069b70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40b2000 tx=0x7f0ce4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:50.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:50.192+0000 7f0ce9ffb640 1 -- 172.21.15.55:0/1108906870 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f0cf0056050 con 0x7f0cf40c73c0 2024-03-19T02:03:50.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:50.192+0000 7f0ce9ffb640 1 -- 172.21.15.55:0/1108906870 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f0cf0028020 con 0x7f0cf40c73c0 2024-03-19T02:03:50.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:50.384+0000 7f0cfb967640 1 -- 172.21.15.55:0/1108906870 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0cf40c8420 con 0x7f0cf40c73c0 2024-03-19T02:03:50.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:50.404+0000 7f0ce9ffb640 1 -- 172.21.15.55:0/1108906870 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f0cf0057fb0 con 0x7f0cf40c73c0 2024-03-19T02:03:50.406 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:50.406 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:50.406 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:50.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:50.408+0000 7f0cfb967640 1 -- 172.21.15.55:0/1108906870 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0cd80676b0 msgr2=0x7f0cd8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:50.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:50.408+0000 7f0cfb967640 1 --2- 172.21.15.55:0/1108906870 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0cd80676b0 0x7f0cd8069b70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40b2000 tx=0x7f0ce4009040 comp rx=0 tx=0).stop 2024-03-19T02:03:50.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:50.408+0000 7f0cfb967640 1 -- 172.21.15.55:0/1108906870 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf40c73c0 msgr2=0x7f0cf4148c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:50.408+0000 7f0cfb967640 1 --2- 172.21.15.55:0/1108906870 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf40c73c0 0x7f0cf4148c60 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0cf000e960 tx=0x7f0cf000ee20 comp rx=0 tx=0).stop 2024-03-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:50.408+0000 7f0cfb967640 1 -- 172.21.15.55:0/1108906870 shutdown_connections 2024-03-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:50.408+0000 7f0cfb967640 1 --2- 172.21.15.55:0/1108906870 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0cd80676b0 0x7f0cd8069b70 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:50.408+0000 7f0cfb967640 1 --2- 172.21.15.55:0/1108906870 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0cf415a410 0x7f0cf4141270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:50.408+0000 7f0cfb967640 1 --2- 172.21.15.55:0/1108906870 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0cf41510f0 0x7f0cf41e20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:50.408+0000 7f0cfb967640 1 --2- 172.21.15.55:0/1108906870 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0cf40c73c0 0x7f0cf4148c60 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:50.408+0000 7f0cfb967640 1 -- 172.21.15.55:0/1108906870 >> 172.21.15.55:0/1108906870 conn(0x7f0cf40cc9c0 msgr2=0x7f0cf40c9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:50.408+0000 7f0cfb967640 1 -- 172.21.15.55:0/1108906870 shutdown_connections 2024-03-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:50.408+0000 7f0cfb967640 1 -- 172.21.15.55:0/1108906870 wait complete. 2024-03-19T02:03:51.027 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:51.027 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:52.028 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:52.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.148+0000 7fb87cf2c640 1 Processor -- start 2024-03-19T02:03:52.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.148+0000 7fb87cf2c640 1 -- start start 2024-03-19T02:03:52.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.148+0000 7fb87cf2c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb878154720 0x7fb87815e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:52.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.148+0000 7fb87cf2c640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fb8780c73c0 0x7fb87815aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:52.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.148+0000 7fb87cf2c640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb878071280 con 0x7fb878154340 2024-03-19T02:03:52.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.148+0000 7fb87cf2c640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fb878073840 con 0x7fb878154720 2024-03-19T02:03:52.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.148+0000 7fb87cf2c640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fb8780c7850 con 0x7fb8780c73c0 2024-03-19T02:03:52.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.148+0000 7fb875d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb878154720 0x7fb87815e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:52.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.148+0000 7fb875d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fb878154720 0x7fb87815e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:35620/0 (socket says 172.21.15.55:35620) 2024-03-19T02:03:52.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.148+0000 7fb875d9b640 1 -- 172.21.15.55:0/4254027485 learned_addr learned my addr 172.21.15.55:0/4254027485 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:52.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.148+0000 7fb876d9d640 1 --2- 172.21.15.55:0/4254027485 >> v2:172.21.15.84:3300/0 conn(0x7fb8780c73c0 0x7fb87815aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:52.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.152+0000 7fb875d9b640 1 -- 172.21.15.55:0/4254027485 >> v2:172.21.15.84:3300/0 conn(0x7fb8780c73c0 msgr2=0x7fb87815aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:52.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.152+0000 7fb875d9b640 1 --2- 172.21.15.55:0/4254027485 >> v2:172.21.15.84:3300/0 conn(0x7fb8780c73c0 0x7fb87815aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:52.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.152+0000 7fb875d9b640 1 -- 172.21.15.55:0/4254027485 >> v1:172.21.15.84:6789/0 conn(0x7fb878154340 legacy=0x7fb878072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:52.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.152+0000 7fb875d9b640 1 -- 172.21.15.55:0/4254027485 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87815d900 con 0x7fb878154720 2024-03-19T02:03:52.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.156+0000 7fb875d9b640 1 --2- 172.21.15.55:0/4254027485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb878154720 0x7fb87815e3a0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fb878154b40 tx=0x7fb86c01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e6480ad76e7c48b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:52.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.160+0000 7fb87559a640 1 -- 172.21.15.55:0/4254027485 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86c0b0070 con 0x7fb878154720 2024-03-19T02:03:52.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.160+0000 7fb87559a640 1 -- 172.21.15.55:0/4254027485 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb86c0ab600 con 0x7fb878154720 2024-03-19T02:03:52.162 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.160+0000 7fb87559a640 1 -- 172.21.15.55:0/4254027485 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86c0ab9d0 con 0x7fb878154720 2024-03-19T02:03:52.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.160+0000 7fb87cf2c640 1 -- 172.21.15.55:0/4254027485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb878154720 msgr2=0x7fb87815e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:52.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.160+0000 7fb87cf2c640 1 --2- 172.21.15.55:0/4254027485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb878154720 0x7fb87815e3a0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fb878154b40 tx=0x7fb86c01c4e0 comp rx=0 tx=0).stop 2024-03-19T02:03:52.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.160+0000 7fb87cf2c640 1 -- 172.21.15.55:0/4254027485 shutdown_connections 2024-03-19T02:03:52.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.160+0000 7fb87cf2c640 1 --2- 172.21.15.55:0/4254027485 >> v2:172.21.15.84:3300/0 conn(0x7fb8780c73c0 0x7fb87815aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:52.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.160+0000 7fb87cf2c640 1 --2- 172.21.15.55:0/4254027485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb878154720 0x7fb87815e3a0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:52.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.160+0000 7fb87cf2c640 1 -- 172.21.15.55:0/4254027485 >> 172.21.15.55:0/4254027485 conn(0x7fb8780cc9c0 msgr2=0x7fb8780ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:52.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.160+0000 7fb87cf2c640 1 -- 172.21.15.55:0/4254027485 shutdown_connections 2024-03-19T02:03:52.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.160+0000 7fb87cf2c640 1 -- 172.21.15.55:0/4254027485 wait complete. 2024-03-19T02:03:52.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.164+0000 7fb87cf2c640 1 Processor -- start 2024-03-19T02:03:52.165 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.164+0000 7fb87cf2c640 1 -- start start 2024-03-19T02:03:52.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.164+0000 7fb87cf2c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb8780c73c0 0x7fb878153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:52.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.164+0000 7fb87cf2c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb878154340 0x7fb878153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:52.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.164+0000 7fb87cf2c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb878154720 0x7fb8781f50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:52.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.164+0000 7fb87cf2c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb878166b70 con 0x7fb878154720 2024-03-19T02:03:52.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.164+0000 7fb87cf2c640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb87815b1e0 con 0x7fb878154340 2024-03-19T02:03:52.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.164+0000 7fb87cf2c640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fb878166cf0 con 0x7fb8780c73c0 2024-03-19T02:03:52.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.164+0000 7fb876d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb878154720 0x7fb8781f50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:52.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.164+0000 7fb87659c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb8780c73c0 0x7fb878153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:52.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.164+0000 7fb876d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb878154720 0x7fb8781f50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:35636/0 (socket says 172.21.15.55:35636) 2024-03-19T02:03:52.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.164+0000 7fb876d9d640 1 -- 172.21.15.55:0/989101823 learned_addr learned my addr 172.21.15.55:0/989101823 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:52.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.164+0000 7fb875d9b640 1 --2- 172.21.15.55:0/989101823 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb878154340 0x7fb878153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:52.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.168+0000 7fb876d9d640 1 -- 172.21.15.55:0/989101823 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb8780c73c0 msgr2=0x7fb878153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:52.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.168+0000 7fb876d9d640 1 --2- 172.21.15.55:0/989101823 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb8780c73c0 0x7fb878153120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:52.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.168+0000 7fb876d9d640 1 -- 172.21.15.55:0/989101823 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb878154340 msgr2=0x7fb878153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:52.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.168+0000 7fb876d9d640 1 --2- 172.21.15.55:0/989101823 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb878154340 0x7fb878153660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:52.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.168+0000 7fb876d9d640 1 -- 172.21.15.55:0/989101823 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87814aa00 con 0x7fb878154720 2024-03-19T02:03:52.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.172+0000 7fb876d9d640 1 --2- 172.21.15.55:0/989101823 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb878154720 0x7fb8781f50f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb87000efc0 tx=0x7fb87000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:52.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.176+0000 7fb866ffd640 1 -- 172.21.15.55:0/989101823 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb870018020 con 0x7fb878154720 2024-03-19T02:03:52.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.176+0000 7fb866ffd640 1 -- 172.21.15.55:0/989101823 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb870014480 con 0x7fb878154720 2024-03-19T02:03:52.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.176+0000 7fb87cf2c640 1 -- 172.21.15.55:0/989101823 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb878149790 con 0x7fb878154720 2024-03-19T02:03:52.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.176+0000 7fb866ffd640 1 -- 172.21.15.55:0/989101823 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb870014870 con 0x7fb878154720 2024-03-19T02:03:52.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.176+0000 7fb87cf2c640 1 -- 172.21.15.55:0/989101823 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8781f4610 con 0x7fb878154720 2024-03-19T02:03:52.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.176+0000 7fb87cf2c640 1 -- 172.21.15.55:0/989101823 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb878068a40 con 0x7fb878154720 2024-03-19T02:03:52.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.212+0000 7fb866ffd640 1 -- 172.21.15.55:0/989101823 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb87001b020 con 0x7fb878154720 2024-03-19T02:03:52.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.212+0000 7fb866ffd640 1 --2- 172.21.15.55:0/989101823 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb8580676e0 0x7fb858069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:52.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.216+0000 7fb87659c640 1 --2- 172.21.15.55:0/989101823 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb8580676e0 0x7fb858069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:52.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.216+0000 7fb87659c640 1 --2- 172.21.15.55:0/989101823 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb8580676e0 0x7fb858069ba0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb8780cbbd0 tx=0x7fb868009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:52.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.220+0000 7fb866ffd640 1 -- 172.21.15.55:0/989101823 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fb870055160 con 0x7fb878154720 2024-03-19T02:03:52.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.516+0000 7fb866ffd640 1 -- 172.21.15.55:0/989101823 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fb870022030 con 0x7fb878154720 2024-03-19T02:03:52.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.728+0000 7fb87cf2c640 1 -- 172.21.15.55:0/989101823 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8780c64e0 con 0x7fb878154720 2024-03-19T02:03:52.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.804+0000 7fb866ffd640 1 -- 172.21.15.55:0/989101823 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fb87814aa00 con 0x7fb878154720 2024-03-19T02:03:52.806 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:52.806 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:52.806 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:52.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.808+0000 7fb87cf2c640 1 -- 172.21.15.55:0/989101823 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb8580676e0 msgr2=0x7fb858069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:52.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.808+0000 7fb87cf2c640 1 --2- 172.21.15.55:0/989101823 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb8580676e0 0x7fb858069ba0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb8780cbbd0 tx=0x7fb868009040 comp rx=0 tx=0).stop 2024-03-19T02:03:52.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.808+0000 7fb87cf2c640 1 -- 172.21.15.55:0/989101823 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb878154720 msgr2=0x7fb8781f50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:52.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.808+0000 7fb87cf2c640 1 --2- 172.21.15.55:0/989101823 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb878154720 0x7fb8781f50f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb87000efc0 tx=0x7fb87000c490 comp rx=0 tx=0).stop 2024-03-19T02:03:52.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.808+0000 7fb87cf2c640 1 -- 172.21.15.55:0/989101823 shutdown_connections 2024-03-19T02:03:52.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.808+0000 7fb87cf2c640 1 --2- 172.21.15.55:0/989101823 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb8580676e0 0x7fb858069ba0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:52.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.808+0000 7fb87cf2c640 1 --2- 172.21.15.55:0/989101823 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb878154720 0x7fb8781f50f0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:52.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.808+0000 7fb87cf2c640 1 --2- 172.21.15.55:0/989101823 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb878154340 0x7fb878153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:52.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.808+0000 7fb87cf2c640 1 --2- 172.21.15.55:0/989101823 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb8780c73c0 0x7fb878153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:52.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.808+0000 7fb87cf2c640 1 -- 172.21.15.55:0/989101823 >> 172.21.15.55:0/989101823 conn(0x7fb8780cc9c0 msgr2=0x7fb8780c56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:52.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.808+0000 7fb87cf2c640 1 -- 172.21.15.55:0/989101823 shutdown_connections 2024-03-19T02:03:52.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:52.808+0000 7fb87cf2c640 1 -- 172.21.15.55:0/989101823 wait complete. 2024-03-19T02:03:53.430 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:53.430 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:54.431 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:54.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.560+0000 7fbe81fea640 1 Processor -- start 2024-03-19T02:03:54.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.560+0000 7fbe81fea640 1 -- start start 2024-03-19T02:03:54.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.560+0000 7fbe81fea640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fbe7c154340 0x7fbe7c15e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:54.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.560+0000 7fbe81fea640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbe7c154720 0x7fbe7c0c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:54.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.560+0000 7fbe81fea640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fbe7c15aca0 0x7fbe7c072e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:54.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.560+0000 7fbe81fea640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fbe7c073840 con 0x7fbe7c154340 2024-03-19T02:03:54.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.560+0000 7fbe81fea640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fbe7c15df10 con 0x7fbe7c15aca0 2024-03-19T02:03:54.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.560+0000 7fbe81fea640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbe7c071280 con 0x7fbe7c154720 2024-03-19T02:03:54.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.560+0000 7fbe7bfff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fbe7c15aca0 0x7fbe7c072e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:54.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.560+0000 7fbe7affd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbe7c154720 0x7fbe7c0c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:54.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.564+0000 7fbe7affd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbe7c154720 0x7fbe7c0c73c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:54898/0 (socket says 172.21.15.55:54898) 2024-03-19T02:03:54.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.564+0000 7fbe7bfff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fbe7c15aca0 0x7fbe7c072e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:39470/0 (socket says 172.21.15.55:39470) 2024-03-19T02:03:54.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.564+0000 7fbe7affd640 1 -- 172.21.15.55:0/4190423265 learned_addr learned my addr 172.21.15.55:0/4190423265 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:54.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.564+0000 7fbe7affd640 1 -- 172.21.15.55:0/4190423265 >> v2:172.21.15.84:3300/0 conn(0x7fbe7c15aca0 msgr2=0x7fbe7c072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:54.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.564+0000 7fbe7affd640 1 --2- 172.21.15.55:0/4190423265 >> v2:172.21.15.84:3300/0 conn(0x7fbe7c15aca0 0x7fbe7c072e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:54.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.564+0000 7fbe7affd640 1 -- 172.21.15.55:0/4190423265 >> v2:172.21.15.55:3300/0 conn(0x7fbe7c154340 msgr2=0x7fbe7c15e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:54.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.564+0000 7fbe7affd640 1 --2- 172.21.15.55:0/4190423265 >> v2:172.21.15.55:3300/0 conn(0x7fbe7c154340 0x7fbe7c15e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:54.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.564+0000 7fbe7affd640 1 -- 172.21.15.55:0/4190423265 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe7c15d900 con 0x7fbe7c154720 2024-03-19T02:03:54.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.568+0000 7fbe7affd640 1 --2- 172.21.15.55:0/4190423265 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbe7c154720 0x7fbe7c0c73c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c0c3a70 tx=0x7fbe7001c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60e558ba5eb47f66 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:54.576 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.576+0000 7fbe7a7fc640 1 -- 172.21.15.55:0/4190423265 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe700b1070 con 0x7fbe7c154720 2024-03-19T02:03:54.577 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.576+0000 7fbe7a7fc640 1 -- 172.21.15.55:0/4190423265 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe700ac630 con 0x7fbe7c154720 2024-03-19T02:03:54.577 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.576+0000 7fbe7a7fc640 1 -- 172.21.15.55:0/4190423265 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe700a8050 con 0x7fbe7c154720 2024-03-19T02:03:54.577 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.576+0000 7fbe81fea640 1 -- 172.21.15.55:0/4190423265 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbe7c154720 msgr2=0x7fbe7c0c73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:54.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.576+0000 7fbe81fea640 1 --2- 172.21.15.55:0/4190423265 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbe7c154720 0x7fbe7c0c73c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c0c3a70 tx=0x7fbe7001c520 comp rx=0 tx=0).stop 2024-03-19T02:03:54.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.576+0000 7fbe81fea640 1 -- 172.21.15.55:0/4190423265 shutdown_connections 2024-03-19T02:03:54.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.576+0000 7fbe81fea640 1 --2- 172.21.15.55:0/4190423265 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbe7c154720 0x7fbe7c0c73c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:54.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.576+0000 7fbe81fea640 1 --2- 172.21.15.55:0/4190423265 >> v2:172.21.15.84:3300/0 conn(0x7fbe7c15aca0 0x7fbe7c072e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:54.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.576+0000 7fbe81fea640 1 --2- 172.21.15.55:0/4190423265 >> v2:172.21.15.55:3300/0 conn(0x7fbe7c154340 0x7fbe7c15e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:54.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.576+0000 7fbe81fea640 1 -- 172.21.15.55:0/4190423265 >> 172.21.15.55:0/4190423265 conn(0x7fbe7c0cc9c0 msgr2=0x7fbe7c0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:54.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.576+0000 7fbe81fea640 1 -- 172.21.15.55:0/4190423265 shutdown_connections 2024-03-19T02:03:54.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.576+0000 7fbe81fea640 1 -- 172.21.15.55:0/4190423265 wait complete. 2024-03-19T02:03:54.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.576+0000 7fbe81fea640 1 Processor -- start 2024-03-19T02:03:54.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.576+0000 7fbe81fea640 1 -- start start 2024-03-19T02:03:54.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.580+0000 7fbe81fea640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbe7c154340 0x7fbe7c161a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:54.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.580+0000 7fbe81fea640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe7c154720 0x7fbe7c157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:54.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.580+0000 7fbe81fea640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe7c15aca0 0x7fbe7c1f6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:54.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.580+0000 7fbe81fea640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7c15b080 con 0x7fbe7c15aca0 2024-03-19T02:03:54.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.580+0000 7fbe81fea640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7c0733d0 con 0x7fbe7c154720 2024-03-19T02:03:54.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.580+0000 7fbe81fea640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbe7c15b200 con 0x7fbe7c154340 2024-03-19T02:03:54.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.580+0000 7fbe7affd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe7c154720 0x7fbe7c157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:54.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.580+0000 7fbe7b7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbe7c154340 0x7fbe7c161a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:54.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.580+0000 7fbe7affd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe7c154720 0x7fbe7c157050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:39482/0 (socket says 172.21.15.55:39482) 2024-03-19T02:03:54.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.580+0000 7fbe7affd640 1 -- 172.21.15.55:0/2087847093 learned_addr learned my addr 172.21.15.55:0/2087847093 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:54.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.584+0000 7fbe7b7fe640 1 -- 172.21.15.55:0/2087847093 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe7c154720 msgr2=0x7fbe7c157050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:54.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.584+0000 7fbe7b7fe640 1 --2- 172.21.15.55:0/2087847093 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe7c154720 0x7fbe7c157050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:54.590 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.584+0000 7fbe7b7fe640 1 -- 172.21.15.55:0/2087847093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe7c15aca0 msgr2=0x7fbe7c1f6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:54.590 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.584+0000 7fbe7b7fe640 1 --2- 172.21.15.55:0/2087847093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe7c15aca0 0x7fbe7c1f6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:54.590 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.584+0000 7fbe7b7fe640 1 -- 172.21.15.55:0/2087847093 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe7c1f6ae0 con 0x7fbe7c154340 2024-03-19T02:03:54.590 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.588+0000 7fbe7b7fe640 1 --2- 172.21.15.55:0/2087847093 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbe7c154340 0x7fbe7c161a00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c00b420 tx=0x7fbe6c00b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:54.597 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.596+0000 7fbe63fff640 1 -- 172.21.15.55:0/2087847093 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c015020 con 0x7fbe7c154340 2024-03-19T02:03:54.597 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.596+0000 7fbe63fff640 1 -- 172.21.15.55:0/2087847093 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe6c011830 con 0x7fbe7c154340 2024-03-19T02:03:54.597 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.596+0000 7fbe81fea640 1 -- 172.21.15.55:0/2087847093 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe7c06be00 con 0x7fbe7c154340 2024-03-19T02:03:54.597 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.596+0000 7fbe63fff640 1 -- 172.21.15.55:0/2087847093 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c011c20 con 0x7fbe7c154340 2024-03-19T02:03:54.597 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.596+0000 7fbe81fea640 1 -- 172.21.15.55:0/2087847093 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe7c156b30 con 0x7fbe7c154340 2024-03-19T02:03:54.597 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.596+0000 7fbe81fea640 1 -- 172.21.15.55:0/2087847093 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe7c068a40 con 0x7fbe7c154340 2024-03-19T02:03:54.636 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.636+0000 7fbe63fff640 1 -- 172.21.15.55:0/2087847093 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fbe6c018020 con 0x7fbe7c154340 2024-03-19T02:03:54.637 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.636+0000 7fbe63fff640 1 --2- 172.21.15.55:0/2087847093 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbe680676b0 0x7fbe68069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:54.637 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.636+0000 7fbe7affd640 1 --2- 172.21.15.55:0/2087847093 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbe680676b0 0x7fbe68069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:54.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.636+0000 7fbe7affd640 1 --2- 172.21.15.55:0/2087847093 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbe680676b0 0x7fbe68069b70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fbe70000c10 tx=0x7fbe700a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:54.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.892+0000 7fbe63fff640 1 -- 172.21.15.55:0/2087847093 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fbe6c054330 con 0x7fbe7c154340 2024-03-19T02:03:54.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:54.916+0000 7fbe63fff640 1 -- 172.21.15.55:0/2087847093 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fbe6c01f020 con 0x7fbe7c154340 2024-03-19T02:03:55.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:55.136+0000 7fbe81fea640 1 -- 172.21.15.55:0/2087847093 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe7c0cb7f0 con 0x7fbe7c154340 2024-03-19T02:03:55.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:55.252+0000 7fbe63fff640 1 -- 172.21.15.55:0/2087847093 <== mon.2 v2:172.21.15.84:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fbe6c056990 con 0x7fbe7c154340 2024-03-19T02:03:55.253 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:55.253 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:55.253 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:55.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:55.256+0000 7fbe81fea640 1 -- 172.21.15.55:0/2087847093 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbe680676b0 msgr2=0x7fbe68069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:55.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:55.256+0000 7fbe81fea640 1 --2- 172.21.15.55:0/2087847093 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbe680676b0 0x7fbe68069b70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fbe70000c10 tx=0x7fbe700a7000 comp rx=0 tx=0).stop 2024-03-19T02:03:55.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:55.256+0000 7fbe81fea640 1 -- 172.21.15.55:0/2087847093 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbe7c154340 msgr2=0x7fbe7c161a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:55.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:55.256+0000 7fbe81fea640 1 --2- 172.21.15.55:0/2087847093 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbe7c154340 0x7fbe7c161a00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c00b420 tx=0x7fbe6c00b8e0 comp rx=0 tx=0).stop 2024-03-19T02:03:55.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:55.256+0000 7fbe81fea640 1 -- 172.21.15.55:0/2087847093 shutdown_connections 2024-03-19T02:03:55.257 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:55.256+0000 7fbe81fea640 1 --2- 172.21.15.55:0/2087847093 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbe680676b0 0x7fbe68069b70 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:55.257 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:55.256+0000 7fbe81fea640 1 --2- 172.21.15.55:0/2087847093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbe7c15aca0 0x7fbe7c1f6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:55.257 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:55.256+0000 7fbe81fea640 1 --2- 172.21.15.55:0/2087847093 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe7c154720 0x7fbe7c157050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:55.257 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:55.256+0000 7fbe81fea640 1 --2- 172.21.15.55:0/2087847093 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbe7c154340 0x7fbe7c161a00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:55.257 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:55.256+0000 7fbe81fea640 1 -- 172.21.15.55:0/2087847093 >> 172.21.15.55:0/2087847093 conn(0x7fbe7c0cc9c0 msgr2=0x7fbe7c159c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:55.257 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:55.256+0000 7fbe81fea640 1 -- 172.21.15.55:0/2087847093 shutdown_connections 2024-03-19T02:03:55.257 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:55.256+0000 7fbe81fea640 1 -- 172.21.15.55:0/2087847093 wait complete. 2024-03-19T02:03:55.879 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:55.879 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:56.880 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:56.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:56.992+0000 7f74ddb5a640 1 Processor -- start 2024-03-19T02:03:56.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:56.992+0000 7f74ddb5a640 1 -- start start 2024-03-19T02:03:56.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:56.992+0000 7f74ddb5a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f74d81544f0 0x7f74d8073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:56.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:56.992+0000 7f74ddb5a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f74d815acb0 0x7f74d80cce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:56.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:56.992+0000 7f74ddb5a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74d80c7fa0 con 0x7f74d806b2a0 2024-03-19T02:03:56.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:56.992+0000 7f74ddb5a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f74d815b210 con 0x7f74d81544f0 2024-03-19T02:03:56.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:56.992+0000 7f74ddb5a640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f74d80c8240 con 0x7f74d815acb0 2024-03-19T02:03:56.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:56.992+0000 7f74d77fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f74d81544f0 0x7f74d8073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:56.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:56.996+0000 7f74d7fff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f74d815acb0 0x7f74d80cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:56.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:56.996+0000 7f74d77fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f74d81544f0 0x7f74d8073540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:35658/0 (socket says 172.21.15.55:35658) 2024-03-19T02:03:56.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:56.996+0000 7f74d77fe640 1 -- 172.21.15.55:0/287234949 learned_addr learned my addr 172.21.15.55:0/287234949 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:56.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:56.996+0000 7f74d77fe640 1 -- 172.21.15.55:0/287234949 >> v2:172.21.15.84:3300/0 conn(0x7f74d815acb0 msgr2=0x7f74d80cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:56.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:56.996+0000 7f74d77fe640 1 --2- 172.21.15.55:0/287234949 >> v2:172.21.15.84:3300/0 conn(0x7f74d815acb0 0x7f74d80cce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:56.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:56.996+0000 7f74d77fe640 1 -- 172.21.15.55:0/287234949 >> v1:172.21.15.84:6789/0 conn(0x7f74d806b2a0 legacy=0x7f74d815e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:56.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:56.996+0000 7f74d77fe640 1 -- 172.21.15.55:0/287234949 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74d815d920 con 0x7f74d81544f0 2024-03-19T02:03:57.003 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.004+0000 7f74d77fe640 1 --2- 172.21.15.55:0/287234949 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74d81544f0 0x7f74d8073540 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f74d80ca740 tx=0x7f74c801ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8830093df2db7d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:57.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.008+0000 7f74d67fc640 1 -- 172.21.15.55:0/287234949 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c80b5070 con 0x7f74d81544f0 2024-03-19T02:03:57.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.008+0000 7f74d67fc640 1 -- 172.21.15.55:0/287234949 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74c80a89d0 con 0x7f74d81544f0 2024-03-19T02:03:57.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.008+0000 7f74d67fc640 1 -- 172.21.15.55:0/287234949 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c80a8da0 con 0x7f74d81544f0 2024-03-19T02:03:57.010 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.008+0000 7f74ddb5a640 1 -- 172.21.15.55:0/287234949 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74d81544f0 msgr2=0x7f74d8073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:57.010 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.008+0000 7f74ddb5a640 1 --2- 172.21.15.55:0/287234949 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74d81544f0 0x7f74d8073540 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f74d80ca740 tx=0x7f74c801ecd0 comp rx=0 tx=0).stop 2024-03-19T02:03:57.010 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.008+0000 7f74ddb5a640 1 -- 172.21.15.55:0/287234949 shutdown_connections 2024-03-19T02:03:57.010 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.008+0000 7f74ddb5a640 1 --2- 172.21.15.55:0/287234949 >> v2:172.21.15.84:3300/0 conn(0x7f74d815acb0 0x7f74d80cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:57.010 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.008+0000 7f74ddb5a640 1 --2- 172.21.15.55:0/287234949 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74d81544f0 0x7f74d8073540 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:57.010 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.008+0000 7f74ddb5a640 1 -- 172.21.15.55:0/287234949 >> 172.21.15.55:0/287234949 conn(0x7f74d806e670 msgr2=0x7f74d806ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:57.010 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.008+0000 7f74ddb5a640 1 -- 172.21.15.55:0/287234949 shutdown_connections 2024-03-19T02:03:57.010 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.008+0000 7f74ddb5a640 1 -- 172.21.15.55:0/287234949 wait complete. 2024-03-19T02:03:57.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.008+0000 7f74ddb5a640 1 Processor -- start 2024-03-19T02:03:57.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.008+0000 7f74ddb5a640 1 -- start start 2024-03-19T02:03:57.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.012+0000 7f74ddb5a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74d806b2a0 0x7f74d8151a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:57.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.012+0000 7f74ddb5a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74d815acb0 0x7f74d80cc000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:57.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.012+0000 7f74ddb5a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f74d81f4510 0x7f74d80cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:57.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.012+0000 7f74ddb5a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f74d80ca120 con 0x7f74d815acb0 2024-03-19T02:03:57.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.012+0000 7f74ddb5a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f74d81548d0 con 0x7f74d806b2a0 2024-03-19T02:03:57.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.012+0000 7f74ddb5a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f74d80ca2a0 con 0x7f74d81f4510 2024-03-19T02:03:57.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.012+0000 7f74d6ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74d815acb0 0x7f74d80cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:57.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.012+0000 7f74d6ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74d815acb0 0x7f74d80cc000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:35660/0 (socket says 172.21.15.55:35660) 2024-03-19T02:03:57.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.012+0000 7f74d6ffd640 1 -- 172.21.15.55:0/921729225 learned_addr learned my addr 172.21.15.55:0/921729225 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:57.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.012+0000 7f74d77fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74d806b2a0 0x7f74d8151a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:57.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.012+0000 7f74d7fff640 1 --2- 172.21.15.55:0/921729225 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f74d81f4510 0x7f74d80cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:57.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.016+0000 7f74d6ffd640 1 -- 172.21.15.55:0/921729225 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f74d81f4510 msgr2=0x7f74d80cc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:57.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.016+0000 7f74d6ffd640 1 --2- 172.21.15.55:0/921729225 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f74d81f4510 0x7f74d80cc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:57.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.016+0000 7f74d6ffd640 1 -- 172.21.15.55:0/921729225 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74d806b2a0 msgr2=0x7f74d8151a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:57.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.016+0000 7f74d6ffd640 1 --2- 172.21.15.55:0/921729225 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74d806b2a0 0x7f74d8151a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:57.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.016+0000 7f74d6ffd640 1 -- 172.21.15.55:0/921729225 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74d81f07a0 con 0x7f74d815acb0 2024-03-19T02:03:57.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.020+0000 7f74d6ffd640 1 --2- 172.21.15.55:0/921729225 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74d815acb0 0x7f74d80cc000 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00c490 tx=0x7f74cc001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:57.026 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.024+0000 7f74bffff640 1 -- 172.21.15.55:0/921729225 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74cc017070 con 0x7f74d815acb0 2024-03-19T02:03:57.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.024+0000 7f74bffff640 1 -- 172.21.15.55:0/921729225 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74cc003600 con 0x7f74d815acb0 2024-03-19T02:03:57.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.024+0000 7f74ddb5a640 1 -- 172.21.15.55:0/921729225 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74d81f0a80 con 0x7f74d815acb0 2024-03-19T02:03:57.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.024+0000 7f74bffff640 1 -- 172.21.15.55:0/921729225 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74cc003a30 con 0x7f74d815acb0 2024-03-19T02:03:57.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.024+0000 7f74ddb5a640 1 -- 172.21.15.55:0/921729225 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74d81f1040 con 0x7f74d815acb0 2024-03-19T02:03:57.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.024+0000 7f74ddb5a640 1 -- 172.21.15.55:0/921729225 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74d8068a40 con 0x7f74d815acb0 2024-03-19T02:03:57.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.304+0000 7f74bffff640 1 -- 172.21.15.55:0/921729225 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f74cc01a070 con 0x7f74d815acb0 2024-03-19T02:03:57.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.308+0000 7f74bffff640 1 --2- 172.21.15.55:0/921729225 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f74c40676e0 0x7f74c4069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:57.308 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.308+0000 7f74d77fe640 1 --2- 172.21.15.55:0/921729225 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f74c40676e0 0x7f74c4069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:57.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.308+0000 7f74d77fe640 1 --2- 172.21.15.55:0/921729225 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f74c40676e0 0x7f74c4069ba0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f74c80b2000 tx=0x7f74c8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:57.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.332+0000 7f74bffff640 1 -- 172.21.15.55:0/921729225 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f74cc0537a0 con 0x7f74d815acb0 2024-03-19T02:03:57.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.332+0000 7f74bffff640 1 -- 172.21.15.55:0/921729225 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f74cc021020 con 0x7f74d815acb0 2024-03-19T02:03:57.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.580+0000 7f74ddb5a640 1 -- 172.21.15.55:0/921729225 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74d8070770 con 0x7f74d815acb0 2024-03-19T02:03:57.593 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.592+0000 7f74bffff640 1 -- 172.21.15.55:0/921729225 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f74cc055700 con 0x7f74d815acb0 2024-03-19T02:03:57.594 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:57.594 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:57.594 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:57.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.596+0000 7f74ddb5a640 1 -- 172.21.15.55:0/921729225 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f74c40676e0 msgr2=0x7f74c4069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:57.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.596+0000 7f74ddb5a640 1 --2- 172.21.15.55:0/921729225 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f74c40676e0 0x7f74c4069ba0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f74c80b2000 tx=0x7f74c8009040 comp rx=0 tx=0).stop 2024-03-19T02:03:57.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.596+0000 7f74ddb5a640 1 -- 172.21.15.55:0/921729225 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74d815acb0 msgr2=0x7f74d80cc000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:57.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.596+0000 7f74ddb5a640 1 --2- 172.21.15.55:0/921729225 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74d815acb0 0x7f74d80cc000 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00c490 tx=0x7f74cc001e80 comp rx=0 tx=0).stop 2024-03-19T02:03:57.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.596+0000 7f74ddb5a640 1 -- 172.21.15.55:0/921729225 shutdown_connections 2024-03-19T02:03:57.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.596+0000 7f74ddb5a640 1 --2- 172.21.15.55:0/921729225 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f74c40676e0 0x7f74c4069ba0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:57.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.596+0000 7f74ddb5a640 1 --2- 172.21.15.55:0/921729225 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f74d81f4510 0x7f74d80cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:57.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.596+0000 7f74ddb5a640 1 --2- 172.21.15.55:0/921729225 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74d815acb0 0x7f74d80cc000 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:57.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.596+0000 7f74ddb5a640 1 --2- 172.21.15.55:0/921729225 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74d806b2a0 0x7f74d8151a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:57.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.596+0000 7f74ddb5a640 1 -- 172.21.15.55:0/921729225 >> 172.21.15.55:0/921729225 conn(0x7f74d806e670 msgr2=0x7f74d815e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:57.597 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.596+0000 7f74ddb5a640 1 -- 172.21.15.55:0/921729225 shutdown_connections 2024-03-19T02:03:57.597 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:57.596+0000 7f74ddb5a640 1 -- 172.21.15.55:0/921729225 wait complete. 2024-03-19T02:03:58.215 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:03:58.216 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:03:59.217 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:03:59.346 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.344+0000 7fd69cc81640 1 Processor -- start 2024-03-19T02:03:59.346 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.344+0000 7fd69cc81640 1 -- start start 2024-03-19T02:03:59.347 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.344+0000 7fd69cc81640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd6980c96b0 0x7fd698070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:59.347 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.344+0000 7fd69cc81640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd6980c9a90 0x7fd6980c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:59.347 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.344+0000 7fd69cc81640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd69815e3a0 con 0x7fd6981540c0 2024-03-19T02:03:59.347 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.344+0000 7fd69cc81640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fd69815e680 con 0x7fd6980c96b0 2024-03-19T02:03:59.347 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.344+0000 7fd69cc81640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd69815a330 con 0x7fd6980c9a90 2024-03-19T02:03:59.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.348+0000 7fd69659c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd6980c96b0 0x7fd698070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:59.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.348+0000 7fd696d9d640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fd6981540c0 0x7fd69815d810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:56284/0 (socket says 172.21.15.55:56284) 2024-03-19T02:03:59.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.348+0000 7fd696d9d640 1 -- 172.21.15.55:0/3279403340 learned_addr learned my addr 172.21.15.55:0/3279403340 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:03:59.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.348+0000 7fd695d9b640 1 --2- 172.21.15.55:0/3279403340 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd6980c9a90 0x7fd6980c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:59.351 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.352+0000 7fd69659c640 1 -- 172.21.15.55:0/3279403340 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd6980c9a90 msgr2=0x7fd6980c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:03:59.351 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.352+0000 7fd69659c640 1 --2- 172.21.15.55:0/3279403340 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd6980c9a90 0x7fd6980c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:59.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.352+0000 7fd69659c640 1 -- 172.21.15.55:0/3279403340 >> v1:172.21.15.84:6789/0 conn(0x7fd6981540c0 legacy=0x7fd69815d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:59.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.352+0000 7fd69659c640 1 -- 172.21.15.55:0/3279403340 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd69815a610 con 0x7fd6980c96b0 2024-03-19T02:03:59.355 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.352+0000 7fd69659c640 1 --2- 172.21.15.55:0/3279403340 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd6980c96b0 0x7fd698070f20 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fd6880097c0 tx=0x7fd68801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=986db43732f0e671 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:59.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.360+0000 7fd69559a640 1 -- 172.21.15.55:0/3279403340 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6880b0070 con 0x7fd6980c96b0 2024-03-19T02:03:59.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.360+0000 7fd69559a640 1 -- 172.21.15.55:0/3279403340 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6880ab600 con 0x7fd6980c96b0 2024-03-19T02:03:59.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.360+0000 7fd69559a640 1 -- 172.21.15.55:0/3279403340 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6880ab9d0 con 0x7fd6980c96b0 2024-03-19T02:03:59.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.360+0000 7fd69cc81640 1 -- 172.21.15.55:0/3279403340 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd6980c96b0 msgr2=0x7fd698070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:59.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.360+0000 7fd69cc81640 1 --2- 172.21.15.55:0/3279403340 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd6980c96b0 0x7fd698070f20 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fd6880097c0 tx=0x7fd68801c410 comp rx=0 tx=0).stop 2024-03-19T02:03:59.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.360+0000 7fd69cc81640 1 -- 172.21.15.55:0/3279403340 shutdown_connections 2024-03-19T02:03:59.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.360+0000 7fd69cc81640 1 --2- 172.21.15.55:0/3279403340 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd6980c9a90 0x7fd6980c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:59.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.360+0000 7fd69cc81640 1 --2- 172.21.15.55:0/3279403340 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd6980c96b0 0x7fd698070f20 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:59.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.360+0000 7fd69cc81640 1 -- 172.21.15.55:0/3279403340 >> 172.21.15.55:0/3279403340 conn(0x7fd6980cc9c0 msgr2=0x7fd6980ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:59.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.360+0000 7fd69cc81640 1 -- 172.21.15.55:0/3279403340 shutdown_connections 2024-03-19T02:03:59.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.360+0000 7fd69cc81640 1 -- 172.21.15.55:0/3279403340 wait complete. 2024-03-19T02:03:59.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.360+0000 7fd69cc81640 1 Processor -- start 2024-03-19T02:03:59.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.360+0000 7fd69cc81640 1 -- start start 2024-03-19T02:03:59.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.360+0000 7fd69cc81640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd6980c96b0 0x7fd698155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:59.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.360+0000 7fd69cc81640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd6980c9a90 0x7fd6981f2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:59.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.360+0000 7fd69cc81640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd6981540c0 0x7fd698151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:59.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.360+0000 7fd69cc81640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd6980c7900 con 0x7fd6981540c0 2024-03-19T02:03:59.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.360+0000 7fd69cc81640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd69815b0f0 con 0x7fd6980c9a90 2024-03-19T02:03:59.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.360+0000 7fd69cc81640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd6980c7a80 con 0x7fd6980c96b0 2024-03-19T02:03:59.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.364+0000 7fd696d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd6981540c0 0x7fd698151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:59.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.364+0000 7fd69659c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd6980c96b0 0x7fd698155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:59.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.364+0000 7fd696d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd6981540c0 0x7fd698151940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:37792/0 (socket says 172.21.15.55:37792) 2024-03-19T02:03:59.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.364+0000 7fd696d9d640 1 -- 172.21.15.55:0/3901886432 learned_addr learned my addr 172.21.15.55:0/3901886432 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:03:59.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.364+0000 7fd695d9b640 1 --2- 172.21.15.55:0/3901886432 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd6980c9a90 0x7fd6981f2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:59.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.368+0000 7fd696d9d640 1 -- 172.21.15.55:0/3901886432 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd6980c96b0 msgr2=0x7fd698155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:59.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.368+0000 7fd696d9d640 1 --2- 172.21.15.55:0/3901886432 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd6980c96b0 0x7fd698155860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:59.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.368+0000 7fd696d9d640 1 -- 172.21.15.55:0/3901886432 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd6980c9a90 msgr2=0x7fd6981f2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:59.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.368+0000 7fd696d9d640 1 --2- 172.21.15.55:0/3901886432 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd6980c9a90 0x7fd6981f2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:59.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.368+0000 7fd696d9d640 1 -- 172.21.15.55:0/3901886432 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6981f76b0 con 0x7fd6981540c0 2024-03-19T02:03:59.370 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.368+0000 7fd696d9d640 1 --2- 172.21.15.55:0/3901886432 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd6981540c0 0x7fd698151940 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd69000cab0 tx=0x7fd69000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:59.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.372+0000 7fd686ffd640 1 -- 172.21.15.55:0/3901886432 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd690017070 con 0x7fd6981540c0 2024-03-19T02:03:59.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.372+0000 7fd686ffd640 1 -- 172.21.15.55:0/3901886432 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd69000ac30 con 0x7fd6981540c0 2024-03-19T02:03:59.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.372+0000 7fd69cc81640 1 -- 172.21.15.55:0/3901886432 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd69814b720 con 0x7fd6981540c0 2024-03-19T02:03:59.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.372+0000 7fd686ffd640 1 -- 172.21.15.55:0/3901886432 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd690012400 con 0x7fd6981540c0 2024-03-19T02:03:59.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.372+0000 7fd69cc81640 1 -- 172.21.15.55:0/3901886432 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd69814b160 con 0x7fd6981540c0 2024-03-19T02:03:59.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.376+0000 7fd69cc81640 1 -- 172.21.15.55:0/3901886432 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd698068a40 con 0x7fd6981540c0 2024-03-19T02:03:59.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.404+0000 7fd686ffd640 1 -- 172.21.15.55:0/3901886432 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fd69001a070 con 0x7fd6981540c0 2024-03-19T02:03:59.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.408+0000 7fd686ffd640 1 --2- 172.21.15.55:0/3901886432 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd6780676b0 0x7fd678069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:03:59.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.408+0000 7fd69659c640 1 --2- 172.21.15.55:0/3901886432 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd6780676b0 0x7fd678069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:03:59.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.408+0000 7fd69659c640 1 --2- 172.21.15.55:0/3901886432 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd6780676b0 0x7fd678069b70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd6880ad000 tx=0x7fd6880a4f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:03:59.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.412+0000 7fd686ffd640 1 -- 172.21.15.55:0/3901886432 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fd690053fc0 con 0x7fd6981540c0 2024-03-19T02:03:59.692 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.692+0000 7fd686ffd640 1 -- 172.21.15.55:0/3901886432 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fd690055f20 con 0x7fd6981540c0 2024-03-19T02:03:59.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.912+0000 7fd69cc81640 1 -- 172.21.15.55:0/3901886432 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6980c8dc0 con 0x7fd6981540c0 2024-03-19T02:03:59.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.932+0000 7fd686ffd640 1 -- 172.21.15.55:0/3901886432 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fd690059fc0 con 0x7fd6981540c0 2024-03-19T02:03:59.933 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:03:59.933 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:03:59.933 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:03:59.935 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.932+0000 7fd69cc81640 1 -- 172.21.15.55:0/3901886432 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd6780676b0 msgr2=0x7fd678069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:59.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.932+0000 7fd69cc81640 1 --2- 172.21.15.55:0/3901886432 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd6780676b0 0x7fd678069b70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd6880ad000 tx=0x7fd6880a4f90 comp rx=0 tx=0).stop 2024-03-19T02:03:59.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.932+0000 7fd69cc81640 1 -- 172.21.15.55:0/3901886432 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd6981540c0 msgr2=0x7fd698151940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:03:59.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.932+0000 7fd69cc81640 1 --2- 172.21.15.55:0/3901886432 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd6981540c0 0x7fd698151940 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd69000cab0 tx=0x7fd69000cf70 comp rx=0 tx=0).stop 2024-03-19T02:03:59.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.932+0000 7fd69cc81640 1 -- 172.21.15.55:0/3901886432 shutdown_connections 2024-03-19T02:03:59.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.932+0000 7fd69cc81640 1 --2- 172.21.15.55:0/3901886432 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd6780676b0 0x7fd678069b70 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:59.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.932+0000 7fd69cc81640 1 --2- 172.21.15.55:0/3901886432 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd6981540c0 0x7fd698151940 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:59.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.932+0000 7fd69cc81640 1 --2- 172.21.15.55:0/3901886432 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd6980c9a90 0x7fd6981f2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:59.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.932+0000 7fd69cc81640 1 --2- 172.21.15.55:0/3901886432 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd6980c96b0 0x7fd698155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:03:59.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.932+0000 7fd69cc81640 1 -- 172.21.15.55:0/3901886432 >> 172.21.15.55:0/3901886432 conn(0x7fd6980cc9c0 msgr2=0x7fd69815d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:03:59.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.932+0000 7fd69cc81640 1 -- 172.21.15.55:0/3901886432 shutdown_connections 2024-03-19T02:03:59.936 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:03:59.932+0000 7fd69cc81640 1 -- 172.21.15.55:0/3901886432 wait complete. 2024-03-19T02:04:00.560 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:00.560 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:01.561 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:01.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.684+0000 7f7ddae3a640 1 Processor -- start 2024-03-19T02:04:01.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.684+0000 7f7ddae3a640 1 -- start start 2024-03-19T02:04:01.688 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.688+0000 7f7ddae3a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7dd4152150 0x7f7dd4158a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:01.689 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.688+0000 7f7ddae3a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7dd4070970 0x7f7dd40c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:01.689 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.688+0000 7f7ddae3a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dd4158f40 con 0x7f7dd40c9600 2024-03-19T02:04:01.689 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.688+0000 7f7ddae3a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f7dd4071130 con 0x7f7dd4070970 2024-03-19T02:04:01.689 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.688+0000 7f7ddae3a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f7dd406f6d0 con 0x7f7dd4152150 2024-03-19T02:04:01.689 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.688+0000 7f7dd8bd6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7dd4152150 0x7f7dd4158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:01.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.688+0000 7f7dd8bd6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7dd4152150 0x7f7dd4158a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:49266/0 (socket says 172.21.15.55:49266) 2024-03-19T02:04:01.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.688+0000 7f7dd8bd6640 1 -- 172.21.15.55:0/1241699491 learned_addr learned my addr 172.21.15.55:0/1241699491 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:01.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.692+0000 7f7dd3fff640 1 --2- 172.21.15.55:0/1241699491 >> v2:172.21.15.55:3300/0 conn(0x7f7dd4070970 0x7f7dd40c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:01.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.692+0000 7f7dd8bd6640 1 -- 172.21.15.55:0/1241699491 >> v2:172.21.15.55:3300/0 conn(0x7f7dd4070970 msgr2=0x7f7dd40c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:01.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.692+0000 7f7dd8bd6640 1 --2- 172.21.15.55:0/1241699491 >> v2:172.21.15.55:3300/0 conn(0x7f7dd4070970 0x7f7dd40c90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:01.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.692+0000 7f7dd8bd6640 1 -- 172.21.15.55:0/1241699491 >> v1:172.21.15.55:6789/0 conn(0x7f7dd40c9600 legacy=0x7f7dd40c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:01.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.692+0000 7f7dd8bd6640 1 -- 172.21.15.55:0/1241699491 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dd4070450 con 0x7f7dd4152150 2024-03-19T02:04:01.697 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.696+0000 7f7dd8bd6640 1 --2- 172.21.15.55:0/1241699491 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7dd4152150 0x7f7dd4158a00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8009870 tx=0x7f7dc801eb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=830e3b602a819bb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:01.703 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.700+0000 7f7dd37fe640 1 -- 172.21.15.55:0/1241699491 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc80b5070 con 0x7f7dd4152150 2024-03-19T02:04:01.703 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7dd37fe640 1 -- 172.21.15.55:0/1241699491 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dc80a87f0 con 0x7f7dd4152150 2024-03-19T02:04:01.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7dd37fe640 1 -- 172.21.15.55:0/1241699491 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc80a8bc0 con 0x7f7dd4152150 2024-03-19T02:04:01.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7ddae3a640 1 -- 172.21.15.55:0/1241699491 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7dd4152150 msgr2=0x7f7dd4158a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:01.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7ddae3a640 1 --2- 172.21.15.55:0/1241699491 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7dd4152150 0x7f7dd4158a00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8009870 tx=0x7f7dc801eb30 comp rx=0 tx=0).stop 2024-03-19T02:04:01.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7ddae3a640 1 -- 172.21.15.55:0/1241699491 shutdown_connections 2024-03-19T02:04:01.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7ddae3a640 1 --2- 172.21.15.55:0/1241699491 >> v2:172.21.15.55:3300/0 conn(0x7f7dd4070970 0x7f7dd40c90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:01.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7ddae3a640 1 --2- 172.21.15.55:0/1241699491 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7dd4152150 0x7f7dd4158a00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:01.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7ddae3a640 1 -- 172.21.15.55:0/1241699491 >> 172.21.15.55:0/1241699491 conn(0x7f7dd4067600 msgr2=0x7f7dd40714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:01.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7ddae3a640 1 -- 172.21.15.55:0/1241699491 shutdown_connections 2024-03-19T02:04:01.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7ddae3a640 1 -- 172.21.15.55:0/1241699491 wait complete. 2024-03-19T02:04:01.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7ddae3a640 1 Processor -- start 2024-03-19T02:04:01.705 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7ddae3a640 1 -- start start 2024-03-19T02:04:01.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7ddae3a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7dd4070970 0x7f7dd41ee7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:01.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7ddae3a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7dd40c9600 0x7f7dd41f1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:01.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7ddae3a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7dd4152150 0x7f7dd4145020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:01.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7ddae3a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd406fc50 con 0x7f7dd40c9600 2024-03-19T02:04:01.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7ddae3a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd406f970 con 0x7f7dd4070970 2024-03-19T02:04:01.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7ddae3a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f7dd406ff30 con 0x7f7dd4152150 2024-03-19T02:04:01.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7dd3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7dd40c9600 0x7f7dd41f1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:01.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7dd8bd6640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7dd4070970 0x7f7dd41ee7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:01.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7dd3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7dd40c9600 0x7f7dd41f1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:37804/0 (socket says 172.21.15.55:37804) 2024-03-19T02:04:01.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.704+0000 7f7dd3fff640 1 -- 172.21.15.55:0/3115208946 learned_addr learned my addr 172.21.15.55:0/3115208946 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:01.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.708+0000 7f7dd93d7640 1 --2- 172.21.15.55:0/3115208946 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7dd4152150 0x7f7dd4145020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:01.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.708+0000 7f7dd3fff640 1 -- 172.21.15.55:0/3115208946 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7dd4152150 msgr2=0x7f7dd4145020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:01.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.708+0000 7f7dd3fff640 1 --2- 172.21.15.55:0/3115208946 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7dd4152150 0x7f7dd4145020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:01.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.708+0000 7f7dd3fff640 1 -- 172.21.15.55:0/3115208946 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7dd4070970 msgr2=0x7f7dd41ee7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:01.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.708+0000 7f7dd3fff640 1 --2- 172.21.15.55:0/3115208946 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7dd4070970 0x7f7dd41ee7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:01.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.708+0000 7f7dd3fff640 1 -- 172.21.15.55:0/3115208946 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dd41e2510 con 0x7f7dd40c9600 2024-03-19T02:04:01.713 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.712+0000 7f7dd3fff640 1 --2- 172.21.15.55:0/3115208946 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7dd40c9600 0x7f7dd41f1b00 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7dc000c7f0 tx=0x7f7dc000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:01.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.716+0000 7f7dd17fa640 1 -- 172.21.15.55:0/3115208946 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc0015070 con 0x7f7dd40c9600 2024-03-19T02:04:01.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.716+0000 7f7dd17fa640 1 -- 172.21.15.55:0/3115208946 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dc00077c0 con 0x7f7dd40c9600 2024-03-19T02:04:01.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.716+0000 7f7dd17fa640 1 -- 172.21.15.55:0/3115208946 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc0007bf0 con 0x7f7dd40c9600 2024-03-19T02:04:01.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.716+0000 7f7ddae3a640 1 -- 172.21.15.55:0/3115208946 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dd41e27f0 con 0x7f7dd40c9600 2024-03-19T02:04:01.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.716+0000 7f7ddae3a640 1 -- 172.21.15.55:0/3115208946 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dd41f1300 con 0x7f7dd40c9600 2024-03-19T02:04:01.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:01.716+0000 7f7ddae3a640 1 -- 172.21.15.55:0/3115208946 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dd4068a40 con 0x7f7dd40c9600 2024-03-19T02:04:02.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:02.040+0000 7f7dd17fa640 1 -- 172.21.15.55:0/3115208946 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f7dc0018070 con 0x7f7dd40c9600 2024-03-19T02:04:02.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:02.040+0000 7f7dd17fa640 1 --2- 172.21.15.55:0/3115208946 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7db8067660 0x7f7db8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:02.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:02.044+0000 7f7dd8bd6640 1 --2- 172.21.15.55:0/3115208946 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7db8067660 0x7f7db8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:02.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:02.044+0000 7f7dd8bd6640 1 --2- 172.21.15.55:0/3115208946 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7db8067660 0x7f7db8069b20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7dc80b2000 tx=0x7f7dc80aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:02.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:02.084+0000 7f7dd17fa640 1 -- 172.21.15.55:0/3115208946 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f7dc0054480 con 0x7f7dd40c9600 2024-03-19T02:04:02.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:02.104+0000 7f7dd17fa640 1 -- 172.21.15.55:0/3115208946 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f7dc00563e0 con 0x7f7dd40c9600 2024-03-19T02:04:02.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:02.316+0000 7f7ddae3a640 1 -- 172.21.15.55:0/3115208946 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7dd40c8690 con 0x7f7dd40c9600 2024-03-19T02:04:02.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:02.344+0000 7f7dd17fa640 1 -- 172.21.15.55:0/3115208946 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f7dc005a480 con 0x7f7dd40c9600 2024-03-19T02:04:02.344 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:02.344 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:02.345 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:02.347 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:02.344+0000 7f7ddae3a640 1 -- 172.21.15.55:0/3115208946 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7db8067660 msgr2=0x7f7db8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:02.347 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:02.344+0000 7f7ddae3a640 1 --2- 172.21.15.55:0/3115208946 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7db8067660 0x7f7db8069b20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7dc80b2000 tx=0x7f7dc80aa070 comp rx=0 tx=0).stop 2024-03-19T02:04:02.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:02.344+0000 7f7ddae3a640 1 -- 172.21.15.55:0/3115208946 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7dd40c9600 msgr2=0x7f7dd41f1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:02.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:02.344+0000 7f7ddae3a640 1 --2- 172.21.15.55:0/3115208946 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7dd40c9600 0x7f7dd41f1b00 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7dc000c7f0 tx=0x7f7dc000ccb0 comp rx=0 tx=0).stop 2024-03-19T02:04:02.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:02.344+0000 7f7ddae3a640 1 -- 172.21.15.55:0/3115208946 shutdown_connections 2024-03-19T02:04:02.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:02.344+0000 7f7ddae3a640 1 --2- 172.21.15.55:0/3115208946 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7db8067660 0x7f7db8069b20 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:02.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:02.344+0000 7f7ddae3a640 1 --2- 172.21.15.55:0/3115208946 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7dd4152150 0x7f7dd4145020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:02.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:02.344+0000 7f7ddae3a640 1 --2- 172.21.15.55:0/3115208946 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7dd40c9600 0x7f7dd41f1b00 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:02.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:02.344+0000 7f7ddae3a640 1 --2- 172.21.15.55:0/3115208946 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7dd4070970 0x7f7dd41ee7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:02.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:02.344+0000 7f7ddae3a640 1 -- 172.21.15.55:0/3115208946 >> 172.21.15.55:0/3115208946 conn(0x7f7dd4067600 msgr2=0x7f7dd4150650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:02.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:02.344+0000 7f7ddae3a640 1 -- 172.21.15.55:0/3115208946 shutdown_connections 2024-03-19T02:04:02.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:02.344+0000 7f7ddae3a640 1 -- 172.21.15.55:0/3115208946 wait complete. 2024-03-19T02:04:02.971 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:02.973 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:03.975 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:04.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.084+0000 7f7977181640 1 Processor -- start 2024-03-19T02:04:04.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.084+0000 7f7977181640 1 -- start start 2024-03-19T02:04:04.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.084+0000 7f7977181640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f797014d6a0 0x7f79700c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:04.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.084+0000 7f7977181640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f79701577f0 0x7f79700c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:04.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.084+0000 7f7977181640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79700c6e80 con 0x7f79700c7900 2024-03-19T02:04:04.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.084+0000 7f7977181640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f79700c4ca0 con 0x7f797014d6a0 2024-03-19T02:04:04.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.084+0000 7f7977181640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f79700c6300 con 0x7f79701577f0 2024-03-19T02:04:04.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.084+0000 7f7974f1d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f797014d6a0 0x7f79700c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:04.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.084+0000 7f7974f1d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f797014d6a0 0x7f79700c73c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:37820/0 (socket says 172.21.15.55:37820) 2024-03-19T02:04:04.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.084+0000 7f7974f1d640 1 -- 172.21.15.55:0/1303092272 learned_addr learned my addr 172.21.15.55:0/1303092272 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:04.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.088+0000 7f797571e640 1 --2- 172.21.15.55:0/1303092272 >> v2:172.21.15.84:3300/0 conn(0x7f79701577f0 0x7f79700c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:04.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.088+0000 7f7974f1d640 1 -- 172.21.15.55:0/1303092272 >> v2:172.21.15.84:3300/0 conn(0x7f79701577f0 msgr2=0x7f79700c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:04.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.088+0000 7f7974f1d640 1 --2- 172.21.15.55:0/1303092272 >> v2:172.21.15.84:3300/0 conn(0x7f79701577f0 0x7f79700c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:04.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.088+0000 7f7974f1d640 1 -- 172.21.15.55:0/1303092272 >> v1:172.21.15.55:6789/0 conn(0x7f79700c7900 legacy=0x7f79701570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:04.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.088+0000 7f7974f1d640 1 -- 172.21.15.55:0/1303092272 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79700c65e0 con 0x7f797014d6a0 2024-03-19T02:04:04.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.092+0000 7f7974f1d640 1 --2- 172.21.15.55:0/1303092272 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f797014d6a0 0x7f79700c73c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f7970068000 tx=0x7f796401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b399fc77b81805 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:04.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.100+0000 7f796f7fe640 1 -- 172.21.15.55:0/1303092272 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79640a6040 con 0x7f797014d6a0 2024-03-19T02:04:04.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.100+0000 7f796f7fe640 1 -- 172.21.15.55:0/1303092272 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79640af070 con 0x7f797014d6a0 2024-03-19T02:04:04.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.100+0000 7f796f7fe640 1 -- 172.21.15.55:0/1303092272 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79640aa860 con 0x7f797014d6a0 2024-03-19T02:04:04.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.100+0000 7f7977181640 1 -- 172.21.15.55:0/1303092272 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f797014d6a0 msgr2=0x7f79700c73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:04.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.100+0000 7f7977181640 1 --2- 172.21.15.55:0/1303092272 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f797014d6a0 0x7f79700c73c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f7970068000 tx=0x7f796401c4e0 comp rx=0 tx=0).stop 2024-03-19T02:04:04.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.100+0000 7f7977181640 1 -- 172.21.15.55:0/1303092272 shutdown_connections 2024-03-19T02:04:04.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.100+0000 7f7977181640 1 --2- 172.21.15.55:0/1303092272 >> v2:172.21.15.84:3300/0 conn(0x7f79701577f0 0x7f79700c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:04.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.100+0000 7f7977181640 1 --2- 172.21.15.55:0/1303092272 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f797014d6a0 0x7f79700c73c0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:04.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.100+0000 7f7977181640 1 -- 172.21.15.55:0/1303092272 >> 172.21.15.55:0/1303092272 conn(0x7f797006c3b0 msgr2=0x7f797006c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:04.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.100+0000 7f7977181640 1 -- 172.21.15.55:0/1303092272 shutdown_connections 2024-03-19T02:04:04.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.100+0000 7f7977181640 1 -- 172.21.15.55:0/1303092272 wait complete. 2024-03-19T02:04:04.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.100+0000 7f7977181640 1 Processor -- start 2024-03-19T02:04:04.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.100+0000 7f7977181640 1 -- start start 2024-03-19T02:04:04.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.104+0000 7f7977181640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f79700c7900 0x7f79701f4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:04.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.104+0000 7f7977181640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f797014d6a0 0x7f79700cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:04.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.104+0000 7f7977181640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79701577f0 0x7f7970142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:04.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.104+0000 7f7977181640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f797014dc00 con 0x7f797014d6a0 2024-03-19T02:04:04.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.104+0000 7f7977181640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f797014da80 con 0x7f79701577f0 2024-03-19T02:04:04.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.104+0000 7f7977181640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f79700c4810 con 0x7f79700c7900 2024-03-19T02:04:04.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.104+0000 7f796ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f797014d6a0 0x7f79700cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:04.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.104+0000 7f796ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f797014d6a0 0x7f79700cbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:37826/0 (socket says 172.21.15.55:37826) 2024-03-19T02:04:04.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.104+0000 7f796ffff640 1 -- 172.21.15.55:0/251720162 learned_addr learned my addr 172.21.15.55:0/251720162 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:04.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.104+0000 7f7974f1d640 1 --2- 172.21.15.55:0/251720162 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f79700c7900 0x7f79701f4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:04.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.104+0000 7f797571e640 1 --2- 172.21.15.55:0/251720162 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79701577f0 0x7f7970142250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:04.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.108+0000 7f796ffff640 1 -- 172.21.15.55:0/251720162 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f79700c7900 msgr2=0x7f79701f4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:04.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.108+0000 7f796ffff640 1 --2- 172.21.15.55:0/251720162 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f79700c7900 0x7f79701f4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:04.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.108+0000 7f796ffff640 1 -- 172.21.15.55:0/251720162 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79701577f0 msgr2=0x7f7970142250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:04.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.108+0000 7f796ffff640 1 --2- 172.21.15.55:0/251720162 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79701577f0 0x7f7970142250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:04.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.108+0000 7f796ffff640 1 -- 172.21.15.55:0/251720162 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79701f5320 con 0x7f797014d6a0 2024-03-19T02:04:04.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.108+0000 7f796ffff640 1 --2- 172.21.15.55:0/251720162 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f797014d6a0 0x7f79700cbba0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f795c00c480 tx=0x7f795c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:04.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.116+0000 7f796d7fa640 1 -- 172.21.15.55:0/251720162 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c012830 con 0x7f797014d6a0 2024-03-19T02:04:04.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.116+0000 7f796d7fa640 1 -- 172.21.15.55:0/251720162 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f795c017070 con 0x7f797014d6a0 2024-03-19T02:04:04.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.116+0000 7f796d7fa640 1 -- 172.21.15.55:0/251720162 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c012c30 con 0x7f797014d6a0 2024-03-19T02:04:04.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.116+0000 7f7977181640 1 -- 172.21.15.55:0/251720162 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7970156520 con 0x7f797014d6a0 2024-03-19T02:04:04.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.116+0000 7f7977181640 1 -- 172.21.15.55:0/251720162 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7970156ae0 con 0x7f797014d6a0 2024-03-19T02:04:04.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.116+0000 7f7977181640 1 -- 172.21.15.55:0/251720162 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7970156ce0 con 0x7f797014d6a0 2024-03-19T02:04:04.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.408+0000 7f796d7fa640 1 -- 172.21.15.55:0/251720162 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f795c01c460 con 0x7f797014d6a0 2024-03-19T02:04:04.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.408+0000 7f796d7fa640 1 --2- 172.21.15.55:0/251720162 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7954067690 0x7f7954069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:04.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.408+0000 7f7974f1d640 1 --2- 172.21.15.55:0/251720162 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7954067690 0x7f7954069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:04.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.412+0000 7f7974f1d640 1 --2- 172.21.15.55:0/251720162 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7954067690 0x7f7954069b50 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f79640ae000 tx=0x7f79640a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:04.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.432+0000 7f796d7fa640 1 -- 172.21.15.55:0/251720162 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f795c054740 con 0x7f797014d6a0 2024-03-19T02:04:04.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.432+0000 7f796d7fa640 1 -- 172.21.15.55:0/251720162 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f7970156ce0 con 0x7f797014d6a0 2024-03-19T02:04:04.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.664+0000 7f7977181640 1 -- 172.21.15.55:0/251720162 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7970068a40 con 0x7f797014d6a0 2024-03-19T02:04:04.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.684+0000 7f796d7fa640 1 -- 172.21.15.55:0/251720162 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f795c0566a0 con 0x7f797014d6a0 2024-03-19T02:04:04.687 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:04.687 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:04.687 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:04.689 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.688+0000 7f7977181640 1 -- 172.21.15.55:0/251720162 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7954067690 msgr2=0x7f7954069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:04.689 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.688+0000 7f7977181640 1 --2- 172.21.15.55:0/251720162 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7954067690 0x7f7954069b50 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f79640ae000 tx=0x7f79640a7000 comp rx=0 tx=0).stop 2024-03-19T02:04:04.689 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.688+0000 7f7977181640 1 -- 172.21.15.55:0/251720162 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f797014d6a0 msgr2=0x7f79700cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:04.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.688+0000 7f7977181640 1 --2- 172.21.15.55:0/251720162 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f797014d6a0 0x7f79700cbba0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f795c00c480 tx=0x7f795c00c940 comp rx=0 tx=0).stop 2024-03-19T02:04:04.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.688+0000 7f7977181640 1 -- 172.21.15.55:0/251720162 shutdown_connections 2024-03-19T02:04:04.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.688+0000 7f7977181640 1 --2- 172.21.15.55:0/251720162 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7954067690 0x7f7954069b50 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:04.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.688+0000 7f7977181640 1 --2- 172.21.15.55:0/251720162 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79701577f0 0x7f7970142250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:04.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.688+0000 7f7977181640 1 --2- 172.21.15.55:0/251720162 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f797014d6a0 0x7f79700cbba0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:04.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.688+0000 7f7977181640 1 --2- 172.21.15.55:0/251720162 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f79700c7900 0x7f79701f4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:04.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.688+0000 7f7977181640 1 -- 172.21.15.55:0/251720162 >> 172.21.15.55:0/251720162 conn(0x7f797006c3b0 msgr2=0x7f797014c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:04.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.688+0000 7f7977181640 1 -- 172.21.15.55:0/251720162 shutdown_connections 2024-03-19T02:04:04.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:04.688+0000 7f7977181640 1 -- 172.21.15.55:0/251720162 wait complete. 2024-03-19T02:04:05.310 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:05.310 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:06.312 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:06.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.424+0000 7f3025282640 1 Processor -- start 2024-03-19T02:04:06.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.424+0000 7f3025282640 1 -- start start 2024-03-19T02:04:06.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.424+0000 7f3025282640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f3020152120 0x7f30200c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:06.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.424+0000 7f3025282640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f302006fff0 con 0x7f30200c9610 2024-03-19T02:04:06.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.424+0000 7f3025282640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30200c39c0 con 0x7f3020070780 2024-03-19T02:04:06.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.424+0000 7f3025282640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f3020152500 con 0x7f3020152120 2024-03-19T02:04:06.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.424+0000 7f301e7fc640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f30200c9610 0x7f3020157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:47670/0 (socket says 172.21.15.55:47670) 2024-03-19T02:04:06.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.424+0000 7f301e7fc640 1 -- 172.21.15.55:0/2818255758 learned_addr learned my addr 172.21.15.55:0/2818255758 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:04:06.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.424+0000 7f301effd640 1 --2- 172.21.15.55:0/2818255758 >> v2:172.21.15.84:3300/0 conn(0x7f3020152120 0x7f30200c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:06.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.428+0000 7f301dffb640 1 -- 172.21.15.55:0/2818255758 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3122463238 0 0) 0x7f302006fff0 con 0x7f30200c9610 2024-03-19T02:04:06.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.428+0000 7f301dffb640 1 -- 172.21.15.55:0/2818255758 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3008003530 con 0x7f30200c9610 2024-03-19T02:04:06.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.432+0000 7f301effd640 1 -- 172.21.15.55:0/2818255758 >> v1:172.21.15.84:6789/0 conn(0x7f3020070780 legacy=0x7f30201586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:06.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.432+0000 7f301effd640 1 -- 172.21.15.55:0/2818255758 >> v1:172.21.15.55:6789/0 conn(0x7f30200c9610 legacy=0x7f3020157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:06.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.432+0000 7f301effd640 1 -- 172.21.15.55:0/2818255758 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30200702b0 con 0x7f3020152120 2024-03-19T02:04:06.438 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.436+0000 7f301effd640 1 --2- 172.21.15.55:0/2818255758 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3020152120 0x7f30200c90d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f30140097c0 tx=0x7f301401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de4cc016887ec02a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:06.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.444+0000 7f301dffb640 1 -- 172.21.15.55:0/2818255758 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30140b0070 con 0x7f3020152120 2024-03-19T02:04:06.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.444+0000 7f301dffb640 1 -- 172.21.15.55:0/2818255758 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30140ab600 con 0x7f3020152120 2024-03-19T02:04:06.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.444+0000 7f301dffb640 1 -- 172.21.15.55:0/2818255758 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30140ab9d0 con 0x7f3020152120 2024-03-19T02:04:06.445 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.444+0000 7f3025282640 1 -- 172.21.15.55:0/2818255758 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3020152120 msgr2=0x7f30200c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:06.445 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.444+0000 7f3025282640 1 --2- 172.21.15.55:0/2818255758 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3020152120 0x7f30200c90d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f30140097c0 tx=0x7f301401c4a0 comp rx=0 tx=0).stop 2024-03-19T02:04:06.445 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.444+0000 7f3025282640 1 -- 172.21.15.55:0/2818255758 shutdown_connections 2024-03-19T02:04:06.445 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.444+0000 7f3025282640 1 --2- 172.21.15.55:0/2818255758 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3020152120 0x7f30200c90d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:06.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.444+0000 7f3025282640 1 -- 172.21.15.55:0/2818255758 >> 172.21.15.55:0/2818255758 conn(0x7f3020067600 msgr2=0x7f3020071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:06.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.444+0000 7f3025282640 1 -- 172.21.15.55:0/2818255758 shutdown_connections 2024-03-19T02:04:06.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.444+0000 7f3025282640 1 -- 172.21.15.55:0/2818255758 wait complete. 2024-03-19T02:04:06.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.444+0000 7f3025282640 1 Processor -- start 2024-03-19T02:04:06.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.444+0000 7f3025282640 1 -- start start 2024-03-19T02:04:06.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.444+0000 7f3025282640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3020070780 0x7f3020140e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:06.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.444+0000 7f3025282640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f30200c9610 0x7f30200b0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:06.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.444+0000 7f3025282640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3020152120 0x7f30201560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:06.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.444+0000 7f3025282640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3020158df0 con 0x7f3020070780 2024-03-19T02:04:06.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.444+0000 7f3025282640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f30200c3530 con 0x7f30200c9610 2024-03-19T02:04:06.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.444+0000 7f3025282640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f30200c86d0 con 0x7f3020152120 2024-03-19T02:04:06.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.448+0000 7f301effd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3020070780 0x7f3020140e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:06.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.448+0000 7f301effd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3020070780 0x7f3020140e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:37846/0 (socket says 172.21.15.55:37846) 2024-03-19T02:04:06.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.448+0000 7f301effd640 1 -- 172.21.15.55:0/1112322790 learned_addr learned my addr 172.21.15.55:0/1112322790 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:06.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.448+0000 7f301f7fe640 1 --2- 172.21.15.55:0/1112322790 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3020152120 0x7f30201560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:06.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.448+0000 7f301e7fc640 1 --2- 172.21.15.55:0/1112322790 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f30200c9610 0x7f30200b0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:06.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.452+0000 7f301effd640 1 -- 172.21.15.55:0/1112322790 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3020152120 msgr2=0x7f30201560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:06.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.452+0000 7f301effd640 1 --2- 172.21.15.55:0/1112322790 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3020152120 0x7f30201560a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:06.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.452+0000 7f301effd640 1 -- 172.21.15.55:0/1112322790 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f30200c9610 msgr2=0x7f30200b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:06.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.452+0000 7f301effd640 1 --2- 172.21.15.55:0/1112322790 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f30200c9610 0x7f30200b0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:06.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.452+0000 7f301effd640 1 -- 172.21.15.55:0/1112322790 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3020141590 con 0x7f3020070780 2024-03-19T02:04:06.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.452+0000 7f301effd640 1 --2- 172.21.15.55:0/1112322790 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3020070780 0x7f3020140e90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f30140190b0 tx=0x7f301401ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:06.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.456+0000 7f30077fe640 1 -- 172.21.15.55:0/1112322790 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30140b0050 con 0x7f3020070780 2024-03-19T02:04:06.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.456+0000 7f30077fe640 1 -- 172.21.15.55:0/1112322790 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30140bc070 con 0x7f3020070780 2024-03-19T02:04:06.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.456+0000 7f3025282640 1 -- 172.21.15.55:0/1112322790 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30200c7030 con 0x7f3020070780 2024-03-19T02:04:06.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.456+0000 7f30077fe640 1 -- 172.21.15.55:0/1112322790 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30140b5830 con 0x7f3020070780 2024-03-19T02:04:06.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.460+0000 7f3025282640 1 -- 172.21.15.55:0/1112322790 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30200c60f0 con 0x7f3020070780 2024-03-19T02:04:06.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.460+0000 7f3025282640 1 -- 172.21.15.55:0/1112322790 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3020068a40 con 0x7f3020070780 2024-03-19T02:04:06.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.772+0000 7f30077fe640 1 -- 172.21.15.55:0/1112322790 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f30140b59d0 con 0x7f3020070780 2024-03-19T02:04:06.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.772+0000 7f30077fe640 1 --2- 172.21.15.55:0/1112322790 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f300c0676b0 0x7f300c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:06.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.772+0000 7f301e7fc640 1 --2- 172.21.15.55:0/1112322790 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f300c0676b0 0x7f300c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:06.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.772+0000 7f301e7fc640 1 --2- 172.21.15.55:0/1112322790 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f300c0676b0 0x7f300c069b70 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3020157fa0 tx=0x7f3010007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:06.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.816+0000 7f30077fe640 1 -- 172.21.15.55:0/1112322790 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f30140ef070 con 0x7f3020070780 2024-03-19T02:04:06.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:06.848+0000 7f30077fe640 1 -- 172.21.15.55:0/1112322790 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f3014016090 con 0x7f3020070780 2024-03-19T02:04:07.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:07.096+0000 7f3025282640 1 -- 172.21.15.55:0/1112322790 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30200c84b0 con 0x7f3020070780 2024-03-19T02:04:07.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:07.204+0000 7f30077fe640 1 -- 172.21.15.55:0/1112322790 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f30140f0fd0 con 0x7f3020070780 2024-03-19T02:04:07.207 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:07.207 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:07.207 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:07.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:07.208+0000 7f3025282640 1 -- 172.21.15.55:0/1112322790 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f300c0676b0 msgr2=0x7f300c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:07.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:07.208+0000 7f3025282640 1 --2- 172.21.15.55:0/1112322790 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f300c0676b0 0x7f300c069b70 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3020157fa0 tx=0x7f3010007d40 comp rx=0 tx=0).stop 2024-03-19T02:04:07.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:07.208+0000 7f3025282640 1 -- 172.21.15.55:0/1112322790 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3020070780 msgr2=0x7f3020140e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:07.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:07.208+0000 7f3025282640 1 --2- 172.21.15.55:0/1112322790 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3020070780 0x7f3020140e90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f30140190b0 tx=0x7f301401ace0 comp rx=0 tx=0).stop 2024-03-19T02:04:07.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:07.208+0000 7f3025282640 1 -- 172.21.15.55:0/1112322790 shutdown_connections 2024-03-19T02:04:07.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:07.208+0000 7f3025282640 1 --2- 172.21.15.55:0/1112322790 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f300c0676b0 0x7f300c069b70 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:07.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:07.208+0000 7f3025282640 1 --2- 172.21.15.55:0/1112322790 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3020152120 0x7f30201560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:07.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:07.208+0000 7f3025282640 1 --2- 172.21.15.55:0/1112322790 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f30200c9610 0x7f30200b0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:07.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:07.208+0000 7f3025282640 1 --2- 172.21.15.55:0/1112322790 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3020070780 0x7f3020140e90 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:07.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:07.208+0000 7f3025282640 1 -- 172.21.15.55:0/1112322790 >> 172.21.15.55:0/1112322790 conn(0x7f3020067600 msgr2=0x7f3020151370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:07.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:07.208+0000 7f3025282640 1 -- 172.21.15.55:0/1112322790 shutdown_connections 2024-03-19T02:04:07.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:07.208+0000 7f3025282640 1 -- 172.21.15.55:0/1112322790 wait complete. 2024-03-19T02:04:07.836 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:07.837 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:08.838 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:08.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.952+0000 7f901bdc6640 1 Processor -- start 2024-03-19T02:04:08.956 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.952+0000 7f901bdc6640 1 -- start start 2024-03-19T02:04:08.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.956+0000 7f901bdc6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f90140c96b0 0x7f901415aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:08.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.956+0000 7f901bdc6640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f90140c9a90 0x7f90140c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:08.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.956+0000 7f901bdc6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9014073350 con 0x7f901415e4d0 2024-03-19T02:04:08.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.956+0000 7f901bdc6640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f90140c7900 con 0x7f90140c9a90 2024-03-19T02:04:08.957 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.956+0000 7f901bdc6640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f901415e040 con 0x7f90140c96b0 2024-03-19T02:04:08.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.956+0000 7f9019361640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f90140c9a90 0x7f90140c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:08.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.956+0000 7f9019b62640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f90140c96b0 0x7f901415aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:08.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.956+0000 7f901a363640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f901415e4d0 0x7f9014070c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:47746/0 (socket says 172.21.15.55:47746) 2024-03-19T02:04:08.958 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.956+0000 7f901a363640 1 -- 172.21.15.55:0/1767049683 learned_addr learned my addr 172.21.15.55:0/1767049683 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:04:08.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.960+0000 7f9019b62640 1 -- 172.21.15.55:0/1767049683 >> v2:172.21.15.84:3300/0 conn(0x7f90140c9a90 msgr2=0x7f90140c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:08.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.960+0000 7f9019b62640 1 --2- 172.21.15.55:0/1767049683 >> v2:172.21.15.84:3300/0 conn(0x7f90140c9a90 0x7f90140c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:08.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.960+0000 7f9019b62640 1 -- 172.21.15.55:0/1767049683 >> v1:172.21.15.84:6789/0 conn(0x7f901415e4d0 legacy=0x7f9014070c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:08.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.960+0000 7f9019b62640 1 -- 172.21.15.55:0/1767049683 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9014073630 con 0x7f90140c96b0 2024-03-19T02:04:08.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.964+0000 7f9019b62640 1 --2- 172.21.15.55:0/1767049683 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f90140c96b0 0x7f901415aea0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f90100097c0 tx=0x7f901001c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaf85254fded0eba server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:08.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.976+0000 7f9018b60640 1 -- 172.21.15.55:0/1767049683 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90100b2070 con 0x7f90140c96b0 2024-03-19T02:04:08.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.976+0000 7f9018b60640 1 -- 172.21.15.55:0/1767049683 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90100ad640 con 0x7f90140c96b0 2024-03-19T02:04:08.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.976+0000 7f9018b60640 1 -- 172.21.15.55:0/1767049683 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90100ada10 con 0x7f90140c96b0 2024-03-19T02:04:08.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.976+0000 7f901bdc6640 1 -- 172.21.15.55:0/1767049683 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f90140c96b0 msgr2=0x7f901415aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:08.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.976+0000 7f901bdc6640 1 --2- 172.21.15.55:0/1767049683 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f90140c96b0 0x7f901415aea0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f90100097c0 tx=0x7f901001c4a0 comp rx=0 tx=0).stop 2024-03-19T02:04:08.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.976+0000 7f901bdc6640 1 -- 172.21.15.55:0/1767049683 shutdown_connections 2024-03-19T02:04:08.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.976+0000 7f901bdc6640 1 --2- 172.21.15.55:0/1767049683 >> v2:172.21.15.84:3300/0 conn(0x7f90140c9a90 0x7f90140c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:08.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.976+0000 7f901bdc6640 1 --2- 172.21.15.55:0/1767049683 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f90140c96b0 0x7f901415aea0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:08.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.976+0000 7f901bdc6640 1 -- 172.21.15.55:0/1767049683 >> 172.21.15.55:0/1767049683 conn(0x7f90140cc9c0 msgr2=0x7f90140ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:08.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.976+0000 7f901bdc6640 1 -- 172.21.15.55:0/1767049683 shutdown_connections 2024-03-19T02:04:08.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.976+0000 7f901bdc6640 1 -- 172.21.15.55:0/1767049683 wait complete. 2024-03-19T02:04:08.979 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.976+0000 7f901bdc6640 1 Processor -- start 2024-03-19T02:04:08.979 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.976+0000 7f901bdc6640 1 -- start start 2024-03-19T02:04:08.980 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.980+0000 7f901bdc6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90140c96b0 0x7f901415e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:08.980 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.980+0000 7f901bdc6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f90140c9a90 0x7f90141f7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:08.980 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.980+0000 7f901bdc6640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f901415e4d0 0x7f901406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:08.980 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.980+0000 7f901bdc6640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f90141548c0 con 0x7f90140c96b0 2024-03-19T02:04:08.980 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.980+0000 7f901bdc6640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f9014154740 con 0x7f901415e4d0 2024-03-19T02:04:08.981 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.980+0000 7f901bdc6640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f9014166bf0 con 0x7f90140c9a90 2024-03-19T02:04:08.981 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.980+0000 7f901a363640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f901415e4d0 0x7f901406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:08.981 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.980+0000 7f9019b62640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90140c96b0 0x7f901415e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:08.981 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.980+0000 7f9019361640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f90140c9a90 0x7f90141f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:08.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.980+0000 7f9019b62640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90140c96b0 0x7f901415e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56864/0 (socket says 172.21.15.55:56864) 2024-03-19T02:04:08.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.980+0000 7f9019b62640 1 -- 172.21.15.55:0/1123602183 learned_addr learned my addr 172.21.15.55:0/1123602183 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:08.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.984+0000 7f9019b62640 1 -- 172.21.15.55:0/1123602183 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f90140c9a90 msgr2=0x7f90141f7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:08.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.984+0000 7f9019b62640 1 --2- 172.21.15.55:0/1123602183 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f90140c9a90 0x7f90141f7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:08.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.984+0000 7f9019b62640 1 -- 172.21.15.55:0/1123602183 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f901415e4d0 msgr2=0x7f901406c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:08.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.984+0000 7f9019b62640 1 --2- 172.21.15.55:0/1123602183 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f901415e4d0 0x7f901406c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:08.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.984+0000 7f9019b62640 1 -- 172.21.15.55:0/1123602183 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f901415f0c0 con 0x7f90140c96b0 2024-03-19T02:04:08.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.988+0000 7f9019b62640 1 --2- 172.21.15.55:0/1123602183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90140c96b0 0x7f901415e9a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f901001a0d0 tx=0x7f9010019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:08.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.992+0000 7f900a7fc640 1 -- 172.21.15.55:0/1123602183 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90100b2040 con 0x7f90140c96b0 2024-03-19T02:04:08.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.992+0000 7f900a7fc640 1 -- 172.21.15.55:0/1123602183 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90100bc070 con 0x7f90140c96b0 2024-03-19T02:04:08.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.992+0000 7f900a7fc640 1 -- 172.21.15.55:0/1123602183 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90100b6630 con 0x7f90140c96b0 2024-03-19T02:04:08.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.992+0000 7f901bdc6640 1 -- 172.21.15.55:0/1123602183 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f901415f3a0 con 0x7f90140c96b0 2024-03-19T02:04:08.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.992+0000 7f901bdc6640 1 -- 172.21.15.55:0/1123602183 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90141f3800 con 0x7f90140c96b0 2024-03-19T02:04:08.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:08.992+0000 7f901bdc6640 1 -- 172.21.15.55:0/1123602183 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9014068a40 con 0x7f90140c96b0 2024-03-19T02:04:09.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:09.272+0000 7f900a7fc640 1 -- 172.21.15.55:0/1123602183 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f90100b67d0 con 0x7f90140c96b0 2024-03-19T02:04:09.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:09.272+0000 7f900a7fc640 1 --2- 172.21.15.55:0/1123602183 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8ff8067660 0x7f8ff8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:09.273 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:09.272+0000 7f9019361640 1 --2- 172.21.15.55:0/1123602183 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8ff8067660 0x7f8ff8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:09.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:09.272+0000 7f9019361640 1 --2- 172.21.15.55:0/1123602183 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8ff8067660 0x7f8ff8069b20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f90141f0e00 tx=0x7f9004009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:09.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:09.296+0000 7f900a7fc640 1 -- 172.21.15.55:0/1123602183 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f90100eeaa0 con 0x7f90140c96b0 2024-03-19T02:04:09.299 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:09.300+0000 7f900a7fc640 1 -- 172.21.15.55:0/1123602183 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f9010016090 con 0x7f90140c96b0 2024-03-19T02:04:09.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:09.556+0000 7f901bdc6640 1 -- 172.21.15.55:0/1123602183 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9014072db0 con 0x7f90140c96b0 2024-03-19T02:04:09.597 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:09.596+0000 7f900a7fc640 1 -- 172.21.15.55:0/1123602183 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f90100f0a00 con 0x7f90140c96b0 2024-03-19T02:04:09.598 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:09.598 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:09.598 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:09.601 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:09.600+0000 7f901bdc6640 1 -- 172.21.15.55:0/1123602183 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8ff8067660 msgr2=0x7f8ff8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:09.601 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:09.600+0000 7f901bdc6640 1 --2- 172.21.15.55:0/1123602183 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8ff8067660 0x7f8ff8069b20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f90141f0e00 tx=0x7f9004009190 comp rx=0 tx=0).stop 2024-03-19T02:04:09.601 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:09.600+0000 7f901bdc6640 1 -- 172.21.15.55:0/1123602183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90140c96b0 msgr2=0x7f901415e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:09.601 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:09.600+0000 7f901bdc6640 1 --2- 172.21.15.55:0/1123602183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90140c96b0 0x7f901415e9a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f901001a0d0 tx=0x7f9010019240 comp rx=0 tx=0).stop 2024-03-19T02:04:09.601 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:09.600+0000 7f901bdc6640 1 -- 172.21.15.55:0/1123602183 shutdown_connections 2024-03-19T02:04:09.601 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:09.600+0000 7f901bdc6640 1 --2- 172.21.15.55:0/1123602183 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8ff8067660 0x7f8ff8069b20 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:09.601 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:09.600+0000 7f901bdc6640 1 --2- 172.21.15.55:0/1123602183 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f901415e4d0 0x7f901406c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:09.601 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:09.600+0000 7f901bdc6640 1 --2- 172.21.15.55:0/1123602183 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f90140c9a90 0x7f90141f7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:09.601 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:09.600+0000 7f901bdc6640 1 --2- 172.21.15.55:0/1123602183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f90140c96b0 0x7f901415e9a0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:09.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:09.600+0000 7f901bdc6640 1 -- 172.21.15.55:0/1123602183 >> 172.21.15.55:0/1123602183 conn(0x7f90140cc9c0 msgr2=0x7f901415cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:09.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:09.600+0000 7f901bdc6640 1 -- 172.21.15.55:0/1123602183 shutdown_connections 2024-03-19T02:04:09.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:09.600+0000 7f901bdc6640 1 -- 172.21.15.55:0/1123602183 wait complete. 2024-03-19T02:04:10.222 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:10.223 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:10.933 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:04:10.936 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:04:11.224 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:11.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.336+0000 7f8a57f35640 1 Processor -- start 2024-03-19T02:04:11.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.336+0000 7f8a57f35640 1 -- start start 2024-03-19T02:04:11.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.340+0000 7f8a57f35640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f8a500c96b0 0x7f8a50070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:11.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.340+0000 7f8a57f35640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8a500c9a90 0x7f8a500c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:11.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.340+0000 7f8a57f35640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a5015e3a0 con 0x7f8a501540c0 2024-03-19T02:04:11.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.340+0000 7f8a57f35640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f8a5015e680 con 0x7f8a500c96b0 2024-03-19T02:04:11.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.340+0000 7f8a57f35640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f8a5015a330 con 0x7f8a500c9a90 2024-03-19T02:04:11.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.340+0000 7f8a55cd1640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f8a500c96b0 0x7f8a50070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:11.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.340+0000 7f8a554d0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8a500c9a90 0x7f8a500c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:11.342 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.340+0000 7f8a564d2640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f8a501540c0 0x7f8a5015d810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:35840/0 (socket says 172.21.15.55:35840) 2024-03-19T02:04:11.342 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.340+0000 7f8a564d2640 1 -- 172.21.15.55:0/385429398 learned_addr learned my addr 172.21.15.55:0/385429398 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:04:11.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.344+0000 7f8a554d0640 1 -- 172.21.15.55:0/385429398 >> v2:172.21.15.55:3300/0 conn(0x7f8a500c96b0 msgr2=0x7f8a50070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:11.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.344+0000 7f8a554d0640 1 --2- 172.21.15.55:0/385429398 >> v2:172.21.15.55:3300/0 conn(0x7f8a500c96b0 0x7f8a50070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:11.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.344+0000 7f8a554d0640 1 -- 172.21.15.55:0/385429398 >> v1:172.21.15.55:6789/0 conn(0x7f8a501540c0 legacy=0x7f8a5015d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:11.346 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.344+0000 7f8a554d0640 1 -- 172.21.15.55:0/385429398 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a5015a610 con 0x7f8a500c9a90 2024-03-19T02:04:11.350 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.348+0000 7f8a554d0640 1 --2- 172.21.15.55:0/385429398 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8a500c9a90 0x7f8a500c73c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8a400097c0 tx=0x7f8a4001c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ef6805e16f2ded3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:11.354 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.352+0000 7f8a54ccf640 1 -- 172.21.15.55:0/385429398 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a400b0070 con 0x7f8a500c9a90 2024-03-19T02:04:11.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.356+0000 7f8a54ccf640 1 -- 172.21.15.55:0/385429398 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a400ab600 con 0x7f8a500c9a90 2024-03-19T02:04:11.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.356+0000 7f8a54ccf640 1 -- 172.21.15.55:0/385429398 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a400ab9d0 con 0x7f8a500c9a90 2024-03-19T02:04:11.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.356+0000 7f8a57f35640 1 -- 172.21.15.55:0/385429398 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8a500c9a90 msgr2=0x7f8a500c73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:11.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.356+0000 7f8a57f35640 1 --2- 172.21.15.55:0/385429398 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8a500c9a90 0x7f8a500c73c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8a400097c0 tx=0x7f8a4001c410 comp rx=0 tx=0).stop 2024-03-19T02:04:11.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.356+0000 7f8a57f35640 1 -- 172.21.15.55:0/385429398 shutdown_connections 2024-03-19T02:04:11.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.356+0000 7f8a57f35640 1 --2- 172.21.15.55:0/385429398 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8a500c9a90 0x7f8a500c73c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:11.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.356+0000 7f8a57f35640 1 --2- 172.21.15.55:0/385429398 >> v2:172.21.15.55:3300/0 conn(0x7f8a500c96b0 0x7f8a50070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:11.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.356+0000 7f8a57f35640 1 -- 172.21.15.55:0/385429398 >> 172.21.15.55:0/385429398 conn(0x7f8a500cc9c0 msgr2=0x7f8a500ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:11.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.356+0000 7f8a57f35640 1 -- 172.21.15.55:0/385429398 shutdown_connections 2024-03-19T02:04:11.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.356+0000 7f8a57f35640 1 -- 172.21.15.55:0/385429398 wait complete. 2024-03-19T02:04:11.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.360+0000 7f8a57f35640 1 Processor -- start 2024-03-19T02:04:11.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.360+0000 7f8a57f35640 1 -- start start 2024-03-19T02:04:11.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.360+0000 7f8a57f35640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8a500c96b0 0x7f8a50149900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:11.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.360+0000 7f8a57f35640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a500c9a90 0x7f8a50149e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:11.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.360+0000 7f8a57f35640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a501540c0 0x7f8a5014d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:11.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.360+0000 7f8a57f35640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8a500c7900 con 0x7f8a501540c0 2024-03-19T02:04:11.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.360+0000 7f8a57f35640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5015b0f0 con 0x7f8a500c9a90 2024-03-19T02:04:11.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.360+0000 7f8a57f35640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f8a500c7a80 con 0x7f8a500c96b0 2024-03-19T02:04:11.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.360+0000 7f8a564d2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a501540c0 0x7f8a5014d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:11.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.360+0000 7f8a554d0640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a500c9a90 0x7f8a50149e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:11.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.360+0000 7f8a564d2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a501540c0 0x7f8a5014d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56880/0 (socket says 172.21.15.55:56880) 2024-03-19T02:04:11.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.360+0000 7f8a564d2640 1 -- 172.21.15.55:0/1414718840 learned_addr learned my addr 172.21.15.55:0/1414718840 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:11.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.360+0000 7f8a55cd1640 1 --2- 172.21.15.55:0/1414718840 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8a500c96b0 0x7f8a50149900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:11.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.364+0000 7f8a564d2640 1 -- 172.21.15.55:0/1414718840 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8a500c96b0 msgr2=0x7f8a50149900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:11.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.364+0000 7f8a564d2640 1 --2- 172.21.15.55:0/1414718840 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8a500c96b0 0x7f8a50149900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:11.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.364+0000 7f8a564d2640 1 -- 172.21.15.55:0/1414718840 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a500c9a90 msgr2=0x7f8a50149e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:11.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.364+0000 7f8a564d2640 1 --2- 172.21.15.55:0/1414718840 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a500c9a90 0x7f8a50149e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:11.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.364+0000 7f8a564d2640 1 -- 172.21.15.55:0/1414718840 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a5014b3c0 con 0x7f8a501540c0 2024-03-19T02:04:11.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.368+0000 7f8a564d2640 1 --2- 172.21.15.55:0/1414718840 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a501540c0 0x7f8a5014d3f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8a4800cbc0 tx=0x7f8a4800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:11.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.372+0000 7f8a467fc640 1 -- 172.21.15.55:0/1414718840 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a48017070 con 0x7f8a501540c0 2024-03-19T02:04:11.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.372+0000 7f8a467fc640 1 -- 172.21.15.55:0/1414718840 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a4800ad40 con 0x7f8a501540c0 2024-03-19T02:04:11.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.372+0000 7f8a57f35640 1 -- 172.21.15.55:0/1414718840 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a5014b6a0 con 0x7f8a501540c0 2024-03-19T02:04:11.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.372+0000 7f8a467fc640 1 -- 172.21.15.55:0/1414718840 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a48012600 con 0x7f8a501540c0 2024-03-19T02:04:11.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.372+0000 7f8a57f35640 1 -- 172.21.15.55:0/1414718840 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a5014c910 con 0x7f8a501540c0 2024-03-19T02:04:11.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.372+0000 7f8a57f35640 1 -- 172.21.15.55:0/1414718840 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a50068a40 con 0x7f8a501540c0 2024-03-19T02:04:11.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.396+0000 7f8a467fc640 1 -- 172.21.15.55:0/1414718840 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f8a4801a070 con 0x7f8a501540c0 2024-03-19T02:04:11.400 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.400+0000 7f8a467fc640 1 --2- 172.21.15.55:0/1414718840 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8a340676b0 0x7f8a34069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:11.400 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.400+0000 7f8a55cd1640 1 --2- 172.21.15.55:0/1414718840 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8a340676b0 0x7f8a34069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:11.400 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.400+0000 7f8a55cd1640 1 --2- 172.21.15.55:0/1414718840 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8a340676b0 0x7f8a34069b70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c009ac0 tx=0x7f8a4c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:11.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.680+0000 7f8a467fc640 1 -- 172.21.15.55:0/1414718840 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f8a48054150 con 0x7f8a501540c0 2024-03-19T02:04:11.682 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.680+0000 7f8a467fc640 1 -- 172.21.15.55:0/1414718840 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f8a480560b0 con 0x7f8a501540c0 2024-03-19T02:04:11.928 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:11.932+0000 7f8a57f35640 1 -- 172.21.15.55:0/1414718840 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a500c8b00 con 0x7f8a501540c0 2024-03-19T02:04:12.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:12.044+0000 7f8a467fc640 1 -- 172.21.15.55:0/1414718840 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f8a4805a150 con 0x7f8a501540c0 2024-03-19T02:04:12.041 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:12.041 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:12.041 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:12.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:12.048+0000 7f8a57f35640 1 -- 172.21.15.55:0/1414718840 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8a340676b0 msgr2=0x7f8a34069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:12.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:12.048+0000 7f8a57f35640 1 --2- 172.21.15.55:0/1414718840 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8a340676b0 0x7f8a34069b70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c009ac0 tx=0x7f8a4c009190 comp rx=0 tx=0).stop 2024-03-19T02:04:12.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:12.048+0000 7f8a57f35640 1 -- 172.21.15.55:0/1414718840 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a501540c0 msgr2=0x7f8a5014d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:12.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:12.048+0000 7f8a57f35640 1 --2- 172.21.15.55:0/1414718840 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a501540c0 0x7f8a5014d3f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8a4800cbc0 tx=0x7f8a4800a430 comp rx=0 tx=0).stop 2024-03-19T02:04:12.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:12.048+0000 7f8a57f35640 1 -- 172.21.15.55:0/1414718840 shutdown_connections 2024-03-19T02:04:12.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:12.048+0000 7f8a57f35640 1 --2- 172.21.15.55:0/1414718840 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8a340676b0 0x7f8a34069b70 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:12.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:12.048+0000 7f8a57f35640 1 --2- 172.21.15.55:0/1414718840 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a501540c0 0x7f8a5014d3f0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:12.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:12.048+0000 7f8a57f35640 1 --2- 172.21.15.55:0/1414718840 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a500c9a90 0x7f8a50149e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:12.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:12.048+0000 7f8a57f35640 1 --2- 172.21.15.55:0/1414718840 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8a500c96b0 0x7f8a50149900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:12.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:12.048+0000 7f8a57f35640 1 -- 172.21.15.55:0/1414718840 >> 172.21.15.55:0/1414718840 conn(0x7f8a500cc9c0 msgr2=0x7f8a500c5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:12.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:12.048+0000 7f8a57f35640 1 -- 172.21.15.55:0/1414718840 shutdown_connections 2024-03-19T02:04:12.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:12.048+0000 7f8a57f35640 1 -- 172.21.15.55:0/1414718840 wait complete. 2024-03-19T02:04:12.666 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:12.666 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:13.668 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:13.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.788+0000 7f8a61e1f640 1 Processor -- start 2024-03-19T02:04:13.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.788+0000 7f8a61e1f640 1 -- start start 2024-03-19T02:04:13.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.788+0000 7f8a61e1f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f8a5c0cd470 0x7f8a5c14e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:13.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.788+0000 7f8a61e1f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f8a5c0c8010 0x7f8a5c14ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:13.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.788+0000 7f8a61e1f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a5c158330 con 0x7f8a5c0cd090 2024-03-19T02:04:13.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.788+0000 7f8a61e1f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f8a5c158e90 con 0x7f8a5c0cd470 2024-03-19T02:04:13.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.788+0000 7f8a61e1f640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f8a5c1585f0 con 0x7f8a5c0c8010 2024-03-19T02:04:13.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.788+0000 7f8a5affd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f8a5c0cd470 0x7f8a5c14e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:13.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.788+0000 7f8a5affd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f8a5c0cd470 0x7f8a5c14e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56890/0 (socket says 172.21.15.55:56890) 2024-03-19T02:04:13.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.788+0000 7f8a5affd640 1 -- 172.21.15.55:0/3419458877 learned_addr learned my addr 172.21.15.55:0/3419458877 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:13.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.792+0000 7f8a5bfff640 1 --2- 172.21.15.55:0/3419458877 >> v2:172.21.15.84:3300/0 conn(0x7f8a5c0c8010 0x7f8a5c14ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:13.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.796+0000 7f8a5bfff640 1 -- 172.21.15.55:0/3419458877 >> v2:172.21.15.55:3300/0 conn(0x7f8a5c0cd470 msgr2=0x7f8a5c14e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:13.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.796+0000 7f8a5bfff640 1 --2- 172.21.15.55:0/3419458877 >> v2:172.21.15.55:3300/0 conn(0x7f8a5c0cd470 0x7f8a5c14e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:13.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.796+0000 7f8a5bfff640 1 -- 172.21.15.55:0/3419458877 >> v1:172.21.15.84:6789/0 conn(0x7f8a5c0cd090 legacy=0x7f8a5c0732b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:13.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.796+0000 7f8a5bfff640 1 -- 172.21.15.55:0/3419458877 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a5c1588d0 con 0x7f8a5c0c8010 2024-03-19T02:04:13.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.800+0000 7f8a5bfff640 1 --2- 172.21.15.55:0/3419458877 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a5c0c8010 0x7f8a5c14ea90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c159060 tx=0x7f8a5401c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a24ef903544f036 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:13.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.808+0000 7f8a5a7fc640 1 -- 172.21.15.55:0/3419458877 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a540b2070 con 0x7f8a5c0c8010 2024-03-19T02:04:13.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.808+0000 7f8a5a7fc640 1 -- 172.21.15.55:0/3419458877 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a540a57f0 con 0x7f8a5c0c8010 2024-03-19T02:04:13.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.808+0000 7f8a5a7fc640 1 -- 172.21.15.55:0/3419458877 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a540a5bc0 con 0x7f8a5c0c8010 2024-03-19T02:04:13.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.808+0000 7f8a61e1f640 1 -- 172.21.15.55:0/3419458877 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a5c0c8010 msgr2=0x7f8a5c14ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:13.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.808+0000 7f8a61e1f640 1 --2- 172.21.15.55:0/3419458877 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a5c0c8010 0x7f8a5c14ea90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c159060 tx=0x7f8a5401c930 comp rx=0 tx=0).stop 2024-03-19T02:04:13.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.808+0000 7f8a61e1f640 1 -- 172.21.15.55:0/3419458877 shutdown_connections 2024-03-19T02:04:13.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.808+0000 7f8a61e1f640 1 --2- 172.21.15.55:0/3419458877 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a5c0c8010 0x7f8a5c14ea90 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:13.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.808+0000 7f8a61e1f640 1 --2- 172.21.15.55:0/3419458877 >> v2:172.21.15.55:3300/0 conn(0x7f8a5c0cd470 0x7f8a5c14e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:13.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.808+0000 7f8a61e1f640 1 -- 172.21.15.55:0/3419458877 >> 172.21.15.55:0/3419458877 conn(0x7f8a5c06e670 msgr2=0x7f8a5c06ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:13.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.808+0000 7f8a61e1f640 1 -- 172.21.15.55:0/3419458877 shutdown_connections 2024-03-19T02:04:13.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.808+0000 7f8a61e1f640 1 -- 172.21.15.55:0/3419458877 wait complete. 2024-03-19T02:04:13.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.812+0000 7f8a61e1f640 1 Processor -- start 2024-03-19T02:04:13.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.812+0000 7f8a61e1f640 1 -- start start 2024-03-19T02:04:13.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.812+0000 7f8a61e1f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a5c0c8010 0x7f8a5c13c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:13.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.812+0000 7f8a61e1f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a5c0cd090 0x7f8a5c1f1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:13.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.812+0000 7f8a61e1f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8a5c0cd470 0x7f8a5c13ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:13.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.812+0000 7f8a61e1f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c164a90 con 0x7f8a5c0cd090 2024-03-19T02:04:13.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.812+0000 7f8a61e1f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c164910 con 0x7f8a5c0c8010 2024-03-19T02:04:13.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.812+0000 7f8a61e1f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f8a5c164c10 con 0x7f8a5c0cd470 2024-03-19T02:04:13.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.812+0000 7f8a5bfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8a5c0cd470 0x7f8a5c13ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:13.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.812+0000 7f8a5b7fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a5c0c8010 0x7f8a5c13c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:13.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.812+0000 7f8a5b7fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a5c0c8010 0x7f8a5c13c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:34078/0 (socket says 172.21.15.55:34078) 2024-03-19T02:04:13.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.812+0000 7f8a5b7fe640 1 -- 172.21.15.55:0/2526714121 learned_addr learned my addr 172.21.15.55:0/2526714121 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:13.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.816+0000 7f8a5b7fe640 1 -- 172.21.15.55:0/2526714121 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8a5c0cd470 msgr2=0x7f8a5c13ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:13.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.816+0000 7f8a5b7fe640 1 --2- 172.21.15.55:0/2526714121 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8a5c0cd470 0x7f8a5c13ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:13.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.816+0000 7f8a5b7fe640 1 -- 172.21.15.55:0/2526714121 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a5c0cd090 msgr2=0x7f8a5c1f1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:13.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.816+0000 7f8a5b7fe640 1 --2- 172.21.15.55:0/2526714121 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a5c0cd090 0x7f8a5c1f1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:13.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.820+0000 7f8a5b7fe640 1 -- 172.21.15.55:0/2526714121 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a5c13c500 con 0x7f8a5c0c8010 2024-03-19T02:04:13.820 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.824+0000 7f8a5b7fe640 1 --2- 172.21.15.55:0/2526714121 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a5c0c8010 0x7f8a5c13c720 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00c450 tx=0x7f8a4c00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:13.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.828+0000 7f8a43fff640 1 -- 172.21.15.55:0/2526714121 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c016020 con 0x7f8a5c0c8010 2024-03-19T02:04:13.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.828+0000 7f8a43fff640 1 -- 172.21.15.55:0/2526714121 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a4c0052d0 con 0x7f8a5c0c8010 2024-03-19T02:04:13.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.828+0000 7f8a43fff640 1 -- 172.21.15.55:0/2526714121 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c007260 con 0x7f8a5c0c8010 2024-03-19T02:04:13.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.828+0000 7f8a61e1f640 1 -- 172.21.15.55:0/2526714121 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a5c1431f0 con 0x7f8a5c0c8010 2024-03-19T02:04:13.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.828+0000 7f8a61e1f640 1 -- 172.21.15.55:0/2526714121 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a5c1f4750 con 0x7f8a5c0c8010 2024-03-19T02:04:13.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.832+0000 7f8a61e1f640 1 -- 172.21.15.55:0/2526714121 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a5c068a40 con 0x7f8a5c0c8010 2024-03-19T02:04:13.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.868+0000 7f8a43fff640 1 -- 172.21.15.55:0/2526714121 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f8a4c007400 con 0x7f8a5c0c8010 2024-03-19T02:04:13.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.868+0000 7f8a43fff640 1 --2- 172.21.15.55:0/2526714121 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8a480676b0 0x7f8a48069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:13.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.868+0000 7f8a5affd640 1 --2- 172.21.15.55:0/2526714121 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8a480676b0 0x7f8a48069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:13.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.872+0000 7f8a5affd640 1 --2- 172.21.15.55:0/2526714121 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8a480676b0 0x7f8a48069b70 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8a50006f50 tx=0x7f8a50006ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:13.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:13.872+0000 7f8a43fff640 1 -- 172.21.15.55:0/2526714121 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f8a4c0545f0 con 0x7f8a5c0c8010 2024-03-19T02:04:14.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:14.192+0000 7f8a43fff640 1 -- 172.21.15.55:0/2526714121 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f8a4c020020 con 0x7f8a5c0c8010 2024-03-19T02:04:14.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:14.400+0000 7f8a61e1f640 1 -- 172.21.15.55:0/2526714121 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a5c14da80 con 0x7f8a5c0c8010 2024-03-19T02:04:14.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:14.436+0000 7f8a43fff640 1 -- 172.21.15.55:0/2526714121 <== mon.1 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f8a4c005480 con 0x7f8a5c0c8010 2024-03-19T02:04:14.433 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:14.433 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:14.433 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:14.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:14.440+0000 7f8a61e1f640 1 -- 172.21.15.55:0/2526714121 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8a480676b0 msgr2=0x7f8a48069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:14.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:14.440+0000 7f8a61e1f640 1 --2- 172.21.15.55:0/2526714121 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8a480676b0 0x7f8a48069b70 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8a50006f50 tx=0x7f8a50006ea0 comp rx=0 tx=0).stop 2024-03-19T02:04:14.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:14.440+0000 7f8a61e1f640 1 -- 172.21.15.55:0/2526714121 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a5c0c8010 msgr2=0x7f8a5c13c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:14.440+0000 7f8a61e1f640 1 --2- 172.21.15.55:0/2526714121 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a5c0c8010 0x7f8a5c13c720 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00c450 tx=0x7f8a4c00c910 comp rx=0 tx=0).stop 2024-03-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:14.440+0000 7f8a61e1f640 1 -- 172.21.15.55:0/2526714121 shutdown_connections 2024-03-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:14.440+0000 7f8a61e1f640 1 --2- 172.21.15.55:0/2526714121 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8a480676b0 0x7f8a48069b70 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:14.440+0000 7f8a61e1f640 1 --2- 172.21.15.55:0/2526714121 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8a5c0cd470 0x7f8a5c13ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:14.440+0000 7f8a61e1f640 1 --2- 172.21.15.55:0/2526714121 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a5c0cd090 0x7f8a5c1f1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:14.440+0000 7f8a61e1f640 1 --2- 172.21.15.55:0/2526714121 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a5c0c8010 0x7f8a5c13c720 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:14.440+0000 7f8a61e1f640 1 -- 172.21.15.55:0/2526714121 >> 172.21.15.55:0/2526714121 conn(0x7f8a5c06e670 msgr2=0x7f8a5c06ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:14.440+0000 7f8a61e1f640 1 -- 172.21.15.55:0/2526714121 shutdown_connections 2024-03-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:14.440+0000 7f8a61e1f640 1 -- 172.21.15.55:0/2526714121 wait complete. 2024-03-19T02:04:15.058 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:15.059 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:16.060 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:16.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.176+0000 7f8bbb11a640 1 Processor -- start 2024-03-19T02:04:16.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.176+0000 7f8bbb11a640 1 -- start start 2024-03-19T02:04:16.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.176+0000 7f8bbb11a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8bb4150f50 0x7f8bb415b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:16.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.176+0000 7f8bbb11a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f8bb40ca350 0x7f8bb40728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:16.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.176+0000 7f8bbb11a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f8bb40c3320 0x7f8bb4157920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:16.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.176+0000 7f8bbb11a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f8bb40c3e00 con 0x7f8bb40c3320 2024-03-19T02:04:16.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.176+0000 7f8bbb11a640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f8bb415ac10 con 0x7f8bb40ca350 2024-03-19T02:04:16.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.176+0000 7f8bbb11a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f8bb4157e60 con 0x7f8bb4150f50 2024-03-19T02:04:16.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.176+0000 7f8bb96b7640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f8bb40c3320 0x7f8bb4157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:16.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.180+0000 7f8bb3fff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f8bb40ca350 0x7f8bb40728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:16.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.180+0000 7f8bb8eb6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8bb4150f50 0x7f8bb415b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:16.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.180+0000 7f8bb96b7640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f8bb40c3320 0x7f8bb4157920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56908/0 (socket says 172.21.15.55:56908) 2024-03-19T02:04:16.176 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.180+0000 7f8bb96b7640 1 -- 172.21.15.55:0/4270424228 learned_addr learned my addr 172.21.15.55:0/4270424228 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:16.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.180+0000 7f8bb96b7640 1 -- 172.21.15.55:0/4270424228 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8bb4150f50 msgr2=0x7f8bb415b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:16.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.180+0000 7f8bb96b7640 1 --2- 172.21.15.55:0/4270424228 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8bb4150f50 0x7f8bb415b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:16.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.180+0000 7f8bb96b7640 1 -- 172.21.15.55:0/4270424228 >> v2:172.21.15.84:3300/0 conn(0x7f8bb40ca350 msgr2=0x7f8bb40728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:16.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.180+0000 7f8bb96b7640 1 --2- 172.21.15.55:0/4270424228 >> v2:172.21.15.84:3300/0 conn(0x7f8bb40ca350 0x7f8bb40728b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:16.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.180+0000 7f8bb96b7640 1 -- 172.21.15.55:0/4270424228 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb4151330 con 0x7f8bb40c3320 2024-03-19T02:04:16.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.184+0000 7f8bb96b7640 1 --2- 172.21.15.55:0/4270424228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8bb40c3320 0x7f8bb4157920 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8bb406d410 tx=0x7f8bac01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c477070bb54dad server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:16.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.188+0000 7f8bb37fe640 1 -- 172.21.15.55:0/4270424228 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac0a5870 con 0x7f8bb40c3320 2024-03-19T02:04:16.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.188+0000 7f8bb37fe640 1 -- 172.21.15.55:0/4270424228 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bac0a5a10 con 0x7f8bb40c3320 2024-03-19T02:04:16.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.188+0000 7f8bb37fe640 1 -- 172.21.15.55:0/4270424228 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac0a5de0 con 0x7f8bb40c3320 2024-03-19T02:04:16.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.192+0000 7f8bbb11a640 1 -- 172.21.15.55:0/4270424228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8bb40c3320 msgr2=0x7f8bb4157920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:16.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.192+0000 7f8bbb11a640 1 --2- 172.21.15.55:0/4270424228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8bb40c3320 0x7f8bb4157920 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8bb406d410 tx=0x7f8bac01c9f0 comp rx=0 tx=0).stop 2024-03-19T02:04:16.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.192+0000 7f8bbb11a640 1 -- 172.21.15.55:0/4270424228 shutdown_connections 2024-03-19T02:04:16.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.192+0000 7f8bbb11a640 1 --2- 172.21.15.55:0/4270424228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8bb40c3320 0x7f8bb4157920 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:16.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.192+0000 7f8bbb11a640 1 --2- 172.21.15.55:0/4270424228 >> v2:172.21.15.84:3300/0 conn(0x7f8bb40ca350 0x7f8bb40728b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:16.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.192+0000 7f8bbb11a640 1 --2- 172.21.15.55:0/4270424228 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8bb4150f50 0x7f8bb415b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:16.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.192+0000 7f8bbb11a640 1 -- 172.21.15.55:0/4270424228 >> 172.21.15.55:0/4270424228 conn(0x7f8bb406e670 msgr2=0x7f8bb406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:16.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.192+0000 7f8bbb11a640 1 -- 172.21.15.55:0/4270424228 shutdown_connections 2024-03-19T02:04:16.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.192+0000 7f8bbb11a640 1 -- 172.21.15.55:0/4270424228 wait complete. 2024-03-19T02:04:16.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.192+0000 7f8bbb11a640 1 Processor -- start 2024-03-19T02:04:16.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.192+0000 7f8bbb11a640 1 -- start start 2024-03-19T02:04:16.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.192+0000 7f8bbb11a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8bb40c3320 0x7f8bb4154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:16.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.192+0000 7f8bbb11a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8bb40ca350 0x7f8bb4143650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:16.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.192+0000 7f8bbb11a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8bb4150f50 0x7f8bb4141480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:16.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.192+0000 7f8bbb11a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb406c3b0 con 0x7f8bb4150f50 2024-03-19T02:04:16.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.192+0000 7f8bbb11a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb415a8a0 con 0x7f8bb40c3320 2024-03-19T02:04:16.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.192+0000 7f8bbb11a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f8bb406c530 con 0x7f8bb40ca350 2024-03-19T02:04:16.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.196+0000 7f8bb96b7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8bb4150f50 0x7f8bb4141480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:16.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.196+0000 7f8bb96b7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8bb4150f50 0x7f8bb4141480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56924/0 (socket says 172.21.15.55:56924) 2024-03-19T02:04:16.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.196+0000 7f8bb96b7640 1 -- 172.21.15.55:0/2997366397 learned_addr learned my addr 172.21.15.55:0/2997366397 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:16.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.196+0000 7f8bb8eb6640 1 --2- 172.21.15.55:0/2997366397 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8bb40c3320 0x7f8bb4154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:16.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.196+0000 7f8bb3fff640 1 --2- 172.21.15.55:0/2997366397 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8bb40ca350 0x7f8bb4143650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:16.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.200+0000 7f8bb96b7640 1 -- 172.21.15.55:0/2997366397 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8bb40ca350 msgr2=0x7f8bb4143650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:16.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.200+0000 7f8bb96b7640 1 --2- 172.21.15.55:0/2997366397 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8bb40ca350 0x7f8bb4143650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:16.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.200+0000 7f8bb96b7640 1 -- 172.21.15.55:0/2997366397 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8bb40c3320 msgr2=0x7f8bb4154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:16.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.200+0000 7f8bb96b7640 1 --2- 172.21.15.55:0/2997366397 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8bb40c3320 0x7f8bb4154150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:16.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.200+0000 7f8bb96b7640 1 -- 172.21.15.55:0/2997366397 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb40c4660 con 0x7f8bb4150f50 2024-03-19T02:04:16.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.204+0000 7f8bb96b7640 1 --2- 172.21.15.55:0/2997366397 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8bb4150f50 0x7f8bb4141480 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8bac0af040 tx=0x7f8bac019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:16.203 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.204+0000 7f8bb17fa640 1 -- 172.21.15.55:0/2997366397 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac0ba070 con 0x7f8bb4150f50 2024-03-19T02:04:16.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.208+0000 7f8bbb11a640 1 -- 172.21.15.55:0/2997366397 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bb41590b0 con 0x7f8bb4150f50 2024-03-19T02:04:16.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.208+0000 7f8bbb11a640 1 -- 172.21.15.55:0/2997366397 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bb4159670 con 0x7f8bb4150f50 2024-03-19T02:04:16.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.208+0000 7f8bbb11a640 1 -- 172.21.15.55:0/2997366397 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bb4068a40 con 0x7f8bb4150f50 2024-03-19T02:04:16.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.208+0000 7f8bb17fa640 1 -- 172.21.15.55:0/2997366397 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bac0a5a10 con 0x7f8bb4150f50 2024-03-19T02:04:16.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.208+0000 7f8bb17fa640 1 -- 172.21.15.55:0/2997366397 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac009510 con 0x7f8bb4150f50 2024-03-19T02:04:16.499 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.500+0000 7f8bb17fa640 1 -- 172.21.15.55:0/2997366397 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f8bac0c5060 con 0x7f8bb4150f50 2024-03-19T02:04:16.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.504+0000 7f8bb17fa640 1 --2- 172.21.15.55:0/2997366397 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8b98067660 0x7f8b98069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:16.501 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.504+0000 7f8bb8eb6640 1 --2- 172.21.15.55:0/2997366397 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8b98067660 0x7f8b98069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:16.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.504+0000 7f8bb8eb6640 1 --2- 172.21.15.55:0/2997366397 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8b98067660 0x7f8b98069b20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009990 tx=0x7f8ba8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:16.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.524+0000 7f8bb17fa640 1 -- 172.21.15.55:0/2997366397 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f8bac0f04c0 con 0x7f8bb4150f50 2024-03-19T02:04:16.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.524+0000 7f8bb17fa640 1 -- 172.21.15.55:0/2997366397 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f8bac0165e0 con 0x7f8bb4150f50 2024-03-19T02:04:16.735 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.736+0000 7f8bbb11a640 1 -- 172.21.15.55:0/2997366397 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8bb415a270 con 0x7f8bb4150f50 2024-03-19T02:04:16.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.812+0000 7f8bb17fa640 1 -- 172.21.15.55:0/2997366397 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f8bac0f2b20 con 0x7f8bb4150f50 2024-03-19T02:04:16.809 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:16.809 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:16.809 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:16.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.812+0000 7f8bbb11a640 1 -- 172.21.15.55:0/2997366397 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8b98067660 msgr2=0x7f8b98069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:16.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.812+0000 7f8bbb11a640 1 --2- 172.21.15.55:0/2997366397 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8b98067660 0x7f8b98069b20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009990 tx=0x7f8ba8008040 comp rx=0 tx=0).stop 2024-03-19T02:04:16.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.812+0000 7f8bbb11a640 1 -- 172.21.15.55:0/2997366397 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8bb4150f50 msgr2=0x7f8bb4141480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:16.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.812+0000 7f8bbb11a640 1 --2- 172.21.15.55:0/2997366397 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8bb4150f50 0x7f8bb4141480 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8bac0af040 tx=0x7f8bac019790 comp rx=0 tx=0).stop 2024-03-19T02:04:16.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.812+0000 7f8bbb11a640 1 -- 172.21.15.55:0/2997366397 shutdown_connections 2024-03-19T02:04:16.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.812+0000 7f8bbb11a640 1 --2- 172.21.15.55:0/2997366397 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f8b98067660 0x7f8b98069b20 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:16.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.812+0000 7f8bbb11a640 1 --2- 172.21.15.55:0/2997366397 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8bb4150f50 0x7f8bb4141480 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:16.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.812+0000 7f8bbb11a640 1 --2- 172.21.15.55:0/2997366397 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8bb40ca350 0x7f8bb4143650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:16.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.812+0000 7f8bbb11a640 1 --2- 172.21.15.55:0/2997366397 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8bb40c3320 0x7f8bb4154150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:16.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.812+0000 7f8bbb11a640 1 -- 172.21.15.55:0/2997366397 >> 172.21.15.55:0/2997366397 conn(0x7f8bb406e670 msgr2=0x7f8bb4155030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:16.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.812+0000 7f8bbb11a640 1 -- 172.21.15.55:0/2997366397 shutdown_connections 2024-03-19T02:04:16.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:16.812+0000 7f8bbb11a640 1 -- 172.21.15.55:0/2997366397 wait complete. 2024-03-19T02:04:17.433 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:17.433 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:18.434 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:18.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.688+0000 7fbace819640 1 Processor -- start 2024-03-19T02:04:18.685 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.688+0000 7fbace819640 1 -- start start 2024-03-19T02:04:18.687 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.688+0000 7fbace819640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbac81510f0 0x7fbac80c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:18.688 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.688+0000 7fbace819640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fbac80c73c0 0x7fbac80c7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:18.688 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.688+0000 7fbace819640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbac80c3f60 con 0x7fbac815a410 2024-03-19T02:04:18.688 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.688+0000 7fbace819640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fbac8070fb0 con 0x7fbac80c73c0 2024-03-19T02:04:18.688 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.688+0000 7fbace819640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbac81514d0 con 0x7fbac81510f0 2024-03-19T02:04:18.688 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.688+0000 7fbac7fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbac81510f0 0x7fbac80c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:18.688 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.688+0000 7fbaccdb6640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fbac80c73c0 0x7fbac80c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:18.688 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.688+0000 7fbac7fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbac81510f0 0x7fbac80c61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:56608/0 (socket says 172.21.15.55:56608) 2024-03-19T02:04:18.688 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.688+0000 7fbac7fff640 1 -- 172.21.15.55:0/2858301776 learned_addr learned my addr 172.21.15.55:0/2858301776 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:18.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.692+0000 7fbac7fff640 1 -- 172.21.15.55:0/2858301776 >> v2:172.21.15.84:3300/0 conn(0x7fbac80c73c0 msgr2=0x7fbac80c7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:18.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.692+0000 7fbac7fff640 1 --2- 172.21.15.55:0/2858301776 >> v2:172.21.15.84:3300/0 conn(0x7fbac80c73c0 0x7fbac80c7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:18.691 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.692+0000 7fbac7fff640 1 -- 172.21.15.55:0/2858301776 >> v1:172.21.15.84:6789/0 conn(0x7fbac815a410 legacy=0x7fbac80708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:18.691 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.692+0000 7fbac7fff640 1 -- 172.21.15.55:0/2858301776 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac80c6870 con 0x7fbac81510f0 2024-03-19T02:04:18.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.696+0000 7fbac7fff640 1 --2- 172.21.15.55:0/2858301776 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbac81510f0 0x7fbac80c61c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbab8009870 tx=0x7fbab801ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74102d925400e325 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:18.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.704+0000 7fbac6ffd640 1 -- 172.21.15.55:0/2858301776 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab80b5070 con 0x7fbac81510f0 2024-03-19T02:04:18.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.704+0000 7fbac6ffd640 1 -- 172.21.15.55:0/2858301776 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbab80a8a50 con 0x7fbac81510f0 2024-03-19T02:04:18.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.704+0000 7fbac6ffd640 1 -- 172.21.15.55:0/2858301776 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab80a8e20 con 0x7fbac81510f0 2024-03-19T02:04:18.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.704+0000 7fbace819640 1 -- 172.21.15.55:0/2858301776 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbac81510f0 msgr2=0x7fbac80c61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:18.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.704+0000 7fbace819640 1 --2- 172.21.15.55:0/2858301776 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbac81510f0 0x7fbac80c61c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbab8009870 tx=0x7fbab801ed50 comp rx=0 tx=0).stop 2024-03-19T02:04:18.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.704+0000 7fbace819640 1 -- 172.21.15.55:0/2858301776 shutdown_connections 2024-03-19T02:04:18.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.704+0000 7fbace819640 1 --2- 172.21.15.55:0/2858301776 >> v2:172.21.15.84:3300/0 conn(0x7fbac80c73c0 0x7fbac80c7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:18.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.704+0000 7fbace819640 1 --2- 172.21.15.55:0/2858301776 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbac81510f0 0x7fbac80c61c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:18.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.704+0000 7fbace819640 1 -- 172.21.15.55:0/2858301776 >> 172.21.15.55:0/2858301776 conn(0x7fbac80cc9c0 msgr2=0x7fbac80ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:18.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.704+0000 7fbace819640 1 -- 172.21.15.55:0/2858301776 shutdown_connections 2024-03-19T02:04:18.704 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.704+0000 7fbace819640 1 -- 172.21.15.55:0/2858301776 wait complete. 2024-03-19T02:04:18.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.708+0000 7fbace819640 1 Processor -- start 2024-03-19T02:04:18.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.708+0000 7fbace819640 1 -- start start 2024-03-19T02:04:18.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.708+0000 7fbace819640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbac80c73c0 0x7fbac81f09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:18.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.708+0000 7fbace819640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbac81510f0 0x7fbac814fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:18.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.708+0000 7fbace819640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbac815a410 0x7fbac8149810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:18.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.708+0000 7fbace819640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbac80c44e0 con 0x7fbac815a410 2024-03-19T02:04:18.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.708+0000 7fbace819640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fbac80c4200 con 0x7fbac81510f0 2024-03-19T02:04:18.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.708+0000 7fbace819640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbac80c6e30 con 0x7fbac80c73c0 2024-03-19T02:04:18.706 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.708+0000 7fbac7fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbac80c73c0 0x7fbac81f09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:18.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.708+0000 7fbaccdb6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbac815a410 0x7fbac8149810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:18.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.708+0000 7fbaccdb6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbac815a410 0x7fbac8149810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:45438/0 (socket says 172.21.15.55:45438) 2024-03-19T02:04:18.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.708+0000 7fbaccdb6640 1 -- 172.21.15.55:0/109598681 learned_addr learned my addr 172.21.15.55:0/109598681 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:18.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.708+0000 7fbac77fe640 1 --2- 172.21.15.55:0/109598681 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbac81510f0 0x7fbac814fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:18.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.712+0000 7fbaccdb6640 1 -- 172.21.15.55:0/109598681 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbac80c73c0 msgr2=0x7fbac81f09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:18.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.712+0000 7fbaccdb6640 1 --2- 172.21.15.55:0/109598681 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbac80c73c0 0x7fbac81f09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:18.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.712+0000 7fbaccdb6640 1 -- 172.21.15.55:0/109598681 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbac81510f0 msgr2=0x7fbac814fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:18.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.712+0000 7fbaccdb6640 1 --2- 172.21.15.55:0/109598681 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbac81510f0 0x7fbac814fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:18.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.712+0000 7fbaccdb6640 1 -- 172.21.15.55:0/109598681 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac81ecdd0 con 0x7fbac815a410 2024-03-19T02:04:18.713 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.716+0000 7fbaccdb6640 1 --2- 172.21.15.55:0/109598681 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbac815a410 0x7fbac8149810 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbac000efc0 tx=0x7fbac000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:18.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.720+0000 7fbac4ff9640 1 -- 172.21.15.55:0/109598681 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0018020 con 0x7fbac815a410 2024-03-19T02:04:18.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.720+0000 7fbac4ff9640 1 -- 172.21.15.55:0/109598681 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbac0014480 con 0x7fbac815a410 2024-03-19T02:04:18.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.720+0000 7fbace819640 1 -- 172.21.15.55:0/109598681 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbac81ed0b0 con 0x7fbac815a410 2024-03-19T02:04:18.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.720+0000 7fbac4ff9640 1 -- 172.21.15.55:0/109598681 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0014870 con 0x7fbac815a410 2024-03-19T02:04:18.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.720+0000 7fbace819640 1 -- 172.21.15.55:0/109598681 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbac81ed670 con 0x7fbac815a410 2024-03-19T02:04:18.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.724+0000 7fbace819640 1 -- 172.21.15.55:0/109598681 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbac8068a40 con 0x7fbac815a410 2024-03-19T02:04:18.974 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.976+0000 7fbac4ff9640 1 -- 172.21.15.55:0/109598681 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fbac001b020 con 0x7fbac815a410 2024-03-19T02:04:18.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.976+0000 7fbac4ff9640 1 --2- 172.21.15.55:0/109598681 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbab40676b0 0x7fbab4069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:18.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.976+0000 7fbac7fff640 1 --2- 172.21.15.55:0/109598681 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbab40676b0 0x7fbab4069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:18.975 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.980+0000 7fbac7fff640 1 --2- 172.21.15.55:0/109598681 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbab40676b0 0x7fbab4069b70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fbab80b2000 tx=0x7fbab80aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:18.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:18.980+0000 7fbac4ff9640 1 -- 172.21.15.55:0/109598681 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fbac0055380 con 0x7fbac815a410 2024-03-19T02:04:19.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:19.004+0000 7fbac4ff9640 1 -- 172.21.15.55:0/109598681 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fbac0022020 con 0x7fbac815a410 2024-03-19T02:04:19.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:19.220+0000 7fbace819640 1 -- 172.21.15.55:0/109598681 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbac80ca860 con 0x7fbac815a410 2024-03-19T02:04:19.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:19.236+0000 7fbac4ff9640 1 -- 172.21.15.55:0/109598681 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fbac0010040 con 0x7fbac815a410 2024-03-19T02:04:19.234 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:19.234 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:19.234 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:19.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:19.240+0000 7fbace819640 1 -- 172.21.15.55:0/109598681 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbab40676b0 msgr2=0x7fbab4069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:19.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:19.240+0000 7fbace819640 1 --2- 172.21.15.55:0/109598681 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbab40676b0 0x7fbab4069b70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fbab80b2000 tx=0x7fbab80aa070 comp rx=0 tx=0).stop 2024-03-19T02:04:19.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:19.240+0000 7fbace819640 1 -- 172.21.15.55:0/109598681 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbac815a410 msgr2=0x7fbac8149810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:19.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:19.240+0000 7fbace819640 1 --2- 172.21.15.55:0/109598681 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbac815a410 0x7fbac8149810 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbac000efc0 tx=0x7fbac000c490 comp rx=0 tx=0).stop 2024-03-19T02:04:19.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:19.240+0000 7fbace819640 1 -- 172.21.15.55:0/109598681 shutdown_connections 2024-03-19T02:04:19.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:19.240+0000 7fbace819640 1 --2- 172.21.15.55:0/109598681 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbab40676b0 0x7fbab4069b70 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:19.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:19.240+0000 7fbace819640 1 --2- 172.21.15.55:0/109598681 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbac815a410 0x7fbac8149810 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:19.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:19.240+0000 7fbace819640 1 --2- 172.21.15.55:0/109598681 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbac81510f0 0x7fbac814fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:19.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:19.240+0000 7fbace819640 1 --2- 172.21.15.55:0/109598681 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbac80c73c0 0x7fbac81f09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:19.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:19.240+0000 7fbace819640 1 -- 172.21.15.55:0/109598681 >> 172.21.15.55:0/109598681 conn(0x7fbac80cc9c0 msgr2=0x7fbac815a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:19.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:19.240+0000 7fbace819640 1 -- 172.21.15.55:0/109598681 shutdown_connections 2024-03-19T02:04:19.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:19.240+0000 7fbace819640 1 -- 172.21.15.55:0/109598681 wait complete. 2024-03-19T02:04:19.862 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:19.862 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:20.864 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:20.991 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:20.991+0000 7f10ef291640 1 Processor -- start 2024-03-19T02:04:20.991 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:20.991+0000 7f10ef291640 1 -- start start 2024-03-19T02:04:20.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:20.995+0000 7f10ef291640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f10e80c9780 0x7f10e80c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:20.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:20.995+0000 7f10ef291640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10e80c8f70 con 0x7f10e813dd00 2024-03-19T02:04:20.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:20.995+0000 7f10ef291640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10e80c9250 con 0x7f10e815b550 2024-03-19T02:04:20.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:20.995+0000 7f10ef291640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f10e81582b0 con 0x7f10e80c9780 2024-03-19T02:04:20.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:20.995+0000 7f10ec82c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f10e80c9780 0x7f10e80c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:20.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:20.995+0000 7f10ec82c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f10e80c9780 0x7f10e80c7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:45450/0 (socket says 172.21.15.55:45450) 2024-03-19T02:04:20.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:20.995+0000 7f10ec82c640 1 -- 172.21.15.55:0/1437655841 learned_addr learned my addr 172.21.15.55:0/1437655841 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:20.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:20.999+0000 7f10dffff640 1 -- 172.21.15.55:0/1437655841 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2249188460 0 0) 0x7f10e80c9250 con 0x7f10e815b550 2024-03-19T02:04:20.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:20.999+0000 7f10dffff640 1 -- 172.21.15.55:0/1437655841 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10d4003530 con 0x7f10e815b550 2024-03-19T02:04:20.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:20.999+0000 7f10ec82c640 1 -- 172.21.15.55:0/1437655841 >> v1:172.21.15.84:6789/0 conn(0x7f10e815b550 legacy=0x7f10e80c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:20.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:20.999+0000 7f10ec82c640 1 -- 172.21.15.55:0/1437655841 >> v1:172.21.15.55:6789/0 conn(0x7f10e813dd00 legacy=0x7f10e80c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:20.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:20.999+0000 7f10ec82c640 1 -- 172.21.15.55:0/1437655841 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e8158590 con 0x7f10e80c9780 2024-03-19T02:04:21.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.007+0000 7f10ec82c640 1 --2- 172.21.15.55:0/1437655841 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10e80c9780 0x7f10e80c7430 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f10d8009870 tx=0x7f10d801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c78fe51251d96eb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:21.008 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.011+0000 7f10dffff640 1 -- 172.21.15.55:0/1437655841 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d80a7070 con 0x7f10e80c9780 2024-03-19T02:04:21.008 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.011+0000 7f10dffff640 1 -- 172.21.15.55:0/1437655841 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10d80b0070 con 0x7f10e80c9780 2024-03-19T02:04:21.008 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.011+0000 7f10dffff640 1 -- 172.21.15.55:0/1437655841 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d80ab890 con 0x7f10e80c9780 2024-03-19T02:04:21.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.011+0000 7f10ef291640 1 -- 172.21.15.55:0/1437655841 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10e80c9780 msgr2=0x7f10e80c7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:21.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.011+0000 7f10ef291640 1 --2- 172.21.15.55:0/1437655841 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10e80c9780 0x7f10e80c7430 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f10d8009870 tx=0x7f10d801c560 comp rx=0 tx=0).stop 2024-03-19T02:04:21.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.011+0000 7f10ef291640 1 -- 172.21.15.55:0/1437655841 shutdown_connections 2024-03-19T02:04:21.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.011+0000 7f10ef291640 1 --2- 172.21.15.55:0/1437655841 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10e80c9780 0x7f10e80c7430 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:21.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.011+0000 7f10ef291640 1 -- 172.21.15.55:0/1437655841 >> 172.21.15.55:0/1437655841 conn(0x7f10e806e5a0 msgr2=0x7f10e8072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:21.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.011+0000 7f10ef291640 1 -- 172.21.15.55:0/1437655841 shutdown_connections 2024-03-19T02:04:21.009 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.011+0000 7f10ef291640 1 -- 172.21.15.55:0/1437655841 wait complete. 2024-03-19T02:04:21.010 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.011+0000 7f10ef291640 1 Processor -- start 2024-03-19T02:04:21.010 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.011+0000 7f10ef291640 1 -- start start 2024-03-19T02:04:21.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.015+0000 7f10ef291640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10e80c9780 0x7f10e8154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:21.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.015+0000 7f10ef291640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f10e813dd00 0x7f10e8150570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:21.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.015+0000 7f10ef291640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10e815b550 0x7f10e8150ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:21.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.015+0000 7f10ef291640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f10e80c5fc0 con 0x7f10e815b550 2024-03-19T02:04:21.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.015+0000 7f10ef291640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f10e80c5e40 con 0x7f10e80c9780 2024-03-19T02:04:21.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.015+0000 7f10ef291640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f10e8166520 con 0x7f10e813dd00 2024-03-19T02:04:21.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.015+0000 7f10ed82e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10e815b550 0x7f10e8150ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:21.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.015+0000 7f10ec82c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f10e813dd00 0x7f10e8150570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:21.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.015+0000 7f10ed82e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10e815b550 0x7f10e8150ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:45460/0 (socket says 172.21.15.55:45460) 2024-03-19T02:04:21.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.015+0000 7f10ed82e640 1 -- 172.21.15.55:0/3602533730 learned_addr learned my addr 172.21.15.55:0/3602533730 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:21.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.015+0000 7f10ed02d640 1 --2- 172.21.15.55:0/3602533730 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10e80c9780 0x7f10e8154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:21.016 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.019+0000 7f10ed82e640 1 -- 172.21.15.55:0/3602533730 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f10e813dd00 msgr2=0x7f10e8150570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:21.016 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.019+0000 7f10ed82e640 1 --2- 172.21.15.55:0/3602533730 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f10e813dd00 0x7f10e8150570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:21.016 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.019+0000 7f10ed82e640 1 -- 172.21.15.55:0/3602533730 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10e80c9780 msgr2=0x7f10e8154db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:21.016 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.019+0000 7f10ed82e640 1 --2- 172.21.15.55:0/3602533730 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10e80c9780 0x7f10e8154db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:21.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.019+0000 7f10ed82e640 1 -- 172.21.15.55:0/3602533730 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e813ee70 con 0x7f10e815b550 2024-03-19T02:04:21.018 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.019+0000 7f10ed82e640 1 --2- 172.21.15.55:0/3602533730 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10e815b550 0x7f10e8150ab0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f10e000c760 tx=0x7f10e000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:21.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.023+0000 7f10ddffb640 1 -- 172.21.15.55:0/3602533730 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e0003b80 con 0x7f10e815b550 2024-03-19T02:04:21.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.023+0000 7f10ddffb640 1 -- 172.21.15.55:0/3602533730 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10e0003d20 con 0x7f10e815b550 2024-03-19T02:04:21.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.023+0000 7f10ef291640 1 -- 172.21.15.55:0/3602533730 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10e813f110 con 0x7f10e815b550 2024-03-19T02:04:21.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.023+0000 7f10ddffb640 1 -- 172.21.15.55:0/3602533730 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e000aa90 con 0x7f10e815b550 2024-03-19T02:04:21.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.027+0000 7f10ef291640 1 -- 172.21.15.55:0/3602533730 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10e813f6d0 con 0x7f10e815b550 2024-03-19T02:04:21.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.027+0000 7f10ef291640 1 -- 172.21.15.55:0/3602533730 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10e8068a40 con 0x7f10e815b550 2024-03-19T02:04:21.296 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.299+0000 7f10ddffb640 1 -- 172.21.15.55:0/3602533730 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f10e000ac30 con 0x7f10e815b550 2024-03-19T02:04:21.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.299+0000 7f10ddffb640 1 --2- 172.21.15.55:0/3602533730 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f10cc0676e0 0x7f10cc069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:21.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.299+0000 7f10ed02d640 1 --2- 172.21.15.55:0/3602533730 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f10cc0676e0 0x7f10cc069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:21.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.299+0000 7f10ed02d640 1 --2- 172.21.15.55:0/3602533730 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f10cc0676e0 0x7f10cc069ba0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f10e4001770 tx=0x7f10e4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:21.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.331+0000 7f10ddffb640 1 -- 172.21.15.55:0/3602533730 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f10e0054790 con 0x7f10e815b550 2024-03-19T02:04:21.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.331+0000 7f10ddffb640 1 -- 172.21.15.55:0/3602533730 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f10e00566f0 con 0x7f10e815b550 2024-03-19T02:04:21.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.567+0000 7f10ef291640 1 -- 172.21.15.55:0/3602533730 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10e81636d0 con 0x7f10e815b550 2024-03-19T02:04:21.624 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.627+0000 7f10ddffb640 1 -- 172.21.15.55:0/3602533730 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f10e005a790 con 0x7f10e815b550 2024-03-19T02:04:21.625 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:21.625 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:21.625 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:21.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.631+0000 7f10ef291640 1 -- 172.21.15.55:0/3602533730 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f10cc0676e0 msgr2=0x7f10cc069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:21.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.631+0000 7f10ef291640 1 --2- 172.21.15.55:0/3602533730 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f10cc0676e0 0x7f10cc069ba0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f10e4001770 tx=0x7f10e4009040 comp rx=0 tx=0).stop 2024-03-19T02:04:21.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.631+0000 7f10ef291640 1 -- 172.21.15.55:0/3602533730 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10e815b550 msgr2=0x7f10e8150ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:21.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.631+0000 7f10ef291640 1 --2- 172.21.15.55:0/3602533730 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10e815b550 0x7f10e8150ab0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f10e000c760 tx=0x7f10e000cc20 comp rx=0 tx=0).stop 2024-03-19T02:04:21.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.631+0000 7f10ef291640 1 -- 172.21.15.55:0/3602533730 shutdown_connections 2024-03-19T02:04:21.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.631+0000 7f10ef291640 1 --2- 172.21.15.55:0/3602533730 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f10cc0676e0 0x7f10cc069ba0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:21.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.631+0000 7f10ef291640 1 --2- 172.21.15.55:0/3602533730 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f10e815b550 0x7f10e8150ab0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:21.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.631+0000 7f10ef291640 1 --2- 172.21.15.55:0/3602533730 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f10e813dd00 0x7f10e8150570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:21.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.631+0000 7f10ef291640 1 --2- 172.21.15.55:0/3602533730 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10e80c9780 0x7f10e8154db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:21.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.631+0000 7f10ef291640 1 -- 172.21.15.55:0/3602533730 >> 172.21.15.55:0/3602533730 conn(0x7f10e806e5a0 msgr2=0x7f10e806b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:21.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.631+0000 7f10ef291640 1 -- 172.21.15.55:0/3602533730 shutdown_connections 2024-03-19T02:04:21.630 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:21.631+0000 7f10ef291640 1 -- 172.21.15.55:0/3602533730 wait complete. 2024-03-19T02:04:22.248 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:22.248 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:23.249 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:23.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.363+0000 7f4c6d0db640 1 Processor -- start 2024-03-19T02:04:23.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.363+0000 7f4c6d0db640 1 -- start start 2024-03-19T02:04:23.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.367+0000 7f4c6d0db640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4c68152120 0x7f4c680c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:23.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.367+0000 7f4c6d0db640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4c680c9610 0x7f4c68157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:23.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.367+0000 7f4c6d0db640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c6806fff0 con 0x7f4c68070780 2024-03-19T02:04:23.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.367+0000 7f4c6d0db640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f4c68152500 con 0x7f4c68152120 2024-03-19T02:04:23.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.367+0000 7f4c6d0db640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4c68070590 con 0x7f4c680c9610 2024-03-19T02:04:23.365 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.367+0000 7f4c66d9d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4c68152120 0x7f4c680c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:23.365 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.367+0000 7f4c6659c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4c680c9610 0x7f4c68157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:23.365 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.367+0000 7f4c66d9d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f4c68152120 0x7f4c680c90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:45474/0 (socket says 172.21.15.55:45474) 2024-03-19T02:04:23.365 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.367+0000 7f4c66d9d640 1 -- 172.21.15.55:0/1031619436 learned_addr learned my addr 172.21.15.55:0/1031619436 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:23.369 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.371+0000 7f4c66d9d640 1 -- 172.21.15.55:0/1031619436 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4c680c9610 msgr2=0x7f4c68157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:23.369 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.371+0000 7f4c66d9d640 1 --2- 172.21.15.55:0/1031619436 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4c680c9610 0x7f4c68157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:23.369 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.371+0000 7f4c66d9d640 1 -- 172.21.15.55:0/1031619436 >> v1:172.21.15.55:6789/0 conn(0x7f4c68070780 legacy=0x7f4c68158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:23.369 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.371+0000 7f4c66d9d640 1 -- 172.21.15.55:0/1031619436 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c680c39c0 con 0x7f4c68152120 2024-03-19T02:04:23.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.375+0000 7f4c66d9d640 1 --2- 172.21.15.55:0/1031619436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c68152120 0x7f4c680c90d0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009870 tx=0x7f4c5801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd6d2bff9f15877f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:23.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.383+0000 7f4c65d9b640 1 -- 172.21.15.55:0/1031619436 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c580b0070 con 0x7f4c68152120 2024-03-19T02:04:23.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.383+0000 7f4c65d9b640 1 -- 172.21.15.55:0/1031619436 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c580ab600 con 0x7f4c68152120 2024-03-19T02:04:23.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.383+0000 7f4c65d9b640 1 -- 172.21.15.55:0/1031619436 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c580ab9d0 con 0x7f4c68152120 2024-03-19T02:04:23.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.383+0000 7f4c6d0db640 1 -- 172.21.15.55:0/1031619436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c68152120 msgr2=0x7f4c680c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:23.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.383+0000 7f4c6d0db640 1 --2- 172.21.15.55:0/1031619436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c68152120 0x7f4c680c90d0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009870 tx=0x7f4c5801c520 comp rx=0 tx=0).stop 2024-03-19T02:04:23.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.383+0000 7f4c6d0db640 1 -- 172.21.15.55:0/1031619436 shutdown_connections 2024-03-19T02:04:23.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.383+0000 7f4c6d0db640 1 --2- 172.21.15.55:0/1031619436 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4c680c9610 0x7f4c68157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:23.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.383+0000 7f4c6d0db640 1 --2- 172.21.15.55:0/1031619436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c68152120 0x7f4c680c90d0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:23.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.383+0000 7f4c6d0db640 1 -- 172.21.15.55:0/1031619436 >> 172.21.15.55:0/1031619436 conn(0x7f4c68067600 msgr2=0x7f4c68071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:23.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.383+0000 7f4c6d0db640 1 -- 172.21.15.55:0/1031619436 shutdown_connections 2024-03-19T02:04:23.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.383+0000 7f4c6d0db640 1 -- 172.21.15.55:0/1031619436 wait complete. 2024-03-19T02:04:23.384 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.387+0000 7f4c6d0db640 1 Processor -- start 2024-03-19T02:04:23.384 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.387+0000 7f4c6d0db640 1 -- start start 2024-03-19T02:04:23.384 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.387+0000 7f4c6d0db640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c68070780 0x7f4c680ce290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:23.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.387+0000 7f4c6d0db640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c680c9610 0x7f4c681f3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:23.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.387+0000 7f4c6d0db640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4c68152120 0x7f4c681f7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:23.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.387+0000 7f4c6d0db640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4c680c86d0 con 0x7f4c680c9610 2024-03-19T02:04:23.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.387+0000 7f4c6d0db640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4c680c31c0 con 0x7f4c68070780 2024-03-19T02:04:23.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.387+0000 7f4c6d0db640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f4c68165170 con 0x7f4c68152120 2024-03-19T02:04:23.385 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.387+0000 7f4c6659c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c680c9610 0x7f4c681f3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:23.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.387+0000 7f4c66d9d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c68070780 0x7f4c680ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:23.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.387+0000 7f4c6659c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c680c9610 0x7f4c681f3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:45488/0 (socket says 172.21.15.55:45488) 2024-03-19T02:04:23.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.387+0000 7f4c6659c640 1 -- 172.21.15.55:0/3791135945 learned_addr learned my addr 172.21.15.55:0/3791135945 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:23.386 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.387+0000 7f4c6759e640 1 --2- 172.21.15.55:0/3791135945 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4c68152120 0x7f4c681f7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:23.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.391+0000 7f4c6659c640 1 -- 172.21.15.55:0/3791135945 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4c68152120 msgr2=0x7f4c681f7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:23.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.391+0000 7f4c6659c640 1 --2- 172.21.15.55:0/3791135945 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4c68152120 0x7f4c681f7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:23.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.391+0000 7f4c6659c640 1 -- 172.21.15.55:0/3791135945 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c68070780 msgr2=0x7f4c680ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:23.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.391+0000 7f4c6659c640 1 --2- 172.21.15.55:0/3791135945 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c68070780 0x7f4c680ce290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:23.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.391+0000 7f4c6659c640 1 -- 172.21.15.55:0/3791135945 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c680c6dc0 con 0x7f4c680c9610 2024-03-19T02:04:23.394 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.395+0000 7f4c6659c640 1 --2- 172.21.15.55:0/3791135945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c680c9610 0x7f4c681f3f10 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00e930 tx=0x7f4c5c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:23.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.399+0000 7f4c4f7fe640 1 -- 172.21.15.55:0/3791135945 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c017070 con 0x7f4c680c9610 2024-03-19T02:04:23.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.399+0000 7f4c4f7fe640 1 -- 172.21.15.55:0/3791135945 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c5c00ca00 con 0x7f4c680c9610 2024-03-19T02:04:23.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.399+0000 7f4c4f7fe640 1 -- 172.21.15.55:0/3791135945 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c00cdf0 con 0x7f4c680c9610 2024-03-19T02:04:23.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.399+0000 7f4c6d0db640 1 -- 172.21.15.55:0/3791135945 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c680c7030 con 0x7f4c680c9610 2024-03-19T02:04:23.399 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.399+0000 7f4c6d0db640 1 -- 172.21.15.55:0/3791135945 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c680c75f0 con 0x7f4c680c9610 2024-03-19T02:04:23.400 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.403+0000 7f4c6d0db640 1 -- 172.21.15.55:0/3791135945 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c681f69f0 con 0x7f4c680c9610 2024-03-19T02:04:23.692 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.695+0000 7f4c4f7fe640 1 -- 172.21.15.55:0/3791135945 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f4c5c01a070 con 0x7f4c680c9610 2024-03-19T02:04:23.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.695+0000 7f4c4f7fe640 1 --2- 172.21.15.55:0/3791135945 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4c540676e0 0x7f4c54069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:23.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.695+0000 7f4c66d9d640 1 --2- 172.21.15.55:0/3791135945 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4c540676e0 0x7f4c54069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:23.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.695+0000 7f4c66d9d640 1 --2- 172.21.15.55:0/3791135945 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4c540676e0 0x7f4c54069ba0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4c580ad000 tx=0x7f4c580a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:23.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.735+0000 7f4c4f7fe640 1 -- 172.21.15.55:0/3791135945 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f4c5c0546b0 con 0x7f4c680c9610 2024-03-19T02:04:23.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:23.747+0000 7f4c4f7fe640 1 -- 172.21.15.55:0/3791135945 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f4c5c056610 con 0x7f4c680c9610 2024-03-19T02:04:24.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:24.003+0000 7f4c6d0db640 1 -- 172.21.15.55:0/3791135945 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4c68068a40 con 0x7f4c680c9610 2024-03-19T02:04:24.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:24.031+0000 7f4c4f7fe640 1 -- 172.21.15.55:0/3791135945 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f4c5c05a6b0 con 0x7f4c680c9610 2024-03-19T02:04:24.029 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:24.029 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:24.029 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:24.031 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:24.031+0000 7f4c6d0db640 1 -- 172.21.15.55:0/3791135945 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4c540676e0 msgr2=0x7f4c54069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:24.031 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:24.031+0000 7f4c6d0db640 1 --2- 172.21.15.55:0/3791135945 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4c540676e0 0x7f4c54069ba0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4c580ad000 tx=0x7f4c580a6000 comp rx=0 tx=0).stop 2024-03-19T02:04:24.031 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:24.031+0000 7f4c6d0db640 1 -- 172.21.15.55:0/3791135945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c680c9610 msgr2=0x7f4c681f3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:24.031 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:24.031+0000 7f4c6d0db640 1 --2- 172.21.15.55:0/3791135945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c680c9610 0x7f4c681f3f10 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00e930 tx=0x7f4c5c00edf0 comp rx=0 tx=0).stop 2024-03-19T02:04:24.031 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:24.031+0000 7f4c6d0db640 1 -- 172.21.15.55:0/3791135945 shutdown_connections 2024-03-19T02:04:24.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:24.031+0000 7f4c6d0db640 1 --2- 172.21.15.55:0/3791135945 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4c540676e0 0x7f4c54069ba0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:24.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:24.031+0000 7f4c6d0db640 1 --2- 172.21.15.55:0/3791135945 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f4c68152120 0x7f4c681f7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:24.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:24.031+0000 7f4c6d0db640 1 --2- 172.21.15.55:0/3791135945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c680c9610 0x7f4c681f3f10 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:24.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:24.031+0000 7f4c6d0db640 1 --2- 172.21.15.55:0/3791135945 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c68070780 0x7f4c680ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:24.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:24.031+0000 7f4c6d0db640 1 -- 172.21.15.55:0/3791135945 >> 172.21.15.55:0/3791135945 conn(0x7f4c68067600 msgr2=0x7f4c6814cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:24.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:24.031+0000 7f4c6d0db640 1 -- 172.21.15.55:0/3791135945 shutdown_connections 2024-03-19T02:04:24.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:24.031+0000 7f4c6d0db640 1 -- 172.21.15.55:0/3791135945 wait complete. 2024-03-19T02:04:24.655 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:24.655 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:25.657 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:25.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.779+0000 7f7f54cd8640 1 Processor -- start 2024-03-19T02:04:25.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.779+0000 7f7f54cd8640 1 -- start start 2024-03-19T02:04:25.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.779+0000 7f7f54cd8640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7f500c96b0 0x7f7f50070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:25.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.779+0000 7f7f54cd8640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f7f501540c0 0x7f7f50154550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:25.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.779+0000 7f7f54cd8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f5015e3a0 con 0x7f7f500c9a90 2024-03-19T02:04:25.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.779+0000 7f7f54cd8640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f7f500c7ad0 con 0x7f7f500c96b0 2024-03-19T02:04:25.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.779+0000 7f7f54cd8640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f7f5015e680 con 0x7f7f501540c0 2024-03-19T02:04:25.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.779+0000 7f7f4ed9d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f7f501540c0 0x7f7f50154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:25.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.783+0000 7f7f4ed9d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f7f501540c0 0x7f7f50154550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:39994/0 (socket says 172.21.15.55:39994) 2024-03-19T02:04:25.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.783+0000 7f7f4ed9d640 1 -- 172.21.15.55:0/2480053969 learned_addr learned my addr 172.21.15.55:0/2480053969 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:25.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.787+0000 7f7f4ed9d640 1 -- 172.21.15.55:0/2480053969 >> v2:172.21.15.55:3300/0 conn(0x7f7f500c96b0 msgr2=0x7f7f50070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:25.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.787+0000 7f7f4ed9d640 1 --2- 172.21.15.55:0/2480053969 >> v2:172.21.15.55:3300/0 conn(0x7f7f500c96b0 0x7f7f50070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:25.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.787+0000 7f7f4ed9d640 1 -- 172.21.15.55:0/2480053969 >> v1:172.21.15.55:6789/0 conn(0x7f7f500c9a90 legacy=0x7f7f500c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:25.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.787+0000 7f7f4ed9d640 1 -- 172.21.15.55:0/2480053969 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f5015a330 con 0x7f7f501540c0 2024-03-19T02:04:25.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.791+0000 7f7f4ed9d640 1 --2- 172.21.15.55:0/2480053969 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f501540c0 0x7f7f50154550 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7f50072890 tx=0x7f7f4801c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=120dc260c2722736 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:25.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.803+0000 7f7f4d59a640 1 -- 172.21.15.55:0/2480053969 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f480a56d0 con 0x7f7f501540c0 2024-03-19T02:04:25.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.803+0000 7f7f4d59a640 1 -- 172.21.15.55:0/2480053969 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f480a5870 con 0x7f7f501540c0 2024-03-19T02:04:25.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.803+0000 7f7f4d59a640 1 -- 172.21.15.55:0/2480053969 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f480a5c40 con 0x7f7f501540c0 2024-03-19T02:04:25.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.803+0000 7f7f54cd8640 1 -- 172.21.15.55:0/2480053969 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f501540c0 msgr2=0x7f7f50154550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:25.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.803+0000 7f7f54cd8640 1 --2- 172.21.15.55:0/2480053969 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f501540c0 0x7f7f50154550 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7f50072890 tx=0x7f7f4801c920 comp rx=0 tx=0).stop 2024-03-19T02:04:25.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.803+0000 7f7f54cd8640 1 -- 172.21.15.55:0/2480053969 shutdown_connections 2024-03-19T02:04:25.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.803+0000 7f7f54cd8640 1 --2- 172.21.15.55:0/2480053969 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f501540c0 0x7f7f50154550 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:25.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.803+0000 7f7f54cd8640 1 --2- 172.21.15.55:0/2480053969 >> v2:172.21.15.55:3300/0 conn(0x7f7f500c96b0 0x7f7f50070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:25.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.803+0000 7f7f54cd8640 1 -- 172.21.15.55:0/2480053969 >> 172.21.15.55:0/2480053969 conn(0x7f7f500cc9c0 msgr2=0x7f7f500ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:25.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.803+0000 7f7f54cd8640 1 -- 172.21.15.55:0/2480053969 shutdown_connections 2024-03-19T02:04:25.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.803+0000 7f7f54cd8640 1 -- 172.21.15.55:0/2480053969 wait complete. 2024-03-19T02:04:25.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.803+0000 7f7f54cd8640 1 Processor -- start 2024-03-19T02:04:25.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.803+0000 7f7f54cd8640 1 -- start start 2024-03-19T02:04:25.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.803+0000 7f7f54cd8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7f500c96b0 0x7f7f501405c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:25.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.803+0000 7f7f54cd8640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f500c9a90 0x7f7f500cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:25.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.803+0000 7f7f54cd8640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7f501540c0 0x7f7f50142f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:25.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.803+0000 7f7f54cd8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7f50166b90 con 0x7f7f500c96b0 2024-03-19T02:04:25.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.803+0000 7f7f54cd8640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5015b0f0 con 0x7f7f500c9a90 2024-03-19T02:04:25.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.803+0000 7f7f54cd8640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f7f50166d10 con 0x7f7f501540c0 2024-03-19T02:04:25.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.807+0000 7f7f4ed9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7f501540c0 0x7f7f50142f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:25.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.807+0000 7f7f4e59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7f500c96b0 0x7f7f501405c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:25.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.807+0000 7f7f4e59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7f500c96b0 0x7f7f501405c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:45494/0 (socket says 172.21.15.55:45494) 2024-03-19T02:04:25.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.807+0000 7f7f4e59c640 1 -- 172.21.15.55:0/1613324427 learned_addr learned my addr 172.21.15.55:0/1613324427 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:25.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.807+0000 7f7f4dd9b640 1 --2- 172.21.15.55:0/1613324427 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f500c9a90 0x7f7f500cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:25.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.811+0000 7f7f4e59c640 1 -- 172.21.15.55:0/1613324427 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7f501540c0 msgr2=0x7f7f50142f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:25.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.811+0000 7f7f4e59c640 1 --2- 172.21.15.55:0/1613324427 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7f501540c0 0x7f7f50142f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:25.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.811+0000 7f7f4e59c640 1 -- 172.21.15.55:0/1613324427 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f500c9a90 msgr2=0x7f7f500cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:25.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.811+0000 7f7f4e59c640 1 --2- 172.21.15.55:0/1613324427 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f500c9a90 0x7f7f500cab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:25.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.811+0000 7f7f4e59c640 1 -- 172.21.15.55:0/1613324427 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f50140b00 con 0x7f7f500c96b0 2024-03-19T02:04:25.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.811+0000 7f7f4e59c640 1 --2- 172.21.15.55:0/1613324427 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7f500c96b0 0x7f7f501405c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7f4000b410 tx=0x7f7f4000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:25.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.815+0000 7f7f3effd640 1 -- 172.21.15.55:0/1613324427 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f40015020 con 0x7f7f500c96b0 2024-03-19T02:04:25.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.815+0000 7f7f3effd640 1 -- 172.21.15.55:0/1613324427 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f40011830 con 0x7f7f500c96b0 2024-03-19T02:04:25.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.815+0000 7f7f3effd640 1 -- 172.21.15.55:0/1613324427 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f40011c20 con 0x7f7f500c96b0 2024-03-19T02:04:25.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.815+0000 7f7f54cd8640 1 -- 172.21.15.55:0/1613324427 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f501528c0 con 0x7f7f500c96b0 2024-03-19T02:04:25.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.815+0000 7f7f54cd8640 1 -- 172.21.15.55:0/1613324427 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f50148bf0 con 0x7f7f500c96b0 2024-03-19T02:04:25.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.815+0000 7f7f54cd8640 1 -- 172.21.15.55:0/1613324427 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f50068a40 con 0x7f7f500c96b0 2024-03-19T02:04:25.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.839+0000 7f7f3effd640 1 -- 172.21.15.55:0/1613324427 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f7f40018020 con 0x7f7f500c96b0 2024-03-19T02:04:25.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.839+0000 7f7f3effd640 1 --2- 172.21.15.55:0/1613324427 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7f300676e0 0x7f7f30069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:25.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.839+0000 7f7f4dd9b640 1 --2- 172.21.15.55:0/1613324427 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7f300676e0 0x7f7f30069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:25.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:25.839+0000 7f7f4dd9b640 1 --2- 172.21.15.55:0/1613324427 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7f300676e0 0x7f7f30069ba0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7f50141950 tx=0x7f7f44009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:26.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:26.111+0000 7f7f3effd640 1 -- 172.21.15.55:0/1613324427 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f7f400542e0 con 0x7f7f500c96b0 2024-03-19T02:04:26.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:26.115+0000 7f7f3effd640 1 -- 172.21.15.55:0/1613324427 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f7f4001f020 con 0x7f7f500c96b0 2024-03-19T02:04:26.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:26.323+0000 7f7f54cd8640 1 -- 172.21.15.55:0/1613324427 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f500ca950 con 0x7f7f500c96b0 2024-03-19T02:04:26.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:26.355+0000 7f7f3effd640 1 -- 172.21.15.55:0/1613324427 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f7f40056240 con 0x7f7f500c96b0 2024-03-19T02:04:26.354 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:26.354 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:26.354 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:26.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:26.359+0000 7f7f54cd8640 1 -- 172.21.15.55:0/1613324427 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7f300676e0 msgr2=0x7f7f30069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:26.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:26.359+0000 7f7f54cd8640 1 --2- 172.21.15.55:0/1613324427 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7f300676e0 0x7f7f30069ba0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7f50141950 tx=0x7f7f44009060 comp rx=0 tx=0).stop 2024-03-19T02:04:26.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:26.359+0000 7f7f54cd8640 1 -- 172.21.15.55:0/1613324427 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7f500c96b0 msgr2=0x7f7f501405c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:26.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:26.359+0000 7f7f54cd8640 1 --2- 172.21.15.55:0/1613324427 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7f500c96b0 0x7f7f501405c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7f4000b410 tx=0x7f7f4000b8d0 comp rx=0 tx=0).stop 2024-03-19T02:04:26.357 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:26.359+0000 7f7f54cd8640 1 -- 172.21.15.55:0/1613324427 shutdown_connections 2024-03-19T02:04:26.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:26.359+0000 7f7f54cd8640 1 --2- 172.21.15.55:0/1613324427 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7f300676e0 0x7f7f30069ba0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:26.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:26.359+0000 7f7f54cd8640 1 --2- 172.21.15.55:0/1613324427 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7f501540c0 0x7f7f50142f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:26.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:26.359+0000 7f7f54cd8640 1 --2- 172.21.15.55:0/1613324427 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f500c9a90 0x7f7f500cab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:26.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:26.359+0000 7f7f54cd8640 1 --2- 172.21.15.55:0/1613324427 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7f500c96b0 0x7f7f501405c0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:26.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:26.359+0000 7f7f54cd8640 1 -- 172.21.15.55:0/1613324427 >> 172.21.15.55:0/1613324427 conn(0x7f7f500cc9c0 msgr2=0x7f7f500c6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:26.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:26.359+0000 7f7f54cd8640 1 -- 172.21.15.55:0/1613324427 shutdown_connections 2024-03-19T02:04:26.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:26.359+0000 7f7f54cd8640 1 -- 172.21.15.55:0/1613324427 wait complete. 2024-03-19T02:04:26.982 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:26.982 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:27.983 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:28.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.111+0000 7f5fca2ff640 1 Processor -- start 2024-03-19T02:04:28.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.111+0000 7f5fca2ff640 1 -- start start 2024-03-19T02:04:28.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.115+0000 7f5fca2ff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f5fc40ca350 0x7f5fc4072900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:28.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.115+0000 7f5fca2ff640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fc40738b0 con 0x7f5fc4150e50 2024-03-19T02:04:28.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.115+0000 7f5fca2ff640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fc4072f40 con 0x7f5fc415a490 2024-03-19T02:04:28.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.115+0000 7f5fca2ff640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f5fc40732f0 con 0x7f5fc40ca350 2024-03-19T02:04:28.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.115+0000 7f5fc3fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f5fc4150e50 0x7f5fc41578b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:34776/0 (socket says 172.21.15.55:34776) 2024-03-19T02:04:28.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.115+0000 7f5fc3fff640 1 -- 172.21.15.55:0/762281346 learned_addr learned my addr 172.21.15.55:0/762281346 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:04:28.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.115+0000 7f5fc889c640 1 --2- 172.21.15.55:0/762281346 >> v2:172.21.15.84:3300/0 conn(0x7f5fc40ca350 0x7f5fc4072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:28.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.115+0000 7f5fc2ffd640 1 -- 172.21.15.55:0/762281346 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1780953553 0 0) 0x7f5fc40738b0 con 0x7f5fc4150e50 2024-03-19T02:04:28.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.115+0000 7f5fc2ffd640 1 -- 172.21.15.55:0/762281346 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fac005740 con 0x7f5fc4150e50 2024-03-19T02:04:28.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.119+0000 7f5fc889c640 1 -- 172.21.15.55:0/762281346 >> v1:172.21.15.84:6789/0 conn(0x7f5fc415a490 legacy=0x7f5fc4070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:28.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.119+0000 7f5fc889c640 1 -- 172.21.15.55:0/762281346 >> v1:172.21.15.55:6789/0 conn(0x7f5fc4150e50 legacy=0x7f5fc41578b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:28.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.119+0000 7f5fc889c640 1 -- 172.21.15.55:0/762281346 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fc40735d0 con 0x7f5fc40ca350 2024-03-19T02:04:28.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.123+0000 7f5fc889c640 1 --2- 172.21.15.55:0/762281346 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5fc40ca350 0x7f5fc4072900 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc009d90 tx=0x7f5fbc01c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cf94f42b3a1face server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:28.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.131+0000 7f5fc2ffd640 1 -- 172.21.15.55:0/762281346 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc0b2070 con 0x7f5fc40ca350 2024-03-19T02:04:28.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.131+0000 7f5fc2ffd640 1 -- 172.21.15.55:0/762281346 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fbc0a57e0 con 0x7f5fc40ca350 2024-03-19T02:04:28.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.131+0000 7f5fc2ffd640 1 -- 172.21.15.55:0/762281346 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc0a5bb0 con 0x7f5fc40ca350 2024-03-19T02:04:28.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.131+0000 7f5fca2ff640 1 -- 172.21.15.55:0/762281346 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5fc40ca350 msgr2=0x7f5fc4072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:28.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.131+0000 7f5fca2ff640 1 --2- 172.21.15.55:0/762281346 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5fc40ca350 0x7f5fc4072900 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc009d90 tx=0x7f5fbc01c930 comp rx=0 tx=0).stop 2024-03-19T02:04:28.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.131+0000 7f5fca2ff640 1 -- 172.21.15.55:0/762281346 shutdown_connections 2024-03-19T02:04:28.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.131+0000 7f5fca2ff640 1 --2- 172.21.15.55:0/762281346 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5fc40ca350 0x7f5fc4072900 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:28.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.131+0000 7f5fca2ff640 1 -- 172.21.15.55:0/762281346 >> 172.21.15.55:0/762281346 conn(0x7f5fc406c3b0 msgr2=0x7f5fc406c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:28.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.131+0000 7f5fca2ff640 1 -- 172.21.15.55:0/762281346 shutdown_connections 2024-03-19T02:04:28.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.131+0000 7f5fca2ff640 1 -- 172.21.15.55:0/762281346 wait complete. 2024-03-19T02:04:28.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.131+0000 7f5fca2ff640 1 Processor -- start 2024-03-19T02:04:28.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.131+0000 7f5fca2ff640 1 -- start start 2024-03-19T02:04:28.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.135+0000 7f5fca2ff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5fc40ca350 0x7f5fc41558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:28.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.135+0000 7f5fca2ff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fc4150e50 0x7f5fc4152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:28.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.135+0000 7f5fca2ff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5fc415a490 0x7f5fc4153010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:28.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.135+0000 7f5fca2ff640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc41513b0 con 0x7f5fc4150e50 2024-03-19T02:04:28.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.135+0000 7f5fca2ff640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc4151230 con 0x7f5fc40ca350 2024-03-19T02:04:28.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.135+0000 7f5fca2ff640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f5fc4070c10 con 0x7f5fc415a490 2024-03-19T02:04:28.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.135+0000 7f5fc37fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fc4150e50 0x7f5fc4152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:28.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.135+0000 7f5fc889c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5fc415a490 0x7f5fc4153010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:28.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.135+0000 7f5fc37fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fc4150e50 0x7f5fc4152ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:45504/0 (socket says 172.21.15.55:45504) 2024-03-19T02:04:28.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.135+0000 7f5fc37fe640 1 -- 172.21.15.55:0/578890722 learned_addr learned my addr 172.21.15.55:0/578890722 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:28.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.135+0000 7f5fc3fff640 1 --2- 172.21.15.55:0/578890722 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5fc40ca350 0x7f5fc41558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:28.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.139+0000 7f5fc37fe640 1 -- 172.21.15.55:0/578890722 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5fc415a490 msgr2=0x7f5fc4153010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:28.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.139+0000 7f5fc37fe640 1 --2- 172.21.15.55:0/578890722 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5fc415a490 0x7f5fc4153010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:28.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.139+0000 7f5fc37fe640 1 -- 172.21.15.55:0/578890722 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5fc40ca350 msgr2=0x7f5fc41558a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:28.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.139+0000 7f5fc37fe640 1 --2- 172.21.15.55:0/578890722 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5fc40ca350 0x7f5fc41558a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:28.137 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.139+0000 7f5fc37fe640 1 -- 172.21.15.55:0/578890722 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fc41f5f40 con 0x7f5fc4150e50 2024-03-19T02:04:28.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.139+0000 7f5fc37fe640 1 --2- 172.21.15.55:0/578890722 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fc4150e50 0x7f5fc4152ad0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800c400 tx=0x7f5fb800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:28.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.143+0000 7f5fc0ff9640 1 -- 172.21.15.55:0/578890722 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8017070 con 0x7f5fc4150e50 2024-03-19T02:04:28.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.143+0000 7f5fc0ff9640 1 -- 172.21.15.55:0/578890722 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fb8003600 con 0x7f5fc4150e50 2024-03-19T02:04:28.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.143+0000 7f5fca2ff640 1 -- 172.21.15.55:0/578890722 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fc41f6220 con 0x7f5fc4150e50 2024-03-19T02:04:28.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.143+0000 7f5fc0ff9640 1 -- 172.21.15.55:0/578890722 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb80039f0 con 0x7f5fc4150e50 2024-03-19T02:04:28.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.143+0000 7f5fca2ff640 1 -- 172.21.15.55:0/578890722 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fc41f67e0 con 0x7f5fc4150e50 2024-03-19T02:04:28.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.147+0000 7f5fca2ff640 1 -- 172.21.15.55:0/578890722 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fc4068a40 con 0x7f5fc4150e50 2024-03-19T02:04:28.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.411+0000 7f5fc0ff9640 1 -- 172.21.15.55:0/578890722 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f5fb801a070 con 0x7f5fc4150e50 2024-03-19T02:04:28.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.411+0000 7f5fc0ff9640 1 --2- 172.21.15.55:0/578890722 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5fb00676b0 0x7f5fb0069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:28.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.411+0000 7f5fc3fff640 1 --2- 172.21.15.55:0/578890722 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5fb00676b0 0x7f5fb0069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:28.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.415+0000 7f5fc3fff640 1 --2- 172.21.15.55:0/578890722 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5fb00676b0 0x7f5fb0069b70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4007fd0 tx=0x7f5fb4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:28.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.427+0000 7f5fc0ff9640 1 -- 172.21.15.55:0/578890722 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f5fb8053b30 con 0x7f5fc4150e50 2024-03-19T02:04:28.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.427+0000 7f5fc0ff9640 1 -- 172.21.15.55:0/578890722 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f5fb8055a90 con 0x7f5fc4150e50 2024-03-19T02:04:28.653 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.655+0000 7f5fca2ff640 1 -- 172.21.15.55:0/578890722 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5fc406d550 con 0x7f5fc4150e50 2024-03-19T02:04:28.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.691+0000 7f5fc0ff9640 1 -- 172.21.15.55:0/578890722 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f5fb8059b30 con 0x7f5fc4150e50 2024-03-19T02:04:28.690 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:28.691 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:28.691 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:28.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.695+0000 7f5fca2ff640 1 -- 172.21.15.55:0/578890722 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5fb00676b0 msgr2=0x7f5fb0069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:28.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.695+0000 7f5fca2ff640 1 --2- 172.21.15.55:0/578890722 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5fb00676b0 0x7f5fb0069b70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4007fd0 tx=0x7f5fb4009040 comp rx=0 tx=0).stop 2024-03-19T02:04:28.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.695+0000 7f5fca2ff640 1 -- 172.21.15.55:0/578890722 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fc4150e50 msgr2=0x7f5fc4152ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:28.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.695+0000 7f5fca2ff640 1 --2- 172.21.15.55:0/578890722 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fc4150e50 0x7f5fc4152ad0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800c400 tx=0x7f5fb800c8c0 comp rx=0 tx=0).stop 2024-03-19T02:04:28.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.695+0000 7f5fca2ff640 1 -- 172.21.15.55:0/578890722 shutdown_connections 2024-03-19T02:04:28.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.695+0000 7f5fca2ff640 1 --2- 172.21.15.55:0/578890722 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5fb00676b0 0x7f5fb0069b70 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:28.693 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.695+0000 7f5fca2ff640 1 --2- 172.21.15.55:0/578890722 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5fc415a490 0x7f5fc4153010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:28.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.695+0000 7f5fca2ff640 1 --2- 172.21.15.55:0/578890722 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5fc4150e50 0x7f5fc4152ad0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:28.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.695+0000 7f5fca2ff640 1 --2- 172.21.15.55:0/578890722 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5fc40ca350 0x7f5fc41558a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:28.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.695+0000 7f5fca2ff640 1 -- 172.21.15.55:0/578890722 >> 172.21.15.55:0/578890722 conn(0x7f5fc406c3b0 msgr2=0x7f5fc406c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:28.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.695+0000 7f5fca2ff640 1 -- 172.21.15.55:0/578890722 shutdown_connections 2024-03-19T02:04:28.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:28.695+0000 7f5fca2ff640 1 -- 172.21.15.55:0/578890722 wait complete. 2024-03-19T02:04:29.313 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:29.313 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:30.314 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:30.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.435+0000 7f7b8b6d9640 1 Processor -- start 2024-03-19T02:04:30.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.435+0000 7f7b8b6d9640 1 -- start start 2024-03-19T02:04:30.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.435+0000 7f7b8b6d9640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f7b84150f50 0x7f7b8415b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:30.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.435+0000 7f7b8b6d9640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7b840c3320 0x7f7b840728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:30.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.435+0000 7f7b8b6d9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b84151330 con 0x7f7b840ca350 2024-03-19T02:04:30.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.435+0000 7f7b8b6d9640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f7b840c3e00 con 0x7f7b840c3320 2024-03-19T02:04:30.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.435+0000 7f7b8b6d9640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f7b8415ac10 con 0x7f7b84150f50 2024-03-19T02:04:30.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.439+0000 7f7b89c76640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7b840c3320 0x7f7b840728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:30.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.439+0000 7f7b89475640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f7b84150f50 0x7f7b8415b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:30.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.439+0000 7f7b89c76640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7b840c3320 0x7f7b840728b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:36976/0 (socket says 172.21.15.55:36976) 2024-03-19T02:04:30.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.439+0000 7f7b89c76640 1 -- 172.21.15.55:0/2821948925 learned_addr learned my addr 172.21.15.55:0/2821948925 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:30.439 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.439+0000 7f7b89c76640 1 -- 172.21.15.55:0/2821948925 >> v2:172.21.15.84:3300/0 conn(0x7f7b84150f50 msgr2=0x7f7b8415b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:30.439 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.439+0000 7f7b89c76640 1 --2- 172.21.15.55:0/2821948925 >> v2:172.21.15.84:3300/0 conn(0x7f7b84150f50 0x7f7b8415b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:30.439 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.439+0000 7f7b89c76640 1 -- 172.21.15.55:0/2821948925 >> v1:172.21.15.84:6789/0 conn(0x7f7b840ca350 legacy=0x7f7b84157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:30.439 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.439+0000 7f7b89c76640 1 -- 172.21.15.55:0/2821948925 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b840c3840 con 0x7f7b840c3320 2024-03-19T02:04:30.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.443+0000 7f7b89c76640 1 --2- 172.21.15.55:0/2821948925 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b840c3320 0x7f7b840728b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f7b840cdbf0 tx=0x7f7b7c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d797c5f97c315441 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:30.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.447+0000 7f7b83fff640 1 -- 172.21.15.55:0/2821948925 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c0aa680 con 0x7f7b840c3320 2024-03-19T02:04:30.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.447+0000 7f7b83fff640 1 -- 172.21.15.55:0/2821948925 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b7c0af070 con 0x7f7b840c3320 2024-03-19T02:04:30.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.447+0000 7f7b83fff640 1 -- 172.21.15.55:0/2821948925 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c0aaa40 con 0x7f7b840c3320 2024-03-19T02:04:30.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.451+0000 7f7b8b6d9640 1 -- 172.21.15.55:0/2821948925 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b840c3320 msgr2=0x7f7b840728b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:30.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.451+0000 7f7b8b6d9640 1 --2- 172.21.15.55:0/2821948925 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b840c3320 0x7f7b840728b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f7b840cdbf0 tx=0x7f7b7c01c520 comp rx=0 tx=0).stop 2024-03-19T02:04:30.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.451+0000 7f7b8b6d9640 1 -- 172.21.15.55:0/2821948925 shutdown_connections 2024-03-19T02:04:30.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.451+0000 7f7b8b6d9640 1 --2- 172.21.15.55:0/2821948925 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b840c3320 0x7f7b840728b0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:30.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.451+0000 7f7b8b6d9640 1 --2- 172.21.15.55:0/2821948925 >> v2:172.21.15.84:3300/0 conn(0x7f7b84150f50 0x7f7b8415b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:30.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.451+0000 7f7b8b6d9640 1 -- 172.21.15.55:0/2821948925 >> 172.21.15.55:0/2821948925 conn(0x7f7b8406e670 msgr2=0x7f7b8406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:30.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.451+0000 7f7b8b6d9640 1 -- 172.21.15.55:0/2821948925 shutdown_connections 2024-03-19T02:04:30.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.451+0000 7f7b8b6d9640 1 -- 172.21.15.55:0/2821948925 wait complete. 2024-03-19T02:04:30.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.451+0000 7f7b8b6d9640 1 Processor -- start 2024-03-19T02:04:30.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.451+0000 7f7b8b6d9640 1 -- start start 2024-03-19T02:04:30.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.451+0000 7f7b8b6d9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b840c3320 0x7f7b84143940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:30.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.451+0000 7f7b8b6d9640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7b840ca350 0x7f7b841eee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:30.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.451+0000 7f7b8b6d9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b84150f50 0x7f7b841402e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:30.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.451+0000 7f7b8b6d9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8406b420 con 0x7f7b84150f50 2024-03-19T02:04:30.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.451+0000 7f7b8b6d9640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8406b2a0 con 0x7f7b840ca350 2024-03-19T02:04:30.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.451+0000 7f7b8b6d9640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f7b8406b5a0 con 0x7f7b840c3320 2024-03-19T02:04:30.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.451+0000 7f7b89c76640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b84150f50 0x7f7b841402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:30.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.451+0000 7f7b89c76640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b84150f50 0x7f7b841402e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:36978/0 (socket says 172.21.15.55:36978) 2024-03-19T02:04:30.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.451+0000 7f7b89c76640 1 -- 172.21.15.55:0/389429009 learned_addr learned my addr 172.21.15.55:0/389429009 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:30.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.451+0000 7f7b88c74640 1 --2- 172.21.15.55:0/389429009 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7b840ca350 0x7f7b841eee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:30.452 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.451+0000 7f7b89475640 1 --2- 172.21.15.55:0/389429009 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b840c3320 0x7f7b84143940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:30.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.455+0000 7f7b89c76640 1 -- 172.21.15.55:0/389429009 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b840c3320 msgr2=0x7f7b84143940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:30.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.455+0000 7f7b89c76640 1 --2- 172.21.15.55:0/389429009 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b840c3320 0x7f7b84143940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:30.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.455+0000 7f7b89c76640 1 -- 172.21.15.55:0/389429009 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7b840ca350 msgr2=0x7f7b841eee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:30.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.455+0000 7f7b89c76640 1 --2- 172.21.15.55:0/389429009 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7b840ca350 0x7f7b841eee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:30.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.455+0000 7f7b89c76640 1 -- 172.21.15.55:0/389429009 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b841f1e20 con 0x7f7b84150f50 2024-03-19T02:04:30.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.459+0000 7f7b89c76640 1 --2- 172.21.15.55:0/389429009 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b84150f50 0x7f7b841402e0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c01a150 tx=0x7f7b7c01ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:30.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.463+0000 7f7b81ffb640 1 -- 172.21.15.55:0/389429009 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c0bd5f0 con 0x7f7b84150f50 2024-03-19T02:04:30.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.463+0000 7f7b81ffb640 1 -- 172.21.15.55:0/389429009 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b7c0bd7b0 con 0x7f7b84150f50 2024-03-19T02:04:30.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.463+0000 7f7b81ffb640 1 -- 172.21.15.55:0/389429009 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c0bdba0 con 0x7f7b84150f50 2024-03-19T02:04:30.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.463+0000 7f7b8b6d9640 1 -- 172.21.15.55:0/389429009 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b841f20e0 con 0x7f7b84150f50 2024-03-19T02:04:30.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.463+0000 7f7b8b6d9640 1 -- 172.21.15.55:0/389429009 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b841f26a0 con 0x7f7b84150f50 2024-03-19T02:04:30.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.467+0000 7f7b8b6d9640 1 -- 172.21.15.55:0/389429009 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b84068a40 con 0x7f7b84150f50 2024-03-19T02:04:30.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.759+0000 7f7b81ffb640 1 -- 172.21.15.55:0/389429009 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f7b7c0a8050 con 0x7f7b84150f50 2024-03-19T02:04:30.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.759+0000 7f7b81ffb640 1 --2- 172.21.15.55:0/389429009 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7b700676e0 0x7f7b70069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:30.757 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.759+0000 7f7b89475640 1 --2- 172.21.15.55:0/389429009 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7b700676e0 0x7f7b70069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:30.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.759+0000 7f7b89475640 1 --2- 172.21.15.55:0/389429009 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7b700676e0 0x7f7b70069ba0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7b74003ab0 tx=0x7f7b74009830 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:30.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.783+0000 7f7b81ffb640 1 -- 172.21.15.55:0/389429009 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f7b7c0eea60 con 0x7f7b84150f50 2024-03-19T02:04:30.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.783+0000 7f7b81ffb640 1 -- 172.21.15.55:0/389429009 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f7b7c0c3060 con 0x7f7b84150f50 2024-03-19T02:04:30.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:30.999+0000 7f7b8b6d9640 1 -- 172.21.15.55:0/389429009 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b840c7600 con 0x7f7b84150f50 2024-03-19T02:04:31.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:31.027+0000 7f7b81ffb640 1 -- 172.21.15.55:0/389429009 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f7b7c0a6040 con 0x7f7b84150f50 2024-03-19T02:04:31.025 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:31.025 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:31.025 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:31.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:31.027+0000 7f7b8b6d9640 1 -- 172.21.15.55:0/389429009 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7b700676e0 msgr2=0x7f7b70069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:31.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:31.027+0000 7f7b8b6d9640 1 --2- 172.21.15.55:0/389429009 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7b700676e0 0x7f7b70069ba0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7b74003ab0 tx=0x7f7b74009830 comp rx=0 tx=0).stop 2024-03-19T02:04:31.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:31.027+0000 7f7b8b6d9640 1 -- 172.21.15.55:0/389429009 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b84150f50 msgr2=0x7f7b841402e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:31.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:31.027+0000 7f7b8b6d9640 1 --2- 172.21.15.55:0/389429009 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b84150f50 0x7f7b841402e0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c01a150 tx=0x7f7b7c01ad60 comp rx=0 tx=0).stop 2024-03-19T02:04:31.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:31.027+0000 7f7b8b6d9640 1 -- 172.21.15.55:0/389429009 shutdown_connections 2024-03-19T02:04:31.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:31.027+0000 7f7b8b6d9640 1 --2- 172.21.15.55:0/389429009 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7b700676e0 0x7f7b70069ba0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:31.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:31.027+0000 7f7b8b6d9640 1 --2- 172.21.15.55:0/389429009 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b84150f50 0x7f7b841402e0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:31.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:31.027+0000 7f7b8b6d9640 1 --2- 172.21.15.55:0/389429009 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7b840ca350 0x7f7b841eee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:31.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:31.027+0000 7f7b8b6d9640 1 --2- 172.21.15.55:0/389429009 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b840c3320 0x7f7b84143940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:31.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:31.027+0000 7f7b8b6d9640 1 -- 172.21.15.55:0/389429009 >> 172.21.15.55:0/389429009 conn(0x7f7b8406e670 msgr2=0x7f7b8414de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:31.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:31.027+0000 7f7b8b6d9640 1 -- 172.21.15.55:0/389429009 shutdown_connections 2024-03-19T02:04:31.028 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:31.027+0000 7f7b8b6d9640 1 -- 172.21.15.55:0/389429009 wait complete. 2024-03-19T02:04:31.647 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:31.647 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:32.648 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:32.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.779+0000 7fefd2ca1640 1 Processor -- start 2024-03-19T02:04:32.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.779+0000 7fefd2ca1640 1 -- start start 2024-03-19T02:04:32.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.779+0000 7fefd2ca1640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fefcc14ec60 0x7fefcc06b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:32.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.779+0000 7fefd2ca1640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefcc158330 con 0x7fefcc0cd470 2024-03-19T02:04:32.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.779+0000 7fefd2ca1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefcc1585f0 con 0x7fefcc0cd090 2024-03-19T02:04:32.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.779+0000 7fefd2ca1640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fefcc158e90 con 0x7fefcc14ec60 2024-03-19T02:04:32.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.779+0000 7fefd123e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fefcc14ec60 0x7fefcc06b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:32.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.779+0000 7fefd123e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fefcc14ec60 0x7fefcc06b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:36994/0 (socket says 172.21.15.55:36994) 2024-03-19T02:04:32.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.779+0000 7fefd123e640 1 -- 172.21.15.55:0/1478126460 learned_addr learned my addr 172.21.15.55:0/1478126460 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:32.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.783+0000 7fefd123e640 1 -- 172.21.15.55:0/1478126460 >> v1:172.21.15.84:6789/0 conn(0x7fefcc0cd090 legacy=0x7fefcc0732b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:32.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.783+0000 7fefd123e640 1 -- 172.21.15.55:0/1478126460 >> v1:172.21.15.55:6789/0 conn(0x7fefcc0cd470 legacy=0x7fefcc14e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:32.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.783+0000 7fefd123e640 1 -- 172.21.15.55:0/1478126460 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefcc1588d0 con 0x7fefcc14ec60 2024-03-19T02:04:32.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.791+0000 7fefd123e640 1 --2- 172.21.15.55:0/1478126460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefcc14ec60 0x7fefcc06b8f0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fefcc159010 tx=0x7fefc401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66229cdc3c479c9a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:32.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.795+0000 7fefcb7fe640 1 -- 172.21.15.55:0/1478126460 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc40a56e0 con 0x7fefcc14ec60 2024-03-19T02:04:32.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.795+0000 7fefcb7fe640 1 -- 172.21.15.55:0/1478126460 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefc40a5880 con 0x7fefcc14ec60 2024-03-19T02:04:32.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.795+0000 7fefcb7fe640 1 -- 172.21.15.55:0/1478126460 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc40a5c50 con 0x7fefcc14ec60 2024-03-19T02:04:32.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.795+0000 7fefd2ca1640 1 -- 172.21.15.55:0/1478126460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefcc14ec60 msgr2=0x7fefcc06b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:32.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.795+0000 7fefd2ca1640 1 --2- 172.21.15.55:0/1478126460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefcc14ec60 0x7fefcc06b8f0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fefcc159010 tx=0x7fefc401ca10 comp rx=0 tx=0).stop 2024-03-19T02:04:32.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.795+0000 7fefd2ca1640 1 -- 172.21.15.55:0/1478126460 shutdown_connections 2024-03-19T02:04:32.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.795+0000 7fefd2ca1640 1 --2- 172.21.15.55:0/1478126460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefcc14ec60 0x7fefcc06b8f0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:32.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.795+0000 7fefd2ca1640 1 -- 172.21.15.55:0/1478126460 >> 172.21.15.55:0/1478126460 conn(0x7fefcc06e670 msgr2=0x7fefcc06ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:32.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.795+0000 7fefd2ca1640 1 -- 172.21.15.55:0/1478126460 shutdown_connections 2024-03-19T02:04:32.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.795+0000 7fefd2ca1640 1 -- 172.21.15.55:0/1478126460 wait complete. 2024-03-19T02:04:32.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.795+0000 7fefd2ca1640 1 Processor -- start 2024-03-19T02:04:32.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.795+0000 7fefd2ca1640 1 -- start start 2024-03-19T02:04:32.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.799+0000 7fefd2ca1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefcc0cd090 0x7fefcc13b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:32.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.799+0000 7fefd2ca1640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fefcc0cd470 0x7fefcc0c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:32.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.799+0000 7fefd2ca1640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fefcc14ec60 0x7fefcc0c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:32.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.799+0000 7fefd2ca1640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fefcc157f80 con 0x7fefcc0cd090 2024-03-19T02:04:32.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.799+0000 7fefd2ca1640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fefcc166130 con 0x7fefcc14ec60 2024-03-19T02:04:32.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.799+0000 7fefd2ca1640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fefcc0c1b60 con 0x7fefcc0cd470 2024-03-19T02:04:32.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.799+0000 7fefd0a3d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefcc0cd090 0x7fefcc13b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:32.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.799+0000 7fefd123e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fefcc14ec60 0x7fefcc0c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:32.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.799+0000 7fefd0a3d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefcc0cd090 0x7fefcc13b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:37000/0 (socket says 172.21.15.55:37000) 2024-03-19T02:04:32.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.799+0000 7fefd0a3d640 1 -- 172.21.15.55:0/3311620042 learned_addr learned my addr 172.21.15.55:0/3311620042 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:32.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.799+0000 7fefcbfff640 1 --2- 172.21.15.55:0/3311620042 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fefcc0cd470 0x7fefcc0c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:32.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.803+0000 7fefd0a3d640 1 -- 172.21.15.55:0/3311620042 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fefcc0cd470 msgr2=0x7fefcc0c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:32.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.803+0000 7fefd0a3d640 1 --2- 172.21.15.55:0/3311620042 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fefcc0cd470 0x7fefcc0c49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:32.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.803+0000 7fefd0a3d640 1 -- 172.21.15.55:0/3311620042 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fefcc14ec60 msgr2=0x7fefcc0c4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:32.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.803+0000 7fefd0a3d640 1 --2- 172.21.15.55:0/3311620042 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fefcc14ec60 0x7fefcc0c4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:32.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.803+0000 7fefd0a3d640 1 -- 172.21.15.55:0/3311620042 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefcc0cbf40 con 0x7fefcc0cd090 2024-03-19T02:04:32.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.803+0000 7fefd0a3d640 1 --2- 172.21.15.55:0/3311620042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefcc0cd090 0x7fefcc13b840 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fefc000c3c0 tx=0x7fefc000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:32.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.807+0000 7fefc97fa640 1 -- 172.21.15.55:0/3311620042 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc0017070 con 0x7fefcc0cd090 2024-03-19T02:04:32.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.807+0000 7fefc97fa640 1 -- 172.21.15.55:0/3311620042 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefc00052d0 con 0x7fefcc0cd090 2024-03-19T02:04:32.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.811+0000 7fefd2ca1640 1 -- 172.21.15.55:0/3311620042 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefcc0c3570 con 0x7fefcc0cd090 2024-03-19T02:04:32.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.811+0000 7fefc97fa640 1 -- 172.21.15.55:0/3311620042 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc0003840 con 0x7fefcc0cd090 2024-03-19T02:04:32.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.811+0000 7fefd2ca1640 1 -- 172.21.15.55:0/3311620042 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefcc13bd80 con 0x7fefcc0cd090 2024-03-19T02:04:32.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:32.811+0000 7fefd2ca1640 1 -- 172.21.15.55:0/3311620042 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefcc068a40 con 0x7fefcc0cd090 2024-03-19T02:04:33.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:33.087+0000 7fefc97fa640 1 -- 172.21.15.55:0/3311620042 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fefc001a070 con 0x7fefcc0cd090 2024-03-19T02:04:33.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:33.087+0000 7fefc97fa640 1 --2- 172.21.15.55:0/3311620042 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fefb0067690 0x7fefb0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:33.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:33.087+0000 7fefcbfff640 1 --2- 172.21.15.55:0/3311620042 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fefb0067690 0x7fefb0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:33.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:33.087+0000 7fefcbfff640 1 --2- 172.21.15.55:0/3311620042 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fefb0067690 0x7fefb0069b50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fefcc14ba30 tx=0x7fefb8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:33.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:33.111+0000 7fefc97fa640 1 -- 172.21.15.55:0/3311620042 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fefc0053400 con 0x7fefcc0cd090 2024-03-19T02:04:33.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:33.111+0000 7fefc97fa640 1 -- 172.21.15.55:0/3311620042 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fefc0021080 con 0x7fefcc0cd090 2024-03-19T02:04:33.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:33.331+0000 7fefd2ca1640 1 -- 172.21.15.55:0/3311620042 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fefcc1418d0 con 0x7fefcc0cd090 2024-03-19T02:04:33.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:33.431+0000 7fefc97fa640 1 -- 172.21.15.55:0/3311620042 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fefc0005480 con 0x7fefcc0cd090 2024-03-19T02:04:33.430 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:33.430 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:33.430 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:33.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:33.435+0000 7fefd2ca1640 1 -- 172.21.15.55:0/3311620042 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fefb0067690 msgr2=0x7fefb0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:33.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:33.435+0000 7fefd2ca1640 1 --2- 172.21.15.55:0/3311620042 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fefb0067690 0x7fefb0069b50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fefcc14ba30 tx=0x7fefb8009040 comp rx=0 tx=0).stop 2024-03-19T02:04:33.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:33.435+0000 7fefd2ca1640 1 -- 172.21.15.55:0/3311620042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefcc0cd090 msgr2=0x7fefcc13b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:33.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:33.435+0000 7fefd2ca1640 1 --2- 172.21.15.55:0/3311620042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefcc0cd090 0x7fefcc13b840 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fefc000c3c0 tx=0x7fefc000c880 comp rx=0 tx=0).stop 2024-03-19T02:04:33.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:33.435+0000 7fefd2ca1640 1 -- 172.21.15.55:0/3311620042 shutdown_connections 2024-03-19T02:04:33.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:33.435+0000 7fefd2ca1640 1 --2- 172.21.15.55:0/3311620042 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fefb0067690 0x7fefb0069b50 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:33.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:33.435+0000 7fefd2ca1640 1 --2- 172.21.15.55:0/3311620042 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fefcc14ec60 0x7fefcc0c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:33.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:33.435+0000 7fefd2ca1640 1 --2- 172.21.15.55:0/3311620042 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fefcc0cd470 0x7fefcc0c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:33.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:33.435+0000 7fefd2ca1640 1 --2- 172.21.15.55:0/3311620042 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefcc0cd090 0x7fefcc13b840 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:33.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:33.435+0000 7fefd2ca1640 1 -- 172.21.15.55:0/3311620042 >> 172.21.15.55:0/3311620042 conn(0x7fefcc06e670 msgr2=0x7fefcc14d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:33.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:33.435+0000 7fefd2ca1640 1 -- 172.21.15.55:0/3311620042 shutdown_connections 2024-03-19T02:04:33.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:33.435+0000 7fefd2ca1640 1 -- 172.21.15.55:0/3311620042 wait complete. 2024-03-19T02:04:34.052 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:34.052 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:35.054 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:35.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.167+0000 7f877e7e5640 1 Processor -- start 2024-03-19T02:04:35.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.167+0000 7f877e7e5640 1 -- start start 2024-03-19T02:04:35.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.167+0000 7f877e7e5640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f8778154060 0x7f877806bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:35.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.167+0000 7f877e7e5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f87780ca0c0 0x7f877815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:35.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.167+0000 7f877e7e5640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87780ccd80 con 0x7f8778070410 2024-03-19T02:04:35.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.167+0000 7f877e7e5640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f87780ccb50 con 0x7f87780ca0c0 2024-03-19T02:04:35.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.167+0000 7f877e7e5640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f87780cd040 con 0x7f8778154060 2024-03-19T02:04:35.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.171+0000 7f87777fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f87780ca0c0 0x7f877815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:35.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.171+0000 7f8777fff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f8778154060 0x7f877806bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:35.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.171+0000 7f87777fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f87780ca0c0 0x7f877815a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:37012/0 (socket says 172.21.15.55:37012) 2024-03-19T02:04:35.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.171+0000 7f87777fe640 1 -- 172.21.15.55:0/2847844356 learned_addr learned my addr 172.21.15.55:0/2847844356 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:35.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.175+0000 7f87777fe640 1 -- 172.21.15.55:0/2847844356 >> v2:172.21.15.84:3300/0 conn(0x7f8778154060 msgr2=0x7f877806bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:35.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.175+0000 7f87777fe640 1 --2- 172.21.15.55:0/2847844356 >> v2:172.21.15.84:3300/0 conn(0x7f8778154060 0x7f877806bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:35.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.175+0000 7f87777fe640 1 -- 172.21.15.55:0/2847844356 >> v1:172.21.15.84:6789/0 conn(0x7f8778070410 legacy=0x7f87780c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:35.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.175+0000 7f87777fe640 1 -- 172.21.15.55:0/2847844356 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87780cd320 con 0x7f87780ca0c0 2024-03-19T02:04:35.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.175+0000 7f87777fe640 1 --2- 172.21.15.55:0/2847844356 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87780ca0c0 0x7f877815a9b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f87780ca050 tx=0x7f876c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aea713c1fb8c961 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:35.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.183+0000 7f8776ffd640 1 -- 172.21.15.55:0/2847844356 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c0b5070 con 0x7f87780ca0c0 2024-03-19T02:04:35.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.183+0000 7f8776ffd640 1 -- 172.21.15.55:0/2847844356 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f876c0a8a10 con 0x7f87780ca0c0 2024-03-19T02:04:35.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.183+0000 7f8776ffd640 1 -- 172.21.15.55:0/2847844356 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c0a8de0 con 0x7f87780ca0c0 2024-03-19T02:04:35.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.183+0000 7f877e7e5640 1 -- 172.21.15.55:0/2847844356 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87780ca0c0 msgr2=0x7f877815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:35.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.183+0000 7f877e7e5640 1 --2- 172.21.15.55:0/2847844356 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87780ca0c0 0x7f877815a9b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f87780ca050 tx=0x7f876c01ed10 comp rx=0 tx=0).stop 2024-03-19T02:04:35.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.183+0000 7f877e7e5640 1 -- 172.21.15.55:0/2847844356 shutdown_connections 2024-03-19T02:04:35.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.183+0000 7f877e7e5640 1 --2- 172.21.15.55:0/2847844356 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87780ca0c0 0x7f877815a9b0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:35.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.183+0000 7f877e7e5640 1 --2- 172.21.15.55:0/2847844356 >> v2:172.21.15.84:3300/0 conn(0x7f8778154060 0x7f877806bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:35.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.183+0000 7f877e7e5640 1 -- 172.21.15.55:0/2847844356 >> 172.21.15.55:0/2847844356 conn(0x7f877806c3b0 msgr2=0x7f877806c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:35.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.183+0000 7f877e7e5640 1 -- 172.21.15.55:0/2847844356 shutdown_connections 2024-03-19T02:04:35.182 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.183+0000 7f877e7e5640 1 -- 172.21.15.55:0/2847844356 wait complete. 2024-03-19T02:04:35.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.183+0000 7f877e7e5640 1 Processor -- start 2024-03-19T02:04:35.183 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.183+0000 7f877e7e5640 1 -- start start 2024-03-19T02:04:35.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.187+0000 7f877e7e5640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8778070410 0x7f87780cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:35.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.187+0000 7f877e7e5640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f87780ca0c0 0x7f877815c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:35.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.187+0000 7f877e7e5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8778154060 0x7f87781f0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:35.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.187+0000 7f877e7e5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f877815a490 con 0x7f8778154060 2024-03-19T02:04:35.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.187+0000 7f877e7e5640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f87780cd8e0 con 0x7f8778070410 2024-03-19T02:04:35.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.187+0000 7f877e7e5640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f877815a140 con 0x7f87780ca0c0 2024-03-19T02:04:35.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.187+0000 7f877cd82640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8778154060 0x7f87781f0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:35.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.187+0000 7f8777fff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8778070410 0x7f87780cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:35.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.187+0000 7f877cd82640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8778154060 0x7f87781f0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:37018/0 (socket says 172.21.15.55:37018) 2024-03-19T02:04:35.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.187+0000 7f877cd82640 1 -- 172.21.15.55:0/1332800696 learned_addr learned my addr 172.21.15.55:0/1332800696 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:35.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.187+0000 7f87777fe640 1 --2- 172.21.15.55:0/1332800696 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f87780ca0c0 0x7f877815c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:35.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.191+0000 7f877cd82640 1 -- 172.21.15.55:0/1332800696 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f87780ca0c0 msgr2=0x7f877815c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:35.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.191+0000 7f877cd82640 1 --2- 172.21.15.55:0/1332800696 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f87780ca0c0 0x7f877815c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:35.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.191+0000 7f877cd82640 1 -- 172.21.15.55:0/1332800696 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8778070410 msgr2=0x7f87780cb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:35.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.191+0000 7f877cd82640 1 --2- 172.21.15.55:0/1332800696 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8778070410 0x7f87780cb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:35.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.191+0000 7f877cd82640 1 -- 172.21.15.55:0/1332800696 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87780cbe80 con 0x7f8778154060 2024-03-19T02:04:35.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.195+0000 7f877cd82640 1 --2- 172.21.15.55:0/1332800696 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8778154060 0x7f87781f0760 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f877000c730 tx=0x7f877000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:35.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.199+0000 7f8774ff9640 1 -- 172.21.15.55:0/1332800696 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8770003b80 con 0x7f8778154060 2024-03-19T02:04:35.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.199+0000 7f8774ff9640 1 -- 172.21.15.55:0/1332800696 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8770003d20 con 0x7f8778154060 2024-03-19T02:04:35.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.199+0000 7f8774ff9640 1 -- 172.21.15.55:0/1332800696 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877000aa30 con 0x7f8778154060 2024-03-19T02:04:35.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.199+0000 7f877e7e5640 1 -- 172.21.15.55:0/1332800696 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87781f3a20 con 0x7f8778154060 2024-03-19T02:04:35.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.199+0000 7f877e7e5640 1 -- 172.21.15.55:0/1332800696 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87781f3fe0 con 0x7f8778154060 2024-03-19T02:04:35.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.199+0000 7f877e7e5640 1 -- 172.21.15.55:0/1332800696 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8778068a40 con 0x7f8778154060 2024-03-19T02:04:35.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.231+0000 7f8774ff9640 1 -- 172.21.15.55:0/1332800696 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f877000abd0 con 0x7f8778154060 2024-03-19T02:04:35.232 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.235+0000 7f8774ff9640 1 --2- 172.21.15.55:0/1332800696 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f87640676b0 0x7f8764069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:35.232 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.235+0000 7f8777fff640 1 --2- 172.21.15.55:0/1332800696 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f87640676b0 0x7f8764069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:35.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.235+0000 7f8777fff640 1 --2- 172.21.15.55:0/1332800696 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f87640676b0 0x7f8764069b70 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8768009ac0 tx=0x7f8768009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:35.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.239+0000 7f8774ff9640 1 -- 172.21.15.55:0/1332800696 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f8770054730 con 0x7f8778154060 2024-03-19T02:04:35.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.523+0000 7f8774ff9640 1 -- 172.21.15.55:0/1332800696 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f87780cbe80 con 0x7f8778154060 2024-03-19T02:04:35.729 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.731+0000 7f877e7e5640 1 -- 172.21.15.55:0/1332800696 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8778159c40 con 0x7f8778154060 2024-03-19T02:04:35.752 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.755+0000 7f8774ff9640 1 -- 172.21.15.55:0/1332800696 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f8770056690 con 0x7f8778154060 2024-03-19T02:04:35.752 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:35.753 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:35.753 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:35.755 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.755+0000 7f877e7e5640 1 -- 172.21.15.55:0/1332800696 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f87640676b0 msgr2=0x7f8764069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:35.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.755+0000 7f877e7e5640 1 --2- 172.21.15.55:0/1332800696 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f87640676b0 0x7f8764069b70 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8768009ac0 tx=0x7f8768009190 comp rx=0 tx=0).stop 2024-03-19T02:04:35.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.755+0000 7f877e7e5640 1 -- 172.21.15.55:0/1332800696 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8778154060 msgr2=0x7f87781f0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:35.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.755+0000 7f877e7e5640 1 --2- 172.21.15.55:0/1332800696 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8778154060 0x7f87781f0760 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f877000c730 tx=0x7f877000cbf0 comp rx=0 tx=0).stop 2024-03-19T02:04:35.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.755+0000 7f877e7e5640 1 -- 172.21.15.55:0/1332800696 shutdown_connections 2024-03-19T02:04:35.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.755+0000 7f877e7e5640 1 --2- 172.21.15.55:0/1332800696 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f87640676b0 0x7f8764069b70 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:35.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.755+0000 7f877e7e5640 1 --2- 172.21.15.55:0/1332800696 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8778154060 0x7f87781f0760 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:35.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.755+0000 7f877e7e5640 1 --2- 172.21.15.55:0/1332800696 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f87780ca0c0 0x7f877815c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:35.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.755+0000 7f877e7e5640 1 --2- 172.21.15.55:0/1332800696 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8778070410 0x7f87780cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:35.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.755+0000 7f877e7e5640 1 -- 172.21.15.55:0/1332800696 >> 172.21.15.55:0/1332800696 conn(0x7f877806c3b0 msgr2=0x7f87780c7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:35.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.755+0000 7f877e7e5640 1 -- 172.21.15.55:0/1332800696 shutdown_connections 2024-03-19T02:04:35.756 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:35.755+0000 7f877e7e5640 1 -- 172.21.15.55:0/1332800696 wait complete. 2024-03-19T02:04:36.376 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:36.377 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:37.377 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:37.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.499+0000 7feacfc7c640 1 Processor -- start 2024-03-19T02:04:37.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.499+0000 7feacfc7c640 1 -- start start 2024-03-19T02:04:37.499 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.499+0000 7feacfc7c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7feac8152120 0x7feac80c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:37.499 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.499+0000 7feacfc7c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7feac80c9610 0x7feac8157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:37.499 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.499+0000 7feacfc7c640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7feac8070780 0x7feac8158510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:37.499 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.499+0000 7feacfc7c640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7feac8152500 con 0x7feac8152120 2024-03-19T02:04:37.499 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.499+0000 7feacfc7c640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7feac8070590 con 0x7feac8070780 2024-03-19T02:04:37.499 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.499+0000 7feacfc7c640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7feac80c3530 con 0x7feac80c9610 2024-03-19T02:04:37.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.499+0000 7feacda18640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7feac8152120 0x7feac80c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:37.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.499+0000 7feacd217640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7feac80c9610 0x7feac8157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:37.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.499+0000 7feace219640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7feac8070780 0x7feac8158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:37.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.499+0000 7feacda18640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7feac8152120 0x7feac80c90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:37034/0 (socket says 172.21.15.55:37034) 2024-03-19T02:04:37.500 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.503+0000 7feacda18640 1 -- 172.21.15.55:0/3582695167 learned_addr learned my addr 172.21.15.55:0/3582695167 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:37.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.503+0000 7feacda18640 1 -- 172.21.15.55:0/3582695167 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7feac80c9610 msgr2=0x7feac8157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:37.502 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.503+0000 7feacda18640 1 --2- 172.21.15.55:0/3582695167 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7feac80c9610 0x7feac8157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:37.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.503+0000 7feacda18640 1 -- 172.21.15.55:0/3582695167 >> v2:172.21.15.84:3300/0 conn(0x7feac8070780 msgr2=0x7feac8158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:37.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.503+0000 7feacda18640 1 --2- 172.21.15.55:0/3582695167 >> v2:172.21.15.84:3300/0 conn(0x7feac8070780 0x7feac8158510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:37.503 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.503+0000 7feacda18640 1 -- 172.21.15.55:0/3582695167 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feac806fff0 con 0x7feac8152120 2024-03-19T02:04:37.505 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.507+0000 7feacda18640 1 --2- 172.21.15.55:0/3582695167 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feac8152120 0x7feac80c90d0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7feac8071f20 tx=0x7feac401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=429032ae879f3cd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:37.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.511+0000 7feacca16640 1 -- 172.21.15.55:0/3582695167 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac40b0070 con 0x7feac8152120 2024-03-19T02:04:37.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.511+0000 7feacca16640 1 -- 172.21.15.55:0/3582695167 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feac40ab600 con 0x7feac8152120 2024-03-19T02:04:37.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.511+0000 7feacca16640 1 -- 172.21.15.55:0/3582695167 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac40ab9d0 con 0x7feac8152120 2024-03-19T02:04:37.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.515+0000 7feacfc7c640 1 -- 172.21.15.55:0/3582695167 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feac8152120 msgr2=0x7feac80c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:37.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.515+0000 7feacfc7c640 1 --2- 172.21.15.55:0/3582695167 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feac8152120 0x7feac80c90d0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7feac8071f20 tx=0x7feac401c520 comp rx=0 tx=0).stop 2024-03-19T02:04:37.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.515+0000 7feacfc7c640 1 -- 172.21.15.55:0/3582695167 shutdown_connections 2024-03-19T02:04:37.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.515+0000 7feacfc7c640 1 --2- 172.21.15.55:0/3582695167 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7feac80c9610 0x7feac8157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:37.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.515+0000 7feacfc7c640 1 --2- 172.21.15.55:0/3582695167 >> v2:172.21.15.84:3300/0 conn(0x7feac8070780 0x7feac8158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:37.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.515+0000 7feacfc7c640 1 --2- 172.21.15.55:0/3582695167 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feac8152120 0x7feac80c90d0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:37.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.515+0000 7feacfc7c640 1 -- 172.21.15.55:0/3582695167 >> 172.21.15.55:0/3582695167 conn(0x7feac8067600 msgr2=0x7feac8071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:37.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.515+0000 7feacfc7c640 1 -- 172.21.15.55:0/3582695167 shutdown_connections 2024-03-19T02:04:37.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.515+0000 7feacfc7c640 1 -- 172.21.15.55:0/3582695167 wait complete. 2024-03-19T02:04:37.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.515+0000 7feacfc7c640 1 Processor -- start 2024-03-19T02:04:37.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.515+0000 7feacfc7c640 1 -- start start 2024-03-19T02:04:37.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.515+0000 7feacfc7c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feac8070780 0x7feac8148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:37.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.515+0000 7feacfc7c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7feac80c9610 0x7feac81f2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:37.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.515+0000 7feacfc7c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feac8152120 0x7feac80cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:37.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.515+0000 7feacfc7c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7feac80c31c0 con 0x7feac8152120 2024-03-19T02:04:37.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.515+0000 7feacfc7c640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7feac80c2ee0 con 0x7feac8070780 2024-03-19T02:04:37.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.515+0000 7feacfc7c640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7feac80c86d0 con 0x7feac80c9610 2024-03-19T02:04:37.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.515+0000 7feace219640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feac8152120 0x7feac80cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:37.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.515+0000 7feacd217640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7feac80c9610 0x7feac81f2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:37.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.515+0000 7feace219640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feac8152120 0x7feac80cddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:37036/0 (socket says 172.21.15.55:37036) 2024-03-19T02:04:37.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.515+0000 7feace219640 1 -- 172.21.15.55:0/1499772784 learned_addr learned my addr 172.21.15.55:0/1499772784 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:37.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.519+0000 7feacda18640 1 --2- 172.21.15.55:0/1499772784 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feac8070780 0x7feac8148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:37.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.519+0000 7feace219640 1 -- 172.21.15.55:0/1499772784 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7feac80c9610 msgr2=0x7feac81f2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:37.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.519+0000 7feace219640 1 --2- 172.21.15.55:0/1499772784 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7feac80c9610 0x7feac81f2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:37.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.519+0000 7feace219640 1 -- 172.21.15.55:0/1499772784 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feac8070780 msgr2=0x7feac8148eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:37.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.519+0000 7feace219640 1 --2- 172.21.15.55:0/1499772784 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feac8070780 0x7feac8148eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:37.520 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.519+0000 7feace219640 1 -- 172.21.15.55:0/1499772784 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feac80cd4a0 con 0x7feac8152120 2024-03-19T02:04:37.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.523+0000 7feace219640 1 --2- 172.21.15.55:0/1499772784 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feac8152120 0x7feac80cddb0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7feac000efc0 tx=0x7feac000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:37.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.527+0000 7feabe7fc640 1 -- 172.21.15.55:0/1499772784 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac0018020 con 0x7feac8152120 2024-03-19T02:04:37.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.527+0000 7feabe7fc640 1 -- 172.21.15.55:0/1499772784 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feac0014480 con 0x7feac8152120 2024-03-19T02:04:37.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.527+0000 7feacfc7c640 1 -- 172.21.15.55:0/1499772784 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feac80cc440 con 0x7feac8152120 2024-03-19T02:04:37.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.527+0000 7feabe7fc640 1 -- 172.21.15.55:0/1499772784 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac0014870 con 0x7feac8152120 2024-03-19T02:04:37.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.527+0000 7feacfc7c640 1 -- 172.21.15.55:0/1499772784 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feac80b09f0 con 0x7feac8152120 2024-03-19T02:04:37.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.531+0000 7feacfc7c640 1 -- 172.21.15.55:0/1499772784 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feac8068a40 con 0x7feac8152120 2024-03-19T02:04:37.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.555+0000 7feabe7fc640 1 -- 172.21.15.55:0/1499772784 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7feac001b020 con 0x7feac8152120 2024-03-19T02:04:37.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.555+0000 7feabe7fc640 1 --2- 172.21.15.55:0/1499772784 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7feaac0676b0 0x7feaac069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:37.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.555+0000 7feacda18640 1 --2- 172.21.15.55:0/1499772784 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7feaac0676b0 0x7feaac069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:37.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.555+0000 7feacda18640 1 --2- 172.21.15.55:0/1499772784 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7feaac0676b0 0x7feaac069b70 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7feac40ad000 tx=0x7feac4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:37.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.855+0000 7feabe7fc640 1 -- 172.21.15.55:0/1499772784 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7feac0055100 con 0x7feac8152120 2024-03-19T02:04:37.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:37.855+0000 7feabe7fc640 1 -- 172.21.15.55:0/1499772784 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7feac0022070 con 0x7feac8152120 2024-03-19T02:04:38.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:38.083+0000 7feacfc7c640 1 -- 172.21.15.55:0/1499772784 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feac80c84b0 con 0x7feac8152120 2024-03-19T02:04:38.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:38.103+0000 7feabe7fc640 1 -- 172.21.15.55:0/1499772784 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7feac0010040 con 0x7feac8152120 2024-03-19T02:04:38.103 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:38.103 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:38.103 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:38.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:38.107+0000 7feacfc7c640 1 -- 172.21.15.55:0/1499772784 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7feaac0676b0 msgr2=0x7feaac069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:38.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:38.107+0000 7feacfc7c640 1 --2- 172.21.15.55:0/1499772784 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7feaac0676b0 0x7feaac069b70 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7feac40ad000 tx=0x7feac4008040 comp rx=0 tx=0).stop 2024-03-19T02:04:38.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:38.107+0000 7feacfc7c640 1 -- 172.21.15.55:0/1499772784 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feac8152120 msgr2=0x7feac80cddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:38.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:38.107+0000 7feacfc7c640 1 --2- 172.21.15.55:0/1499772784 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feac8152120 0x7feac80cddb0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7feac000efc0 tx=0x7feac000c490 comp rx=0 tx=0).stop 2024-03-19T02:04:38.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:38.107+0000 7feacfc7c640 1 -- 172.21.15.55:0/1499772784 shutdown_connections 2024-03-19T02:04:38.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:38.107+0000 7feacfc7c640 1 --2- 172.21.15.55:0/1499772784 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7feaac0676b0 0x7feaac069b70 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:38.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:38.107+0000 7feacfc7c640 1 --2- 172.21.15.55:0/1499772784 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feac8152120 0x7feac80cddb0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:38.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:38.107+0000 7feacfc7c640 1 --2- 172.21.15.55:0/1499772784 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7feac80c9610 0x7feac81f2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:38.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:38.107+0000 7feacfc7c640 1 --2- 172.21.15.55:0/1499772784 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feac8070780 0x7feac8148eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:38.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:38.107+0000 7feacfc7c640 1 -- 172.21.15.55:0/1499772784 >> 172.21.15.55:0/1499772784 conn(0x7feac8067600 msgr2=0x7feac8157fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:38.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:38.107+0000 7feacfc7c640 1 -- 172.21.15.55:0/1499772784 shutdown_connections 2024-03-19T02:04:38.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:38.107+0000 7feacfc7c640 1 -- 172.21.15.55:0/1499772784 wait complete. 2024-03-19T02:04:38.727 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:38.727 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:39.728 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:39.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.843+0000 7f56f8c0e640 1 Processor -- start 2024-03-19T02:04:39.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.843+0000 7f56f8c0e640 1 -- start start 2024-03-19T02:04:39.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.843+0000 7f56f8c0e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56f4154290 0x7f56f415e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:39.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.843+0000 7f56f8c0e640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f56f40ca310 0x7f56f4154670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:39.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.843+0000 7f56f8c0e640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f56f415ad40 0x7f56f4072900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:39.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.843+0000 7f56f8c0e640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f56f415b2c0 con 0x7f56f40ca310 2024-03-19T02:04:39.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.843+0000 7f56f8c0e640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f56f406c1d0 con 0x7f56f415ad40 2024-03-19T02:04:39.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.843+0000 7f56f8c0e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f56f415e030 con 0x7f56f4154290 2024-03-19T02:04:39.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.843+0000 7f56f1d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f56f40ca310 0x7f56f4154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:39.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.843+0000 7f56f259c640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56f4154290 0x7f56f415e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:39.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.843+0000 7f56f1d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f56f40ca310 0x7f56f4154670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:45832/0 (socket says 172.21.15.55:45832) 2024-03-19T02:04:39.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.843+0000 7f56f1d9b640 1 -- 172.21.15.55:0/1673584779 learned_addr learned my addr 172.21.15.55:0/1673584779 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:39.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.843+0000 7f56f2d9d640 1 --2- 172.21.15.55:0/1673584779 >> v2:172.21.15.84:3300/0 conn(0x7f56f415ad40 0x7f56f4072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:39.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.847+0000 7f56f1d9b640 1 -- 172.21.15.55:0/1673584779 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56f4154290 msgr2=0x7f56f415e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:39.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.847+0000 7f56f1d9b640 1 --2- 172.21.15.55:0/1673584779 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56f4154290 0x7f56f415e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:39.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.847+0000 7f56f1d9b640 1 -- 172.21.15.55:0/1673584779 >> v2:172.21.15.84:3300/0 conn(0x7f56f415ad40 msgr2=0x7f56f4072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:39.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.847+0000 7f56f1d9b640 1 --2- 172.21.15.55:0/1673584779 >> v2:172.21.15.84:3300/0 conn(0x7f56f415ad40 0x7f56f4072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:39.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.847+0000 7f56f1d9b640 1 -- 172.21.15.55:0/1673584779 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f406bc90 con 0x7f56f40ca310 2024-03-19T02:04:39.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.851+0000 7f56f1d9b640 1 --2- 172.21.15.55:0/1673584779 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56f40ca310 0x7f56f4154670 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f56f406ff90 tx=0x7f56e801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b30b011062d5e85 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:39.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.855+0000 7f56f159a640 1 -- 172.21.15.55:0/1673584779 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e80aa660 con 0x7f56f40ca310 2024-03-19T02:04:39.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.855+0000 7f56f159a640 1 -- 172.21.15.55:0/1673584779 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56e80af070 con 0x7f56f40ca310 2024-03-19T02:04:39.854 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.855+0000 7f56f159a640 1 -- 172.21.15.55:0/1673584779 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e80aaa20 con 0x7f56f40ca310 2024-03-19T02:04:39.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.855+0000 7f56f8c0e640 1 -- 172.21.15.55:0/1673584779 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56f40ca310 msgr2=0x7f56f4154670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:39.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.855+0000 7f56f8c0e640 1 --2- 172.21.15.55:0/1673584779 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56f40ca310 0x7f56f4154670 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f56f406ff90 tx=0x7f56e801c520 comp rx=0 tx=0).stop 2024-03-19T02:04:39.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.855+0000 7f56f8c0e640 1 -- 172.21.15.55:0/1673584779 shutdown_connections 2024-03-19T02:04:39.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.855+0000 7f56f8c0e640 1 --2- 172.21.15.55:0/1673584779 >> v2:172.21.15.84:3300/0 conn(0x7f56f415ad40 0x7f56f4072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:39.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.855+0000 7f56f8c0e640 1 --2- 172.21.15.55:0/1673584779 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56f40ca310 0x7f56f4154670 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:39.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.855+0000 7f56f8c0e640 1 --2- 172.21.15.55:0/1673584779 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56f4154290 0x7f56f415e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:39.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.855+0000 7f56f8c0e640 1 -- 172.21.15.55:0/1673584779 >> 172.21.15.55:0/1673584779 conn(0x7f56f406c580 msgr2=0x7f56f406c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:39.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.855+0000 7f56f8c0e640 1 -- 172.21.15.55:0/1673584779 shutdown_connections 2024-03-19T02:04:39.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.855+0000 7f56f8c0e640 1 -- 172.21.15.55:0/1673584779 wait complete. 2024-03-19T02:04:39.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.859+0000 7f56f8c0e640 1 Processor -- start 2024-03-19T02:04:39.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.859+0000 7f56f8c0e640 1 -- start start 2024-03-19T02:04:39.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.859+0000 7f56f8c0e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f56f40ca310 0x7f56f40cd310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:39.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.859+0000 7f56f8c0e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56f4154290 0x7f56f414f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:39.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.859+0000 7f56f8c0e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56f415ad40 0x7f56f414fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:39.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.859+0000 7f56f8c0e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f56f40ccb50 con 0x7f56f415ad40 2024-03-19T02:04:39.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.859+0000 7f56f8c0e640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f56f406bf30 con 0x7f56f40ca310 2024-03-19T02:04:39.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.859+0000 7f56f8c0e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f56f40cccd0 con 0x7f56f4154290 2024-03-19T02:04:39.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.859+0000 7f56f2d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56f415ad40 0x7f56f414fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:39.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.859+0000 7f56f259c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f56f40ca310 0x7f56f40cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:39.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.859+0000 7f56f2d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56f415ad40 0x7f56f414fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:45834/0 (socket says 172.21.15.55:45834) 2024-03-19T02:04:39.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.859+0000 7f56f2d9d640 1 -- 172.21.15.55:0/3389854410 learned_addr learned my addr 172.21.15.55:0/3389854410 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:39.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.863+0000 7f56f2d9d640 1 -- 172.21.15.55:0/3389854410 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56f4154290 msgr2=0x7f56f414f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:39.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.863+0000 7f56f2d9d640 1 --2- 172.21.15.55:0/3389854410 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56f4154290 0x7f56f414f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:39.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.863+0000 7f56f2d9d640 1 -- 172.21.15.55:0/3389854410 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f56f40ca310 msgr2=0x7f56f40cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:39.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.863+0000 7f56f2d9d640 1 --2- 172.21.15.55:0/3389854410 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f56f40ca310 0x7f56f40cd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:39.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.863+0000 7f56f2d9d640 1 -- 172.21.15.55:0/3389854410 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f4155290 con 0x7f56f415ad40 2024-03-19T02:04:39.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.867+0000 7f56f2d9d640 1 --2- 172.21.15.55:0/3389854410 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56f415ad40 0x7f56f414fb40 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f56ec00efc0 tx=0x7f56ec00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:39.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.871+0000 7f56e2ffd640 1 -- 172.21.15.55:0/3389854410 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec018020 con 0x7f56f415ad40 2024-03-19T02:04:39.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.871+0000 7f56e2ffd640 1 -- 172.21.15.55:0/3389854410 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56ec014480 con 0x7f56f415ad40 2024-03-19T02:04:39.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.871+0000 7f56f8c0e640 1 -- 172.21.15.55:0/3389854410 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56f4155570 con 0x7f56f415ad40 2024-03-19T02:04:39.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.871+0000 7f56e2ffd640 1 -- 172.21.15.55:0/3389854410 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec014870 con 0x7f56f415ad40 2024-03-19T02:04:39.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.871+0000 7f56f8c0e640 1 -- 172.21.15.55:0/3389854410 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56f4155b30 con 0x7f56f415ad40 2024-03-19T02:04:39.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.871+0000 7f56f8c0e640 1 -- 172.21.15.55:0/3389854410 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f4068a40 con 0x7f56f415ad40 2024-03-19T02:04:39.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.895+0000 7f56e2ffd640 1 -- 172.21.15.55:0/3389854410 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f56ec01b020 con 0x7f56f415ad40 2024-03-19T02:04:39.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.895+0000 7f56e2ffd640 1 --2- 172.21.15.55:0/3389854410 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f56d40676b0 0x7f56d4069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:39.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.895+0000 7f56f259c640 1 --2- 172.21.15.55:0/3389854410 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f56d40676b0 0x7f56d4069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:39.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:39.895+0000 7f56f259c640 1 --2- 172.21.15.55:0/3389854410 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f56d40676b0 0x7f56d4069b70 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f56e4009ae0 tx=0x7f56e4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:40.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:40.215+0000 7f56e2ffd640 1 -- 172.21.15.55:0/3389854410 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f56ec055140 con 0x7f56f415ad40 2024-03-19T02:04:40.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:40.215+0000 7f56e2ffd640 1 -- 172.21.15.55:0/3389854410 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f56ec022070 con 0x7f56f415ad40 2024-03-19T02:04:40.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:40.427+0000 7f56f8c0e640 1 -- 172.21.15.55:0/3389854410 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56f4073650 con 0x7f56f415ad40 2024-03-19T02:04:40.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:40.447+0000 7f56e2ffd640 1 -- 172.21.15.55:0/3389854410 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f56ec010040 con 0x7f56f415ad40 2024-03-19T02:04:40.447 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:40.448 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:40.448 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:40.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:40.451+0000 7f56f8c0e640 1 -- 172.21.15.55:0/3389854410 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f56d40676b0 msgr2=0x7f56d4069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:40.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:40.451+0000 7f56f8c0e640 1 --2- 172.21.15.55:0/3389854410 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f56d40676b0 0x7f56d4069b70 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f56e4009ae0 tx=0x7f56e4009190 comp rx=0 tx=0).stop 2024-03-19T02:04:40.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:40.451+0000 7f56f8c0e640 1 -- 172.21.15.55:0/3389854410 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56f415ad40 msgr2=0x7f56f414fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:40.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:40.451+0000 7f56f8c0e640 1 --2- 172.21.15.55:0/3389854410 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56f415ad40 0x7f56f414fb40 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f56ec00efc0 tx=0x7f56ec00c490 comp rx=0 tx=0).stop 2024-03-19T02:04:40.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:40.451+0000 7f56f8c0e640 1 -- 172.21.15.55:0/3389854410 shutdown_connections 2024-03-19T02:04:40.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:40.451+0000 7f56f8c0e640 1 --2- 172.21.15.55:0/3389854410 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f56d40676b0 0x7f56d4069b70 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:40.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:40.451+0000 7f56f8c0e640 1 --2- 172.21.15.55:0/3389854410 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56f415ad40 0x7f56f414fb40 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:40.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:40.451+0000 7f56f8c0e640 1 --2- 172.21.15.55:0/3389854410 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56f4154290 0x7f56f414f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:40.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:40.451+0000 7f56f8c0e640 1 --2- 172.21.15.55:0/3389854410 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f56f40ca310 0x7f56f40cd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:40.450 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:40.451+0000 7f56f8c0e640 1 -- 172.21.15.55:0/3389854410 >> 172.21.15.55:0/3389854410 conn(0x7f56f406c580 msgr2=0x7f56f41538d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:40.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:40.451+0000 7f56f8c0e640 1 -- 172.21.15.55:0/3389854410 shutdown_connections 2024-03-19T02:04:40.451 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:40.451+0000 7f56f8c0e640 1 -- 172.21.15.55:0/3389854410 wait complete. 2024-03-19T02:04:41.027 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:04:41.030 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:04:41.069 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:41.069 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:42.070 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:42.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.207+0000 7fea2fb79640 1 Processor -- start 2024-03-19T02:04:42.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.207+0000 7fea2fb79640 1 -- start start 2024-03-19T02:04:42.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.207+0000 7fea2fb79640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fea28152120 0x7fea280c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:42.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.207+0000 7fea2fb79640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fea28070780 0x7fea281586e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:42.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.207+0000 7fea2fb79640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea2806fff0 con 0x7fea280c9610 2024-03-19T02:04:42.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.207+0000 7fea2fb79640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fea28152500 con 0x7fea28152120 2024-03-19T02:04:42.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.207+0000 7fea2fb79640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fea28070590 con 0x7fea28070780 2024-03-19T02:04:42.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.207+0000 7fea2d915640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fea28152120 0x7fea280c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:42.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.211+0000 7fea2d915640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fea28152120 0x7fea280c90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:45840/0 (socket says 172.21.15.55:45840) 2024-03-19T02:04:42.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.211+0000 7fea2d915640 1 -- 172.21.15.55:0/4197005791 learned_addr learned my addr 172.21.15.55:0/4197005791 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:42.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.211+0000 7fea2e116640 1 --2- 172.21.15.55:0/4197005791 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fea28070780 0x7fea281586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:42.210 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.211+0000 7fea2d915640 1 -- 172.21.15.55:0/4197005791 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fea28070780 msgr2=0x7fea281586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:42.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.211+0000 7fea2d915640 1 --2- 172.21.15.55:0/4197005791 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fea28070780 0x7fea281586e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:42.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.211+0000 7fea2d915640 1 -- 172.21.15.55:0/4197005791 >> v1:172.21.15.84:6789/0 conn(0x7fea280c9610 legacy=0x7fea28157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:42.211 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.211+0000 7fea2d915640 1 -- 172.21.15.55:0/4197005791 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea280c39c0 con 0x7fea28152120 2024-03-19T02:04:42.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.215+0000 7fea2d915640 1 --2- 172.21.15.55:0/4197005791 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea28152120 0x7fea280c90d0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fea24009870 tx=0x7fea2401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5273706f67e8e9e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:42.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.219+0000 7fea2c913640 1 -- 172.21.15.55:0/4197005791 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea240b2070 con 0x7fea28152120 2024-03-19T02:04:42.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.219+0000 7fea2c913640 1 -- 172.21.15.55:0/4197005791 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea240a9070 con 0x7fea28152120 2024-03-19T02:04:42.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.219+0000 7fea2c913640 1 -- 172.21.15.55:0/4197005791 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea240ad880 con 0x7fea28152120 2024-03-19T02:04:42.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.219+0000 7fea2fb79640 1 -- 172.21.15.55:0/4197005791 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea28152120 msgr2=0x7fea280c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:42.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.219+0000 7fea2fb79640 1 --2- 172.21.15.55:0/4197005791 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea28152120 0x7fea280c90d0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fea24009870 tx=0x7fea2401c560 comp rx=0 tx=0).stop 2024-03-19T02:04:42.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.219+0000 7fea2fb79640 1 -- 172.21.15.55:0/4197005791 shutdown_connections 2024-03-19T02:04:42.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.219+0000 7fea2fb79640 1 --2- 172.21.15.55:0/4197005791 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fea28070780 0x7fea281586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:42.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.219+0000 7fea2fb79640 1 --2- 172.21.15.55:0/4197005791 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea28152120 0x7fea280c90d0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:42.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.219+0000 7fea2fb79640 1 -- 172.21.15.55:0/4197005791 >> 172.21.15.55:0/4197005791 conn(0x7fea28067600 msgr2=0x7fea28071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:42.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.219+0000 7fea2fb79640 1 -- 172.21.15.55:0/4197005791 shutdown_connections 2024-03-19T02:04:42.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.219+0000 7fea2fb79640 1 -- 172.21.15.55:0/4197005791 wait complete. 2024-03-19T02:04:42.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.223+0000 7fea2fb79640 1 Processor -- start 2024-03-19T02:04:42.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.223+0000 7fea2fb79640 1 -- start start 2024-03-19T02:04:42.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.223+0000 7fea2fb79640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea28070780 0x7fea28150730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:42.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.223+0000 7fea2fb79640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea280c9610 0x7fea28141660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:42.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.223+0000 7fea2fb79640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fea28152120 0x7fea28141ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:42.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.223+0000 7fea2fb79640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fea280c86d0 con 0x7fea280c9610 2024-03-19T02:04:42.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.223+0000 7fea2fb79640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fea280c31c0 con 0x7fea28070780 2024-03-19T02:04:42.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.223+0000 7fea2fb79640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fea28165170 con 0x7fea28152120 2024-03-19T02:04:42.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.223+0000 7fea2d114640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea280c9610 0x7fea28141660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:42.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.223+0000 7fea2d915640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea28070780 0x7fea28150730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:42.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.223+0000 7fea2d114640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea280c9610 0x7fea28141660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:45854/0 (socket says 172.21.15.55:45854) 2024-03-19T02:04:42.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.223+0000 7fea2d114640 1 -- 172.21.15.55:0/1398692066 learned_addr learned my addr 172.21.15.55:0/1398692066 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:42.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.223+0000 7fea2e116640 1 --2- 172.21.15.55:0/1398692066 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fea28152120 0x7fea28141ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:42.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.227+0000 7fea2d114640 1 -- 172.21.15.55:0/1398692066 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fea28152120 msgr2=0x7fea28141ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:42.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.227+0000 7fea2d114640 1 --2- 172.21.15.55:0/1398692066 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fea28152120 0x7fea28141ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:42.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.227+0000 7fea2d114640 1 -- 172.21.15.55:0/1398692066 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea28070780 msgr2=0x7fea28150730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:42.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.227+0000 7fea2d114640 1 --2- 172.21.15.55:0/1398692066 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea28070780 0x7fea28150730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:42.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.227+0000 7fea2d114640 1 -- 172.21.15.55:0/1398692066 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea28141180 con 0x7fea280c9610 2024-03-19T02:04:42.229 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.231+0000 7fea2d114640 1 --2- 172.21.15.55:0/1398692066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea280c9610 0x7fea28141660 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fea1800c460 tx=0x7fea1800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:42.232 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.235+0000 7fea1e7fc640 1 -- 172.21.15.55:0/1398692066 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea18016020 con 0x7fea280c9610 2024-03-19T02:04:42.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.235+0000 7fea1e7fc640 1 -- 172.21.15.55:0/1398692066 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea180052d0 con 0x7fea280c9610 2024-03-19T02:04:42.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.235+0000 7fea2fb79640 1 -- 172.21.15.55:0/1398692066 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea28142700 con 0x7fea280c9610 2024-03-19T02:04:42.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.235+0000 7fea1e7fc640 1 -- 172.21.15.55:0/1398692066 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea18007260 con 0x7fea280c9610 2024-03-19T02:04:42.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.235+0000 7fea2fb79640 1 -- 172.21.15.55:0/1398692066 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea28142cc0 con 0x7fea280c9610 2024-03-19T02:04:42.233 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.235+0000 7fea2fb79640 1 -- 172.21.15.55:0/1398692066 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea28068a40 con 0x7fea280c9610 2024-03-19T02:04:42.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.511+0000 7fea1e7fc640 1 -- 172.21.15.55:0/1398692066 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fea18007400 con 0x7fea280c9610 2024-03-19T02:04:42.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.511+0000 7fea1e7fc640 1 --2- 172.21.15.55:0/1398692066 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fea0c067690 0x7fea0c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:42.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.511+0000 7fea2d915640 1 --2- 172.21.15.55:0/1398692066 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fea0c067690 0x7fea0c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:42.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.511+0000 7fea2d915640 1 --2- 172.21.15.55:0/1398692066 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fea0c067690 0x7fea0c069b50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fea240af000 tx=0x7fea240a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:42.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.555+0000 7fea1e7fc640 1 -- 172.21.15.55:0/1398692066 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fea180545f0 con 0x7fea280c9610 2024-03-19T02:04:42.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.559+0000 7fea1e7fc640 1 -- 172.21.15.55:0/1398692066 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fea18020020 con 0x7fea280c9610 2024-03-19T02:04:42.778 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.779+0000 7fea2fb79640 1 -- 172.21.15.55:0/1398692066 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea28141350 con 0x7fea280c9610 2024-03-19T02:04:42.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.807+0000 7fea1e7fc640 1 -- 172.21.15.55:0/1398692066 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fea18005480 con 0x7fea280c9610 2024-03-19T02:04:42.809 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:42.809 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:42.809 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:42.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.811+0000 7fea2fb79640 1 -- 172.21.15.55:0/1398692066 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fea0c067690 msgr2=0x7fea0c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:42.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.811+0000 7fea2fb79640 1 --2- 172.21.15.55:0/1398692066 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fea0c067690 0x7fea0c069b50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fea240af000 tx=0x7fea240a7000 comp rx=0 tx=0).stop 2024-03-19T02:04:42.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.811+0000 7fea2fb79640 1 -- 172.21.15.55:0/1398692066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea280c9610 msgr2=0x7fea28141660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:42.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.811+0000 7fea2fb79640 1 --2- 172.21.15.55:0/1398692066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea280c9610 0x7fea28141660 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fea1800c460 tx=0x7fea1800c920 comp rx=0 tx=0).stop 2024-03-19T02:04:42.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.811+0000 7fea2fb79640 1 -- 172.21.15.55:0/1398692066 shutdown_connections 2024-03-19T02:04:42.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.811+0000 7fea2fb79640 1 --2- 172.21.15.55:0/1398692066 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fea0c067690 0x7fea0c069b50 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:42.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.811+0000 7fea2fb79640 1 --2- 172.21.15.55:0/1398692066 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fea28152120 0x7fea28141ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:42.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.811+0000 7fea2fb79640 1 --2- 172.21.15.55:0/1398692066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea280c9610 0x7fea28141660 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:42.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.811+0000 7fea2fb79640 1 --2- 172.21.15.55:0/1398692066 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea28070780 0x7fea28150730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:42.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.811+0000 7fea2fb79640 1 -- 172.21.15.55:0/1398692066 >> 172.21.15.55:0/1398692066 conn(0x7fea28067600 msgr2=0x7fea280c9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:42.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.811+0000 7fea2fb79640 1 -- 172.21.15.55:0/1398692066 shutdown_connections 2024-03-19T02:04:42.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:42.811+0000 7fea2fb79640 1 -- 172.21.15.55:0/1398692066 wait complete. 2024-03-19T02:04:43.432 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:43.432 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:44.433 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:44.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.547+0000 7fc3a6db9640 1 Processor -- start 2024-03-19T02:04:44.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.547+0000 7fc3a6db9640 1 -- start start 2024-03-19T02:04:44.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.547+0000 7fc3a6db9640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc3a00c9a90 0x7fc3a0070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:44.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.547+0000 7fc3a6db9640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3a015e3a0 con 0x7fc3a01540c0 2024-03-19T02:04:44.547 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.547+0000 7fc3a6db9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3a015e680 con 0x7fc3a00c96b0 2024-03-19T02:04:44.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.547+0000 7fc3a6db9640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fc3a00c7ad0 con 0x7fc3a00c9a90 2024-03-19T02:04:44.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.551+0000 7fc39ffff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc3a00c9a90 0x7fc3a0070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:44.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.551+0000 7fc3a4b55640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fc3a00c96b0 0x7fc3a00c73c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:57830/0 (socket says 172.21.15.55:57830) 2024-03-19T02:04:44.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.551+0000 7fc3a4b55640 1 -- 172.21.15.55:0/2709564259 learned_addr learned my addr 172.21.15.55:0/2709564259 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:04:44.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.555+0000 7fc39f7fe640 1 -- 172.21.15.55:0/2709564259 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2979436361 0 0) 0x7fc3a015e680 con 0x7fc3a00c96b0 2024-03-19T02:04:44.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.555+0000 7fc39f7fe640 1 -- 172.21.15.55:0/2709564259 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc390003530 con 0x7fc3a00c96b0 2024-03-19T02:04:44.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.555+0000 7fc39ffff640 1 -- 172.21.15.55:0/2709564259 >> v1:172.21.15.84:6789/0 conn(0x7fc3a00c96b0 legacy=0x7fc3a00c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:44.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.555+0000 7fc39ffff640 1 -- 172.21.15.55:0/2709564259 >> v1:172.21.15.55:6789/0 conn(0x7fc3a01540c0 legacy=0x7fc3a015d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:44.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.555+0000 7fc39ffff640 1 -- 172.21.15.55:0/2709564259 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a015a330 con 0x7fc3a00c9a90 2024-03-19T02:04:44.561 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.563+0000 7fc39ffff640 1 --2- 172.21.15.55:0/2709564259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3a00c9a90 0x7fc3a0070f20 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc38c0097c0 tx=0x7fc38c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7781e26b0269f6c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:44.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.567+0000 7fc39f7fe640 1 -- 172.21.15.55:0/2709564259 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c0b0070 con 0x7fc3a00c9a90 2024-03-19T02:04:44.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.567+0000 7fc39f7fe640 1 -- 172.21.15.55:0/2709564259 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc38c0ab600 con 0x7fc3a00c9a90 2024-03-19T02:04:44.566 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.567+0000 7fc39f7fe640 1 -- 172.21.15.55:0/2709564259 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c0ab9d0 con 0x7fc3a00c9a90 2024-03-19T02:04:44.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.567+0000 7fc3a6db9640 1 -- 172.21.15.55:0/2709564259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3a00c9a90 msgr2=0x7fc3a0070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:44.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.567+0000 7fc3a6db9640 1 --2- 172.21.15.55:0/2709564259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3a00c9a90 0x7fc3a0070f20 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc38c0097c0 tx=0x7fc38c01c410 comp rx=0 tx=0).stop 2024-03-19T02:04:44.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.567+0000 7fc3a6db9640 1 -- 172.21.15.55:0/2709564259 shutdown_connections 2024-03-19T02:04:44.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.567+0000 7fc3a6db9640 1 --2- 172.21.15.55:0/2709564259 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3a00c9a90 0x7fc3a0070f20 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:44.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.567+0000 7fc3a6db9640 1 -- 172.21.15.55:0/2709564259 >> 172.21.15.55:0/2709564259 conn(0x7fc3a00cc9c0 msgr2=0x7fc3a00ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:44.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.567+0000 7fc3a6db9640 1 -- 172.21.15.55:0/2709564259 shutdown_connections 2024-03-19T02:04:44.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.567+0000 7fc3a6db9640 1 -- 172.21.15.55:0/2709564259 wait complete. 2024-03-19T02:04:44.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.571+0000 7fc3a6db9640 1 Processor -- start 2024-03-19T02:04:44.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.571+0000 7fc3a6db9640 1 -- start start 2024-03-19T02:04:44.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.571+0000 7fc3a6db9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3a00c96b0 0x7fc3a0152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:44.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.571+0000 7fc3a6db9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3a00c9a90 0x7fc3a0144fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:44.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.571+0000 7fc3a6db9640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3a01540c0 0x7fc3a0145500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:44.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.571+0000 7fc3a6db9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a01548e0 con 0x7fc3a00c96b0 2024-03-19T02:04:44.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.571+0000 7fc3a6db9640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a015b0f0 con 0x7fc3a01540c0 2024-03-19T02:04:44.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.571+0000 7fc3a6db9640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc3a015df20 con 0x7fc3a00c9a90 2024-03-19T02:04:44.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.571+0000 7fc3a4b55640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3a00c96b0 0x7fc3a0152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:44.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.571+0000 7fc39ffff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3a00c9a90 0x7fc3a0144fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:44.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.571+0000 7fc3a4b55640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3a00c96b0 0x7fc3a0152e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:45866/0 (socket says 172.21.15.55:45866) 2024-03-19T02:04:44.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.571+0000 7fc3a4b55640 1 -- 172.21.15.55:0/2950187778 learned_addr learned my addr 172.21.15.55:0/2950187778 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:44.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.571+0000 7fc3a5356640 1 --2- 172.21.15.55:0/2950187778 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3a01540c0 0x7fc3a0145500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:44.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.575+0000 7fc3a4b55640 1 -- 172.21.15.55:0/2950187778 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3a00c9a90 msgr2=0x7fc3a0144fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:44.573 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.575+0000 7fc3a4b55640 1 --2- 172.21.15.55:0/2950187778 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3a00c9a90 0x7fc3a0144fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:44.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.575+0000 7fc3a4b55640 1 -- 172.21.15.55:0/2950187778 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3a01540c0 msgr2=0x7fc3a0145500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:44.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.575+0000 7fc3a4b55640 1 --2- 172.21.15.55:0/2950187778 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3a01540c0 0x7fc3a0145500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:44.574 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.575+0000 7fc3a4b55640 1 -- 172.21.15.55:0/2950187778 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a0148ab0 con 0x7fc3a00c96b0 2024-03-19T02:04:44.576 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.579+0000 7fc3a4b55640 1 --2- 172.21.15.55:0/2950187778 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3a00c96b0 0x7fc3a0152e00 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc39400c420 tx=0x7fc39400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:44.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.583+0000 7fc39d7fa640 1 -- 172.21.15.55:0/2950187778 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc394017070 con 0x7fc3a00c96b0 2024-03-19T02:04:44.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.583+0000 7fc39d7fa640 1 -- 172.21.15.55:0/2950187778 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc394005cf0 con 0x7fc3a00c96b0 2024-03-19T02:04:44.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.583+0000 7fc39d7fa640 1 -- 172.21.15.55:0/2950187778 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3940072d0 con 0x7fc3a00c96b0 2024-03-19T02:04:44.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.583+0000 7fc3a6db9640 1 -- 172.21.15.55:0/2950187778 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3a0148d50 con 0x7fc3a00c96b0 2024-03-19T02:04:44.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.583+0000 7fc3a6db9640 1 -- 172.21.15.55:0/2950187778 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3a0149310 con 0x7fc3a00c96b0 2024-03-19T02:04:44.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.583+0000 7fc3a6db9640 1 -- 172.21.15.55:0/2950187778 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3a0068a40 con 0x7fc3a00c96b0 2024-03-19T02:04:44.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.871+0000 7fc39d7fa640 1 -- 172.21.15.55:0/2950187778 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc39401a070 con 0x7fc3a00c96b0 2024-03-19T02:04:44.871 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.871+0000 7fc39d7fa640 1 --2- 172.21.15.55:0/2950187778 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3840676b0 0x7fc384069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:44.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.875+0000 7fc39ffff640 1 --2- 172.21.15.55:0/2950187778 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3840676b0 0x7fc384069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:44.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.875+0000 7fc39ffff640 1 --2- 172.21.15.55:0/2950187778 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3840676b0 0x7fc384069b70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc38c0ad000 tx=0x7fc38c0a4f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:44.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.895+0000 7fc39d7fa640 1 -- 172.21.15.55:0/2950187778 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fc394053770 con 0x7fc3a00c96b0 2024-03-19T02:04:44.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:44.895+0000 7fc39d7fa640 1 -- 172.21.15.55:0/2950187778 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc3940556d0 con 0x7fc3a00c96b0 2024-03-19T02:04:45.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:45.107+0000 7fc3a6db9640 1 -- 172.21.15.55:0/2950187778 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3a0149550 con 0x7fc3a00c96b0 2024-03-19T02:04:45.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:45.123+0000 7fc39d7fa640 1 -- 172.21.15.55:0/2950187778 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fc394059770 con 0x7fc3a00c96b0 2024-03-19T02:04:45.124 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:45.124 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:45.124 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:45.127+0000 7fc3a6db9640 1 -- 172.21.15.55:0/2950187778 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3840676b0 msgr2=0x7fc384069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:45.127+0000 7fc3a6db9640 1 --2- 172.21.15.55:0/2950187778 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3840676b0 0x7fc384069b70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc38c0ad000 tx=0x7fc38c0a4f90 comp rx=0 tx=0).stop 2024-03-19T02:04:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:45.127+0000 7fc3a6db9640 1 -- 172.21.15.55:0/2950187778 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3a00c96b0 msgr2=0x7fc3a0152e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:45.127+0000 7fc3a6db9640 1 --2- 172.21.15.55:0/2950187778 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3a00c96b0 0x7fc3a0152e00 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc39400c420 tx=0x7fc39400c8e0 comp rx=0 tx=0).stop 2024-03-19T02:04:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:45.127+0000 7fc3a6db9640 1 -- 172.21.15.55:0/2950187778 shutdown_connections 2024-03-19T02:04:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:45.127+0000 7fc3a6db9640 1 --2- 172.21.15.55:0/2950187778 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc3840676b0 0x7fc384069b70 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:45.127+0000 7fc3a6db9640 1 --2- 172.21.15.55:0/2950187778 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3a01540c0 0x7fc3a0145500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:45.127+0000 7fc3a6db9640 1 --2- 172.21.15.55:0/2950187778 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc3a00c9a90 0x7fc3a0144fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:45.127+0000 7fc3a6db9640 1 --2- 172.21.15.55:0/2950187778 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc3a00c96b0 0x7fc3a0152e00 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:45.127+0000 7fc3a6db9640 1 -- 172.21.15.55:0/2950187778 >> 172.21.15.55:0/2950187778 conn(0x7fc3a00cc9c0 msgr2=0x7fc3a014a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:45.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:45.127+0000 7fc3a6db9640 1 -- 172.21.15.55:0/2950187778 shutdown_connections 2024-03-19T02:04:45.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:45.127+0000 7fc3a6db9640 1 -- 172.21.15.55:0/2950187778 wait complete. 2024-03-19T02:04:45.751 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:45.751 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:46.752 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:46.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.867+0000 7fbbe7dd5640 1 Processor -- start 2024-03-19T02:04:46.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.867+0000 7fbbe7dd5640 1 -- start start 2024-03-19T02:04:46.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.867+0000 7fbbe7dd5640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbe0154340 0x7fbbe015e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:46.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.867+0000 7fbbe7dd5640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbe0071280 con 0x7fbbe0154720 2024-03-19T02:04:46.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.867+0000 7fbbe7dd5640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbe015d900 con 0x7fbbe00c73c0 2024-03-19T02:04:46.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.867+0000 7fbbe7dd5640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbbe0073840 con 0x7fbbe0154340 2024-03-19T02:04:46.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.871+0000 7fbbe5370640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fbbe0154720 0x7fbbe0072e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:43716/0 (socket says 172.21.15.55:43716) 2024-03-19T02:04:46.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.871+0000 7fbbe5370640 1 -- 172.21.15.55:0/1758537432 learned_addr learned my addr 172.21.15.55:0/1758537432 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:04:46.868 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.871+0000 7fbbe5b71640 1 --2- 172.21.15.55:0/1758537432 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbe0154340 0x7fbbe015e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:46.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.875+0000 7fbbe4b6f640 1 -- 172.21.15.55:0/1758537432 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 789097197 0 0) 0x7fbbe0071280 con 0x7fbbe0154720 2024-03-19T02:04:46.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.875+0000 7fbbe4b6f640 1 -- 172.21.15.55:0/1758537432 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbcc003530 con 0x7fbbe0154720 2024-03-19T02:04:46.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.875+0000 7fbbe5b71640 1 -- 172.21.15.55:0/1758537432 >> v1:172.21.15.84:6789/0 conn(0x7fbbe00c73c0 legacy=0x7fbbe00c91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:46.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.875+0000 7fbbe5b71640 1 -- 172.21.15.55:0/1758537432 >> v1:172.21.15.55:6789/0 conn(0x7fbbe0154720 legacy=0x7fbbe0072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:46.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.875+0000 7fbbe5b71640 1 -- 172.21.15.55:0/1758537432 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbe0070cc0 con 0x7fbbe0154340 2024-03-19T02:04:46.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.879+0000 7fbbe5b71640 1 --2- 172.21.15.55:0/1758537432 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbe0154340 0x7fbbe015e3a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0097f0 tx=0x7fbbdc01c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ced72d20664e118d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:46.881 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.883+0000 7fbbe4b6f640 1 -- 172.21.15.55:0/1758537432 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc0b0070 con 0x7fbbe0154340 2024-03-19T02:04:46.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.887+0000 7fbbe4b6f640 1 -- 172.21.15.55:0/1758537432 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbdc0ab600 con 0x7fbbe0154340 2024-03-19T02:04:46.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.887+0000 7fbbe4b6f640 1 -- 172.21.15.55:0/1758537432 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc0ab9b0 con 0x7fbbe0154340 2024-03-19T02:04:46.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.887+0000 7fbbe7dd5640 1 -- 172.21.15.55:0/1758537432 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbe0154340 msgr2=0x7fbbe015e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:46.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.887+0000 7fbbe7dd5640 1 --2- 172.21.15.55:0/1758537432 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbe0154340 0x7fbbe015e3a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0097f0 tx=0x7fbbdc01c4e0 comp rx=0 tx=0).stop 2024-03-19T02:04:46.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.887+0000 7fbbe7dd5640 1 -- 172.21.15.55:0/1758537432 shutdown_connections 2024-03-19T02:04:46.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.887+0000 7fbbe7dd5640 1 --2- 172.21.15.55:0/1758537432 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbe0154340 0x7fbbe015e3a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:46.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.887+0000 7fbbe7dd5640 1 -- 172.21.15.55:0/1758537432 >> 172.21.15.55:0/1758537432 conn(0x7fbbe00cc9c0 msgr2=0x7fbbe00ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:46.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.887+0000 7fbbe7dd5640 1 -- 172.21.15.55:0/1758537432 shutdown_connections 2024-03-19T02:04:46.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.887+0000 7fbbe7dd5640 1 -- 172.21.15.55:0/1758537432 wait complete. 2024-03-19T02:04:46.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.887+0000 7fbbe7dd5640 1 Processor -- start 2024-03-19T02:04:46.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.887+0000 7fbbe7dd5640 1 -- start start 2024-03-19T02:04:46.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.887+0000 7fbbe7dd5640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbbe00c73c0 0x7fbbe01f3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:46.887 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.887+0000 7fbbe7dd5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbe0154340 0x7fbbe01599e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:46.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.887+0000 7fbbe7dd5640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbe0154720 0x7fbbe0156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:46.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.887+0000 7fbbe7dd5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe0168390 con 0x7fbbe0154340 2024-03-19T02:04:46.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.887+0000 7fbbe7dd5640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe015b0b0 con 0x7fbbe00c73c0 2024-03-19T02:04:46.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.887+0000 7fbbe7dd5640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fbbe0168510 con 0x7fbbe0154720 2024-03-19T02:04:46.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.891+0000 7fbbe6372640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbe0154720 0x7fbbe0156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:46.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.891+0000 7fbbe5370640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbe0154340 0x7fbbe01599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:46.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.891+0000 7fbbe5370640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbe0154340 0x7fbbe01599e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:45872/0 (socket says 172.21.15.55:45872) 2024-03-19T02:04:46.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.891+0000 7fbbe5370640 1 -- 172.21.15.55:0/2930787775 learned_addr learned my addr 172.21.15.55:0/2930787775 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:46.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.891+0000 7fbbe5b71640 1 --2- 172.21.15.55:0/2930787775 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbbe00c73c0 0x7fbbe01f3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:46.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.895+0000 7fbbe5370640 1 -- 172.21.15.55:0/2930787775 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbe0154720 msgr2=0x7fbbe0156540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:46.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.895+0000 7fbbe5370640 1 --2- 172.21.15.55:0/2930787775 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbe0154720 0x7fbbe0156540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:46.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.895+0000 7fbbe5370640 1 -- 172.21.15.55:0/2930787775 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbbe00c73c0 msgr2=0x7fbbe01f3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:46.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.895+0000 7fbbe5370640 1 --2- 172.21.15.55:0/2930787775 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbbe00c73c0 0x7fbbe01f3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:46.892 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.895+0000 7fbbe5370640 1 -- 172.21.15.55:0/2930787775 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbe0155d60 con 0x7fbbe0154340 2024-03-19T02:04:46.896 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.899+0000 7fbbe5370640 1 --2- 172.21.15.55:0/2930787775 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbe0154340 0x7fbbe01599e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fbbd000c450 tx=0x7fbbd000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:46.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.903+0000 7fbbd67fc640 1 -- 172.21.15.55:0/2930787775 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0017070 con 0x7fbbe0154340 2024-03-19T02:04:46.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.903+0000 7fbbd67fc640 1 -- 172.21.15.55:0/2930787775 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbd0005cf0 con 0x7fbbe0154340 2024-03-19T02:04:46.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.903+0000 7fbbe7dd5640 1 -- 172.21.15.55:0/2930787775 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbe0157410 con 0x7fbbe0154340 2024-03-19T02:04:46.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.903+0000 7fbbd67fc640 1 -- 172.21.15.55:0/2930787775 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0007270 con 0x7fbbe0154340 2024-03-19T02:04:46.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.903+0000 7fbbe7dd5640 1 -- 172.21.15.55:0/2930787775 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbe01579d0 con 0x7fbbe0154340 2024-03-19T02:04:46.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.903+0000 7fbbe7dd5640 1 -- 172.21.15.55:0/2930787775 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbe0068a40 con 0x7fbbe0154340 2024-03-19T02:04:46.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.923+0000 7fbbd67fc640 1 -- 172.21.15.55:0/2930787775 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fbbd001a070 con 0x7fbbe0154340 2024-03-19T02:04:46.922 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.923+0000 7fbbd67fc640 1 --2- 172.21.15.55:0/2930787775 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbbc40676e0 0x7fbbc4069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:46.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.923+0000 7fbbe5b71640 1 --2- 172.21.15.55:0/2930787775 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbbc40676e0 0x7fbbc4069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:46.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.923+0000 7fbbe5b71640 1 --2- 172.21.15.55:0/2930787775 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbbc40676e0 0x7fbbc4069ba0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0ad000 tx=0x7fbbdc0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:46.925 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:46.927+0000 7fbbd67fc640 1 -- 172.21.15.55:0/2930787775 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fbbd0053730 con 0x7fbbe0154340 2024-03-19T02:04:47.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:47.203+0000 7fbbd67fc640 1 -- 172.21.15.55:0/2930787775 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fbbd0055690 con 0x7fbbe0154340 2024-03-19T02:04:47.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:47.407+0000 7fbbe7dd5640 1 -- 172.21.15.55:0/2930787775 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbbe015a780 con 0x7fbbe0154340 2024-03-19T02:04:47.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:47.431+0000 7fbbd67fc640 1 -- 172.21.15.55:0/2930787775 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fbbd0059730 con 0x7fbbe0154340 2024-03-19T02:04:47.432 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:47.432 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:47.432 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:47.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:47.435+0000 7fbbe7dd5640 1 -- 172.21.15.55:0/2930787775 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbbc40676e0 msgr2=0x7fbbc4069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:47.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:47.435+0000 7fbbe7dd5640 1 --2- 172.21.15.55:0/2930787775 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbbc40676e0 0x7fbbc4069ba0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0ad000 tx=0x7fbbdc0a6000 comp rx=0 tx=0).stop 2024-03-19T02:04:47.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:47.435+0000 7fbbe7dd5640 1 -- 172.21.15.55:0/2930787775 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbe0154340 msgr2=0x7fbbe01599e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:47.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:47.435+0000 7fbbe7dd5640 1 --2- 172.21.15.55:0/2930787775 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbe0154340 0x7fbbe01599e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fbbd000c450 tx=0x7fbbd000c910 comp rx=0 tx=0).stop 2024-03-19T02:04:47.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:47.435+0000 7fbbe7dd5640 1 -- 172.21.15.55:0/2930787775 shutdown_connections 2024-03-19T02:04:47.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:47.435+0000 7fbbe7dd5640 1 --2- 172.21.15.55:0/2930787775 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fbbc40676e0 0x7fbbc4069ba0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:47.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:47.435+0000 7fbbe7dd5640 1 --2- 172.21.15.55:0/2930787775 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fbbe0154720 0x7fbbe0156540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:47.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:47.435+0000 7fbbe7dd5640 1 --2- 172.21.15.55:0/2930787775 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbe0154340 0x7fbbe01599e0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:47.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:47.435+0000 7fbbe7dd5640 1 --2- 172.21.15.55:0/2930787775 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbbe00c73c0 0x7fbbe01f3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:47.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:47.435+0000 7fbbe7dd5640 1 -- 172.21.15.55:0/2930787775 >> 172.21.15.55:0/2930787775 conn(0x7fbbe00cc9c0 msgr2=0x7fbbe00cb450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:47.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:47.435+0000 7fbbe7dd5640 1 -- 172.21.15.55:0/2930787775 shutdown_connections 2024-03-19T02:04:47.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:47.435+0000 7fbbe7dd5640 1 -- 172.21.15.55:0/2930787775 wait complete. 2024-03-19T02:04:48.061 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:48.061 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:49.063 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:49.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.179+0000 7fcc225a3640 1 Processor -- start 2024-03-19T02:04:49.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.179+0000 7fcc225a3640 1 -- start start 2024-03-19T02:04:49.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.179+0000 7fcc225a3640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fcc1c0c96b0 0x7fcc1c070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:49.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.179+0000 7fcc225a3640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc1c0c9a90 0x7fcc1c0c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:49.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.179+0000 7fcc225a3640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc1c15e3a0 con 0x7fcc1c1540c0 2024-03-19T02:04:49.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.179+0000 7fcc225a3640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fcc1c15e680 con 0x7fcc1c0c96b0 2024-03-19T02:04:49.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.179+0000 7fcc225a3640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fcc1c15a330 con 0x7fcc1c0c9a90 2024-03-19T02:04:49.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.183+0000 7fcc1b7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc1c0c9a90 0x7fcc1c0c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:49.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.183+0000 7fcc1bfff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fcc1c0c96b0 0x7fcc1c070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:49.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.183+0000 7fcc1b7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc1c0c9a90 0x7fcc1c0c73c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:47706/0 (socket says 172.21.15.55:47706) 2024-03-19T02:04:49.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.183+0000 7fcc1b7fe640 1 -- 172.21.15.55:0/745926068 learned_addr learned my addr 172.21.15.55:0/745926068 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:49.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.187+0000 7fcc1b7fe640 1 -- 172.21.15.55:0/745926068 >> v2:172.21.15.84:3300/0 conn(0x7fcc1c0c96b0 msgr2=0x7fcc1c070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:49.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.187+0000 7fcc1b7fe640 1 --2- 172.21.15.55:0/745926068 >> v2:172.21.15.84:3300/0 conn(0x7fcc1c0c96b0 0x7fcc1c070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:49.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.187+0000 7fcc1b7fe640 1 -- 172.21.15.55:0/745926068 >> v1:172.21.15.84:6789/0 conn(0x7fcc1c1540c0 legacy=0x7fcc1c15d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:49.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.187+0000 7fcc1b7fe640 1 -- 172.21.15.55:0/745926068 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1c15a610 con 0x7fcc1c0c9a90 2024-03-19T02:04:49.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.191+0000 7fcc1b7fe640 1 --2- 172.21.15.55:0/745926068 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc1c0c9a90 0x7fcc1c0c73c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009870 tx=0x7fcc1001c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e209f6e5703c4730 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:49.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.195+0000 7fcc1affd640 1 -- 172.21.15.55:0/745926068 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc100b2070 con 0x7fcc1c0c9a90 2024-03-19T02:04:49.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.195+0000 7fcc1affd640 1 -- 172.21.15.55:0/745926068 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc100a9070 con 0x7fcc1c0c9a90 2024-03-19T02:04:49.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.195+0000 7fcc1affd640 1 -- 172.21.15.55:0/745926068 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc100ad880 con 0x7fcc1c0c9a90 2024-03-19T02:04:49.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.195+0000 7fcc225a3640 1 -- 172.21.15.55:0/745926068 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc1c0c9a90 msgr2=0x7fcc1c0c73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:49.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.195+0000 7fcc225a3640 1 --2- 172.21.15.55:0/745926068 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc1c0c9a90 0x7fcc1c0c73c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009870 tx=0x7fcc1001c490 comp rx=0 tx=0).stop 2024-03-19T02:04:49.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.199+0000 7fcc225a3640 1 -- 172.21.15.55:0/745926068 shutdown_connections 2024-03-19T02:04:49.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.199+0000 7fcc225a3640 1 --2- 172.21.15.55:0/745926068 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc1c0c9a90 0x7fcc1c0c73c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:49.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.199+0000 7fcc225a3640 1 --2- 172.21.15.55:0/745926068 >> v2:172.21.15.84:3300/0 conn(0x7fcc1c0c96b0 0x7fcc1c070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:49.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.199+0000 7fcc225a3640 1 -- 172.21.15.55:0/745926068 >> 172.21.15.55:0/745926068 conn(0x7fcc1c0cc9c0 msgr2=0x7fcc1c0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:49.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.199+0000 7fcc225a3640 1 -- 172.21.15.55:0/745926068 shutdown_connections 2024-03-19T02:04:49.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.199+0000 7fcc225a3640 1 -- 172.21.15.55:0/745926068 wait complete. 2024-03-19T02:04:49.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.199+0000 7fcc225a3640 1 Processor -- start 2024-03-19T02:04:49.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.199+0000 7fcc225a3640 1 -- start start 2024-03-19T02:04:49.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.199+0000 7fcc225a3640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc1c0c96b0 0x7fcc1c145850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:49.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.199+0000 7fcc225a3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc1c0c9a90 0x7fcc1c146f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:49.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.199+0000 7fcc225a3640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc1c1540c0 0x7fcc1c1474c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:49.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.199+0000 7fcc225a3640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1c0c7900 con 0x7fcc1c0c9a90 2024-03-19T02:04:49.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.199+0000 7fcc225a3640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1c15b0f0 con 0x7fcc1c1540c0 2024-03-19T02:04:49.199 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.199+0000 7fcc225a3640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fcc1c0c7a80 con 0x7fcc1c0c96b0 2024-03-19T02:04:49.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.203+0000 7fcc1b7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc1c0c9a90 0x7fcc1c146f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:49.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.203+0000 7fcc1bfff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc1c0c96b0 0x7fcc1c145850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:49.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.203+0000 7fcc20b40640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc1c1540c0 0x7fcc1c1474c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:49.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.203+0000 7fcc1b7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc1c0c9a90 0x7fcc1c146f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:35232/0 (socket says 172.21.15.55:35232) 2024-03-19T02:04:49.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.203+0000 7fcc1b7fe640 1 -- 172.21.15.55:0/931156697 learned_addr learned my addr 172.21.15.55:0/931156697 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:49.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.207+0000 7fcc1b7fe640 1 -- 172.21.15.55:0/931156697 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc1c0c96b0 msgr2=0x7fcc1c145850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:49.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.207+0000 7fcc1b7fe640 1 --2- 172.21.15.55:0/931156697 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc1c0c96b0 0x7fcc1c145850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:49.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.207+0000 7fcc1b7fe640 1 -- 172.21.15.55:0/931156697 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc1c1540c0 msgr2=0x7fcc1c1474c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:49.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.207+0000 7fcc1b7fe640 1 --2- 172.21.15.55:0/931156697 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc1c1540c0 0x7fcc1c1474c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:49.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.207+0000 7fcc1b7fe640 1 -- 172.21.15.55:0/931156697 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1c147ab0 con 0x7fcc1c0c9a90 2024-03-19T02:04:49.209 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.211+0000 7fcc1b7fe640 1 --2- 172.21.15.55:0/931156697 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc1c0c9a90 0x7fcc1c146f80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fcc100b0040 tx=0x7fcc10019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:49.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.215+0000 7fcc18ff9640 1 -- 172.21.15.55:0/931156697 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc100bb070 con 0x7fcc1c0c9a90 2024-03-19T02:04:49.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.215+0000 7fcc18ff9640 1 -- 172.21.15.55:0/931156697 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc100add40 con 0x7fcc1c0c9a90 2024-03-19T02:04:49.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.215+0000 7fcc225a3640 1 -- 172.21.15.55:0/931156697 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc1c143200 con 0x7fcc1c0c9a90 2024-03-19T02:04:49.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.215+0000 7fcc18ff9640 1 -- 172.21.15.55:0/931156697 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc100b2040 con 0x7fcc1c0c9a90 2024-03-19T02:04:49.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.215+0000 7fcc225a3640 1 -- 172.21.15.55:0/931156697 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc1c1437c0 con 0x7fcc1c0c9a90 2024-03-19T02:04:49.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.215+0000 7fcc225a3640 1 -- 172.21.15.55:0/931156697 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc1c068a40 con 0x7fcc1c0c9a90 2024-03-19T02:04:49.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.235+0000 7fcc18ff9640 1 -- 172.21.15.55:0/931156697 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fcc100be070 con 0x7fcc1c0c9a90 2024-03-19T02:04:49.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.235+0000 7fcc18ff9640 1 --2- 172.21.15.55:0/931156697 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcc08067690 0x7fcc08069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:49.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.239+0000 7fcc1bfff640 1 --2- 172.21.15.55:0/931156697 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcc08067690 0x7fcc08069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:49.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.239+0000 7fcc1bfff640 1 --2- 172.21.15.55:0/931156697 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcc08067690 0x7fcc08069b50 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009ac0 tx=0x7fcc0c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:49.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.563+0000 7fcc18ff9640 1 -- 172.21.15.55:0/931156697 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fcc100ecf40 con 0x7fcc1c0c9a90 2024-03-19T02:04:49.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.563+0000 7fcc18ff9640 1 -- 172.21.15.55:0/931156697 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fcc10016110 con 0x7fcc1c0c9a90 2024-03-19T02:04:49.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.783+0000 7fcc225a3640 1 -- 172.21.15.55:0/931156697 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc1c152110 con 0x7fcc1c0c9a90 2024-03-19T02:04:49.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.819+0000 7fcc18ff9640 1 -- 172.21.15.55:0/931156697 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fcc100eeea0 con 0x7fcc1c0c9a90 2024-03-19T02:04:49.818 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:49.818 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:49.819 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:49.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.823+0000 7fcc225a3640 1 -- 172.21.15.55:0/931156697 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcc08067690 msgr2=0x7fcc08069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:49.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.823+0000 7fcc225a3640 1 --2- 172.21.15.55:0/931156697 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcc08067690 0x7fcc08069b50 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009ac0 tx=0x7fcc0c009190 comp rx=0 tx=0).stop 2024-03-19T02:04:49.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.823+0000 7fcc225a3640 1 -- 172.21.15.55:0/931156697 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc1c0c9a90 msgr2=0x7fcc1c146f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:49.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.823+0000 7fcc225a3640 1 --2- 172.21.15.55:0/931156697 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc1c0c9a90 0x7fcc1c146f80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fcc100b0040 tx=0x7fcc10019230 comp rx=0 tx=0).stop 2024-03-19T02:04:49.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.823+0000 7fcc225a3640 1 -- 172.21.15.55:0/931156697 shutdown_connections 2024-03-19T02:04:49.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.823+0000 7fcc225a3640 1 --2- 172.21.15.55:0/931156697 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fcc08067690 0x7fcc08069b50 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:49.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.823+0000 7fcc225a3640 1 --2- 172.21.15.55:0/931156697 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc1c1540c0 0x7fcc1c1474c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:49.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.823+0000 7fcc225a3640 1 --2- 172.21.15.55:0/931156697 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc1c0c9a90 0x7fcc1c146f80 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:49.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.823+0000 7fcc225a3640 1 --2- 172.21.15.55:0/931156697 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fcc1c0c96b0 0x7fcc1c145850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:49.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.823+0000 7fcc225a3640 1 -- 172.21.15.55:0/931156697 >> 172.21.15.55:0/931156697 conn(0x7fcc1c0cc9c0 msgr2=0x7fcc1c153230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:49.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.823+0000 7fcc225a3640 1 -- 172.21.15.55:0/931156697 shutdown_connections 2024-03-19T02:04:49.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:49.823+0000 7fcc225a3640 1 -- 172.21.15.55:0/931156697 wait complete. 2024-03-19T02:04:50.442 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:50.442 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:51.443 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:51.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.563+0000 7f49592d3640 1 Processor -- start 2024-03-19T02:04:51.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.563+0000 7f49592d3640 1 -- start start 2024-03-19T02:04:51.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.563+0000 7f49592d3640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f49540c61c0 0x7f49540708a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:51.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.563+0000 7f49592d3640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49540c73c0 0x7f49540c7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:51.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.563+0000 7f49592d3640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49540c3f60 con 0x7f49541510f0 2024-03-19T02:04:51.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.563+0000 7f49592d3640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f49540c6630 con 0x7f49540c61c0 2024-03-19T02:04:51.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.563+0000 7f49592d3640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f49541514d0 con 0x7f49540c73c0 2024-03-19T02:04:51.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.567+0000 7f49527fc640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f49540c61c0 0x7f49540708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:51.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.567+0000 7f49527fc640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f49540c61c0 0x7f49540708a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:35246/0 (socket says 172.21.15.55:35246) 2024-03-19T02:04:51.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.567+0000 7f49527fc640 1 -- 172.21.15.55:0/3350987867 learned_addr learned my addr 172.21.15.55:0/3350987867 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:51.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.567+0000 7f49537fe640 1 --2- 172.21.15.55:0/3350987867 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49540c73c0 0x7f49540c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:51.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.571+0000 7f49527fc640 1 -- 172.21.15.55:0/3350987867 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49540c73c0 msgr2=0x7f49540c7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:51.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.571+0000 7f49527fc640 1 --2- 172.21.15.55:0/3350987867 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49540c73c0 0x7f49540c7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:51.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.571+0000 7f49527fc640 1 -- 172.21.15.55:0/3350987867 >> v1:172.21.15.55:6789/0 conn(0x7f49541510f0 legacy=0x7f495415a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:51.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.571+0000 7f49527fc640 1 -- 172.21.15.55:0/3350987867 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49540c6870 con 0x7f49540c61c0 2024-03-19T02:04:51.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.575+0000 7f49527fc640 1 --2- 172.21.15.55:0/3350987867 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49540c61c0 0x7f49540708a0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f4948009870 tx=0x7f494801ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17c8532b6277a71f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:51.576 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.579+0000 7f4951ffb640 1 -- 172.21.15.55:0/3350987867 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49480b5070 con 0x7f49540c61c0 2024-03-19T02:04:51.576 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.579+0000 7f4951ffb640 1 -- 172.21.15.55:0/3350987867 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49480a8a50 con 0x7f49540c61c0 2024-03-19T02:04:51.576 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.579+0000 7f4951ffb640 1 -- 172.21.15.55:0/3350987867 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49480a8e20 con 0x7f49540c61c0 2024-03-19T02:04:51.577 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.579+0000 7f49592d3640 1 -- 172.21.15.55:0/3350987867 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49540c61c0 msgr2=0x7f49540708a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:51.577 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.579+0000 7f49592d3640 1 --2- 172.21.15.55:0/3350987867 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49540c61c0 0x7f49540708a0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f4948009870 tx=0x7f494801ed50 comp rx=0 tx=0).stop 2024-03-19T02:04:51.577 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.579+0000 7f49592d3640 1 -- 172.21.15.55:0/3350987867 shutdown_connections 2024-03-19T02:04:51.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.579+0000 7f49592d3640 1 --2- 172.21.15.55:0/3350987867 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49540c73c0 0x7f49540c7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:51.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.579+0000 7f49592d3640 1 --2- 172.21.15.55:0/3350987867 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49540c61c0 0x7f49540708a0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:51.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.579+0000 7f49592d3640 1 -- 172.21.15.55:0/3350987867 >> 172.21.15.55:0/3350987867 conn(0x7f49540cc9c0 msgr2=0x7f49540ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:51.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.579+0000 7f49592d3640 1 -- 172.21.15.55:0/3350987867 shutdown_connections 2024-03-19T02:04:51.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.579+0000 7f49592d3640 1 -- 172.21.15.55:0/3350987867 wait complete. 2024-03-19T02:04:51.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.579+0000 7f49592d3640 1 Processor -- start 2024-03-19T02:04:51.578 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.579+0000 7f49592d3640 1 -- start start 2024-03-19T02:04:51.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.579+0000 7f49592d3640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49540c61c0 0x7f49541f4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:51.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.579+0000 7f49592d3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49540c73c0 0x7f49541e4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:51.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.579+0000 7f49592d3640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f49541510f0 0x7f495414f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:51.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.579+0000 7f49592d3640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f49540c44e0 con 0x7f49540c73c0 2024-03-19T02:04:51.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.579+0000 7f49592d3640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f49540c4200 con 0x7f49541510f0 2024-03-19T02:04:51.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.579+0000 7f49592d3640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f49540c6e30 con 0x7f49540c61c0 2024-03-19T02:04:51.580 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.583+0000 7f49527fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49540c73c0 0x7f49541e4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:51.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.583+0000 7f49527fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49540c73c0 0x7f49541e4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:35256/0 (socket says 172.21.15.55:35256) 2024-03-19T02:04:51.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.583+0000 7f49527fc640 1 -- 172.21.15.55:0/3527989061 learned_addr learned my addr 172.21.15.55:0/3527989061 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:51.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.583+0000 7f4952ffd640 1 --2- 172.21.15.55:0/3527989061 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49540c61c0 0x7f49541f4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:51.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.583+0000 7f49537fe640 1 --2- 172.21.15.55:0/3527989061 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f49541510f0 0x7f495414f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:51.586 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.587+0000 7f49527fc640 1 -- 172.21.15.55:0/3527989061 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49540c61c0 msgr2=0x7f49541f4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:51.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.587+0000 7f49527fc640 1 --2- 172.21.15.55:0/3527989061 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49540c61c0 0x7f49541f4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:51.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.587+0000 7f49527fc640 1 -- 172.21.15.55:0/3527989061 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f49541510f0 msgr2=0x7f495414f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:51.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.587+0000 7f49527fc640 1 --2- 172.21.15.55:0/3527989061 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f49541510f0 0x7f495414f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:51.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.587+0000 7f49527fc640 1 -- 172.21.15.55:0/3527989061 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49540cb370 con 0x7f49540c73c0 2024-03-19T02:04:51.590 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.591+0000 7f49527fc640 1 --2- 172.21.15.55:0/3527989061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49540c73c0 0x7f49541e4ca0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f494801b960 tx=0x7f494801baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:51.595 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.595+0000 7f493b7fe640 1 -- 172.21.15.55:0/3527989061 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49480b5040 con 0x7f49540c73c0 2024-03-19T02:04:51.595 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.595+0000 7f493b7fe640 1 -- 172.21.15.55:0/3527989061 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49480a8850 con 0x7f49540c73c0 2024-03-19T02:04:51.595 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.595+0000 7f49592d3640 1 -- 172.21.15.55:0/3527989061 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49540cb650 con 0x7f49540c73c0 2024-03-19T02:04:51.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.595+0000 7f493b7fe640 1 -- 172.21.15.55:0/3527989061 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49480b1d20 con 0x7f49540c73c0 2024-03-19T02:04:51.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.595+0000 7f49592d3640 1 -- 172.21.15.55:0/3527989061 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49540cbc10 con 0x7f49540c73c0 2024-03-19T02:04:51.596 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.599+0000 7f49592d3640 1 -- 172.21.15.55:0/3527989061 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4954068a40 con 0x7f49540c73c0 2024-03-19T02:04:51.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.879+0000 7f493b7fe640 1 -- 172.21.15.55:0/3527989061 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f49480c0020 con 0x7f49540c73c0 2024-03-19T02:04:51.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.879+0000 7f493b7fe640 1 --2- 172.21.15.55:0/3527989061 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f49400676b0 0x7f4940069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:51.878 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.879+0000 7f4952ffd640 1 --2- 172.21.15.55:0/3527989061 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f49400676b0 0x7f4940069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:51.879 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.879+0000 7f4952ffd640 1 --2- 172.21.15.55:0/3527989061 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f49400676b0 0x7f4940069b70 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4944001810 tx=0x7f4944009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:51.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.899+0000 7f493b7fe640 1 -- 172.21.15.55:0/3527989061 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f49480a8a00 con 0x7f49540c73c0 2024-03-19T02:04:51.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:51.903+0000 7f493b7fe640 1 -- 172.21.15.55:0/3527989061 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f49480c8080 con 0x7f49540c73c0 2024-03-19T02:04:52.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:52.123+0000 7f49592d3640 1 -- 172.21.15.55:0/3527989061 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49540cbe50 con 0x7f49540c73c0 2024-03-19T02:04:52.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:52.147+0000 7f493b7fe640 1 -- 172.21.15.55:0/3527989061 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f49480188b0 con 0x7f49540c73c0 2024-03-19T02:04:52.147 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:52.147 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:52.147 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:52.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:52.151+0000 7f49592d3640 1 -- 172.21.15.55:0/3527989061 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f49400676b0 msgr2=0x7f4940069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:52.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:52.151+0000 7f49592d3640 1 --2- 172.21.15.55:0/3527989061 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f49400676b0 0x7f4940069b70 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4944001810 tx=0x7f4944009040 comp rx=0 tx=0).stop 2024-03-19T02:04:52.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:52.151+0000 7f49592d3640 1 -- 172.21.15.55:0/3527989061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49540c73c0 msgr2=0x7f49541e4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:52.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:52.151+0000 7f49592d3640 1 --2- 172.21.15.55:0/3527989061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49540c73c0 0x7f49541e4ca0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f494801b960 tx=0x7f494801baf0 comp rx=0 tx=0).stop 2024-03-19T02:04:52.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:52.151+0000 7f49592d3640 1 -- 172.21.15.55:0/3527989061 shutdown_connections 2024-03-19T02:04:52.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:52.151+0000 7f49592d3640 1 --2- 172.21.15.55:0/3527989061 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f49400676b0 0x7f4940069b70 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:52.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:52.151+0000 7f49592d3640 1 --2- 172.21.15.55:0/3527989061 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f49541510f0 0x7f495414f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:52.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:52.151+0000 7f49592d3640 1 --2- 172.21.15.55:0/3527989061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f49540c73c0 0x7f49541e4ca0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:52.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:52.151+0000 7f49592d3640 1 --2- 172.21.15.55:0/3527989061 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f49540c61c0 0x7f49541f4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:52.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:52.151+0000 7f49592d3640 1 -- 172.21.15.55:0/3527989061 >> 172.21.15.55:0/3527989061 conn(0x7f49540cc9c0 msgr2=0x7f4954154a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:52.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:52.151+0000 7f49592d3640 1 -- 172.21.15.55:0/3527989061 shutdown_connections 2024-03-19T02:04:52.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:52.151+0000 7f49592d3640 1 -- 172.21.15.55:0/3527989061 wait complete. 2024-03-19T02:04:52.771 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:52.771 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:53.772 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:53.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.903+0000 7f685ffad640 1 Processor -- start 2024-03-19T02:04:53.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.903+0000 7f685ffad640 1 -- start start 2024-03-19T02:04:53.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.903+0000 7f685ffad640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68580cbf30 0x7f68580cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:53.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.903+0000 7f685ffad640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f685806c6f0 0x7f68580cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:53.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.903+0000 7f685ffad640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f685806cb90 con 0x7f685815a3b0 2024-03-19T02:04:53.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.903+0000 7f685ffad640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f68580ce640 con 0x7f685806c6f0 2024-03-19T02:04:53.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.903+0000 7f685ffad640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f68580cda30 con 0x7f68580cbf30 2024-03-19T02:04:53.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.903+0000 7f685e54a640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f685815a3b0 0x7f685815a830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:38472/0 (socket says 172.21.15.55:38472) 2024-03-19T02:04:53.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.903+0000 7f685e54a640 1 -- 172.21.15.55:0/2345256734 learned_addr learned my addr 172.21.15.55:0/2345256734 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:04:53.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.903+0000 7f685dd49640 1 --2- 172.21.15.55:0/2345256734 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68580cbf30 0x7f68580cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:53.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.907+0000 7f685d548640 1 --2- 172.21.15.55:0/2345256734 >> v2:172.21.15.84:3300/0 conn(0x7f685806c6f0 0x7f68580cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:53.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.907+0000 7f685cd47640 1 -- 172.21.15.55:0/2345256734 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 399247102 0 0) 0x7f685806cb90 con 0x7f685815a3b0 2024-03-19T02:04:53.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.907+0000 7f685cd47640 1 -- 172.21.15.55:0/2345256734 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6844003530 con 0x7f685815a3b0 2024-03-19T02:04:53.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.911+0000 7f685dd49640 1 -- 172.21.15.55:0/2345256734 >> v2:172.21.15.84:3300/0 conn(0x7f685806c6f0 msgr2=0x7f68580cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:53.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.911+0000 7f685dd49640 1 --2- 172.21.15.55:0/2345256734 >> v2:172.21.15.84:3300/0 conn(0x7f685806c6f0 0x7f68580cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:53.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.911+0000 7f685dd49640 1 -- 172.21.15.55:0/2345256734 >> v1:172.21.15.55:6789/0 conn(0x7f685815a3b0 legacy=0x7f685815a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:53.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.911+0000 7f685dd49640 1 -- 172.21.15.55:0/2345256734 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68580cdd10 con 0x7f68580cbf30 2024-03-19T02:04:53.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.915+0000 7f685dd49640 1 --2- 172.21.15.55:0/2345256734 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68580cbf30 0x7f68580cead0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f68540097c0 tx=0x7f685401c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7d15fd08690d1a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:53.919 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.919+0000 7f685cd47640 1 -- 172.21.15.55:0/2345256734 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68540b0070 con 0x7f68580cbf30 2024-03-19T02:04:53.919 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.919+0000 7f685cd47640 1 -- 172.21.15.55:0/2345256734 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68540ab600 con 0x7f68580cbf30 2024-03-19T02:04:53.919 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.919+0000 7f685cd47640 1 -- 172.21.15.55:0/2345256734 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68540ab9d0 con 0x7f68580cbf30 2024-03-19T02:04:53.920 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.919+0000 7f685ffad640 1 -- 172.21.15.55:0/2345256734 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68580cbf30 msgr2=0x7f68580cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:53.920 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.919+0000 7f685ffad640 1 --2- 172.21.15.55:0/2345256734 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68580cbf30 0x7f68580cead0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f68540097c0 tx=0x7f685401c410 comp rx=0 tx=0).stop 2024-03-19T02:04:53.920 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.919+0000 7f685ffad640 1 -- 172.21.15.55:0/2345256734 shutdown_connections 2024-03-19T02:04:53.920 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.919+0000 7f685ffad640 1 --2- 172.21.15.55:0/2345256734 >> v2:172.21.15.84:3300/0 conn(0x7f685806c6f0 0x7f68580cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:53.920 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.919+0000 7f685ffad640 1 --2- 172.21.15.55:0/2345256734 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68580cbf30 0x7f68580cead0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:53.920 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.919+0000 7f685ffad640 1 -- 172.21.15.55:0/2345256734 >> 172.21.15.55:0/2345256734 conn(0x7f68580b16a0 msgr2=0x7f68580b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:53.920 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.919+0000 7f685ffad640 1 -- 172.21.15.55:0/2345256734 shutdown_connections 2024-03-19T02:04:53.920 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.919+0000 7f685ffad640 1 -- 172.21.15.55:0/2345256734 wait complete. 2024-03-19T02:04:53.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.923+0000 7f685ffad640 1 Processor -- start 2024-03-19T02:04:53.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.923+0000 7f685ffad640 1 -- start start 2024-03-19T02:04:53.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.923+0000 7f685ffad640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f685806c6f0 0x7f685814fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:53.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.923+0000 7f685ffad640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68580cbf30 0x7f68581f6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:53.921 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.923+0000 7f685ffad640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f685815a3b0 0x7f685814d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:53.922 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.923+0000 7f685ffad640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f68580cc310 con 0x7f685806c6f0 2024-03-19T02:04:53.922 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.923+0000 7f685ffad640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f68580b23b0 con 0x7f68580cbf30 2024-03-19T02:04:53.922 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.923+0000 7f685ffad640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f68580cc490 con 0x7f685815a3b0 2024-03-19T02:04:53.922 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.923+0000 7f685dd49640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f685806c6f0 0x7f685814fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:53.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.923+0000 7f685e54a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f685815a3b0 0x7f685814d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:53.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.923+0000 7f685dd49640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f685806c6f0 0x7f685814fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:35264/0 (socket says 172.21.15.55:35264) 2024-03-19T02:04:53.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.923+0000 7f685dd49640 1 -- 172.21.15.55:0/3415592947 learned_addr learned my addr 172.21.15.55:0/3415592947 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:53.923 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.923+0000 7f685d548640 1 --2- 172.21.15.55:0/3415592947 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68580cbf30 0x7f68581f6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:53.926 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.927+0000 7f685dd49640 1 -- 172.21.15.55:0/3415592947 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f685815a3b0 msgr2=0x7f685814d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:53.926 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.927+0000 7f685dd49640 1 --2- 172.21.15.55:0/3415592947 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f685815a3b0 0x7f685814d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:53.926 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.927+0000 7f685dd49640 1 -- 172.21.15.55:0/3415592947 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68580cbf30 msgr2=0x7f68581f6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:53.926 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.927+0000 7f685dd49640 1 --2- 172.21.15.55:0/3415592947 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68580cbf30 0x7f68581f6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:53.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.927+0000 7f685dd49640 1 -- 172.21.15.55:0/3415592947 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6858150240 con 0x7f685806c6f0 2024-03-19T02:04:53.929 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.931+0000 7f685dd49640 1 --2- 172.21.15.55:0/3415592947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f685806c6f0 0x7f685814fd00 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f685401a040 tx=0x7f68540191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:53.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.935+0000 7f684e7fc640 1 -- 172.21.15.55:0/3415592947 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68540b0040 con 0x7f685806c6f0 2024-03-19T02:04:53.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.935+0000 7f684e7fc640 1 -- 172.21.15.55:0/3415592947 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68540bb070 con 0x7f685806c6f0 2024-03-19T02:04:53.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.935+0000 7f685ffad640 1 -- 172.21.15.55:0/3415592947 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68580afb90 con 0x7f685806c6f0 2024-03-19T02:04:53.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.935+0000 7f684e7fc640 1 -- 172.21.15.55:0/3415592947 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68540b56c0 con 0x7f685806c6f0 2024-03-19T02:04:53.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.935+0000 7f685ffad640 1 -- 172.21.15.55:0/3415592947 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68580af8b0 con 0x7f685806c6f0 2024-03-19T02:04:53.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:53.935+0000 7f685ffad640 1 -- 172.21.15.55:0/3415592947 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6858068330 con 0x7f685806c6f0 2024-03-19T02:04:54.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:54.215+0000 7f684e7fc640 1 -- 172.21.15.55:0/3415592947 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f68540b3070 con 0x7f685806c6f0 2024-03-19T02:04:54.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:54.215+0000 7f684e7fc640 1 --2- 172.21.15.55:0/3415592947 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f683c0676b0 0x7f683c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:54.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:54.215+0000 7f685d548640 1 --2- 172.21.15.55:0/3415592947 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f683c0676b0 0x7f683c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:54.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:54.215+0000 7f685d548640 1 --2- 172.21.15.55:0/3415592947 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f683c0676b0 0x7f683c069b70 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6858164c20 tx=0x7f6848009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:54.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:54.243+0000 7f684e7fc640 1 -- 172.21.15.55:0/3415592947 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f68540ee500 con 0x7f685806c6f0 2024-03-19T02:04:54.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:54.243+0000 7f684e7fc640 1 -- 172.21.15.55:0/3415592947 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f6854016090 con 0x7f685806c6f0 2024-03-19T02:04:54.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:54.467+0000 7f685ffad640 1 -- 172.21.15.55:0/3415592947 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68580b2190 con 0x7f685806c6f0 2024-03-19T02:04:54.493 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:54.495+0000 7f684e7fc640 1 -- 172.21.15.55:0/3415592947 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f68540f0460 con 0x7f685806c6f0 2024-03-19T02:04:54.493 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:54.493 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:54.494 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:54.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:54.499+0000 7f685ffad640 1 -- 172.21.15.55:0/3415592947 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f683c0676b0 msgr2=0x7f683c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:54.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:54.499+0000 7f685ffad640 1 --2- 172.21.15.55:0/3415592947 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f683c0676b0 0x7f683c069b70 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6858164c20 tx=0x7f6848009190 comp rx=0 tx=0).stop 2024-03-19T02:04:54.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:54.499+0000 7f685ffad640 1 -- 172.21.15.55:0/3415592947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f685806c6f0 msgr2=0x7f685814fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:54.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:54.499+0000 7f685ffad640 1 --2- 172.21.15.55:0/3415592947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f685806c6f0 0x7f685814fd00 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f685401a040 tx=0x7f68540191b0 comp rx=0 tx=0).stop 2024-03-19T02:04:54.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:54.499+0000 7f685ffad640 1 -- 172.21.15.55:0/3415592947 shutdown_connections 2024-03-19T02:04:54.497 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:54.499+0000 7f685ffad640 1 --2- 172.21.15.55:0/3415592947 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f683c0676b0 0x7f683c069b70 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:54.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:54.499+0000 7f685ffad640 1 --2- 172.21.15.55:0/3415592947 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f685815a3b0 0x7f685814d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:54.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:54.499+0000 7f685ffad640 1 --2- 172.21.15.55:0/3415592947 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68580cbf30 0x7f68581f6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:54.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:54.499+0000 7f685ffad640 1 --2- 172.21.15.55:0/3415592947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f685806c6f0 0x7f685814fd00 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:54.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:54.499+0000 7f685ffad640 1 -- 172.21.15.55:0/3415592947 >> 172.21.15.55:0/3415592947 conn(0x7f68580b16a0 msgr2=0x7f68580cab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:54.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:54.499+0000 7f685ffad640 1 -- 172.21.15.55:0/3415592947 shutdown_connections 2024-03-19T02:04:54.498 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:54.499+0000 7f685ffad640 1 -- 172.21.15.55:0/3415592947 wait complete. 2024-03-19T02:04:55.118 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:55.118 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:56.120 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:56.241 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.243+0000 7fd603a2e640 1 Processor -- start 2024-03-19T02:04:56.242 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.243+0000 7fd603a2e640 1 -- start start 2024-03-19T02:04:56.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.243+0000 7fd603a2e640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fd5fc1546a0 0x7fd5fc0c8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:56.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.243+0000 7fd603a2e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd5fc0ca0c0 0x7fd5fc0ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:56.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.243+0000 7fd603a2e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5fc0ca550 con 0x7fd5fc072c10 2024-03-19T02:04:56.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.243+0000 7fd603a2e640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fd5fc06bab0 con 0x7fd5fc1546a0 2024-03-19T02:04:56.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.243+0000 7fd603a2e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd5fc06bd90 con 0x7fd5fc0ca0c0 2024-03-19T02:04:56.243 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.243+0000 7fd601fcb640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd5fc0ca0c0 0x7fd5fc0ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:56.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.247+0000 7fd6017ca640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fd5fc1546a0 0x7fd5fc0c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:56.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.247+0000 7fd601fcb640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd5fc0ca0c0 0x7fd5fc0ccb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:47760/0 (socket says 172.21.15.55:47760) 2024-03-19T02:04:56.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.247+0000 7fd601fcb640 1 -- 172.21.15.55:0/2174375704 learned_addr learned my addr 172.21.15.55:0/2174375704 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:56.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.251+0000 7fd601fcb640 1 -- 172.21.15.55:0/2174375704 >> v2:172.21.15.84:3300/0 conn(0x7fd5fc1546a0 msgr2=0x7fd5fc0c8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:56.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.251+0000 7fd601fcb640 1 --2- 172.21.15.55:0/2174375704 >> v2:172.21.15.84:3300/0 conn(0x7fd5fc1546a0 0x7fd5fc0c8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:56.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.251+0000 7fd601fcb640 1 -- 172.21.15.55:0/2174375704 >> v1:172.21.15.84:6789/0 conn(0x7fd5fc072c10 legacy=0x7fd5fc15e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:56.248 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.251+0000 7fd601fcb640 1 -- 172.21.15.55:0/2174375704 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5fc06c070 con 0x7fd5fc0ca0c0 2024-03-19T02:04:56.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.255+0000 7fd601fcb640 1 --2- 172.21.15.55:0/2174375704 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd5fc0ca0c0 0x7fd5fc0ccb90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4009e50 tx=0x7fd5f401ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d8446d1387a0cfd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:56.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.263+0000 7fd5effff640 1 -- 172.21.15.55:0/2174375704 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f40a5700 con 0x7fd5fc0ca0c0 2024-03-19T02:04:56.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.263+0000 7fd5effff640 1 -- 172.21.15.55:0/2174375704 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5f40a58a0 con 0x7fd5fc0ca0c0 2024-03-19T02:04:56.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.263+0000 7fd5effff640 1 -- 172.21.15.55:0/2174375704 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f40a5c70 con 0x7fd5fc0ca0c0 2024-03-19T02:04:56.262 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.263+0000 7fd603a2e640 1 -- 172.21.15.55:0/2174375704 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd5fc0ca0c0 msgr2=0x7fd5fc0ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:56.262 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.263+0000 7fd603a2e640 1 --2- 172.21.15.55:0/2174375704 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd5fc0ca0c0 0x7fd5fc0ccb90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4009e50 tx=0x7fd5f401ca30 comp rx=0 tx=0).stop 2024-03-19T02:04:56.262 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.263+0000 7fd603a2e640 1 -- 172.21.15.55:0/2174375704 shutdown_connections 2024-03-19T02:04:56.262 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.263+0000 7fd603a2e640 1 --2- 172.21.15.55:0/2174375704 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd5fc0ca0c0 0x7fd5fc0ccb90 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:56.262 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.263+0000 7fd603a2e640 1 --2- 172.21.15.55:0/2174375704 >> v2:172.21.15.84:3300/0 conn(0x7fd5fc1546a0 0x7fd5fc0c8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:56.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.263+0000 7fd603a2e640 1 -- 172.21.15.55:0/2174375704 >> 172.21.15.55:0/2174375704 conn(0x7fd5fc0703c0 msgr2=0x7fd5fc0707d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:56.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.263+0000 7fd603a2e640 1 -- 172.21.15.55:0/2174375704 shutdown_connections 2024-03-19T02:04:56.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.263+0000 7fd603a2e640 1 -- 172.21.15.55:0/2174375704 wait complete. 2024-03-19T02:04:56.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.267+0000 7fd603a2e640 1 Processor -- start 2024-03-19T02:04:56.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.267+0000 7fd603a2e640 1 -- start start 2024-03-19T02:04:56.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.267+0000 7fd603a2e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd5fc072c10 0x7fd5fc1428b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:56.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.267+0000 7fd603a2e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5fc0ca0c0 0x7fd5fc0caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:56.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.267+0000 7fd603a2e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5fc1546a0 0x7fd5fc0cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:56.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.267+0000 7fd603a2e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc0c7dd0 con 0x7fd5fc1546a0 2024-03-19T02:04:56.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.267+0000 7fd603a2e640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc154a80 con 0x7fd5fc0ca0c0 2024-03-19T02:04:56.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.267+0000 7fd603a2e640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd5fc0c7f50 con 0x7fd5fc072c10 2024-03-19T02:04:56.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.267+0000 7fd601fcb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5fc1546a0 0x7fd5fc0cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:56.267 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.267+0000 7fd601fcb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5fc1546a0 0x7fd5fc0cb4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:35280/0 (socket says 172.21.15.55:35280) 2024-03-19T02:04:56.267 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.267+0000 7fd601fcb640 1 -- 172.21.15.55:0/176767229 learned_addr learned my addr 172.21.15.55:0/176767229 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:56.267 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.267+0000 7fd6017ca640 1 --2- 172.21.15.55:0/176767229 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd5fc072c10 0x7fd5fc1428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:56.267 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.267+0000 7fd600fc9640 1 --2- 172.21.15.55:0/176767229 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5fc0ca0c0 0x7fd5fc0caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:56.270 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.271+0000 7fd601fcb640 1 -- 172.21.15.55:0/176767229 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd5fc072c10 msgr2=0x7fd5fc1428b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:56.270 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.271+0000 7fd601fcb640 1 --2- 172.21.15.55:0/176767229 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd5fc072c10 0x7fd5fc1428b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:56.270 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.271+0000 7fd601fcb640 1 -- 172.21.15.55:0/176767229 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5fc0ca0c0 msgr2=0x7fd5fc0caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:56.270 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.271+0000 7fd601fcb640 1 --2- 172.21.15.55:0/176767229 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5fc0ca0c0 0x7fd5fc0caf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:56.270 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.271+0000 7fd601fcb640 1 -- 172.21.15.55:0/176767229 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5fc15a170 con 0x7fd5fc1546a0 2024-03-19T02:04:56.272 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.275+0000 7fd601fcb640 1 --2- 172.21.15.55:0/176767229 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5fc1546a0 0x7fd5fc0cb4d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fd5f40af040 tx=0x7fd5f40197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:56.277 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.279+0000 7fd5edffb640 1 -- 172.21.15.55:0/176767229 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f40a5700 con 0x7fd5fc1546a0 2024-03-19T02:04:56.277 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.279+0000 7fd5edffb640 1 -- 172.21.15.55:0/176767229 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5f40be070 con 0x7fd5fc1546a0 2024-03-19T02:04:56.277 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.279+0000 7fd5edffb640 1 -- 172.21.15.55:0/176767229 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f40b88b0 con 0x7fd5fc1546a0 2024-03-19T02:04:56.278 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.279+0000 7fd603a2e640 1 -- 172.21.15.55:0/176767229 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5fc15a450 con 0x7fd5fc1546a0 2024-03-19T02:04:56.278 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.279+0000 7fd603a2e640 1 -- 172.21.15.55:0/176767229 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5fc15aa10 con 0x7fd5fc1546a0 2024-03-19T02:04:56.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.279+0000 7fd603a2e640 1 -- 172.21.15.55:0/176767229 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5fc068a40 con 0x7fd5fc1546a0 2024-03-19T02:04:56.587 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.587+0000 7fd5edffb640 1 -- 172.21.15.55:0/176767229 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fd5f40b8a50 con 0x7fd5fc1546a0 2024-03-19T02:04:56.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.591+0000 7fd5edffb640 1 --2- 172.21.15.55:0/176767229 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd5e00676b0 0x7fd5e0069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:56.588 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.591+0000 7fd6017ca640 1 --2- 172.21.15.55:0/176767229 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd5e00676b0 0x7fd5e0069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:56.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.591+0000 7fd6017ca640 1 --2- 172.21.15.55:0/176767229 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd5e00676b0 0x7fd5e0069b70 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0009b60 tx=0x7fd5f0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:56.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.631+0000 7fd5edffb640 1 -- 172.21.15.55:0/176767229 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fd5f40ef8e0 con 0x7fd5fc1546a0 2024-03-19T02:04:56.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.655+0000 7fd5edffb640 1 -- 172.21.15.55:0/176767229 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fd5f4016620 con 0x7fd5fc1546a0 2024-03-19T02:04:56.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.867+0000 7fd603a2e640 1 -- 172.21.15.55:0/176767229 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5fc158a50 con 0x7fd5fc1546a0 2024-03-19T02:04:56.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.887+0000 7fd5edffb640 1 -- 172.21.15.55:0/176767229 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fd5f40f1840 con 0x7fd5fc1546a0 2024-03-19T02:04:56.886 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:56.886 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:56.887 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:56.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.891+0000 7fd603a2e640 1 -- 172.21.15.55:0/176767229 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd5e00676b0 msgr2=0x7fd5e0069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:56.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.891+0000 7fd603a2e640 1 --2- 172.21.15.55:0/176767229 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd5e00676b0 0x7fd5e0069b70 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0009b60 tx=0x7fd5f0009190 comp rx=0 tx=0).stop 2024-03-19T02:04:56.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.891+0000 7fd603a2e640 1 -- 172.21.15.55:0/176767229 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5fc1546a0 msgr2=0x7fd5fc0cb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:56.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.891+0000 7fd603a2e640 1 --2- 172.21.15.55:0/176767229 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5fc1546a0 0x7fd5fc0cb4d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fd5f40af040 tx=0x7fd5f40197d0 comp rx=0 tx=0).stop 2024-03-19T02:04:56.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.891+0000 7fd603a2e640 1 -- 172.21.15.55:0/176767229 shutdown_connections 2024-03-19T02:04:56.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.891+0000 7fd603a2e640 1 --2- 172.21.15.55:0/176767229 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd5e00676b0 0x7fd5e0069b70 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:56.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.891+0000 7fd603a2e640 1 --2- 172.21.15.55:0/176767229 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5fc1546a0 0x7fd5fc0cb4d0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:56.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.891+0000 7fd603a2e640 1 --2- 172.21.15.55:0/176767229 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5fc0ca0c0 0x7fd5fc0caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:56.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.891+0000 7fd603a2e640 1 --2- 172.21.15.55:0/176767229 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd5fc072c10 0x7fd5fc1428b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:56.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.891+0000 7fd603a2e640 1 -- 172.21.15.55:0/176767229 >> 172.21.15.55:0/176767229 conn(0x7fd5fc0703c0 msgr2=0x7fd5fc0c41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:56.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.891+0000 7fd603a2e640 1 -- 172.21.15.55:0/176767229 shutdown_connections 2024-03-19T02:04:56.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:56.891+0000 7fd603a2e640 1 -- 172.21.15.55:0/176767229 wait complete. 2024-03-19T02:04:57.515 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:57.516 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:04:58.517 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:04:58.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.627+0000 7f509e2ad640 1 Processor -- start 2024-03-19T02:04:58.627 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.627+0000 7f509e2ad640 1 -- start start 2024-03-19T02:04:58.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.627+0000 7f509e2ad640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f50980c2b70 0x7f50980c3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:58.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.631+0000 7f509e2ad640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f509806f6d0 con 0x7f5098070970 2024-03-19T02:04:58.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.631+0000 7f509e2ad640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5098070450 con 0x7f5098152150 2024-03-19T02:04:58.628 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.631+0000 7f509e2ad640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f5098071130 con 0x7f50980c2b70 2024-03-19T02:04:58.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.631+0000 7f509c84a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f50980c2b70 0x7f50980c3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:58.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.631+0000 7f5097fff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f5098152150 0x7f5098158a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:60742/0 (socket says 172.21.15.55:60742) 2024-03-19T02:04:58.629 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.631+0000 7f5097fff640 1 -- 172.21.15.55:0/326623382 learned_addr learned my addr 172.21.15.55:0/326623382 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:04:58.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.635+0000 7f509c84a640 1 -- 172.21.15.55:0/326623382 >> v1:172.21.15.84:6789/0 conn(0x7f5098152150 legacy=0x7f5098158a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:58.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.635+0000 7f509c84a640 1 -- 172.21.15.55:0/326623382 >> v1:172.21.15.55:6789/0 conn(0x7f5098070970 legacy=0x7f50980c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:04:58.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.635+0000 7f509c84a640 1 -- 172.21.15.55:0/326623382 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5098070e50 con 0x7f50980c2b70 2024-03-19T02:04:58.639 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.639+0000 7f509c84a640 1 --2- 172.21.15.55:0/326623382 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f50980c2b70 0x7f50980c3000 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5090009dc0 tx=0x7f509001f140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d39114c6659e7b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:58.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.647+0000 7f5096ffd640 1 -- 172.21.15.55:0/326623382 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50900b4070 con 0x7f50980c2b70 2024-03-19T02:04:58.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.647+0000 7f5096ffd640 1 -- 172.21.15.55:0/326623382 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50900af600 con 0x7f50980c2b70 2024-03-19T02:04:58.646 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.647+0000 7f5096ffd640 1 -- 172.21.15.55:0/326623382 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50900af9d0 con 0x7f50980c2b70 2024-03-19T02:04:58.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.647+0000 7f509e2ad640 1 -- 172.21.15.55:0/326623382 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f50980c2b70 msgr2=0x7f50980c3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:58.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.647+0000 7f509e2ad640 1 --2- 172.21.15.55:0/326623382 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f50980c2b70 0x7f50980c3000 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5090009dc0 tx=0x7f509001f140 comp rx=0 tx=0).stop 2024-03-19T02:04:58.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.647+0000 7f509e2ad640 1 -- 172.21.15.55:0/326623382 shutdown_connections 2024-03-19T02:04:58.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.647+0000 7f509e2ad640 1 --2- 172.21.15.55:0/326623382 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f50980c2b70 0x7f50980c3000 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:58.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.647+0000 7f509e2ad640 1 -- 172.21.15.55:0/326623382 >> 172.21.15.55:0/326623382 conn(0x7f5098067600 msgr2=0x7f50980714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:58.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.647+0000 7f509e2ad640 1 -- 172.21.15.55:0/326623382 shutdown_connections 2024-03-19T02:04:58.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.647+0000 7f509e2ad640 1 -- 172.21.15.55:0/326623382 wait complete. 2024-03-19T02:04:58.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.651+0000 7f509e2ad640 1 Processor -- start 2024-03-19T02:04:58.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.651+0000 7f509e2ad640 1 -- start start 2024-03-19T02:04:58.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.651+0000 7f509e2ad640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5098070970 0x7f50980c6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:58.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.651+0000 7f509e2ad640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f50980c2b70 0x7f509814b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:58.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.651+0000 7f509e2ad640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5098152150 0x7f509806cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:58.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.651+0000 7f509e2ad640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f509806ff30 con 0x7f5098152150 2024-03-19T02:04:58.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.651+0000 7f509e2ad640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f509806fc50 con 0x7f50980c2b70 2024-03-19T02:04:58.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.651+0000 7f509e2ad640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f50980c97d0 con 0x7f5098070970 2024-03-19T02:04:58.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.651+0000 7f509c84a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5098152150 0x7f509806cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:58.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.651+0000 7f50977fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f50980c2b70 0x7f509814b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:58.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.651+0000 7f509c84a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5098152150 0x7f509806cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:35294/0 (socket says 172.21.15.55:35294) 2024-03-19T02:04:58.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.651+0000 7f509c84a640 1 -- 172.21.15.55:0/2130504610 learned_addr learned my addr 172.21.15.55:0/2130504610 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:04:58.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.655+0000 7f509c84a640 1 -- 172.21.15.55:0/2130504610 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5098070970 msgr2=0x7f50980c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:58.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.655+0000 7f509c84a640 1 --2- 172.21.15.55:0/2130504610 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5098070970 0x7f50980c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:58.654 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.655+0000 7f509c84a640 1 -- 172.21.15.55:0/2130504610 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f50980c2b70 msgr2=0x7f509814b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:58.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.655+0000 7f509c84a640 1 --2- 172.21.15.55:0/2130504610 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f50980c2b70 0x7f509814b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:58.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.655+0000 7f509c84a640 1 -- 172.21.15.55:0/2130504610 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50980c4f20 con 0x7f5098152150 2024-03-19T02:04:58.660 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.663+0000 7f509c84a640 1 --2- 172.21.15.55:0/2130504610 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5098152150 0x7f509806cce0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5090009eb0 tx=0x7f509001bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:58.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.667+0000 7f5094ff9640 1 -- 172.21.15.55:0/2130504610 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50900b4040 con 0x7f5098152150 2024-03-19T02:04:58.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.667+0000 7f5094ff9640 1 -- 172.21.15.55:0/2130504610 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50900c2070 con 0x7f5098152150 2024-03-19T02:04:58.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.667+0000 7f509e2ad640 1 -- 172.21.15.55:0/2130504610 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50981f06f0 con 0x7f5098152150 2024-03-19T02:04:58.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.667+0000 7f5094ff9640 1 -- 172.21.15.55:0/2130504610 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50900ba870 con 0x7f5098152150 2024-03-19T02:04:58.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.667+0000 7f509e2ad640 1 -- 172.21.15.55:0/2130504610 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50981f0990 con 0x7f5098152150 2024-03-19T02:04:58.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.667+0000 7f509e2ad640 1 -- 172.21.15.55:0/2130504610 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5098068a40 con 0x7f5098152150 2024-03-19T02:04:58.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.699+0000 7f5094ff9640 1 -- 172.21.15.55:0/2130504610 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f50900baa10 con 0x7f5098152150 2024-03-19T02:04:58.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.699+0000 7f5094ff9640 1 --2- 172.21.15.55:0/2130504610 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f50780676b0 0x7f5078069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:04:58.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.703+0000 7f5097fff640 1 --2- 172.21.15.55:0/2130504610 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f50780676b0 0x7f5078069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:04:58.700 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.703+0000 7f5097fff640 1 --2- 172.21.15.55:0/2130504610 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f50780676b0 0x7f5078069b70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5088000f70 tx=0x7f5088007a90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:04:58.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.990+0000 7f5094ff9640 1 -- 172.21.15.55:0/2130504610 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f50900f33a0 con 0x7f5098152150 2024-03-19T02:04:58.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:58.990+0000 7f5094ff9640 1 -- 172.21.15.55:0/2130504610 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f50980c4f20 con 0x7f5098152150 2024-03-19T02:04:59.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:59.218+0000 7f509e2ad640 1 -- 172.21.15.55:0/2130504610 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5098143370 con 0x7f5098152150 2024-03-19T02:04:59.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:59.234+0000 7f5094ff9640 1 -- 172.21.15.55:0/2130504610 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f5090018ca0 con 0x7f5098152150 2024-03-19T02:04:59.235 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:04:59.235 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:04:59.235 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:04:59.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:59.234+0000 7f509e2ad640 1 -- 172.21.15.55:0/2130504610 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f50780676b0 msgr2=0x7f5078069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:59.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:59.234+0000 7f509e2ad640 1 --2- 172.21.15.55:0/2130504610 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f50780676b0 0x7f5078069b70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5088000f70 tx=0x7f5088007a90 comp rx=0 tx=0).stop 2024-03-19T02:04:59.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:59.234+0000 7f509e2ad640 1 -- 172.21.15.55:0/2130504610 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5098152150 msgr2=0x7f509806cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:04:59.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:59.234+0000 7f509e2ad640 1 --2- 172.21.15.55:0/2130504610 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5098152150 0x7f509806cce0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5090009eb0 tx=0x7f509001bee0 comp rx=0 tx=0).stop 2024-03-19T02:04:59.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:59.238+0000 7f509e2ad640 1 -- 172.21.15.55:0/2130504610 shutdown_connections 2024-03-19T02:04:59.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:59.238+0000 7f509e2ad640 1 --2- 172.21.15.55:0/2130504610 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f50780676b0 0x7f5078069b70 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:59.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:59.238+0000 7f509e2ad640 1 --2- 172.21.15.55:0/2130504610 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5098152150 0x7f509806cce0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:59.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:59.238+0000 7f509e2ad640 1 --2- 172.21.15.55:0/2130504610 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f50980c2b70 0x7f509814b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:59.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:59.238+0000 7f509e2ad640 1 --2- 172.21.15.55:0/2130504610 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5098070970 0x7f50980c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:04:59.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:59.238+0000 7f509e2ad640 1 -- 172.21.15.55:0/2130504610 >> 172.21.15.55:0/2130504610 conn(0x7f5098067600 msgr2=0x7f5098157760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:04:59.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:59.238+0000 7f509e2ad640 1 -- 172.21.15.55:0/2130504610 shutdown_connections 2024-03-19T02:04:59.237 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:04:59.238+0000 7f509e2ad640 1 -- 172.21.15.55:0/2130504610 wait complete. 2024-03-19T02:04:59.856 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:04:59.856 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:00.857 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:00.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.970+0000 7fe5f1f6b640 1 Processor -- start 2024-03-19T02:05:00.971 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.970+0000 7fe5f1f6b640 1 -- start start 2024-03-19T02:05:00.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.974+0000 7fe5f1f6b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe5ec0c6e60 0x7fe5ec06b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:00.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.974+0000 7fe5f1f6b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5ec13d010 con 0x7fe5ec070670 2024-03-19T02:05:00.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.974+0000 7fe5f1f6b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5ec0ccbf0 con 0x7fe5ec070a50 2024-03-19T02:05:00.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.974+0000 7fe5f1f6b640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fe5ec0cce90 con 0x7fe5ec0c6e60 2024-03-19T02:05:00.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.974+0000 7fe5ebfff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fe5ec0c6e60 0x7fe5ec06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:00.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.974+0000 7fe5eaffd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fe5ec070a50 0x7fe5ec0c6750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:60572/0 (socket says 172.21.15.55:60572) 2024-03-19T02:05:00.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.974+0000 7fe5eaffd640 1 -- 172.21.15.55:0/3636083585 learned_addr learned my addr 172.21.15.55:0/3636083585 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:05:00.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.978+0000 7fe5ebfff640 1 -- 172.21.15.55:0/3636083585 >> v1:172.21.15.84:6789/0 conn(0x7fe5ec070a50 legacy=0x7fe5ec0c6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:00.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.978+0000 7fe5ebfff640 1 -- 172.21.15.55:0/3636083585 >> v1:172.21.15.55:6789/0 conn(0x7fe5ec070670 legacy=0x7fe5ec13d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:00.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.978+0000 7fe5ebfff640 1 -- 172.21.15.55:0/3636083585 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5ec0cd170 con 0x7fe5ec0c6e60 2024-03-19T02:05:00.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.982+0000 7fe5ebfff640 1 --2- 172.21.15.55:0/3636083585 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5ec0c6e60 0x7fe5ec06b380 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4009d90 tx=0x7fe5e401f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c97708c925eac43e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:00.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.986+0000 7fe5ea7fc640 1 -- 172.21.15.55:0/3636083585 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e40b4070 con 0x7fe5ec0c6e60 2024-03-19T02:05:00.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.986+0000 7fe5ea7fc640 1 -- 172.21.15.55:0/3636083585 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5e40af600 con 0x7fe5ec0c6e60 2024-03-19T02:05:00.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.986+0000 7fe5ea7fc640 1 -- 172.21.15.55:0/3636083585 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e40af9d0 con 0x7fe5ec0c6e60 2024-03-19T02:05:00.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.986+0000 7fe5f1f6b640 1 -- 172.21.15.55:0/3636083585 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5ec0c6e60 msgr2=0x7fe5ec06b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:00.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.986+0000 7fe5f1f6b640 1 --2- 172.21.15.55:0/3636083585 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5ec0c6e60 0x7fe5ec06b380 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4009d90 tx=0x7fe5e401f150 comp rx=0 tx=0).stop 2024-03-19T02:05:00.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.986+0000 7fe5f1f6b640 1 -- 172.21.15.55:0/3636083585 shutdown_connections 2024-03-19T02:05:00.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.986+0000 7fe5f1f6b640 1 --2- 172.21.15.55:0/3636083585 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5ec0c6e60 0x7fe5ec06b380 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:00.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.986+0000 7fe5f1f6b640 1 -- 172.21.15.55:0/3636083585 >> 172.21.15.55:0/3636083585 conn(0x7fe5ec06c3b0 msgr2=0x7fe5ec06c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:00.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.990+0000 7fe5f1f6b640 1 -- 172.21.15.55:0/3636083585 shutdown_connections 2024-03-19T02:05:00.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.990+0000 7fe5f1f6b640 1 -- 172.21.15.55:0/3636083585 wait complete. 2024-03-19T02:05:00.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.990+0000 7fe5f1f6b640 1 Processor -- start 2024-03-19T02:05:00.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.990+0000 7fe5f1f6b640 1 -- start start 2024-03-19T02:05:00.991 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.990+0000 7fe5f1f6b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5ec070670 0x7fe5ec13c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:00.991 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.990+0000 7fe5f1f6b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe5ec070a50 0x7fe5ec1e1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:00.991 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.990+0000 7fe5f1f6b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe5ec0c6e60 0x7fe5ec1e37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:00.991 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.990+0000 7fe5f1f6b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec13dbf0 con 0x7fe5ec070670 2024-03-19T02:05:00.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.990+0000 7fe5f1f6b640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec0cd730 con 0x7fe5ec070a50 2024-03-19T02:05:00.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.990+0000 7fe5f1f6b640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fe5ec13dd70 con 0x7fe5ec0c6e60 2024-03-19T02:05:00.992 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.990+0000 7fe5eb7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5ec070670 0x7fe5ec13c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:00.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.994+0000 7fe5eb7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5ec070670 0x7fe5ec13c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33972/0 (socket says 172.21.15.55:33972) 2024-03-19T02:05:00.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.994+0000 7fe5eb7fe640 1 -- 172.21.15.55:0/3612736079 learned_addr learned my addr 172.21.15.55:0/3612736079 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:00.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.994+0000 7fe5ebfff640 1 --2- 172.21.15.55:0/3612736079 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe5ec0c6e60 0x7fe5ec1e37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:00.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.994+0000 7fe5eaffd640 1 --2- 172.21.15.55:0/3612736079 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe5ec070a50 0x7fe5ec1e1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:00.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.998+0000 7fe5eb7fe640 1 -- 172.21.15.55:0/3612736079 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe5ec0c6e60 msgr2=0x7fe5ec1e37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:00.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.998+0000 7fe5eb7fe640 1 --2- 172.21.15.55:0/3612736079 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe5ec0c6e60 0x7fe5ec1e37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:00.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.998+0000 7fe5eb7fe640 1 -- 172.21.15.55:0/3612736079 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe5ec070a50 msgr2=0x7fe5ec1e1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:00.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.998+0000 7fe5eb7fe640 1 --2- 172.21.15.55:0/3612736079 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe5ec070a50 0x7fe5ec1e1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:00.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:00.998+0000 7fe5eb7fe640 1 -- 172.21.15.55:0/3612736079 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5ec13c680 con 0x7fe5ec070670 2024-03-19T02:05:01.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.002+0000 7fe5eb7fe640 1 --2- 172.21.15.55:0/3612736079 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5ec070670 0x7fe5ec13c140 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc00c450 tx=0x7fe5dc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:01.004 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.006+0000 7fe5d3fff640 1 -- 172.21.15.55:0/3612736079 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc016020 con 0x7fe5ec070670 2024-03-19T02:05:01.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.006+0000 7fe5d3fff640 1 -- 172.21.15.55:0/3612736079 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5dc0052d0 con 0x7fe5ec070670 2024-03-19T02:05:01.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.006+0000 7fe5f1f6b640 1 -- 172.21.15.55:0/3612736079 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5ec1ee760 con 0x7fe5ec070670 2024-03-19T02:05:01.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.006+0000 7fe5d3fff640 1 -- 172.21.15.55:0/3612736079 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc007260 con 0x7fe5ec070670 2024-03-19T02:05:01.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.006+0000 7fe5f1f6b640 1 -- 172.21.15.55:0/3612736079 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5ec1e2700 con 0x7fe5ec070670 2024-03-19T02:05:01.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.006+0000 7fe5f1f6b640 1 -- 172.21.15.55:0/3612736079 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5ec068a40 con 0x7fe5ec070670 2024-03-19T02:05:01.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.042+0000 7fe5d3fff640 1 -- 172.21.15.55:0/3612736079 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fe5dc007400 con 0x7fe5ec070670 2024-03-19T02:05:01.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.042+0000 7fe5d3fff640 1 --2- 172.21.15.55:0/3612736079 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe5d80676e0 0x7fe5d8069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:01.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.042+0000 7fe5eaffd640 1 --2- 172.21.15.55:0/3612736079 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe5d80676e0 0x7fe5d8069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:01.042 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.042+0000 7fe5eaffd640 1 --2- 172.21.15.55:0/3612736079 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe5d80676e0 0x7fe5d8069ba0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec0ccab0 tx=0x7fe5e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:01.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.046+0000 7fe5d3fff640 1 -- 172.21.15.55:0/3612736079 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fe5dc054650 con 0x7fe5ec070670 2024-03-19T02:05:01.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.342+0000 7fe5d3fff640 1 -- 172.21.15.55:0/3612736079 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fe5dc020020 con 0x7fe5ec070670 2024-03-19T02:05:01.576 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.578+0000 7fe5f1f6b640 1 -- 172.21.15.55:0/3612736079 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5ec155950 con 0x7fe5ec070670 2024-03-19T02:05:01.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.602+0000 7fe5d3fff640 1 -- 172.21.15.55:0/3612736079 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fe5dc005480 con 0x7fe5ec070670 2024-03-19T02:05:01.602 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:01.602 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:01.603 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:01.605 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.606+0000 7fe5f1f6b640 1 -- 172.21.15.55:0/3612736079 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe5d80676e0 msgr2=0x7fe5d8069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:01.605 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.606+0000 7fe5f1f6b640 1 --2- 172.21.15.55:0/3612736079 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe5d80676e0 0x7fe5d8069ba0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec0ccab0 tx=0x7fe5e0009040 comp rx=0 tx=0).stop 2024-03-19T02:05:01.605 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.606+0000 7fe5f1f6b640 1 -- 172.21.15.55:0/3612736079 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5ec070670 msgr2=0x7fe5ec13c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:01.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.606+0000 7fe5f1f6b640 1 --2- 172.21.15.55:0/3612736079 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5ec070670 0x7fe5ec13c140 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc00c450 tx=0x7fe5dc00c910 comp rx=0 tx=0).stop 2024-03-19T02:05:01.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.606+0000 7fe5f1f6b640 1 -- 172.21.15.55:0/3612736079 shutdown_connections 2024-03-19T02:05:01.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.606+0000 7fe5f1f6b640 1 --2- 172.21.15.55:0/3612736079 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fe5d80676e0 0x7fe5d8069ba0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:01.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.606+0000 7fe5f1f6b640 1 --2- 172.21.15.55:0/3612736079 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fe5ec0c6e60 0x7fe5ec1e37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:01.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.606+0000 7fe5f1f6b640 1 --2- 172.21.15.55:0/3612736079 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe5ec070a50 0x7fe5ec1e1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:01.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.606+0000 7fe5f1f6b640 1 --2- 172.21.15.55:0/3612736079 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5ec070670 0x7fe5ec13c140 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:01.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.606+0000 7fe5f1f6b640 1 -- 172.21.15.55:0/3612736079 >> 172.21.15.55:0/3612736079 conn(0x7fe5ec06c3b0 msgr2=0x7fe5ec1564c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:01.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.606+0000 7fe5f1f6b640 1 -- 172.21.15.55:0/3612736079 shutdown_connections 2024-03-19T02:05:01.606 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:01.606+0000 7fe5f1f6b640 1 -- 172.21.15.55:0/3612736079 wait complete. 2024-03-19T02:05:02.229 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:02.229 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:03.230 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:03.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.458+0000 7fd0e5da7640 1 Processor -- start 2024-03-19T02:05:03.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.458+0000 7fd0e5da7640 1 -- start start 2024-03-19T02:05:03.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.462+0000 7fd0e5da7640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd0e0070780 0x7fd0e015b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:03.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.462+0000 7fd0e5da7640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0e006fff0 con 0x7fd0e0152120 2024-03-19T02:05:03.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.462+0000 7fd0e5da7640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0e00c39c0 con 0x7fd0e015bb00 2024-03-19T02:05:03.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.462+0000 7fd0e5da7640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fd0e0152500 con 0x7fd0e0070780 2024-03-19T02:05:03.461 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.462+0000 7fd0deffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd0e0070780 0x7fd0e015b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:03.461 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.462+0000 7fd0deffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd0e0070780 0x7fd0e015b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33976/0 (socket says 172.21.15.55:33976) 2024-03-19T02:05:03.461 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.462+0000 7fd0deffd640 1 -- 172.21.15.55:0/2034645739 learned_addr learned my addr 172.21.15.55:0/2034645739 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:03.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.466+0000 7fd0deffd640 1 -- 172.21.15.55:0/2034645739 >> v1:172.21.15.84:6789/0 conn(0x7fd0e015bb00 legacy=0x7fd0e015bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:03.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.466+0000 7fd0deffd640 1 -- 172.21.15.55:0/2034645739 >> v1:172.21.15.55:6789/0 conn(0x7fd0e0152120 legacy=0x7fd0e00c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:03.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.466+0000 7fd0deffd640 1 -- 172.21.15.55:0/2034645739 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0e00702b0 con 0x7fd0e0070780 2024-03-19T02:05:03.467 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.466+0000 7fd0deffd640 1 --2- 172.21.15.55:0/2034645739 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e0070780 0x7fd0e015b5c0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4009870 tx=0x7fd0d401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa3d78b9ca8fe696 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:03.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.474+0000 7fd0de7fc640 1 -- 172.21.15.55:0/2034645739 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d40b0070 con 0x7fd0e0070780 2024-03-19T02:05:03.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.474+0000 7fd0de7fc640 1 -- 172.21.15.55:0/2034645739 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0d40ab600 con 0x7fd0e0070780 2024-03-19T02:05:03.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.474+0000 7fd0de7fc640 1 -- 172.21.15.55:0/2034645739 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d40ab9d0 con 0x7fd0e0070780 2024-03-19T02:05:03.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.474+0000 7fd0e5da7640 1 -- 172.21.15.55:0/2034645739 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e0070780 msgr2=0x7fd0e015b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:03.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.474+0000 7fd0e5da7640 1 --2- 172.21.15.55:0/2034645739 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e0070780 0x7fd0e015b5c0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4009870 tx=0x7fd0d401c520 comp rx=0 tx=0).stop 2024-03-19T02:05:03.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.474+0000 7fd0e5da7640 1 -- 172.21.15.55:0/2034645739 shutdown_connections 2024-03-19T02:05:03.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.474+0000 7fd0e5da7640 1 --2- 172.21.15.55:0/2034645739 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e0070780 0x7fd0e015b5c0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:03.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.474+0000 7fd0e5da7640 1 -- 172.21.15.55:0/2034645739 >> 172.21.15.55:0/2034645739 conn(0x7fd0e0067600 msgr2=0x7fd0e0071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:03.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.474+0000 7fd0e5da7640 1 -- 172.21.15.55:0/2034645739 shutdown_connections 2024-03-19T02:05:03.475 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.474+0000 7fd0e5da7640 1 -- 172.21.15.55:0/2034645739 wait complete. 2024-03-19T02:05:03.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.474+0000 7fd0e5da7640 1 Processor -- start 2024-03-19T02:05:03.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.474+0000 7fd0e5da7640 1 -- start start 2024-03-19T02:05:03.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.478+0000 7fd0e5da7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e0070780 0x7fd0e01478d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:03.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.478+0000 7fd0e5da7640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0e0152120 0x7fd0e01f7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:03.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.478+0000 7fd0e5da7640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd0e015bb00 0x7fd0e006cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:03.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.478+0000 7fd0e5da7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e00c97e0 con 0x7fd0e0070780 2024-03-19T02:05:03.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.478+0000 7fd0e5da7640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e00c3530 con 0x7fd0e0152120 2024-03-19T02:05:03.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.478+0000 7fd0e5da7640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd0e00c86d0 con 0x7fd0e015bb00 2024-03-19T02:05:03.477 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.478+0000 7fd0df7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e0070780 0x7fd0e01478d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:03.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.478+0000 7fd0df7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e0070780 0x7fd0e01478d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33982/0 (socket says 172.21.15.55:33982) 2024-03-19T02:05:03.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.478+0000 7fd0df7fe640 1 -- 172.21.15.55:0/1842944578 learned_addr learned my addr 172.21.15.55:0/1842944578 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:03.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.482+0000 7fd0df7fe640 1 -- 172.21.15.55:0/1842944578 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd0e015bb00 msgr2=0x7fd0e006cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:03.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.482+0000 7fd0df7fe640 1 --2- 172.21.15.55:0/1842944578 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd0e015bb00 0x7fd0e006cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:03.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.482+0000 7fd0df7fe640 1 -- 172.21.15.55:0/1842944578 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0e0152120 msgr2=0x7fd0e01f7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:03.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.482+0000 7fd0df7fe640 1 --2- 172.21.15.55:0/1842944578 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0e0152120 0x7fd0e01f7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:03.481 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.482+0000 7fd0df7fe640 1 -- 172.21.15.55:0/1842944578 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0e01f4ad0 con 0x7fd0e0070780 2024-03-19T02:05:03.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.486+0000 7fd0df7fe640 1 --2- 172.21.15.55:0/1842944578 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e0070780 0x7fd0e01478d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000b440 tx=0x7fd0d000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:03.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.490+0000 7fd0c7fff640 1 -- 172.21.15.55:0/1842944578 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0015020 con 0x7fd0e0070780 2024-03-19T02:05:03.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.490+0000 7fd0c7fff640 1 -- 172.21.15.55:0/1842944578 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0d0011830 con 0x7fd0e0070780 2024-03-19T02:05:03.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.490+0000 7fd0e5da7640 1 -- 172.21.15.55:0/1842944578 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0e00c65f0 con 0x7fd0e0070780 2024-03-19T02:05:03.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.490+0000 7fd0c7fff640 1 -- 172.21.15.55:0/1842944578 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0011c20 con 0x7fd0e0070780 2024-03-19T02:05:03.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.490+0000 7fd0e5da7640 1 -- 172.21.15.55:0/1842944578 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0e01f3ff0 con 0x7fd0e0070780 2024-03-19T02:05:03.491 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.490+0000 7fd0e5da7640 1 -- 172.21.15.55:0/1842944578 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0e0068a40 con 0x7fd0e0070780 2024-03-19T02:05:03.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.770+0000 7fd0c7fff640 1 -- 172.21.15.55:0/1842944578 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fd0d0018020 con 0x7fd0e0070780 2024-03-19T02:05:03.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.770+0000 7fd0c7fff640 1 --2- 172.21.15.55:0/1842944578 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd0cc067690 0x7fd0cc069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:03.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.770+0000 7fd0deffd640 1 --2- 172.21.15.55:0/1842944578 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd0cc067690 0x7fd0cc069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:03.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.774+0000 7fd0deffd640 1 --2- 172.21.15.55:0/1842944578 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd0cc067690 0x7fd0cc069b50 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd0d40ad000 tx=0x7fd0d4004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:03.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.790+0000 7fd0c7fff640 1 -- 172.21.15.55:0/1842944578 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fd0d0053fb0 con 0x7fd0e0070780 2024-03-19T02:05:03.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:03.790+0000 7fd0c7fff640 1 -- 172.21.15.55:0/1842944578 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fd0d001f030 con 0x7fd0e0070780 2024-03-19T02:05:04.019 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:04.018+0000 7fd0e5da7640 1 -- 172.21.15.55:0/1842944578 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0e00c93e0 con 0x7fd0e0070780 2024-03-19T02:05:04.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:04.034+0000 7fd0c7fff640 1 -- 172.21.15.55:0/1842944578 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fd0d0056610 con 0x7fd0e0070780 2024-03-19T02:05:04.034 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:04.035 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:04.035 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:04.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:04.038+0000 7fd0e5da7640 1 -- 172.21.15.55:0/1842944578 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd0cc067690 msgr2=0x7fd0cc069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:04.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:04.038+0000 7fd0e5da7640 1 --2- 172.21.15.55:0/1842944578 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd0cc067690 0x7fd0cc069b50 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd0d40ad000 tx=0x7fd0d4004740 comp rx=0 tx=0).stop 2024-03-19T02:05:04.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:04.038+0000 7fd0e5da7640 1 -- 172.21.15.55:0/1842944578 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e0070780 msgr2=0x7fd0e01478d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:04.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:04.038+0000 7fd0e5da7640 1 --2- 172.21.15.55:0/1842944578 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e0070780 0x7fd0e01478d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000b440 tx=0x7fd0d000b900 comp rx=0 tx=0).stop 2024-03-19T02:05:04.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:04.038+0000 7fd0e5da7640 1 -- 172.21.15.55:0/1842944578 shutdown_connections 2024-03-19T02:05:04.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:04.038+0000 7fd0e5da7640 1 --2- 172.21.15.55:0/1842944578 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd0cc067690 0x7fd0cc069b50 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:04.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:04.038+0000 7fd0e5da7640 1 --2- 172.21.15.55:0/1842944578 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd0e015bb00 0x7fd0e006cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:04.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:04.038+0000 7fd0e5da7640 1 --2- 172.21.15.55:0/1842944578 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0e0152120 0x7fd0e01f7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:04.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:04.038+0000 7fd0e5da7640 1 --2- 172.21.15.55:0/1842944578 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd0e0070780 0x7fd0e01478d0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:04.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:04.038+0000 7fd0e5da7640 1 -- 172.21.15.55:0/1842944578 >> 172.21.15.55:0/1842944578 conn(0x7fd0e0067600 msgr2=0x7fd0e00c9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:04.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:04.038+0000 7fd0e5da7640 1 -- 172.21.15.55:0/1842944578 shutdown_connections 2024-03-19T02:05:04.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:04.038+0000 7fd0e5da7640 1 -- 172.21.15.55:0/1842944578 wait complete. 2024-03-19T02:05:04.659 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:04.660 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:05.661 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:05.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.782+0000 7f56f476b640 1 Processor -- start 2024-03-19T02:05:05.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.782+0000 7f56f476b640 1 -- start start 2024-03-19T02:05:05.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.782+0000 7f56f476b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56ec152120 0x7f56ec0c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:05.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.782+0000 7f56f476b640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f56ec0c9610 0x7f56ec157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:05.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.782+0000 7f56f476b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56ec06fff0 con 0x7f56ec070780 2024-03-19T02:05:05.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.782+0000 7f56f476b640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f56ec152500 con 0x7f56ec0c9610 2024-03-19T02:05:05.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.782+0000 7f56f476b640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f56ec070590 con 0x7f56ec152120 2024-03-19T02:05:05.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.782+0000 7f56f2507640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56ec152120 0x7f56ec0c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:05.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.782+0000 7f56f1d06640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f56ec0c9610 0x7f56ec157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:05.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.786+0000 7f56f2d08640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f56ec070780 0x7f56ec158510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:60596/0 (socket says 172.21.15.55:60596) 2024-03-19T02:05:05.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.786+0000 7f56f2d08640 1 -- 172.21.15.55:0/2941858695 learned_addr learned my addr 172.21.15.55:0/2941858695 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:05:05.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.790+0000 7f56f2507640 1 -- 172.21.15.55:0/2941858695 >> v2:172.21.15.84:3300/0 conn(0x7f56ec0c9610 msgr2=0x7f56ec157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:05.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.790+0000 7f56f2507640 1 --2- 172.21.15.55:0/2941858695 >> v2:172.21.15.84:3300/0 conn(0x7f56ec0c9610 0x7f56ec157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:05.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.790+0000 7f56f2507640 1 -- 172.21.15.55:0/2941858695 >> v1:172.21.15.84:6789/0 conn(0x7f56ec070780 legacy=0x7f56ec158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:05.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.790+0000 7f56f2507640 1 -- 172.21.15.55:0/2941858695 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56ec0c39c0 con 0x7f56ec152120 2024-03-19T02:05:05.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.790+0000 7f56f2507640 1 --2- 172.21.15.55:0/2941858695 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56ec152120 0x7f56ec0c90d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f56e80097c0 tx=0x7f56e801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dce701a310a096d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:05.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.798+0000 7f56f1505640 1 -- 172.21.15.55:0/2941858695 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e80b0070 con 0x7f56ec152120 2024-03-19T02:05:05.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.798+0000 7f56f1505640 1 -- 172.21.15.55:0/2941858695 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56e80ab600 con 0x7f56ec152120 2024-03-19T02:05:05.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.798+0000 7f56f1505640 1 -- 172.21.15.55:0/2941858695 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e80ab9d0 con 0x7f56ec152120 2024-03-19T02:05:05.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.798+0000 7f56f476b640 1 -- 172.21.15.55:0/2941858695 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56ec152120 msgr2=0x7f56ec0c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:05.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.798+0000 7f56f476b640 1 --2- 172.21.15.55:0/2941858695 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56ec152120 0x7f56ec0c90d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f56e80097c0 tx=0x7f56e801c4a0 comp rx=0 tx=0).stop 2024-03-19T02:05:05.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.798+0000 7f56f476b640 1 -- 172.21.15.55:0/2941858695 shutdown_connections 2024-03-19T02:05:05.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.798+0000 7f56f476b640 1 --2- 172.21.15.55:0/2941858695 >> v2:172.21.15.84:3300/0 conn(0x7f56ec0c9610 0x7f56ec157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:05.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.798+0000 7f56f476b640 1 --2- 172.21.15.55:0/2941858695 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56ec152120 0x7f56ec0c90d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:05.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.798+0000 7f56f476b640 1 -- 172.21.15.55:0/2941858695 >> 172.21.15.55:0/2941858695 conn(0x7f56ec067600 msgr2=0x7f56ec071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:05.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.798+0000 7f56f476b640 1 -- 172.21.15.55:0/2941858695 shutdown_connections 2024-03-19T02:05:05.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.798+0000 7f56f476b640 1 -- 172.21.15.55:0/2941858695 wait complete. 2024-03-19T02:05:05.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.798+0000 7f56f476b640 1 Processor -- start 2024-03-19T02:05:05.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.798+0000 7f56f476b640 1 -- start start 2024-03-19T02:05:05.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.798+0000 7f56f476b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56ec070780 0x7f56ec0ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:05.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.798+0000 7f56f476b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f56ec0c9610 0x7f56ec140310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:05.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.798+0000 7f56f476b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56ec152120 0x7f56ec140850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:05.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.798+0000 7f56f476b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f56ec0c86d0 con 0x7f56ec070780 2024-03-19T02:05:05.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.798+0000 7f56f476b640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f56ec0c31c0 con 0x7f56ec0c9610 2024-03-19T02:05:05.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.798+0000 7f56f476b640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f56ec165170 con 0x7f56ec152120 2024-03-19T02:05:05.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.798+0000 7f56f2507640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56ec070780 0x7f56ec0ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:05.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.802+0000 7f56f1d06640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f56ec0c9610 0x7f56ec140310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:05.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.802+0000 7f56f2507640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56ec070780 0x7f56ec0ca1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33994/0 (socket says 172.21.15.55:33994) 2024-03-19T02:05:05.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.802+0000 7f56f2507640 1 -- 172.21.15.55:0/1189927249 learned_addr learned my addr 172.21.15.55:0/1189927249 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:05.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.802+0000 7f56f2d08640 1 --2- 172.21.15.55:0/1189927249 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56ec152120 0x7f56ec140850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:05.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.806+0000 7f56f2507640 1 -- 172.21.15.55:0/1189927249 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56ec152120 msgr2=0x7f56ec140850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:05.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.806+0000 7f56f2507640 1 --2- 172.21.15.55:0/1189927249 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56ec152120 0x7f56ec140850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:05.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.806+0000 7f56f2507640 1 -- 172.21.15.55:0/1189927249 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f56ec0c9610 msgr2=0x7f56ec140310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:05.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.806+0000 7f56f2507640 1 --2- 172.21.15.55:0/1189927249 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f56ec0c9610 0x7f56ec140310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:05.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.806+0000 7f56f2507640 1 -- 172.21.15.55:0/1189927249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56ec1f37e0 con 0x7f56ec070780 2024-03-19T02:05:05.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.810+0000 7f56f2507640 1 --2- 172.21.15.55:0/1189927249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56ec070780 0x7f56ec0ca1b0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f56e801a0d0 tx=0x7f56e8019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:05.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.814+0000 7f56e2ffd640 1 -- 172.21.15.55:0/1189927249 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e80b0040 con 0x7f56ec070780 2024-03-19T02:05:05.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.814+0000 7f56e2ffd640 1 -- 172.21.15.55:0/1189927249 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56e80be070 con 0x7f56ec070780 2024-03-19T02:05:05.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.814+0000 7f56f476b640 1 -- 172.21.15.55:0/1189927249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56ec0cdb50 con 0x7f56ec070780 2024-03-19T02:05:05.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.814+0000 7f56e2ffd640 1 -- 172.21.15.55:0/1189927249 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e80b6680 con 0x7f56ec070780 2024-03-19T02:05:05.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.814+0000 7f56f476b640 1 -- 172.21.15.55:0/1189927249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56ec1f3a60 con 0x7f56ec070780 2024-03-19T02:05:05.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.814+0000 7f56f476b640 1 -- 172.21.15.55:0/1189927249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56ec1f46e0 con 0x7f56ec070780 2024-03-19T02:05:05.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.834+0000 7f56e2ffd640 1 -- 172.21.15.55:0/1189927249 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f56e80b6820 con 0x7f56ec070780 2024-03-19T02:05:05.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.838+0000 7f56e2ffd640 1 --2- 172.21.15.55:0/1189927249 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f56d00676b0 0x7f56d0069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:05.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.838+0000 7f56f1d06640 1 --2- 172.21.15.55:0/1189927249 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f56d00676b0 0x7f56d0069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:05.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:05.838+0000 7f56f1d06640 1 --2- 172.21.15.55:0/1189927249 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f56d00676b0 0x7f56d0069b70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f56ec14d8a0 tx=0x7f56dc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:06.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:06.098+0000 7f56e2ffd640 1 -- 172.21.15.55:0/1189927249 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f56e80eee60 con 0x7f56ec070780 2024-03-19T02:05:06.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:06.098+0000 7f56e2ffd640 1 -- 172.21.15.55:0/1189927249 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f56ec1f46e0 con 0x7f56ec070780 2024-03-19T02:05:06.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:06.298+0000 7f56f476b640 1 -- 172.21.15.55:0/1189927249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56ec068a40 con 0x7f56ec070780 2024-03-19T02:05:06.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:06.310+0000 7f56e2ffd640 1 -- 172.21.15.55:0/1189927249 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f56e80c8020 con 0x7f56ec070780 2024-03-19T02:05:06.312 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:06.312 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:06.312 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:06.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:06.314+0000 7f56f476b640 1 -- 172.21.15.55:0/1189927249 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f56d00676b0 msgr2=0x7f56d0069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:06.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:06.314+0000 7f56f476b640 1 --2- 172.21.15.55:0/1189927249 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f56d00676b0 0x7f56d0069b70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f56ec14d8a0 tx=0x7f56dc009190 comp rx=0 tx=0).stop 2024-03-19T02:05:06.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:06.314+0000 7f56f476b640 1 -- 172.21.15.55:0/1189927249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56ec070780 msgr2=0x7f56ec0ca1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:06.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:06.314+0000 7f56f476b640 1 --2- 172.21.15.55:0/1189927249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56ec070780 0x7f56ec0ca1b0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f56e801a0d0 tx=0x7f56e8019240 comp rx=0 tx=0).stop 2024-03-19T02:05:06.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:06.314+0000 7f56f476b640 1 -- 172.21.15.55:0/1189927249 shutdown_connections 2024-03-19T02:05:06.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:06.314+0000 7f56f476b640 1 --2- 172.21.15.55:0/1189927249 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f56d00676b0 0x7f56d0069b70 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:06.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:06.314+0000 7f56f476b640 1 --2- 172.21.15.55:0/1189927249 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f56ec152120 0x7f56ec140850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:06.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:06.314+0000 7f56f476b640 1 --2- 172.21.15.55:0/1189927249 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f56ec0c9610 0x7f56ec140310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:06.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:06.314+0000 7f56f476b640 1 --2- 172.21.15.55:0/1189927249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56ec070780 0x7f56ec0ca1b0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:06.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:06.314+0000 7f56f476b640 1 -- 172.21.15.55:0/1189927249 >> 172.21.15.55:0/1189927249 conn(0x7f56ec067600 msgr2=0x7f56ec146830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:06.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:06.314+0000 7f56f476b640 1 -- 172.21.15.55:0/1189927249 shutdown_connections 2024-03-19T02:05:06.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:06.314+0000 7f56f476b640 1 -- 172.21.15.55:0/1189927249 wait complete. 2024-03-19T02:05:06.934 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:06.935 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:07.935 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:08.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.066+0000 7fa0beb74640 1 Processor -- start 2024-03-19T02:05:08.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.066+0000 7fa0beb74640 1 -- start start 2024-03-19T02:05:08.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.066+0000 7fa0beb74640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa0b806c6f0 0x7fa0b815d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:08.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.066+0000 7fa0beb74640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa0b80cf1e0 0x7fa0b815de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:08.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.066+0000 7fa0beb74640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0b806cb90 con 0x7fa0b80cbf30 2024-03-19T02:05:08.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.066+0000 7fa0beb74640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fa0b80ce640 con 0x7fa0b806c6f0 2024-03-19T02:05:08.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.066+0000 7fa0beb74640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fa0b80cda30 con 0x7fa0b80cf1e0 2024-03-19T02:05:08.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.066+0000 7fa0b7fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa0b806c6f0 0x7fa0b815d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:08.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.066+0000 7fa0b7fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa0b806c6f0 0x7fa0b815d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34006/0 (socket says 172.21.15.55:34006) 2024-03-19T02:05:08.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.066+0000 7fa0b7fff640 1 -- 172.21.15.55:0/1201513080 learned_addr learned my addr 172.21.15.55:0/1201513080 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:08.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.066+0000 7fa0bd111640 1 --2- 172.21.15.55:0/1201513080 >> v2:172.21.15.84:3300/0 conn(0x7fa0b80cf1e0 0x7fa0b815de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:08.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.070+0000 7fa0b7fff640 1 -- 172.21.15.55:0/1201513080 >> v2:172.21.15.84:3300/0 conn(0x7fa0b80cf1e0 msgr2=0x7fa0b815de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:08.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.070+0000 7fa0b7fff640 1 --2- 172.21.15.55:0/1201513080 >> v2:172.21.15.84:3300/0 conn(0x7fa0b80cf1e0 0x7fa0b815de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:08.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.070+0000 7fa0b7fff640 1 -- 172.21.15.55:0/1201513080 >> v1:172.21.15.84:6789/0 conn(0x7fa0b80cbf30 legacy=0x7fa0b80cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:08.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.070+0000 7fa0b7fff640 1 -- 172.21.15.55:0/1201513080 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b80cdd10 con 0x7fa0b806c6f0 2024-03-19T02:05:08.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.074+0000 7fa0b7fff640 1 --2- 172.21.15.55:0/1201513080 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0b806c6f0 0x7fa0b815d8d0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80af5b0 tx=0x7fa0ac01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d924f328ad26e24 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:08.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.078+0000 7fa0b77fe640 1 -- 172.21.15.55:0/1201513080 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac0ab620 con 0x7fa0b806c6f0 2024-03-19T02:05:08.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.078+0000 7fa0b77fe640 1 -- 172.21.15.55:0/1201513080 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0ac0b0070 con 0x7fa0b806c6f0 2024-03-19T02:05:08.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.078+0000 7fa0b77fe640 1 -- 172.21.15.55:0/1201513080 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac0ab9e0 con 0x7fa0b806c6f0 2024-03-19T02:05:08.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.078+0000 7fa0beb74640 1 -- 172.21.15.55:0/1201513080 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0b806c6f0 msgr2=0x7fa0b815d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:08.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.078+0000 7fa0beb74640 1 --2- 172.21.15.55:0/1201513080 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0b806c6f0 0x7fa0b815d8d0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80af5b0 tx=0x7fa0ac01c420 comp rx=0 tx=0).stop 2024-03-19T02:05:08.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.078+0000 7fa0beb74640 1 -- 172.21.15.55:0/1201513080 shutdown_connections 2024-03-19T02:05:08.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.078+0000 7fa0beb74640 1 --2- 172.21.15.55:0/1201513080 >> v2:172.21.15.84:3300/0 conn(0x7fa0b80cf1e0 0x7fa0b815de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:08.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.078+0000 7fa0beb74640 1 --2- 172.21.15.55:0/1201513080 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0b806c6f0 0x7fa0b815d8d0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:08.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.078+0000 7fa0beb74640 1 -- 172.21.15.55:0/1201513080 >> 172.21.15.55:0/1201513080 conn(0x7fa0b80b16a0 msgr2=0x7fa0b80b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:08.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.078+0000 7fa0beb74640 1 -- 172.21.15.55:0/1201513080 shutdown_connections 2024-03-19T02:05:08.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.078+0000 7fa0beb74640 1 -- 172.21.15.55:0/1201513080 wait complete. 2024-03-19T02:05:08.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.082+0000 7fa0beb74640 1 Processor -- start 2024-03-19T02:05:08.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.082+0000 7fa0beb74640 1 -- start start 2024-03-19T02:05:08.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.082+0000 7fa0beb74640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa0b806c6f0 0x7fa0b8155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:08.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.082+0000 7fa0beb74640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0b80cbf30 0x7fa0b80c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:08.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.082+0000 7fa0beb74640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa0b80cf1e0 0x7fa0b80c5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:08.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.082+0000 7fa0beb74640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b80b23b0 con 0x7fa0b80cbf30 2024-03-19T02:05:08.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.082+0000 7fa0beb74640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b806ce50 con 0x7fa0b80cf1e0 2024-03-19T02:05:08.082 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.082+0000 7fa0beb74640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa0b80cc310 con 0x7fa0b806c6f0 2024-03-19T02:05:08.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.082+0000 7fa0b7fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0b80cbf30 0x7fa0b80c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:08.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.082+0000 7fa0b7fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0b80cbf30 0x7fa0b80c56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34010/0 (socket says 172.21.15.55:34010) 2024-03-19T02:05:08.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.082+0000 7fa0b7fff640 1 -- 172.21.15.55:0/3190617514 learned_addr learned my addr 172.21.15.55:0/3190617514 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:08.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.082+0000 7fa0bd111640 1 --2- 172.21.15.55:0/3190617514 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa0b80cf1e0 0x7fa0b80c5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:08.083 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.082+0000 7fa0bc910640 1 --2- 172.21.15.55:0/3190617514 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa0b806c6f0 0x7fa0b8155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:08.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.086+0000 7fa0b7fff640 1 -- 172.21.15.55:0/3190617514 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa0b806c6f0 msgr2=0x7fa0b8155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:08.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.086+0000 7fa0b7fff640 1 --2- 172.21.15.55:0/3190617514 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa0b806c6f0 0x7fa0b8155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:08.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.086+0000 7fa0b7fff640 1 -- 172.21.15.55:0/3190617514 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa0b80cf1e0 msgr2=0x7fa0b80c5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:08.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.086+0000 7fa0b7fff640 1 --2- 172.21.15.55:0/3190617514 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa0b80cf1e0 0x7fa0b80c5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:08.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.086+0000 7fa0b7fff640 1 -- 172.21.15.55:0/3190617514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b8155000 con 0x7fa0b80cbf30 2024-03-19T02:05:08.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.090+0000 7fa0b7fff640 1 --2- 172.21.15.55:0/3190617514 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0b80cbf30 0x7fa0b80c56a0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac0ad040 tx=0x7fa0ac01ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:08.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.094+0000 7fa0b57fa640 1 -- 172.21.15.55:0/3190617514 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac0abc50 con 0x7fa0b80cbf30 2024-03-19T02:05:08.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.094+0000 7fa0b57fa640 1 -- 172.21.15.55:0/3190617514 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0ac0abe10 con 0x7fa0b80cbf30 2024-03-19T02:05:08.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.094+0000 7fa0beb74640 1 -- 172.21.15.55:0/3190617514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0b8154d20 con 0x7fa0b80cbf30 2024-03-19T02:05:08.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.094+0000 7fa0b57fa640 1 -- 172.21.15.55:0/3190617514 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac0be630 con 0x7fa0b80cbf30 2024-03-19T02:05:08.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.094+0000 7fa0beb74640 1 -- 172.21.15.55:0/3190617514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0b80c42a0 con 0x7fa0b80cbf30 2024-03-19T02:05:08.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.094+0000 7fa0beb74640 1 -- 172.21.15.55:0/3190617514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0b8068330 con 0x7fa0b80cbf30 2024-03-19T02:05:08.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.118+0000 7fa0b57fa640 1 -- 172.21.15.55:0/3190617514 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa0ac0c7440 con 0x7fa0b80cbf30 2024-03-19T02:05:08.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.118+0000 7fa0b57fa640 1 --2- 172.21.15.55:0/3190617514 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa0a4067660 0x7fa0a4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:08.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.118+0000 7fa0bc910640 1 --2- 172.21.15.55:0/3190617514 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa0a4067660 0x7fa0a4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:08.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.118+0000 7fa0bc910640 1 --2- 172.21.15.55:0/3190617514 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa0a4067660 0x7fa0a4069b20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa0a80098e0 tx=0x7fa0a8009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:08.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.126+0000 7fa0b57fa640 1 -- 172.21.15.55:0/3190617514 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fa0ac0ee7f0 con 0x7fa0b80cbf30 2024-03-19T02:05:08.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.458+0000 7fa0b57fa640 1 -- 172.21.15.55:0/3190617514 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa0ac0160d0 con 0x7fa0b80cbf30 2024-03-19T02:05:08.725 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.726+0000 7fa0beb74640 1 -- 172.21.15.55:0/3190617514 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0b81566f0 con 0x7fa0b80cbf30 2024-03-19T02:05:08.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.794+0000 7fa0b57fa640 1 -- 172.21.15.55:0/3190617514 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fa0ac0f0750 con 0x7fa0b80cbf30 2024-03-19T02:05:08.794 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:08.794 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:08.794 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:08.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.798+0000 7fa0beb74640 1 -- 172.21.15.55:0/3190617514 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa0a4067660 msgr2=0x7fa0a4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:08.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.798+0000 7fa0beb74640 1 --2- 172.21.15.55:0/3190617514 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa0a4067660 0x7fa0a4069b20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa0a80098e0 tx=0x7fa0a8009060 comp rx=0 tx=0).stop 2024-03-19T02:05:08.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.798+0000 7fa0beb74640 1 -- 172.21.15.55:0/3190617514 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0b80cbf30 msgr2=0x7fa0b80c56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:08.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.798+0000 7fa0beb74640 1 --2- 172.21.15.55:0/3190617514 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0b80cbf30 0x7fa0b80c56a0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac0ad040 tx=0x7fa0ac01ac60 comp rx=0 tx=0).stop 2024-03-19T02:05:08.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.798+0000 7fa0beb74640 1 -- 172.21.15.55:0/3190617514 shutdown_connections 2024-03-19T02:05:08.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.798+0000 7fa0beb74640 1 --2- 172.21.15.55:0/3190617514 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa0a4067660 0x7fa0a4069b20 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:08.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.798+0000 7fa0beb74640 1 --2- 172.21.15.55:0/3190617514 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa0b80cf1e0 0x7fa0b80c5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:08.797 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.798+0000 7fa0beb74640 1 --2- 172.21.15.55:0/3190617514 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa0b80cbf30 0x7fa0b80c56a0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:08.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.798+0000 7fa0beb74640 1 --2- 172.21.15.55:0/3190617514 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa0b806c6f0 0x7fa0b8155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:08.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.798+0000 7fa0beb74640 1 -- 172.21.15.55:0/3190617514 >> 172.21.15.55:0/3190617514 conn(0x7fa0b80b16a0 msgr2=0x7fa0b80ca880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:08.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.798+0000 7fa0beb74640 1 -- 172.21.15.55:0/3190617514 shutdown_connections 2024-03-19T02:05:08.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:08.798+0000 7fa0beb74640 1 -- 172.21.15.55:0/3190617514 wait complete. 2024-03-19T02:05:09.426 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:09.427 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:10.428 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:10.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.554+0000 7f68fa4ec640 1 Processor -- start 2024-03-19T02:05:10.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.554+0000 7f68fa4ec640 1 -- start start 2024-03-19T02:05:10.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.554+0000 7f68fa4ec640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f68f4154650 0x7f68f415ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:10.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.554+0000 7f68fa4ec640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f68f415e370 0x7f68f40c9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:10.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.554+0000 7f68fa4ec640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f4073790 con 0x7f68f4154270 2024-03-19T02:05:10.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.554+0000 7f68fa4ec640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f68f415d7d0 con 0x7f68f415e370 2024-03-19T02:05:10.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.554+0000 7f68fa4ec640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f68f415b090 con 0x7f68f4154650 2024-03-19T02:05:10.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.558+0000 7f68f8a89640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f68f415e370 0x7f68f40c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:10.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.558+0000 7f68f37fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f68f4154650 0x7f68f415ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:10.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.558+0000 7f68f8a89640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f68f415e370 0x7f68f40c9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59812/0 (socket says 172.21.15.55:59812) 2024-03-19T02:05:10.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.558+0000 7f68f8a89640 1 -- 172.21.15.55:0/2764346081 learned_addr learned my addr 172.21.15.55:0/2764346081 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:10.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.558+0000 7f68f8a89640 1 -- 172.21.15.55:0/2764346081 >> v2:172.21.15.84:3300/0 conn(0x7f68f4154650 msgr2=0x7f68f415ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:10.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.558+0000 7f68f8a89640 1 --2- 172.21.15.55:0/2764346081 >> v2:172.21.15.84:3300/0 conn(0x7f68f4154650 0x7f68f415ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:10.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.558+0000 7f68f8a89640 1 -- 172.21.15.55:0/2764346081 >> v1:172.21.15.84:6789/0 conn(0x7f68f4154270 legacy=0x7f68f415dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:10.560 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.558+0000 7f68f8a89640 1 -- 172.21.15.55:0/2764346081 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f4073370 con 0x7f68f415e370 2024-03-19T02:05:10.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.562+0000 7f68f8a89640 1 --2- 172.21.15.55:0/2764346081 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f415e370 0x7f68f40c9150 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f68f406fa20 tx=0x7f68ec01f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac0f44358c6794a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:10.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.566+0000 7f68f2ffd640 1 -- 172.21.15.55:0/2764346081 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec0b4070 con 0x7f68f415e370 2024-03-19T02:05:10.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.566+0000 7f68f2ffd640 1 -- 172.21.15.55:0/2764346081 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68ec0af400 con 0x7f68f415e370 2024-03-19T02:05:10.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.566+0000 7f68f2ffd640 1 -- 172.21.15.55:0/2764346081 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec0af7d0 con 0x7f68f415e370 2024-03-19T02:05:10.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.566+0000 7f68fa4ec640 1 -- 172.21.15.55:0/2764346081 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f415e370 msgr2=0x7f68f40c9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:10.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.566+0000 7f68fa4ec640 1 --2- 172.21.15.55:0/2764346081 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f415e370 0x7f68f40c9150 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f68f406fa20 tx=0x7f68ec01f1a0 comp rx=0 tx=0).stop 2024-03-19T02:05:10.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.566+0000 7f68fa4ec640 1 -- 172.21.15.55:0/2764346081 shutdown_connections 2024-03-19T02:05:10.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.566+0000 7f68fa4ec640 1 --2- 172.21.15.55:0/2764346081 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f415e370 0x7f68f40c9150 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:10.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.566+0000 7f68fa4ec640 1 --2- 172.21.15.55:0/2764346081 >> v2:172.21.15.84:3300/0 conn(0x7f68f4154650 0x7f68f415ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:10.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.566+0000 7f68fa4ec640 1 -- 172.21.15.55:0/2764346081 >> 172.21.15.55:0/2764346081 conn(0x7f68f40cc9c0 msgr2=0x7f68f40ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:10.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.570+0000 7f68fa4ec640 1 -- 172.21.15.55:0/2764346081 shutdown_connections 2024-03-19T02:05:10.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.570+0000 7f68fa4ec640 1 -- 172.21.15.55:0/2764346081 wait complete. 2024-03-19T02:05:10.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.570+0000 7f68fa4ec640 1 Processor -- start 2024-03-19T02:05:10.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.570+0000 7f68fa4ec640 1 -- start start 2024-03-19T02:05:10.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.570+0000 7f68fa4ec640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68f4154270 0x7f68f415c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:10.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.570+0000 7f68fa4ec640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68f415e370 0x7f68f40c5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:10.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.570+0000 7f68fa4ec640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f414df00 0x7f68f40c5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:10.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.570+0000 7f68fa4ec640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f68f4070ea0 con 0x7f68f414df00 2024-03-19T02:05:10.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.570+0000 7f68fa4ec640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f68f4070bc0 con 0x7f68f415e370 2024-03-19T02:05:10.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.570+0000 7f68fa4ec640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f68f4072d50 con 0x7f68f4154270 2024-03-19T02:05:10.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.570+0000 7f68f8a89640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f414df00 0x7f68f40c5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:10.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.570+0000 7f68f37fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68f415e370 0x7f68f40c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:10.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.570+0000 7f68f8a89640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f414df00 0x7f68f40c5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59828/0 (socket says 172.21.15.55:59828) 2024-03-19T02:05:10.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.570+0000 7f68f8a89640 1 -- 172.21.15.55:0/3562569056 learned_addr learned my addr 172.21.15.55:0/3562569056 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:10.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.570+0000 7f68f3fff640 1 --2- 172.21.15.55:0/3562569056 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68f4154270 0x7f68f415c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:10.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.574+0000 7f68f8a89640 1 -- 172.21.15.55:0/3562569056 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68f4154270 msgr2=0x7f68f415c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:10.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.574+0000 7f68f8a89640 1 --2- 172.21.15.55:0/3562569056 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68f4154270 0x7f68f415c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:10.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.574+0000 7f68f8a89640 1 -- 172.21.15.55:0/3562569056 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68f415e370 msgr2=0x7f68f40c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:10.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.574+0000 7f68f8a89640 1 --2- 172.21.15.55:0/3562569056 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68f415e370 0x7f68f40c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:10.575 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.574+0000 7f68f8a89640 1 -- 172.21.15.55:0/3562569056 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f41f0210 con 0x7f68f414df00 2024-03-19T02:05:10.577 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.578+0000 7f68f8a89640 1 --2- 172.21.15.55:0/3562569056 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f414df00 0x7f68f40c5b50 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f68ec01cdd0 tx=0x7f68ec01bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:10.582 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.582+0000 7f68f0ff9640 1 -- 172.21.15.55:0/3562569056 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec0b4040 con 0x7f68f414df00 2024-03-19T02:05:10.582 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.582+0000 7f68f0ff9640 1 -- 172.21.15.55:0/3562569056 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68ec0c1070 con 0x7f68f414df00 2024-03-19T02:05:10.582 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.582+0000 7f68f0ff9640 1 -- 172.21.15.55:0/3562569056 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec0ba470 con 0x7f68f414df00 2024-03-19T02:05:10.582 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.582+0000 7f68fa4ec640 1 -- 172.21.15.55:0/3562569056 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68f41f04f0 con 0x7f68f414df00 2024-03-19T02:05:10.582 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.582+0000 7f68fa4ec640 1 -- 172.21.15.55:0/3562569056 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68f41f0ab0 con 0x7f68f414df00 2024-03-19T02:05:10.583 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.582+0000 7f68fa4ec640 1 -- 172.21.15.55:0/3562569056 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68f4068330 con 0x7f68f414df00 2024-03-19T02:05:10.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.602+0000 7f68f0ff9640 1 -- 172.21.15.55:0/3562569056 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f68ec0ba610 con 0x7f68f414df00 2024-03-19T02:05:10.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.602+0000 7f68f0ff9640 1 --2- 172.21.15.55:0/3562569056 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68e00676b0 0x7f68e0069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:10.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.606+0000 7f68f3fff640 1 --2- 172.21.15.55:0/3562569056 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68e00676b0 0x7f68e0069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:10.604 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.606+0000 7f68f3fff640 1 --2- 172.21.15.55:0/3562569056 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68e00676b0 0x7f68e0069b70 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f68e4007fd0 tx=0x7f68e4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:10.890 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.890+0000 7f68f0ff9640 1 -- 172.21.15.55:0/3562569056 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f68ec0f2bd0 con 0x7f68f414df00 2024-03-19T02:05:10.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:10.890+0000 7f68f0ff9640 1 -- 172.21.15.55:0/3562569056 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f68ec018d00 con 0x7f68f414df00 2024-03-19T02:05:11.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:11.110+0000 7f68fa4ec640 1 -- 172.21.15.55:0/3562569056 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68f40cc150 con 0x7f68f414df00 2024-03-19T02:05:11.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:11.134+0000 7f68f0ff9640 1 -- 172.21.15.55:0/3562569056 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f68ec0f4b30 con 0x7f68f414df00 2024-03-19T02:05:11.141 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:11.141 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:11.142 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:11.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:11.138+0000 7f68fa4ec640 1 -- 172.21.15.55:0/3562569056 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68e00676b0 msgr2=0x7f68e0069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:11.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:11.138+0000 7f68fa4ec640 1 --2- 172.21.15.55:0/3562569056 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68e00676b0 0x7f68e0069b70 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f68e4007fd0 tx=0x7f68e4009040 comp rx=0 tx=0).stop 2024-03-19T02:05:11.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:11.138+0000 7f68fa4ec640 1 -- 172.21.15.55:0/3562569056 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f414df00 msgr2=0x7f68f40c5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:11.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:11.138+0000 7f68fa4ec640 1 --2- 172.21.15.55:0/3562569056 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f414df00 0x7f68f40c5b50 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f68ec01cdd0 tx=0x7f68ec01bf40 comp rx=0 tx=0).stop 2024-03-19T02:05:11.142 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:11.138+0000 7f68fa4ec640 1 -- 172.21.15.55:0/3562569056 shutdown_connections 2024-03-19T02:05:11.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:11.138+0000 7f68fa4ec640 1 --2- 172.21.15.55:0/3562569056 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68e00676b0 0x7f68e0069b70 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:11.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:11.138+0000 7f68fa4ec640 1 --2- 172.21.15.55:0/3562569056 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68f414df00 0x7f68f40c5b50 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:11.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:11.138+0000 7f68fa4ec640 1 --2- 172.21.15.55:0/3562569056 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68f415e370 0x7f68f40c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:11.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:11.138+0000 7f68fa4ec640 1 --2- 172.21.15.55:0/3562569056 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68f4154270 0x7f68f415c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:11.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:11.138+0000 7f68fa4ec640 1 -- 172.21.15.55:0/3562569056 >> 172.21.15.55:0/3562569056 conn(0x7f68f40cc9c0 msgr2=0x7f68f4153050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:11.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:11.138+0000 7f68fa4ec640 1 -- 172.21.15.55:0/3562569056 shutdown_connections 2024-03-19T02:05:11.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:11.138+0000 7f68fa4ec640 1 -- 172.21.15.55:0/3562569056 wait complete. 2024-03-19T02:05:11.150 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:05:11.152 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:05:11.759 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:11.759 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:12.760 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:12.882 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.882+0000 7fa2697b5640 1 Processor -- start 2024-03-19T02:05:12.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.882+0000 7fa2697b5640 1 -- start start 2024-03-19T02:05:12.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.882+0000 7fa2697b5640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fa26406c9f0 0x7fa2640c92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:12.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.882+0000 7fa2697b5640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa2640c9800 0x7fa2641427f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:12.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.882+0000 7fa2697b5640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa264070e80 con 0x7fa26406c610 2024-03-19T02:05:12.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.882+0000 7fa2697b5640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fa26406fb00 con 0x7fa26406c9f0 2024-03-19T02:05:12.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.882+0000 7fa2697b5640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa26406fde0 con 0x7fa2640c9800 2024-03-19T02:05:12.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.886+0000 7fa262ffd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fa26406c610 0x7fa26415ad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:41550/0 (socket says 172.21.15.55:41550) 2024-03-19T02:05:12.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.886+0000 7fa262ffd640 1 -- 172.21.15.55:0/3990744019 learned_addr learned my addr 172.21.15.55:0/3990744019 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:05:12.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.886+0000 7fa2627fc640 1 --2- 172.21.15.55:0/3990744019 >> v2:172.21.15.84:3300/0 conn(0x7fa26406c9f0 0x7fa2640c92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:12.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.886+0000 7fa2637fe640 1 --2- 172.21.15.55:0/3990744019 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa2640c9800 0x7fa2641427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:12.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.886+0000 7fa261ffb640 1 -- 172.21.15.55:0/3990744019 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2836796836 0 0) 0x7fa264070e80 con 0x7fa26406c610 2024-03-19T02:05:12.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.886+0000 7fa261ffb640 1 -- 172.21.15.55:0/3990744019 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa24c003530 con 0x7fa26406c610 2024-03-19T02:05:12.888 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.890+0000 7fa2627fc640 1 -- 172.21.15.55:0/3990744019 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa2640c9800 msgr2=0x7fa2641427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:12.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.890+0000 7fa2627fc640 1 --2- 172.21.15.55:0/3990744019 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa2640c9800 0x7fa2641427f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:12.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.890+0000 7fa2627fc640 1 -- 172.21.15.55:0/3990744019 >> v1:172.21.15.55:6789/0 conn(0x7fa26406c610 legacy=0x7fa26415ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:12.889 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.890+0000 7fa2627fc640 1 -- 172.21.15.55:0/3990744019 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2640700c0 con 0x7fa26406c9f0 2024-03-19T02:05:12.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.894+0000 7fa2627fc640 1 --2- 172.21.15.55:0/3990744019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa26406c9f0 0x7fa2640c92c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa2580097c0 tx=0x7fa25801ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c525679520d8a53 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:12.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.898+0000 7fa261ffb640 1 -- 172.21.15.55:0/3990744019 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2580b5070 con 0x7fa26406c9f0 2024-03-19T02:05:12.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.902+0000 7fa261ffb640 1 -- 172.21.15.55:0/3990744019 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2580a8990 con 0x7fa26406c9f0 2024-03-19T02:05:12.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.902+0000 7fa2697b5640 1 -- 172.21.15.55:0/3990744019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa26406c9f0 msgr2=0x7fa2640c92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:12.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.902+0000 7fa2697b5640 1 --2- 172.21.15.55:0/3990744019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa26406c9f0 0x7fa2640c92c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa2580097c0 tx=0x7fa25801ec90 comp rx=0 tx=0).stop 2024-03-19T02:05:12.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.902+0000 7fa2697b5640 1 -- 172.21.15.55:0/3990744019 shutdown_connections 2024-03-19T02:05:12.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.902+0000 7fa2697b5640 1 --2- 172.21.15.55:0/3990744019 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa2640c9800 0x7fa2641427f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:12.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.902+0000 7fa2697b5640 1 --2- 172.21.15.55:0/3990744019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa26406c9f0 0x7fa2640c92c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:12.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.902+0000 7fa2697b5640 1 -- 172.21.15.55:0/3990744019 >> 172.21.15.55:0/3990744019 conn(0x7fa264067600 msgr2=0x7fa264071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:12.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.902+0000 7fa2697b5640 1 -- 172.21.15.55:0/3990744019 shutdown_connections 2024-03-19T02:05:12.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.902+0000 7fa2697b5640 1 -- 172.21.15.55:0/3990744019 wait complete. 2024-03-19T02:05:12.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.902+0000 7fa2697b5640 1 Processor -- start 2024-03-19T02:05:12.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.902+0000 7fa2697b5640 1 -- start start 2024-03-19T02:05:12.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.902+0000 7fa2697b5640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa26406c610 0x7fa26413aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:12.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.902+0000 7fa2697b5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa26406c9f0 0x7fa26415c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:12.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.902+0000 7fa2697b5640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2640c9800 0x7fa26415c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:12.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.902+0000 7fa2697b5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa264070960 con 0x7fa26406c9f0 2024-03-19T02:05:12.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.902+0000 7fa2697b5640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa264070680 con 0x7fa2640c9800 2024-03-19T02:05:12.905 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.902+0000 7fa2697b5640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa26406ceb0 con 0x7fa26406c610 2024-03-19T02:05:12.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.906+0000 7fa2627fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa26406c9f0 0x7fa26415c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:12.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.906+0000 7fa262ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa26406c610 0x7fa26413aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:12.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.906+0000 7fa2627fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa26406c9f0 0x7fa26415c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59836/0 (socket says 172.21.15.55:59836) 2024-03-19T02:05:12.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.906+0000 7fa2627fc640 1 -- 172.21.15.55:0/3786413457 learned_addr learned my addr 172.21.15.55:0/3786413457 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:12.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.906+0000 7fa2637fe640 1 --2- 172.21.15.55:0/3786413457 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2640c9800 0x7fa26415c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:12.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.910+0000 7fa2627fc640 1 -- 172.21.15.55:0/3786413457 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa26406c610 msgr2=0x7fa26413aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:12.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.910+0000 7fa2627fc640 1 --2- 172.21.15.55:0/3786413457 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa26406c610 0x7fa26413aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:12.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.910+0000 7fa2627fc640 1 -- 172.21.15.55:0/3786413457 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2640c9800 msgr2=0x7fa26415c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:12.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.910+0000 7fa2627fc640 1 --2- 172.21.15.55:0/3786413457 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2640c9800 0x7fa26415c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:12.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.910+0000 7fa2627fc640 1 -- 172.21.15.55:0/3786413457 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa26413afa0 con 0x7fa26406c9f0 2024-03-19T02:05:12.911 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.910+0000 7fa2627fc640 1 --2- 172.21.15.55:0/3786413457 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa26406c9f0 0x7fa26415c170 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fa2580b3040 tx=0x7fa25801ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:12.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.914+0000 7fa24b7fe640 1 -- 172.21.15.55:0/3786413457 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2580b5040 con 0x7fa26406c9f0 2024-03-19T02:05:12.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.914+0000 7fa24b7fe640 1 -- 172.21.15.55:0/3786413457 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2580bf070 con 0x7fa26406c9f0 2024-03-19T02:05:12.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.914+0000 7fa24b7fe640 1 -- 172.21.15.55:0/3786413457 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa258003620 con 0x7fa26406c9f0 2024-03-19T02:05:12.915 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.914+0000 7fa2697b5640 1 -- 172.21.15.55:0/3786413457 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2641f0440 con 0x7fa26406c9f0 2024-03-19T02:05:12.916 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.914+0000 7fa2697b5640 1 -- 172.21.15.55:0/3786413457 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa26415b690 con 0x7fa26406c9f0 2024-03-19T02:05:12.916 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.918+0000 7fa2497fa640 1 -- 172.21.15.55:0/3786413457 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa23c0052a0 con 0x7fa26406c9f0 2024-03-19T02:05:12.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.942+0000 7fa24b7fe640 1 -- 172.21.15.55:0/3786413457 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa2580037c0 con 0x7fa26406c9f0 2024-03-19T02:05:12.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.942+0000 7fa24b7fe640 1 --2- 172.21.15.55:0/3786413457 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa250067690 0x7fa250069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:12.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.942+0000 7fa262ffd640 1 --2- 172.21.15.55:0/3786413457 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa250067690 0x7fa250069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:12.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:12.942+0000 7fa262ffd640 1 --2- 172.21.15.55:0/3786413457 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa250067690 0x7fa250069b50 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa254007fd0 tx=0x7fa254009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:13.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:13.238+0000 7fa24b7fe640 1 -- 172.21.15.55:0/3786413457 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fa2580f1cf0 con 0x7fa26406c9f0 2024-03-19T02:05:13.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:13.238+0000 7fa24b7fe640 1 -- 172.21.15.55:0/3786413457 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa2580187f0 con 0x7fa26406c9f0 2024-03-19T02:05:13.479 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:13.478+0000 7fa2497fa640 1 -- 172.21.15.55:0/3786413457 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa23c005590 con 0x7fa26406c9f0 2024-03-19T02:05:13.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:13.566+0000 7fa24b7fe640 1 -- 172.21.15.55:0/3786413457 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fa2580f3c50 con 0x7fa26406c9f0 2024-03-19T02:05:13.567 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:13.567 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:13.567 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:13.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:13.570+0000 7fa2497fa640 1 -- 172.21.15.55:0/3786413457 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa250067690 msgr2=0x7fa250069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:13.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:13.570+0000 7fa2497fa640 1 --2- 172.21.15.55:0/3786413457 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa250067690 0x7fa250069b50 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa254007fd0 tx=0x7fa254009040 comp rx=0 tx=0).stop 2024-03-19T02:05:13.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:13.570+0000 7fa2497fa640 1 -- 172.21.15.55:0/3786413457 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa26406c9f0 msgr2=0x7fa26415c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:13.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:13.570+0000 7fa2497fa640 1 --2- 172.21.15.55:0/3786413457 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa26406c9f0 0x7fa26415c170 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fa2580b3040 tx=0x7fa25801ba30 comp rx=0 tx=0).stop 2024-03-19T02:05:13.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:13.570+0000 7fa2497fa640 1 -- 172.21.15.55:0/3786413457 shutdown_connections 2024-03-19T02:05:13.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:13.570+0000 7fa2497fa640 1 --2- 172.21.15.55:0/3786413457 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa250067690 0x7fa250069b50 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:13.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:13.570+0000 7fa2497fa640 1 --2- 172.21.15.55:0/3786413457 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2640c9800 0x7fa26415c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:13.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:13.570+0000 7fa2497fa640 1 --2- 172.21.15.55:0/3786413457 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa26406c9f0 0x7fa26415c170 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:13.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:13.570+0000 7fa2497fa640 1 --2- 172.21.15.55:0/3786413457 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa26406c610 0x7fa26413aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:13.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:13.570+0000 7fa2497fa640 1 -- 172.21.15.55:0/3786413457 >> 172.21.15.55:0/3786413457 conn(0x7fa264067600 msgr2=0x7fa26413db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:13.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:13.570+0000 7fa2497fa640 1 -- 172.21.15.55:0/3786413457 shutdown_connections 2024-03-19T02:05:13.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:13.570+0000 7fa2497fa640 1 -- 172.21.15.55:0/3786413457 wait complete. 2024-03-19T02:05:14.191 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:14.191 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:15.193 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:15.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.322+0000 7ff0cef6d640 1 Processor -- start 2024-03-19T02:05:15.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.322+0000 7ff0cef6d640 1 -- start start 2024-03-19T02:05:15.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.326+0000 7ff0cef6d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0c8154340 0x7ff0c815e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:15.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.326+0000 7ff0cef6d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff0c8154720 0x7ff0c80c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:15.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.326+0000 7ff0cef6d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff0c815aca0 0x7ff0c8072e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:15.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.326+0000 7ff0cef6d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7ff0c8073840 con 0x7ff0c815aca0 2024-03-19T02:05:15.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.326+0000 7ff0cef6d640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7ff0c815df10 con 0x7ff0c8154720 2024-03-19T02:05:15.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.326+0000 7ff0cef6d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff0c8071280 con 0x7ff0c8154340 2024-03-19T02:05:15.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.326+0000 7ff0cd50a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff0c815aca0 0x7ff0c8072e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:15.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.326+0000 7ff0c7fff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff0c8154720 0x7ff0c80c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:15.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.326+0000 7ff0cd50a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff0c815aca0 0x7ff0c8072e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59840/0 (socket says 172.21.15.55:59840) 2024-03-19T02:05:15.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.326+0000 7ff0cd50a640 1 -- 172.21.15.55:0/2944114730 learned_addr learned my addr 172.21.15.55:0/2944114730 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:15.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.326+0000 7ff0ccd09640 1 --2- 172.21.15.55:0/2944114730 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0c8154340 0x7ff0c815e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:15.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.330+0000 7ff0cd50a640 1 -- 172.21.15.55:0/2944114730 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0c8154340 msgr2=0x7ff0c815e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:15.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.330+0000 7ff0cd50a640 1 --2- 172.21.15.55:0/2944114730 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0c8154340 0x7ff0c815e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:15.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.330+0000 7ff0cd50a640 1 -- 172.21.15.55:0/2944114730 >> v2:172.21.15.84:3300/0 conn(0x7ff0c8154720 msgr2=0x7ff0c80c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:15.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.330+0000 7ff0cd50a640 1 --2- 172.21.15.55:0/2944114730 >> v2:172.21.15.84:3300/0 conn(0x7ff0c8154720 0x7ff0c80c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:15.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.330+0000 7ff0cd50a640 1 -- 172.21.15.55:0/2944114730 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c815d900 con 0x7ff0c815aca0 2024-03-19T02:05:15.331 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.330+0000 7ff0cd50a640 1 --2- 172.21.15.55:0/2944114730 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0c815aca0 0x7ff0c8072e90 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8071fc0 tx=0x7ff0c001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10ca5cba682d8b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:15.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.334+0000 7ff0c77fe640 1 -- 172.21.15.55:0/2944114730 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c00b1070 con 0x7ff0c815aca0 2024-03-19T02:05:15.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.338+0000 7ff0c77fe640 1 -- 172.21.15.55:0/2944114730 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0c00ac630 con 0x7ff0c815aca0 2024-03-19T02:05:15.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.338+0000 7ff0c77fe640 1 -- 172.21.15.55:0/2944114730 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c00a8050 con 0x7ff0c815aca0 2024-03-19T02:05:15.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.338+0000 7ff0cef6d640 1 -- 172.21.15.55:0/2944114730 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0c815aca0 msgr2=0x7ff0c8072e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:15.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.338+0000 7ff0cef6d640 1 --2- 172.21.15.55:0/2944114730 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0c815aca0 0x7ff0c8072e90 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8071fc0 tx=0x7ff0c001c520 comp rx=0 tx=0).stop 2024-03-19T02:05:15.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.338+0000 7ff0cef6d640 1 -- 172.21.15.55:0/2944114730 shutdown_connections 2024-03-19T02:05:15.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.338+0000 7ff0cef6d640 1 --2- 172.21.15.55:0/2944114730 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0c815aca0 0x7ff0c8072e90 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:15.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.338+0000 7ff0cef6d640 1 --2- 172.21.15.55:0/2944114730 >> v2:172.21.15.84:3300/0 conn(0x7ff0c8154720 0x7ff0c80c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:15.337 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.338+0000 7ff0cef6d640 1 --2- 172.21.15.55:0/2944114730 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0c8154340 0x7ff0c815e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:15.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.338+0000 7ff0cef6d640 1 -- 172.21.15.55:0/2944114730 >> 172.21.15.55:0/2944114730 conn(0x7ff0c80cc9c0 msgr2=0x7ff0c80ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:15.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.338+0000 7ff0cef6d640 1 -- 172.21.15.55:0/2944114730 shutdown_connections 2024-03-19T02:05:15.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.338+0000 7ff0cef6d640 1 -- 172.21.15.55:0/2944114730 wait complete. 2024-03-19T02:05:15.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.338+0000 7ff0cef6d640 1 Processor -- start 2024-03-19T02:05:15.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.338+0000 7ff0cef6d640 1 -- start start 2024-03-19T02:05:15.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.338+0000 7ff0cef6d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0c8154340 0x7ff0c81f3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:15.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.338+0000 7ff0cef6d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0c8154720 0x7ff0c8153490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:15.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.338+0000 7ff0cef6d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0c815aca0 0x7ff0c8162a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:15.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.338+0000 7ff0cef6d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c815b100 con 0x7ff0c8154340 2024-03-19T02:05:15.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.338+0000 7ff0cef6d640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c80733d0 con 0x7ff0c815aca0 2024-03-19T02:05:15.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.338+0000 7ff0cef6d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff0c806b2a0 con 0x7ff0c8154720 2024-03-19T02:05:15.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.342+0000 7ff0ccd09640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0c8154340 0x7ff0c81f3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:15.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.342+0000 7ff0cd50a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0c815aca0 0x7ff0c8162a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:15.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.342+0000 7ff0ccd09640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0c8154340 0x7ff0c81f3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59850/0 (socket says 172.21.15.55:59850) 2024-03-19T02:05:15.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.342+0000 7ff0ccd09640 1 -- 172.21.15.55:0/3879887540 learned_addr learned my addr 172.21.15.55:0/3879887540 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:15.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.342+0000 7ff0c7fff640 1 --2- 172.21.15.55:0/3879887540 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0c8154720 0x7ff0c8153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:15.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.346+0000 7ff0ccd09640 1 -- 172.21.15.55:0/3879887540 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0c8154720 msgr2=0x7ff0c8153490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:15.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.346+0000 7ff0ccd09640 1 --2- 172.21.15.55:0/3879887540 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0c8154720 0x7ff0c8153490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:15.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.346+0000 7ff0ccd09640 1 -- 172.21.15.55:0/3879887540 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0c815aca0 msgr2=0x7ff0c8162a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:15.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.346+0000 7ff0ccd09640 1 --2- 172.21.15.55:0/3879887540 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0c815aca0 0x7ff0c8162a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:15.345 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.346+0000 7ff0ccd09640 1 -- 172.21.15.55:0/3879887540 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c8161f90 con 0x7ff0c8154340 2024-03-19T02:05:15.347 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.346+0000 7ff0ccd09640 1 --2- 172.21.15.55:0/3879887540 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0c8154340 0x7ff0c81f3940 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7ff0b800ef00 tx=0x7ff0b800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:15.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.354+0000 7ff0c57fa640 1 -- 172.21.15.55:0/3879887540 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b8018020 con 0x7ff0c8154340 2024-03-19T02:05:15.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.354+0000 7ff0c57fa640 1 -- 172.21.15.55:0/3879887540 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0b8014400 con 0x7ff0c8154340 2024-03-19T02:05:15.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.354+0000 7ff0cef6d640 1 -- 172.21.15.55:0/3879887540 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0c8164230 con 0x7ff0c8154340 2024-03-19T02:05:15.352 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.354+0000 7ff0c57fa640 1 -- 172.21.15.55:0/3879887540 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b80147d0 con 0x7ff0c8154340 2024-03-19T02:05:15.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.354+0000 7ff0cef6d640 1 -- 172.21.15.55:0/3879887540 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0c81647f0 con 0x7ff0c8154340 2024-03-19T02:05:15.353 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.354+0000 7ff0cef6d640 1 -- 172.21.15.55:0/3879887540 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0c8068a40 con 0x7ff0c8154340 2024-03-19T02:05:15.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.378+0000 7ff0c57fa640 1 -- 172.21.15.55:0/3879887540 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ff0b801b020 con 0x7ff0c8154340 2024-03-19T02:05:15.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.378+0000 7ff0c57fa640 1 --2- 172.21.15.55:0/3879887540 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff0a8067660 0x7ff0a8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:15.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.378+0000 7ff0c7fff640 1 --2- 172.21.15.55:0/3879887540 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff0a8067660 0x7ff0a8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:15.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.378+0000 7ff0c7fff640 1 --2- 172.21.15.55:0/3879887540 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff0a8067660 0x7ff0a8069b20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff0c80c5a30 tx=0x7ff0b0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:15.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.678+0000 7ff0c57fa640 1 -- 172.21.15.55:0/3879887540 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7ff0b8054d20 con 0x7ff0c8154340 2024-03-19T02:05:15.679 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.678+0000 7ff0c57fa640 1 -- 172.21.15.55:0/3879887540 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ff0b8022070 con 0x7ff0c8154340 2024-03-19T02:05:15.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.902+0000 7ff0cef6d640 1 -- 172.21.15.55:0/3879887540 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0c815c980 con 0x7ff0c8154340 2024-03-19T02:05:15.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.962+0000 7ff0c57fa640 1 -- 172.21.15.55:0/3879887540 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7ff0b8056c80 con 0x7ff0c8154340 2024-03-19T02:05:15.962 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:15.962 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:15.963 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:15.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.966+0000 7ff0cef6d640 1 -- 172.21.15.55:0/3879887540 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff0a8067660 msgr2=0x7ff0a8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:15.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.966+0000 7ff0cef6d640 1 --2- 172.21.15.55:0/3879887540 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff0a8067660 0x7ff0a8069b20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff0c80c5a30 tx=0x7ff0b0009190 comp rx=0 tx=0).stop 2024-03-19T02:05:15.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.966+0000 7ff0cef6d640 1 -- 172.21.15.55:0/3879887540 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0c8154340 msgr2=0x7ff0c81f3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:15.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.966+0000 7ff0cef6d640 1 --2- 172.21.15.55:0/3879887540 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0c8154340 0x7ff0c81f3940 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7ff0b800ef00 tx=0x7ff0b800b480 comp rx=0 tx=0).stop 2024-03-19T02:05:15.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.966+0000 7ff0cef6d640 1 -- 172.21.15.55:0/3879887540 shutdown_connections 2024-03-19T02:05:15.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.966+0000 7ff0cef6d640 1 --2- 172.21.15.55:0/3879887540 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff0a8067660 0x7ff0a8069b20 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:15.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.966+0000 7ff0cef6d640 1 --2- 172.21.15.55:0/3879887540 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0c815aca0 0x7ff0c8162a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:15.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.966+0000 7ff0cef6d640 1 --2- 172.21.15.55:0/3879887540 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff0c8154720 0x7ff0c8153490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:15.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.966+0000 7ff0cef6d640 1 --2- 172.21.15.55:0/3879887540 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0c8154340 0x7ff0c81f3940 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:15.966 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.966+0000 7ff0cef6d640 1 -- 172.21.15.55:0/3879887540 >> 172.21.15.55:0/3879887540 conn(0x7ff0c80cc9c0 msgr2=0x7ff0c8159ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:15.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.966+0000 7ff0cef6d640 1 -- 172.21.15.55:0/3879887540 shutdown_connections 2024-03-19T02:05:15.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:15.966+0000 7ff0cef6d640 1 -- 172.21.15.55:0/3879887540 wait complete. 2024-03-19T02:05:16.591 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:16.591 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:17.592 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:17.722 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.722+0000 7f08aed07640 1 Processor -- start 2024-03-19T02:05:17.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.722+0000 7f08aed07640 1 -- start start 2024-03-19T02:05:17.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.722+0000 7f08aed07640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f08a806e470 0x7f08a80c71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:17.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.722+0000 7f08aed07640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08a815af80 con 0x7f08a806e850 2024-03-19T02:05:17.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.722+0000 7f08aed07640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08a80c79b0 con 0x7f08a80c8770 2024-03-19T02:05:17.723 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.722+0000 7f08aed07640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f08a815ad30 con 0x7f08a806e470 2024-03-19T02:05:17.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.726+0000 7f08a7fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f08a806e850 0x7f08a806b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:41558/0 (socket says 172.21.15.55:41558) 2024-03-19T02:05:17.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.726+0000 7f08a7fff640 1 -- 172.21.15.55:0/637553574 learned_addr learned my addr 172.21.15.55:0/637553574 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:05:17.724 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.726+0000 7f08acaa3640 1 --2- 172.21.15.55:0/637553574 >> v2:172.21.15.84:3300/0 conn(0x7f08a806e470 0x7f08a80c71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:17.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.726+0000 7f08a77fe640 1 -- 172.21.15.55:0/637553574 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 526943044 0 0) 0x7f08a815af80 con 0x7f08a806e850 2024-03-19T02:05:17.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.726+0000 7f08a77fe640 1 -- 172.21.15.55:0/637553574 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0890003530 con 0x7f08a806e850 2024-03-19T02:05:17.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.730+0000 7f08acaa3640 1 -- 172.21.15.55:0/637553574 >> v1:172.21.15.84:6789/0 conn(0x7f08a80c8770 legacy=0x7f08a8154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:17.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.730+0000 7f08acaa3640 1 -- 172.21.15.55:0/637553574 >> v1:172.21.15.55:6789/0 conn(0x7f08a806e850 legacy=0x7f08a806b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:17.728 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.730+0000 7f08acaa3640 1 -- 172.21.15.55:0/637553574 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a80c7c90 con 0x7f08a806e470 2024-03-19T02:05:17.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.734+0000 7f08acaa3640 1 --2- 172.21.15.55:0/637553574 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08a806e470 0x7f08a80c71a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f08980097f0 tx=0x7f089801c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b0fdfa4320892ef server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:17.736 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.738+0000 7f08a77fe640 1 -- 172.21.15.55:0/637553574 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08980b0070 con 0x7f08a806e470 2024-03-19T02:05:17.739 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.738+0000 7f08a77fe640 1 -- 172.21.15.55:0/637553574 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08980ab600 con 0x7f08a806e470 2024-03-19T02:05:17.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.742+0000 7f08aed07640 1 -- 172.21.15.55:0/637553574 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08a806e470 msgr2=0x7f08a80c71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:17.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.742+0000 7f08aed07640 1 --2- 172.21.15.55:0/637553574 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08a806e470 0x7f08a80c71a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f08980097f0 tx=0x7f089801c460 comp rx=0 tx=0).stop 2024-03-19T02:05:17.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.742+0000 7f08aed07640 1 -- 172.21.15.55:0/637553574 shutdown_connections 2024-03-19T02:05:17.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.742+0000 7f08aed07640 1 --2- 172.21.15.55:0/637553574 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08a806e470 0x7f08a80c71a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:17.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.742+0000 7f08aed07640 1 -- 172.21.15.55:0/637553574 >> 172.21.15.55:0/637553574 conn(0x7f08a80cc270 msgr2=0x7f08a80cc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:17.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.742+0000 7f08aed07640 1 -- 172.21.15.55:0/637553574 shutdown_connections 2024-03-19T02:05:17.741 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.742+0000 7f08aed07640 1 -- 172.21.15.55:0/637553574 wait complete. 2024-03-19T02:05:17.742 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.742+0000 7f08aed07640 1 Processor -- start 2024-03-19T02:05:17.742 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.742+0000 7f08aed07640 1 -- start start 2024-03-19T02:05:17.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.742+0000 7f08aed07640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f08a806e470 0x7f08a81f6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:17.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.742+0000 7f08aed07640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f08a806e850 0x7f08a815db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:17.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.742+0000 7f08aed07640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08a80c8770 0x7f08a814c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:17.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.742+0000 7f08aed07640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f08a8154880 con 0x7f08a806e850 2024-03-19T02:05:17.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.742+0000 7f08aed07640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f08a815b260 con 0x7f08a80c8770 2024-03-19T02:05:17.744 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.742+0000 7f08aed07640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f08a8154a00 con 0x7f08a806e470 2024-03-19T02:05:17.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.746+0000 7f08a7fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f08a806e850 0x7f08a815db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:17.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.746+0000 7f08acaa3640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f08a806e470 0x7f08a81f6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:17.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.746+0000 7f08a7fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f08a806e850 0x7f08a815db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59860/0 (socket says 172.21.15.55:59860) 2024-03-19T02:05:17.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.746+0000 7f08a7fff640 1 -- 172.21.15.55:0/4028670250 learned_addr learned my addr 172.21.15.55:0/4028670250 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:17.745 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.746+0000 7f08ad2a4640 1 --2- 172.21.15.55:0/4028670250 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08a80c8770 0x7f08a814c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:17.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.750+0000 7f08a7fff640 1 -- 172.21.15.55:0/4028670250 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f08a806e470 msgr2=0x7f08a81f6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:17.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.750+0000 7f08a7fff640 1 --2- 172.21.15.55:0/4028670250 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f08a806e470 0x7f08a81f6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:17.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.750+0000 7f08a7fff640 1 -- 172.21.15.55:0/4028670250 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08a80c8770 msgr2=0x7f08a814c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:17.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.750+0000 7f08a7fff640 1 --2- 172.21.15.55:0/4028670250 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08a80c8770 0x7f08a814c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:17.750 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.750+0000 7f08a7fff640 1 -- 172.21.15.55:0/4028670250 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a81f73c0 con 0x7f08a806e850 2024-03-19T02:05:17.753 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.754+0000 7f08a7fff640 1 --2- 172.21.15.55:0/4028670250 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f08a806e850 0x7f08a815db10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f089c00c450 tx=0x7f089c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:17.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.758+0000 7f08a57fa640 1 -- 172.21.15.55:0/4028670250 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089c017070 con 0x7f08a806e850 2024-03-19T02:05:17.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.758+0000 7f08a57fa640 1 -- 172.21.15.55:0/4028670250 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f089c005cf0 con 0x7f08a806e850 2024-03-19T02:05:17.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.758+0000 7f08a57fa640 1 -- 172.21.15.55:0/4028670250 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089c007290 con 0x7f08a806e850 2024-03-19T02:05:17.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.758+0000 7f08aed07640 1 -- 172.21.15.55:0/4028670250 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a81f76a0 con 0x7f08a806e850 2024-03-19T02:05:17.758 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.758+0000 7f08aed07640 1 -- 172.21.15.55:0/4028670250 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a815d310 con 0x7f08a806e850 2024-03-19T02:05:17.759 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:17.758+0000 7f08aed07640 1 -- 172.21.15.55:0/4028670250 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08a8068a40 con 0x7f08a806e850 2024-03-19T02:05:18.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:18.086+0000 7f08a57fa640 1 -- 172.21.15.55:0/4028670250 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f089c01a070 con 0x7f08a806e850 2024-03-19T02:05:18.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:18.086+0000 7f08a57fa640 1 --2- 172.21.15.55:0/4028670250 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0894067690 0x7f0894069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:18.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:18.086+0000 7f08acaa3640 1 --2- 172.21.15.55:0/4028670250 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0894067690 0x7f0894069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:18.086 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:18.086+0000 7f08acaa3640 1 --2- 172.21.15.55:0/4028670250 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0894067690 0x7f0894069b50 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f08980ad000 tx=0x7f08980a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:18.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:18.126+0000 7f08a57fa640 1 -- 172.21.15.55:0/4028670250 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f089c053560 con 0x7f08a806e850 2024-03-19T02:05:18.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:18.126+0000 7f08a57fa640 1 -- 172.21.15.55:0/4028670250 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f089c021080 con 0x7f08a806e850 2024-03-19T02:05:18.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:18.346+0000 7f08aed07640 1 -- 172.21.15.55:0/4028670250 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08a80cc650 con 0x7f08a806e850 2024-03-19T02:05:18.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:18.362+0000 7f08a57fa640 1 -- 172.21.15.55:0/4028670250 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f089c0554c0 con 0x7f08a806e850 2024-03-19T02:05:18.362 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:18.362 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:18.362 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:18.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:18.366+0000 7f08aed07640 1 -- 172.21.15.55:0/4028670250 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0894067690 msgr2=0x7f0894069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:18.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:18.366+0000 7f08aed07640 1 --2- 172.21.15.55:0/4028670250 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0894067690 0x7f0894069b50 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f08980ad000 tx=0x7f08980a6000 comp rx=0 tx=0).stop 2024-03-19T02:05:18.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:18.366+0000 7f08aed07640 1 -- 172.21.15.55:0/4028670250 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f08a806e850 msgr2=0x7f08a815db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:18.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:18.366+0000 7f08aed07640 1 --2- 172.21.15.55:0/4028670250 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f08a806e850 0x7f08a815db10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f089c00c450 tx=0x7f089c00c910 comp rx=0 tx=0).stop 2024-03-19T02:05:18.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:18.366+0000 7f08aed07640 1 -- 172.21.15.55:0/4028670250 shutdown_connections 2024-03-19T02:05:18.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:18.366+0000 7f08aed07640 1 --2- 172.21.15.55:0/4028670250 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0894067690 0x7f0894069b50 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:18.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:18.366+0000 7f08aed07640 1 --2- 172.21.15.55:0/4028670250 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08a80c8770 0x7f08a814c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:18.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:18.366+0000 7f08aed07640 1 --2- 172.21.15.55:0/4028670250 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f08a806e850 0x7f08a815db10 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:18.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:18.366+0000 7f08aed07640 1 --2- 172.21.15.55:0/4028670250 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f08a806e470 0x7f08a81f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:18.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:18.366+0000 7f08aed07640 1 -- 172.21.15.55:0/4028670250 >> 172.21.15.55:0/4028670250 conn(0x7f08a80cc270 msgr2=0x7f08a806b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:18.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:18.366+0000 7f08aed07640 1 -- 172.21.15.55:0/4028670250 shutdown_connections 2024-03-19T02:05:18.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:18.366+0000 7f08aed07640 1 -- 172.21.15.55:0/4028670250 wait complete. 2024-03-19T02:05:18.986 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:18.986 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:19.987 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:20.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.106+0000 7f0c612c2640 1 Processor -- start 2024-03-19T02:05:20.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.106+0000 7f0c612c2640 1 -- start start 2024-03-19T02:05:20.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.106+0000 7f0c612c2640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f0c5c1546a0 0x7f0c5c15ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:20.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.106+0000 7f0c612c2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0c5c0c8750 0x7f0c5c0cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:20.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.106+0000 7f0c612c2640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0c5c0ce020 0x7f0c5c06cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:20.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.106+0000 7f0c612c2640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f0c5c15e7f0 con 0x7f0c5c0ce020 2024-03-19T02:05:20.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.106+0000 7f0c612c2640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f0c5c15b2a0 con 0x7f0c5c1546a0 2024-03-19T02:05:20.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.106+0000 7f0c612c2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f0c5c154a80 con 0x7f0c5c0c8750 2024-03-19T02:05:20.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.110+0000 7f0c5b7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0c5c0ce020 0x7f0c5c06cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:20.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.110+0000 7f0c5a7fc640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0c5c0c8750 0x7f0c5c0cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:20.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.110+0000 7f0c5b7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0c5c0ce020 0x7f0c5c06cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33326/0 (socket says 172.21.15.55:33326) 2024-03-19T02:05:20.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.110+0000 7f0c5b7fe640 1 -- 172.21.15.55:0/1185768947 learned_addr learned my addr 172.21.15.55:0/1185768947 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:20.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.110+0000 7f0c5affd640 1 --2- 172.21.15.55:0/1185768947 >> v2:172.21.15.84:3300/0 conn(0x7f0c5c1546a0 0x7f0c5c15ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:20.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.110+0000 7f0c5b7fe640 1 -- 172.21.15.55:0/1185768947 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0c5c0c8750 msgr2=0x7f0c5c0cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:20.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.110+0000 7f0c5b7fe640 1 --2- 172.21.15.55:0/1185768947 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0c5c0c8750 0x7f0c5c0cdae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:20.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.110+0000 7f0c5b7fe640 1 -- 172.21.15.55:0/1185768947 >> v2:172.21.15.84:3300/0 conn(0x7f0c5c1546a0 msgr2=0x7f0c5c15ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:20.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.110+0000 7f0c5b7fe640 1 --2- 172.21.15.55:0/1185768947 >> v2:172.21.15.84:3300/0 conn(0x7f0c5c1546a0 0x7f0c5c15ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:20.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.110+0000 7f0c5b7fe640 1 -- 172.21.15.55:0/1185768947 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c5c15dc70 con 0x7f0c5c0ce020 2024-03-19T02:05:20.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.114+0000 7f0c5b7fe640 1 --2- 172.21.15.55:0/1185768947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c5c0ce020 0x7f0c5c06cea0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c072010 tx=0x7f0c5401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b3726c236c729b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:20.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.118+0000 7f0c59ffb640 1 -- 172.21.15.55:0/1185768947 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c540a5740 con 0x7f0c5c0ce020 2024-03-19T02:05:20.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.118+0000 7f0c59ffb640 1 -- 172.21.15.55:0/1185768947 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c540a58e0 con 0x7f0c5c0ce020 2024-03-19T02:05:20.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.118+0000 7f0c59ffb640 1 -- 172.21.15.55:0/1185768947 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c540a5cb0 con 0x7f0c5c0ce020 2024-03-19T02:05:20.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.122+0000 7f0c612c2640 1 -- 172.21.15.55:0/1185768947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c5c0ce020 msgr2=0x7f0c5c06cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:20.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.122+0000 7f0c612c2640 1 --2- 172.21.15.55:0/1185768947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c5c0ce020 0x7f0c5c06cea0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c072010 tx=0x7f0c5401c9d0 comp rx=0 tx=0).stop 2024-03-19T02:05:20.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.122+0000 7f0c612c2640 1 -- 172.21.15.55:0/1185768947 shutdown_connections 2024-03-19T02:05:20.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.122+0000 7f0c612c2640 1 --2- 172.21.15.55:0/1185768947 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0c5c0c8750 0x7f0c5c0cdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:20.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.122+0000 7f0c612c2640 1 --2- 172.21.15.55:0/1185768947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c5c0ce020 0x7f0c5c06cea0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:20.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.122+0000 7f0c612c2640 1 --2- 172.21.15.55:0/1185768947 >> v2:172.21.15.84:3300/0 conn(0x7f0c5c1546a0 0x7f0c5c15ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:20.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.122+0000 7f0c612c2640 1 -- 172.21.15.55:0/1185768947 >> 172.21.15.55:0/1185768947 conn(0x7f0c5c0714d0 msgr2=0x7f0c5c0718e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:20.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.122+0000 7f0c612c2640 1 -- 172.21.15.55:0/1185768947 shutdown_connections 2024-03-19T02:05:20.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.122+0000 7f0c612c2640 1 -- 172.21.15.55:0/1185768947 wait complete. 2024-03-19T02:05:20.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.122+0000 7f0c612c2640 1 Processor -- start 2024-03-19T02:05:20.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.122+0000 7f0c612c2640 1 -- start start 2024-03-19T02:05:20.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.122+0000 7f0c612c2640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0c5c0c8750 0x7f0c5c14e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:20.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.122+0000 7f0c612c2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c5c0ce020 0x7f0c5c152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:20.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.122+0000 7f0c612c2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0c5c1546a0 0x7f0c5c1f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:20.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.122+0000 7f0c612c2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5c06d7f0 con 0x7f0c5c0ce020 2024-03-19T02:05:20.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.122+0000 7f0c612c2640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5c06c980 con 0x7f0c5c0c8750 2024-03-19T02:05:20.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.122+0000 7f0c612c2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f0c5c15d8a0 con 0x7f0c5c1546a0 2024-03-19T02:05:20.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.126+0000 7f0c5a7fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c5c0ce020 0x7f0c5c152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:20.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.126+0000 7f0c5affd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0c5c0c8750 0x7f0c5c14e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:20.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.126+0000 7f0c5b7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0c5c1546a0 0x7f0c5c1f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:20.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.126+0000 7f0c5a7fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c5c0ce020 0x7f0c5c152a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33334/0 (socket says 172.21.15.55:33334) 2024-03-19T02:05:20.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.126+0000 7f0c5a7fc640 1 -- 172.21.15.55:0/660183226 learned_addr learned my addr 172.21.15.55:0/660183226 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:20.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.130+0000 7f0c5a7fc640 1 -- 172.21.15.55:0/660183226 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0c5c1546a0 msgr2=0x7f0c5c1f2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:20.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.130+0000 7f0c5a7fc640 1 --2- 172.21.15.55:0/660183226 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0c5c1546a0 0x7f0c5c1f2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:20.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.130+0000 7f0c5a7fc640 1 -- 172.21.15.55:0/660183226 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0c5c0c8750 msgr2=0x7f0c5c14e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:20.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.130+0000 7f0c5a7fc640 1 --2- 172.21.15.55:0/660183226 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0c5c0c8750 0x7f0c5c14e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:20.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.130+0000 7f0c5a7fc640 1 -- 172.21.15.55:0/660183226 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c5c1f62a0 con 0x7f0c5c0ce020 2024-03-19T02:05:20.133 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.134+0000 7f0c5a7fc640 1 --2- 172.21.15.55:0/660183226 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c5c0ce020 0x7f0c5c152a20 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0c5000e8a0 tx=0x7f0c5000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:20.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.138+0000 7f0c437fe640 1 -- 172.21.15.55:0/660183226 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c50017070 con 0x7f0c5c0ce020 2024-03-19T02:05:20.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.138+0000 7f0c437fe640 1 -- 172.21.15.55:0/660183226 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c5000ee90 con 0x7f0c5c0ce020 2024-03-19T02:05:20.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.138+0000 7f0c612c2640 1 -- 172.21.15.55:0/660183226 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c5c14fd00 con 0x7f0c5c0ce020 2024-03-19T02:05:20.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.138+0000 7f0c437fe640 1 -- 172.21.15.55:0/660183226 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5000cb80 con 0x7f0c5c0ce020 2024-03-19T02:05:20.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.138+0000 7f0c612c2640 1 -- 172.21.15.55:0/660183226 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c5c1f6820 con 0x7f0c5c0ce020 2024-03-19T02:05:20.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.138+0000 7f0c612c2640 1 -- 172.21.15.55:0/660183226 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c5c068330 con 0x7f0c5c0ce020 2024-03-19T02:05:20.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.174+0000 7f0c437fe640 1 -- 172.21.15.55:0/660183226 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f0c5001a070 con 0x7f0c5c0ce020 2024-03-19T02:05:20.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.174+0000 7f0c437fe640 1 --2- 172.21.15.55:0/660183226 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0c480676b0 0x7f0c48069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:20.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.174+0000 7f0c5affd640 1 --2- 172.21.15.55:0/660183226 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0c480676b0 0x7f0c48069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:20.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.174+0000 7f0c5affd640 1 --2- 172.21.15.55:0/660183226 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0c480676b0 0x7f0c48069b70 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c009b60 tx=0x7f0c4c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:20.177 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.178+0000 7f0c437fe640 1 -- 172.21.15.55:0/660183226 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f0c50054490 con 0x7f0c5c0ce020 2024-03-19T02:05:20.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.534+0000 7f0c437fe640 1 -- 172.21.15.55:0/660183226 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f0c500563f0 con 0x7f0c5c0ce020 2024-03-19T02:05:20.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.854+0000 7f0c612c2640 1 -- 172.21.15.55:0/660183226 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c5c0c2c10 con 0x7f0c5c0ce020 2024-03-19T02:05:20.870 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.870+0000 7f0c437fe640 1 -- 172.21.15.55:0/660183226 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f0c5005a490 con 0x7f0c5c0ce020 2024-03-19T02:05:20.871 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:20.871 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:20.871 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:20.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.874+0000 7f0c612c2640 1 -- 172.21.15.55:0/660183226 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0c480676b0 msgr2=0x7f0c48069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:20.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.874+0000 7f0c612c2640 1 --2- 172.21.15.55:0/660183226 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0c480676b0 0x7f0c48069b70 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c009b60 tx=0x7f0c4c009190 comp rx=0 tx=0).stop 2024-03-19T02:05:20.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.874+0000 7f0c612c2640 1 -- 172.21.15.55:0/660183226 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c5c0ce020 msgr2=0x7f0c5c152a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:20.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.874+0000 7f0c612c2640 1 --2- 172.21.15.55:0/660183226 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c5c0ce020 0x7f0c5c152a20 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0c5000e8a0 tx=0x7f0c5000ed60 comp rx=0 tx=0).stop 2024-03-19T02:05:20.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.874+0000 7f0c612c2640 1 -- 172.21.15.55:0/660183226 shutdown_connections 2024-03-19T02:05:20.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.874+0000 7f0c612c2640 1 --2- 172.21.15.55:0/660183226 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0c480676b0 0x7f0c48069b70 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:20.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.874+0000 7f0c612c2640 1 --2- 172.21.15.55:0/660183226 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0c5c1546a0 0x7f0c5c1f2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:20.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.874+0000 7f0c612c2640 1 --2- 172.21.15.55:0/660183226 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c5c0ce020 0x7f0c5c152a20 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:20.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.874+0000 7f0c612c2640 1 --2- 172.21.15.55:0/660183226 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0c5c0c8750 0x7f0c5c14e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:20.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.874+0000 7f0c612c2640 1 -- 172.21.15.55:0/660183226 >> 172.21.15.55:0/660183226 conn(0x7f0c5c0714d0 msgr2=0x7f0c5c1535a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:20.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.874+0000 7f0c612c2640 1 -- 172.21.15.55:0/660183226 shutdown_connections 2024-03-19T02:05:20.874 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:20.874+0000 7f0c612c2640 1 -- 172.21.15.55:0/660183226 wait complete. 2024-03-19T02:05:21.497 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:21.497 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:22.498 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:22.615 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.614+0000 7f1d7cc3d640 1 Processor -- start 2024-03-19T02:05:22.615 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.614+0000 7f1d7cc3d640 1 -- start start 2024-03-19T02:05:22.616 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.614+0000 7f1d7cc3d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1d78152120 0x7f1d780c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:22.616 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.614+0000 7f1d7cc3d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f1d780c9610 0x7f1d78157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:22.616 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.614+0000 7f1d7cc3d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1d78070780 0x7f1d78158510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:22.616 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.614+0000 7f1d7cc3d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f1d78152500 con 0x7f1d78152120 2024-03-19T02:05:22.616 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.614+0000 7f1d7cc3d640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f1d78070590 con 0x7f1d780c9610 2024-03-19T02:05:22.616 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.614+0000 7f1d7cc3d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1d780c3530 con 0x7f1d78070780 2024-03-19T02:05:22.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.618+0000 7f1d76d9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1d78070780 0x7f1d78158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:22.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.618+0000 7f1d75d9b640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f1d780c9610 0x7f1d78157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:22.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.618+0000 7f1d76d9d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1d78070780 0x7f1d78158510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:48470/0 (socket says 172.21.15.55:48470) 2024-03-19T02:05:22.617 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.618+0000 7f1d76d9d640 1 -- 172.21.15.55:0/3593145463 learned_addr learned my addr 172.21.15.55:0/3593145463 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:22.621 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.622+0000 7f1d76d9d640 1 -- 172.21.15.55:0/3593145463 >> v2:172.21.15.84:3300/0 conn(0x7f1d780c9610 msgr2=0x7f1d78157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:22.621 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.622+0000 7f1d76d9d640 1 --2- 172.21.15.55:0/3593145463 >> v2:172.21.15.84:3300/0 conn(0x7f1d780c9610 0x7f1d78157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:22.621 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.622+0000 7f1d76d9d640 1 -- 172.21.15.55:0/3593145463 >> v2:172.21.15.55:3300/0 conn(0x7f1d78152120 msgr2=0x7f1d780c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:22.621 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.622+0000 7f1d76d9d640 1 --2- 172.21.15.55:0/3593145463 >> v2:172.21.15.55:3300/0 conn(0x7f1d78152120 0x7f1d780c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:22.621 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.622+0000 7f1d76d9d640 1 -- 172.21.15.55:0/3593145463 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d7806fff0 con 0x7f1d78070780 2024-03-19T02:05:22.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.626+0000 7f1d76d9d640 1 --2- 172.21.15.55:0/3593145463 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1d78070780 0x7f1d78158510 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1d780cb130 tx=0x7f1d7001c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed220ca4f853dc12 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:22.632 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.634+0000 7f1d7559a640 1 -- 172.21.15.55:0/3593145463 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d700a5670 con 0x7f1d78070780 2024-03-19T02:05:22.632 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.634+0000 7f1d7559a640 1 -- 172.21.15.55:0/3593145463 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d700a5810 con 0x7f1d78070780 2024-03-19T02:05:22.632 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.634+0000 7f1d7559a640 1 -- 172.21.15.55:0/3593145463 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d700a5be0 con 0x7f1d78070780 2024-03-19T02:05:22.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.634+0000 7f1d7cc3d640 1 -- 172.21.15.55:0/3593145463 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1d78070780 msgr2=0x7f1d78158510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:22.633 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.634+0000 7f1d7cc3d640 1 --2- 172.21.15.55:0/3593145463 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1d78070780 0x7f1d78158510 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1d780cb130 tx=0x7f1d7001c9f0 comp rx=0 tx=0).stop 2024-03-19T02:05:22.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.634+0000 7f1d7cc3d640 1 -- 172.21.15.55:0/3593145463 shutdown_connections 2024-03-19T02:05:22.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.634+0000 7f1d7cc3d640 1 --2- 172.21.15.55:0/3593145463 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1d78070780 0x7f1d78158510 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:22.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.634+0000 7f1d7cc3d640 1 --2- 172.21.15.55:0/3593145463 >> v2:172.21.15.84:3300/0 conn(0x7f1d780c9610 0x7f1d78157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:22.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.634+0000 7f1d7cc3d640 1 --2- 172.21.15.55:0/3593145463 >> v2:172.21.15.55:3300/0 conn(0x7f1d78152120 0x7f1d780c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:22.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.634+0000 7f1d7cc3d640 1 -- 172.21.15.55:0/3593145463 >> 172.21.15.55:0/3593145463 conn(0x7f1d78067600 msgr2=0x7f1d78071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:22.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.634+0000 7f1d7cc3d640 1 -- 172.21.15.55:0/3593145463 shutdown_connections 2024-03-19T02:05:22.634 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.634+0000 7f1d7cc3d640 1 -- 172.21.15.55:0/3593145463 wait complete. 2024-03-19T02:05:22.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.634+0000 7f1d7cc3d640 1 Processor -- start 2024-03-19T02:05:22.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.634+0000 7f1d7cc3d640 1 -- start start 2024-03-19T02:05:22.636 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.638+0000 7f1d7cc3d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1d78070780 0x7f1d780ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:22.636 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.638+0000 7f1d7cc3d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1d780c9610 0x7f1d780ce180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:22.637 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.638+0000 7f1d7cc3d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d78152120 0x7f1d7815ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:22.637 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.638+0000 7f1d7cc3d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1d780c31c0 con 0x7f1d78152120 2024-03-19T02:05:22.637 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.638+0000 7f1d7cc3d640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f1d780c2ee0 con 0x7f1d780c9610 2024-03-19T02:05:22.637 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.638+0000 7f1d7cc3d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1d780c86d0 con 0x7f1d78070780 2024-03-19T02:05:22.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.638+0000 7f1d76d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d78152120 0x7f1d7815ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:22.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.638+0000 7f1d76d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d78152120 0x7f1d7815ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33350/0 (socket says 172.21.15.55:33350) 2024-03-19T02:05:22.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.638+0000 7f1d76d9d640 1 -- 172.21.15.55:0/2665704860 learned_addr learned my addr 172.21.15.55:0/2665704860 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:22.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.638+0000 7f1d75d9b640 1 --2- 172.21.15.55:0/2665704860 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1d780c9610 0x7f1d780ce180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:22.638 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.638+0000 7f1d7659c640 1 --2- 172.21.15.55:0/2665704860 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1d78070780 0x7f1d780ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:22.641 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.642+0000 7f1d76d9d640 1 -- 172.21.15.55:0/2665704860 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1d78070780 msgr2=0x7f1d780ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:22.641 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.642+0000 7f1d76d9d640 1 --2- 172.21.15.55:0/2665704860 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1d78070780 0x7f1d780ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:22.641 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.642+0000 7f1d76d9d640 1 -- 172.21.15.55:0/2665704860 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1d780c9610 msgr2=0x7f1d780ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:22.642 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.642+0000 7f1d76d9d640 1 --2- 172.21.15.55:0/2665704860 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1d780c9610 0x7f1d780ce180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:22.642 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.642+0000 7f1d76d9d640 1 -- 172.21.15.55:0/2665704860 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d780cbed0 con 0x7f1d78152120 2024-03-19T02:05:22.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.646+0000 7f1d76d9d640 1 --2- 172.21.15.55:0/2665704860 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d78152120 0x7f1d7815ac10 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f1d700af040 tx=0x7f1d70019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:22.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.650+0000 7f1d66ffd640 1 -- 172.21.15.55:0/2665704860 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d700a5670 con 0x7f1d78152120 2024-03-19T02:05:22.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.650+0000 7f1d66ffd640 1 -- 172.21.15.55:0/2665704860 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d700be070 con 0x7f1d78152120 2024-03-19T02:05:22.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.650+0000 7f1d66ffd640 1 -- 172.21.15.55:0/2665704860 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d700b88b0 con 0x7f1d78152120 2024-03-19T02:05:22.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.650+0000 7f1d7cc3d640 1 -- 172.21.15.55:0/2665704860 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d781f5680 con 0x7f1d78152120 2024-03-19T02:05:22.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.650+0000 7f1d7cc3d640 1 -- 172.21.15.55:0/2665704860 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d781f4ba0 con 0x7f1d78152120 2024-03-19T02:05:22.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.650+0000 7f1d7cc3d640 1 -- 172.21.15.55:0/2665704860 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d78068a40 con 0x7f1d78152120 2024-03-19T02:05:22.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.970+0000 7f1d66ffd640 1 -- 172.21.15.55:0/2665704860 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1d700b8a50 con 0x7f1d78152120 2024-03-19T02:05:22.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.974+0000 7f1d66ffd640 1 --2- 172.21.15.55:0/2665704860 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1d58067660 0x7f1d58069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:22.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.974+0000 7f1d7659c640 1 --2- 172.21.15.55:0/2665704860 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1d58067660 0x7f1d58069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:22.973 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:22.974+0000 7f1d7659c640 1 --2- 172.21.15.55:0/2665704860 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1d58067660 0x7f1d58069b20 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f1d680098f0 tx=0x7f1d68006f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:23.008 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:23.010+0000 7f1d66ffd640 1 -- 172.21.15.55:0/2665704860 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f1d700ef950 con 0x7f1d78152120 2024-03-19T02:05:23.008 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:23.010+0000 7f1d66ffd640 1 -- 172.21.15.55:0/2665704860 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1d700165e0 con 0x7f1d78152120 2024-03-19T02:05:23.232 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:23.230+0000 7f1d7cc3d640 1 -- 172.21.15.55:0/2665704860 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d780c84b0 con 0x7f1d78152120 2024-03-19T02:05:23.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:23.286+0000 7f1d66ffd640 1 -- 172.21.15.55:0/2665704860 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f1d700f18b0 con 0x7f1d78152120 2024-03-19T02:05:23.287 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:23.288 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:23.288 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:23.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:23.290+0000 7f1d7cc3d640 1 -- 172.21.15.55:0/2665704860 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1d58067660 msgr2=0x7f1d58069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:23.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:23.290+0000 7f1d7cc3d640 1 --2- 172.21.15.55:0/2665704860 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1d58067660 0x7f1d58069b20 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f1d680098f0 tx=0x7f1d68006f90 comp rx=0 tx=0).stop 2024-03-19T02:05:23.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:23.290+0000 7f1d7cc3d640 1 -- 172.21.15.55:0/2665704860 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d78152120 msgr2=0x7f1d7815ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:23.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:23.290+0000 7f1d7cc3d640 1 --2- 172.21.15.55:0/2665704860 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d78152120 0x7f1d7815ac10 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f1d700af040 tx=0x7f1d70019790 comp rx=0 tx=0).stop 2024-03-19T02:05:23.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:23.290+0000 7f1d7cc3d640 1 -- 172.21.15.55:0/2665704860 shutdown_connections 2024-03-19T02:05:23.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:23.290+0000 7f1d7cc3d640 1 --2- 172.21.15.55:0/2665704860 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1d58067660 0x7f1d58069b20 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:23.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:23.290+0000 7f1d7cc3d640 1 --2- 172.21.15.55:0/2665704860 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d78152120 0x7f1d7815ac10 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:23.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:23.290+0000 7f1d7cc3d640 1 --2- 172.21.15.55:0/2665704860 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1d780c9610 0x7f1d780ce180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:23.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:23.290+0000 7f1d7cc3d640 1 --2- 172.21.15.55:0/2665704860 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1d78070780 0x7f1d780ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:23.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:23.290+0000 7f1d7cc3d640 1 -- 172.21.15.55:0/2665704860 >> 172.21.15.55:0/2665704860 conn(0x7f1d78067600 msgr2=0x7f1d7814d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:23.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:23.290+0000 7f1d7cc3d640 1 -- 172.21.15.55:0/2665704860 shutdown_connections 2024-03-19T02:05:23.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:23.290+0000 7f1d7cc3d640 1 -- 172.21.15.55:0/2665704860 wait complete. 2024-03-19T02:05:23.909 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:23.910 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:24.911 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:25.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.022+0000 7fa5d28b9640 1 Processor -- start 2024-03-19T02:05:25.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.022+0000 7fa5d28b9640 1 -- start start 2024-03-19T02:05:25.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.022+0000 7fa5d28b9640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa5cc1544f0 0x7fa5cc073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:25.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.022+0000 7fa5d28b9640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5cc0c7fa0 con 0x7fa5cc15acb0 2024-03-19T02:05:25.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.022+0000 7fa5d28b9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5cc0c8240 con 0x7fa5cc06b2a0 2024-03-19T02:05:25.023 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.022+0000 7fa5d28b9640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fa5cc15b210 con 0x7fa5cc1544f0 2024-03-19T02:05:25.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.022+0000 7fa5cbfff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa5cc1544f0 0x7fa5cc073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:25.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.026+0000 7fa5cbfff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa5cc1544f0 0x7fa5cc073540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33356/0 (socket says 172.21.15.55:33356) 2024-03-19T02:05:25.024 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.026+0000 7fa5cbfff640 1 -- 172.21.15.55:0/3251776819 learned_addr learned my addr 172.21.15.55:0/3251776819 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:25.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.026+0000 7fa5cbfff640 1 -- 172.21.15.55:0/3251776819 >> v1:172.21.15.84:6789/0 conn(0x7fa5cc06b2a0 legacy=0x7fa5cc15e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:25.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.026+0000 7fa5cbfff640 1 -- 172.21.15.55:0/3251776819 >> v1:172.21.15.55:6789/0 conn(0x7fa5cc15acb0 legacy=0x7fa5cc0cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:25.027 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.026+0000 7fa5cbfff640 1 -- 172.21.15.55:0/3251776819 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5cc15d920 con 0x7fa5cc1544f0 2024-03-19T02:05:25.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.030+0000 7fa5cbfff640 1 --2- 172.21.15.55:0/3251776819 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5cc1544f0 0x7fa5cc073540 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc0ca790 tx=0x7fa5bc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992e367fed41a9ab server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:25.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.034+0000 7fa5caffd640 1 -- 172.21.15.55:0/3251776819 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc0b5070 con 0x7fa5cc1544f0 2024-03-19T02:05:25.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.034+0000 7fa5caffd640 1 -- 172.21.15.55:0/3251776819 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5bc0a89f0 con 0x7fa5cc1544f0 2024-03-19T02:05:25.036 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.034+0000 7fa5caffd640 1 -- 172.21.15.55:0/3251776819 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc0a8dc0 con 0x7fa5cc1544f0 2024-03-19T02:05:25.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.038+0000 7fa5d28b9640 1 -- 172.21.15.55:0/3251776819 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5cc1544f0 msgr2=0x7fa5cc073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:25.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.038+0000 7fa5d28b9640 1 --2- 172.21.15.55:0/3251776819 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5cc1544f0 0x7fa5cc073540 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc0ca790 tx=0x7fa5bc01ecf0 comp rx=0 tx=0).stop 2024-03-19T02:05:25.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.038+0000 7fa5d28b9640 1 -- 172.21.15.55:0/3251776819 shutdown_connections 2024-03-19T02:05:25.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.038+0000 7fa5d28b9640 1 --2- 172.21.15.55:0/3251776819 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5cc1544f0 0x7fa5cc073540 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:25.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.038+0000 7fa5d28b9640 1 -- 172.21.15.55:0/3251776819 >> 172.21.15.55:0/3251776819 conn(0x7fa5cc06e670 msgr2=0x7fa5cc06ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:25.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.038+0000 7fa5d28b9640 1 -- 172.21.15.55:0/3251776819 shutdown_connections 2024-03-19T02:05:25.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.038+0000 7fa5d28b9640 1 -- 172.21.15.55:0/3251776819 wait complete. 2024-03-19T02:05:25.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.038+0000 7fa5d28b9640 1 Processor -- start 2024-03-19T02:05:25.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.038+0000 7fa5d28b9640 1 -- start start 2024-03-19T02:05:25.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.038+0000 7fa5d28b9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5cc06b2a0 0x7fa5cc1f0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:25.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.038+0000 7fa5d28b9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5cc1544f0 0x7fa5cc0c4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:25.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.038+0000 7fa5d28b9640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc15acb0 0x7fa5cc15c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:25.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.038+0000 7fa5d28b9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc0cd540 con 0x7fa5cc1544f0 2024-03-19T02:05:25.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.038+0000 7fa5d28b9640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1548d0 con 0x7fa5cc15acb0 2024-03-19T02:05:25.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.038+0000 7fa5d28b9640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa5cc0cd6c0 con 0x7fa5cc06b2a0 2024-03-19T02:05:25.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.038+0000 7fa5cb7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5cc1544f0 0x7fa5cc0c4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:25.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.038+0000 7fa5cb7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5cc1544f0 0x7fa5cc0c4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33364/0 (socket says 172.21.15.55:33364) 2024-03-19T02:05:25.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.038+0000 7fa5cb7fe640 1 -- 172.21.15.55:0/1583979631 learned_addr learned my addr 172.21.15.55:0/1583979631 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:25.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.038+0000 7fa5cbfff640 1 --2- 172.21.15.55:0/1583979631 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5cc06b2a0 0x7fa5cc1f0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:25.040 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.042+0000 7fa5d0e56640 1 --2- 172.21.15.55:0/1583979631 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc15acb0 0x7fa5cc15c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:25.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.046+0000 7fa5cb7fe640 1 -- 172.21.15.55:0/1583979631 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5cc06b2a0 msgr2=0x7fa5cc1f0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:25.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.046+0000 7fa5cb7fe640 1 --2- 172.21.15.55:0/1583979631 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5cc06b2a0 0x7fa5cc1f0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:25.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.046+0000 7fa5cb7fe640 1 -- 172.21.15.55:0/1583979631 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc15acb0 msgr2=0x7fa5cc15c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:25.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.046+0000 7fa5cb7fe640 1 --2- 172.21.15.55:0/1583979631 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc15acb0 0x7fa5cc15c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:25.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.046+0000 7fa5cb7fe640 1 -- 172.21.15.55:0/1583979631 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5cc0c6260 con 0x7fa5cc1544f0 2024-03-19T02:05:25.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.050+0000 7fa5cb7fe640 1 --2- 172.21.15.55:0/1583979631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5cc1544f0 0x7fa5cc0c4230 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fa5c000c450 tx=0x7fa5c000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:25.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.054+0000 7fa5c8ff9640 1 -- 172.21.15.55:0/1583979631 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0016020 con 0x7fa5cc1544f0 2024-03-19T02:05:25.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.054+0000 7fa5c8ff9640 1 -- 172.21.15.55:0/1583979631 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5c00052d0 con 0x7fa5cc1544f0 2024-03-19T02:05:25.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.054+0000 7fa5d28b9640 1 -- 172.21.15.55:0/1583979631 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5cc0c6540 con 0x7fa5cc1544f0 2024-03-19T02:05:25.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.054+0000 7fa5c8ff9640 1 -- 172.21.15.55:0/1583979631 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0007260 con 0x7fa5cc1544f0 2024-03-19T02:05:25.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.054+0000 7fa5d28b9640 1 -- 172.21.15.55:0/1583979631 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5cc1f0710 con 0x7fa5cc1544f0 2024-03-19T02:05:25.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.054+0000 7fa5d28b9640 1 -- 172.21.15.55:0/1583979631 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5cc068a40 con 0x7fa5cc1544f0 2024-03-19T02:05:25.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.090+0000 7fa5c8ff9640 1 -- 172.21.15.55:0/1583979631 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa5c0007400 con 0x7fa5cc1544f0 2024-03-19T02:05:25.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.090+0000 7fa5c8ff9640 1 --2- 172.21.15.55:0/1583979631 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa5b8067690 0x7fa5b8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:25.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.090+0000 7fa5cbfff640 1 --2- 172.21.15.55:0/1583979631 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa5b8067690 0x7fa5b8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:25.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.090+0000 7fa5cbfff640 1 --2- 172.21.15.55:0/1583979631 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa5b8067690 0x7fa5b8069b50 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc0b2000 tx=0x7fa5bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:25.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.374+0000 7fa5c8ff9640 1 -- 172.21.15.55:0/1583979631 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fa5c00545f0 con 0x7fa5cc1544f0 2024-03-19T02:05:25.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.374+0000 7fa5c8ff9640 1 -- 172.21.15.55:0/1583979631 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa5c0020020 con 0x7fa5cc1544f0 2024-03-19T02:05:25.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.646+0000 7fa5d28b9640 1 -- 172.21.15.55:0/1583979631 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5cc0c7be0 con 0x7fa5cc1544f0 2024-03-19T02:05:25.673 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.674+0000 7fa5c8ff9640 1 -- 172.21.15.55:0/1583979631 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fa5c0005480 con 0x7fa5cc1544f0 2024-03-19T02:05:25.673 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:25.674 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:25.674 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:25.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.678+0000 7fa5d28b9640 1 -- 172.21.15.55:0/1583979631 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa5b8067690 msgr2=0x7fa5b8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:25.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.678+0000 7fa5d28b9640 1 --2- 172.21.15.55:0/1583979631 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa5b8067690 0x7fa5b8069b50 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc0b2000 tx=0x7fa5bc008040 comp rx=0 tx=0).stop 2024-03-19T02:05:25.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.678+0000 7fa5d28b9640 1 -- 172.21.15.55:0/1583979631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5cc1544f0 msgr2=0x7fa5cc0c4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:25.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.678+0000 7fa5d28b9640 1 --2- 172.21.15.55:0/1583979631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5cc1544f0 0x7fa5cc0c4230 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fa5c000c450 tx=0x7fa5c000c910 comp rx=0 tx=0).stop 2024-03-19T02:05:25.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.678+0000 7fa5d28b9640 1 -- 172.21.15.55:0/1583979631 shutdown_connections 2024-03-19T02:05:25.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.678+0000 7fa5d28b9640 1 --2- 172.21.15.55:0/1583979631 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa5b8067690 0x7fa5b8069b50 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:25.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.678+0000 7fa5d28b9640 1 --2- 172.21.15.55:0/1583979631 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc15acb0 0x7fa5cc15c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:25.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.678+0000 7fa5d28b9640 1 --2- 172.21.15.55:0/1583979631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa5cc1544f0 0x7fa5cc0c4230 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:25.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.678+0000 7fa5d28b9640 1 --2- 172.21.15.55:0/1583979631 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa5cc06b2a0 0x7fa5cc1f0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:25.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.678+0000 7fa5d28b9640 1 -- 172.21.15.55:0/1583979631 >> 172.21.15.55:0/1583979631 conn(0x7fa5cc06e670 msgr2=0x7fa5cc0caa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:25.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.678+0000 7fa5d28b9640 1 -- 172.21.15.55:0/1583979631 shutdown_connections 2024-03-19T02:05:25.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:25.678+0000 7fa5d28b9640 1 -- 172.21.15.55:0/1583979631 wait complete. 2024-03-19T02:05:26.297 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:26.297 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:27.299 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:27.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.410+0000 7f9263450640 1 Processor -- start 2024-03-19T02:05:27.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.410+0000 7f9263450640 1 -- start start 2024-03-19T02:05:27.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.414+0000 7f9263450640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f925c0c62c0 0x7f925c0cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:27.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.414+0000 7f9263450640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f925c0cc4f0 0x7f925c156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:27.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.414+0000 7f9263450640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f925c06cac0 con 0x7f925c13cb80 2024-03-19T02:05:27.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.414+0000 7f9263450640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f925c0c6df0 con 0x7f925c0cc4f0 2024-03-19T02:05:27.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.414+0000 7f9263450640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f925c0c35c0 con 0x7f925c0c62c0 2024-03-19T02:05:27.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.414+0000 7f92619ed640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f925c0cc4f0 0x7f925c156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:27.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.414+0000 7f92609eb640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f925c0c62c0 0x7f925c0cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:27.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.414+0000 7f92619ed640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f925c0cc4f0 0x7f925c156ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33372/0 (socket says 172.21.15.55:33372) 2024-03-19T02:05:27.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.414+0000 7f92619ed640 1 -- 172.21.15.55:0/1822952553 learned_addr learned my addr 172.21.15.55:0/1822952553 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:27.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.418+0000 7f92619ed640 1 -- 172.21.15.55:0/1822952553 >> v2:172.21.15.84:3300/0 conn(0x7f925c0c62c0 msgr2=0x7f925c0cbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:27.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.418+0000 7f92619ed640 1 --2- 172.21.15.55:0/1822952553 >> v2:172.21.15.84:3300/0 conn(0x7f925c0c62c0 0x7f925c0cbfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:27.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.418+0000 7f92619ed640 1 -- 172.21.15.55:0/1822952553 >> v1:172.21.15.55:6789/0 conn(0x7f925c13cb80 legacy=0x7f925c06c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:27.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.418+0000 7f92619ed640 1 -- 172.21.15.55:0/1822952553 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f925c0c38a0 con 0x7f925c0cc4f0 2024-03-19T02:05:27.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.422+0000 7f92619ed640 1 --2- 172.21.15.55:0/1822952553 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f925c0cc4f0 0x7f925c156ec0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f925c067860 tx=0x7f925401f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d20d635392d9ef server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:27.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.426+0000 7f9253fff640 1 -- 172.21.15.55:0/1822952553 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92540b4070 con 0x7f925c0cc4f0 2024-03-19T02:05:27.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.426+0000 7f9253fff640 1 -- 172.21.15.55:0/1822952553 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92540af400 con 0x7f925c0cc4f0 2024-03-19T02:05:27.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.426+0000 7f9253fff640 1 -- 172.21.15.55:0/1822952553 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92540af7d0 con 0x7f925c0cc4f0 2024-03-19T02:05:27.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.426+0000 7f9263450640 1 -- 172.21.15.55:0/1822952553 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f925c0cc4f0 msgr2=0x7f925c156ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:27.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.426+0000 7f9263450640 1 --2- 172.21.15.55:0/1822952553 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f925c0cc4f0 0x7f925c156ec0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f925c067860 tx=0x7f925401f1a0 comp rx=0 tx=0).stop 2024-03-19T02:05:27.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.426+0000 7f9263450640 1 -- 172.21.15.55:0/1822952553 shutdown_connections 2024-03-19T02:05:27.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.426+0000 7f9263450640 1 --2- 172.21.15.55:0/1822952553 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f925c0cc4f0 0x7f925c156ec0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:27.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.426+0000 7f9263450640 1 --2- 172.21.15.55:0/1822952553 >> v2:172.21.15.84:3300/0 conn(0x7f925c0c62c0 0x7f925c0cbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:27.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.426+0000 7f9263450640 1 -- 172.21.15.55:0/1822952553 >> 172.21.15.55:0/1822952553 conn(0x7f925c06e5a0 msgr2=0x7f925c072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:27.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.426+0000 7f9263450640 1 -- 172.21.15.55:0/1822952553 shutdown_connections 2024-03-19T02:05:27.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.426+0000 7f9263450640 1 -- 172.21.15.55:0/1822952553 wait complete. 2024-03-19T02:05:27.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.426+0000 7f9263450640 1 Processor -- start 2024-03-19T02:05:27.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.426+0000 7f9263450640 1 -- start start 2024-03-19T02:05:27.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.430+0000 7f9263450640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f925c0c62c0 0x7f925c1f06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:27.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.430+0000 7f9263450640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f925c0cc4f0 0x7f925c13e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:27.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.430+0000 7f9263450640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f925c13cb80 0x7f925c1e2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:27.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.430+0000 7f9263450640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f925c0c70d0 con 0x7f925c13cb80 2024-03-19T02:05:27.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.430+0000 7f9263450640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f925c0c3e60 con 0x7f925c0cc4f0 2024-03-19T02:05:27.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.430+0000 7f9263450640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f925c0c6a20 con 0x7f925c0c62c0 2024-03-19T02:05:27.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.430+0000 7f92619ed640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f925c13cb80 0x7f925c1e2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:27.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.430+0000 7f92609eb640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f925c0cc4f0 0x7f925c13e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:27.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.430+0000 7f92619ed640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f925c13cb80 0x7f925c1e2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33386/0 (socket says 172.21.15.55:33386) 2024-03-19T02:05:27.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.430+0000 7f92619ed640 1 -- 172.21.15.55:0/1860979545 learned_addr learned my addr 172.21.15.55:0/1860979545 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:27.430 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.430+0000 7f92611ec640 1 --2- 172.21.15.55:0/1860979545 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f925c0c62c0 0x7f925c1f06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:27.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.434+0000 7f92619ed640 1 -- 172.21.15.55:0/1860979545 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f925c0c62c0 msgr2=0x7f925c1f06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:27.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.434+0000 7f92619ed640 1 --2- 172.21.15.55:0/1860979545 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f925c0c62c0 0x7f925c1f06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:27.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.434+0000 7f92619ed640 1 -- 172.21.15.55:0/1860979545 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f925c0cc4f0 msgr2=0x7f925c13e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:27.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.434+0000 7f92619ed640 1 --2- 172.21.15.55:0/1860979545 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f925c0cc4f0 0x7f925c13e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:27.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.434+0000 7f92619ed640 1 -- 172.21.15.55:0/1860979545 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f925c13f060 con 0x7f925c13cb80 2024-03-19T02:05:27.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.434+0000 7f92619ed640 1 --2- 172.21.15.55:0/1860979545 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f925c13cb80 0x7f925c1e2480 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f925401cdd0 tx=0x7f925401bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:27.440 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.438+0000 7f9251ffb640 1 -- 172.21.15.55:0/1860979545 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92540b4040 con 0x7f925c13cb80 2024-03-19T02:05:27.440 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.438+0000 7f9251ffb640 1 -- 172.21.15.55:0/1860979545 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92540c0070 con 0x7f925c13cb80 2024-03-19T02:05:27.440 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.438+0000 7f9251ffb640 1 -- 172.21.15.55:0/1860979545 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9254009510 con 0x7f925c13cb80 2024-03-19T02:05:27.440 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.438+0000 7f9263450640 1 -- 172.21.15.55:0/1860979545 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f925c13f340 con 0x7f925c13cb80 2024-03-19T02:05:27.440 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.438+0000 7f9263450640 1 -- 172.21.15.55:0/1860979545 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f925c1e19a0 con 0x7f925c13cb80 2024-03-19T02:05:27.441 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.442+0000 7f9263450640 1 -- 172.21.15.55:0/1860979545 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f925c068a40 con 0x7f925c13cb80 2024-03-19T02:05:27.717 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.718+0000 7f9251ffb640 1 -- 172.21.15.55:0/1860979545 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f92540ba420 con 0x7f925c13cb80 2024-03-19T02:05:27.718 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.718+0000 7f9251ffb640 1 --2- 172.21.15.55:0/1860979545 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92400676b0 0x7f9240069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:27.719 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.718+0000 7f92611ec640 1 --2- 172.21.15.55:0/1860979545 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92400676b0 0x7f9240069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:27.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.722+0000 7f92611ec640 1 --2- 172.21.15.55:0/1860979545 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92400676b0 0x7f9240069b70 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f92580098e0 tx=0x7f9258009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:27.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.746+0000 7f9251ffb640 1 -- 172.21.15.55:0/1860979545 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f92540f2cc0 con 0x7f925c13cb80 2024-03-19T02:05:27.748 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.750+0000 7f9251ffb640 1 -- 172.21.15.55:0/1860979545 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f9254018d00 con 0x7f925c13cb80 2024-03-19T02:05:27.983 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:27.982+0000 7f9263450640 1 -- 172.21.15.55:0/1860979545 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f925c072360 con 0x7f925c13cb80 2024-03-19T02:05:28.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:28.018+0000 7f9251ffb640 1 -- 172.21.15.55:0/1860979545 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f92540f4c20 con 0x7f925c13cb80 2024-03-19T02:05:28.017 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:28.017 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:28.017 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:28.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:28.018+0000 7f9263450640 1 -- 172.21.15.55:0/1860979545 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92400676b0 msgr2=0x7f9240069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:28.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:28.018+0000 7f9263450640 1 --2- 172.21.15.55:0/1860979545 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92400676b0 0x7f9240069b70 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f92580098e0 tx=0x7f9258009060 comp rx=0 tx=0).stop 2024-03-19T02:05:28.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:28.018+0000 7f9263450640 1 -- 172.21.15.55:0/1860979545 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f925c13cb80 msgr2=0x7f925c1e2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:28.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:28.018+0000 7f9263450640 1 --2- 172.21.15.55:0/1860979545 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f925c13cb80 0x7f925c1e2480 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f925401cdd0 tx=0x7f925401bf40 comp rx=0 tx=0).stop 2024-03-19T02:05:28.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:28.018+0000 7f9263450640 1 -- 172.21.15.55:0/1860979545 shutdown_connections 2024-03-19T02:05:28.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:28.018+0000 7f9263450640 1 --2- 172.21.15.55:0/1860979545 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f92400676b0 0x7f9240069b70 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:28.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:28.018+0000 7f9263450640 1 --2- 172.21.15.55:0/1860979545 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f925c13cb80 0x7f925c1e2480 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:28.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:28.018+0000 7f9263450640 1 --2- 172.21.15.55:0/1860979545 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f925c0cc4f0 0x7f925c13e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:28.020 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:28.018+0000 7f9263450640 1 --2- 172.21.15.55:0/1860979545 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f925c0c62c0 0x7f925c1f06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:28.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:28.018+0000 7f9263450640 1 -- 172.21.15.55:0/1860979545 >> 172.21.15.55:0/1860979545 conn(0x7f925c06e5a0 msgr2=0x7f925c156730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:28.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:28.018+0000 7f9263450640 1 -- 172.21.15.55:0/1860979545 shutdown_connections 2024-03-19T02:05:28.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:28.018+0000 7f9263450640 1 -- 172.21.15.55:0/1860979545 wait complete. 2024-03-19T02:05:28.638 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:28.638 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:29.640 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:29.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.770+0000 7f1eeb0c2640 1 Processor -- start 2024-03-19T02:05:29.770 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.770+0000 7f1eeb0c2640 1 -- start start 2024-03-19T02:05:29.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.770+0000 7f1eeb0c2640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f1ee4070780 0x7f1ee415b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:29.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.770+0000 7f1eeb0c2640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1ee415bb00 0x7f1ee415bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:29.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.770+0000 7f1eeb0c2640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ee415c4d0 con 0x7f1ee4152120 2024-03-19T02:05:29.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.770+0000 7f1eeb0c2640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f1ee4152500 con 0x7f1ee415bb00 2024-03-19T02:05:29.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.770+0000 7f1eeb0c2640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f1ee4070590 con 0x7f1ee4070780 2024-03-19T02:05:29.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.770+0000 7f1ee965f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1ee415bb00 0x7f1ee415bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:29.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.770+0000 7f1ee3fff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f1ee4070780 0x7f1ee415b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:29.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.774+0000 7f1ee8e5e640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f1ee4152120 0x7f1ee40c90d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:34572/0 (socket says 172.21.15.55:34572) 2024-03-19T02:05:29.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.774+0000 7f1ee8e5e640 1 -- 172.21.15.55:0/2626098839 learned_addr learned my addr 172.21.15.55:0/2626098839 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:05:29.776 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.778+0000 7f1ee965f640 1 -- 172.21.15.55:0/2626098839 >> v2:172.21.15.84:3300/0 conn(0x7f1ee4070780 msgr2=0x7f1ee415b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:29.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.778+0000 7f1ee965f640 1 --2- 172.21.15.55:0/2626098839 >> v2:172.21.15.84:3300/0 conn(0x7f1ee4070780 0x7f1ee415b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:29.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.778+0000 7f1ee965f640 1 -- 172.21.15.55:0/2626098839 >> v1:172.21.15.84:6789/0 conn(0x7f1ee4152120 legacy=0x7f1ee40c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:29.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.778+0000 7f1ee965f640 1 -- 172.21.15.55:0/2626098839 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ee406fff0 con 0x7f1ee415bb00 2024-03-19T02:05:29.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.782+0000 7f1ee965f640 1 --2- 172.21.15.55:0/2626098839 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ee415bb00 0x7f1ee415bf90 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f1ee40cb130 tx=0x7f1edc01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf1a786549458f0e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:29.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.786+0000 7f1ee37fe640 1 -- 172.21.15.55:0/2626098839 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1edc0b2070 con 0x7f1ee415bb00 2024-03-19T02:05:29.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.786+0000 7f1ee37fe640 1 -- 172.21.15.55:0/2626098839 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1edc0a57f0 con 0x7f1ee415bb00 2024-03-19T02:05:29.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.786+0000 7f1ee37fe640 1 -- 172.21.15.55:0/2626098839 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1edc0a5bc0 con 0x7f1ee415bb00 2024-03-19T02:05:29.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.790+0000 7f1eeb0c2640 1 -- 172.21.15.55:0/2626098839 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ee415bb00 msgr2=0x7f1ee415bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:29.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.790+0000 7f1eeb0c2640 1 --2- 172.21.15.55:0/2626098839 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ee415bb00 0x7f1ee415bf90 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f1ee40cb130 tx=0x7f1edc01c930 comp rx=0 tx=0).stop 2024-03-19T02:05:29.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.790+0000 7f1eeb0c2640 1 -- 172.21.15.55:0/2626098839 shutdown_connections 2024-03-19T02:05:29.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.790+0000 7f1eeb0c2640 1 --2- 172.21.15.55:0/2626098839 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ee415bb00 0x7f1ee415bf90 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:29.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.790+0000 7f1eeb0c2640 1 --2- 172.21.15.55:0/2626098839 >> v2:172.21.15.84:3300/0 conn(0x7f1ee4070780 0x7f1ee415b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:29.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.790+0000 7f1eeb0c2640 1 -- 172.21.15.55:0/2626098839 >> 172.21.15.55:0/2626098839 conn(0x7f1ee4067600 msgr2=0x7f1ee4071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:29.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.790+0000 7f1eeb0c2640 1 -- 172.21.15.55:0/2626098839 shutdown_connections 2024-03-19T02:05:29.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.790+0000 7f1eeb0c2640 1 -- 172.21.15.55:0/2626098839 wait complete. 2024-03-19T02:05:29.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.790+0000 7f1eeb0c2640 1 Processor -- start 2024-03-19T02:05:29.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.790+0000 7f1eeb0c2640 1 -- start start 2024-03-19T02:05:29.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.790+0000 7f1eeb0c2640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1ee4070780 0x7f1ee41f3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:29.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.790+0000 7f1eeb0c2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ee4152120 0x7f1ee414ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:29.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.790+0000 7f1eeb0c2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ee415bb00 0x7f1ee415a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:29.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.790+0000 7f1eeb0c2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee40c31c0 con 0x7f1ee4152120 2024-03-19T02:05:29.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.790+0000 7f1eeb0c2640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee40c2ee0 con 0x7f1ee4070780 2024-03-19T02:05:29.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.790+0000 7f1eeb0c2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1ee40c97e0 con 0x7f1ee415bb00 2024-03-19T02:05:29.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.794+0000 7f1ee8e5e640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1ee4070780 0x7f1ee41f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:29.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.794+0000 7f1ee3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ee4152120 0x7f1ee414ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:29.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.794+0000 7f1ee965f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ee415bb00 0x7f1ee415a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:29.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.794+0000 7f1ee3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ee4152120 0x7f1ee414ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41794/0 (socket says 172.21.15.55:41794) 2024-03-19T02:05:29.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.794+0000 7f1ee3fff640 1 -- 172.21.15.55:0/1184587270 learned_addr learned my addr 172.21.15.55:0/1184587270 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:29.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.794+0000 7f1ee3fff640 1 -- 172.21.15.55:0/1184587270 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ee415bb00 msgr2=0x7f1ee415a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:29.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.794+0000 7f1ee3fff640 1 --2- 172.21.15.55:0/1184587270 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ee415bb00 0x7f1ee415a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:29.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.794+0000 7f1ee3fff640 1 -- 172.21.15.55:0/1184587270 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1ee4070780 msgr2=0x7f1ee41f3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:29.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.794+0000 7f1ee3fff640 1 --2- 172.21.15.55:0/1184587270 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1ee4070780 0x7f1ee41f3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:29.796 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.794+0000 7f1ee3fff640 1 -- 172.21.15.55:0/1184587270 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ee414bfb0 con 0x7f1ee4152120 2024-03-19T02:05:29.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.798+0000 7f1ee3fff640 1 --2- 172.21.15.55:0/1184587270 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ee4152120 0x7f1ee414ccb0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f1ed000e960 tx=0x7f1ed000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:29.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.802+0000 7f1ee17fa640 1 -- 172.21.15.55:0/1184587270 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0017070 con 0x7f1ee4152120 2024-03-19T02:05:29.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.802+0000 7f1ee17fa640 1 -- 172.21.15.55:0/1184587270 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ed000ca00 con 0x7f1ee4152120 2024-03-19T02:05:29.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.802+0000 7f1eeb0c2640 1 -- 172.21.15.55:0/1184587270 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ee4162090 con 0x7f1ee4152120 2024-03-19T02:05:29.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.802+0000 7f1ee17fa640 1 -- 172.21.15.55:0/1184587270 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed000cdb0 con 0x7f1ee4152120 2024-03-19T02:05:29.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.802+0000 7f1eeb0c2640 1 -- 172.21.15.55:0/1184587270 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ee414c4b0 con 0x7f1ee4152120 2024-03-19T02:05:29.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:29.802+0000 7f1eeb0c2640 1 -- 172.21.15.55:0/1184587270 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ee4068a40 con 0x7f1ee4152120 2024-03-19T02:05:30.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:30.094+0000 7f1ee17fa640 1 -- 172.21.15.55:0/1184587270 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1ed001a070 con 0x7f1ee4152120 2024-03-19T02:05:30.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:30.094+0000 7f1ee17fa640 1 --2- 172.21.15.55:0/1184587270 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ec80676b0 0x7f1ec8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:30.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:30.094+0000 7f1ee8e5e640 1 --2- 172.21.15.55:0/1184587270 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ec80676b0 0x7f1ec8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:30.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:30.094+0000 7f1ee8e5e640 1 --2- 172.21.15.55:0/1184587270 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ec80676b0 0x7f1ec8069b70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8000f70 tx=0x7f1ed8007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:30.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:30.110+0000 7f1ee17fa640 1 -- 172.21.15.55:0/1184587270 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f1ed0054720 con 0x7f1ee4152120 2024-03-19T02:05:30.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:30.110+0000 7f1ee17fa640 1 -- 172.21.15.55:0/1184587270 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1ed0056680 con 0x7f1ee4152120 2024-03-19T02:05:30.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:30.322+0000 7f1eeb0c2640 1 -- 172.21.15.55:0/1184587270 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ee40c84b0 con 0x7f1ee4152120 2024-03-19T02:05:30.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:30.334+0000 7f1ee17fa640 1 -- 172.21.15.55:0/1184587270 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f1ed005a720 con 0x7f1ee4152120 2024-03-19T02:05:30.336 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:30.336 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:30.336 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:30.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:30.338+0000 7f1eeb0c2640 1 -- 172.21.15.55:0/1184587270 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ec80676b0 msgr2=0x7f1ec8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:30.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:30.338+0000 7f1eeb0c2640 1 --2- 172.21.15.55:0/1184587270 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ec80676b0 0x7f1ec8069b70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8000f70 tx=0x7f1ed8007e90 comp rx=0 tx=0).stop 2024-03-19T02:05:30.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:30.338+0000 7f1eeb0c2640 1 -- 172.21.15.55:0/1184587270 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ee4152120 msgr2=0x7f1ee414ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:30.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:30.338+0000 7f1eeb0c2640 1 --2- 172.21.15.55:0/1184587270 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ee4152120 0x7f1ee414ccb0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f1ed000e960 tx=0x7f1ed000ee20 comp rx=0 tx=0).stop 2024-03-19T02:05:30.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:30.338+0000 7f1eeb0c2640 1 -- 172.21.15.55:0/1184587270 shutdown_connections 2024-03-19T02:05:30.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:30.338+0000 7f1eeb0c2640 1 --2- 172.21.15.55:0/1184587270 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1ec80676b0 0x7f1ec8069b70 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:30.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:30.338+0000 7f1eeb0c2640 1 --2- 172.21.15.55:0/1184587270 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1ee415bb00 0x7f1ee415a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:30.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:30.338+0000 7f1eeb0c2640 1 --2- 172.21.15.55:0/1184587270 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ee4152120 0x7f1ee414ccb0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:30.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:30.338+0000 7f1eeb0c2640 1 --2- 172.21.15.55:0/1184587270 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1ee4070780 0x7f1ee41f3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:30.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:30.338+0000 7f1eeb0c2640 1 -- 172.21.15.55:0/1184587270 >> 172.21.15.55:0/1184587270 conn(0x7f1ee4067600 msgr2=0x7f1ee41508a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:30.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:30.338+0000 7f1eeb0c2640 1 -- 172.21.15.55:0/1184587270 shutdown_connections 2024-03-19T02:05:30.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:30.338+0000 7f1eeb0c2640 1 -- 172.21.15.55:0/1184587270 wait complete. 2024-03-19T02:05:30.964 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:30.964 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:31.965 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:32.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.098+0000 7f2f77bdf640 1 Processor -- start 2024-03-19T02:05:32.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.098+0000 7f2f77bdf640 1 -- start start 2024-03-19T02:05:32.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.098+0000 7f2f77bdf640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f2f70070f30 0x7f2f70158060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:32.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.098+0000 7f2f77bdf640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f701585a0 0x7f2f7006b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:32.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.098+0000 7f2f77bdf640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f700c3e10 con 0x7f2f700c3a30 2024-03-19T02:05:32.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.098+0000 7f2f77bdf640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f2f7013e240 con 0x7f2f70070f30 2024-03-19T02:05:32.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.098+0000 7f2f77bdf640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2f700c8390 con 0x7f2f701585a0 2024-03-19T02:05:32.099 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.098+0000 7f2f7517a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f2f70070f30 0x7f2f70158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:32.100 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.098+0000 7f2f7597b640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f2f700c3a30 0x7f2f700c7600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:47396/0 (socket says 172.21.15.55:47396) 2024-03-19T02:05:32.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.098+0000 7f2f7597b640 1 -- 172.21.15.55:0/2118875301 learned_addr learned my addr 172.21.15.55:0/2118875301 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:05:32.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.098+0000 7f2f7617c640 1 --2- 172.21.15.55:0/2118875301 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f701585a0 0x7f2f7006b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:32.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.102+0000 7f2f74979640 1 -- 172.21.15.55:0/2118875301 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 22707659 0 0) 0x7f2f700c3e10 con 0x7f2f700c3a30 2024-03-19T02:05:32.102 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.102+0000 7f2f74979640 1 -- 172.21.15.55:0/2118875301 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f5c003530 con 0x7f2f700c3a30 2024-03-19T02:05:32.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.102+0000 7f2f7517a640 1 -- 172.21.15.55:0/2118875301 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f701585a0 msgr2=0x7f2f7006b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:32.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.102+0000 7f2f7517a640 1 --2- 172.21.15.55:0/2118875301 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f701585a0 0x7f2f7006b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:32.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.102+0000 7f2f7517a640 1 -- 172.21.15.55:0/2118875301 >> v1:172.21.15.55:6789/0 conn(0x7f2f700c3a30 legacy=0x7f2f700c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:32.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.102+0000 7f2f7517a640 1 -- 172.21.15.55:0/2118875301 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f700c80b0 con 0x7f2f70070f30 2024-03-19T02:05:32.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.106+0000 7f2f7517a640 1 --2- 172.21.15.55:0/2118875301 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f70070f30 0x7f2f70158060 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2f600097f0 tx=0x7f2f6001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b06c5fca747caa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:32.113 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.114+0000 7f2f74979640 1 -- 172.21.15.55:0/2118875301 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f600b1070 con 0x7f2f70070f30 2024-03-19T02:05:32.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.114+0000 7f2f74979640 1 -- 172.21.15.55:0/2118875301 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f600ac410 con 0x7f2f70070f30 2024-03-19T02:05:32.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.114+0000 7f2f74979640 1 -- 172.21.15.55:0/2118875301 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f600ac7e0 con 0x7f2f70070f30 2024-03-19T02:05:32.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.114+0000 7f2f77bdf640 1 -- 172.21.15.55:0/2118875301 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f70070f30 msgr2=0x7f2f70158060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:32.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.114+0000 7f2f77bdf640 1 --2- 172.21.15.55:0/2118875301 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f70070f30 0x7f2f70158060 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2f600097f0 tx=0x7f2f6001c4a0 comp rx=0 tx=0).stop 2024-03-19T02:05:32.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.114+0000 7f2f77bdf640 1 -- 172.21.15.55:0/2118875301 shutdown_connections 2024-03-19T02:05:32.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.114+0000 7f2f77bdf640 1 --2- 172.21.15.55:0/2118875301 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f701585a0 0x7f2f7006b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:32.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.114+0000 7f2f77bdf640 1 --2- 172.21.15.55:0/2118875301 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f70070f30 0x7f2f70158060 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:32.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.114+0000 7f2f77bdf640 1 -- 172.21.15.55:0/2118875301 >> 172.21.15.55:0/2118875301 conn(0x7f2f70070410 msgr2=0x7f2f70070820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:32.115 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.114+0000 7f2f77bdf640 1 -- 172.21.15.55:0/2118875301 shutdown_connections 2024-03-19T02:05:32.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.114+0000 7f2f77bdf640 1 -- 172.21.15.55:0/2118875301 wait complete. 2024-03-19T02:05:32.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.114+0000 7f2f77bdf640 1 Processor -- start 2024-03-19T02:05:32.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.114+0000 7f2f77bdf640 1 -- start start 2024-03-19T02:05:32.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.118+0000 7f2f77bdf640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f70070f30 0x7f2f7014c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:32.117 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.118+0000 7f2f77bdf640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f700c3a30 0x7f2f7014b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:32.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.118+0000 7f2f77bdf640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f701585a0 0x7f2f70137410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:32.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.118+0000 7f2f77bdf640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2f700c7e90 con 0x7f2f701585a0 2024-03-19T02:05:32.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.118+0000 7f2f77bdf640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2f700c7d10 con 0x7f2f70070f30 2024-03-19T02:05:32.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.118+0000 7f2f77bdf640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2f70165200 con 0x7f2f700c3a30 2024-03-19T02:05:32.118 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.118+0000 7f2f7617c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f701585a0 0x7f2f70137410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:32.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.118+0000 7f2f7617c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f701585a0 0x7f2f70137410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41804/0 (socket says 172.21.15.55:41804) 2024-03-19T02:05:32.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.118+0000 7f2f7517a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f700c3a30 0x7f2f7014b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:32.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.118+0000 7f2f7617c640 1 -- 172.21.15.55:0/128056764 learned_addr learned my addr 172.21.15.55:0/128056764 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:32.119 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.118+0000 7f2f7597b640 1 --2- 172.21.15.55:0/128056764 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f70070f30 0x7f2f7014c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:32.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.122+0000 7f2f7617c640 1 -- 172.21.15.55:0/128056764 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f700c3a30 msgr2=0x7f2f7014b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:32.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.122+0000 7f2f7617c640 1 --2- 172.21.15.55:0/128056764 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f700c3a30 0x7f2f7014b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:32.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.122+0000 7f2f7617c640 1 -- 172.21.15.55:0/128056764 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f70070f30 msgr2=0x7f2f7014c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:32.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.122+0000 7f2f7617c640 1 --2- 172.21.15.55:0/128056764 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f70070f30 0x7f2f7014c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:32.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.122+0000 7f2f7617c640 1 -- 172.21.15.55:0/128056764 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f7014d080 con 0x7f2f701585a0 2024-03-19T02:05:32.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.126+0000 7f2f7617c640 1 --2- 172.21.15.55:0/128056764 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f701585a0 0x7f2f70137410 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f2f68009d30 tx=0x7f2f6800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:32.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.126+0000 7f2f667fc640 1 -- 172.21.15.55:0/128056764 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f68018020 con 0x7f2f701585a0 2024-03-19T02:05:32.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.126+0000 7f2f77bdf640 1 -- 172.21.15.55:0/128056764 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f7014d340 con 0x7f2f701585a0 2024-03-19T02:05:32.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.126+0000 7f2f77bdf640 1 -- 172.21.15.55:0/128056764 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f7014d900 con 0x7f2f701585a0 2024-03-19T02:05:32.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.126+0000 7f2f77bdf640 1 -- 172.21.15.55:0/128056764 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f70068a40 con 0x7f2f701585a0 2024-03-19T02:05:32.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.338+0000 7f2f667fc640 1 -- 172.21.15.55:0/128056764 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f68014480 con 0x7f2f701585a0 2024-03-19T02:05:32.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.406+0000 7f2f667fc640 1 -- 172.21.15.55:0/128056764 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f68014830 con 0x7f2f701585a0 2024-03-19T02:05:32.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.410+0000 7f2f667fc640 1 -- 172.21.15.55:0/128056764 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f2f6801b020 con 0x7f2f701585a0 2024-03-19T02:05:32.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.410+0000 7f2f667fc640 1 --2- 172.21.15.55:0/128056764 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f54067690 0x7f2f54069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:32.409 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.410+0000 7f2f7597b640 1 --2- 172.21.15.55:0/128056764 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f54067690 0x7f2f54069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:32.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.410+0000 7f2f7597b640 1 --2- 172.21.15.55:0/128056764 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f54067690 0x7f2f54069b50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c007fd0 tx=0x7f2f6c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:32.422 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.422+0000 7f2f667fc640 1 -- 172.21.15.55:0/128056764 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f2f68055340 con 0x7f2f701585a0 2024-03-19T02:05:32.423 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.422+0000 7f2f667fc640 1 -- 172.21.15.55:0/128056764 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2f68022070 con 0x7f2f701585a0 2024-03-19T02:05:32.635 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.634+0000 7f2f77bdf640 1 -- 172.21.15.55:0/128056764 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f7006b4e0 con 0x7f2f701585a0 2024-03-19T02:05:32.661 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.662+0000 7f2f667fc640 1 -- 172.21.15.55:0/128056764 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f2f68010040 con 0x7f2f701585a0 2024-03-19T02:05:32.662 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:32.662 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:32.662 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:32.663 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.662+0000 7f2f77bdf640 1 -- 172.21.15.55:0/128056764 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f54067690 msgr2=0x7f2f54069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:32.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.662+0000 7f2f77bdf640 1 --2- 172.21.15.55:0/128056764 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f54067690 0x7f2f54069b50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c007fd0 tx=0x7f2f6c009040 comp rx=0 tx=0).stop 2024-03-19T02:05:32.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.662+0000 7f2f77bdf640 1 -- 172.21.15.55:0/128056764 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f701585a0 msgr2=0x7f2f70137410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:32.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.662+0000 7f2f77bdf640 1 --2- 172.21.15.55:0/128056764 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f701585a0 0x7f2f70137410 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f2f68009d30 tx=0x7f2f6800c490 comp rx=0 tx=0).stop 2024-03-19T02:05:32.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.662+0000 7f2f77bdf640 1 -- 172.21.15.55:0/128056764 shutdown_connections 2024-03-19T02:05:32.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.662+0000 7f2f77bdf640 1 --2- 172.21.15.55:0/128056764 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2f54067690 0x7f2f54069b50 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:32.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.662+0000 7f2f77bdf640 1 --2- 172.21.15.55:0/128056764 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2f701585a0 0x7f2f70137410 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:32.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.662+0000 7f2f77bdf640 1 --2- 172.21.15.55:0/128056764 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2f700c3a30 0x7f2f7014b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:32.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.662+0000 7f2f77bdf640 1 --2- 172.21.15.55:0/128056764 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f70070f30 0x7f2f7014c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:32.664 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.662+0000 7f2f77bdf640 1 -- 172.21.15.55:0/128056764 >> 172.21.15.55:0/128056764 conn(0x7f2f70070410 msgr2=0x7f2f701570a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:32.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.662+0000 7f2f77bdf640 1 -- 172.21.15.55:0/128056764 shutdown_connections 2024-03-19T02:05:32.665 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:32.662+0000 7f2f77bdf640 1 -- 172.21.15.55:0/128056764 wait complete. 2024-03-19T02:05:33.286 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:33.286 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:34.287 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:34.401 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.402+0000 7f65cb73f640 1 Processor -- start 2024-03-19T02:05:34.402 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.402+0000 7f65cb73f640 1 -- start start 2024-03-19T02:05:34.402 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.402+0000 7f65cb73f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f65c40c5890 0x7f65c406c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:34.402 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.402+0000 7f65cb73f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f65c40c6fd0 0x7f65c4150d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:34.402 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.402+0000 7f65cb73f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65c41404a0 con 0x7f65c40c6bf0 2024-03-19T02:05:34.402 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.402+0000 7f65cb73f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f65c40c5e10 con 0x7f65c40c6fd0 2024-03-19T02:05:34.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.402+0000 7f65cb73f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f65c40c60f0 con 0x7f65c40c5890 2024-03-19T02:05:34.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.402+0000 7f65c9cdc640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f65c40c6fd0 0x7f65c4150d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:34.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.402+0000 7f65c94db640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f65c40c6bf0 0x7f65c415a480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:34576/0 (socket says 172.21.15.55:34576) 2024-03-19T02:05:34.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.402+0000 7f65c94db640 1 -- 172.21.15.55:0/1067632796 learned_addr learned my addr 172.21.15.55:0/1067632796 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:05:34.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.406+0000 7f65c9cdc640 1 -- 172.21.15.55:0/1067632796 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f65c40c5890 msgr2=0x7f65c406c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:34.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.406+0000 7f65c9cdc640 1 --2- 172.21.15.55:0/1067632796 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f65c40c5890 0x7f65c406c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:34.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.406+0000 7f65c9cdc640 1 -- 172.21.15.55:0/1067632796 >> v1:172.21.15.84:6789/0 conn(0x7f65c40c6bf0 legacy=0x7f65c415a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:34.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.406+0000 7f65c9cdc640 1 -- 172.21.15.55:0/1067632796 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c40c63d0 con 0x7f65c40c6fd0 2024-03-19T02:05:34.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.410+0000 7f65c9cdc640 1 --2- 172.21.15.55:0/1067632796 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65c40c6fd0 0x7f65c4150d60 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f65bc009d90 tx=0x7f65bc01c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97a57df5854d708f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:34.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.414+0000 7f65bbfff640 1 -- 172.21.15.55:0/1067632796 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65bc0b2070 con 0x7f65c40c6fd0 2024-03-19T02:05:34.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.414+0000 7f65bbfff640 1 -- 172.21.15.55:0/1067632796 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65bc0a5a20 con 0x7f65c40c6fd0 2024-03-19T02:05:34.416 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.414+0000 7f65bbfff640 1 -- 172.21.15.55:0/1067632796 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65bc0a5df0 con 0x7f65c40c6fd0 2024-03-19T02:05:34.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.418+0000 7f65cb73f640 1 -- 172.21.15.55:0/1067632796 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65c40c6fd0 msgr2=0x7f65c4150d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:34.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.418+0000 7f65cb73f640 1 --2- 172.21.15.55:0/1067632796 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65c40c6fd0 0x7f65c4150d60 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f65bc009d90 tx=0x7f65bc01c940 comp rx=0 tx=0).stop 2024-03-19T02:05:34.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.418+0000 7f65cb73f640 1 -- 172.21.15.55:0/1067632796 shutdown_connections 2024-03-19T02:05:34.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.418+0000 7f65cb73f640 1 --2- 172.21.15.55:0/1067632796 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65c40c6fd0 0x7f65c4150d60 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:34.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.418+0000 7f65cb73f640 1 --2- 172.21.15.55:0/1067632796 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f65c40c5890 0x7f65c406c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:34.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.418+0000 7f65cb73f640 1 -- 172.21.15.55:0/1067632796 >> 172.21.15.55:0/1067632796 conn(0x7f65c406e5a0 msgr2=0x7f65c4072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:34.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.418+0000 7f65cb73f640 1 -- 172.21.15.55:0/1067632796 shutdown_connections 2024-03-19T02:05:34.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.418+0000 7f65cb73f640 1 -- 172.21.15.55:0/1067632796 wait complete. 2024-03-19T02:05:34.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.418+0000 7f65cb73f640 1 Processor -- start 2024-03-19T02:05:34.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.418+0000 7f65cb73f640 1 -- start start 2024-03-19T02:05:34.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.418+0000 7f65cb73f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f65c40c5890 0x7f65c41f42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:34.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.418+0000 7f65cb73f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f65c40c6bf0 0x7f65c41385e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:34.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.418+0000 7f65cb73f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65c40c6fd0 0x7f65c4136a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:34.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.418+0000 7f65cb73f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f65c406c8f0 con 0x7f65c40c6fd0 2024-03-19T02:05:34.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.418+0000 7f65cb73f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f65c41516b0 con 0x7f65c40c6bf0 2024-03-19T02:05:34.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.418+0000 7f65cb73f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f65c406ca70 con 0x7f65c40c5890 2024-03-19T02:05:34.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.422+0000 7f65c9cdc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65c40c6fd0 0x7f65c4136a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:34.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.422+0000 7f65c9cdc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65c40c6fd0 0x7f65c4136a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41826/0 (socket says 172.21.15.55:41826) 2024-03-19T02:05:34.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.422+0000 7f65c9cdc640 1 -- 172.21.15.55:0/4262100054 learned_addr learned my addr 172.21.15.55:0/4262100054 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:34.421 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.422+0000 7f65c8cda640 1 --2- 172.21.15.55:0/4262100054 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f65c40c6bf0 0x7f65c41385e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:34.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.426+0000 7f65c9cdc640 1 -- 172.21.15.55:0/4262100054 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f65c40c5890 msgr2=0x7f65c41f42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:34.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.426+0000 7f65c9cdc640 1 --2- 172.21.15.55:0/4262100054 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f65c40c5890 0x7f65c41f42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:34.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.426+0000 7f65c9cdc640 1 -- 172.21.15.55:0/4262100054 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f65c40c6bf0 msgr2=0x7f65c41385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:34.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.426+0000 7f65c9cdc640 1 --2- 172.21.15.55:0/4262100054 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f65c40c6bf0 0x7f65c41385e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:34.425 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.426+0000 7f65c9cdc640 1 -- 172.21.15.55:0/4262100054 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c413e8a0 con 0x7f65c40c6fd0 2024-03-19T02:05:34.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.426+0000 7f65c9cdc640 1 --2- 172.21.15.55:0/4262100054 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65c40c6fd0 0x7f65c4136a30 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f65bc01aff0 tx=0x7f65bc0196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:34.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.430+0000 7f65b9ffb640 1 -- 172.21.15.55:0/4262100054 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65bc0b2050 con 0x7f65c40c6fd0 2024-03-19T02:05:34.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.434+0000 7f65b9ffb640 1 -- 172.21.15.55:0/4262100054 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65bc0be070 con 0x7f65c40c6fd0 2024-03-19T02:05:34.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.434+0000 7f65b9ffb640 1 -- 172.21.15.55:0/4262100054 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65bc0adb70 con 0x7f65c40c6fd0 2024-03-19T02:05:34.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.434+0000 7f65cb73f640 1 -- 172.21.15.55:0/4262100054 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65c413eb80 con 0x7f65c40c6fd0 2024-03-19T02:05:34.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.434+0000 7f65cb73f640 1 -- 172.21.15.55:0/4262100054 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65c413f140 con 0x7f65c40c6fd0 2024-03-19T02:05:34.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.434+0000 7f65cb73f640 1 -- 172.21.15.55:0/4262100054 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65c4068a40 con 0x7f65c40c6fd0 2024-03-19T02:05:34.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.718+0000 7f65b9ffb640 1 -- 172.21.15.55:0/4262100054 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f65bc0add10 con 0x7f65c40c6fd0 2024-03-19T02:05:34.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.722+0000 7f65b9ffb640 1 --2- 172.21.15.55:0/4262100054 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f65a80676b0 0x7f65a8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:34.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.722+0000 7f65c94db640 1 --2- 172.21.15.55:0/4262100054 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f65a80676b0 0x7f65a8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:34.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.722+0000 7f65c94db640 1 --2- 172.21.15.55:0/4262100054 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f65a80676b0 0x7f65a8069b70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f65c0000f70 tx=0x7f65c0007a90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:34.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.746+0000 7f65b9ffb640 1 -- 172.21.15.55:0/4262100054 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f65bc0efb80 con 0x7f65c40c6fd0 2024-03-19T02:05:34.746 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.746+0000 7f65b9ffb640 1 -- 172.21.15.55:0/4262100054 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f65bc016560 con 0x7f65c40c6fd0 2024-03-19T02:05:34.972 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:34.970+0000 7f65cb73f640 1 -- 172.21.15.55:0/4262100054 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65c40c9590 con 0x7f65c40c6fd0 2024-03-19T02:05:35.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:35.018+0000 7f65b9ffb640 1 -- 172.21.15.55:0/4262100054 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f65bc0f21e0 con 0x7f65c40c6fd0 2024-03-19T02:05:35.018 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:35.018 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:35.018 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:35.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:35.022+0000 7f65cb73f640 1 -- 172.21.15.55:0/4262100054 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f65a80676b0 msgr2=0x7f65a8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:35.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:35.022+0000 7f65cb73f640 1 --2- 172.21.15.55:0/4262100054 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f65a80676b0 0x7f65a8069b70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f65c0000f70 tx=0x7f65c0007a90 comp rx=0 tx=0).stop 2024-03-19T02:05:35.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:35.022+0000 7f65cb73f640 1 -- 172.21.15.55:0/4262100054 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65c40c6fd0 msgr2=0x7f65c4136a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:35.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:35.022+0000 7f65cb73f640 1 --2- 172.21.15.55:0/4262100054 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65c40c6fd0 0x7f65c4136a30 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f65bc01aff0 tx=0x7f65bc0196e0 comp rx=0 tx=0).stop 2024-03-19T02:05:35.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:35.022+0000 7f65cb73f640 1 -- 172.21.15.55:0/4262100054 shutdown_connections 2024-03-19T02:05:35.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:35.022+0000 7f65cb73f640 1 --2- 172.21.15.55:0/4262100054 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f65a80676b0 0x7f65a8069b70 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:35.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:35.022+0000 7f65cb73f640 1 --2- 172.21.15.55:0/4262100054 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f65c40c6fd0 0x7f65c4136a30 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:35.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:35.022+0000 7f65cb73f640 1 --2- 172.21.15.55:0/4262100054 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f65c40c6bf0 0x7f65c41385e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:35.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:35.022+0000 7f65cb73f640 1 --2- 172.21.15.55:0/4262100054 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f65c40c5890 0x7f65c41f42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:35.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:35.022+0000 7f65cb73f640 1 -- 172.21.15.55:0/4262100054 >> 172.21.15.55:0/4262100054 conn(0x7f65c406e5a0 msgr2=0x7f65c4156560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:35.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:35.022+0000 7f65cb73f640 1 -- 172.21.15.55:0/4262100054 shutdown_connections 2024-03-19T02:05:35.022 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:35.022+0000 7f65cb73f640 1 -- 172.21.15.55:0/4262100054 wait complete. 2024-03-19T02:05:35.642 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:35.643 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:36.644 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:36.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.762+0000 7f3065678640 1 Processor -- start 2024-03-19T02:05:36.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.762+0000 7f3065678640 1 -- start start 2024-03-19T02:05:36.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.762+0000 7f3065678640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3060070f20 0x7f306015d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:36.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.762+0000 7f3065678640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f306015e3a0 con 0x7f30600c96b0 2024-03-19T02:05:36.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.762+0000 7f3065678640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f306015e680 con 0x7f30600c9a90 2024-03-19T02:05:36.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.762+0000 7f3065678640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f30600c7ad0 con 0x7f3060070f20 2024-03-19T02:05:36.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.766+0000 7f305f7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3060070f20 0x7f306015d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:36.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.766+0000 7f305effd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f30600c96b0 0x7f30600c73c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:47400/0 (socket says 172.21.15.55:47400) 2024-03-19T02:05:36.764 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.766+0000 7f305effd640 1 -- 172.21.15.55:0/1234446772 learned_addr learned my addr 172.21.15.55:0/1234446772 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:05:36.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.766+0000 7f305dffb640 1 -- 172.21.15.55:0/1234446772 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1251662682 0 0) 0x7f306015e3a0 con 0x7f30600c96b0 2024-03-19T02:05:36.767 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.766+0000 7f305dffb640 1 -- 172.21.15.55:0/1234446772 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3048003530 con 0x7f30600c96b0 2024-03-19T02:05:36.768 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.770+0000 7f305f7fe640 1 -- 172.21.15.55:0/1234446772 >> v1:172.21.15.84:6789/0 conn(0x7f30600c9a90 legacy=0x7f30601540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:36.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.770+0000 7f305f7fe640 1 -- 172.21.15.55:0/1234446772 >> v1:172.21.15.55:6789/0 conn(0x7f30600c96b0 legacy=0x7f30600c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:36.769 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.770+0000 7f305f7fe640 1 -- 172.21.15.55:0/1234446772 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f306015a330 con 0x7f3060070f20 2024-03-19T02:05:36.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.774+0000 7f305f7fe640 1 --2- 172.21.15.55:0/1234446772 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3060070f20 0x7f306015d810 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3058009dc0 tx=0x7f305801c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe6f4b347fae85a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:36.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.778+0000 7f305dffb640 1 -- 172.21.15.55:0/1234446772 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30580b2070 con 0x7f3060070f20 2024-03-19T02:05:36.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.778+0000 7f305dffb640 1 -- 172.21.15.55:0/1234446772 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30580a57c0 con 0x7f3060070f20 2024-03-19T02:05:36.780 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.778+0000 7f305dffb640 1 -- 172.21.15.55:0/1234446772 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30580a5b90 con 0x7f3060070f20 2024-03-19T02:05:36.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.782+0000 7f3065678640 1 -- 172.21.15.55:0/1234446772 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3060070f20 msgr2=0x7f306015d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:36.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.782+0000 7f3065678640 1 --2- 172.21.15.55:0/1234446772 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3060070f20 0x7f306015d810 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3058009dc0 tx=0x7f305801c900 comp rx=0 tx=0).stop 2024-03-19T02:05:36.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.782+0000 7f3065678640 1 -- 172.21.15.55:0/1234446772 shutdown_connections 2024-03-19T02:05:36.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.782+0000 7f3065678640 1 --2- 172.21.15.55:0/1234446772 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3060070f20 0x7f306015d810 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:36.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.782+0000 7f3065678640 1 -- 172.21.15.55:0/1234446772 >> 172.21.15.55:0/1234446772 conn(0x7f30600cc9c0 msgr2=0x7f30600ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:36.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.782+0000 7f3065678640 1 -- 172.21.15.55:0/1234446772 shutdown_connections 2024-03-19T02:05:36.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.782+0000 7f3065678640 1 -- 172.21.15.55:0/1234446772 wait complete. 2024-03-19T02:05:36.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.782+0000 7f3065678640 1 Processor -- start 2024-03-19T02:05:36.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.782+0000 7f3065678640 1 -- start start 2024-03-19T02:05:36.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.782+0000 7f3065678640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3060070f20 0x7f30601593c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:36.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.782+0000 7f3065678640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30600c96b0 0x7f30600cba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:36.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.782+0000 7f3065678640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f30600c9a90 0x7f30600cbf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:36.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.782+0000 7f3065678640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f30601547d0 con 0x7f30600c96b0 2024-03-19T02:05:36.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.782+0000 7f3065678640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f306015b0f0 con 0x7f3060070f20 2024-03-19T02:05:36.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.782+0000 7f3065678640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3060154950 con 0x7f30600c9a90 2024-03-19T02:05:36.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.782+0000 7f305f7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f30600c9a90 0x7f30600cbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:36.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.782+0000 7f305e7fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30600c96b0 0x7f30600cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:36.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.782+0000 7f305f7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f30600c9a90 0x7f30600cbf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:38208/0 (socket says 172.21.15.55:38208) 2024-03-19T02:05:36.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.782+0000 7f305f7fe640 1 -- 172.21.15.55:0/812754010 learned_addr learned my addr 172.21.15.55:0/812754010 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:36.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.782+0000 7f305effd640 1 --2- 172.21.15.55:0/812754010 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3060070f20 0x7f30601593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:36.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.786+0000 7f305e7fc640 1 -- 172.21.15.55:0/812754010 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f30600c9a90 msgr2=0x7f30600cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:36.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.786+0000 7f305e7fc640 1 --2- 172.21.15.55:0/812754010 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f30600c9a90 0x7f30600cbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:36.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.786+0000 7f305e7fc640 1 -- 172.21.15.55:0/812754010 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3060070f20 msgr2=0x7f30601593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:36.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.786+0000 7f305e7fc640 1 --2- 172.21.15.55:0/812754010 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3060070f20 0x7f30601593c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:36.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.786+0000 7f305e7fc640 1 -- 172.21.15.55:0/812754010 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30601f7180 con 0x7f30600c96b0 2024-03-19T02:05:36.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.790+0000 7f305e7fc640 1 --2- 172.21.15.55:0/812754010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30600c96b0 0x7f30600cba20 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f30600c9eb0 tx=0x7f305400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:36.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.794+0000 7f30477fe640 1 -- 172.21.15.55:0/812754010 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3054017070 con 0x7f30600c96b0 2024-03-19T02:05:36.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.794+0000 7f30477fe640 1 -- 172.21.15.55:0/812754010 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30540052d0 con 0x7f30600c96b0 2024-03-19T02:05:36.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.794+0000 7f3065678640 1 -- 172.21.15.55:0/812754010 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30601f7440 con 0x7f30600c96b0 2024-03-19T02:05:36.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.794+0000 7f30477fe640 1 -- 172.21.15.55:0/812754010 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3054003840 con 0x7f30600c96b0 2024-03-19T02:05:36.794 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.794+0000 7f3065678640 1 -- 172.21.15.55:0/812754010 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30601f7a00 con 0x7f30600c96b0 2024-03-19T02:05:36.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.794+0000 7f30457fa640 1 -- 172.21.15.55:0/812754010 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3060068a40 con 0x7f30600c96b0 2024-03-19T02:05:36.820 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.818+0000 7f30477fe640 1 -- 172.21.15.55:0/812754010 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f305401a070 con 0x7f30600c96b0 2024-03-19T02:05:36.820 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.818+0000 7f30477fe640 1 --2- 172.21.15.55:0/812754010 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f304c0676b0 0x7f304c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:36.820 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.822+0000 7f305effd640 1 --2- 172.21.15.55:0/812754010 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f304c0676b0 0x7f304c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:36.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:36.822+0000 7f305effd640 1 --2- 172.21.15.55:0/812754010 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f304c0676b0 0x7f304c069b70 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3050007fd0 tx=0x7f3050009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:37.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:37.122+0000 7f30477fe640 1 -- 172.21.15.55:0/812754010 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f30540535f0 con 0x7f30600c96b0 2024-03-19T02:05:37.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:37.122+0000 7f30477fe640 1 -- 172.21.15.55:0/812754010 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f3054021020 con 0x7f30600c96b0 2024-03-19T02:05:37.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:37.346+0000 7f30457fa640 1 -- 172.21.15.55:0/812754010 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30600c8c00 con 0x7f30600c96b0 2024-03-19T02:05:37.410 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:37.410+0000 7f30477fe640 1 -- 172.21.15.55:0/812754010 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f3054005480 con 0x7f30600c96b0 2024-03-19T02:05:37.410 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:37.410 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:37.411 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:37.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:37.414+0000 7f30457fa640 1 -- 172.21.15.55:0/812754010 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f304c0676b0 msgr2=0x7f304c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:37.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:37.414+0000 7f30457fa640 1 --2- 172.21.15.55:0/812754010 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f304c0676b0 0x7f304c069b70 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3050007fd0 tx=0x7f3050009040 comp rx=0 tx=0).stop 2024-03-19T02:05:37.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:37.414+0000 7f30457fa640 1 -- 172.21.15.55:0/812754010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30600c96b0 msgr2=0x7f30600cba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:37.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:37.414+0000 7f30457fa640 1 --2- 172.21.15.55:0/812754010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30600c96b0 0x7f30600cba20 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f30600c9eb0 tx=0x7f305400c820 comp rx=0 tx=0).stop 2024-03-19T02:05:37.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:37.414+0000 7f30457fa640 1 -- 172.21.15.55:0/812754010 shutdown_connections 2024-03-19T02:05:37.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:37.414+0000 7f30457fa640 1 --2- 172.21.15.55:0/812754010 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f304c0676b0 0x7f304c069b70 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:37.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:37.414+0000 7f30457fa640 1 --2- 172.21.15.55:0/812754010 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f30600c9a90 0x7f30600cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:37.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:37.414+0000 7f30457fa640 1 --2- 172.21.15.55:0/812754010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30600c96b0 0x7f30600cba20 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:37.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:37.414+0000 7f30457fa640 1 --2- 172.21.15.55:0/812754010 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3060070f20 0x7f30601593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:37.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:37.414+0000 7f30457fa640 1 -- 172.21.15.55:0/812754010 >> 172.21.15.55:0/812754010 conn(0x7f30600cc9c0 msgr2=0x7f3060072de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:37.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:37.414+0000 7f30457fa640 1 -- 172.21.15.55:0/812754010 shutdown_connections 2024-03-19T02:05:37.414 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:37.414+0000 7f30457fa640 1 -- 172.21.15.55:0/812754010 wait complete. 2024-03-19T02:05:38.033 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:38.033 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:39.034 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:39.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.150+0000 7fd83d807640 1 Processor -- start 2024-03-19T02:05:39.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.150+0000 7fd83d807640 1 -- start start 2024-03-19T02:05:39.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.150+0000 7fd83d807640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd83815a0e0 0x7fd8380cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:39.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.150+0000 7fd83d807640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fd8380cc4f0 0x7fd8380c9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:39.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.150+0000 7fd83d807640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd83815a6b0 con 0x7fd8381544b0 2024-03-19T02:05:39.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.150+0000 7fd83d807640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fd83815a990 con 0x7fd83815a0e0 2024-03-19T02:05:39.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.150+0000 7fd83d807640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fd83815ac70 con 0x7fd8380cc4f0 2024-03-19T02:05:39.150 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.150+0000 7fd8367fc640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd83815a0e0 0x7fd8380cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:39.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.150+0000 7fd8367fc640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fd83815a0e0 0x7fd8380cbfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55834/0 (socket says 172.21.15.55:55834) 2024-03-19T02:05:39.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.150+0000 7fd8367fc640 1 -- 172.21.15.55:0/2615923722 learned_addr learned my addr 172.21.15.55:0/2615923722 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:39.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.150+0000 7fd8377fe640 1 --2- 172.21.15.55:0/2615923722 >> v2:172.21.15.84:3300/0 conn(0x7fd8380cc4f0 0x7fd8380c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:39.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.154+0000 7fd8367fc640 1 -- 172.21.15.55:0/2615923722 >> v2:172.21.15.84:3300/0 conn(0x7fd8380cc4f0 msgr2=0x7fd8380c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:39.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.154+0000 7fd8367fc640 1 --2- 172.21.15.55:0/2615923722 >> v2:172.21.15.84:3300/0 conn(0x7fd8380cc4f0 0x7fd8380c9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:39.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.154+0000 7fd8367fc640 1 -- 172.21.15.55:0/2615923722 >> v1:172.21.15.55:6789/0 conn(0x7fd8381544b0 legacy=0x7fd83806b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:39.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.154+0000 7fd8367fc640 1 -- 172.21.15.55:0/2615923722 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd83815af50 con 0x7fd83815a0e0 2024-03-19T02:05:39.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.158+0000 7fd8367fc640 1 --2- 172.21.15.55:0/2615923722 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd83815a0e0 0x7fd8380cbfb0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fd8380c9ed0 tx=0x7fd82c01ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e008fe3b5cb565fe server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:39.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.162+0000 7fd835ffb640 1 -- 172.21.15.55:0/2615923722 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82c0b5070 con 0x7fd83815a0e0 2024-03-19T02:05:39.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.162+0000 7fd835ffb640 1 -- 172.21.15.55:0/2615923722 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd82c0a89d0 con 0x7fd83815a0e0 2024-03-19T02:05:39.163 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.162+0000 7fd835ffb640 1 -- 172.21.15.55:0/2615923722 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82c0a8da0 con 0x7fd83815a0e0 2024-03-19T02:05:39.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.162+0000 7fd83d807640 1 -- 172.21.15.55:0/2615923722 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd83815a0e0 msgr2=0x7fd8380cbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:39.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.162+0000 7fd83d807640 1 --2- 172.21.15.55:0/2615923722 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd83815a0e0 0x7fd8380cbfb0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fd8380c9ed0 tx=0x7fd82c01ecd0 comp rx=0 tx=0).stop 2024-03-19T02:05:39.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.162+0000 7fd83d807640 1 -- 172.21.15.55:0/2615923722 shutdown_connections 2024-03-19T02:05:39.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.162+0000 7fd83d807640 1 --2- 172.21.15.55:0/2615923722 >> v2:172.21.15.84:3300/0 conn(0x7fd8380cc4f0 0x7fd8380c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:39.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.162+0000 7fd83d807640 1 --2- 172.21.15.55:0/2615923722 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd83815a0e0 0x7fd8380cbfb0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:39.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.162+0000 7fd83d807640 1 -- 172.21.15.55:0/2615923722 >> 172.21.15.55:0/2615923722 conn(0x7fd83806e5a0 msgr2=0x7fd838072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:39.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.162+0000 7fd83d807640 1 -- 172.21.15.55:0/2615923722 shutdown_connections 2024-03-19T02:05:39.164 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.166+0000 7fd83d807640 1 -- 172.21.15.55:0/2615923722 wait complete. 2024-03-19T02:05:39.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.166+0000 7fd83d807640 1 Processor -- start 2024-03-19T02:05:39.166 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.166+0000 7fd83d807640 1 -- start start 2024-03-19T02:05:39.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.166+0000 7fd83d807640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8380cc4f0 0x7fd8381534a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:39.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.166+0000 7fd83d807640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8381544b0 0x7fd83814b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:39.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.166+0000 7fd83d807640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd83815a0e0 0x7fd8380c47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:39.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.166+0000 7fd83d807640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd8380c8760 con 0x7fd8380cc4f0 2024-03-19T02:05:39.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.166+0000 7fd83d807640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd8380c6be0 con 0x7fd8381544b0 2024-03-19T02:05:39.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.166+0000 7fd83d807640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fd8380c88e0 con 0x7fd83815a0e0 2024-03-19T02:05:39.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.166+0000 7fd836ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8380cc4f0 0x7fd8381534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:39.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.170+0000 7fd836ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8380cc4f0 0x7fd8381534a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55850/0 (socket says 172.21.15.55:55850) 2024-03-19T02:05:39.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.170+0000 7fd836ffd640 1 -- 172.21.15.55:0/3719821065 learned_addr learned my addr 172.21.15.55:0/3719821065 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:39.168 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.170+0000 7fd8367fc640 1 --2- 172.21.15.55:0/3719821065 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8381544b0 0x7fd83814b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:39.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.170+0000 7fd8377fe640 1 --2- 172.21.15.55:0/3719821065 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd83815a0e0 0x7fd8380c47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:39.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.174+0000 7fd836ffd640 1 -- 172.21.15.55:0/3719821065 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd83815a0e0 msgr2=0x7fd8380c47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:39.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.174+0000 7fd836ffd640 1 --2- 172.21.15.55:0/3719821065 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd83815a0e0 0x7fd8380c47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:39.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.174+0000 7fd836ffd640 1 -- 172.21.15.55:0/3719821065 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8381544b0 msgr2=0x7fd83814b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:39.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.174+0000 7fd836ffd640 1 --2- 172.21.15.55:0/3719821065 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8381544b0 0x7fd83814b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:39.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.174+0000 7fd836ffd640 1 -- 172.21.15.55:0/3719821065 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8381f40c0 con 0x7fd8380cc4f0 2024-03-19T02:05:39.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.174+0000 7fd836ffd640 1 --2- 172.21.15.55:0/3719821065 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8380cc4f0 0x7fd8381534a0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd82800c490 tx=0x7fd828001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:39.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.178+0000 7fd81ffff640 1 -- 172.21.15.55:0/3719821065 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd828017070 con 0x7fd8380cc4f0 2024-03-19T02:05:39.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.178+0000 7fd81ffff640 1 -- 172.21.15.55:0/3719821065 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd828003600 con 0x7fd8380cc4f0 2024-03-19T02:05:39.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.178+0000 7fd83d807640 1 -- 172.21.15.55:0/3719821065 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8381f4380 con 0x7fd8380cc4f0 2024-03-19T02:05:39.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.178+0000 7fd81ffff640 1 -- 172.21.15.55:0/3719821065 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8280039f0 con 0x7fd8380cc4f0 2024-03-19T02:05:39.179 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.178+0000 7fd83d807640 1 -- 172.21.15.55:0/3719821065 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8381f0660 con 0x7fd8380cc4f0 2024-03-19T02:05:39.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.178+0000 7fd83d807640 1 -- 172.21.15.55:0/3719821065 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd838068a40 con 0x7fd8380cc4f0 2024-03-19T02:05:39.472 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.474+0000 7fd81ffff640 1 -- 172.21.15.55:0/3719821065 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fd82801a070 con 0x7fd8380cc4f0 2024-03-19T02:05:39.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.474+0000 7fd81ffff640 1 --2- 172.21.15.55:0/3719821065 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd8240676b0 0x7fd824069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:39.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.474+0000 7fd8367fc640 1 --2- 172.21.15.55:0/3719821065 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd8240676b0 0x7fd824069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:39.474 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.474+0000 7fd8367fc640 1 --2- 172.21.15.55:0/3719821065 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd8240676b0 0x7fd824069b70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd82c0b2000 tx=0x7fd82c0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:39.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.510+0000 7fd81ffff640 1 -- 172.21.15.55:0/3719821065 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fd828053790 con 0x7fd8380cc4f0 2024-03-19T02:05:39.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.510+0000 7fd81ffff640 1 -- 172.21.15.55:0/3719821065 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fd828021020 con 0x7fd8380cc4f0 2024-03-19T02:05:39.720 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.718+0000 7fd83d807640 1 -- 172.21.15.55:0/3719821065 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd83806bb10 con 0x7fd8380cc4f0 2024-03-19T02:05:39.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.782+0000 7fd81ffff640 1 -- 172.21.15.55:0/3719821065 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fd8280556f0 con 0x7fd8380cc4f0 2024-03-19T02:05:39.782 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:39.782 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:39.782 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:39.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.786+0000 7fd83d807640 1 -- 172.21.15.55:0/3719821065 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd8240676b0 msgr2=0x7fd824069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:39.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.786+0000 7fd83d807640 1 --2- 172.21.15.55:0/3719821065 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd8240676b0 0x7fd824069b70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd82c0b2000 tx=0x7fd82c0aa070 comp rx=0 tx=0).stop 2024-03-19T02:05:39.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.786+0000 7fd83d807640 1 -- 172.21.15.55:0/3719821065 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8380cc4f0 msgr2=0x7fd8381534a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:39.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.786+0000 7fd83d807640 1 --2- 172.21.15.55:0/3719821065 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8380cc4f0 0x7fd8381534a0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd82800c490 tx=0x7fd828001e80 comp rx=0 tx=0).stop 2024-03-19T02:05:39.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.786+0000 7fd83d807640 1 -- 172.21.15.55:0/3719821065 shutdown_connections 2024-03-19T02:05:39.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.786+0000 7fd83d807640 1 --2- 172.21.15.55:0/3719821065 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fd8240676b0 0x7fd824069b70 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:39.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.786+0000 7fd83d807640 1 --2- 172.21.15.55:0/3719821065 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fd83815a0e0 0x7fd8380c47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:39.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.786+0000 7fd83d807640 1 --2- 172.21.15.55:0/3719821065 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8381544b0 0x7fd83814b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:39.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.786+0000 7fd83d807640 1 --2- 172.21.15.55:0/3719821065 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8380cc4f0 0x7fd8381534a0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:39.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.786+0000 7fd83d807640 1 -- 172.21.15.55:0/3719821065 >> 172.21.15.55:0/3719821065 conn(0x7fd83806e5a0 msgr2=0x7fd8380ca920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:39.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.786+0000 7fd83d807640 1 -- 172.21.15.55:0/3719821065 shutdown_connections 2024-03-19T02:05:39.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:39.786+0000 7fd83d807640 1 -- 172.21.15.55:0/3719821065 wait complete. 2024-03-19T02:05:40.407 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:40.407 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:41.236 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:05:41.240 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:05:41.408 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:41.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.529+0000 7ff6047a4640 1 Processor -- start 2024-03-19T02:05:41.531 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.529+0000 7ff6047a4640 1 -- start start 2024-03-19T02:05:41.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.529+0000 7ff6047a4640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff5fc0c9780 0x7ff5fc0c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:41.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.529+0000 7ff6047a4640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5fc0c8f70 con 0x7ff5fc15b550 2024-03-19T02:05:41.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.529+0000 7ff6047a4640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5fc0c9250 con 0x7ff5fc13dd00 2024-03-19T02:05:41.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.529+0000 7ff6047a4640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7ff5fc1582b0 con 0x7ff5fc0c9780 2024-03-19T02:05:41.532 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.533+0000 7ff601d3f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff5fc0c9780 0x7ff5fc0c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:41.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.533+0000 7ff601d3f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7ff5fc0c9780 0x7ff5fc0c7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55862/0 (socket says 172.21.15.55:55862) 2024-03-19T02:05:41.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.533+0000 7ff601d3f640 1 -- 172.21.15.55:0/2974736302 learned_addr learned my addr 172.21.15.55:0/2974736302 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:41.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.537+0000 7ff60153e640 1 -- 172.21.15.55:0/2974736302 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2893565478 0 0) 0x7ff5fc0c9250 con 0x7ff5fc13dd00 2024-03-19T02:05:41.538 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.537+0000 7ff60153e640 1 -- 172.21.15.55:0/2974736302 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5e8003530 con 0x7ff5fc13dd00 2024-03-19T02:05:41.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.537+0000 7ff601d3f640 1 -- 172.21.15.55:0/2974736302 >> v1:172.21.15.84:6789/0 conn(0x7ff5fc13dd00 legacy=0x7ff5fc0c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:41.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.537+0000 7ff601d3f640 1 -- 172.21.15.55:0/2974736302 >> v1:172.21.15.55:6789/0 conn(0x7ff5fc15b550 legacy=0x7ff5fc0c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:41.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.537+0000 7ff601d3f640 1 -- 172.21.15.55:0/2974736302 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5fc158590 con 0x7ff5fc0c9780 2024-03-19T02:05:41.546 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.545+0000 7ff601d3f640 1 --2- 172.21.15.55:0/2974736302 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5fc0c9780 0x7ff5fc0c7430 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec009870 tx=0x7ff5ec01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1147fd90d5bb225 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:41.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.549+0000 7ff60153e640 1 -- 172.21.15.55:0/2974736302 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec0a7070 con 0x7ff5fc0c9780 2024-03-19T02:05:41.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.549+0000 7ff60153e640 1 -- 172.21.15.55:0/2974736302 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5ec0b0070 con 0x7ff5fc0c9780 2024-03-19T02:05:41.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.549+0000 7ff60153e640 1 -- 172.21.15.55:0/2974736302 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec0ab890 con 0x7ff5fc0c9780 2024-03-19T02:05:41.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.549+0000 7ff6047a4640 1 -- 172.21.15.55:0/2974736302 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5fc0c9780 msgr2=0x7ff5fc0c7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:41.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.549+0000 7ff6047a4640 1 --2- 172.21.15.55:0/2974736302 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5fc0c9780 0x7ff5fc0c7430 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec009870 tx=0x7ff5ec01c560 comp rx=0 tx=0).stop 2024-03-19T02:05:41.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.549+0000 7ff6047a4640 1 -- 172.21.15.55:0/2974736302 shutdown_connections 2024-03-19T02:05:41.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.549+0000 7ff6047a4640 1 --2- 172.21.15.55:0/2974736302 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5fc0c9780 0x7ff5fc0c7430 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:41.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.549+0000 7ff6047a4640 1 -- 172.21.15.55:0/2974736302 >> 172.21.15.55:0/2974736302 conn(0x7ff5fc06e5a0 msgr2=0x7ff5fc072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:41.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.549+0000 7ff6047a4640 1 -- 172.21.15.55:0/2974736302 shutdown_connections 2024-03-19T02:05:41.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.549+0000 7ff6047a4640 1 -- 172.21.15.55:0/2974736302 wait complete. 2024-03-19T02:05:41.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.553+0000 7ff6047a4640 1 Processor -- start 2024-03-19T02:05:41.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.553+0000 7ff6047a4640 1 -- start start 2024-03-19T02:05:41.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.553+0000 7ff6047a4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5fc0c9780 0x7ff5fc154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:41.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.553+0000 7ff6047a4640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff5fc13dd00 0x7ff5fc15a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:41.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.553+0000 7ff6047a4640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff5fc15b550 0x7ff5fc15aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:41.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.553+0000 7ff6047a4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff5fc0c5fc0 con 0x7ff5fc0c9780 2024-03-19T02:05:41.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.553+0000 7ff6047a4640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff5fc0c5e40 con 0x7ff5fc13dd00 2024-03-19T02:05:41.553 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.553+0000 7ff6047a4640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff5fc166520 con 0x7ff5fc15b550 2024-03-19T02:05:41.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.553+0000 7ff602540640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5fc0c9780 0x7ff5fc154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:41.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.553+0000 7ff602540640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5fc0c9780 0x7ff5fc154270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55864/0 (socket says 172.21.15.55:55864) 2024-03-19T02:05:41.554 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.553+0000 7ff602540640 1 -- 172.21.15.55:0/3020996290 learned_addr learned my addr 172.21.15.55:0/3020996290 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:41.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.553+0000 7ff602d41640 1 --2- 172.21.15.55:0/3020996290 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff5fc15b550 0x7ff5fc15aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:41.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.553+0000 7ff601d3f640 1 --2- 172.21.15.55:0/3020996290 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff5fc13dd00 0x7ff5fc15a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:41.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.557+0000 7ff602540640 1 -- 172.21.15.55:0/3020996290 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff5fc15b550 msgr2=0x7ff5fc15aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:41.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.557+0000 7ff602540640 1 --2- 172.21.15.55:0/3020996290 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff5fc15b550 0x7ff5fc15aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:41.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.557+0000 7ff602540640 1 -- 172.21.15.55:0/3020996290 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff5fc13dd00 msgr2=0x7ff5fc15a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:41.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.557+0000 7ff602540640 1 --2- 172.21.15.55:0/3020996290 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff5fc13dd00 0x7ff5fc15a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:41.558 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.557+0000 7ff602540640 1 -- 172.21.15.55:0/3020996290 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5fc0671a0 con 0x7ff5fc0c9780 2024-03-19T02:05:41.559 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.557+0000 7ff602540640 1 --2- 172.21.15.55:0/3020996290 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5fc0c9780 0x7ff5fc154270 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff5f800c460 tx=0x7ff5f800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:41.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.561+0000 7ff5f2ffd640 1 -- 172.21.15.55:0/3020996290 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f80052d0 con 0x7ff5fc0c9780 2024-03-19T02:05:41.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.561+0000 7ff5f2ffd640 1 -- 172.21.15.55:0/3020996290 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5f8005490 con 0x7ff5fc0c9780 2024-03-19T02:05:41.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.561+0000 7ff5f2ffd640 1 -- 172.21.15.55:0/3020996290 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f8012850 con 0x7ff5fc0c9780 2024-03-19T02:05:41.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.565+0000 7ff6047a4640 1 -- 172.21.15.55:0/3020996290 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5fc135380 con 0x7ff5fc0c9780 2024-03-19T02:05:41.564 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.565+0000 7ff6047a4640 1 -- 172.21.15.55:0/3020996290 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5fc153880 con 0x7ff5fc0c9780 2024-03-19T02:05:41.565 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.565+0000 7ff6047a4640 1 -- 172.21.15.55:0/3020996290 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5fc068a40 con 0x7ff5fc0c9780 2024-03-19T02:05:41.585 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.585+0000 7ff5f2ffd640 1 -- 172.21.15.55:0/3020996290 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ff5f80129f0 con 0x7ff5fc0c9780 2024-03-19T02:05:41.585 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.585+0000 7ff5f2ffd640 1 --2- 172.21.15.55:0/3020996290 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff5e00676b0 0x7ff5e0069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:41.585 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.585+0000 7ff601d3f640 1 --2- 172.21.15.55:0/3020996290 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff5e00676b0 0x7ff5e0069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:41.586 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.585+0000 7ff601d3f640 1 --2- 172.21.15.55:0/3020996290 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff5e00676b0 0x7ff5e0069b70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec0af000 tx=0x7ff5ec0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:41.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.865+0000 7ff5f2ffd640 1 -- 172.21.15.55:0/3020996290 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7ff5f8054590 con 0x7ff5fc0c9780 2024-03-19T02:05:41.869 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:41.865+0000 7ff5f2ffd640 1 -- 172.21.15.55:0/3020996290 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ff5f80564f0 con 0x7ff5fc0c9780 2024-03-19T02:05:42.085 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:42.081+0000 7ff6047a4640 1 -- 172.21.15.55:0/3020996290 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5fc1636d0 con 0x7ff5fc0c9780 2024-03-19T02:05:42.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:42.101+0000 7ff5f2ffd640 1 -- 172.21.15.55:0/3020996290 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7ff5f805a590 con 0x7ff5fc0c9780 2024-03-19T02:05:42.107 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:42.107 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:42.108 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:42.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:42.105+0000 7ff6047a4640 1 -- 172.21.15.55:0/3020996290 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff5e00676b0 msgr2=0x7ff5e0069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:42.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:42.105+0000 7ff6047a4640 1 --2- 172.21.15.55:0/3020996290 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff5e00676b0 0x7ff5e0069b70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec0af000 tx=0x7ff5ec0a7000 comp rx=0 tx=0).stop 2024-03-19T02:05:42.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:42.105+0000 7ff6047a4640 1 -- 172.21.15.55:0/3020996290 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5fc0c9780 msgr2=0x7ff5fc154270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:42.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:42.105+0000 7ff6047a4640 1 --2- 172.21.15.55:0/3020996290 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5fc0c9780 0x7ff5fc154270 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff5f800c460 tx=0x7ff5f800c920 comp rx=0 tx=0).stop 2024-03-19T02:05:42.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:42.105+0000 7ff6047a4640 1 -- 172.21.15.55:0/3020996290 shutdown_connections 2024-03-19T02:05:42.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:42.105+0000 7ff6047a4640 1 --2- 172.21.15.55:0/3020996290 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff5e00676b0 0x7ff5e0069b70 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:42.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:42.105+0000 7ff6047a4640 1 --2- 172.21.15.55:0/3020996290 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff5fc15b550 0x7ff5fc15aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:42.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:42.105+0000 7ff6047a4640 1 --2- 172.21.15.55:0/3020996290 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff5fc13dd00 0x7ff5fc15a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:42.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:42.105+0000 7ff6047a4640 1 --2- 172.21.15.55:0/3020996290 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5fc0c9780 0x7ff5fc154270 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:42.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:42.105+0000 7ff6047a4640 1 -- 172.21.15.55:0/3020996290 >> 172.21.15.55:0/3020996290 conn(0x7ff5fc06e5a0 msgr2=0x7ff5fc13b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:42.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:42.105+0000 7ff6047a4640 1 -- 172.21.15.55:0/3020996290 shutdown_connections 2024-03-19T02:05:42.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:42.105+0000 7ff6047a4640 1 -- 172.21.15.55:0/3020996290 wait complete. 2024-03-19T02:05:42.726 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:42.727 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:43.728 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:43.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.841+0000 7f1b8deb3640 1 Processor -- start 2024-03-19T02:05:43.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.841+0000 7f1b8deb3640 1 -- start start 2024-03-19T02:05:43.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.841+0000 7f1b8deb3640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b8814eaa0 0x7f1b8806b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:43.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.841+0000 7f1b8deb3640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1b881585f0 0x7f1b88158a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:43.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.841+0000 7f1b8deb3640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b8814ee80 con 0x7f1b8806bec0 2024-03-19T02:05:43.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.841+0000 7f1b8deb3640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f1b880c7540 con 0x7f1b881585f0 2024-03-19T02:05:43.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.841+0000 7f1b8deb3640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1b880c77c0 con 0x7f1b8814eaa0 2024-03-19T02:05:43.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.845+0000 7f1b87fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1b881585f0 0x7f1b88158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:43.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.845+0000 7f1b87fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f1b881585f0 0x7f1b88158a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55872/0 (socket says 172.21.15.55:55872) 2024-03-19T02:05:43.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.845+0000 7f1b87fff640 1 -- 172.21.15.55:0/2702324825 learned_addr learned my addr 172.21.15.55:0/2702324825 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:43.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.845+0000 7f1b877fe640 1 --2- 172.21.15.55:0/2702324825 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b8814eaa0 0x7f1b8806b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:43.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.845+0000 7f1b87fff640 1 -- 172.21.15.55:0/2702324825 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b8814eaa0 msgr2=0x7f1b8806b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:43.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.845+0000 7f1b87fff640 1 --2- 172.21.15.55:0/2702324825 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b8814eaa0 0x7f1b8806b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:43.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.845+0000 7f1b87fff640 1 -- 172.21.15.55:0/2702324825 >> v1:172.21.15.84:6789/0 conn(0x7f1b8806bec0 legacy=0x7f1b880c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:43.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.845+0000 7f1b87fff640 1 -- 172.21.15.55:0/2702324825 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b880c7aa0 con 0x7f1b881585f0 2024-03-19T02:05:43.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.849+0000 7f1b87fff640 1 --2- 172.21.15.55:0/2702324825 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b881585f0 0x7f1b88158a40 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f1b80009e50 tx=0x7f1b8001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e1742fa9a5280 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:43.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.857+0000 7f1b867fc640 1 -- 172.21.15.55:0/2702324825 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b800a5770 con 0x7f1b881585f0 2024-03-19T02:05:43.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.857+0000 7f1b867fc640 1 -- 172.21.15.55:0/2702324825 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b800a5910 con 0x7f1b881585f0 2024-03-19T02:05:43.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.857+0000 7f1b867fc640 1 -- 172.21.15.55:0/2702324825 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b800a5ce0 con 0x7f1b881585f0 2024-03-19T02:05:43.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.857+0000 7f1b8deb3640 1 -- 172.21.15.55:0/2702324825 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b881585f0 msgr2=0x7f1b88158a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:43.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.857+0000 7f1b8deb3640 1 --2- 172.21.15.55:0/2702324825 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b881585f0 0x7f1b88158a40 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f1b80009e50 tx=0x7f1b8001c9a0 comp rx=0 tx=0).stop 2024-03-19T02:05:43.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.857+0000 7f1b8deb3640 1 -- 172.21.15.55:0/2702324825 shutdown_connections 2024-03-19T02:05:43.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.857+0000 7f1b8deb3640 1 --2- 172.21.15.55:0/2702324825 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b881585f0 0x7f1b88158a40 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:43.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.857+0000 7f1b8deb3640 1 --2- 172.21.15.55:0/2702324825 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b8814eaa0 0x7f1b8806b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:43.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.857+0000 7f1b8deb3640 1 -- 172.21.15.55:0/2702324825 >> 172.21.15.55:0/2702324825 conn(0x7f1b880cc9c0 msgr2=0x7f1b880ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:43.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.857+0000 7f1b8deb3640 1 -- 172.21.15.55:0/2702324825 shutdown_connections 2024-03-19T02:05:43.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.857+0000 7f1b8deb3640 1 -- 172.21.15.55:0/2702324825 wait complete. 2024-03-19T02:05:43.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.857+0000 7f1b8deb3640 1 Processor -- start 2024-03-19T02:05:43.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.857+0000 7f1b8deb3640 1 -- start start 2024-03-19T02:05:43.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.861+0000 7f1b8deb3640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b8806bec0 0x7f1b88160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:43.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.861+0000 7f1b8deb3640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b8814eaa0 0x7f1b880c56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:43.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.861+0000 7f1b8deb3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b881585f0 0x7f1b88146fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:43.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.861+0000 7f1b8deb3640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1b880c6960 con 0x7f1b881585f0 2024-03-19T02:05:43.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.861+0000 7f1b8deb3640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f1b880c67e0 con 0x7f1b8806bec0 2024-03-19T02:05:43.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.861+0000 7f1b8deb3640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f1b880c6bf0 con 0x7f1b8814eaa0 2024-03-19T02:05:43.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.861+0000 7f1b86ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b8814eaa0 0x7f1b880c56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:43.861 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.861+0000 7f1b87fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b881585f0 0x7f1b88146fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:43.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.861+0000 7f1b86ffd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b8814eaa0 0x7f1b880c56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:59166/0 (socket says 172.21.15.55:59166) 2024-03-19T02:05:43.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.861+0000 7f1b86ffd640 1 -- 172.21.15.55:0/1600163147 learned_addr learned my addr 172.21.15.55:0/1600163147 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:43.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.861+0000 7f1b877fe640 1 --2- 172.21.15.55:0/1600163147 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b8806bec0 0x7f1b88160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:43.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.865+0000 7f1b87fff640 1 -- 172.21.15.55:0/1600163147 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b8814eaa0 msgr2=0x7f1b880c56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:43.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.865+0000 7f1b87fff640 1 --2- 172.21.15.55:0/1600163147 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b8814eaa0 0x7f1b880c56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:43.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.865+0000 7f1b87fff640 1 -- 172.21.15.55:0/1600163147 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b8806bec0 msgr2=0x7f1b88160ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:43.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.865+0000 7f1b87fff640 1 --2- 172.21.15.55:0/1600163147 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b8806bec0 0x7f1b88160ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:43.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.865+0000 7f1b87fff640 1 -- 172.21.15.55:0/1600163147 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b88147de0 con 0x7f1b881585f0 2024-03-19T02:05:43.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.865+0000 7f1b87fff640 1 --2- 172.21.15.55:0/1600163147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b881585f0 0x7f1b88146fa0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f1b800ab000 tx=0x7f1b80019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:43.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.869+0000 7f1b6ffff640 1 -- 172.21.15.55:0/1600163147 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b800bc070 con 0x7f1b881585f0 2024-03-19T02:05:43.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.869+0000 7f1b6ffff640 1 -- 172.21.15.55:0/1600163147 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b800b8640 con 0x7f1b881585f0 2024-03-19T02:05:43.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.869+0000 7f1b8deb3640 1 -- 172.21.15.55:0/1600163147 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b880cc360 con 0x7f1b881585f0 2024-03-19T02:05:43.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.869+0000 7f1b6ffff640 1 -- 172.21.15.55:0/1600163147 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b80009510 con 0x7f1b881585f0 2024-03-19T02:05:43.872 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.869+0000 7f1b8deb3640 1 -- 172.21.15.55:0/1600163147 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b881f32c0 con 0x7f1b881585f0 2024-03-19T02:05:43.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:43.873+0000 7f1b8deb3640 1 -- 172.21.15.55:0/1600163147 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b88068a40 con 0x7f1b881585f0 2024-03-19T02:05:44.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:44.153+0000 7f1b6ffff640 1 -- 172.21.15.55:0/1600163147 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f1b800b8a30 con 0x7f1b881585f0 2024-03-19T02:05:44.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:44.153+0000 7f1b6ffff640 1 --2- 172.21.15.55:0/1600163147 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1b740676b0 0x7f1b74069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:44.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:44.153+0000 7f1b877fe640 1 --2- 172.21.15.55:0/1600163147 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1b740676b0 0x7f1b74069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:44.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:44.157+0000 7f1b877fe640 1 --2- 172.21.15.55:0/1600163147 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1b740676b0 0x7f1b74069b70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1b78009b60 tx=0x7f1b78009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:44.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:44.189+0000 7f1b6ffff640 1 -- 172.21.15.55:0/1600163147 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f1b800eedf0 con 0x7f1b881585f0 2024-03-19T02:05:44.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:44.189+0000 7f1b6ffff640 1 -- 172.21.15.55:0/1600163147 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f1b80016620 con 0x7f1b881585f0 2024-03-19T02:05:44.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:44.401+0000 7f1b8deb3640 1 -- 172.21.15.55:0/1600163147 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b880729d0 con 0x7f1b881585f0 2024-03-19T02:05:44.424 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:44.421+0000 7f1b6ffff640 1 -- 172.21.15.55:0/1600163147 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f1b800f0d50 con 0x7f1b881585f0 2024-03-19T02:05:44.425 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:44.425 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:44.425 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:44.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:44.425+0000 7f1b8deb3640 1 -- 172.21.15.55:0/1600163147 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1b740676b0 msgr2=0x7f1b74069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:44.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:44.425+0000 7f1b8deb3640 1 --2- 172.21.15.55:0/1600163147 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1b740676b0 0x7f1b74069b70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1b78009b60 tx=0x7f1b78009190 comp rx=0 tx=0).stop 2024-03-19T02:05:44.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:44.425+0000 7f1b8deb3640 1 -- 172.21.15.55:0/1600163147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b881585f0 msgr2=0x7f1b88146fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:44.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:44.425+0000 7f1b8deb3640 1 --2- 172.21.15.55:0/1600163147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b881585f0 0x7f1b88146fa0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f1b800ab000 tx=0x7f1b80019740 comp rx=0 tx=0).stop 2024-03-19T02:05:44.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:44.425+0000 7f1b8deb3640 1 -- 172.21.15.55:0/1600163147 shutdown_connections 2024-03-19T02:05:44.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:44.425+0000 7f1b8deb3640 1 --2- 172.21.15.55:0/1600163147 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f1b740676b0 0x7f1b74069b70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:44.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:44.425+0000 7f1b8deb3640 1 --2- 172.21.15.55:0/1600163147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b881585f0 0x7f1b88146fa0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:44.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:44.425+0000 7f1b8deb3640 1 --2- 172.21.15.55:0/1600163147 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f1b8814eaa0 0x7f1b880c56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:44.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:44.425+0000 7f1b8deb3640 1 --2- 172.21.15.55:0/1600163147 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b8806bec0 0x7f1b88160ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:44.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:44.425+0000 7f1b8deb3640 1 -- 172.21.15.55:0/1600163147 >> 172.21.15.55:0/1600163147 conn(0x7f1b880cc9c0 msgr2=0x7f1b880703e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:44.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:44.425+0000 7f1b8deb3640 1 -- 172.21.15.55:0/1600163147 shutdown_connections 2024-03-19T02:05:44.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:44.425+0000 7f1b8deb3640 1 -- 172.21.15.55:0/1600163147 wait complete. 2024-03-19T02:05:45.049 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:45.049 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:46.050 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:46.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.169+0000 7f7b86c8a640 1 Processor -- start 2024-03-19T02:05:46.169 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.169+0000 7f7b86c8a640 1 -- start start 2024-03-19T02:05:46.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.169+0000 7f7b86c8a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7b8006c610 0x7f7b8015ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:46.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.169+0000 7f7b86c8a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b80070e80 con 0x7f7b8006ceb0 2024-03-19T02:05:46.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.169+0000 7f7b86c8a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b8015b250 con 0x7f7b8006c9f0 2024-03-19T02:05:46.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.169+0000 7f7b86c8a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f7b8006fb00 con 0x7f7b8006c610 2024-03-19T02:05:46.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.169+0000 7f7b84a26640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7b8006c610 0x7f7b8015ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:46.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.169+0000 7f7b84a26640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f7b8006c610 0x7f7b8015ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55888/0 (socket says 172.21.15.55:55888) 2024-03-19T02:05:46.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.169+0000 7f7b84a26640 1 -- 172.21.15.55:0/2451407100 learned_addr learned my addr 172.21.15.55:0/2451407100 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:46.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.173+0000 7f7b84a26640 1 -- 172.21.15.55:0/2451407100 >> v1:172.21.15.84:6789/0 conn(0x7f7b8006c9f0 legacy=0x7f7b800c92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:46.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.173+0000 7f7b84a26640 1 -- 172.21.15.55:0/2451407100 >> v1:172.21.15.55:6789/0 conn(0x7f7b8006ceb0 legacy=0x7f7b801427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:46.178 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.173+0000 7f7b84a26640 1 -- 172.21.15.55:0/2451407100 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b8006fde0 con 0x7f7b8006c610 2024-03-19T02:05:46.180 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.177+0000 7f7b84a26640 1 --2- 172.21.15.55:0/2451407100 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b8006c610 0x7f7b8015ad10 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009870 tx=0x7f7b7401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dbc59337a188a22 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:46.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.185+0000 7f7b7f7fe640 1 -- 172.21.15.55:0/2451407100 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b740ab610 con 0x7f7b8006c610 2024-03-19T02:05:46.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.185+0000 7f7b7f7fe640 1 -- 172.21.15.55:0/2451407100 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b740b0070 con 0x7f7b8006c610 2024-03-19T02:05:46.184 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.185+0000 7f7b7f7fe640 1 -- 172.21.15.55:0/2451407100 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b740ab9d0 con 0x7f7b8006c610 2024-03-19T02:05:46.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.185+0000 7f7b86c8a640 1 -- 172.21.15.55:0/2451407100 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b8006c610 msgr2=0x7f7b8015ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:46.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.185+0000 7f7b86c8a640 1 --2- 172.21.15.55:0/2451407100 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b8006c610 0x7f7b8015ad10 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009870 tx=0x7f7b7401c520 comp rx=0 tx=0).stop 2024-03-19T02:05:46.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.185+0000 7f7b86c8a640 1 -- 172.21.15.55:0/2451407100 shutdown_connections 2024-03-19T02:05:46.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.185+0000 7f7b86c8a640 1 --2- 172.21.15.55:0/2451407100 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b8006c610 0x7f7b8015ad10 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:46.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.185+0000 7f7b86c8a640 1 -- 172.21.15.55:0/2451407100 >> 172.21.15.55:0/2451407100 conn(0x7f7b80067600 msgr2=0x7f7b80071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:46.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.185+0000 7f7b86c8a640 1 -- 172.21.15.55:0/2451407100 shutdown_connections 2024-03-19T02:05:46.187 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.185+0000 7f7b86c8a640 1 -- 172.21.15.55:0/2451407100 wait complete. 2024-03-19T02:05:46.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.189+0000 7f7b86c8a640 1 Processor -- start 2024-03-19T02:05:46.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.189+0000 7f7b86c8a640 1 -- start start 2024-03-19T02:05:46.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.189+0000 7f7b86c8a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b8006c610 0x7f7b800cbc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:46.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.189+0000 7f7b86c8a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7b8006c9f0 0x7f7b800c6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:46.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.189+0000 7f7b86c8a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b8006ceb0 0x7f7b800c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:46.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.189+0000 7f7b86c8a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7b800c99d0 con 0x7f7b8006ceb0 2024-03-19T02:05:46.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.189+0000 7f7b86c8a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f7b80070960 con 0x7f7b8006c9f0 2024-03-19T02:05:46.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.189+0000 7f7b86c8a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f7b80142f00 con 0x7f7b8006c610 2024-03-19T02:05:46.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.189+0000 7f7b85227640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b8006ceb0 0x7f7b800c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:46.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.189+0000 7f7b84a26640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b8006c610 0x7f7b800cbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:46.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.189+0000 7f7b85227640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b8006ceb0 0x7f7b800c6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55898/0 (socket says 172.21.15.55:55898) 2024-03-19T02:05:46.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.189+0000 7f7b85227640 1 -- 172.21.15.55:0/724678633 learned_addr learned my addr 172.21.15.55:0/724678633 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:46.191 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.189+0000 7f7b7ffff640 1 --2- 172.21.15.55:0/724678633 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7b8006c9f0 0x7f7b800c6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:46.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.193+0000 7f7b85227640 1 -- 172.21.15.55:0/724678633 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b8006c610 msgr2=0x7f7b800cbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:46.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.193+0000 7f7b85227640 1 --2- 172.21.15.55:0/724678633 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b8006c610 0x7f7b800cbc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:46.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.193+0000 7f7b85227640 1 -- 172.21.15.55:0/724678633 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7b8006c9f0 msgr2=0x7f7b800c6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:46.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.193+0000 7f7b85227640 1 --2- 172.21.15.55:0/724678633 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7b8006c9f0 0x7f7b800c6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:46.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.193+0000 7f7b85227640 1 -- 172.21.15.55:0/724678633 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b800c5030 con 0x7f7b8006ceb0 2024-03-19T02:05:46.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.197+0000 7f7b85227640 1 --2- 172.21.15.55:0/724678633 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b8006ceb0 0x7f7b800c6f60 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f7b7800b9c0 tx=0x7f7b7800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:46.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.197+0000 7f7b7d7fa640 1 -- 172.21.15.55:0/724678633 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b78019070 con 0x7f7b8006ceb0 2024-03-19T02:05:46.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.197+0000 7f7b7d7fa640 1 -- 172.21.15.55:0/724678633 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b7800cba0 con 0x7f7b8006ceb0 2024-03-19T02:05:46.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.197+0000 7f7b86c8a640 1 -- 172.21.15.55:0/724678633 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b800cc860 con 0x7f7b8006ceb0 2024-03-19T02:05:46.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.197+0000 7f7b7d7fa640 1 -- 172.21.15.55:0/724678633 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b78014440 con 0x7f7b8006ceb0 2024-03-19T02:05:46.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.201+0000 7f7b86c8a640 1 -- 172.21.15.55:0/724678633 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b800cce20 con 0x7f7b8006ceb0 2024-03-19T02:05:46.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.201+0000 7f7b86c8a640 1 -- 172.21.15.55:0/724678633 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b80068a40 con 0x7f7b8006ceb0 2024-03-19T02:05:46.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.221+0000 7f7b7d7fa640 1 -- 172.21.15.55:0/724678633 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f7b7801c070 con 0x7f7b8006ceb0 2024-03-19T02:05:46.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.221+0000 7f7b7d7fa640 1 --2- 172.21.15.55:0/724678633 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7b64067660 0x7f7b64069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:46.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.221+0000 7f7b84a26640 1 --2- 172.21.15.55:0/724678633 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7b64067660 0x7f7b64069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:46.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.221+0000 7f7b84a26640 1 --2- 172.21.15.55:0/724678633 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7b64067660 0x7f7b64069b20 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7b740af000 tx=0x7f7b740a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:46.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.533+0000 7f7b7d7fa640 1 -- 172.21.15.55:0/724678633 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f7b78054070 con 0x7f7b8006ceb0 2024-03-19T02:05:46.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.533+0000 7f7b7d7fa640 1 -- 172.21.15.55:0/724678633 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f7b78023020 con 0x7f7b8006ceb0 2024-03-19T02:05:46.763 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.761+0000 7f7b86c8a640 1 -- 172.21.15.55:0/724678633 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b800c8170 con 0x7f7b8006ceb0 2024-03-19T02:05:46.781 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.781+0000 7f7b7d7fa640 1 -- 172.21.15.55:0/724678633 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f7b78055fd0 con 0x7f7b8006ceb0 2024-03-19T02:05:46.781 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:46.781 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:46.782 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:46.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.781+0000 7f7b86c8a640 1 -- 172.21.15.55:0/724678633 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7b64067660 msgr2=0x7f7b64069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:46.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.781+0000 7f7b86c8a640 1 --2- 172.21.15.55:0/724678633 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7b64067660 0x7f7b64069b20 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7b740af000 tx=0x7f7b740a7000 comp rx=0 tx=0).stop 2024-03-19T02:05:46.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.781+0000 7f7b86c8a640 1 -- 172.21.15.55:0/724678633 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b8006ceb0 msgr2=0x7f7b800c6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:46.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.781+0000 7f7b86c8a640 1 --2- 172.21.15.55:0/724678633 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b8006ceb0 0x7f7b800c6f60 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f7b7800b9c0 tx=0x7f7b7800be80 comp rx=0 tx=0).stop 2024-03-19T02:05:46.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.781+0000 7f7b86c8a640 1 -- 172.21.15.55:0/724678633 shutdown_connections 2024-03-19T02:05:46.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.781+0000 7f7b86c8a640 1 --2- 172.21.15.55:0/724678633 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f7b64067660 0x7f7b64069b20 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:46.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.781+0000 7f7b86c8a640 1 --2- 172.21.15.55:0/724678633 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7b8006ceb0 0x7f7b800c6f60 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:46.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.781+0000 7f7b86c8a640 1 --2- 172.21.15.55:0/724678633 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7b8006c9f0 0x7f7b800c6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:46.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.781+0000 7f7b86c8a640 1 --2- 172.21.15.55:0/724678633 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7b8006c610 0x7f7b800cbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:46.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.781+0000 7f7b86c8a640 1 -- 172.21.15.55:0/724678633 >> 172.21.15.55:0/724678633 conn(0x7f7b80067600 msgr2=0x7f7b800c4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:46.784 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.781+0000 7f7b86c8a640 1 -- 172.21.15.55:0/724678633 shutdown_connections 2024-03-19T02:05:46.785 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:46.781+0000 7f7b86c8a640 1 -- 172.21.15.55:0/724678633 wait complete. 2024-03-19T02:05:47.404 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:47.404 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:48.405 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:48.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.513+0000 7feaa044d640 1 Processor -- start 2024-03-19T02:05:48.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.513+0000 7feaa044d640 1 -- start start 2024-03-19T02:05:48.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.513+0000 7feaa044d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fea980c52b0 0x7fea98151d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:48.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.513+0000 7feaa044d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fea980c89f0 0x7fea98152260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:48.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.513+0000 7feaa044d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea980724c0 con 0x7fea980c5690 2024-03-19T02:05:48.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.513+0000 7feaa044d640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fea9813d290 con 0x7fea980c52b0 2024-03-19T02:05:48.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.513+0000 7feaa044d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fea9813d570 con 0x7fea980c89f0 2024-03-19T02:05:48.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.517+0000 7fea9e1e9640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fea980c52b0 0x7fea98151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:48.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.517+0000 7fea9d9e8640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fea980c89f0 0x7fea98152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:48.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.517+0000 7fea9e9ea640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fea980c5690 0x7fea98158070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:54570/0 (socket says 172.21.15.55:54570) 2024-03-19T02:05:48.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.517+0000 7fea9e9ea640 1 -- 172.21.15.55:0/1578278542 learned_addr learned my addr 172.21.15.55:0/1578278542 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:05:48.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.521+0000 7fea9d9e8640 1 -- 172.21.15.55:0/1578278542 >> v2:172.21.15.84:3300/0 conn(0x7fea980c52b0 msgr2=0x7fea98151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:48.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.521+0000 7fea9d9e8640 1 --2- 172.21.15.55:0/1578278542 >> v2:172.21.15.84:3300/0 conn(0x7fea980c52b0 0x7fea98151d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:48.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.521+0000 7fea9d9e8640 1 -- 172.21.15.55:0/1578278542 >> v1:172.21.15.84:6789/0 conn(0x7fea980c5690 legacy=0x7fea98158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:48.522 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.521+0000 7fea9d9e8640 1 -- 172.21.15.55:0/1578278542 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea9813d850 con 0x7fea980c89f0 2024-03-19T02:05:48.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.525+0000 7fea9d9e8640 1 --2- 172.21.15.55:0/1578278542 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fea980c89f0 0x7fea98152260 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fea880097c0 tx=0x7fea8801ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8df680fa2d43936d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:48.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.533+0000 7fea9d1e7640 1 -- 172.21.15.55:0/1578278542 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea880b5070 con 0x7fea980c89f0 2024-03-19T02:05:48.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.533+0000 7fea9d1e7640 1 -- 172.21.15.55:0/1578278542 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea880a8990 con 0x7fea980c89f0 2024-03-19T02:05:48.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.533+0000 7fea9d1e7640 1 -- 172.21.15.55:0/1578278542 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea880a8d60 con 0x7fea980c89f0 2024-03-19T02:05:48.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.533+0000 7feaa044d640 1 -- 172.21.15.55:0/1578278542 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fea980c89f0 msgr2=0x7fea98152260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:48.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.533+0000 7feaa044d640 1 --2- 172.21.15.55:0/1578278542 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fea980c89f0 0x7fea98152260 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fea880097c0 tx=0x7fea8801ec90 comp rx=0 tx=0).stop 2024-03-19T02:05:48.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.533+0000 7feaa044d640 1 -- 172.21.15.55:0/1578278542 shutdown_connections 2024-03-19T02:05:48.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.533+0000 7feaa044d640 1 --2- 172.21.15.55:0/1578278542 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fea980c89f0 0x7fea98152260 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:48.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.533+0000 7feaa044d640 1 --2- 172.21.15.55:0/1578278542 >> v2:172.21.15.84:3300/0 conn(0x7fea980c52b0 0x7fea98151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:48.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.533+0000 7feaa044d640 1 -- 172.21.15.55:0/1578278542 >> 172.21.15.55:0/1578278542 conn(0x7fea9806f3b0 msgr2=0x7fea9806f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:48.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.533+0000 7feaa044d640 1 -- 172.21.15.55:0/1578278542 shutdown_connections 2024-03-19T02:05:48.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.533+0000 7feaa044d640 1 -- 172.21.15.55:0/1578278542 wait complete. 2024-03-19T02:05:48.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.533+0000 7feaa044d640 1 Processor -- start 2024-03-19T02:05:48.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.533+0000 7feaa044d640 1 -- start start 2024-03-19T02:05:48.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.533+0000 7feaa044d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea980c52b0 0x7fea980c2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:48.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.533+0000 7feaa044d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fea980c5690 0x7fea98156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:48.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.533+0000 7feaa044d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea980c89f0 0x7fea98140540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:48.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.533+0000 7feaa044d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fea98158780 con 0x7fea980c89f0 2024-03-19T02:05:48.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.533+0000 7feaa044d640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fea9813de10 con 0x7fea980c52b0 2024-03-19T02:05:48.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.533+0000 7feaa044d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fea98158900 con 0x7fea980c5690 2024-03-19T02:05:48.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.537+0000 7fea9e9ea640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea980c89f0 0x7fea98140540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:48.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.537+0000 7fea9e1e9640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea980c52b0 0x7fea980c2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:48.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.537+0000 7fea9e9ea640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea980c89f0 0x7fea98140540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:55914/0 (socket says 172.21.15.55:55914) 2024-03-19T02:05:48.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.537+0000 7fea9e9ea640 1 -- 172.21.15.55:0/4095975996 learned_addr learned my addr 172.21.15.55:0/4095975996 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:48.537 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.537+0000 7fea9d9e8640 1 --2- 172.21.15.55:0/4095975996 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fea980c5690 0x7fea98156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:48.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.541+0000 7fea9e9ea640 1 -- 172.21.15.55:0/4095975996 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fea980c5690 msgr2=0x7fea98156e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:48.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.541+0000 7fea9e9ea640 1 --2- 172.21.15.55:0/4095975996 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fea980c5690 0x7fea98156e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:48.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.541+0000 7fea9e9ea640 1 -- 172.21.15.55:0/4095975996 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea980c52b0 msgr2=0x7fea980c2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:48.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.541+0000 7fea9e9ea640 1 --2- 172.21.15.55:0/4095975996 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea980c52b0 0x7fea980c2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:48.542 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.541+0000 7fea9e9ea640 1 -- 172.21.15.55:0/4095975996 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea981f3fa0 con 0x7fea980c89f0 2024-03-19T02:05:48.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.545+0000 7fea9e9ea640 1 --2- 172.21.15.55:0/4095975996 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea980c89f0 0x7fea98140540 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fea9000c370 tx=0x7fea9000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:48.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.549+0000 7fea8effd640 1 -- 172.21.15.55:0/4095975996 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea90017070 con 0x7fea980c89f0 2024-03-19T02:05:48.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.549+0000 7fea8effd640 1 -- 172.21.15.55:0/4095975996 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea90005cf0 con 0x7fea980c89f0 2024-03-19T02:05:48.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.549+0000 7fea8effd640 1 -- 172.21.15.55:0/4095975996 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea900072d0 con 0x7fea980c89f0 2024-03-19T02:05:48.550 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.549+0000 7feaa044d640 1 -- 172.21.15.55:0/4095975996 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea981f4240 con 0x7fea980c89f0 2024-03-19T02:05:48.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.549+0000 7feaa044d640 1 -- 172.21.15.55:0/4095975996 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea98156360 con 0x7fea980c89f0 2024-03-19T02:05:48.551 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.549+0000 7feaa044d640 1 -- 172.21.15.55:0/4095975996 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea98068330 con 0x7fea980c89f0 2024-03-19T02:05:48.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.821+0000 7fea8effd640 1 -- 172.21.15.55:0/4095975996 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fea9001a070 con 0x7fea980c89f0 2024-03-19T02:05:48.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.825+0000 7fea8effd640 1 --2- 172.21.15.55:0/4095975996 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fea7c0676e0 0x7fea7c069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:48.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.825+0000 7fea9e1e9640 1 --2- 172.21.15.55:0/4095975996 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fea7c0676e0 0x7fea7c069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:48.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.825+0000 7fea9e1e9640 1 --2- 172.21.15.55:0/4095975996 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fea7c0676e0 0x7fea7c069ba0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fea940113e0 tx=0x7fea9400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:48.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.849+0000 7fea8effd640 1 -- 172.21.15.55:0/4095975996 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fea90053a40 con 0x7fea980c89f0 2024-03-19T02:05:48.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:48.849+0000 7fea8effd640 1 -- 172.21.15.55:0/4095975996 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fea900559a0 con 0x7fea980c89f0 2024-03-19T02:05:49.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:49.077+0000 7feaa044d640 1 -- 172.21.15.55:0/4095975996 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea9813c600 con 0x7fea980c89f0 2024-03-19T02:05:49.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:49.153+0000 7fea8effd640 1 -- 172.21.15.55:0/4095975996 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fea90059a40 con 0x7fea980c89f0 2024-03-19T02:05:49.153 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:49.153 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:49.154 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:49.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:49.157+0000 7feaa044d640 1 -- 172.21.15.55:0/4095975996 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fea7c0676e0 msgr2=0x7fea7c069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:49.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:49.157+0000 7feaa044d640 1 --2- 172.21.15.55:0/4095975996 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fea7c0676e0 0x7fea7c069ba0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fea940113e0 tx=0x7fea9400b040 comp rx=0 tx=0).stop 2024-03-19T02:05:49.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:49.157+0000 7feaa044d640 1 -- 172.21.15.55:0/4095975996 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea980c89f0 msgr2=0x7fea98140540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:49.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:49.157+0000 7feaa044d640 1 --2- 172.21.15.55:0/4095975996 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea980c89f0 0x7fea98140540 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fea9000c370 tx=0x7fea9000c830 comp rx=0 tx=0).stop 2024-03-19T02:05:49.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:49.157+0000 7feaa044d640 1 -- 172.21.15.55:0/4095975996 shutdown_connections 2024-03-19T02:05:49.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:49.157+0000 7feaa044d640 1 --2- 172.21.15.55:0/4095975996 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fea7c0676e0 0x7fea7c069ba0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:49.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:49.157+0000 7feaa044d640 1 --2- 172.21.15.55:0/4095975996 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea980c89f0 0x7fea98140540 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:49.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:49.157+0000 7feaa044d640 1 --2- 172.21.15.55:0/4095975996 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fea980c5690 0x7fea98156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:49.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:49.157+0000 7feaa044d640 1 --2- 172.21.15.55:0/4095975996 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea980c52b0 0x7fea980c2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:49.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:49.157+0000 7feaa044d640 1 -- 172.21.15.55:0/4095975996 >> 172.21.15.55:0/4095975996 conn(0x7fea9806f3b0 msgr2=0x7fea980c74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:49.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:49.157+0000 7feaa044d640 1 -- 172.21.15.55:0/4095975996 shutdown_connections 2024-03-19T02:05:49.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:49.157+0000 7feaa044d640 1 -- 172.21.15.55:0/4095975996 wait complete. 2024-03-19T02:05:49.777 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:49.777 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:50.778 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:50.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.905+0000 7f3f44029640 1 Processor -- start 2024-03-19T02:05:50.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.905+0000 7f3f44029640 1 -- start start 2024-03-19T02:05:50.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.909+0000 7f3f44029640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f3f3c14eaa0 0x7f3f3c06b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:50.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.909+0000 7f3f44029640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3f3c1585f0 0x7f3f3c158a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:50.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.909+0000 7f3f44029640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f3c14ee80 con 0x7f3f3c06bec0 2024-03-19T02:05:50.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.909+0000 7f3f44029640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f3f3c158f50 con 0x7f3f3c14eaa0 2024-03-19T02:05:50.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.909+0000 7f3f44029640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3f3c0c7540 con 0x7f3f3c1585f0 2024-03-19T02:05:50.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.909+0000 7f3f41dc5640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f3f3c14eaa0 0x7f3f3c06b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:50.910 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.909+0000 7f3f425c6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3f3c1585f0 0x7f3f3c158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:50.910 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.909+0000 7f3f41dc5640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f3f3c14eaa0 0x7f3f3c06b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:37822/0 (socket says 172.21.15.55:37822) 2024-03-19T02:05:50.910 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.909+0000 7f3f41dc5640 1 -- 172.21.15.55:0/272274785 learned_addr learned my addr 172.21.15.55:0/272274785 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:50.913 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.913+0000 7f3f425c6640 1 -- 172.21.15.55:0/272274785 >> v2:172.21.15.84:3300/0 conn(0x7f3f3c14eaa0 msgr2=0x7f3f3c06b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:50.913 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.913+0000 7f3f425c6640 1 --2- 172.21.15.55:0/272274785 >> v2:172.21.15.84:3300/0 conn(0x7f3f3c14eaa0 0x7f3f3c06b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:50.913 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.913+0000 7f3f425c6640 1 -- 172.21.15.55:0/272274785 >> v1:172.21.15.84:6789/0 conn(0x7f3f3c06bec0 legacy=0x7f3f3c0c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:50.913 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.913+0000 7f3f425c6640 1 -- 172.21.15.55:0/272274785 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3c0c77c0 con 0x7f3f3c1585f0 2024-03-19T02:05:50.919 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.917+0000 7f3f425c6640 1 --2- 172.21.15.55:0/272274785 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3f3c1585f0 0x7f3f3c158a10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3f34009d90 tx=0x7f3f3401c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44308fe48facd218 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:50.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.925+0000 7f3f40dc3640 1 -- 172.21.15.55:0/272274785 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f340b2070 con 0x7f3f3c1585f0 2024-03-19T02:05:50.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.925+0000 7f3f40dc3640 1 -- 172.21.15.55:0/272274785 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f340a59a0 con 0x7f3f3c1585f0 2024-03-19T02:05:50.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.925+0000 7f3f40dc3640 1 -- 172.21.15.55:0/272274785 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f340a5d70 con 0x7f3f3c1585f0 2024-03-19T02:05:50.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.929+0000 7f3f44029640 1 -- 172.21.15.55:0/272274785 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3f3c1585f0 msgr2=0x7f3f3c158a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:50.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.929+0000 7f3f44029640 1 --2- 172.21.15.55:0/272274785 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3f3c1585f0 0x7f3f3c158a10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3f34009d90 tx=0x7f3f3401c8e0 comp rx=0 tx=0).stop 2024-03-19T02:05:50.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.929+0000 7f3f44029640 1 -- 172.21.15.55:0/272274785 shutdown_connections 2024-03-19T02:05:50.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.929+0000 7f3f44029640 1 --2- 172.21.15.55:0/272274785 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3f3c1585f0 0x7f3f3c158a10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:50.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.929+0000 7f3f44029640 1 --2- 172.21.15.55:0/272274785 >> v2:172.21.15.84:3300/0 conn(0x7f3f3c14eaa0 0x7f3f3c06b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:50.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.929+0000 7f3f44029640 1 -- 172.21.15.55:0/272274785 >> 172.21.15.55:0/272274785 conn(0x7f3f3c0cc9c0 msgr2=0x7f3f3c0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:50.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.929+0000 7f3f44029640 1 -- 172.21.15.55:0/272274785 shutdown_connections 2024-03-19T02:05:50.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.929+0000 7f3f44029640 1 -- 172.21.15.55:0/272274785 wait complete. 2024-03-19T02:05:50.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.929+0000 7f3f44029640 1 Processor -- start 2024-03-19T02:05:50.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.929+0000 7f3f44029640 1 -- start start 2024-03-19T02:05:50.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.929+0000 7f3f44029640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3f3c06bec0 0x7f3f3c14d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:50.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.929+0000 7f3f44029640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3f3c14eaa0 0x7f3f3c1f4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:50.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.929+0000 7f3f44029640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f3c1585f0 0x7f3f3c1437b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:50.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.929+0000 7f3f44029640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3c0c67e0 con 0x7f3f3c1585f0 2024-03-19T02:05:50.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.929+0000 7f3f44029640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3c0c8060 con 0x7f3f3c06bec0 2024-03-19T02:05:50.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.929+0000 7f3f44029640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3f3c0c6960 con 0x7f3f3c14eaa0 2024-03-19T02:05:50.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.929+0000 7f3f425c6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f3c1585f0 0x7f3f3c1437b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:50.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.929+0000 7f3f41dc5640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3f3c06bec0 0x7f3f3c14d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:50.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.929+0000 7f3f425c6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f3c1585f0 0x7f3f3c1437b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56108/0 (socket says 172.21.15.55:56108) 2024-03-19T02:05:50.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.929+0000 7f3f425c6640 1 -- 172.21.15.55:0/3962237267 learned_addr learned my addr 172.21.15.55:0/3962237267 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:50.934 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.929+0000 7f3f415c4640 1 --2- 172.21.15.55:0/3962237267 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3f3c14eaa0 0x7f3f3c1f4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:50.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.937+0000 7f3f425c6640 1 -- 172.21.15.55:0/3962237267 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3f3c14eaa0 msgr2=0x7f3f3c1f4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:50.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.937+0000 7f3f425c6640 1 --2- 172.21.15.55:0/3962237267 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3f3c14eaa0 0x7f3f3c1f4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:50.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.937+0000 7f3f425c6640 1 -- 172.21.15.55:0/3962237267 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3f3c06bec0 msgr2=0x7f3f3c14d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:50.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.937+0000 7f3f425c6640 1 --2- 172.21.15.55:0/3962237267 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3f3c06bec0 0x7f3f3c14d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:50.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.937+0000 7f3f425c6640 1 -- 172.21.15.55:0/3962237267 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3c1532c0 con 0x7f3f3c1585f0 2024-03-19T02:05:50.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.941+0000 7f3f425c6640 1 --2- 172.21.15.55:0/3962237267 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f3c1585f0 0x7f3f3c1437b0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f3f3401a510 tx=0x7f3f34019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:50.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.945+0000 7f3f327fc640 1 -- 172.21.15.55:0/3962237267 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f340b2040 con 0x7f3f3c1585f0 2024-03-19T02:05:50.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.945+0000 7f3f327fc640 1 -- 172.21.15.55:0/3962237267 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f340bd070 con 0x7f3f3c1585f0 2024-03-19T02:05:50.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.945+0000 7f3f327fc640 1 -- 172.21.15.55:0/3962237267 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f340ad880 con 0x7f3f3c1585f0 2024-03-19T02:05:50.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.945+0000 7f3f44029640 1 -- 172.21.15.55:0/3962237267 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f3c0c9db0 con 0x7f3f3c1585f0 2024-03-19T02:05:50.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.945+0000 7f3f44029640 1 -- 172.21.15.55:0/3962237267 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f3c1549d0 con 0x7f3f3c1585f0 2024-03-19T02:05:50.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:50.949+0000 7f3f44029640 1 -- 172.21.15.55:0/3962237267 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f3c068a40 con 0x7f3f3c1585f0 2024-03-19T02:05:51.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:51.261+0000 7f3f327fc640 1 -- 172.21.15.55:0/3962237267 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f3f340ada20 con 0x7f3f3c1585f0 2024-03-19T02:05:51.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:51.261+0000 7f3f327fc640 1 --2- 172.21.15.55:0/3962237267 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3f200676b0 0x7f3f20069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:51.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:51.261+0000 7f3f41dc5640 1 --2- 172.21.15.55:0/3962237267 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3f200676b0 0x7f3f20069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:51.262 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:51.261+0000 7f3f41dc5640 1 --2- 172.21.15.55:0/3962237267 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3f200676b0 0x7f3f20069b70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009ab0 tx=0x7f3f38009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:51.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:51.301+0000 7f3f327fc640 1 -- 172.21.15.55:0/3962237267 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f3f34002020 con 0x7f3f3c1585f0 2024-03-19T02:05:51.301 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:51.301+0000 7f3f327fc640 1 -- 172.21.15.55:0/3962237267 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f3f34016560 con 0x7f3f3c1585f0 2024-03-19T02:05:51.517 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:51.517+0000 7f3f44029640 1 -- 172.21.15.55:0/3962237267 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f3c0729d0 con 0x7f3f3c1585f0 2024-03-19T02:05:51.594 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:51.593+0000 7f3f327fc640 1 -- 172.21.15.55:0/3962237267 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f3f340f0e50 con 0x7f3f3c1585f0 2024-03-19T02:05:51.594 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:51.595 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:51.595 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:51.598 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:51.597+0000 7f3f44029640 1 -- 172.21.15.55:0/3962237267 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3f200676b0 msgr2=0x7f3f20069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:51.598 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:51.597+0000 7f3f44029640 1 --2- 172.21.15.55:0/3962237267 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3f200676b0 0x7f3f20069b70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009ab0 tx=0x7f3f38009240 comp rx=0 tx=0).stop 2024-03-19T02:05:51.598 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:51.597+0000 7f3f44029640 1 -- 172.21.15.55:0/3962237267 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f3c1585f0 msgr2=0x7f3f3c1437b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:51.598 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:51.597+0000 7f3f44029640 1 --2- 172.21.15.55:0/3962237267 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f3c1585f0 0x7f3f3c1437b0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f3f3401a510 tx=0x7f3f34019680 comp rx=0 tx=0).stop 2024-03-19T02:05:51.598 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:51.597+0000 7f3f44029640 1 -- 172.21.15.55:0/3962237267 shutdown_connections 2024-03-19T02:05:51.598 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:51.597+0000 7f3f44029640 1 --2- 172.21.15.55:0/3962237267 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3f200676b0 0x7f3f20069b70 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:51.598 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:51.597+0000 7f3f44029640 1 --2- 172.21.15.55:0/3962237267 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f3c1585f0 0x7f3f3c1437b0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:51.598 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:51.597+0000 7f3f44029640 1 --2- 172.21.15.55:0/3962237267 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3f3c14eaa0 0x7f3f3c1f4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:51.598 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:51.597+0000 7f3f44029640 1 --2- 172.21.15.55:0/3962237267 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3f3c06bec0 0x7f3f3c14d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:51.598 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:51.597+0000 7f3f44029640 1 -- 172.21.15.55:0/3962237267 >> 172.21.15.55:0/3962237267 conn(0x7f3f3c0cc9c0 msgr2=0x7f3f3c1485c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:51.598 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:51.597+0000 7f3f44029640 1 -- 172.21.15.55:0/3962237267 shutdown_connections 2024-03-19T02:05:51.598 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:51.597+0000 7f3f44029640 1 -- 172.21.15.55:0/3962237267 wait complete. 2024-03-19T02:05:52.218 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:52.218 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:53.220 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:53.342 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.341+0000 7f874c601640 1 Processor -- start 2024-03-19T02:05:53.342 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.341+0000 7f874c601640 1 -- start start 2024-03-19T02:05:53.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.341+0000 7f874c601640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f87440c2420 0x7f8744156770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:53.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.341+0000 7f874c601640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8744150c70 0x7f8744158510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:53.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.341+0000 7f874c601640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8744156e70 con 0x7f87440c2800 2024-03-19T02:05:53.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.341+0000 7f874c601640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f87441576f0 con 0x7f87440c2420 2024-03-19T02:05:53.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.341+0000 7f874c601640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f8744157130 con 0x7f8744150c70 2024-03-19T02:05:53.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.341+0000 7f8749b9c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f87440c2800 0x7f8744150560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:52502/0 (socket says 172.21.15.55:52502) 2024-03-19T02:05:53.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.341+0000 7f8749b9c640 1 -- 172.21.15.55:0/819124020 learned_addr learned my addr 172.21.15.55:0/819124020 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:05:53.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.345+0000 7f874ab9e640 1 --2- 172.21.15.55:0/819124020 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8744150c70 0x7f8744158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:53.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.345+0000 7f874a39d640 1 --2- 172.21.15.55:0/819124020 >> v2:172.21.15.84:3300/0 conn(0x7f87440c2420 0x7f8744156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:53.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.345+0000 7f874939b640 1 -- 172.21.15.55:0/819124020 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2952532049 0 0) 0x7f8744156e70 con 0x7f87440c2800 2024-03-19T02:05:53.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.345+0000 7f874939b640 1 -- 172.21.15.55:0/819124020 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8730003530 con 0x7f87440c2800 2024-03-19T02:05:53.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.345+0000 7f874ab9e640 1 -- 172.21.15.55:0/819124020 >> v2:172.21.15.84:3300/0 conn(0x7f87440c2420 msgr2=0x7f8744156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:53.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.345+0000 7f874ab9e640 1 --2- 172.21.15.55:0/819124020 >> v2:172.21.15.84:3300/0 conn(0x7f87440c2420 0x7f8744156770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:53.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.345+0000 7f874ab9e640 1 -- 172.21.15.55:0/819124020 >> v1:172.21.15.55:6789/0 conn(0x7f87440c2800 legacy=0x7f8744150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:53.348 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.345+0000 7f874ab9e640 1 -- 172.21.15.55:0/819124020 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8744157410 con 0x7f8744150c70 2024-03-19T02:05:53.351 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.349+0000 7f874ab9e640 1 --2- 172.21.15.55:0/819124020 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8744150c70 0x7f8744158510 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f873c009d90 tx=0x7f873c01c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32148fcaff872859 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:53.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.357+0000 7f874939b640 1 -- 172.21.15.55:0/819124020 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f873c0b2070 con 0x7f8744150c70 2024-03-19T02:05:53.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.357+0000 7f874939b640 1 -- 172.21.15.55:0/819124020 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f873c0a5940 con 0x7f8744150c70 2024-03-19T02:05:53.358 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.357+0000 7f874939b640 1 -- 172.21.15.55:0/819124020 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f873c0a5d10 con 0x7f8744150c70 2024-03-19T02:05:53.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.357+0000 7f874c601640 1 -- 172.21.15.55:0/819124020 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8744150c70 msgr2=0x7f8744158510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:53.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.357+0000 7f874c601640 1 --2- 172.21.15.55:0/819124020 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8744150c70 0x7f8744158510 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f873c009d90 tx=0x7f873c01c970 comp rx=0 tx=0).stop 2024-03-19T02:05:53.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.357+0000 7f874c601640 1 -- 172.21.15.55:0/819124020 shutdown_connections 2024-03-19T02:05:53.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.357+0000 7f874c601640 1 --2- 172.21.15.55:0/819124020 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8744150c70 0x7f8744158510 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:53.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.357+0000 7f874c601640 1 --2- 172.21.15.55:0/819124020 >> v2:172.21.15.84:3300/0 conn(0x7f87440c2420 0x7f8744156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:53.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.357+0000 7f874c601640 1 -- 172.21.15.55:0/819124020 >> 172.21.15.55:0/819124020 conn(0x7f874406f3b0 msgr2=0x7f874406f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:53.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.357+0000 7f874c601640 1 -- 172.21.15.55:0/819124020 shutdown_connections 2024-03-19T02:05:53.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.357+0000 7f874c601640 1 -- 172.21.15.55:0/819124020 wait complete. 2024-03-19T02:05:53.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.361+0000 7f874c601640 1 Processor -- start 2024-03-19T02:05:53.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.361+0000 7f874c601640 1 -- start start 2024-03-19T02:05:53.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.361+0000 7f874c601640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87440c2420 0x7f8744140930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:53.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.361+0000 7f874c601640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87440c2800 0x7f87441f6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:53.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.361+0000 7f874c601640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8744150c70 0x7f87441f68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:53.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.361+0000 7f874c601640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8744158e20 con 0x7f87440c2800 2024-03-19T02:05:53.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.361+0000 7f874c601640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f8744158ca0 con 0x7f87440c2420 2024-03-19T02:05:53.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.361+0000 7f874c601640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f8744161010 con 0x7f8744150c70 2024-03-19T02:05:53.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.361+0000 7f874ab9e640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8744150c70 0x7f87441f68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:53.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.361+0000 7f8749b9c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87440c2800 0x7f87441f6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:53.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.361+0000 7f8749b9c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87440c2800 0x7f87441f6370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56112/0 (socket says 172.21.15.55:56112) 2024-03-19T02:05:53.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.361+0000 7f8749b9c640 1 -- 172.21.15.55:0/2779242249 learned_addr learned my addr 172.21.15.55:0/2779242249 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:53.362 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.361+0000 7f874a39d640 1 --2- 172.21.15.55:0/2779242249 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87440c2420 0x7f8744140930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:53.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.365+0000 7f8749b9c640 1 -- 172.21.15.55:0/2779242249 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8744150c70 msgr2=0x7f87441f68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:53.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.365+0000 7f8749b9c640 1 --2- 172.21.15.55:0/2779242249 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8744150c70 0x7f87441f68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:53.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.365+0000 7f8749b9c640 1 -- 172.21.15.55:0/2779242249 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87440c2420 msgr2=0x7f8744140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:53.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.365+0000 7f8749b9c640 1 --2- 172.21.15.55:0/2779242249 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87440c2420 0x7f8744140930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:53.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.365+0000 7f8749b9c640 1 -- 172.21.15.55:0/2779242249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8744071c70 con 0x7f87440c2800 2024-03-19T02:05:53.368 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.369+0000 7f8749b9c640 1 --2- 172.21.15.55:0/2779242249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87440c2800 0x7f87441f6370 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f873400c480 tx=0x7f873400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:53.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.373+0000 7f873affd640 1 -- 172.21.15.55:0/2779242249 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8734017070 con 0x7f87440c2800 2024-03-19T02:05:53.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.373+0000 7f873affd640 1 -- 172.21.15.55:0/2779242249 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8734005cf0 con 0x7f87440c2800 2024-03-19T02:05:53.373 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.373+0000 7f873affd640 1 -- 172.21.15.55:0/2779242249 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8734007290 con 0x7f87440c2800 2024-03-19T02:05:53.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.373+0000 7f874c601640 1 -- 172.21.15.55:0/2779242249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8744071f50 con 0x7f87440c2800 2024-03-19T02:05:53.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.373+0000 7f874c601640 1 -- 172.21.15.55:0/2779242249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f874413f060 con 0x7f87440c2800 2024-03-19T02:05:53.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.373+0000 7f874c601640 1 -- 172.21.15.55:0/2779242249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8744068330 con 0x7f87440c2800 2024-03-19T02:05:53.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.405+0000 7f873affd640 1 -- 172.21.15.55:0/2779242249 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f873401a070 con 0x7f87440c2800 2024-03-19T02:05:53.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.405+0000 7f873affd640 1 --2- 172.21.15.55:0/2779242249 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f87280676e0 0x7f8728069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:53.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.409+0000 7f874a39d640 1 --2- 172.21.15.55:0/2779242249 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f87280676e0 0x7f8728069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:53.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.409+0000 7f874a39d640 1 --2- 172.21.15.55:0/2779242249 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f87280676e0 0x7f8728069ba0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8740009b60 tx=0x7f8740009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:53.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.409+0000 7f873affd640 1 -- 172.21.15.55:0/2779242249 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f87340537a0 con 0x7f87440c2800 2024-03-19T02:05:53.686 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.685+0000 7f873affd640 1 -- 172.21.15.55:0/2779242249 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f8734021020 con 0x7f87440c2800 2024-03-19T02:05:53.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.893+0000 7f874c601640 1 -- 172.21.15.55:0/2779242249 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87441566c0 con 0x7f87440c2800 2024-03-19T02:05:53.933 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.933+0000 7f873affd640 1 -- 172.21.15.55:0/2779242249 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f8734055700 con 0x7f87440c2800 2024-03-19T02:05:53.933 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:53.933 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:53.933 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:53.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.933+0000 7f874c601640 1 -- 172.21.15.55:0/2779242249 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f87280676e0 msgr2=0x7f8728069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:53.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.933+0000 7f874c601640 1 --2- 172.21.15.55:0/2779242249 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f87280676e0 0x7f8728069ba0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8740009b60 tx=0x7f8740009190 comp rx=0 tx=0).stop 2024-03-19T02:05:53.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.933+0000 7f874c601640 1 -- 172.21.15.55:0/2779242249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87440c2800 msgr2=0x7f87441f6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:53.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.933+0000 7f874c601640 1 --2- 172.21.15.55:0/2779242249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87440c2800 0x7f87441f6370 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f873400c480 tx=0x7f873400c940 comp rx=0 tx=0).stop 2024-03-19T02:05:53.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.937+0000 7f874c601640 1 -- 172.21.15.55:0/2779242249 shutdown_connections 2024-03-19T02:05:53.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.937+0000 7f874c601640 1 --2- 172.21.15.55:0/2779242249 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f87280676e0 0x7f8728069ba0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:53.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.937+0000 7f874c601640 1 --2- 172.21.15.55:0/2779242249 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f8744150c70 0x7f87441f68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:53.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.937+0000 7f874c601640 1 --2- 172.21.15.55:0/2779242249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87440c2800 0x7f87441f6370 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:53.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.937+0000 7f874c601640 1 --2- 172.21.15.55:0/2779242249 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87440c2420 0x7f8744140930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:53.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.937+0000 7f874c601640 1 -- 172.21.15.55:0/2779242249 >> 172.21.15.55:0/2779242249 conn(0x7f874406f3b0 msgr2=0x7f874406f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:53.937 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.937+0000 7f874c601640 1 -- 172.21.15.55:0/2779242249 shutdown_connections 2024-03-19T02:05:53.938 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:53.937+0000 7f874c601640 1 -- 172.21.15.55:0/2779242249 wait complete. 2024-03-19T02:05:54.563 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:54.563 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:55.564 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:55.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.693+0000 7f093a91a640 1 Processor -- start 2024-03-19T02:05:55.694 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.693+0000 7f093a91a640 1 -- start start 2024-03-19T02:05:55.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.693+0000 7f093a91a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0934070780 0x7f093415b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:55.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.693+0000 7f093a91a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f093415bb00 0x7f093415bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:55.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.693+0000 7f093a91a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f093415c4d0 con 0x7f0934152120 2024-03-19T02:05:55.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.693+0000 7f093a91a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f0934152500 con 0x7f0934070780 2024-03-19T02:05:55.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.693+0000 7f093a91a640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f0934070590 con 0x7f093415bb00 2024-03-19T02:05:55.695 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.693+0000 7f09337fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0934070780 0x7f093415b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:55.696 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.697+0000 7f09337fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0934070780 0x7f093415b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56114/0 (socket says 172.21.15.55:56114) 2024-03-19T02:05:55.696 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.697+0000 7f09337fe640 1 -- 172.21.15.55:0/631699551 learned_addr learned my addr 172.21.15.55:0/631699551 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:55.696 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.697+0000 7f0938eb7640 1 --2- 172.21.15.55:0/631699551 >> v2:172.21.15.84:3300/0 conn(0x7f093415bb00 0x7f093415bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:55.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.697+0000 7f09337fe640 1 -- 172.21.15.55:0/631699551 >> v2:172.21.15.84:3300/0 conn(0x7f093415bb00 msgr2=0x7f093415bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:55.698 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.697+0000 7f09337fe640 1 --2- 172.21.15.55:0/631699551 >> v2:172.21.15.84:3300/0 conn(0x7f093415bb00 0x7f093415bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:55.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.697+0000 7f09337fe640 1 -- 172.21.15.55:0/631699551 >> v1:172.21.15.84:6789/0 conn(0x7f0934152120 legacy=0x7f09340c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:55.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.697+0000 7f09337fe640 1 -- 172.21.15.55:0/631699551 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f093406fff0 con 0x7f0934070780 2024-03-19T02:05:55.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.701+0000 7f09337fe640 1 --2- 172.21.15.55:0/631699551 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0934070780 0x7f093415b5c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f09340cb650 tx=0x7f092801c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aafb9579aa391e32 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:55.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.705+0000 7f0932ffd640 1 -- 172.21.15.55:0/631699551 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09280b0070 con 0x7f0934070780 2024-03-19T02:05:55.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.705+0000 7f0932ffd640 1 -- 172.21.15.55:0/631699551 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09280ab600 con 0x7f0934070780 2024-03-19T02:05:55.707 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.705+0000 7f0932ffd640 1 -- 172.21.15.55:0/631699551 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09280ab9d0 con 0x7f0934070780 2024-03-19T02:05:55.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.705+0000 7f093a91a640 1 -- 172.21.15.55:0/631699551 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0934070780 msgr2=0x7f093415b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:55.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.705+0000 7f093a91a640 1 --2- 172.21.15.55:0/631699551 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0934070780 0x7f093415b5c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f09340cb650 tx=0x7f092801c300 comp rx=0 tx=0).stop 2024-03-19T02:05:55.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.709+0000 7f093a91a640 1 -- 172.21.15.55:0/631699551 shutdown_connections 2024-03-19T02:05:55.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.709+0000 7f093a91a640 1 --2- 172.21.15.55:0/631699551 >> v2:172.21.15.84:3300/0 conn(0x7f093415bb00 0x7f093415bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:55.708 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.709+0000 7f093a91a640 1 --2- 172.21.15.55:0/631699551 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0934070780 0x7f093415b5c0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:55.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.709+0000 7f093a91a640 1 -- 172.21.15.55:0/631699551 >> 172.21.15.55:0/631699551 conn(0x7f0934067600 msgr2=0x7f0934071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:55.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.709+0000 7f093a91a640 1 -- 172.21.15.55:0/631699551 shutdown_connections 2024-03-19T02:05:55.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.709+0000 7f093a91a640 1 -- 172.21.15.55:0/631699551 wait complete. 2024-03-19T02:05:55.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.709+0000 7f093a91a640 1 Processor -- start 2024-03-19T02:05:55.709 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.709+0000 7f093a91a640 1 -- start start 2024-03-19T02:05:55.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.709+0000 7f093a91a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0934070780 0x7f09340c76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:55.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.709+0000 7f093a91a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0934152120 0x7f093415a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:55.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.709+0000 7f093a91a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f093415bb00 0x7f09341441a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:55.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.709+0000 7f093a91a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f09340c31c0 con 0x7f0934152120 2024-03-19T02:05:55.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.709+0000 7f093a91a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f09340c2ee0 con 0x7f093415bb00 2024-03-19T02:05:55.710 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.709+0000 7f093a91a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f09340c97e0 con 0x7f0934070780 2024-03-19T02:05:55.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.709+0000 7f09337fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0934152120 0x7f093415a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:55.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.709+0000 7f0938eb7640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f093415bb00 0x7f09341441a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:55.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.709+0000 7f09337fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0934152120 0x7f093415a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56130/0 (socket says 172.21.15.55:56130) 2024-03-19T02:05:55.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.709+0000 7f09337fe640 1 -- 172.21.15.55:0/4216294072 learned_addr learned my addr 172.21.15.55:0/4216294072 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:55.711 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.709+0000 7f0933fff640 1 --2- 172.21.15.55:0/4216294072 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0934070780 0x7f09340c76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:55.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.713+0000 7f09337fe640 1 -- 172.21.15.55:0/4216294072 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0934070780 msgr2=0x7f09340c76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:55.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.713+0000 7f09337fe640 1 --2- 172.21.15.55:0/4216294072 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0934070780 0x7f09340c76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:55.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.713+0000 7f09337fe640 1 -- 172.21.15.55:0/4216294072 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f093415bb00 msgr2=0x7f09341441a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:55.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.713+0000 7f09337fe640 1 --2- 172.21.15.55:0/4216294072 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f093415bb00 0x7f09341441a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:55.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.713+0000 7f09337fe640 1 -- 172.21.15.55:0/4216294072 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09340c5c10 con 0x7f0934152120 2024-03-19T02:05:55.717 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.717+0000 7f09337fe640 1 --2- 172.21.15.55:0/4216294072 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0934152120 0x7f093415a650 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f0928018f10 tx=0x7f09280190a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:55.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.721+0000 7f0930ff9640 1 -- 172.21.15.55:0/4216294072 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09280b0050 con 0x7f0934152120 2024-03-19T02:05:55.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.721+0000 7f0930ff9640 1 -- 172.21.15.55:0/4216294072 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09280ba070 con 0x7f0934152120 2024-03-19T02:05:55.721 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.721+0000 7f0930ff9640 1 -- 172.21.15.55:0/4216294072 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09280b47f0 con 0x7f0934152120 2024-03-19T02:05:55.722 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.721+0000 7f093a91a640 1 -- 172.21.15.55:0/4216294072 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09340c5ef0 con 0x7f0934152120 2024-03-19T02:05:55.722 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.721+0000 7f093a91a640 1 -- 172.21.15.55:0/4216294072 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09340c64b0 con 0x7f0934152120 2024-03-19T02:05:55.722 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:55.721+0000 7f093a91a640 1 -- 172.21.15.55:0/4216294072 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0934068a40 con 0x7f0934152120 2024-03-19T02:05:56.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:56.057+0000 7f0930ff9640 1 -- 172.21.15.55:0/4216294072 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f09280b4990 con 0x7f0934152120 2024-03-19T02:05:56.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:56.057+0000 7f0930ff9640 1 --2- 172.21.15.55:0/4216294072 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0920067660 0x7f0920069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:56.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:56.057+0000 7f0933fff640 1 --2- 172.21.15.55:0/4216294072 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0920067660 0x7f0920069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:56.058 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:56.057+0000 7f0933fff640 1 --2- 172.21.15.55:0/4216294072 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0920067660 0x7f0920069b20 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f09240098e0 tx=0x7f0924009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:56.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:56.077+0000 7f0930ff9640 1 -- 172.21.15.55:0/4216294072 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f09280ee6e0 con 0x7f0934152120 2024-03-19T02:05:56.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:56.077+0000 7f0930ff9640 1 -- 172.21.15.55:0/4216294072 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f093415c4d0 con 0x7f0934152120 2024-03-19T02:05:56.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:56.289+0000 7f093a91a640 1 -- 172.21.15.55:0/4216294072 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09341505b0 con 0x7f0934152120 2024-03-19T02:05:56.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:56.321+0000 7f0930ff9640 1 -- 172.21.15.55:0/4216294072 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f09280f0640 con 0x7f0934152120 2024-03-19T02:05:56.324 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:56.325 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:56.325 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:56.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:56.325+0000 7f093a91a640 1 -- 172.21.15.55:0/4216294072 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0920067660 msgr2=0x7f0920069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:56.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:56.325+0000 7f093a91a640 1 --2- 172.21.15.55:0/4216294072 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0920067660 0x7f0920069b20 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f09240098e0 tx=0x7f0924009060 comp rx=0 tx=0).stop 2024-03-19T02:05:56.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:56.325+0000 7f093a91a640 1 -- 172.21.15.55:0/4216294072 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0934152120 msgr2=0x7f093415a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:56.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:56.325+0000 7f093a91a640 1 --2- 172.21.15.55:0/4216294072 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0934152120 0x7f093415a650 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f0928018f10 tx=0x7f09280190a0 comp rx=0 tx=0).stop 2024-03-19T02:05:56.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:56.325+0000 7f093a91a640 1 -- 172.21.15.55:0/4216294072 shutdown_connections 2024-03-19T02:05:56.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:56.325+0000 7f093a91a640 1 --2- 172.21.15.55:0/4216294072 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0920067660 0x7f0920069b20 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:56.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:56.325+0000 7f093a91a640 1 --2- 172.21.15.55:0/4216294072 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f093415bb00 0x7f09341441a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:56.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:56.325+0000 7f093a91a640 1 --2- 172.21.15.55:0/4216294072 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0934152120 0x7f093415a650 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:56.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:56.325+0000 7f093a91a640 1 --2- 172.21.15.55:0/4216294072 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0934070780 0x7f09340c76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:56.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:56.325+0000 7f093a91a640 1 -- 172.21.15.55:0/4216294072 >> 172.21.15.55:0/4216294072 conn(0x7f0934067600 msgr2=0x7f09340cc660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:56.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:56.325+0000 7f093a91a640 1 -- 172.21.15.55:0/4216294072 shutdown_connections 2024-03-19T02:05:56.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:56.325+0000 7f093a91a640 1 -- 172.21.15.55:0/4216294072 wait complete. 2024-03-19T02:05:56.947 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:56.947 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:05:57.948 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:05:58.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.077+0000 7f76325a0640 1 Processor -- start 2024-03-19T02:05:58.078 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.077+0000 7f76325a0640 1 -- start start 2024-03-19T02:05:58.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.077+0000 7f76325a0640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f762c0720b0 0x7f762c157e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:58.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.077+0000 7f76325a0640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f762c06f1f0 con 0x7f762c0c8de0 2024-03-19T02:05:58.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.077+0000 7f76325a0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f762c06f4b0 con 0x7f762c152230 2024-03-19T02:05:58.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.077+0000 7f76325a0640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f762c0700c0 con 0x7f762c0720b0 2024-03-19T02:05:58.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.077+0000 7f762b7fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f762c0720b0 0x7f762c157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:58.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.081+0000 7f762bfff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f762c152230 0x7f762c159350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:49950/0 (socket says 172.21.15.55:49950) 2024-03-19T02:05:58.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.081+0000 7f762bfff640 1 -- 172.21.15.55:0/609285314 learned_addr learned my addr 172.21.15.55:0/609285314 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:05:58.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.081+0000 7f762b7fe640 1 -- 172.21.15.55:0/609285314 >> v1:172.21.15.84:6789/0 conn(0x7f762c152230 legacy=0x7f762c159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:58.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.081+0000 7f762b7fe640 1 -- 172.21.15.55:0/609285314 >> v1:172.21.15.55:6789/0 conn(0x7f762c0c8de0 legacy=0x7f762c0c2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:05:58.084 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.081+0000 7f762b7fe640 1 -- 172.21.15.55:0/609285314 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f762c06f790 con 0x7f762c0720b0 2024-03-19T02:05:58.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.089+0000 7f762b7fe640 1 --2- 172.21.15.55:0/609285314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c0720b0 0x7f762c157e60 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f76200097c0 tx=0x7f762001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba71fe85ba34f12c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:58.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.093+0000 7f762affd640 1 -- 172.21.15.55:0/609285314 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76200b5070 con 0x7f762c0720b0 2024-03-19T02:05:58.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.093+0000 7f762affd640 1 -- 172.21.15.55:0/609285314 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76200a8950 con 0x7f762c0720b0 2024-03-19T02:05:58.093 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.093+0000 7f762affd640 1 -- 172.21.15.55:0/609285314 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76200a8d20 con 0x7f762c0720b0 2024-03-19T02:05:58.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.093+0000 7f76325a0640 1 -- 172.21.15.55:0/609285314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c0720b0 msgr2=0x7f762c157e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:58.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.093+0000 7f76325a0640 1 --2- 172.21.15.55:0/609285314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c0720b0 0x7f762c157e60 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f76200097c0 tx=0x7f762001ec50 comp rx=0 tx=0).stop 2024-03-19T02:05:58.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.093+0000 7f76325a0640 1 -- 172.21.15.55:0/609285314 shutdown_connections 2024-03-19T02:05:58.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.093+0000 7f76325a0640 1 --2- 172.21.15.55:0/609285314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c0720b0 0x7f762c157e60 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:58.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.093+0000 7f76325a0640 1 -- 172.21.15.55:0/609285314 >> 172.21.15.55:0/609285314 conn(0x7f762c071590 msgr2=0x7f762c0719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:58.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.093+0000 7f76325a0640 1 -- 172.21.15.55:0/609285314 shutdown_connections 2024-03-19T02:05:58.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.093+0000 7f76325a0640 1 -- 172.21.15.55:0/609285314 wait complete. 2024-03-19T02:05:58.095 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.093+0000 7f76325a0640 1 Processor -- start 2024-03-19T02:05:58.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.093+0000 7f76325a0640 1 -- start start 2024-03-19T02:05:58.096 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.097+0000 7f76325a0640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f762c0720b0 0x7f762c14dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:58.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.097+0000 7f76325a0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f762c0c8de0 0x7f762c15a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:58.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.097+0000 7f76325a0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c152230 0x7f762c1f1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:58.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.097+0000 7f76325a0640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f762c070550 con 0x7f762c152230 2024-03-19T02:05:58.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.097+0000 7f76325a0640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f762c06fd50 con 0x7f762c0720b0 2024-03-19T02:05:58.097 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.097+0000 7f76325a0640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f762c0706d0 con 0x7f762c0c8de0 2024-03-19T02:05:58.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.097+0000 7f7630b3d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c152230 0x7f762c1f1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:58.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.097+0000 7f762b7fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f762c0c8de0 0x7f762c15a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:58.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.097+0000 7f7630b3d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c152230 0x7f762c1f1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56150/0 (socket says 172.21.15.55:56150) 2024-03-19T02:05:58.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.097+0000 7f7630b3d640 1 -- 172.21.15.55:0/3135412150 learned_addr learned my addr 172.21.15.55:0/3135412150 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:05:58.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.097+0000 7f762bfff640 1 --2- 172.21.15.55:0/3135412150 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f762c0720b0 0x7f762c14dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:58.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.101+0000 7f7630b3d640 1 -- 172.21.15.55:0/3135412150 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f762c0c8de0 msgr2=0x7f762c15a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:58.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.101+0000 7f7630b3d640 1 --2- 172.21.15.55:0/3135412150 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f762c0c8de0 0x7f762c15a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:58.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.101+0000 7f7630b3d640 1 -- 172.21.15.55:0/3135412150 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f762c0720b0 msgr2=0x7f762c14dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:58.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.101+0000 7f7630b3d640 1 --2- 172.21.15.55:0/3135412150 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f762c0720b0 0x7f762c14dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:58.101 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.101+0000 7f7630b3d640 1 -- 172.21.15.55:0/3135412150 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f762c1f1820 con 0x7f762c152230 2024-03-19T02:05:58.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.105+0000 7f7630b3d640 1 --2- 172.21.15.55:0/3135412150 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c152230 0x7f762c1f1fb0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f762400b9c0 tx=0x7f762400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:58.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.109+0000 7f7628ff9640 1 -- 172.21.15.55:0/3135412150 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7624019070 con 0x7f762c152230 2024-03-19T02:05:58.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.109+0000 7f7628ff9640 1 -- 172.21.15.55:0/3135412150 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f762400cba0 con 0x7f762c152230 2024-03-19T02:05:58.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.109+0000 7f7628ff9640 1 -- 172.21.15.55:0/3135412150 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7624014440 con 0x7f762c152230 2024-03-19T02:05:58.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.109+0000 7f76325a0640 1 -- 172.21.15.55:0/3135412150 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f762c0cbb60 con 0x7f762c152230 2024-03-19T02:05:58.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.109+0000 7f76325a0640 1 -- 172.21.15.55:0/3135412150 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f762c0cc120 con 0x7f762c152230 2024-03-19T02:05:58.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.109+0000 7f76325a0640 1 -- 172.21.15.55:0/3135412150 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f762c068a40 con 0x7f762c152230 2024-03-19T02:05:58.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.129+0000 7f7628ff9640 1 -- 172.21.15.55:0/3135412150 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f7624007370 con 0x7f762c152230 2024-03-19T02:05:58.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.129+0000 7f7628ff9640 1 --2- 172.21.15.55:0/3135412150 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f76180676e0 0x7f7618069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:05:58.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.129+0000 7f762bfff640 1 --2- 172.21.15.55:0/3135412150 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f76180676e0 0x7f7618069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:05:58.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.133+0000 7f762bfff640 1 --2- 172.21.15.55:0/3135412150 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f76180676e0 0x7f7618069ba0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f761c005890 tx=0x7f761c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:05:58.411 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.409+0000 7f7628ff9640 1 -- 172.21.15.55:0/3135412150 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f7624053e50 con 0x7f762c152230 2024-03-19T02:05:58.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.409+0000 7f7628ff9640 1 -- 172.21.15.55:0/3135412150 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f7624055db0 con 0x7f762c152230 2024-03-19T02:05:58.636 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.633+0000 7f76325a0640 1 -- 172.21.15.55:0/3135412150 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f762c157660 con 0x7f762c152230 2024-03-19T02:05:58.699 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.697+0000 7f7628ff9640 1 -- 172.21.15.55:0/3135412150 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f7624059e50 con 0x7f762c152230 2024-03-19T02:05:58.700 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:05:58.700 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:05:58.700 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:05:58.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.701+0000 7f76325a0640 1 -- 172.21.15.55:0/3135412150 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f76180676e0 msgr2=0x7f7618069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:58.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.701+0000 7f76325a0640 1 --2- 172.21.15.55:0/3135412150 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f76180676e0 0x7f7618069ba0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f761c005890 tx=0x7f761c009040 comp rx=0 tx=0).stop 2024-03-19T02:05:58.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.701+0000 7f76325a0640 1 -- 172.21.15.55:0/3135412150 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c152230 msgr2=0x7f762c1f1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:05:58.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.701+0000 7f76325a0640 1 --2- 172.21.15.55:0/3135412150 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c152230 0x7f762c1f1fb0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f762400b9c0 tx=0x7f762400be80 comp rx=0 tx=0).stop 2024-03-19T02:05:58.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.701+0000 7f76325a0640 1 -- 172.21.15.55:0/3135412150 shutdown_connections 2024-03-19T02:05:58.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.701+0000 7f76325a0640 1 --2- 172.21.15.55:0/3135412150 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f76180676e0 0x7f7618069ba0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:58.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.701+0000 7f76325a0640 1 --2- 172.21.15.55:0/3135412150 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f762c152230 0x7f762c1f1fb0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:58.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.701+0000 7f76325a0640 1 --2- 172.21.15.55:0/3135412150 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f762c0c8de0 0x7f762c15a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:58.702 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.701+0000 7f76325a0640 1 --2- 172.21.15.55:0/3135412150 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f762c0720b0 0x7f762c14dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:05:58.703 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.701+0000 7f76325a0640 1 -- 172.21.15.55:0/3135412150 >> 172.21.15.55:0/3135412150 conn(0x7f762c071590 msgr2=0x7f762c071970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:05:58.703 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.701+0000 7f76325a0640 1 -- 172.21.15.55:0/3135412150 shutdown_connections 2024-03-19T02:05:58.703 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:05:58.701+0000 7f76325a0640 1 -- 172.21.15.55:0/3135412150 wait complete. 2024-03-19T02:05:59.324 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:05:59.325 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:06:00.326 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:06:00.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.453+0000 7fc7af825640 1 Processor -- start 2024-03-19T02:06:00.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.453+0000 7fc7af825640 1 -- start start 2024-03-19T02:06:00.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.453+0000 7fc7af825640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc7a8154290 0x7fc7a815e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:00.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.453+0000 7fc7af825640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7a815ad40 0x7fc7a8154670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:00.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.453+0000 7fc7af825640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7a806bc90 con 0x7fc7a80ca310 2024-03-19T02:06:00.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.453+0000 7fc7af825640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fc7a815b2c0 con 0x7fc7a8154290 2024-03-19T02:06:00.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.453+0000 7fc7af825640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc7a806c1d0 con 0x7fc7a815ad40 2024-03-19T02:06:00.456 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.457+0000 7fc7ad5c1640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc7a8154290 0x7fc7a815e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:00.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.457+0000 7fc7addc2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7a815ad40 0x7fc7a8154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:00.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.457+0000 7fc7ad5c1640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc7a8154290 0x7fc7a815e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:44270/0 (socket says 172.21.15.55:44270) 2024-03-19T02:06:00.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.457+0000 7fc7ad5c1640 1 -- 172.21.15.55:0/2798857489 learned_addr learned my addr 172.21.15.55:0/2798857489 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:00.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.457+0000 7fc7ad5c1640 1 -- 172.21.15.55:0/2798857489 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7a815ad40 msgr2=0x7fc7a8154670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:00.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.457+0000 7fc7ad5c1640 1 --2- 172.21.15.55:0/2798857489 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7a815ad40 0x7fc7a8154670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:00.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.457+0000 7fc7ad5c1640 1 -- 172.21.15.55:0/2798857489 >> v1:172.21.15.55:6789/0 conn(0x7fc7a80ca310 legacy=0x7fc7a8070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:00.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.457+0000 7fc7ad5c1640 1 -- 172.21.15.55:0/2798857489 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7a806bf30 con 0x7fc7a8154290 2024-03-19T02:06:00.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.461+0000 7fc7ad5c1640 1 --2- 172.21.15.55:0/2798857489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7a8154290 0x7fc7a815e4c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4009870 tx=0x7fc7a401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47cf461cd2cc29fe server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:00.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.465+0000 7fc79ffff640 1 -- 172.21.15.55:0/2798857489 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a40ab5f0 con 0x7fc7a8154290 2024-03-19T02:06:00.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.465+0000 7fc79ffff640 1 -- 172.21.15.55:0/2798857489 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7a40b0070 con 0x7fc7a8154290 2024-03-19T02:06:00.468 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.465+0000 7fc79ffff640 1 -- 172.21.15.55:0/2798857489 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a40ab9b0 con 0x7fc7a8154290 2024-03-19T02:06:00.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.469+0000 7fc7af825640 1 -- 172.21.15.55:0/2798857489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7a8154290 msgr2=0x7fc7a815e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:00.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.469+0000 7fc7af825640 1 --2- 172.21.15.55:0/2798857489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7a8154290 0x7fc7a815e4c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4009870 tx=0x7fc7a401c520 comp rx=0 tx=0).stop 2024-03-19T02:06:00.469 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.469+0000 7fc7af825640 1 -- 172.21.15.55:0/2798857489 shutdown_connections 2024-03-19T02:06:00.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.469+0000 7fc7af825640 1 --2- 172.21.15.55:0/2798857489 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7a815ad40 0x7fc7a8154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:00.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.469+0000 7fc7af825640 1 --2- 172.21.15.55:0/2798857489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7a8154290 0x7fc7a815e4c0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:00.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.469+0000 7fc7af825640 1 -- 172.21.15.55:0/2798857489 >> 172.21.15.55:0/2798857489 conn(0x7fc7a806c580 msgr2=0x7fc7a806c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:00.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.469+0000 7fc7af825640 1 -- 172.21.15.55:0/2798857489 shutdown_connections 2024-03-19T02:06:00.470 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.469+0000 7fc7af825640 1 -- 172.21.15.55:0/2798857489 wait complete. 2024-03-19T02:06:00.471 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.469+0000 7fc7af825640 1 Processor -- start 2024-03-19T02:06:00.471 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.469+0000 7fc7af825640 1 -- start start 2024-03-19T02:06:00.472 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.469+0000 7fc7af825640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc7a80ca310 0x7fc7a80c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:00.472 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.469+0000 7fc7af825640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7a8154290 0x7fc7a80c5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:00.472 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.469+0000 7fc7af825640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7a815ad40 0x7fc7a80c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:00.472 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.469+0000 7fc7af825640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a80b2460 con 0x7fc7a815ad40 2024-03-19T02:06:00.472 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.469+0000 7fc7af825640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a80b22e0 con 0x7fc7a80ca310 2024-03-19T02:06:00.472 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.469+0000 7fc7af825640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc7a8070d50 con 0x7fc7a8154290 2024-03-19T02:06:00.472 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.469+0000 7fc7addc2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7a815ad40 0x7fc7a80c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:00.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.469+0000 7fc7ad5c1640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc7a80ca310 0x7fc7a80c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:00.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.469+0000 7fc7addc2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7a815ad40 0x7fc7a80c5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:44272/0 (socket says 172.21.15.55:44272) 2024-03-19T02:06:00.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.469+0000 7fc7addc2640 1 -- 172.21.15.55:0/3193331168 learned_addr learned my addr 172.21.15.55:0/3193331168 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:00.473 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.473+0000 7fc7acdc0640 1 --2- 172.21.15.55:0/3193331168 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7a8154290 0x7fc7a80c5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:00.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.473+0000 7fc7addc2640 1 -- 172.21.15.55:0/3193331168 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7a8154290 msgr2=0x7fc7a80c5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:00.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.473+0000 7fc7addc2640 1 --2- 172.21.15.55:0/3193331168 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7a8154290 0x7fc7a80c5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:00.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.473+0000 7fc7addc2640 1 -- 172.21.15.55:0/3193331168 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc7a80ca310 msgr2=0x7fc7a80c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:00.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.473+0000 7fc7addc2640 1 --2- 172.21.15.55:0/3193331168 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc7a80ca310 0x7fc7a80c7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:00.476 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.473+0000 7fc7addc2640 1 -- 172.21.15.55:0/3193331168 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7a8152b80 con 0x7fc7a815ad40 2024-03-19T02:06:00.478 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.477+0000 7fc7addc2640 1 --2- 172.21.15.55:0/3193331168 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7a815ad40 0x7fc7a80c5ad0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0009dd0 tx=0x7fc7a000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:00.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.481+0000 7fc79dffb640 1 -- 172.21.15.55:0/3193331168 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0018020 con 0x7fc7a815ad40 2024-03-19T02:06:00.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.481+0000 7fc79dffb640 1 -- 172.21.15.55:0/3193331168 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7a0014480 con 0x7fc7a815ad40 2024-03-19T02:06:00.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.481+0000 7fc79dffb640 1 -- 172.21.15.55:0/3193331168 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0014870 con 0x7fc7a815ad40 2024-03-19T02:06:00.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.481+0000 7fc7af825640 1 -- 172.21.15.55:0/3193331168 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7a8153b10 con 0x7fc7a815ad40 2024-03-19T02:06:00.483 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.481+0000 7fc7af825640 1 -- 172.21.15.55:0/3193331168 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7a80c6fd0 con 0x7fc7a815ad40 2024-03-19T02:06:00.484 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.481+0000 7fc7af825640 1 -- 172.21.15.55:0/3193331168 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7a8068a40 con 0x7fc7a815ad40 2024-03-19T02:06:00.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.509+0000 7fc79dffb640 1 -- 172.21.15.55:0/3193331168 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc7a001b020 con 0x7fc7a815ad40 2024-03-19T02:06:00.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.509+0000 7fc79dffb640 1 --2- 172.21.15.55:0/3193331168 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc78c067660 0x7fc78c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:00.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.509+0000 7fc7ad5c1640 1 --2- 172.21.15.55:0/3193331168 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc78c067660 0x7fc78c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:00.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.509+0000 7fc7ad5c1640 1 --2- 172.21.15.55:0/3193331168 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc78c067660 0x7fc78c069b20 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc7a40af000 tx=0x7fc7a40a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:00.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.513+0000 7fc79dffb640 1 -- 172.21.15.55:0/3193331168 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fc7a0055430 con 0x7fc7a815ad40 2024-03-19T02:06:00.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:00.805+0000 7fc79dffb640 1 -- 172.21.15.55:0/3193331168 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc7a0022030 con 0x7fc7a815ad40 2024-03-19T02:06:01.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:01.029+0000 7fc7af825640 1 -- 172.21.15.55:0/3193331168 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7a806b960 con 0x7fc7a815ad40 2024-03-19T02:06:01.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:01.061+0000 7fc79dffb640 1 -- 172.21.15.55:0/3193331168 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fc7a0010040 con 0x7fc7a815ad40 2024-03-19T02:06:01.063 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:06:01.063 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:06:01.063 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:06:01.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:01.065+0000 7fc7af825640 1 -- 172.21.15.55:0/3193331168 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc78c067660 msgr2=0x7fc78c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:01.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:01.065+0000 7fc7af825640 1 --2- 172.21.15.55:0/3193331168 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc78c067660 0x7fc78c069b20 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc7a40af000 tx=0x7fc7a40a7000 comp rx=0 tx=0).stop 2024-03-19T02:06:01.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:01.065+0000 7fc7af825640 1 -- 172.21.15.55:0/3193331168 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7a815ad40 msgr2=0x7fc7a80c5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:01.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:01.065+0000 7fc7af825640 1 --2- 172.21.15.55:0/3193331168 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7a815ad40 0x7fc7a80c5ad0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0009dd0 tx=0x7fc7a000c490 comp rx=0 tx=0).stop 2024-03-19T02:06:01.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:01.065+0000 7fc7af825640 1 -- 172.21.15.55:0/3193331168 shutdown_connections 2024-03-19T02:06:01.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:01.065+0000 7fc7af825640 1 --2- 172.21.15.55:0/3193331168 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc78c067660 0x7fc78c069b20 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:01.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:01.065+0000 7fc7af825640 1 --2- 172.21.15.55:0/3193331168 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc7a815ad40 0x7fc7a80c5ad0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:01.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:01.065+0000 7fc7af825640 1 --2- 172.21.15.55:0/3193331168 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc7a8154290 0x7fc7a80c5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:01.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:01.065+0000 7fc7af825640 1 --2- 172.21.15.55:0/3193331168 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc7a80ca310 0x7fc7a80c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:01.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:01.065+0000 7fc7af825640 1 -- 172.21.15.55:0/3193331168 >> 172.21.15.55:0/3193331168 conn(0x7fc7a806c580 msgr2=0x7fc7a814d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:01.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:01.065+0000 7fc7af825640 1 -- 172.21.15.55:0/3193331168 shutdown_connections 2024-03-19T02:06:01.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:01.065+0000 7fc7af825640 1 -- 172.21.15.55:0/3193331168 wait complete. 2024-03-19T02:06:01.686 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:06:01.687 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:06:02.687 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:06:02.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.805+0000 7fda9f615640 1 Processor -- start 2024-03-19T02:06:02.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.805+0000 7fda9f615640 1 -- start start 2024-03-19T02:06:02.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.805+0000 7fda9f615640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fda980ca350 0x7fda9815b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:02.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.805+0000 7fda9f615640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fda980c3320 0x7fda980728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:02.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.805+0000 7fda9f615640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda98151330 con 0x7fda98150f50 2024-03-19T02:06:02.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.805+0000 7fda9f615640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fda980c3e00 con 0x7fda980c3320 2024-03-19T02:06:02.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.805+0000 7fda9f615640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fda9815ac10 con 0x7fda980ca350 2024-03-19T02:06:02.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.805+0000 7fda9dbb2640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fda980c3320 0x7fda980728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:02.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.805+0000 7fda9cbb0640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fda980ca350 0x7fda9815b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:02.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.805+0000 7fda9d3b1640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fda98150f50 0x7fda98157920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:40450/0 (socket says 172.21.15.55:40450) 2024-03-19T02:06:02.808 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.805+0000 7fda9d3b1640 1 -- 172.21.15.55:0/1338743023 learned_addr learned my addr 172.21.15.55:0/1338743023 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:06:02.810 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.809+0000 7fda9dbb2640 1 -- 172.21.15.55:0/1338743023 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fda980ca350 msgr2=0x7fda9815b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:02.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.809+0000 7fda9dbb2640 1 --2- 172.21.15.55:0/1338743023 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fda980ca350 0x7fda9815b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:02.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.809+0000 7fda9dbb2640 1 -- 172.21.15.55:0/1338743023 >> v1:172.21.15.84:6789/0 conn(0x7fda98150f50 legacy=0x7fda98157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:02.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.809+0000 7fda9dbb2640 1 -- 172.21.15.55:0/1338743023 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda980c3840 con 0x7fda980c3320 2024-03-19T02:06:02.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.813+0000 7fda9dbb2640 1 --2- 172.21.15.55:0/1338743023 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda980c3320 0x7fda980728b0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fda90009d90 tx=0x7fda9001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1885d27b1164235c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:02.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.817+0000 7fda8ffff640 1 -- 172.21.15.55:0/1338743023 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda900b2070 con 0x7fda980c3320 2024-03-19T02:06:02.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.817+0000 7fda8ffff640 1 -- 172.21.15.55:0/1338743023 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda900a5a30 con 0x7fda980c3320 2024-03-19T02:06:02.820 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.817+0000 7fda8ffff640 1 -- 172.21.15.55:0/1338743023 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda900a5e00 con 0x7fda980c3320 2024-03-19T02:06:02.820 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.817+0000 7fda9f615640 1 -- 172.21.15.55:0/1338743023 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda980c3320 msgr2=0x7fda980728b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:02.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.817+0000 7fda9f615640 1 --2- 172.21.15.55:0/1338743023 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda980c3320 0x7fda980728b0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fda90009d90 tx=0x7fda9001c970 comp rx=0 tx=0).stop 2024-03-19T02:06:02.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.817+0000 7fda9f615640 1 -- 172.21.15.55:0/1338743023 shutdown_connections 2024-03-19T02:06:02.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.817+0000 7fda9f615640 1 --2- 172.21.15.55:0/1338743023 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda980c3320 0x7fda980728b0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:02.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.817+0000 7fda9f615640 1 --2- 172.21.15.55:0/1338743023 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fda980ca350 0x7fda9815b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:02.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.817+0000 7fda9f615640 1 -- 172.21.15.55:0/1338743023 >> 172.21.15.55:0/1338743023 conn(0x7fda9806e670 msgr2=0x7fda9806ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:02.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.817+0000 7fda9f615640 1 -- 172.21.15.55:0/1338743023 shutdown_connections 2024-03-19T02:06:02.821 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.821+0000 7fda9f615640 1 -- 172.21.15.55:0/1338743023 wait complete. 2024-03-19T02:06:02.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.821+0000 7fda9f615640 1 Processor -- start 2024-03-19T02:06:02.822 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.821+0000 7fda9f615640 1 -- start start 2024-03-19T02:06:02.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.821+0000 7fda9f615640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda980c3320 0x7fda980cbde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:02.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.821+0000 7fda9f615640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fda980ca350 0x7fda9806ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:02.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.821+0000 7fda9f615640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fda98150f50 0x7fda98141520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:02.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.821+0000 7fda9f615640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fda981639b0 con 0x7fda980c3320 2024-03-19T02:06:02.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.821+0000 7fda9f615640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fda98163830 con 0x7fda98150f50 2024-03-19T02:06:02.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.821+0000 7fda9f615640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fda98163b30 con 0x7fda980ca350 2024-03-19T02:06:02.824 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.821+0000 7fda9d3b1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda980c3320 0x7fda980cbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:02.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.825+0000 7fda9d3b1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda980c3320 0x7fda980cbde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:44280/0 (socket says 172.21.15.55:44280) 2024-03-19T02:06:02.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.825+0000 7fda9d3b1640 1 -- 172.21.15.55:0/3643497743 learned_addr learned my addr 172.21.15.55:0/3643497743 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:02.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.825+0000 7fda9dbb2640 1 --2- 172.21.15.55:0/3643497743 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fda98150f50 0x7fda98141520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:02.825 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.825+0000 7fda9cbb0640 1 --2- 172.21.15.55:0/3643497743 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fda980ca350 0x7fda9806ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:02.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.825+0000 7fda9d3b1640 1 -- 172.21.15.55:0/3643497743 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fda980ca350 msgr2=0x7fda9806ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:02.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.825+0000 7fda9d3b1640 1 --2- 172.21.15.55:0/3643497743 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fda980ca350 0x7fda9806ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:02.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.825+0000 7fda9d3b1640 1 -- 172.21.15.55:0/3643497743 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fda98150f50 msgr2=0x7fda98141520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:02.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.825+0000 7fda9d3b1640 1 --2- 172.21.15.55:0/3643497743 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fda98150f50 0x7fda98141520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:02.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.825+0000 7fda9d3b1640 1 -- 172.21.15.55:0/3643497743 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda9814e0c0 con 0x7fda980c3320 2024-03-19T02:06:02.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.829+0000 7fda9d3b1640 1 --2- 172.21.15.55:0/3643497743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda980c3320 0x7fda980cbde0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fda9400c3a0 tx=0x7fda9400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:02.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.833+0000 7fda8dffb640 1 -- 172.21.15.55:0/3643497743 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda94017070 con 0x7fda980c3320 2024-03-19T02:06:02.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.833+0000 7fda8dffb640 1 -- 172.21.15.55:0/3643497743 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda94005cf0 con 0x7fda980c3320 2024-03-19T02:06:02.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.833+0000 7fda9f615640 1 -- 172.21.15.55:0/3643497743 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda98146f90 con 0x7fda980c3320 2024-03-19T02:06:02.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.833+0000 7fda8dffb640 1 -- 172.21.15.55:0/3643497743 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda940072f0 con 0x7fda980c3320 2024-03-19T02:06:02.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.833+0000 7fda9f615640 1 -- 172.21.15.55:0/3643497743 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda9806b4f0 con 0x7fda980c3320 2024-03-19T02:06:02.836 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:02.837+0000 7fda9f615640 1 -- 172.21.15.55:0/3643497743 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda98068a40 con 0x7fda980c3320 2024-03-19T02:06:03.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:03.125+0000 7fda8dffb640 1 -- 172.21.15.55:0/3643497743 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fda9401a070 con 0x7fda980c3320 2024-03-19T02:06:03.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:03.129+0000 7fda8dffb640 1 --2- 172.21.15.55:0/3643497743 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fda7c0676b0 0x7fda7c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:03.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:03.129+0000 7fda9cbb0640 1 --2- 172.21.15.55:0/3643497743 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fda7c0676b0 0x7fda7c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:03.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:03.129+0000 7fda9cbb0640 1 --2- 172.21.15.55:0/3643497743 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fda7c0676b0 0x7fda7c069b70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fda9806c310 tx=0x7fda88009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:03.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:03.165+0000 7fda8dffb640 1 -- 172.21.15.55:0/3643497743 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fda940536d0 con 0x7fda980c3320 2024-03-19T02:06:03.167 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:03.165+0000 7fda8dffb640 1 -- 172.21.15.55:0/3643497743 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fda94021020 con 0x7fda980c3320 2024-03-19T02:06:03.391 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:03.389+0000 7fda9f615640 1 -- 172.21.15.55:0/3643497743 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda9806b6c0 con 0x7fda980c3320 2024-03-19T02:06:03.485 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:03.485+0000 7fda8dffb640 1 -- 172.21.15.55:0/3643497743 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fda94055630 con 0x7fda980c3320 2024-03-19T02:06:03.486 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:06:03.486 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:06:03.486 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:06:03.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:03.489+0000 7fda9f615640 1 -- 172.21.15.55:0/3643497743 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fda7c0676b0 msgr2=0x7fda7c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:03.489 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:03.489+0000 7fda9f615640 1 --2- 172.21.15.55:0/3643497743 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fda7c0676b0 0x7fda7c069b70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fda9806c310 tx=0x7fda88009190 comp rx=0 tx=0).stop 2024-03-19T02:06:03.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:03.489+0000 7fda9f615640 1 -- 172.21.15.55:0/3643497743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda980c3320 msgr2=0x7fda980cbde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:03.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:03.489+0000 7fda9f615640 1 --2- 172.21.15.55:0/3643497743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda980c3320 0x7fda980cbde0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fda9400c3a0 tx=0x7fda9400c860 comp rx=0 tx=0).stop 2024-03-19T02:06:03.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:03.489+0000 7fda9f615640 1 -- 172.21.15.55:0/3643497743 shutdown_connections 2024-03-19T02:06:03.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:03.489+0000 7fda9f615640 1 --2- 172.21.15.55:0/3643497743 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fda7c0676b0 0x7fda7c069b70 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:03.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:03.489+0000 7fda9f615640 1 --2- 172.21.15.55:0/3643497743 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fda98150f50 0x7fda98141520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:03.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:03.489+0000 7fda9f615640 1 --2- 172.21.15.55:0/3643497743 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fda980ca350 0x7fda9806ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:03.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:03.489+0000 7fda9f615640 1 --2- 172.21.15.55:0/3643497743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda980c3320 0x7fda980cbde0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:03.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:03.489+0000 7fda9f615640 1 -- 172.21.15.55:0/3643497743 >> 172.21.15.55:0/3643497743 conn(0x7fda9806e670 msgr2=0x7fda98155e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:03.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:03.489+0000 7fda9f615640 1 -- 172.21.15.55:0/3643497743 shutdown_connections 2024-03-19T02:06:03.490 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:03.489+0000 7fda9f615640 1 -- 172.21.15.55:0/3643497743 wait complete. 2024-03-19T02:06:04.113 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:06:04.113 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:06:05.114 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:06:05.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.233+0000 7f3c69e29640 1 Processor -- start 2024-03-19T02:06:05.234 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.233+0000 7f3c69e29640 1 -- start start 2024-03-19T02:06:05.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.233+0000 7f3c69e29640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3c64070f20 0x7f3c6415d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:05.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.233+0000 7f3c69e29640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c6415e3a0 con 0x7f3c640c9a90 2024-03-19T02:06:05.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.233+0000 7f3c69e29640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c6415e680 con 0x7f3c640c96b0 2024-03-19T02:06:05.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.233+0000 7f3c69e29640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3c640c7ad0 con 0x7f3c64070f20 2024-03-19T02:06:05.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.233+0000 7f3c62ffd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f3c640c9a90 0x7f3c641540c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:60856/0 (socket says 172.21.15.55:60856) 2024-03-19T02:06:05.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.233+0000 7f3c62ffd640 1 -- 172.21.15.55:0/1398586161 learned_addr learned my addr 172.21.15.55:0/1398586161 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:06:05.236 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.237+0000 7f3c63fff640 1 --2- 172.21.15.55:0/1398586161 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3c64070f20 0x7f3c6415d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:05.238 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.237+0000 7f3c627fc640 1 -- 172.21.15.55:0/1398586161 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1843428642 0 0) 0x7f3c6415e3a0 con 0x7f3c640c9a90 2024-03-19T02:06:05.239 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.237+0000 7f3c627fc640 1 -- 172.21.15.55:0/1398586161 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c4c003530 con 0x7f3c640c9a90 2024-03-19T02:06:05.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.237+0000 7f3c63fff640 1 -- 172.21.15.55:0/1398586161 >> v1:172.21.15.84:6789/0 conn(0x7f3c640c96b0 legacy=0x7f3c640c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:05.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.237+0000 7f3c63fff640 1 -- 172.21.15.55:0/1398586161 >> v1:172.21.15.55:6789/0 conn(0x7f3c640c9a90 legacy=0x7f3c641540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:05.240 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.237+0000 7f3c63fff640 1 -- 172.21.15.55:0/1398586161 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c6415a330 con 0x7f3c64070f20 2024-03-19T02:06:05.244 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.245+0000 7f3c63fff640 1 --2- 172.21.15.55:0/1398586161 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3c64070f20 0x7f3c6415d810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c009dc0 tx=0x7f3c5c01c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5fa6cfd002f77f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:05.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.253+0000 7f3c627fc640 1 -- 172.21.15.55:0/1398586161 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c0b2070 con 0x7f3c64070f20 2024-03-19T02:06:05.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.253+0000 7f3c627fc640 1 -- 172.21.15.55:0/1398586161 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c5c0a57c0 con 0x7f3c64070f20 2024-03-19T02:06:05.253 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.253+0000 7f3c627fc640 1 -- 172.21.15.55:0/1398586161 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c0a5b90 con 0x7f3c64070f20 2024-03-19T02:06:05.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.253+0000 7f3c69e29640 1 -- 172.21.15.55:0/1398586161 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3c64070f20 msgr2=0x7f3c6415d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:05.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.253+0000 7f3c69e29640 1 --2- 172.21.15.55:0/1398586161 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3c64070f20 0x7f3c6415d810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c009dc0 tx=0x7f3c5c01c900 comp rx=0 tx=0).stop 2024-03-19T02:06:05.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.253+0000 7f3c69e29640 1 -- 172.21.15.55:0/1398586161 shutdown_connections 2024-03-19T02:06:05.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.253+0000 7f3c69e29640 1 --2- 172.21.15.55:0/1398586161 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3c64070f20 0x7f3c6415d810 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:05.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.253+0000 7f3c69e29640 1 -- 172.21.15.55:0/1398586161 >> 172.21.15.55:0/1398586161 conn(0x7f3c640cc9c0 msgr2=0x7f3c640ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:05.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.253+0000 7f3c69e29640 1 -- 172.21.15.55:0/1398586161 shutdown_connections 2024-03-19T02:06:05.254 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.253+0000 7f3c69e29640 1 -- 172.21.15.55:0/1398586161 wait complete. 2024-03-19T02:06:05.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.253+0000 7f3c69e29640 1 Processor -- start 2024-03-19T02:06:05.255 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.253+0000 7f3c69e29640 1 -- start start 2024-03-19T02:06:05.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.253+0000 7f3c69e29640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3c64070f20 0x7f3c64159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:05.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.253+0000 7f3c69e29640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3c640c96b0 0x7f3c64148220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:05.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.253+0000 7f3c69e29640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c640c9a90 0x7f3c641f75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:05.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.253+0000 7f3c69e29640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3c641547d0 con 0x7f3c640c9a90 2024-03-19T02:06:05.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.253+0000 7f3c69e29640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6415b0f0 con 0x7f3c640c96b0 2024-03-19T02:06:05.256 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.253+0000 7f3c69e29640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3c64154950 con 0x7f3c64070f20 2024-03-19T02:06:05.257 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.257+0000 7f3c637fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3c64070f20 0x7f3c64159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:05.257 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.257+0000 7f3c62ffd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3c640c96b0 0x7f3c64148220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:05.257 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.257+0000 7f3c63fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c640c9a90 0x7f3c641f75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:05.257 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.257+0000 7f3c63fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c640c9a90 0x7f3c641f75a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:44294/0 (socket says 172.21.15.55:44294) 2024-03-19T02:06:05.257 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.257+0000 7f3c63fff640 1 -- 172.21.15.55:0/2605599580 learned_addr learned my addr 172.21.15.55:0/2605599580 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:05.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.261+0000 7f3c63fff640 1 -- 172.21.15.55:0/2605599580 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3c64070f20 msgr2=0x7f3c64159d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:05.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.261+0000 7f3c63fff640 1 --2- 172.21.15.55:0/2605599580 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3c64070f20 0x7f3c64159d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:05.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.261+0000 7f3c63fff640 1 -- 172.21.15.55:0/2605599580 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3c640c96b0 msgr2=0x7f3c64148220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:05.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.261+0000 7f3c63fff640 1 --2- 172.21.15.55:0/2605599580 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3c640c96b0 0x7f3c64148220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:05.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.261+0000 7f3c63fff640 1 -- 172.21.15.55:0/2605599580 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c64147d30 con 0x7f3c640c9a90 2024-03-19T02:06:05.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.261+0000 7f3c63fff640 1 --2- 172.21.15.55:0/2605599580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c640c9a90 0x7f3c641f75a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c01cde0 tx=0x7f3c5c0196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:05.268 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.265+0000 7f3c4bfff640 1 -- 172.21.15.55:0/2605599580 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c0b2050 con 0x7f3c640c9a90 2024-03-19T02:06:05.268 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.265+0000 7f3c4bfff640 1 -- 172.21.15.55:0/2605599580 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c5c0be070 con 0x7f3c640c9a90 2024-03-19T02:06:05.268 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.265+0000 7f3c69e29640 1 -- 172.21.15.55:0/2605599580 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c64145370 con 0x7f3c640c9a90 2024-03-19T02:06:05.268 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.265+0000 7f3c4bfff640 1 -- 172.21.15.55:0/2605599580 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c009450 con 0x7f3c640c9a90 2024-03-19T02:06:05.268 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.269+0000 7f3c69e29640 1 -- 172.21.15.55:0/2605599580 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c6415cd50 con 0x7f3c640c9a90 2024-03-19T02:06:05.269 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.269+0000 7f3c69e29640 1 -- 172.21.15.55:0/2605599580 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c64068a40 con 0x7f3c640c9a90 2024-03-19T02:06:05.666 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.665+0000 7f3c4bfff640 1 -- 172.21.15.55:0/2605599580 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f3c5c0ad870 con 0x7f3c640c9a90 2024-03-19T02:06:05.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.665+0000 7f3c4bfff640 1 --2- 172.21.15.55:0/2605599580 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3c500676b0 0x7f3c50069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:05.667 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.665+0000 7f3c637fe640 1 --2- 172.21.15.55:0/2605599580 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3c500676b0 0x7f3c50069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:05.668 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.665+0000 7f3c637fe640 1 --2- 172.21.15.55:0/2605599580 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3c500676b0 0x7f3c50069b70 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3c54001770 tx=0x7f3c54009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:05.690 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.689+0000 7f3c4bfff640 1 -- 172.21.15.55:0/2605599580 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f3c5c0ef8a0 con 0x7f3c640c9a90 2024-03-19T02:06:05.691 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.689+0000 7f3c4bfff640 1 -- 172.21.15.55:0/2605599580 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f3c5c016580 con 0x7f3c640c9a90 2024-03-19T02:06:05.904 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:05.901+0000 7f3c69e29640 1 -- 172.21.15.55:0/2605599580 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c640c8c80 con 0x7f3c640c9a90 2024-03-19T02:06:06.026 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:06.025+0000 7f3c4bfff640 1 -- 172.21.15.55:0/2605599580 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7f3c5c0f1800 con 0x7f3c640c9a90 2024-03-19T02:06:06.026 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:06:06.026 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:06:06.027 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:06:06.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:06.029+0000 7f3c69e29640 1 -- 172.21.15.55:0/2605599580 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3c500676b0 msgr2=0x7f3c50069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:06.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:06.029+0000 7f3c69e29640 1 --2- 172.21.15.55:0/2605599580 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3c500676b0 0x7f3c50069b70 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3c54001770 tx=0x7f3c54009040 comp rx=0 tx=0).stop 2024-03-19T02:06:06.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:06.029+0000 7f3c69e29640 1 -- 172.21.15.55:0/2605599580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c640c9a90 msgr2=0x7f3c641f75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:06.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:06.029+0000 7f3c69e29640 1 --2- 172.21.15.55:0/2605599580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c640c9a90 0x7f3c641f75a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c01cde0 tx=0x7f3c5c0196a0 comp rx=0 tx=0).stop 2024-03-19T02:06:06.031 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:06.029+0000 7f3c69e29640 1 -- 172.21.15.55:0/2605599580 shutdown_connections 2024-03-19T02:06:06.031 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:06.029+0000 7f3c69e29640 1 --2- 172.21.15.55:0/2605599580 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3c500676b0 0x7f3c50069b70 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:06.031 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:06.029+0000 7f3c69e29640 1 --2- 172.21.15.55:0/2605599580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c640c9a90 0x7f3c641f75a0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:06.031 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:06.029+0000 7f3c69e29640 1 --2- 172.21.15.55:0/2605599580 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3c640c96b0 0x7f3c64148220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:06.031 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:06.029+0000 7f3c69e29640 1 --2- 172.21.15.55:0/2605599580 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3c64070f20 0x7f3c64159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:06.031 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:06.029+0000 7f3c69e29640 1 -- 172.21.15.55:0/2605599580 >> 172.21.15.55:0/2605599580 conn(0x7f3c640cc9c0 msgr2=0x7f3c640c5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:06.031 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:06.029+0000 7f3c69e29640 1 -- 172.21.15.55:0/2605599580 shutdown_connections 2024-03-19T02:06:06.031 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:06.029+0000 7f3c69e29640 1 -- 172.21.15.55:0/2605599580 wait complete. 2024-03-19T02:06:06.653 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:06:06.653 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:06:07.654 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:06:07.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.785+0000 7fa4d8c40640 1 Processor -- start 2024-03-19T02:06:07.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.785+0000 7fa4d8c40640 1 -- start start 2024-03-19T02:06:07.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.785+0000 7fa4d8c40640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa4d40c60d0 0x7fa4d40c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:07.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.785+0000 7fa4d8c40640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa4d41585f0 0x7fa4d4158a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:07.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.785+0000 7fa4d8c40640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4d414ee80 con 0x7fa4d414eaa0 2024-03-19T02:06:07.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.785+0000 7fa4d8c40640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fa4d4158f50 con 0x7fa4d40c60d0 2024-03-19T02:06:07.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.785+0000 7fa4d8c40640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa4d40c7540 con 0x7fa4d41585f0 2024-03-19T02:06:07.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.785+0000 7fa4d1d9b640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fa4d40c60d0 0x7fa4d40c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:07.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.785+0000 7fa4d259c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fa4d414eaa0 0x7fa4d406b980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:40462/0 (socket says 172.21.15.55:40462) 2024-03-19T02:06:07.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.785+0000 7fa4d259c640 1 -- 172.21.15.55:0/4007073998 learned_addr learned my addr 172.21.15.55:0/4007073998 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:06:07.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.789+0000 7fa4d2d9d640 1 --2- 172.21.15.55:0/4007073998 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa4d41585f0 0x7fa4d4158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:07.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.789+0000 7fa4d1d9b640 1 -- 172.21.15.55:0/4007073998 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa4d41585f0 msgr2=0x7fa4d4158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:07.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.789+0000 7fa4d1d9b640 1 --2- 172.21.15.55:0/4007073998 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa4d41585f0 0x7fa4d4158a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:07.791 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.789+0000 7fa4d1d9b640 1 -- 172.21.15.55:0/4007073998 >> v1:172.21.15.84:6789/0 conn(0x7fa4d414eaa0 legacy=0x7fa4d406b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:07.792 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.789+0000 7fa4d1d9b640 1 -- 172.21.15.55:0/4007073998 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d40c77c0 con 0x7fa4d40c60d0 2024-03-19T02:06:07.795 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.793+0000 7fa4d1d9b640 1 --2- 172.21.15.55:0/4007073998 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4d40c60d0 0x7fa4d40c6540 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fa4c80097c0 tx=0x7fa4c801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f565d16c899cdcb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:07.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.797+0000 7fa4d159a640 1 -- 172.21.15.55:0/4007073998 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c80b0070 con 0x7fa4d40c60d0 2024-03-19T02:06:07.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.797+0000 7fa4d159a640 1 -- 172.21.15.55:0/4007073998 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4c80ab600 con 0x7fa4d40c60d0 2024-03-19T02:06:07.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.797+0000 7fa4d159a640 1 -- 172.21.15.55:0/4007073998 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c80ab9d0 con 0x7fa4d40c60d0 2024-03-19T02:06:07.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.801+0000 7fa4d8c40640 1 -- 172.21.15.55:0/4007073998 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4d40c60d0 msgr2=0x7fa4d40c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:07.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.801+0000 7fa4d8c40640 1 --2- 172.21.15.55:0/4007073998 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4d40c60d0 0x7fa4d40c6540 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fa4c80097c0 tx=0x7fa4c801c410 comp rx=0 tx=0).stop 2024-03-19T02:06:07.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.801+0000 7fa4d8c40640 1 -- 172.21.15.55:0/4007073998 shutdown_connections 2024-03-19T02:06:07.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.801+0000 7fa4d8c40640 1 --2- 172.21.15.55:0/4007073998 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa4d41585f0 0x7fa4d4158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:07.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.801+0000 7fa4d8c40640 1 --2- 172.21.15.55:0/4007073998 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4d40c60d0 0x7fa4d40c6540 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:07.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.801+0000 7fa4d8c40640 1 -- 172.21.15.55:0/4007073998 >> 172.21.15.55:0/4007073998 conn(0x7fa4d40cc9c0 msgr2=0x7fa4d40ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:07.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.801+0000 7fa4d8c40640 1 -- 172.21.15.55:0/4007073998 shutdown_connections 2024-03-19T02:06:07.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.801+0000 7fa4d8c40640 1 -- 172.21.15.55:0/4007073998 wait complete. 2024-03-19T02:06:07.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.801+0000 7fa4d8c40640 1 Processor -- start 2024-03-19T02:06:07.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.801+0000 7fa4d8c40640 1 -- start start 2024-03-19T02:06:07.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.805+0000 7fa4d8c40640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4d40c60d0 0x7fa4d413c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:07.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.805+0000 7fa4d8c40640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4d414eaa0 0x7fa4d413c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:07.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.805+0000 7fa4d8c40640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa4d41585f0 0x7fa4d40cb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:07.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.805+0000 7fa4d8c40640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d406c090 con 0x7fa4d414eaa0 2024-03-19T02:06:07.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.805+0000 7fa4d8c40640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d40c8060 con 0x7fa4d40c60d0 2024-03-19T02:06:07.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.805+0000 7fa4d8c40640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fa4d406c210 con 0x7fa4d41585f0 2024-03-19T02:06:07.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.805+0000 7fa4d1d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4d414eaa0 0x7fa4d413c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:07.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.805+0000 7fa4d1d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4d414eaa0 0x7fa4d413c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:44318/0 (socket says 172.21.15.55:44318) 2024-03-19T02:06:07.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.805+0000 7fa4d1d9b640 1 -- 172.21.15.55:0/2161380856 learned_addr learned my addr 172.21.15.55:0/2161380856 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:07.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.805+0000 7fa4d2d9d640 1 --2- 172.21.15.55:0/2161380856 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa4d41585f0 0x7fa4d40cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:07.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.805+0000 7fa4d259c640 1 --2- 172.21.15.55:0/2161380856 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4d40c60d0 0x7fa4d413c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:07.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.809+0000 7fa4d1d9b640 1 -- 172.21.15.55:0/2161380856 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa4d41585f0 msgr2=0x7fa4d40cb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:07.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.809+0000 7fa4d1d9b640 1 --2- 172.21.15.55:0/2161380856 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa4d41585f0 0x7fa4d40cb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:07.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.809+0000 7fa4d1d9b640 1 -- 172.21.15.55:0/2161380856 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4d40c60d0 msgr2=0x7fa4d413c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:07.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.809+0000 7fa4d1d9b640 1 --2- 172.21.15.55:0/2161380856 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4d40c60d0 0x7fa4d413c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:07.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.809+0000 7fa4d1d9b640 1 -- 172.21.15.55:0/2161380856 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d414cb00 con 0x7fa4d414eaa0 2024-03-19T02:06:07.811 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.809+0000 7fa4d1d9b640 1 --2- 172.21.15.55:0/2161380856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4d414eaa0 0x7fa4d413c730 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fa4c801a040 tx=0x7fa4c80191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:07.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.813+0000 7fa4c2ffd640 1 -- 172.21.15.55:0/2161380856 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c80ba070 con 0x7fa4d414eaa0 2024-03-19T02:06:07.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.813+0000 7fa4c2ffd640 1 -- 172.21.15.55:0/2161380856 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4c80b43f0 con 0x7fa4d414eaa0 2024-03-19T02:06:07.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.813+0000 7fa4d8c40640 1 -- 172.21.15.55:0/2161380856 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4d414cdc0 con 0x7fa4d414eaa0 2024-03-19T02:06:07.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.817+0000 7fa4c2ffd640 1 -- 172.21.15.55:0/2161380856 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c80b0040 con 0x7fa4d414eaa0 2024-03-19T02:06:07.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.817+0000 7fa4d8c40640 1 -- 172.21.15.55:0/2161380856 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4d414d380 con 0x7fa4d414eaa0 2024-03-19T02:06:07.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.817+0000 7fa4d8c40640 1 -- 172.21.15.55:0/2161380856 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4d4068a40 con 0x7fa4d414eaa0 2024-03-19T02:06:07.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.841+0000 7fa4c2ffd640 1 -- 172.21.15.55:0/2161380856 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fa4c80bd070 con 0x7fa4d414eaa0 2024-03-19T02:06:07.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.841+0000 7fa4c2ffd640 1 --2- 172.21.15.55:0/2161380856 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa4b4067690 0x7fa4b4069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:07.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.841+0000 7fa4d259c640 1 --2- 172.21.15.55:0/2161380856 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa4b4067690 0x7fa4b4069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:07.843 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:07.841+0000 7fa4d259c640 1 --2- 172.21.15.55:0/2161380856 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa4b4067690 0x7fa4b4069b50 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4000f70 tx=0x7fa4c4007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:08.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:08.121+0000 7fa4c2ffd640 1 -- 172.21.15.55:0/2161380856 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fa4c80ece10 con 0x7fa4d414eaa0 2024-03-19T02:06:08.123 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:08.121+0000 7fa4c2ffd640 1 -- 172.21.15.55:0/2161380856 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fa4c8016090 con 0x7fa4d414eaa0 2024-03-19T02:06:08.341 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:08.341+0000 7fa4d8c40640 1 -- 172.21.15.55:0/2161380856 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4d4074970 con 0x7fa4d414eaa0 2024-03-19T02:06:08.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:08.361+0000 7fa4c2ffd640 1 -- 172.21.15.55:0/2161380856 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6278 (secure 0 0 0) 0x7fa4c80eed70 con 0x7fa4d414eaa0 2024-03-19T02:06:08.364 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:06:08.364 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T01:51:09.453331+0000","modified":"2024-03-19T01:55:32.490986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4255,"mds_1":4200,"mds_2":4237,"mds_3":4223,"mds_4":4209},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4200":{"gid":4200,"name":"e","rank":1,"incarnation":21,"state":"up:resolve","state_seq":36,"addr":"172.21.15.55:6833/3674676035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":3674676035},{"type":"v1","addr":"172.21.15.55:6833","nonce":3674676035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4209":{"gid":4209,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.55:6835/3445165511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":3445165511},{"type":"v1","addr":"172.21.15.55:6835","nonce":3445165511}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:06.221231+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4223":{"gid":4223,"name":"b","rank":3,"incarnation":17,"state":"up:active","state_seq":24,"addr":"172.21.15.84:6837/1680195484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1680195484},{"type":"v1","addr":"172.21.15.84:6837","nonce":1680195484}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:53:01.199479+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4237":{"gid":4237,"name":"d","rank":2,"incarnation":15,"state":"up:active","state_seq":23,"addr":"172.21.15.84:6835/1290835674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":1290835674},{"type":"v1","addr":"172.21.15.84:6835","nonce":1290835674}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:52:58.374300+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4255":{"gid":4255,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":9,"addr":"172.21.15.55:6837/2236779395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":2236779395},{"type":"v1","addr":"172.21.15.55:6837","nonce":2236779395}]},"join_fscid":-1,"laggy_since":"2024-03-19T01:55:32.490982+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4255,"qdb_cluster":[4255,4237,4223,4209]},"id":1}]} 2024-03-19T02:06:08.364 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 26 2024-03-19T02:06:08.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:08.365+0000 7fa4d8c40640 1 -- 172.21.15.55:0/2161380856 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa4b4067690 msgr2=0x7fa4b4069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:08.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:08.365+0000 7fa4d8c40640 1 --2- 172.21.15.55:0/2161380856 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa4b4067690 0x7fa4b4069b50 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4000f70 tx=0x7fa4c4007e90 comp rx=0 tx=0).stop 2024-03-19T02:06:08.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:08.365+0000 7fa4d8c40640 1 -- 172.21.15.55:0/2161380856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4d414eaa0 msgr2=0x7fa4d413c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:08.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:08.365+0000 7fa4d8c40640 1 --2- 172.21.15.55:0/2161380856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4d414eaa0 0x7fa4d413c730 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fa4c801a040 tx=0x7fa4c80191b0 comp rx=0 tx=0).stop 2024-03-19T02:06:08.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:08.365+0000 7fa4d8c40640 1 -- 172.21.15.55:0/2161380856 shutdown_connections 2024-03-19T02:06:08.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:08.365+0000 7fa4d8c40640 1 --2- 172.21.15.55:0/2161380856 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fa4b4067690 0x7fa4b4069b50 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:08.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:08.365+0000 7fa4d8c40640 1 --2- 172.21.15.55:0/2161380856 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fa4d41585f0 0x7fa4d40cb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:08.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:08.365+0000 7fa4d8c40640 1 --2- 172.21.15.55:0/2161380856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4d414eaa0 0x7fa4d413c730 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:08.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:08.365+0000 7fa4d8c40640 1 --2- 172.21.15.55:0/2161380856 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4d40c60d0 0x7fa4d413c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:08.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:08.365+0000 7fa4d8c40640 1 -- 172.21.15.55:0/2161380856 >> 172.21.15.55:0/2161380856 conn(0x7fa4d40cc9c0 msgr2=0x7fa4d40ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:08.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:08.365+0000 7fa4d8c40640 1 -- 172.21.15.55:0/2161380856 shutdown_connections 2024-03-19T02:06:08.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:08.365+0000 7fa4d8c40640 1 -- 172.21.15.55:0/2161380856 wait complete. 2024-03-19T02:06:08.987 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T01:51:09.453331+0000', 'modified': '2024-03-19T01:55:32.490986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4255, 'mds_1': 4200, 'mds_2': 4237, 'mds_3': 4223, 'mds_4': 4209}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4200': {'gid': 4200, 'name': 'e', 'rank': 1, 'incarnation': 21, 'state': 'up:resolve', 'state_seq': 36, 'addr': '172.21.15.55:6833/3674676035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6832', 'nonce': 3674676035}, {'type': 'v1', 'addr': '172.21.15.55:6833', 'nonce': 3674676035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4209': {'gid': 4209, 'name': 'a', 'rank': 4, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.55:6835/3445165511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6834', 'nonce': 3445165511}, {'type': 'v1', 'addr': '172.21.15.55:6835', 'nonce': 3445165511}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:06.221231+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4223': {'gid': 4223, 'name': 'b', 'rank': 3, 'incarnation': 17, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.84:6837/1680195484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6836', 'nonce': 1680195484}, {'type': 'v1', 'addr': '172.21.15.84:6837', 'nonce': 1680195484}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:53:01.199479+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4237': {'gid': 4237, 'name': 'd', 'rank': 2, 'incarnation': 15, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.84:6835/1290835674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6834', 'nonce': 1290835674}, {'type': 'v1', 'addr': '172.21.15.84:6835', 'nonce': 1290835674}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:52:58.374300+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4255': {'gid': 4255, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.55:6837/2236779395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6836', 'nonce': 2236779395}, {'type': 'v1', 'addr': '172.21.15.55:6837', 'nonce': 2236779395}]}, 'join_fscid': -1, 'laggy_since': '2024-03-19T01:55:32.490982+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4255, 'qdb_cluster': [4255, 4237, 4223, 4209]} 2024-03-19T02:06:08.987 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4200:up:resolve 2024-03-19T02:06:09.953 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:16:22.111 27555== Warning: unimplemented fcntl command: 1036 2024-03-19T02:06:09.962 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:16:22.121 27555== Warning: unimplemented fcntl command: 1036 2024-03-19T02:06:09.974 INFO:tasks.ceph.mon.b.smithi084.stderr:==00:00:16:22.069 27626== Warning: unimplemented fcntl command: 1036 2024-03-19T02:06:09.978 INFO:tasks.ceph.mon.b.smithi084.stderr:==00:00:16:22.073 27626== Warning: unimplemented fcntl command: 1036 2024-03-19T02:06:09.989 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 26, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-19T01:51:09.453331+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 26, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4200": { "addr": "172.21.15.55:6833/3674676035", "addrs": { "addrvec": [ { "addr": "172.21.15.55:6832", "nonce": 3674676035, "type": "v2" }, { "addr": "172.21.15.55:6833", "nonce": 3674676035, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4200, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:resolve", "state_seq": 36 }, "gid_4209": { "addr": "172.21.15.55:6835/3445165511", "addrs": { "addrvec": [ { "addr": "172.21.15.55:6834", "nonce": 3445165511, "type": "v2" }, { "addr": "172.21.15.55:6835", "nonce": 3445165511, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4209, "incarnation": 19, "join_fscid": -1, "laggy_since": "2024-03-19T01:53:06.221231+0000", "name": "a", "rank": 4, "state": "up:active", "state_seq": 22 }, "gid_4223": { "addr": "172.21.15.84:6837/1680195484", "addrs": { "addrvec": [ { "addr": "172.21.15.84:6836", "nonce": 1680195484, "type": "v2" }, { "addr": "172.21.15.84:6837", "nonce": 1680195484, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4223, "incarnation": 17, "join_fscid": -1, "laggy_since": "2024-03-19T01:53:01.199479+0000", "name": "b", "rank": 3, "state": "up:active", "state_seq": 24 }, "gid_4237": { "addr": "172.21.15.84:6835/1290835674", "addrs": { "addrvec": [ { "addr": "172.21.15.84:6834", "nonce": 1290835674, "type": "v2" }, { "addr": "172.21.15.84:6835", "nonce": 1290835674, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4237, "incarnation": 15, "join_fscid": -1, "laggy_since": "2024-03-19T01:52:58.374300+0000", "name": "d", "rank": 2, "state": "up:active", "state_seq": 23 }, "gid_4255": { "addr": "172.21.15.55:6837/2236779395", "addrs": { "addrvec": [ { "addr": "172.21.15.55:6836", "nonce": 2236779395, "type": "v2" }, { "addr": "172.21.15.55:6837", "nonce": 2236779395, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4255, "incarnation": 5, "join_fscid": -1, "laggy_since": "2024-03-19T01:55:32.490982+0000", "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-19T01:55:32.490986+0000", "qdb_cluster": [ 4255, 4237, 4223, 4209 ], "qdb_leader": 4255, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4255, "mds_1": 4200, "mds_2": 4237, "mds_3": 4223, "mds_4": 4209 } } } ], "standbys": [] } 2024-03-19T02:06:09.989 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-19T02:06:09.989 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:06:10.066 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:16:22.161 27627== Warning: unimplemented fcntl command: 1036 2024-03-19T02:06:10.068 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:16:22.163 27627== Warning: unimplemented fcntl command: 1036 2024-03-19T02:06:10.072 INFO:tasks.ceph.mon.b.smithi084.stderr:==00:00:16:22.167 27626== Warning: unimplemented fcntl command: 1036 2024-03-19T02:06:10.084 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:16:22.243 27555== Warning: unimplemented fcntl command: 1036 2024-03-19T02:06:10.103 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.101+0000 7f5131722640 1 Processor -- start 2024-03-19T02:06:10.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.101+0000 7f5131722640 1 -- start start 2024-03-19T02:06:10.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.105+0000 7f5131722640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f512c0cd090 0x7f512c0732b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:10.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.105+0000 7f5131722640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f512c158330 con 0x7f512c0cd470 2024-03-19T02:06:10.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.105+0000 7f5131722640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f512c1585f0 con 0x7f512c14ec60 2024-03-19T02:06:10.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.105+0000 7f5131722640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f512c158e90 con 0x7f512c0cd090 2024-03-19T02:06:10.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.109+0000 7f512b7fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f512c14ec60 0x7f512c06b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:55584/0 (socket says 172.21.15.55:55584) 2024-03-19T02:06:10.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.109+0000 7f512b7fe640 1 -- 172.21.15.55:0/3534537927 learned_addr learned my addr 172.21.15.55:0/3534537927 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:06:10.116 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.117+0000 7f512affd640 1 --2- 172.21.15.55:0/3534537927 >> v2:172.21.15.55:3300/0 conn(0x7f512c0cd090 0x7f512c0732b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:10.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.121+0000 7f512affd640 1 -- 172.21.15.55:0/3534537927 >> v1:172.21.15.84:6789/0 conn(0x7f512c14ec60 legacy=0x7f512c06b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:10.124 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.121+0000 7f512affd640 1 -- 172.21.15.55:0/3534537927 >> v1:172.21.15.55:6789/0 conn(0x7f512c0cd470 legacy=0x7f512c14e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:10.125 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.125+0000 7f512affd640 1 -- 172.21.15.55:0/3534537927 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f512c1588d0 con 0x7f512c0cd090 2024-03-19T02:06:10.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.145+0000 7f512affd640 1 --2- 172.21.15.55:0/3534537927 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f512c0cd090 0x7f512c0732b0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f512c159010 tx=0x7f511c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96d9dd4e973b8a78 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:10.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.169+0000 7f512a7fc640 1 -- 172.21.15.55:0/3534537927 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c0b0070 con 0x7f512c0cd090 2024-03-19T02:06:10.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.169+0000 7f512a7fc640 1 -- 172.21.15.55:0/3534537927 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f511c004230 con 0x7f512c0cd090 2024-03-19T02:06:10.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.169+0000 7f512a7fc640 1 -- 172.21.15.55:0/3534537927 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c003620 con 0x7f512c0cd090 2024-03-19T02:06:10.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.169+0000 7f5131722640 1 -- 172.21.15.55:0/3534537927 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f512c0cd090 msgr2=0x7f512c0732b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:10.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.169+0000 7f5131722640 1 --2- 172.21.15.55:0/3534537927 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f512c0cd090 0x7f512c0732b0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f512c159010 tx=0x7f511c01c460 comp rx=0 tx=0).stop 2024-03-19T02:06:10.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.169+0000 7f5131722640 1 -- 172.21.15.55:0/3534537927 shutdown_connections 2024-03-19T02:06:10.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.169+0000 7f5131722640 1 --2- 172.21.15.55:0/3534537927 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f512c0cd090 0x7f512c0732b0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:10.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.169+0000 7f5131722640 1 -- 172.21.15.55:0/3534537927 >> 172.21.15.55:0/3534537927 conn(0x7f512c06e670 msgr2=0x7f512c06ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:10.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.169+0000 7f5131722640 1 -- 172.21.15.55:0/3534537927 shutdown_connections 2024-03-19T02:06:10.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.169+0000 7f5131722640 1 -- 172.21.15.55:0/3534537927 wait complete. 2024-03-19T02:06:10.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.173+0000 7f5131722640 1 Processor -- start 2024-03-19T02:06:10.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.173+0000 7f5131722640 1 -- start start 2024-03-19T02:06:10.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.173+0000 7f5131722640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f512c0cd090 0x7f512c0c9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:10.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.173+0000 7f5131722640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f512c0cd470 0x7f512c0c32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:10.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.173+0000 7f5131722640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f512c14ec60 0x7f512c0c37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:10.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.173+0000 7f5131722640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f512c0737f0 con 0x7f512c0cd090 2024-03-19T02:06:10.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.173+0000 7f5131722640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f512c166130 con 0x7f512c0cd470 2024-03-19T02:06:10.175 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.173+0000 7f5131722640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f512c146350 con 0x7f512c14ec60 2024-03-19T02:06:10.181 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.181+0000 7f512affd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f512c0cd090 0x7f512c0c9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:10.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.185+0000 7f512affd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f512c0cd090 0x7f512c0c9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57146/0 (socket says 172.21.15.55:57146) 2024-03-19T02:06:10.185 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.185+0000 7f512affd640 1 -- 172.21.15.55:0/1305171632 learned_addr learned my addr 172.21.15.55:0/1305171632 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:10.192 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.193+0000 7f51227fc640 1 --2- 172.21.15.55:0/1305171632 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f512c0cd470 0x7f512c0c32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:10.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.193+0000 7f512affd640 1 -- 172.21.15.55:0/1305171632 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f512c14ec60 msgr2=0x7f512c0c37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:10.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.193+0000 7f512affd640 1 --2- 172.21.15.55:0/1305171632 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f512c14ec60 0x7f512c0c37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:10.193 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.193+0000 7f512affd640 1 -- 172.21.15.55:0/1305171632 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f512c0cd470 msgr2=0x7f512c0c32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:10.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.193+0000 7f512affd640 1 --2- 172.21.15.55:0/1305171632 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f512c0cd470 0x7f512c0c32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:10.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.193+0000 7f512affd640 1 -- 172.21.15.55:0/1305171632 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f512c147500 con 0x7f512c0cd090 2024-03-19T02:06:10.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.201+0000 7f512affd640 1 --2- 172.21.15.55:0/1305171632 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f512c0cd090 0x7f512c0c9070 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f511c0ae040 tx=0x7f511c019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:10.229 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:16:22.324 27627== Warning: unimplemented fcntl command: 1036 2024-03-19T02:06:10.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.233+0000 7f5123fff640 1 -- 172.21.15.55:0/1305171632 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c0b9070 con 0x7f512c0cd090 2024-03-19T02:06:10.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.233+0000 7f5123fff640 1 -- 172.21.15.55:0/1305171632 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f511c0043e0 con 0x7f512c0cd090 2024-03-19T02:06:10.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.233+0000 7f5123fff640 1 -- 172.21.15.55:0/1305171632 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c0b0040 con 0x7f512c0cd090 2024-03-19T02:06:10.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.233+0000 7f5131722640 1 -- 172.21.15.55:0/1305171632 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f512c1467b0 con 0x7f512c0cd090 2024-03-19T02:06:10.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.233+0000 7f5131722640 1 -- 172.21.15.55:0/1305171632 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f512c146d70 con 0x7f512c0cd090 2024-03-19T02:06:10.235 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.233+0000 7f5131722640 1 -- 172.21.15.55:0/1305171632 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f512c068a40 con 0x7f512c0cd090 2024-03-19T02:06:10.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.561+0000 7f5123fff640 1 -- 172.21.15.55:0/1305171632 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f511c0bc070 con 0x7f512c0cd090 2024-03-19T02:06:10.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.561+0000 7f5123fff640 1 --2- 172.21.15.55:0/1305171632 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f510c0676b0 0x7f510c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:10.562 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.561+0000 7f51227fc640 1 --2- 172.21.15.55:0/1305171632 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f510c0676b0 0x7f510c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:10.563 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.561+0000 7f51227fc640 1 --2- 172.21.15.55:0/1305171632 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f510c0676b0 0x7f510c069b70 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f512c1522e0 tx=0x7f5114008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:10.591 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.589+0000 7f5123fff640 1 -- 172.21.15.55:0/1305171632 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f511c0ee4d0 con 0x7f512c0cd090 2024-03-19T02:06:10.591 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.589+0000 7f5123fff640 1 -- 172.21.15.55:0/1305171632 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f511c016050 con 0x7f512c0cd090 2024-03-19T02:06:10.762 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.761+0000 7f5131722640 1 -- 172.21.15.55:0/1305171632 --> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f512c156b70 con 0x7f510c0676b0 2024-03-19T02:06:10.779 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.777+0000 7f5123fff640 1 -- 172.21.15.55:0/1305171632 <== mgr.4105 v2:172.21.15.84:6800/998533783 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332628 (secure 0 0 0) 0x7f512c156b70 con 0x7f510c0676b0 2024-03-19T02:06:10.784 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:06:10.785 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-19T02:06:10.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.781+0000 7f5131722640 1 -- 172.21.15.55:0/1305171632 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f510c0676b0 msgr2=0x7f510c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:10.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.781+0000 7f5131722640 1 --2- 172.21.15.55:0/1305171632 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f510c0676b0 0x7f510c069b70 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f512c1522e0 tx=0x7f5114008040 comp rx=0 tx=0).stop 2024-03-19T02:06:10.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.781+0000 7f5131722640 1 -- 172.21.15.55:0/1305171632 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f512c0cd090 msgr2=0x7f512c0c9070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:10.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.781+0000 7f5131722640 1 --2- 172.21.15.55:0/1305171632 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f512c0cd090 0x7f512c0c9070 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f511c0ae040 tx=0x7f511c019200 comp rx=0 tx=0).stop 2024-03-19T02:06:10.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.785+0000 7f5131722640 1 -- 172.21.15.55:0/1305171632 shutdown_connections 2024-03-19T02:06:10.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.785+0000 7f5131722640 1 --2- 172.21.15.55:0/1305171632 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f510c0676b0 0x7f510c069b70 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:10.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.785+0000 7f5131722640 1 --2- 172.21.15.55:0/1305171632 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f512c14ec60 0x7f512c0c37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:10.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.785+0000 7f5131722640 1 --2- 172.21.15.55:0/1305171632 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f512c0cd470 0x7f512c0c32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:10.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.785+0000 7f5131722640 1 --2- 172.21.15.55:0/1305171632 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f512c0cd090 0x7f512c0c9070 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:10.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.785+0000 7f5131722640 1 -- 172.21.15.55:0/1305171632 >> 172.21.15.55:0/1305171632 conn(0x7f512c06e670 msgr2=0x7f512c156d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:10.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.785+0000 7f5131722640 1 -- 172.21.15.55:0/1305171632 shutdown_connections 2024-03-19T02:06:10.787 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.785+0000 7f5131722640 1 -- 172.21.15.55:0/1305171632 wait complete. 2024-03-19T02:06:10.801 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-03-19T02:06:10.603886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":368,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":306,"ondisk_log_size":306,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237212,"kb_used_data":4392,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737508,"statfs":{"total":773094113280,"available":772851208192,"internally_reserved":0,"allocated":4497408,"data_stored":2535616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004437"},"pg_stats":[{"pgid":"2.1f","version":"29'13","reported_seq":61,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.576125+0000","last_change":"2024-03-19T01:51:06.982533+0000","last_active":"2024-03-19T02:03:38.576125+0000","last_peered":"2024-03-19T02:03:38.576125+0000","last_clean":"2024-03-19T02:03:38.576125+0000","last_became_active":"2024-03-19T01:51:06.981802+0000","last_became_peered":"2024-03-19T01:51:06.981802+0000","last_unstale":"2024-03-19T02:03:38.576125+0000","last_undegraded":"2024-03-19T02:03:38.576125+0000","last_fullsized":"2024-03-19T02:03:38.576125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:10:01.431003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:11.629408+0000","last_change":"2024-03-19T01:51:09.172057+0000","last_active":"2024-03-19T02:03:11.629408+0000","last_peered":"2024-03-19T02:03:11.629408+0000","last_clean":"2024-03-19T02:03:11.629408+0000","last_became_active":"2024-03-19T01:51:09.171753+0000","last_became_peered":"2024-03-19T01:51:09.171753+0000","last_unstale":"2024-03-19T02:03:11.629408+0000","last_undegraded":"2024-03-19T02:03:11.629408+0000","last_fullsized":"2024-03-19T02:03:11.629408+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:50.575804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:21.630796+0000","last_change":"2024-03-19T01:51:16.537753+0000","last_active":"2024-03-19T02:03:21.630796+0000","last_peered":"2024-03-19T02:03:21.630796+0000","last_clean":"2024-03-19T02:03:21.630796+0000","last_became_active":"2024-03-19T01:51:16.534200+0000","last_became_peered":"2024-03-19T01:51:16.534200+0000","last_unstale":"2024-03-19T02:03:21.630796+0000","last_undegraded":"2024-03-19T02:03:21.630796+0000","last_fullsized":"2024-03-19T02:03:21.630796+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:18:34.208854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":50,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:06.628579+0000","last_change":"2024-03-19T01:51:06.962968+0000","last_active":"2024-03-19T02:03:06.628579+0000","last_peered":"2024-03-19T02:03:06.628579+0000","last_clean":"2024-03-19T02:03:06.628579+0000","last_became_active":"2024-03-19T01:51:06.961856+0000","last_became_peered":"2024-03-19T01:51:06.961856+0000","last_unstale":"2024-03-19T02:03:06.628579+0000","last_undegraded":"2024-03-19T02:03:06.628579+0000","last_fullsized":"2024-03-19T02:03:06.628579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:49.676651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.576098+0000","last_change":"2024-03-19T01:51:09.160338+0000","last_active":"2024-03-19T02:03:38.576098+0000","last_peered":"2024-03-19T02:03:38.576098+0000","last_clean":"2024-03-19T02:03:38.576098+0000","last_became_active":"2024-03-19T01:51:09.158682+0000","last_became_peered":"2024-03-19T01:51:09.158682+0000","last_unstale":"2024-03-19T02:03:38.576098+0000","last_undegraded":"2024-03-19T02:03:38.576098+0000","last_fullsized":"2024-03-19T02:03:38.576098+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":39,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:23.616915+0000","last_change":"2024-03-19T01:52:14.299568+0000","last_active":"2024-03-19T02:03:23.616915+0000","last_peered":"2024-03-19T02:03:23.616915+0000","last_clean":"2024-03-19T02:03:23.616915+0000","last_became_active":"2024-03-19T01:52:14.299067+0000","last_became_peered":"2024-03-19T01:52:14.299067+0000","last_unstale":"2024-03-19T02:03:23.616915+0000","last_undegraded":"2024-03-19T02:03:23.616915+0000","last_fullsized":"2024-03-19T02:03:23.616915+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:15:10.967874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"35'12","reported_seq":59,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603963+0000","last_change":"2024-03-19T01:51:07.044049+0000","last_active":"2024-03-19T02:03:38.603963+0000","last_peered":"2024-03-19T02:03:38.603963+0000","last_clean":"2024-03-19T02:03:38.603963+0000","last_became_active":"2024-03-19T01:51:07.043209+0000","last_became_peered":"2024-03-19T01:51:07.043209+0000","last_unstale":"2024-03-19T02:03:38.603963+0000","last_undegraded":"2024-03-19T02:03:38.603963+0000","last_fullsized":"2024-03-19T02:03:38.603963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:42.992419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:11.654315+0000","last_change":"2024-03-19T01:51:09.179615+0000","last_active":"2024-03-19T02:01:11.654315+0000","last_peered":"2024-03-19T02:01:11.654315+0000","last_clean":"2024-03-19T02:01:11.654315+0000","last_became_active":"2024-03-19T01:51:09.173962+0000","last_became_peered":"2024-03-19T01:51:09.173962+0000","last_unstale":"2024-03-19T02:01:11.654315+0000","last_undegraded":"2024-03-19T02:01:11.654315+0000","last_fullsized":"2024-03-19T02:01:11.654315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:06:12.676868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":38,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:18.601453+0000","last_change":"2024-03-19T01:53:15.447584+0000","last_active":"2024-03-19T02:01:18.601453+0000","last_peered":"2024-03-19T02:01:18.601453+0000","last_clean":"2024-03-19T02:01:18.601453+0000","last_became_active":"2024-03-19T01:53:15.447002+0000","last_became_peered":"2024-03-19T01:53:15.447002+0000","last_unstale":"2024-03-19T02:01:18.601453+0000","last_undegraded":"2024-03-19T02:01:18.601453+0000","last_fullsized":"2024-03-19T02:01:18.601453+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:02:57.393473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6,1],"acting":[4,2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"29'5","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603940+0000","last_change":"2024-03-19T01:51:06.983291+0000","last_active":"2024-03-19T02:03:38.603940+0000","last_peered":"2024-03-19T02:03:38.603940+0000","last_clean":"2024-03-19T02:03:38.603940+0000","last_became_active":"2024-03-19T01:51:06.982309+0000","last_became_peered":"2024-03-19T01:51:06.982309+0000","last_unstale":"2024-03-19T02:03:38.603940+0000","last_undegraded":"2024-03-19T02:03:38.603940+0000","last_fullsized":"2024-03-19T02:03:38.603940+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:27.305286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:11.654335+0000","last_change":"2024-03-19T01:51:09.160738+0000","last_active":"2024-03-19T02:01:11.654335+0000","last_peered":"2024-03-19T02:01:11.654335+0000","last_clean":"2024-03-19T02:01:11.654335+0000","last_became_active":"2024-03-19T01:51:09.159951+0000","last_became_peered":"2024-03-19T01:51:09.159951+0000","last_unstale":"2024-03-19T02:01:11.654335+0000","last_undegraded":"2024-03-19T02:01:11.654335+0000","last_fullsized":"2024-03-19T02:01:11.654335+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:30.620041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":31,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:18.601421+0000","last_change":"2024-03-19T01:51:16.577378+0000","last_active":"2024-03-19T02:01:18.601421+0000","last_peered":"2024-03-19T02:01:18.601421+0000","last_clean":"2024-03-19T02:01:18.601421+0000","last_became_active":"2024-03-19T01:51:16.576256+0000","last_became_peered":"2024-03-19T01:51:16.576256+0000","last_unstale":"2024-03-19T02:01:18.601421+0000","last_undegraded":"2024-03-19T02:01:18.601421+0000","last_fullsized":"2024-03-19T02:01:18.601421+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:16.096547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.494171+0000","last_change":"2024-03-19T01:51:09.185969+0000","last_active":"2024-03-19T02:01:58.494171+0000","last_peered":"2024-03-19T02:01:58.494171+0000","last_clean":"2024-03-19T02:01:58.494171+0000","last_became_active":"2024-03-19T01:51:09.185378+0000","last_became_peered":"2024-03-19T01:51:09.185378+0000","last_unstale":"2024-03-19T02:01:58.494171+0000","last_undegraded":"2024-03-19T02:01:58.494171+0000","last_fullsized":"2024-03-19T02:01:58.494171+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":51,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.594329+0000","last_change":"2024-03-19T01:51:06.988067+0000","last_active":"2024-03-19T02:02:23.594329+0000","last_peered":"2024-03-19T02:02:23.594329+0000","last_clean":"2024-03-19T02:02:23.594329+0000","last_became_active":"2024-03-19T01:51:06.987450+0000","last_became_peered":"2024-03-19T01:51:06.987450+0000","last_unstale":"2024-03-19T02:02:23.594329+0000","last_undegraded":"2024-03-19T02:02:23.594329+0000","last_fullsized":"2024-03-19T02:02:23.594329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:56.962922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:33.610729+0000","last_change":"2024-03-19T01:51:16.583190+0000","last_active":"2024-03-19T02:02:33.610729+0000","last_peered":"2024-03-19T02:02:33.610729+0000","last_clean":"2024-03-19T02:02:33.610729+0000","last_became_active":"2024-03-19T01:51:16.582532+0000","last_became_peered":"2024-03-19T01:51:16.582532+0000","last_unstale":"2024-03-19T02:02:33.610729+0000","last_undegraded":"2024-03-19T02:02:33.610729+0000","last_fullsized":"2024-03-19T02:02:33.610729+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:16:49.817940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:26.624005+0000","last_change":"2024-03-19T01:51:09.174687+0000","last_active":"2024-03-19T02:02:26.624005+0000","last_peered":"2024-03-19T02:02:26.624005+0000","last_clean":"2024-03-19T02:02:26.624005+0000","last_became_active":"2024-03-19T01:51:09.174409+0000","last_became_peered":"2024-03-19T01:51:09.174409+0000","last_unstale":"2024-03-19T02:02:26.624005+0000","last_undegraded":"2024-03-19T02:02:26.624005+0000","last_fullsized":"2024-03-19T02:02:26.624005+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:33:07.160350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"35'3","reported_seq":51,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.504638+0000","last_change":"2024-03-19T01:51:06.955434+0000","last_active":"2024-03-19T02:01:58.504638+0000","last_peered":"2024-03-19T02:01:58.504638+0000","last_clean":"2024-03-19T02:01:58.504638+0000","last_became_active":"2024-03-19T01:51:06.954635+0000","last_became_peered":"2024-03-19T01:51:06.954635+0000","last_unstale":"2024-03-19T02:01:58.504638+0000","last_undegraded":"2024-03-19T02:01:58.504638+0000","last_fullsized":"2024-03-19T02:01:58.504638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:41:39.176905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:33.610667+0000","last_change":"2024-03-19T01:51:16.588604+0000","last_active":"2024-03-19T02:02:33.610667+0000","last_peered":"2024-03-19T02:02:33.610667+0000","last_clean":"2024-03-19T02:02:33.610667+0000","last_became_active":"2024-03-19T01:51:16.587483+0000","last_became_peered":"2024-03-19T01:51:16.587483+0000","last_unstale":"2024-03-19T02:02:33.610667+0000","last_undegraded":"2024-03-19T02:02:33.610667+0000","last_fullsized":"2024-03-19T02:02:33.610667+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:51.658547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:01.620586+0000","last_change":"2024-03-19T01:51:09.173987+0000","last_active":"2024-03-19T02:02:01.620586+0000","last_peered":"2024-03-19T02:02:01.620586+0000","last_clean":"2024-03-19T02:02:01.620586+0000","last_became_active":"2024-03-19T01:51:09.173648+0000","last_became_peered":"2024-03-19T01:51:09.173648+0000","last_unstale":"2024-03-19T02:02:01.620586+0000","last_undegraded":"2024-03-19T02:02:01.620586+0000","last_fullsized":"2024-03-19T02:02:01.620586+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:13:17.277014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"29'7","reported_seq":55,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.504602+0000","last_change":"2024-03-19T01:51:06.981820+0000","last_active":"2024-03-19T02:01:58.504602+0000","last_peered":"2024-03-19T02:01:58.504602+0000","last_clean":"2024-03-19T02:01:58.504602+0000","last_became_active":"2024-03-19T01:51:06.981087+0000","last_became_peered":"2024-03-19T01:51:06.981087+0000","last_unstale":"2024-03-19T02:01:58.504602+0000","last_undegraded":"2024-03-19T02:01:58.504602+0000","last_fullsized":"2024-03-19T02:01:58.504602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:25:59.116138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:33.509725+0000","last_change":"2024-03-19T01:51:16.579017+0000","last_active":"2024-03-19T02:02:33.509725+0000","last_peered":"2024-03-19T02:02:33.509725+0000","last_clean":"2024-03-19T02:02:33.509725+0000","last_became_active":"2024-03-19T01:51:16.573845+0000","last_became_peered":"2024-03-19T01:51:16.573845+0000","last_unstale":"2024-03-19T02:02:33.509725+0000","last_undegraded":"2024-03-19T02:02:33.509725+0000","last_fullsized":"2024-03-19T02:02:33.509725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:42:30.142766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.607044+0000","last_change":"2024-03-19T01:51:09.192886+0000","last_active":"2024-03-19T02:01:58.607044+0000","last_peered":"2024-03-19T02:01:58.607044+0000","last_clean":"2024-03-19T02:01:58.607044+0000","last_became_active":"2024-03-19T01:51:09.185970+0000","last_became_peered":"2024-03-19T01:51:09.185970+0000","last_unstale":"2024-03-19T02:01:58.607044+0000","last_undegraded":"2024-03-19T02:01:58.607044+0000","last_fullsized":"2024-03-19T02:01:58.607044+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:05.196985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":49,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.591524+0000","last_change":"2024-03-19T01:51:06.984875+0000","last_active":"2024-03-19T02:01:58.591524+0000","last_peered":"2024-03-19T02:01:58.591524+0000","last_clean":"2024-03-19T02:01:58.591524+0000","last_became_active":"2024-03-19T01:51:06.983858+0000","last_became_peered":"2024-03-19T01:51:06.983858+0000","last_unstale":"2024-03-19T02:01:58.591524+0000","last_undegraded":"2024-03-19T02:01:58.591524+0000","last_fullsized":"2024-03-19T02:01:58.591524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:14.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:33.610746+0000","last_change":"2024-03-19T01:51:16.591614+0000","last_active":"2024-03-19T02:02:33.610746+0000","last_peered":"2024-03-19T02:02:33.610746+0000","last_clean":"2024-03-19T02:02:33.610746+0000","last_became_active":"2024-03-19T01:51:16.590968+0000","last_became_peered":"2024-03-19T01:51:16.590968+0000","last_unstale":"2024-03-19T02:02:33.610746+0000","last_undegraded":"2024-03-19T02:02:33.610746+0000","last_fullsized":"2024-03-19T02:02:33.610746+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:50.144136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.564107+0000","last_change":"2024-03-19T01:51:09.168941+0000","last_active":"2024-03-19T02:01:58.564107+0000","last_peered":"2024-03-19T02:01:58.564107+0000","last_clean":"2024-03-19T02:01:58.564107+0000","last_became_active":"2024-03-19T01:51:09.168243+0000","last_became_peered":"2024-03-19T01:51:09.168243+0000","last_unstale":"2024-03-19T02:01:58.564107+0000","last_undegraded":"2024-03-19T02:01:58.564107+0000","last_fullsized":"2024-03-19T02:01:58.564107+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'7","reported_seq":54,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.494241+0000","last_change":"2024-03-19T01:51:07.042870+0000","last_active":"2024-03-19T02:01:58.494241+0000","last_peered":"2024-03-19T02:01:58.494241+0000","last_clean":"2024-03-19T02:01:58.494241+0000","last_became_active":"2024-03-19T01:51:07.042327+0000","last_became_peered":"2024-03-19T01:51:07.042327+0000","last_unstale":"2024-03-19T02:01:58.494241+0000","last_undegraded":"2024-03-19T02:01:58.494241+0000","last_fullsized":"2024-03-19T02:01:58.494241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:13.137673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.619433+0000","last_change":"2024-03-19T01:51:16.587730+0000","last_active":"2024-03-19T02:03:48.619433+0000","last_peered":"2024-03-19T02:03:48.619433+0000","last_clean":"2024-03-19T02:03:48.619433+0000","last_became_active":"2024-03-19T01:51:16.586614+0000","last_became_peered":"2024-03-19T01:51:16.586614+0000","last_unstale":"2024-03-19T02:03:48.619433+0000","last_undegraded":"2024-03-19T02:03:48.619433+0000","last_fullsized":"2024-03-19T02:03:48.619433+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:44.830637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:01.620647+0000","last_change":"2024-03-19T01:51:09.186397+0000","last_active":"2024-03-19T02:02:01.620647+0000","last_peered":"2024-03-19T02:02:01.620647+0000","last_clean":"2024-03-19T02:02:01.620647+0000","last_became_active":"2024-03-19T01:51:09.186033+0000","last_became_peered":"2024-03-19T01:51:09.186033+0000","last_unstale":"2024-03-19T02:02:01.620647+0000","last_undegraded":"2024-03-19T02:02:01.620647+0000","last_fullsized":"2024-03-19T02:02:01.620647+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:03:04.997711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"29'7","reported_seq":54,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.504687+0000","last_change":"2024-03-19T01:51:07.043697+0000","last_active":"2024-03-19T02:01:58.504687+0000","last_peered":"2024-03-19T02:01:58.504687+0000","last_clean":"2024-03-19T02:01:58.504687+0000","last_became_active":"2024-03-19T01:51:07.043185+0000","last_became_peered":"2024-03-19T01:51:07.043185+0000","last_unstale":"2024-03-19T02:01:58.504687+0000","last_undegraded":"2024-03-19T02:01:58.504687+0000","last_fullsized":"2024-03-19T02:01:58.504687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:19.016579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:16.630056+0000","last_change":"2024-03-19T01:51:16.535771+0000","last_active":"2024-03-19T02:03:16.630056+0000","last_peered":"2024-03-19T02:03:16.630056+0000","last_clean":"2024-03-19T02:03:16.630056+0000","last_became_active":"2024-03-19T01:51:16.533661+0000","last_became_peered":"2024-03-19T01:51:16.533661+0000","last_unstale":"2024-03-19T02:03:16.630056+0000","last_undegraded":"2024-03-19T02:03:16.630056+0000","last_fullsized":"2024-03-19T02:03:16.630056+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:55.283782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.518547+0000","last_change":"2024-03-19T01:51:09.157526+0000","last_active":"2024-03-19T02:03:38.518547+0000","last_peered":"2024-03-19T02:03:38.518547+0000","last_clean":"2024-03-19T02:03:38.518547+0000","last_became_active":"2024-03-19T01:51:09.156336+0000","last_became_peered":"2024-03-19T01:51:09.156336+0000","last_unstale":"2024-03-19T02:03:38.518547+0000","last_undegraded":"2024-03-19T02:03:38.518547+0000","last_fullsized":"2024-03-19T02:03:38.518547+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'4","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603980+0000","last_change":"2024-03-19T01:51:06.982994+0000","last_active":"2024-03-19T02:03:38.603980+0000","last_peered":"2024-03-19T02:03:38.603980+0000","last_clean":"2024-03-19T02:03:38.603980+0000","last_became_active":"2024-03-19T01:51:06.982133+0000","last_became_peered":"2024-03-19T01:51:06.982133+0000","last_unstale":"2024-03-19T02:03:38.603980+0000","last_undegraded":"2024-03-19T02:03:38.603980+0000","last_fullsized":"2024-03-19T02:03:38.603980+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:03.574098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.577343+0000","last_change":"2024-03-19T01:51:16.578493+0000","last_active":"2024-03-19T02:03:48.577343+0000","last_peered":"2024-03-19T02:03:48.577343+0000","last_clean":"2024-03-19T02:03:48.577343+0000","last_became_active":"2024-03-19T01:51:16.578040+0000","last_became_peered":"2024-03-19T01:51:16.578040+0000","last_unstale":"2024-03-19T02:03:48.577343+0000","last_undegraded":"2024-03-19T02:03:48.577343+0000","last_fullsized":"2024-03-19T02:03:48.577343+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:45.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:41.673779+0000","last_change":"2024-03-19T01:51:09.180758+0000","last_active":"2024-03-19T02:03:41.673779+0000","last_peered":"2024-03-19T02:03:41.673779+0000","last_clean":"2024-03-19T02:03:41.673779+0000","last_became_active":"2024-03-19T01:51:09.180326+0000","last_became_peered":"2024-03-19T01:51:09.180326+0000","last_unstale":"2024-03-19T02:03:41.673779+0000","last_undegraded":"2024-03-19T02:03:41.673779+0000","last_fullsized":"2024-03-19T02:03:41.673779+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:56:07.046285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"29'1","reported_seq":49,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.518603+0000","last_change":"2024-03-19T01:51:06.962643+0000","last_active":"2024-03-19T02:03:38.518603+0000","last_peered":"2024-03-19T02:03:38.518603+0000","last_clean":"2024-03-19T02:03:38.518603+0000","last_became_active":"2024-03-19T01:51:06.961862+0000","last_became_peered":"2024-03-19T01:51:06.961862+0000","last_unstale":"2024-03-19T02:03:38.518603+0000","last_undegraded":"2024-03-19T02:03:38.518603+0000","last_fullsized":"2024-03-19T02:03:38.518603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.520313+0000","last_change":"2024-03-19T01:51:16.577730+0000","last_active":"2024-03-19T02:03:48.520313+0000","last_peered":"2024-03-19T02:03:48.520313+0000","last_clean":"2024-03-19T02:03:48.520313+0000","last_became_active":"2024-03-19T01:51:16.576955+0000","last_became_peered":"2024-03-19T01:51:16.576955+0000","last_unstale":"2024-03-19T02:03:48.520313+0000","last_undegraded":"2024-03-19T02:03:48.520313+0000","last_fullsized":"2024-03-19T02:03:48.520313+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:35.409504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.618478+0000","last_change":"2024-03-19T01:51:09.183678+0000","last_active":"2024-03-19T02:03:38.618478+0000","last_peered":"2024-03-19T02:03:38.618478+0000","last_clean":"2024-03-19T02:03:38.618478+0000","last_became_active":"2024-03-19T01:51:09.176780+0000","last_became_peered":"2024-03-19T01:51:09.176780+0000","last_unstale":"2024-03-19T02:03:38.618478+0000","last_undegraded":"2024-03-19T02:03:38.618478+0000","last_fullsized":"2024-03-19T02:03:38.618478+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:11:02.502333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'5","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:36.673161+0000","last_change":"2024-03-19T01:51:06.956307+0000","last_active":"2024-03-19T02:03:36.673161+0000","last_peered":"2024-03-19T02:03:36.673161+0000","last_clean":"2024-03-19T02:03:36.673161+0000","last_became_active":"2024-03-19T01:51:06.955529+0000","last_became_peered":"2024-03-19T01:51:06.955529+0000","last_unstale":"2024-03-19T02:03:36.673161+0000","last_undegraded":"2024-03-19T02:03:36.673161+0000","last_fullsized":"2024-03-19T02:03:36.673161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:10.486012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.520349+0000","last_change":"2024-03-19T01:51:16.550718+0000","last_active":"2024-03-19T02:03:48.520349+0000","last_peered":"2024-03-19T02:03:48.520349+0000","last_clean":"2024-03-19T02:03:48.520349+0000","last_became_active":"2024-03-19T01:51:16.550115+0000","last_became_peered":"2024-03-19T01:51:16.550115+0000","last_unstale":"2024-03-19T02:03:48.520349+0000","last_undegraded":"2024-03-19T02:03:48.520349+0000","last_fullsized":"2024-03-19T02:03:48.520349+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:43.172796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:08.514405+0000","last_change":"2024-03-19T01:51:09.155718+0000","last_active":"2024-03-19T02:03:08.514405+0000","last_peered":"2024-03-19T02:03:08.514405+0000","last_clean":"2024-03-19T02:03:08.514405+0000","last_became_active":"2024-03-19T01:51:09.155014+0000","last_became_peered":"2024-03-19T01:51:09.155014+0000","last_unstale":"2024-03-19T02:03:08.514405+0000","last_undegraded":"2024-03-19T02:03:08.514405+0000","last_fullsized":"2024-03-19T02:03:08.514405+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"33'6","reported_seq":54,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:36.559085+0000","last_change":"2024-03-19T01:51:06.955372+0000","last_active":"2024-03-19T02:03:36.559085+0000","last_peered":"2024-03-19T02:03:36.559085+0000","last_clean":"2024-03-19T02:03:36.559085+0000","last_became_active":"2024-03-19T01:51:06.954637+0000","last_became_peered":"2024-03-19T01:51:06.954637+0000","last_unstale":"2024-03-19T02:03:36.559085+0000","last_undegraded":"2024-03-19T02:03:36.559085+0000","last_fullsized":"2024-03-19T02:03:36.559085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:38:20.129564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:08.505884+0000","last_change":"2024-03-19T01:51:16.575157+0000","last_active":"2024-03-19T02:02:08.505884+0000","last_peered":"2024-03-19T02:02:08.505884+0000","last_clean":"2024-03-19T02:02:08.505884+0000","last_became_active":"2024-03-19T01:51:16.574575+0000","last_became_peered":"2024-03-19T01:51:16.574575+0000","last_unstale":"2024-03-19T02:02:08.505884+0000","last_undegraded":"2024-03-19T02:02:08.505884+0000","last_fullsized":"2024-03-19T02:02:08.505884+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:15:17.718630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.575915+0000","last_change":"2024-03-19T01:51:09.192610+0000","last_active":"2024-03-19T02:03:38.575915+0000","last_peered":"2024-03-19T02:03:38.575915+0000","last_clean":"2024-03-19T02:03:38.575915+0000","last_became_active":"2024-03-19T01:51:09.192280+0000","last_became_peered":"2024-03-19T01:51:09.192280+0000","last_unstale":"2024-03-19T02:03:38.575915+0000","last_undegraded":"2024-03-19T02:03:38.575915+0000","last_fullsized":"2024-03-19T02:03:38.575915+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:15.454555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"33'5","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:08.600010+0000","last_change":"2024-03-19T01:51:06.986895+0000","last_active":"2024-03-19T02:03:08.600010+0000","last_peered":"2024-03-19T02:03:08.600010+0000","last_clean":"2024-03-19T02:03:08.600010+0000","last_became_active":"2024-03-19T01:51:06.985965+0000","last_became_peered":"2024-03-19T01:51:06.985965+0000","last_unstale":"2024-03-19T02:03:08.600010+0000","last_undegraded":"2024-03-19T02:03:08.600010+0000","last_fullsized":"2024-03-19T02:03:08.600010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:08.565447+0000","last_change":"2024-03-19T01:51:16.505278+0000","last_active":"2024-03-19T02:02:08.565447+0000","last_peered":"2024-03-19T02:02:08.565447+0000","last_clean":"2024-03-19T02:02:08.565447+0000","last_became_active":"2024-03-19T01:51:16.504908+0000","last_became_peered":"2024-03-19T01:51:16.504908+0000","last_unstale":"2024-03-19T02:02:08.565447+0000","last_undegraded":"2024-03-19T02:02:08.565447+0000","last_fullsized":"2024-03-19T02:02:08.565447+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:42:30.142766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:41.633422+0000","last_change":"2024-03-19T01:51:09.179725+0000","last_active":"2024-03-19T02:03:41.633422+0000","last_peered":"2024-03-19T02:03:41.633422+0000","last_clean":"2024-03-19T02:03:41.633422+0000","last_became_active":"2024-03-19T01:51:09.179235+0000","last_became_peered":"2024-03-19T01:51:09.179235+0000","last_unstale":"2024-03-19T02:03:41.633422+0000","last_undegraded":"2024-03-19T02:03:41.633422+0000","last_fullsized":"2024-03-19T02:03:41.633422+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:31.395833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'6","reported_seq":55,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:36.673191+0000","last_change":"2024-03-19T01:51:06.953474+0000","last_active":"2024-03-19T02:03:36.673191+0000","last_peered":"2024-03-19T02:03:36.673191+0000","last_clean":"2024-03-19T02:03:36.673191+0000","last_became_active":"2024-03-19T01:51:06.952898+0000","last_became_peered":"2024-03-19T01:51:06.952898+0000","last_unstale":"2024-03-19T02:03:36.673191+0000","last_undegraded":"2024-03-19T02:03:36.673191+0000","last_fullsized":"2024-03-19T02:03:36.673191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:38.808918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":203,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:28.594947+0000","last_change":"2024-03-19T01:50:26.097484+0000","last_active":"2024-03-19T02:02:28.594947+0000","last_peered":"2024-03-19T02:02:28.594947+0000","last_clean":"2024-03-19T02:02:28.594947+0000","last_became_active":"2024-03-19T01:50:26.095496+0000","last_became_peered":"2024-03-19T01:50:26.095496+0000","last_unstale":"2024-03-19T02:02:28.594947+0000","last_undegraded":"2024-03-19T02:02:28.594947+0000","last_fullsized":"2024-03-19T02:02:28.594947+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:50:24.112070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:50:24.112070+0000","last_clean_scrub_stamp":"2024-03-19T01:50:24.112070+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:05:00.843775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":40,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.507121+0000","last_change":"2024-03-19T01:52:14.333085+0000","last_active":"2024-03-19T02:03:48.507121+0000","last_peered":"2024-03-19T02:03:48.507121+0000","last_clean":"2024-03-19T02:03:48.507121+0000","last_became_active":"2024-03-19T01:52:14.332547+0000","last_became_peered":"2024-03-19T01:52:14.332547+0000","last_unstale":"2024-03-19T02:03:48.507121+0000","last_undegraded":"2024-03-19T02:03:48.507121+0000","last_fullsized":"2024-03-19T02:03:48.507121+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:36.377719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4,7],"acting":[6,0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.594377+0000","last_change":"2024-03-19T01:51:09.196648+0000","last_active":"2024-03-19T02:02:23.594377+0000","last_peered":"2024-03-19T02:02:23.594377+0000","last_clean":"2024-03-19T02:02:23.594377+0000","last_became_active":"2024-03-19T01:51:09.196093+0000","last_became_peered":"2024-03-19T01:51:09.196093+0000","last_unstale":"2024-03-19T02:02:23.594377+0000","last_undegraded":"2024-03-19T02:02:23.594377+0000","last_fullsized":"2024-03-19T02:02:23.594377+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:54:09.655994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"29'5","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:21.549717+0000","last_change":"2024-03-19T01:51:06.950023+0000","last_active":"2024-03-19T02:02:21.549717+0000","last_peered":"2024-03-19T02:02:21.549717+0000","last_clean":"2024-03-19T02:02:21.549717+0000","last_became_active":"2024-03-19T01:51:06.949234+0000","last_became_peered":"2024-03-19T01:51:06.949234+0000","last_unstale":"2024-03-19T02:02:21.549717+0000","last_undegraded":"2024-03-19T02:02:21.549717+0000","last_fullsized":"2024-03-19T02:02:21.549717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:54:13.131325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:08.495157+0000","last_change":"2024-03-19T01:51:16.554046+0000","last_active":"2024-03-19T02:02:08.495157+0000","last_peered":"2024-03-19T02:02:08.495157+0000","last_clean":"2024-03-19T02:02:08.495157+0000","last_became_active":"2024-03-19T01:51:16.553595+0000","last_became_peered":"2024-03-19T01:51:16.553595+0000","last_unstale":"2024-03-19T02:02:08.495157+0000","last_undegraded":"2024-03-19T02:02:08.495157+0000","last_fullsized":"2024-03-19T02:02:08.495157+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:25.684226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:26.557693+0000","last_change":"2024-03-19T01:52:12.918926+0000","last_active":"2024-03-19T02:03:26.557693+0000","last_peered":"2024-03-19T02:03:26.557693+0000","last_clean":"2024-03-19T02:03:26.557693+0000","last_became_active":"2024-03-19T01:52:12.918458+0000","last_became_peered":"2024-03-19T01:52:12.918458+0000","last_unstale":"2024-03-19T02:03:26.557693+0000","last_undegraded":"2024-03-19T02:03:26.557693+0000","last_fullsized":"2024-03-19T02:03:26.557693+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:56.318137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"29'3","reported_seq":50,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.508282+0000","last_change":"2024-03-19T01:51:06.938396+0000","last_active":"2024-03-19T02:02:23.508282+0000","last_peered":"2024-03-19T02:02:23.508282+0000","last_clean":"2024-03-19T02:02:23.508282+0000","last_became_active":"2024-03-19T01:51:06.937580+0000","last_became_peered":"2024-03-19T01:51:06.937580+0000","last_unstale":"2024-03-19T02:02:23.508282+0000","last_undegraded":"2024-03-19T02:02:23.508282+0000","last_fullsized":"2024-03-19T02:02:23.508282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:08.307376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:33.568500+0000","last_change":"2024-03-19T01:51:16.582111+0000","last_active":"2024-03-19T02:02:33.568500+0000","last_peered":"2024-03-19T02:02:33.568500+0000","last_clean":"2024-03-19T02:02:33.568500+0000","last_became_active":"2024-03-19T01:51:16.581746+0000","last_became_peered":"2024-03-19T01:51:16.581746+0000","last_unstale":"2024-03-19T02:02:33.568500+0000","last_undegraded":"2024-03-19T02:02:33.568500+0000","last_fullsized":"2024-03-19T02:02:33.568500+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:19.442287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.594430+0000","last_change":"2024-03-19T01:51:09.195225+0000","last_active":"2024-03-19T02:02:23.594430+0000","last_peered":"2024-03-19T02:02:23.594430+0000","last_clean":"2024-03-19T02:02:23.594430+0000","last_became_active":"2024-03-19T01:51:09.194645+0000","last_became_peered":"2024-03-19T01:51:09.194645+0000","last_unstale":"2024-03-19T02:02:23.594430+0000","last_undegraded":"2024-03-19T02:02:23.594430+0000","last_fullsized":"2024-03-19T02:02:23.594430+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:50:04.024767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"29'5","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:21.549762+0000","last_change":"2024-03-19T01:51:06.955976+0000","last_active":"2024-03-19T02:02:21.549762+0000","last_peered":"2024-03-19T02:02:21.549762+0000","last_clean":"2024-03-19T02:02:21.549762+0000","last_became_active":"2024-03-19T01:51:06.954951+0000","last_became_peered":"2024-03-19T01:51:06.954951+0000","last_unstale":"2024-03-19T02:02:21.549762+0000","last_undegraded":"2024-03-19T02:02:21.549762+0000","last_fullsized":"2024-03-19T02:02:21.549762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:47.536586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:06.661555+0000","last_change":"2024-03-19T01:51:16.586904+0000","last_active":"2024-03-19T02:02:06.661555+0000","last_peered":"2024-03-19T02:02:06.661555+0000","last_clean":"2024-03-19T02:02:06.661555+0000","last_became_active":"2024-03-19T01:51:16.586298+0000","last_became_peered":"2024-03-19T01:51:16.586298+0000","last_unstale":"2024-03-19T02:02:06.661555+0000","last_undegraded":"2024-03-19T02:02:06.661555+0000","last_fullsized":"2024-03-19T02:02:06.661555+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:42.703415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.594405+0000","last_change":"2024-03-19T01:51:09.195400+0000","last_active":"2024-03-19T02:02:23.594405+0000","last_peered":"2024-03-19T02:02:23.594405+0000","last_clean":"2024-03-19T02:02:23.594405+0000","last_became_active":"2024-03-19T01:51:09.195111+0000","last_became_peered":"2024-03-19T01:51:09.195111+0000","last_unstale":"2024-03-19T02:02:23.594405+0000","last_undegraded":"2024-03-19T02:02:23.594405+0000","last_fullsized":"2024-03-19T02:02:23.594405+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:12.084475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"35'2","reported_seq":49,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:40.608950+0000","last_change":"2024-03-19T01:51:06.963650+0000","last_active":"2024-03-19T02:03:40.608950+0000","last_peered":"2024-03-19T02:03:40.608950+0000","last_clean":"2024-03-19T02:03:40.608950+0000","last_became_active":"2024-03-19T01:51:06.963028+0000","last_became_peered":"2024-03-19T01:51:06.963028+0000","last_unstale":"2024-03-19T02:03:40.608950+0000","last_undegraded":"2024-03-19T02:03:40.608950+0000","last_fullsized":"2024-03-19T02:03:40.608950+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:59:26.369856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:08.608117+0000","last_change":"2024-03-19T01:51:16.588077+0000","last_active":"2024-03-19T02:02:08.608117+0000","last_peered":"2024-03-19T02:02:08.608117+0000","last_clean":"2024-03-19T02:02:08.608117+0000","last_became_active":"2024-03-19T01:51:16.587459+0000","last_became_peered":"2024-03-19T01:51:16.587459+0000","last_unstale":"2024-03-19T02:02:08.608117+0000","last_undegraded":"2024-03-19T02:02:08.608117+0000","last_fullsized":"2024-03-19T02:02:08.608117+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:52.667114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.597258+0000","last_change":"2024-03-19T01:51:09.195758+0000","last_active":"2024-03-19T02:02:48.597258+0000","last_peered":"2024-03-19T02:02:48.597258+0000","last_clean":"2024-03-19T02:02:48.597258+0000","last_became_active":"2024-03-19T01:51:09.195139+0000","last_became_peered":"2024-03-19T01:51:09.195139+0000","last_unstale":"2024-03-19T02:02:48.597258+0000","last_undegraded":"2024-03-19T02:02:48.597258+0000","last_fullsized":"2024-03-19T02:02:48.597258+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:18.326414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"29'17","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:46.552835+0000","last_change":"2024-03-19T01:51:06.941200+0000","last_active":"2024-03-19T02:02:46.552835+0000","last_peered":"2024-03-19T02:02:46.552835+0000","last_clean":"2024-03-19T02:02:46.552835+0000","last_became_active":"2024-03-19T01:51:06.940594+0000","last_became_peered":"2024-03-19T01:51:06.940594+0000","last_unstale":"2024-03-19T02:02:46.552835+0000","last_undegraded":"2024-03-19T02:02:46.552835+0000","last_fullsized":"2024-03-19T02:02:46.552835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:25:05.442423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:23.574385+0000","last_change":"2024-03-19T01:51:16.548804+0000","last_active":"2024-03-19T02:03:23.574385+0000","last_peered":"2024-03-19T02:03:23.574385+0000","last_clean":"2024-03-19T02:03:23.574385+0000","last_became_active":"2024-03-19T01:51:16.547800+0000","last_became_peered":"2024-03-19T01:51:16.547800+0000","last_unstale":"2024-03-19T02:03:23.574385+0000","last_undegraded":"2024-03-19T02:03:23.574385+0000","last_fullsized":"2024-03-19T02:03:23.574385+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:43.172796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.499920+0000","last_change":"2024-03-19T01:51:09.167987+0000","last_active":"2024-03-19T02:02:48.499920+0000","last_peered":"2024-03-19T02:02:48.499920+0000","last_clean":"2024-03-19T02:02:48.499920+0000","last_became_active":"2024-03-19T01:51:09.167670+0000","last_became_peered":"2024-03-19T01:51:09.167670+0000","last_unstale":"2024-03-19T02:02:48.499920+0000","last_undegraded":"2024-03-19T02:02:48.499920+0000","last_fullsized":"2024-03-19T02:02:48.499920+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"29'5","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.499949+0000","last_change":"2024-03-19T01:51:07.048197+0000","last_active":"2024-03-19T02:02:48.499949+0000","last_peered":"2024-03-19T02:02:48.499949+0000","last_clean":"2024-03-19T02:02:48.499949+0000","last_became_active":"2024-03-19T01:51:07.047785+0000","last_became_peered":"2024-03-19T01:51:07.047785+0000","last_unstale":"2024-03-19T02:02:48.499949+0000","last_undegraded":"2024-03-19T02:02:48.499949+0000","last_fullsized":"2024-03-19T02:02:48.499949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:52.323249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:21.630885+0000","last_change":"2024-03-19T01:51:16.583394+0000","last_active":"2024-03-19T02:03:21.630885+0000","last_peered":"2024-03-19T02:03:21.630885+0000","last_clean":"2024-03-19T02:03:21.630885+0000","last_became_active":"2024-03-19T01:51:16.582860+0000","last_became_peered":"2024-03-19T01:51:16.582860+0000","last_unstale":"2024-03-19T02:03:21.630885+0000","last_undegraded":"2024-03-19T02:03:21.630885+0000","last_fullsized":"2024-03-19T02:03:21.630885+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:37.669591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.597290+0000","last_change":"2024-03-19T01:51:09.195008+0000","last_active":"2024-03-19T02:02:48.597290+0000","last_peered":"2024-03-19T02:02:48.597290+0000","last_clean":"2024-03-19T02:02:48.597290+0000","last_became_active":"2024-03-19T01:51:09.194728+0000","last_became_peered":"2024-03-19T01:51:09.194728+0000","last_unstale":"2024-03-19T02:02:48.597290+0000","last_undegraded":"2024-03-19T02:02:48.597290+0000","last_fullsized":"2024-03-19T02:02:48.597290+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:15:10.360817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"33'6","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:46.666678+0000","last_change":"2024-03-19T01:51:06.956905+0000","last_active":"2024-03-19T02:02:46.666678+0000","last_peered":"2024-03-19T02:02:46.666678+0000","last_clean":"2024-03-19T02:02:46.666678+0000","last_became_active":"2024-03-19T01:51:06.955991+0000","last_became_peered":"2024-03-19T01:51:06.955991+0000","last_unstale":"2024-03-19T02:02:46.666678+0000","last_undegraded":"2024-03-19T02:02:46.666678+0000","last_fullsized":"2024-03-19T02:02:46.666678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:02:45.970853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":39,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:56.627291+0000","last_change":"2024-03-19T01:52:14.332240+0000","last_active":"2024-03-19T02:02:56.627291+0000","last_peered":"2024-03-19T02:02:56.627291+0000","last_clean":"2024-03-19T02:02:56.627291+0000","last_became_active":"2024-03-19T01:52:14.331292+0000","last_became_peered":"2024-03-19T01:52:14.331292+0000","last_unstale":"2024-03-19T02:02:56.627291+0000","last_undegraded":"2024-03-19T02:02:56.627291+0000","last_fullsized":"2024-03-19T02:02:56.627291+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:14:19.465379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.570250+0000","last_change":"2024-03-19T01:51:09.188003+0000","last_active":"2024-03-19T02:02:48.570250+0000","last_peered":"2024-03-19T02:02:48.570250+0000","last_clean":"2024-03-19T02:02:48.570250+0000","last_became_active":"2024-03-19T01:51:09.187275+0000","last_became_peered":"2024-03-19T01:51:09.187275+0000","last_unstale":"2024-03-19T02:02:48.570250+0000","last_undegraded":"2024-03-19T02:02:48.570250+0000","last_fullsized":"2024-03-19T02:02:48.570250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:10.624259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"29'9","reported_seq":56,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.570286+0000","last_change":"2024-03-19T01:51:06.964413+0000","last_active":"2024-03-19T02:02:48.570286+0000","last_peered":"2024-03-19T02:02:48.570286+0000","last_clean":"2024-03-19T02:02:48.570286+0000","last_became_active":"2024-03-19T01:51:06.963763+0000","last_became_peered":"2024-03-19T01:51:06.963763+0000","last_unstale":"2024-03-19T02:02:48.570286+0000","last_undegraded":"2024-03-19T02:02:48.570286+0000","last_fullsized":"2024-03-19T02:02:48.570286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:21.671644+0000","last_change":"2024-03-19T01:51:16.574401+0000","last_active":"2024-03-19T02:03:21.671644+0000","last_peered":"2024-03-19T02:03:21.671644+0000","last_clean":"2024-03-19T02:03:21.671644+0000","last_became_active":"2024-03-19T01:51:16.573744+0000","last_became_peered":"2024-03-19T01:51:16.573744+0000","last_unstale":"2024-03-19T02:03:21.671644+0000","last_undegraded":"2024-03-19T02:03:21.671644+0000","last_fullsized":"2024-03-19T02:03:21.671644+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:41.414631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.600865+0000","last_change":"2024-03-19T01:51:09.187207+0000","last_active":"2024-03-19T02:03:13.600865+0000","last_peered":"2024-03-19T02:03:13.600865+0000","last_clean":"2024-03-19T02:03:13.600865+0000","last_became_active":"2024-03-19T01:51:09.175898+0000","last_became_peered":"2024-03-19T01:51:09.175898+0000","last_unstale":"2024-03-19T02:03:13.600865+0000","last_undegraded":"2024-03-19T02:03:13.600865+0000","last_fullsized":"2024-03-19T02:03:13.600865+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:38.387069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'14","reported_seq":61,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.499975+0000","last_change":"2024-03-19T01:51:07.042663+0000","last_active":"2024-03-19T02:02:48.499975+0000","last_peered":"2024-03-19T02:02:48.499975+0000","last_clean":"2024-03-19T02:02:48.499975+0000","last_became_active":"2024-03-19T01:51:07.041262+0000","last_became_peered":"2024-03-19T01:51:07.041262+0000","last_unstale":"2024-03-19T02:02:48.499975+0000","last_undegraded":"2024-03-19T02:02:48.499975+0000","last_fullsized":"2024-03-19T02:02:48.499975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:58.513211+0000","last_change":"2024-03-19T01:51:16.549744+0000","last_active":"2024-03-19T02:02:58.513211+0000","last_peered":"2024-03-19T02:02:58.513211+0000","last_clean":"2024-03-19T02:02:58.513211+0000","last_became_active":"2024-03-19T01:51:16.549347+0000","last_became_peered":"2024-03-19T01:51:16.549347+0000","last_unstale":"2024-03-19T02:02:58.513211+0000","last_undegraded":"2024-03-19T02:02:58.513211+0000","last_fullsized":"2024-03-19T02:02:58.513211+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:19.442287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.612688+0000","last_change":"2024-03-19T01:51:09.194165+0000","last_active":"2024-03-19T02:02:48.612688+0000","last_peered":"2024-03-19T02:02:48.612688+0000","last_clean":"2024-03-19T02:02:48.612688+0000","last_became_active":"2024-03-19T01:51:09.183607+0000","last_became_peered":"2024-03-19T01:51:09.183607+0000","last_unstale":"2024-03-19T02:02:48.612688+0000","last_undegraded":"2024-03-19T02:02:48.612688+0000","last_fullsized":"2024-03-19T02:02:48.612688+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:09.211794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'5","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.514982+0000","last_change":"2024-03-19T01:51:06.952515+0000","last_active":"2024-03-19T02:03:13.514982+0000","last_peered":"2024-03-19T02:03:13.514982+0000","last_clean":"2024-03-19T02:03:13.514982+0000","last_became_active":"2024-03-19T01:51:06.951700+0000","last_became_peered":"2024-03-19T01:51:06.951700+0000","last_unstale":"2024-03-19T02:03:13.514982+0000","last_undegraded":"2024-03-19T02:03:13.514982+0000","last_fullsized":"2024-03-19T02:03:13.514982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:13.137673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:58.613767+0000","last_change":"2024-03-19T01:51:16.587027+0000","last_active":"2024-03-19T02:02:58.613767+0000","last_peered":"2024-03-19T02:02:58.613767+0000","last_clean":"2024-03-19T02:02:58.613767+0000","last_became_active":"2024-03-19T01:51:16.586514+0000","last_became_peered":"2024-03-19T01:51:16.586514+0000","last_unstale":"2024-03-19T02:02:58.613767+0000","last_undegraded":"2024-03-19T02:02:58.613767+0000","last_fullsized":"2024-03-19T02:02:58.613767+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:06.637617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.573086+0000","last_change":"2024-03-19T01:51:09.172554+0000","last_active":"2024-03-19T02:03:13.573086+0000","last_peered":"2024-03-19T02:03:13.573086+0000","last_clean":"2024-03-19T02:03:13.573086+0000","last_became_active":"2024-03-19T01:51:09.171814+0000","last_became_peered":"2024-03-19T01:51:09.171814+0000","last_unstale":"2024-03-19T02:03:13.573086+0000","last_undegraded":"2024-03-19T02:03:13.573086+0000","last_fullsized":"2024-03-19T02:03:13.573086+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'4","reported_seq":51,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:11.670336+0000","last_change":"2024-03-19T01:51:07.043404+0000","last_active":"2024-03-19T02:03:11.670336+0000","last_peered":"2024-03-19T02:03:11.670336+0000","last_clean":"2024-03-19T02:03:11.670336+0000","last_became_active":"2024-03-19T01:51:07.043069+0000","last_became_peered":"2024-03-19T01:51:07.043069+0000","last_unstale":"2024-03-19T02:03:11.670336+0000","last_undegraded":"2024-03-19T02:03:11.670336+0000","last_fullsized":"2024-03-19T02:03:11.670336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:13:14.960132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:56.627225+0000","last_change":"2024-03-19T01:51:16.581797+0000","last_active":"2024-03-19T02:02:56.627225+0000","last_peered":"2024-03-19T02:02:56.627225+0000","last_clean":"2024-03-19T02:02:56.627225+0000","last_became_active":"2024-03-19T01:51:16.581174+0000","last_became_peered":"2024-03-19T01:51:16.581174+0000","last_unstale":"2024-03-19T02:02:56.627225+0000","last_undegraded":"2024-03-19T02:02:56.627225+0000","last_fullsized":"2024-03-19T02:02:56.627225+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:29:34.911347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:16.671090+0000","last_change":"2024-03-19T01:51:09.194372+0000","last_active":"2024-03-19T02:03:16.671090+0000","last_peered":"2024-03-19T02:03:16.671090+0000","last_clean":"2024-03-19T02:03:16.671090+0000","last_became_active":"2024-03-19T01:51:09.193656+0000","last_became_peered":"2024-03-19T01:51:09.193656+0000","last_unstale":"2024-03-19T02:03:16.671090+0000","last_undegraded":"2024-03-19T02:03:16.671090+0000","last_fullsized":"2024-03-19T02:03:16.671090+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:23.464803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'4","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.502733+0000","last_change":"2024-03-19T01:51:07.046089+0000","last_active":"2024-03-19T02:03:13.502733+0000","last_peered":"2024-03-19T02:03:13.502733+0000","last_clean":"2024-03-19T02:03:13.502733+0000","last_became_active":"2024-03-19T01:51:07.045342+0000","last_became_peered":"2024-03-19T01:51:07.045342+0000","last_unstale":"2024-03-19T02:03:13.502733+0000","last_undegraded":"2024-03-19T02:03:13.502733+0000","last_fullsized":"2024-03-19T02:03:13.502733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:26.105915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:56.627262+0000","last_change":"2024-03-19T01:51:16.572559+0000","last_active":"2024-03-19T02:02:56.627262+0000","last_peered":"2024-03-19T02:02:56.627262+0000","last_clean":"2024-03-19T02:02:56.627262+0000","last_became_active":"2024-03-19T01:51:16.572283+0000","last_became_peered":"2024-03-19T01:51:16.572283+0000","last_unstale":"2024-03-19T02:02:56.627262+0000","last_undegraded":"2024-03-19T02:02:56.627262+0000","last_fullsized":"2024-03-19T02:02:56.627262+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:42.703415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:18.573781+0000","last_change":"2024-03-19T01:51:16.534088+0000","last_active":"2024-03-19T02:03:18.573781+0000","last_peered":"2024-03-19T02:03:18.573781+0000","last_clean":"2024-03-19T02:03:18.573781+0000","last_became_active":"2024-03-19T01:51:16.533665+0000","last_became_peered":"2024-03-19T01:51:16.533665+0000","last_unstale":"2024-03-19T02:03:18.573781+0000","last_undegraded":"2024-03-19T02:03:18.573781+0000","last_fullsized":"2024-03-19T02:03:18.573781+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:35.409504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.514931+0000","last_change":"2024-03-19T01:51:09.168372+0000","last_active":"2024-03-19T02:03:13.514931+0000","last_peered":"2024-03-19T02:03:13.514931+0000","last_clean":"2024-03-19T02:03:13.514931+0000","last_became_active":"2024-03-19T01:51:09.167626+0000","last_became_peered":"2024-03-19T01:51:09.167626+0000","last_unstale":"2024-03-19T02:03:13.514931+0000","last_undegraded":"2024-03-19T02:03:13.514931+0000","last_fullsized":"2024-03-19T02:03:13.514931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"33'4","reported_seq":51,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:11.670233+0000","last_change":"2024-03-19T01:51:06.951225+0000","last_active":"2024-03-19T02:03:11.670233+0000","last_peered":"2024-03-19T02:03:11.670233+0000","last_clean":"2024-03-19T02:03:11.670233+0000","last_became_active":"2024-03-19T01:51:06.950826+0000","last_became_peered":"2024-03-19T01:51:06.950826+0000","last_unstale":"2024-03-19T02:03:11.670233+0000","last_undegraded":"2024-03-19T02:03:11.670233+0000","last_fullsized":"2024-03-19T02:03:11.670233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:24.949925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.507160+0000","last_change":"2024-03-19T01:51:16.581997+0000","last_active":"2024-03-19T02:03:48.507160+0000","last_peered":"2024-03-19T02:03:48.507160+0000","last_clean":"2024-03-19T02:03:48.507160+0000","last_became_active":"2024-03-19T01:51:16.581412+0000","last_became_peered":"2024-03-19T01:51:16.581412+0000","last_unstale":"2024-03-19T02:03:48.507160+0000","last_undegraded":"2024-03-19T02:03:48.507160+0000","last_fullsized":"2024-03-19T02:03:48.507160+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:45.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:16.630092+0000","last_change":"2024-03-19T01:51:09.161686+0000","last_active":"2024-03-19T02:03:16.630092+0000","last_peered":"2024-03-19T02:03:16.630092+0000","last_clean":"2024-03-19T02:03:16.630092+0000","last_became_active":"2024-03-19T01:51:09.160978+0000","last_became_peered":"2024-03-19T01:51:09.160978+0000","last_unstale":"2024-03-19T02:03:16.630092+0000","last_undegraded":"2024-03-19T02:03:16.630092+0000","last_fullsized":"2024-03-19T02:03:16.630092+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:27.266807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'3","reported_seq":50,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:11.629313+0000","last_change":"2024-03-19T01:51:07.047363+0000","last_active":"2024-03-19T02:03:11.629313+0000","last_peered":"2024-03-19T02:03:11.629313+0000","last_clean":"2024-03-19T02:03:11.629313+0000","last_became_active":"2024-03-19T01:51:07.046886+0000","last_became_peered":"2024-03-19T01:51:07.046886+0000","last_unstale":"2024-03-19T02:03:11.629313+0000","last_undegraded":"2024-03-19T02:03:11.629313+0000","last_fullsized":"2024-03-19T02:03:11.629313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:19.418603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"29'7","reported_seq":54,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.505874+0000","last_change":"2024-03-19T01:51:07.045496+0000","last_active":"2024-03-19T02:03:38.505874+0000","last_peered":"2024-03-19T02:03:38.505874+0000","last_clean":"2024-03-19T02:03:38.505874+0000","last_became_active":"2024-03-19T01:51:07.044554+0000","last_became_peered":"2024-03-19T01:51:07.044554+0000","last_unstale":"2024-03-19T02:03:38.505874+0000","last_undegraded":"2024-03-19T02:03:38.505874+0000","last_fullsized":"2024-03-19T02:03:38.505874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:14.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.576001+0000","last_change":"2024-03-19T01:51:09.184934+0000","last_active":"2024-03-19T02:03:38.576001+0000","last_peered":"2024-03-19T02:03:38.576001+0000","last_clean":"2024-03-19T02:03:38.576001+0000","last_became_active":"2024-03-19T01:51:09.184666+0000","last_became_peered":"2024-03-19T01:51:09.184666+0000","last_unstale":"2024-03-19T02:03:38.576001+0000","last_undegraded":"2024-03-19T02:03:38.576001+0000","last_fullsized":"2024-03-19T02:03:38.576001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:42.786323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":38,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:16.540961+0000","last_change":"2024-03-19T01:52:14.298105+0000","last_active":"2024-03-19T02:01:16.540961+0000","last_peered":"2024-03-19T02:01:16.540961+0000","last_clean":"2024-03-19T02:01:16.540961+0000","last_became_active":"2024-03-19T01:52:14.297612+0000","last_became_peered":"2024-03-19T01:52:14.297612+0000","last_unstale":"2024-03-19T02:01:16.540961+0000","last_undegraded":"2024-03-19T02:01:16.540961+0000","last_fullsized":"2024-03-19T02:01:16.540961+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:38.753646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,5],"acting":[2,0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"33'7","reported_seq":57,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:36.632647+0000","last_change":"2024-03-19T01:51:06.982280+0000","last_active":"2024-03-19T02:03:36.632647+0000","last_peered":"2024-03-19T02:03:36.632647+0000","last_clean":"2024-03-19T02:03:36.632647+0000","last_became_active":"2024-03-19T01:51:06.981704+0000","last_became_peered":"2024-03-19T01:51:06.981704+0000","last_unstale":"2024-03-19T02:03:36.632647+0000","last_undegraded":"2024-03-19T02:03:36.632647+0000","last_fullsized":"2024-03-19T02:03:36.632647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:22:12.657985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.618444+0000","last_change":"2024-03-19T01:51:09.184624+0000","last_active":"2024-03-19T02:03:38.618444+0000","last_peered":"2024-03-19T02:03:38.618444+0000","last_clean":"2024-03-19T02:03:38.618444+0000","last_became_active":"2024-03-19T01:51:09.183976+0000","last_became_peered":"2024-03-19T01:51:09.183976+0000","last_unstale":"2024-03-19T02:03:38.618444+0000","last_undegraded":"2024-03-19T02:03:38.618444+0000","last_fullsized":"2024-03-19T02:03:38.618444+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:24:29.313729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":38,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:16.614098+0000","last_change":"2024-03-19T01:52:14.299182+0000","last_active":"2024-03-19T02:01:16.614098+0000","last_peered":"2024-03-19T02:01:16.614098+0000","last_clean":"2024-03-19T02:01:16.614098+0000","last_became_active":"2024-03-19T01:52:14.297976+0000","last_became_peered":"2024-03-19T01:52:14.297976+0000","last_unstale":"2024-03-19T02:01:16.614098+0000","last_undegraded":"2024-03-19T02:01:16.614098+0000","last_fullsized":"2024-03-19T02:01:16.614098+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:31.057362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4,2],"acting":[3,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.505794+0000","last_change":"2024-03-19T01:51:09.169493+0000","last_active":"2024-03-19T02:03:38.505794+0000","last_peered":"2024-03-19T02:03:38.505794+0000","last_clean":"2024-03-19T02:03:38.505794+0000","last_became_active":"2024-03-19T01:51:09.168231+0000","last_became_peered":"2024-03-19T01:51:09.168231+0000","last_unstale":"2024-03-19T02:03:38.505794+0000","last_undegraded":"2024-03-19T02:03:38.505794+0000","last_fullsized":"2024-03-19T02:03:38.505794+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:10.624259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:41.559618+0000","last_change":"2024-03-19T01:51:09.155483+0000","last_active":"2024-03-19T02:03:41.559618+0000","last_peered":"2024-03-19T02:03:41.559618+0000","last_clean":"2024-03-19T02:03:41.559618+0000","last_became_active":"2024-03-19T01:51:09.154545+0000","last_became_peered":"2024-03-19T01:51:09.154545+0000","last_unstale":"2024-03-19T02:03:41.559618+0000","last_undegraded":"2024-03-19T02:03:41.559618+0000","last_fullsized":"2024-03-19T02:03:41.559618+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:25:09.517323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:41.673722+0000","last_change":"2024-03-19T01:51:09.180498+0000","last_active":"2024-03-19T02:03:41.673722+0000","last_peered":"2024-03-19T02:03:41.673722+0000","last_clean":"2024-03-19T02:03:41.673722+0000","last_became_active":"2024-03-19T01:51:09.179840+0000","last_became_peered":"2024-03-19T01:51:09.179840+0000","last_unstale":"2024-03-19T02:03:41.673722+0000","last_undegraded":"2024-03-19T02:03:41.673722+0000","last_fullsized":"2024-03-19T02:03:41.673722+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:01:21.716571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.575976+0000","last_change":"2024-03-19T01:51:09.170389+0000","last_active":"2024-03-19T02:03:38.575976+0000","last_peered":"2024-03-19T02:03:38.575976+0000","last_clean":"2024-03-19T02:03:38.575976+0000","last_became_active":"2024-03-19T01:51:09.163974+0000","last_became_peered":"2024-03-19T01:51:09.163974+0000","last_unstale":"2024-03-19T02:03:38.575976+0000","last_undegraded":"2024-03-19T02:03:38.575976+0000","last_fullsized":"2024-03-19T02:03:38.575976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:01.620620+0000","last_change":"2024-03-19T01:51:09.177060+0000","last_active":"2024-03-19T02:02:01.620620+0000","last_peered":"2024-03-19T02:02:01.620620+0000","last_clean":"2024-03-19T02:02:01.620620+0000","last_became_active":"2024-03-19T01:51:09.176474+0000","last_became_peered":"2024-03-19T01:51:09.176474+0000","last_unstale":"2024-03-19T02:02:01.620620+0000","last_undegraded":"2024-03-19T02:02:01.620620+0000","last_fullsized":"2024-03-19T02:02:01.620620+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:56.318137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.504665+0000","last_change":"2024-03-19T01:51:09.168741+0000","last_active":"2024-03-19T02:01:58.504665+0000","last_peered":"2024-03-19T02:01:58.504665+0000","last_clean":"2024-03-19T02:01:58.504665+0000","last_became_active":"2024-03-19T01:51:09.167982+0000","last_became_peered":"2024-03-19T01:51:09.167982+0000","last_unstale":"2024-03-19T02:01:58.504665+0000","last_undegraded":"2024-03-19T02:01:58.504665+0000","last_fullsized":"2024-03-19T02:01:58.504665+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.494217+0000","last_change":"2024-03-19T01:51:09.189814+0000","last_active":"2024-03-19T02:01:58.494217+0000","last_peered":"2024-03-19T02:01:58.494217+0000","last_clean":"2024-03-19T02:01:58.494217+0000","last_became_active":"2024-03-19T01:51:09.189128+0000","last_became_peered":"2024-03-19T01:51:09.189128+0000","last_unstale":"2024-03-19T02:01:58.494217+0000","last_undegraded":"2024-03-19T02:01:58.494217+0000","last_fullsized":"2024-03-19T02:01:58.494217+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:01.620637+0000","last_change":"2024-03-19T01:51:09.183989+0000","last_active":"2024-03-19T02:02:01.620637+0000","last_peered":"2024-03-19T02:02:01.620637+0000","last_clean":"2024-03-19T02:02:01.620637+0000","last_became_active":"2024-03-19T01:51:09.183417+0000","last_became_peered":"2024-03-19T01:51:09.183417+0000","last_unstale":"2024-03-19T02:02:01.620637+0000","last_undegraded":"2024-03-19T02:02:01.620637+0000","last_fullsized":"2024-03-19T02:02:01.620637+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:31.405618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:01.660628+0000","last_change":"2024-03-19T01:51:09.172717+0000","last_active":"2024-03-19T02:02:01.660628+0000","last_peered":"2024-03-19T02:02:01.660628+0000","last_clean":"2024-03-19T02:02:01.660628+0000","last_became_active":"2024-03-19T01:51:09.165872+0000","last_became_peered":"2024-03-19T01:51:09.165872+0000","last_unstale":"2024-03-19T02:02:01.660628+0000","last_undegraded":"2024-03-19T02:02:01.660628+0000","last_fullsized":"2024-03-19T02:02:01.660628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:10.995492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:26.557659+0000","last_change":"2024-03-19T01:52:12.916894+0000","last_active":"2024-03-19T02:03:26.557659+0000","last_peered":"2024-03-19T02:03:26.557659+0000","last_clean":"2024-03-19T02:03:26.557659+0000","last_became_active":"2024-03-19T01:52:12.916608+0000","last_became_peered":"2024-03-19T01:52:12.916608+0000","last_unstale":"2024-03-19T02:03:26.557659+0000","last_undegraded":"2024-03-19T02:03:26.557659+0000","last_fullsized":"2024-03-19T02:03:26.557659+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:31.405618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:26.664130+0000","last_change":"2024-03-19T01:51:09.195141+0000","last_active":"2024-03-19T02:02:26.664130+0000","last_peered":"2024-03-19T02:02:26.664130+0000","last_clean":"2024-03-19T02:02:26.664130+0000","last_became_active":"2024-03-19T01:51:09.194536+0000","last_became_peered":"2024-03-19T01:51:09.194536+0000","last_unstale":"2024-03-19T02:02:26.664130+0000","last_undegraded":"2024-03-19T02:02:26.664130+0000","last_fullsized":"2024-03-19T02:02:26.664130+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:31.395833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.567024+0000","last_change":"2024-03-19T01:51:09.189197+0000","last_active":"2024-03-19T02:02:23.567024+0000","last_peered":"2024-03-19T02:02:23.567024+0000","last_clean":"2024-03-19T02:02:23.567024+0000","last_became_active":"2024-03-19T01:51:09.188565+0000","last_became_peered":"2024-03-19T01:51:09.188565+0000","last_unstale":"2024-03-19T02:02:23.567024+0000","last_undegraded":"2024-03-19T02:02:23.567024+0000","last_fullsized":"2024-03-19T02:02:23.567024+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.497100+0000","last_change":"2024-03-19T01:51:09.168581+0000","last_active":"2024-03-19T02:02:23.497100+0000","last_peered":"2024-03-19T02:02:23.497100+0000","last_clean":"2024-03-19T02:02:23.497100+0000","last_became_active":"2024-03-19T01:51:09.167812+0000","last_became_peered":"2024-03-19T01:51:09.167812+0000","last_unstale":"2024-03-19T02:02:23.497100+0000","last_undegraded":"2024-03-19T02:02:23.497100+0000","last_fullsized":"2024-03-19T02:02:23.497100+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:28.517089+0000","last_change":"2024-03-19T01:52:12.915401+0000","last_active":"2024-03-19T02:03:28.517089+0000","last_peered":"2024-03-19T02:03:28.517089+0000","last_clean":"2024-03-19T02:03:28.517089+0000","last_became_active":"2024-03-19T01:52:12.915089+0000","last_became_peered":"2024-03-19T01:52:12.915089+0000","last_unstale":"2024-03-19T02:03:28.517089+0000","last_undegraded":"2024-03-19T02:03:28.517089+0000","last_fullsized":"2024-03-19T02:03:28.517089+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:54:01.116865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:51.553572+0000","last_change":"2024-03-19T01:51:09.168451+0000","last_active":"2024-03-19T02:02:51.553572+0000","last_peered":"2024-03-19T02:02:51.553572+0000","last_clean":"2024-03-19T02:02:51.553572+0000","last_became_active":"2024-03-19T01:51:09.167744+0000","last_became_peered":"2024-03-19T01:51:09.167744+0000","last_unstale":"2024-03-19T02:02:51.553572+0000","last_undegraded":"2024-03-19T02:02:51.553572+0000","last_fullsized":"2024-03-19T02:02:51.553572+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:21.735485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:51.667789+0000","last_change":"2024-03-19T01:51:09.196067+0000","last_active":"2024-03-19T02:02:51.667789+0000","last_peered":"2024-03-19T02:02:51.667789+0000","last_clean":"2024-03-19T02:02:51.667789+0000","last_became_active":"2024-03-19T01:51:09.195488+0000","last_became_peered":"2024-03-19T01:51:09.195488+0000","last_unstale":"2024-03-19T02:02:51.667789+0000","last_undegraded":"2024-03-19T02:02:51.667789+0000","last_fullsized":"2024-03-19T02:02:51.667789+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:07:44.120161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:51.667650+0000","last_change":"2024-03-19T01:51:09.193831+0000","last_active":"2024-03-19T02:02:51.667650+0000","last_peered":"2024-03-19T02:02:51.667650+0000","last_clean":"2024-03-19T02:02:51.667650+0000","last_became_active":"2024-03-19T01:51:09.192777+0000","last_became_peered":"2024-03-19T01:51:09.192777+0000","last_unstale":"2024-03-19T02:02:51.667650+0000","last_undegraded":"2024-03-19T02:02:51.667650+0000","last_fullsized":"2024-03-19T02:02:51.667650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:56.318137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.511885+0000","last_change":"2024-03-19T01:51:09.193581+0000","last_active":"2024-03-19T02:02:48.511885+0000","last_peered":"2024-03-19T02:02:48.511885+0000","last_clean":"2024-03-19T02:02:48.511885+0000","last_became_active":"2024-03-19T01:51:09.192677+0000","last_became_peered":"2024-03-19T01:51:09.192677+0000","last_unstale":"2024-03-19T02:02:48.511885+0000","last_undegraded":"2024-03-19T02:02:48.511885+0000","last_fullsized":"2024-03-19T02:02:48.511885+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.570213+0000","last_change":"2024-03-19T01:51:09.165568+0000","last_active":"2024-03-19T02:02:48.570213+0000","last_peered":"2024-03-19T02:02:48.570213+0000","last_clean":"2024-03-19T02:02:48.570213+0000","last_became_active":"2024-03-19T01:51:09.164753+0000","last_became_peered":"2024-03-19T01:51:09.164753+0000","last_unstale":"2024-03-19T02:02:48.570213+0000","last_undegraded":"2024-03-19T02:02:48.570213+0000","last_fullsized":"2024-03-19T02:02:48.570213+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:28.422798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:16.630124+0000","last_change":"2024-03-19T01:51:09.170898+0000","last_active":"2024-03-19T02:03:16.630124+0000","last_peered":"2024-03-19T02:03:16.630124+0000","last_clean":"2024-03-19T02:03:16.630124+0000","last_became_active":"2024-03-19T01:51:09.170153+0000","last_became_peered":"2024-03-19T01:51:09.170153+0000","last_unstale":"2024-03-19T02:03:16.630124+0000","last_undegraded":"2024-03-19T02:03:16.630124+0000","last_fullsized":"2024-03-19T02:03:16.630124+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:41.125800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.615856+0000","last_change":"2024-03-19T01:51:09.185953+0000","last_active":"2024-03-19T02:03:13.615856+0000","last_peered":"2024-03-19T02:03:13.615856+0000","last_clean":"2024-03-19T02:03:13.615856+0000","last_became_active":"2024-03-19T01:51:09.177996+0000","last_became_peered":"2024-03-19T01:51:09.177996+0000","last_unstale":"2024-03-19T02:03:13.615856+0000","last_undegraded":"2024-03-19T02:03:13.615856+0000","last_fullsized":"2024-03-19T02:03:13.615856+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:39:06.045413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.502685+0000","last_change":"2024-03-19T01:51:09.183841+0000","last_active":"2024-03-19T02:03:13.502685+0000","last_peered":"2024-03-19T02:03:13.502685+0000","last_clean":"2024-03-19T02:03:13.502685+0000","last_became_active":"2024-03-19T01:51:09.183045+0000","last_became_peered":"2024-03-19T01:51:09.183045+0000","last_unstale":"2024-03-19T02:03:13.502685+0000","last_undegraded":"2024-03-19T02:03:13.502685+0000","last_fullsized":"2024-03-19T02:03:13.502685+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.502713+0000","last_change":"2024-03-19T01:51:09.184022+0000","last_active":"2024-03-19T02:03:13.502713+0000","last_peered":"2024-03-19T02:03:13.502713+0000","last_clean":"2024-03-19T02:03:13.502713+0000","last_became_active":"2024-03-19T01:51:09.183351+0000","last_became_peered":"2024-03-19T01:51:09.183351+0000","last_unstale":"2024-03-19T02:03:13.502713+0000","last_undegraded":"2024-03-19T02:03:13.502713+0000","last_fullsized":"2024-03-19T02:03:13.502713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.600945+0000","last_change":"2024-03-19T01:52:12.922818+0000","last_active":"2024-03-19T02:03:13.600945+0000","last_peered":"2024-03-19T02:03:13.600945+0000","last_clean":"2024-03-19T02:03:13.600945+0000","last_became_active":"2024-03-19T01:52:12.922473+0000","last_became_peered":"2024-03-19T01:52:12.922473+0000","last_unstale":"2024-03-19T02:03:13.600945+0000","last_undegraded":"2024-03-19T02:03:13.600945+0000","last_fullsized":"2024-03-19T02:03:13.600945+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:35.798061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:11.613507+0000","last_change":"2024-03-19T01:51:09.180013+0000","last_active":"2024-03-19T02:01:11.613507+0000","last_peered":"2024-03-19T02:01:11.613507+0000","last_clean":"2024-03-19T02:01:11.613507+0000","last_became_active":"2024-03-19T01:51:09.179541+0000","last_became_peered":"2024-03-19T01:51:09.179541+0000","last_unstale":"2024-03-19T02:01:11.613507+0000","last_undegraded":"2024-03-19T02:01:11.613507+0000","last_fullsized":"2024-03-19T02:01:11.613507+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:19:25.933028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603886+0000","last_change":"2024-03-19T01:51:09.188671+0000","last_active":"2024-03-19T02:03:38.603886+0000","last_peered":"2024-03-19T02:03:38.603886+0000","last_clean":"2024-03-19T02:03:38.603886+0000","last_became_active":"2024-03-19T01:51:09.177627+0000","last_became_peered":"2024-03-19T01:51:09.177627+0000","last_unstale":"2024-03-19T02:03:38.603886+0000","last_undegraded":"2024-03-19T02:03:38.603886+0000","last_fullsized":"2024-03-19T02:03:38.603886+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:53:50.539837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:11.654275+0000","last_change":"2024-03-19T01:51:09.180735+0000","last_active":"2024-03-19T02:01:11.654275+0000","last_peered":"2024-03-19T02:01:11.654275+0000","last_clean":"2024-03-19T02:01:11.654275+0000","last_became_active":"2024-03-19T01:51:09.173235+0000","last_became_peered":"2024-03-19T01:51:09.173235+0000","last_unstale":"2024-03-19T02:01:11.654275+0000","last_undegraded":"2024-03-19T02:01:11.654275+0000","last_fullsized":"2024-03-19T02:01:11.654275+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:31.405618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603918+0000","last_change":"2024-03-19T01:51:09.188346+0000","last_active":"2024-03-19T02:03:38.603918+0000","last_peered":"2024-03-19T02:03:38.603918+0000","last_clean":"2024-03-19T02:03:38.603918+0000","last_became_active":"2024-03-19T01:51:09.187703+0000","last_became_peered":"2024-03-19T01:51:09.187703+0000","last_unstale":"2024-03-19T02:03:38.603918+0000","last_undegraded":"2024-03-19T02:03:38.603918+0000","last_fullsized":"2024-03-19T02:03:38.603918+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:28.051691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:11.548313+0000","last_change":"2024-03-19T01:52:12.919846+0000","last_active":"2024-03-19T02:02:11.548313+0000","last_peered":"2024-03-19T02:02:11.548313+0000","last_clean":"2024-03-19T02:02:11.548313+0000","last_became_active":"2024-03-19T01:52:12.918835+0000","last_became_peered":"2024-03-19T01:52:12.918835+0000","last_unstale":"2024-03-19T02:02:11.548313+0000","last_undegraded":"2024-03-19T02:02:11.548313+0000","last_fullsized":"2024-03-19T02:02:11.548313+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:03:04.997711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.576027+0000","last_change":"2024-03-19T01:51:09.161476+0000","last_active":"2024-03-19T02:03:38.576027+0000","last_peered":"2024-03-19T02:03:38.576027+0000","last_clean":"2024-03-19T02:03:38.576027+0000","last_became_active":"2024-03-19T01:51:09.158609+0000","last_became_peered":"2024-03-19T01:51:09.158609+0000","last_unstale":"2024-03-19T02:03:38.576027+0000","last_undegraded":"2024-03-19T02:03:38.576027+0000","last_fullsized":"2024-03-19T02:03:38.576027+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.518711+0000","last_change":"2024-03-19T01:51:09.165938+0000","last_active":"2024-03-19T02:03:38.518711+0000","last_peered":"2024-03-19T02:03:38.518711+0000","last_clean":"2024-03-19T02:03:38.518711+0000","last_became_active":"2024-03-19T01:51:09.165207+0000","last_became_peered":"2024-03-19T01:51:09.165207+0000","last_unstale":"2024-03-19T02:03:38.518711+0000","last_undegraded":"2024-03-19T02:03:38.518711+0000","last_fullsized":"2024-03-19T02:03:38.518711+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603999+0000","last_change":"2024-03-19T01:51:09.176023+0000","last_active":"2024-03-19T02:03:38.603999+0000","last_peered":"2024-03-19T02:03:38.603999+0000","last_clean":"2024-03-19T02:03:38.603999+0000","last_became_active":"2024-03-19T01:51:09.175682+0000","last_became_peered":"2024-03-19T01:51:09.175682+0000","last_unstale":"2024-03-19T02:03:38.603999+0000","last_undegraded":"2024-03-19T02:03:38.603999+0000","last_fullsized":"2024-03-19T02:03:38.603999+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:35.814983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":209,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":189,"ondisk_log_size":189,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509633,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343828,"statfs":{"total":96636764160,"available":96608079872,"internally_reserved":0,"allocated":1011712,"data_stored":759262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509635,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":171426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509634,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":168124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509635,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31580,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":409600,"data_stored":168326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509634,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":1003520,"data_stored":758036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542337,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":430080,"data_stored":170726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542338,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31588,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340252,"statfs":{"total":96636764160,"available":96604418048,"internally_reserved":0,"allocated":413696,"data_stored":171460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542337,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31588,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340252,"statfs":{"total":96636764160,"available":96604418048,"internally_reserved":0,"allocated":413696,"data_stored":168256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:06:10.806 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:06:10.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.949+0000 7fb547a49640 1 Processor -- start 2024-03-19T02:06:10.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.949+0000 7fb547a49640 1 -- start start 2024-03-19T02:06:10.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.949+0000 7fb547a49640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb5400cbf30 0x7fb5400cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:10.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.949+0000 7fb547a49640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fb54006c6f0 0x7fb5400cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:10.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.949+0000 7fb547a49640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb54006cb90 con 0x7fb54015a3b0 2024-03-19T02:06:10.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.949+0000 7fb547a49640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7fb5400ce640 con 0x7fb54006c6f0 2024-03-19T02:06:10.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.949+0000 7fb547a49640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fb5400cda30 con 0x7fb5400cbf30 2024-03-19T02:06:10.961 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.961+0000 7fb544fe4640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fb54006c6f0 0x7fb5400cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:10.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.961+0000 7fb544fe4640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7fb54006c6f0 0x7fb5400cf010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:51516/0 (socket says 172.21.15.55:51516) 2024-03-19T02:06:10.962 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.961+0000 7fb544fe4640 1 -- 172.21.15.55:0/4114360751 learned_addr learned my addr 172.21.15.55:0/4114360751 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:10.967 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.965+0000 7fb544fe4640 1 -- 172.21.15.55:0/4114360751 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb5400cbf30 msgr2=0x7fb5400cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:10.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.965+0000 7fb544fe4640 1 --2- 172.21.15.55:0/4114360751 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb5400cbf30 0x7fb5400cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:10.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.965+0000 7fb544fe4640 1 -- 172.21.15.55:0/4114360751 >> v1:172.21.15.84:6789/0 conn(0x7fb54015a3b0 legacy=0x7fb54015a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:10.968 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.965+0000 7fb544fe4640 1 -- 172.21.15.55:0/4114360751 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5400cdd10 con 0x7fb54006c6f0 2024-03-19T02:06:10.976 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.973+0000 7fb544fe4640 1 --2- 172.21.15.55:0/4114360751 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb54006c6f0 0x7fb5400cf010 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb530009870 tx=0x7fb53001c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=722c175310aff459 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:10.984 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.981+0000 7fb537fff640 1 -- 172.21.15.55:0/4114360751 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5300b1070 con 0x7fb54006c6f0 2024-03-19T02:06:10.984 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.981+0000 7fb537fff640 1 -- 172.21.15.55:0/4114360751 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5300a8050 con 0x7fb54006c6f0 2024-03-19T02:06:10.984 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.981+0000 7fb537fff640 1 -- 172.21.15.55:0/4114360751 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5300ac860 con 0x7fb54006c6f0 2024-03-19T02:06:10.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.985+0000 7fb547a49640 1 -- 172.21.15.55:0/4114360751 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb54006c6f0 msgr2=0x7fb5400cf010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:10.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.985+0000 7fb547a49640 1 --2- 172.21.15.55:0/4114360751 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb54006c6f0 0x7fb5400cf010 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb530009870 tx=0x7fb53001c4d0 comp rx=0 tx=0).stop 2024-03-19T02:06:10.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.985+0000 7fb547a49640 1 -- 172.21.15.55:0/4114360751 shutdown_connections 2024-03-19T02:06:10.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.985+0000 7fb547a49640 1 --2- 172.21.15.55:0/4114360751 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb54006c6f0 0x7fb5400cf010 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:10.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.985+0000 7fb547a49640 1 --2- 172.21.15.55:0/4114360751 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb5400cbf30 0x7fb5400cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:10.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.985+0000 7fb547a49640 1 -- 172.21.15.55:0/4114360751 >> 172.21.15.55:0/4114360751 conn(0x7fb5400b16a0 msgr2=0x7fb5400b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:10.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.985+0000 7fb547a49640 1 -- 172.21.15.55:0/4114360751 shutdown_connections 2024-03-19T02:06:10.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.985+0000 7fb547a49640 1 -- 172.21.15.55:0/4114360751 wait complete. 2024-03-19T02:06:10.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.985+0000 7fb547a49640 1 Processor -- start 2024-03-19T02:06:10.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.985+0000 7fb547a49640 1 -- start start 2024-03-19T02:06:10.988 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.989+0000 7fb547a49640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb54006c6f0 0x7fb5400c8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:10.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.989+0000 7fb547a49640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb5400cbf30 0x7fb5401f3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:10.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.989+0000 7fb547a49640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb54015a3b0 0x7fb5401f6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:10.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.989+0000 7fb547a49640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb5400cc310 con 0x7fb54015a3b0 2024-03-19T02:06:10.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.989+0000 7fb547a49640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb5400b23b0 con 0x7fb5400cbf30 2024-03-19T02:06:10.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.989+0000 7fb547a49640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fb5400cc490 con 0x7fb54006c6f0 2024-03-19T02:06:10.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.989+0000 7fb544fe4640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb5400cbf30 0x7fb5401f3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:10.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.989+0000 7fb544fe4640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb5400cbf30 0x7fb5401f3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:51518/0 (socket says 172.21.15.55:51518) 2024-03-19T02:06:10.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.989+0000 7fb544fe4640 1 -- 172.21.15.55:0/3380773489 learned_addr learned my addr 172.21.15.55:0/3380773489 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:10.997 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.997+0000 7fb544fe4640 1 -- 172.21.15.55:0/3380773489 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb54006c6f0 msgr2=0x7fb5400c8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:10.997 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.997+0000 7fb544fe4640 1 --2- 172.21.15.55:0/3380773489 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb54006c6f0 0x7fb5400c8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:10.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.997+0000 7fb544fe4640 1 -- 172.21.15.55:0/3380773489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb54015a3b0 msgr2=0x7fb5401f6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:10.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.997+0000 7fb544fe4640 1 --2- 172.21.15.55:0/3380773489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb54015a3b0 0x7fb5401f6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:10.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:10.997+0000 7fb544fe4640 1 -- 172.21.15.55:0/3380773489 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb54015b510 con 0x7fb5400cbf30 2024-03-19T02:06:11.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.001+0000 7fb544fe4640 1 --2- 172.21.15.55:0/3380773489 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb5400cbf30 0x7fb5401f3ad0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb53001ab80 tx=0x7fb53001ad10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:11.015 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.013+0000 7fb535ffb640 1 -- 172.21.15.55:0/3380773489 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5300bb020 con 0x7fb5400cbf30 2024-03-19T02:06:11.015 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.013+0000 7fb535ffb640 1 -- 172.21.15.55:0/3380773489 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5300b7420 con 0x7fb5400cbf30 2024-03-19T02:06:11.015 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.013+0000 7fb535ffb640 1 -- 172.21.15.55:0/3380773489 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5300b1040 con 0x7fb5400cbf30 2024-03-19T02:06:11.015 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.013+0000 7fb547a49640 1 -- 172.21.15.55:0/3380773489 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb540153140 con 0x7fb5400cbf30 2024-03-19T02:06:11.015 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.013+0000 7fb547a49640 1 -- 172.21.15.55:0/3380773489 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb54015b7b0 con 0x7fb5400cbf30 2024-03-19T02:06:11.016 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.017+0000 7fb547a49640 1 -- 172.21.15.55:0/3380773489 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb540068330 con 0x7fb5400cbf30 2024-03-19T02:06:11.282 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:06:11.285 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:06:11.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.401+0000 7fb535ffb640 1 -- 172.21.15.55:0/3380773489 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb5300be020 con 0x7fb5400cbf30 2024-03-19T02:06:11.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.405+0000 7fb535ffb640 1 --2- 172.21.15.55:0/3380773489 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb524067660 0x7fb524069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:11.405 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.405+0000 7fb5457e5640 1 --2- 172.21.15.55:0/3380773489 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb524067660 0x7fb524069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:11.406 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.405+0000 7fb5457e5640 1 --2- 172.21.15.55:0/3380773489 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb524067660 0x7fb524069b20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb53c009690 tx=0x7fb53c006c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:11.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.413+0000 7fb535ffb640 1 -- 172.21.15.55:0/3380773489 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fb5300b6e40 con 0x7fb5400cbf30 2024-03-19T02:06:11.431 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.429+0000 7fb535ffb640 1 -- 172.21.15.55:0/3380773489 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fb5300c5030 con 0x7fb5400cbf30 2024-03-19T02:06:11.626 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.625+0000 7fb547a49640 1 -- 172.21.15.55:0/3380773489 --> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb5400b2190 con 0x7fb524067660 2024-03-19T02:06:11.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.641+0000 7fb535ffb640 1 -- 172.21.15.55:0/3380773489 <== mgr.4105 v2:172.21.15.84:6800/998533783 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332628 (secure 0 0 0) 0x7fb5400b2190 con 0x7fb524067660 2024-03-19T02:06:11.647 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:06:11.649 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-19T02:06:11.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.645+0000 7fb547a49640 1 -- 172.21.15.55:0/3380773489 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb524067660 msgr2=0x7fb524069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:11.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.645+0000 7fb547a49640 1 --2- 172.21.15.55:0/3380773489 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb524067660 0x7fb524069b20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb53c009690 tx=0x7fb53c006c00 comp rx=0 tx=0).stop 2024-03-19T02:06:11.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.645+0000 7fb547a49640 1 -- 172.21.15.55:0/3380773489 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb5400cbf30 msgr2=0x7fb5401f3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:11.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.645+0000 7fb547a49640 1 --2- 172.21.15.55:0/3380773489 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb5400cbf30 0x7fb5401f3ad0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb53001ab80 tx=0x7fb53001ad10 comp rx=0 tx=0).stop 2024-03-19T02:06:11.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.645+0000 7fb547a49640 1 -- 172.21.15.55:0/3380773489 shutdown_connections 2024-03-19T02:06:11.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.645+0000 7fb547a49640 1 --2- 172.21.15.55:0/3380773489 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb524067660 0x7fb524069b20 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:11.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.645+0000 7fb547a49640 1 --2- 172.21.15.55:0/3380773489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb54015a3b0 0x7fb5401f6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:11.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.645+0000 7fb547a49640 1 --2- 172.21.15.55:0/3380773489 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb5400cbf30 0x7fb5401f3ad0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:11.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.645+0000 7fb547a49640 1 --2- 172.21.15.55:0/3380773489 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb54006c6f0 0x7fb5400c8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:11.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.645+0000 7fb547a49640 1 -- 172.21.15.55:0/3380773489 >> 172.21.15.55:0/3380773489 conn(0x7fb5400b16a0 msgr2=0x7fb540150d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:11.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.645+0000 7fb547a49640 1 -- 172.21.15.55:0/3380773489 shutdown_connections 2024-03-19T02:06:11.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.645+0000 7fb547a49640 1 -- 172.21.15.55:0/3380773489 wait complete. 2024-03-19T02:06:11.663 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-03-19T02:06:10.603886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":368,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":306,"ondisk_log_size":306,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237212,"kb_used_data":4392,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737508,"statfs":{"total":773094113280,"available":772851208192,"internally_reserved":0,"allocated":4497408,"data_stored":2535616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004437"},"pg_stats":[{"pgid":"2.1f","version":"29'13","reported_seq":61,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.576125+0000","last_change":"2024-03-19T01:51:06.982533+0000","last_active":"2024-03-19T02:03:38.576125+0000","last_peered":"2024-03-19T02:03:38.576125+0000","last_clean":"2024-03-19T02:03:38.576125+0000","last_became_active":"2024-03-19T01:51:06.981802+0000","last_became_peered":"2024-03-19T01:51:06.981802+0000","last_unstale":"2024-03-19T02:03:38.576125+0000","last_undegraded":"2024-03-19T02:03:38.576125+0000","last_fullsized":"2024-03-19T02:03:38.576125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:10:01.431003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:11.629408+0000","last_change":"2024-03-19T01:51:09.172057+0000","last_active":"2024-03-19T02:03:11.629408+0000","last_peered":"2024-03-19T02:03:11.629408+0000","last_clean":"2024-03-19T02:03:11.629408+0000","last_became_active":"2024-03-19T01:51:09.171753+0000","last_became_peered":"2024-03-19T01:51:09.171753+0000","last_unstale":"2024-03-19T02:03:11.629408+0000","last_undegraded":"2024-03-19T02:03:11.629408+0000","last_fullsized":"2024-03-19T02:03:11.629408+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:50.575804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:21.630796+0000","last_change":"2024-03-19T01:51:16.537753+0000","last_active":"2024-03-19T02:03:21.630796+0000","last_peered":"2024-03-19T02:03:21.630796+0000","last_clean":"2024-03-19T02:03:21.630796+0000","last_became_active":"2024-03-19T01:51:16.534200+0000","last_became_peered":"2024-03-19T01:51:16.534200+0000","last_unstale":"2024-03-19T02:03:21.630796+0000","last_undegraded":"2024-03-19T02:03:21.630796+0000","last_fullsized":"2024-03-19T02:03:21.630796+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:18:34.208854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":50,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:06.628579+0000","last_change":"2024-03-19T01:51:06.962968+0000","last_active":"2024-03-19T02:03:06.628579+0000","last_peered":"2024-03-19T02:03:06.628579+0000","last_clean":"2024-03-19T02:03:06.628579+0000","last_became_active":"2024-03-19T01:51:06.961856+0000","last_became_peered":"2024-03-19T01:51:06.961856+0000","last_unstale":"2024-03-19T02:03:06.628579+0000","last_undegraded":"2024-03-19T02:03:06.628579+0000","last_fullsized":"2024-03-19T02:03:06.628579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:49.676651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.576098+0000","last_change":"2024-03-19T01:51:09.160338+0000","last_active":"2024-03-19T02:03:38.576098+0000","last_peered":"2024-03-19T02:03:38.576098+0000","last_clean":"2024-03-19T02:03:38.576098+0000","last_became_active":"2024-03-19T01:51:09.158682+0000","last_became_peered":"2024-03-19T01:51:09.158682+0000","last_unstale":"2024-03-19T02:03:38.576098+0000","last_undegraded":"2024-03-19T02:03:38.576098+0000","last_fullsized":"2024-03-19T02:03:38.576098+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":39,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:23.616915+0000","last_change":"2024-03-19T01:52:14.299568+0000","last_active":"2024-03-19T02:03:23.616915+0000","last_peered":"2024-03-19T02:03:23.616915+0000","last_clean":"2024-03-19T02:03:23.616915+0000","last_became_active":"2024-03-19T01:52:14.299067+0000","last_became_peered":"2024-03-19T01:52:14.299067+0000","last_unstale":"2024-03-19T02:03:23.616915+0000","last_undegraded":"2024-03-19T02:03:23.616915+0000","last_fullsized":"2024-03-19T02:03:23.616915+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:15:10.967874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"35'12","reported_seq":59,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603963+0000","last_change":"2024-03-19T01:51:07.044049+0000","last_active":"2024-03-19T02:03:38.603963+0000","last_peered":"2024-03-19T02:03:38.603963+0000","last_clean":"2024-03-19T02:03:38.603963+0000","last_became_active":"2024-03-19T01:51:07.043209+0000","last_became_peered":"2024-03-19T01:51:07.043209+0000","last_unstale":"2024-03-19T02:03:38.603963+0000","last_undegraded":"2024-03-19T02:03:38.603963+0000","last_fullsized":"2024-03-19T02:03:38.603963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:42.992419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:11.654315+0000","last_change":"2024-03-19T01:51:09.179615+0000","last_active":"2024-03-19T02:01:11.654315+0000","last_peered":"2024-03-19T02:01:11.654315+0000","last_clean":"2024-03-19T02:01:11.654315+0000","last_became_active":"2024-03-19T01:51:09.173962+0000","last_became_peered":"2024-03-19T01:51:09.173962+0000","last_unstale":"2024-03-19T02:01:11.654315+0000","last_undegraded":"2024-03-19T02:01:11.654315+0000","last_fullsized":"2024-03-19T02:01:11.654315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:06:12.676868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":38,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:18.601453+0000","last_change":"2024-03-19T01:53:15.447584+0000","last_active":"2024-03-19T02:01:18.601453+0000","last_peered":"2024-03-19T02:01:18.601453+0000","last_clean":"2024-03-19T02:01:18.601453+0000","last_became_active":"2024-03-19T01:53:15.447002+0000","last_became_peered":"2024-03-19T01:53:15.447002+0000","last_unstale":"2024-03-19T02:01:18.601453+0000","last_undegraded":"2024-03-19T02:01:18.601453+0000","last_fullsized":"2024-03-19T02:01:18.601453+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:02:57.393473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6,1],"acting":[4,2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"29'5","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603940+0000","last_change":"2024-03-19T01:51:06.983291+0000","last_active":"2024-03-19T02:03:38.603940+0000","last_peered":"2024-03-19T02:03:38.603940+0000","last_clean":"2024-03-19T02:03:38.603940+0000","last_became_active":"2024-03-19T01:51:06.982309+0000","last_became_peered":"2024-03-19T01:51:06.982309+0000","last_unstale":"2024-03-19T02:03:38.603940+0000","last_undegraded":"2024-03-19T02:03:38.603940+0000","last_fullsized":"2024-03-19T02:03:38.603940+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:27.305286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:11.654335+0000","last_change":"2024-03-19T01:51:09.160738+0000","last_active":"2024-03-19T02:01:11.654335+0000","last_peered":"2024-03-19T02:01:11.654335+0000","last_clean":"2024-03-19T02:01:11.654335+0000","last_became_active":"2024-03-19T01:51:09.159951+0000","last_became_peered":"2024-03-19T01:51:09.159951+0000","last_unstale":"2024-03-19T02:01:11.654335+0000","last_undegraded":"2024-03-19T02:01:11.654335+0000","last_fullsized":"2024-03-19T02:01:11.654335+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:30.620041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":31,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:18.601421+0000","last_change":"2024-03-19T01:51:16.577378+0000","last_active":"2024-03-19T02:01:18.601421+0000","last_peered":"2024-03-19T02:01:18.601421+0000","last_clean":"2024-03-19T02:01:18.601421+0000","last_became_active":"2024-03-19T01:51:16.576256+0000","last_became_peered":"2024-03-19T01:51:16.576256+0000","last_unstale":"2024-03-19T02:01:18.601421+0000","last_undegraded":"2024-03-19T02:01:18.601421+0000","last_fullsized":"2024-03-19T02:01:18.601421+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:16.096547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.494171+0000","last_change":"2024-03-19T01:51:09.185969+0000","last_active":"2024-03-19T02:01:58.494171+0000","last_peered":"2024-03-19T02:01:58.494171+0000","last_clean":"2024-03-19T02:01:58.494171+0000","last_became_active":"2024-03-19T01:51:09.185378+0000","last_became_peered":"2024-03-19T01:51:09.185378+0000","last_unstale":"2024-03-19T02:01:58.494171+0000","last_undegraded":"2024-03-19T02:01:58.494171+0000","last_fullsized":"2024-03-19T02:01:58.494171+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":51,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.594329+0000","last_change":"2024-03-19T01:51:06.988067+0000","last_active":"2024-03-19T02:02:23.594329+0000","last_peered":"2024-03-19T02:02:23.594329+0000","last_clean":"2024-03-19T02:02:23.594329+0000","last_became_active":"2024-03-19T01:51:06.987450+0000","last_became_peered":"2024-03-19T01:51:06.987450+0000","last_unstale":"2024-03-19T02:02:23.594329+0000","last_undegraded":"2024-03-19T02:02:23.594329+0000","last_fullsized":"2024-03-19T02:02:23.594329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:56.962922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:33.610729+0000","last_change":"2024-03-19T01:51:16.583190+0000","last_active":"2024-03-19T02:02:33.610729+0000","last_peered":"2024-03-19T02:02:33.610729+0000","last_clean":"2024-03-19T02:02:33.610729+0000","last_became_active":"2024-03-19T01:51:16.582532+0000","last_became_peered":"2024-03-19T01:51:16.582532+0000","last_unstale":"2024-03-19T02:02:33.610729+0000","last_undegraded":"2024-03-19T02:02:33.610729+0000","last_fullsized":"2024-03-19T02:02:33.610729+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:16:49.817940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:26.624005+0000","last_change":"2024-03-19T01:51:09.174687+0000","last_active":"2024-03-19T02:02:26.624005+0000","last_peered":"2024-03-19T02:02:26.624005+0000","last_clean":"2024-03-19T02:02:26.624005+0000","last_became_active":"2024-03-19T01:51:09.174409+0000","last_became_peered":"2024-03-19T01:51:09.174409+0000","last_unstale":"2024-03-19T02:02:26.624005+0000","last_undegraded":"2024-03-19T02:02:26.624005+0000","last_fullsized":"2024-03-19T02:02:26.624005+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:33:07.160350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"35'3","reported_seq":51,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.504638+0000","last_change":"2024-03-19T01:51:06.955434+0000","last_active":"2024-03-19T02:01:58.504638+0000","last_peered":"2024-03-19T02:01:58.504638+0000","last_clean":"2024-03-19T02:01:58.504638+0000","last_became_active":"2024-03-19T01:51:06.954635+0000","last_became_peered":"2024-03-19T01:51:06.954635+0000","last_unstale":"2024-03-19T02:01:58.504638+0000","last_undegraded":"2024-03-19T02:01:58.504638+0000","last_fullsized":"2024-03-19T02:01:58.504638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:41:39.176905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:33.610667+0000","last_change":"2024-03-19T01:51:16.588604+0000","last_active":"2024-03-19T02:02:33.610667+0000","last_peered":"2024-03-19T02:02:33.610667+0000","last_clean":"2024-03-19T02:02:33.610667+0000","last_became_active":"2024-03-19T01:51:16.587483+0000","last_became_peered":"2024-03-19T01:51:16.587483+0000","last_unstale":"2024-03-19T02:02:33.610667+0000","last_undegraded":"2024-03-19T02:02:33.610667+0000","last_fullsized":"2024-03-19T02:02:33.610667+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:51.658547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:01.620586+0000","last_change":"2024-03-19T01:51:09.173987+0000","last_active":"2024-03-19T02:02:01.620586+0000","last_peered":"2024-03-19T02:02:01.620586+0000","last_clean":"2024-03-19T02:02:01.620586+0000","last_became_active":"2024-03-19T01:51:09.173648+0000","last_became_peered":"2024-03-19T01:51:09.173648+0000","last_unstale":"2024-03-19T02:02:01.620586+0000","last_undegraded":"2024-03-19T02:02:01.620586+0000","last_fullsized":"2024-03-19T02:02:01.620586+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:13:17.277014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"29'7","reported_seq":55,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.504602+0000","last_change":"2024-03-19T01:51:06.981820+0000","last_active":"2024-03-19T02:01:58.504602+0000","last_peered":"2024-03-19T02:01:58.504602+0000","last_clean":"2024-03-19T02:01:58.504602+0000","last_became_active":"2024-03-19T01:51:06.981087+0000","last_became_peered":"2024-03-19T01:51:06.981087+0000","last_unstale":"2024-03-19T02:01:58.504602+0000","last_undegraded":"2024-03-19T02:01:58.504602+0000","last_fullsized":"2024-03-19T02:01:58.504602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:25:59.116138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:33.509725+0000","last_change":"2024-03-19T01:51:16.579017+0000","last_active":"2024-03-19T02:02:33.509725+0000","last_peered":"2024-03-19T02:02:33.509725+0000","last_clean":"2024-03-19T02:02:33.509725+0000","last_became_active":"2024-03-19T01:51:16.573845+0000","last_became_peered":"2024-03-19T01:51:16.573845+0000","last_unstale":"2024-03-19T02:02:33.509725+0000","last_undegraded":"2024-03-19T02:02:33.509725+0000","last_fullsized":"2024-03-19T02:02:33.509725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:42:30.142766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.607044+0000","last_change":"2024-03-19T01:51:09.192886+0000","last_active":"2024-03-19T02:01:58.607044+0000","last_peered":"2024-03-19T02:01:58.607044+0000","last_clean":"2024-03-19T02:01:58.607044+0000","last_became_active":"2024-03-19T01:51:09.185970+0000","last_became_peered":"2024-03-19T01:51:09.185970+0000","last_unstale":"2024-03-19T02:01:58.607044+0000","last_undegraded":"2024-03-19T02:01:58.607044+0000","last_fullsized":"2024-03-19T02:01:58.607044+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:05.196985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":49,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.591524+0000","last_change":"2024-03-19T01:51:06.984875+0000","last_active":"2024-03-19T02:01:58.591524+0000","last_peered":"2024-03-19T02:01:58.591524+0000","last_clean":"2024-03-19T02:01:58.591524+0000","last_became_active":"2024-03-19T01:51:06.983858+0000","last_became_peered":"2024-03-19T01:51:06.983858+0000","last_unstale":"2024-03-19T02:01:58.591524+0000","last_undegraded":"2024-03-19T02:01:58.591524+0000","last_fullsized":"2024-03-19T02:01:58.591524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:14.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:33.610746+0000","last_change":"2024-03-19T01:51:16.591614+0000","last_active":"2024-03-19T02:02:33.610746+0000","last_peered":"2024-03-19T02:02:33.610746+0000","last_clean":"2024-03-19T02:02:33.610746+0000","last_became_active":"2024-03-19T01:51:16.590968+0000","last_became_peered":"2024-03-19T01:51:16.590968+0000","last_unstale":"2024-03-19T02:02:33.610746+0000","last_undegraded":"2024-03-19T02:02:33.610746+0000","last_fullsized":"2024-03-19T02:02:33.610746+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:50.144136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.564107+0000","last_change":"2024-03-19T01:51:09.168941+0000","last_active":"2024-03-19T02:01:58.564107+0000","last_peered":"2024-03-19T02:01:58.564107+0000","last_clean":"2024-03-19T02:01:58.564107+0000","last_became_active":"2024-03-19T01:51:09.168243+0000","last_became_peered":"2024-03-19T01:51:09.168243+0000","last_unstale":"2024-03-19T02:01:58.564107+0000","last_undegraded":"2024-03-19T02:01:58.564107+0000","last_fullsized":"2024-03-19T02:01:58.564107+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'7","reported_seq":54,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.494241+0000","last_change":"2024-03-19T01:51:07.042870+0000","last_active":"2024-03-19T02:01:58.494241+0000","last_peered":"2024-03-19T02:01:58.494241+0000","last_clean":"2024-03-19T02:01:58.494241+0000","last_became_active":"2024-03-19T01:51:07.042327+0000","last_became_peered":"2024-03-19T01:51:07.042327+0000","last_unstale":"2024-03-19T02:01:58.494241+0000","last_undegraded":"2024-03-19T02:01:58.494241+0000","last_fullsized":"2024-03-19T02:01:58.494241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:13.137673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.619433+0000","last_change":"2024-03-19T01:51:16.587730+0000","last_active":"2024-03-19T02:03:48.619433+0000","last_peered":"2024-03-19T02:03:48.619433+0000","last_clean":"2024-03-19T02:03:48.619433+0000","last_became_active":"2024-03-19T01:51:16.586614+0000","last_became_peered":"2024-03-19T01:51:16.586614+0000","last_unstale":"2024-03-19T02:03:48.619433+0000","last_undegraded":"2024-03-19T02:03:48.619433+0000","last_fullsized":"2024-03-19T02:03:48.619433+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:44.830637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:01.620647+0000","last_change":"2024-03-19T01:51:09.186397+0000","last_active":"2024-03-19T02:02:01.620647+0000","last_peered":"2024-03-19T02:02:01.620647+0000","last_clean":"2024-03-19T02:02:01.620647+0000","last_became_active":"2024-03-19T01:51:09.186033+0000","last_became_peered":"2024-03-19T01:51:09.186033+0000","last_unstale":"2024-03-19T02:02:01.620647+0000","last_undegraded":"2024-03-19T02:02:01.620647+0000","last_fullsized":"2024-03-19T02:02:01.620647+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:03:04.997711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"29'7","reported_seq":54,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.504687+0000","last_change":"2024-03-19T01:51:07.043697+0000","last_active":"2024-03-19T02:01:58.504687+0000","last_peered":"2024-03-19T02:01:58.504687+0000","last_clean":"2024-03-19T02:01:58.504687+0000","last_became_active":"2024-03-19T01:51:07.043185+0000","last_became_peered":"2024-03-19T01:51:07.043185+0000","last_unstale":"2024-03-19T02:01:58.504687+0000","last_undegraded":"2024-03-19T02:01:58.504687+0000","last_fullsized":"2024-03-19T02:01:58.504687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:19.016579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:16.630056+0000","last_change":"2024-03-19T01:51:16.535771+0000","last_active":"2024-03-19T02:03:16.630056+0000","last_peered":"2024-03-19T02:03:16.630056+0000","last_clean":"2024-03-19T02:03:16.630056+0000","last_became_active":"2024-03-19T01:51:16.533661+0000","last_became_peered":"2024-03-19T01:51:16.533661+0000","last_unstale":"2024-03-19T02:03:16.630056+0000","last_undegraded":"2024-03-19T02:03:16.630056+0000","last_fullsized":"2024-03-19T02:03:16.630056+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:55.283782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.518547+0000","last_change":"2024-03-19T01:51:09.157526+0000","last_active":"2024-03-19T02:03:38.518547+0000","last_peered":"2024-03-19T02:03:38.518547+0000","last_clean":"2024-03-19T02:03:38.518547+0000","last_became_active":"2024-03-19T01:51:09.156336+0000","last_became_peered":"2024-03-19T01:51:09.156336+0000","last_unstale":"2024-03-19T02:03:38.518547+0000","last_undegraded":"2024-03-19T02:03:38.518547+0000","last_fullsized":"2024-03-19T02:03:38.518547+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'4","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603980+0000","last_change":"2024-03-19T01:51:06.982994+0000","last_active":"2024-03-19T02:03:38.603980+0000","last_peered":"2024-03-19T02:03:38.603980+0000","last_clean":"2024-03-19T02:03:38.603980+0000","last_became_active":"2024-03-19T01:51:06.982133+0000","last_became_peered":"2024-03-19T01:51:06.982133+0000","last_unstale":"2024-03-19T02:03:38.603980+0000","last_undegraded":"2024-03-19T02:03:38.603980+0000","last_fullsized":"2024-03-19T02:03:38.603980+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:03.574098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.577343+0000","last_change":"2024-03-19T01:51:16.578493+0000","last_active":"2024-03-19T02:03:48.577343+0000","last_peered":"2024-03-19T02:03:48.577343+0000","last_clean":"2024-03-19T02:03:48.577343+0000","last_became_active":"2024-03-19T01:51:16.578040+0000","last_became_peered":"2024-03-19T01:51:16.578040+0000","last_unstale":"2024-03-19T02:03:48.577343+0000","last_undegraded":"2024-03-19T02:03:48.577343+0000","last_fullsized":"2024-03-19T02:03:48.577343+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:45.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:41.673779+0000","last_change":"2024-03-19T01:51:09.180758+0000","last_active":"2024-03-19T02:03:41.673779+0000","last_peered":"2024-03-19T02:03:41.673779+0000","last_clean":"2024-03-19T02:03:41.673779+0000","last_became_active":"2024-03-19T01:51:09.180326+0000","last_became_peered":"2024-03-19T01:51:09.180326+0000","last_unstale":"2024-03-19T02:03:41.673779+0000","last_undegraded":"2024-03-19T02:03:41.673779+0000","last_fullsized":"2024-03-19T02:03:41.673779+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:56:07.046285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"29'1","reported_seq":49,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.518603+0000","last_change":"2024-03-19T01:51:06.962643+0000","last_active":"2024-03-19T02:03:38.518603+0000","last_peered":"2024-03-19T02:03:38.518603+0000","last_clean":"2024-03-19T02:03:38.518603+0000","last_became_active":"2024-03-19T01:51:06.961862+0000","last_became_peered":"2024-03-19T01:51:06.961862+0000","last_unstale":"2024-03-19T02:03:38.518603+0000","last_undegraded":"2024-03-19T02:03:38.518603+0000","last_fullsized":"2024-03-19T02:03:38.518603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.520313+0000","last_change":"2024-03-19T01:51:16.577730+0000","last_active":"2024-03-19T02:03:48.520313+0000","last_peered":"2024-03-19T02:03:48.520313+0000","last_clean":"2024-03-19T02:03:48.520313+0000","last_became_active":"2024-03-19T01:51:16.576955+0000","last_became_peered":"2024-03-19T01:51:16.576955+0000","last_unstale":"2024-03-19T02:03:48.520313+0000","last_undegraded":"2024-03-19T02:03:48.520313+0000","last_fullsized":"2024-03-19T02:03:48.520313+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:35.409504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.618478+0000","last_change":"2024-03-19T01:51:09.183678+0000","last_active":"2024-03-19T02:03:38.618478+0000","last_peered":"2024-03-19T02:03:38.618478+0000","last_clean":"2024-03-19T02:03:38.618478+0000","last_became_active":"2024-03-19T01:51:09.176780+0000","last_became_peered":"2024-03-19T01:51:09.176780+0000","last_unstale":"2024-03-19T02:03:38.618478+0000","last_undegraded":"2024-03-19T02:03:38.618478+0000","last_fullsized":"2024-03-19T02:03:38.618478+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:11:02.502333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'5","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:36.673161+0000","last_change":"2024-03-19T01:51:06.956307+0000","last_active":"2024-03-19T02:03:36.673161+0000","last_peered":"2024-03-19T02:03:36.673161+0000","last_clean":"2024-03-19T02:03:36.673161+0000","last_became_active":"2024-03-19T01:51:06.955529+0000","last_became_peered":"2024-03-19T01:51:06.955529+0000","last_unstale":"2024-03-19T02:03:36.673161+0000","last_undegraded":"2024-03-19T02:03:36.673161+0000","last_fullsized":"2024-03-19T02:03:36.673161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:10.486012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.520349+0000","last_change":"2024-03-19T01:51:16.550718+0000","last_active":"2024-03-19T02:03:48.520349+0000","last_peered":"2024-03-19T02:03:48.520349+0000","last_clean":"2024-03-19T02:03:48.520349+0000","last_became_active":"2024-03-19T01:51:16.550115+0000","last_became_peered":"2024-03-19T01:51:16.550115+0000","last_unstale":"2024-03-19T02:03:48.520349+0000","last_undegraded":"2024-03-19T02:03:48.520349+0000","last_fullsized":"2024-03-19T02:03:48.520349+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:43.172796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:08.514405+0000","last_change":"2024-03-19T01:51:09.155718+0000","last_active":"2024-03-19T02:03:08.514405+0000","last_peered":"2024-03-19T02:03:08.514405+0000","last_clean":"2024-03-19T02:03:08.514405+0000","last_became_active":"2024-03-19T01:51:09.155014+0000","last_became_peered":"2024-03-19T01:51:09.155014+0000","last_unstale":"2024-03-19T02:03:08.514405+0000","last_undegraded":"2024-03-19T02:03:08.514405+0000","last_fullsized":"2024-03-19T02:03:08.514405+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"33'6","reported_seq":54,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:36.559085+0000","last_change":"2024-03-19T01:51:06.955372+0000","last_active":"2024-03-19T02:03:36.559085+0000","last_peered":"2024-03-19T02:03:36.559085+0000","last_clean":"2024-03-19T02:03:36.559085+0000","last_became_active":"2024-03-19T01:51:06.954637+0000","last_became_peered":"2024-03-19T01:51:06.954637+0000","last_unstale":"2024-03-19T02:03:36.559085+0000","last_undegraded":"2024-03-19T02:03:36.559085+0000","last_fullsized":"2024-03-19T02:03:36.559085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:38:20.129564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:08.505884+0000","last_change":"2024-03-19T01:51:16.575157+0000","last_active":"2024-03-19T02:02:08.505884+0000","last_peered":"2024-03-19T02:02:08.505884+0000","last_clean":"2024-03-19T02:02:08.505884+0000","last_became_active":"2024-03-19T01:51:16.574575+0000","last_became_peered":"2024-03-19T01:51:16.574575+0000","last_unstale":"2024-03-19T02:02:08.505884+0000","last_undegraded":"2024-03-19T02:02:08.505884+0000","last_fullsized":"2024-03-19T02:02:08.505884+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:15:17.718630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.575915+0000","last_change":"2024-03-19T01:51:09.192610+0000","last_active":"2024-03-19T02:03:38.575915+0000","last_peered":"2024-03-19T02:03:38.575915+0000","last_clean":"2024-03-19T02:03:38.575915+0000","last_became_active":"2024-03-19T01:51:09.192280+0000","last_became_peered":"2024-03-19T01:51:09.192280+0000","last_unstale":"2024-03-19T02:03:38.575915+0000","last_undegraded":"2024-03-19T02:03:38.575915+0000","last_fullsized":"2024-03-19T02:03:38.575915+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:15.454555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"33'5","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:08.600010+0000","last_change":"2024-03-19T01:51:06.986895+0000","last_active":"2024-03-19T02:03:08.600010+0000","last_peered":"2024-03-19T02:03:08.600010+0000","last_clean":"2024-03-19T02:03:08.600010+0000","last_became_active":"2024-03-19T01:51:06.985965+0000","last_became_peered":"2024-03-19T01:51:06.985965+0000","last_unstale":"2024-03-19T02:03:08.600010+0000","last_undegraded":"2024-03-19T02:03:08.600010+0000","last_fullsized":"2024-03-19T02:03:08.600010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:08.565447+0000","last_change":"2024-03-19T01:51:16.505278+0000","last_active":"2024-03-19T02:02:08.565447+0000","last_peered":"2024-03-19T02:02:08.565447+0000","last_clean":"2024-03-19T02:02:08.565447+0000","last_became_active":"2024-03-19T01:51:16.504908+0000","last_became_peered":"2024-03-19T01:51:16.504908+0000","last_unstale":"2024-03-19T02:02:08.565447+0000","last_undegraded":"2024-03-19T02:02:08.565447+0000","last_fullsized":"2024-03-19T02:02:08.565447+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:42:30.142766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:41.633422+0000","last_change":"2024-03-19T01:51:09.179725+0000","last_active":"2024-03-19T02:03:41.633422+0000","last_peered":"2024-03-19T02:03:41.633422+0000","last_clean":"2024-03-19T02:03:41.633422+0000","last_became_active":"2024-03-19T01:51:09.179235+0000","last_became_peered":"2024-03-19T01:51:09.179235+0000","last_unstale":"2024-03-19T02:03:41.633422+0000","last_undegraded":"2024-03-19T02:03:41.633422+0000","last_fullsized":"2024-03-19T02:03:41.633422+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:31.395833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'6","reported_seq":55,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:36.673191+0000","last_change":"2024-03-19T01:51:06.953474+0000","last_active":"2024-03-19T02:03:36.673191+0000","last_peered":"2024-03-19T02:03:36.673191+0000","last_clean":"2024-03-19T02:03:36.673191+0000","last_became_active":"2024-03-19T01:51:06.952898+0000","last_became_peered":"2024-03-19T01:51:06.952898+0000","last_unstale":"2024-03-19T02:03:36.673191+0000","last_undegraded":"2024-03-19T02:03:36.673191+0000","last_fullsized":"2024-03-19T02:03:36.673191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:38.808918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":203,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:28.594947+0000","last_change":"2024-03-19T01:50:26.097484+0000","last_active":"2024-03-19T02:02:28.594947+0000","last_peered":"2024-03-19T02:02:28.594947+0000","last_clean":"2024-03-19T02:02:28.594947+0000","last_became_active":"2024-03-19T01:50:26.095496+0000","last_became_peered":"2024-03-19T01:50:26.095496+0000","last_unstale":"2024-03-19T02:02:28.594947+0000","last_undegraded":"2024-03-19T02:02:28.594947+0000","last_fullsized":"2024-03-19T02:02:28.594947+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:50:24.112070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:50:24.112070+0000","last_clean_scrub_stamp":"2024-03-19T01:50:24.112070+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:05:00.843775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":40,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.507121+0000","last_change":"2024-03-19T01:52:14.333085+0000","last_active":"2024-03-19T02:03:48.507121+0000","last_peered":"2024-03-19T02:03:48.507121+0000","last_clean":"2024-03-19T02:03:48.507121+0000","last_became_active":"2024-03-19T01:52:14.332547+0000","last_became_peered":"2024-03-19T01:52:14.332547+0000","last_unstale":"2024-03-19T02:03:48.507121+0000","last_undegraded":"2024-03-19T02:03:48.507121+0000","last_fullsized":"2024-03-19T02:03:48.507121+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:36.377719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4,7],"acting":[6,0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.594377+0000","last_change":"2024-03-19T01:51:09.196648+0000","last_active":"2024-03-19T02:02:23.594377+0000","last_peered":"2024-03-19T02:02:23.594377+0000","last_clean":"2024-03-19T02:02:23.594377+0000","last_became_active":"2024-03-19T01:51:09.196093+0000","last_became_peered":"2024-03-19T01:51:09.196093+0000","last_unstale":"2024-03-19T02:02:23.594377+0000","last_undegraded":"2024-03-19T02:02:23.594377+0000","last_fullsized":"2024-03-19T02:02:23.594377+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:54:09.655994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"29'5","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:21.549717+0000","last_change":"2024-03-19T01:51:06.950023+0000","last_active":"2024-03-19T02:02:21.549717+0000","last_peered":"2024-03-19T02:02:21.549717+0000","last_clean":"2024-03-19T02:02:21.549717+0000","last_became_active":"2024-03-19T01:51:06.949234+0000","last_became_peered":"2024-03-19T01:51:06.949234+0000","last_unstale":"2024-03-19T02:02:21.549717+0000","last_undegraded":"2024-03-19T02:02:21.549717+0000","last_fullsized":"2024-03-19T02:02:21.549717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:54:13.131325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:08.495157+0000","last_change":"2024-03-19T01:51:16.554046+0000","last_active":"2024-03-19T02:02:08.495157+0000","last_peered":"2024-03-19T02:02:08.495157+0000","last_clean":"2024-03-19T02:02:08.495157+0000","last_became_active":"2024-03-19T01:51:16.553595+0000","last_became_peered":"2024-03-19T01:51:16.553595+0000","last_unstale":"2024-03-19T02:02:08.495157+0000","last_undegraded":"2024-03-19T02:02:08.495157+0000","last_fullsized":"2024-03-19T02:02:08.495157+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:25.684226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:26.557693+0000","last_change":"2024-03-19T01:52:12.918926+0000","last_active":"2024-03-19T02:03:26.557693+0000","last_peered":"2024-03-19T02:03:26.557693+0000","last_clean":"2024-03-19T02:03:26.557693+0000","last_became_active":"2024-03-19T01:52:12.918458+0000","last_became_peered":"2024-03-19T01:52:12.918458+0000","last_unstale":"2024-03-19T02:03:26.557693+0000","last_undegraded":"2024-03-19T02:03:26.557693+0000","last_fullsized":"2024-03-19T02:03:26.557693+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:56.318137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"29'3","reported_seq":50,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.508282+0000","last_change":"2024-03-19T01:51:06.938396+0000","last_active":"2024-03-19T02:02:23.508282+0000","last_peered":"2024-03-19T02:02:23.508282+0000","last_clean":"2024-03-19T02:02:23.508282+0000","last_became_active":"2024-03-19T01:51:06.937580+0000","last_became_peered":"2024-03-19T01:51:06.937580+0000","last_unstale":"2024-03-19T02:02:23.508282+0000","last_undegraded":"2024-03-19T02:02:23.508282+0000","last_fullsized":"2024-03-19T02:02:23.508282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:08.307376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:33.568500+0000","last_change":"2024-03-19T01:51:16.582111+0000","last_active":"2024-03-19T02:02:33.568500+0000","last_peered":"2024-03-19T02:02:33.568500+0000","last_clean":"2024-03-19T02:02:33.568500+0000","last_became_active":"2024-03-19T01:51:16.581746+0000","last_became_peered":"2024-03-19T01:51:16.581746+0000","last_unstale":"2024-03-19T02:02:33.568500+0000","last_undegraded":"2024-03-19T02:02:33.568500+0000","last_fullsized":"2024-03-19T02:02:33.568500+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:19.442287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.594430+0000","last_change":"2024-03-19T01:51:09.195225+0000","last_active":"2024-03-19T02:02:23.594430+0000","last_peered":"2024-03-19T02:02:23.594430+0000","last_clean":"2024-03-19T02:02:23.594430+0000","last_became_active":"2024-03-19T01:51:09.194645+0000","last_became_peered":"2024-03-19T01:51:09.194645+0000","last_unstale":"2024-03-19T02:02:23.594430+0000","last_undegraded":"2024-03-19T02:02:23.594430+0000","last_fullsized":"2024-03-19T02:02:23.594430+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:50:04.024767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"29'5","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:21.549762+0000","last_change":"2024-03-19T01:51:06.955976+0000","last_active":"2024-03-19T02:02:21.549762+0000","last_peered":"2024-03-19T02:02:21.549762+0000","last_clean":"2024-03-19T02:02:21.549762+0000","last_became_active":"2024-03-19T01:51:06.954951+0000","last_became_peered":"2024-03-19T01:51:06.954951+0000","last_unstale":"2024-03-19T02:02:21.549762+0000","last_undegraded":"2024-03-19T02:02:21.549762+0000","last_fullsized":"2024-03-19T02:02:21.549762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:47.536586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:06.661555+0000","last_change":"2024-03-19T01:51:16.586904+0000","last_active":"2024-03-19T02:02:06.661555+0000","last_peered":"2024-03-19T02:02:06.661555+0000","last_clean":"2024-03-19T02:02:06.661555+0000","last_became_active":"2024-03-19T01:51:16.586298+0000","last_became_peered":"2024-03-19T01:51:16.586298+0000","last_unstale":"2024-03-19T02:02:06.661555+0000","last_undegraded":"2024-03-19T02:02:06.661555+0000","last_fullsized":"2024-03-19T02:02:06.661555+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:42.703415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.594405+0000","last_change":"2024-03-19T01:51:09.195400+0000","last_active":"2024-03-19T02:02:23.594405+0000","last_peered":"2024-03-19T02:02:23.594405+0000","last_clean":"2024-03-19T02:02:23.594405+0000","last_became_active":"2024-03-19T01:51:09.195111+0000","last_became_peered":"2024-03-19T01:51:09.195111+0000","last_unstale":"2024-03-19T02:02:23.594405+0000","last_undegraded":"2024-03-19T02:02:23.594405+0000","last_fullsized":"2024-03-19T02:02:23.594405+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:12.084475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"35'2","reported_seq":49,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:40.608950+0000","last_change":"2024-03-19T01:51:06.963650+0000","last_active":"2024-03-19T02:03:40.608950+0000","last_peered":"2024-03-19T02:03:40.608950+0000","last_clean":"2024-03-19T02:03:40.608950+0000","last_became_active":"2024-03-19T01:51:06.963028+0000","last_became_peered":"2024-03-19T01:51:06.963028+0000","last_unstale":"2024-03-19T02:03:40.608950+0000","last_undegraded":"2024-03-19T02:03:40.608950+0000","last_fullsized":"2024-03-19T02:03:40.608950+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:59:26.369856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:08.608117+0000","last_change":"2024-03-19T01:51:16.588077+0000","last_active":"2024-03-19T02:02:08.608117+0000","last_peered":"2024-03-19T02:02:08.608117+0000","last_clean":"2024-03-19T02:02:08.608117+0000","last_became_active":"2024-03-19T01:51:16.587459+0000","last_became_peered":"2024-03-19T01:51:16.587459+0000","last_unstale":"2024-03-19T02:02:08.608117+0000","last_undegraded":"2024-03-19T02:02:08.608117+0000","last_fullsized":"2024-03-19T02:02:08.608117+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:52.667114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.597258+0000","last_change":"2024-03-19T01:51:09.195758+0000","last_active":"2024-03-19T02:02:48.597258+0000","last_peered":"2024-03-19T02:02:48.597258+0000","last_clean":"2024-03-19T02:02:48.597258+0000","last_became_active":"2024-03-19T01:51:09.195139+0000","last_became_peered":"2024-03-19T01:51:09.195139+0000","last_unstale":"2024-03-19T02:02:48.597258+0000","last_undegraded":"2024-03-19T02:02:48.597258+0000","last_fullsized":"2024-03-19T02:02:48.597258+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:18.326414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"29'17","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:46.552835+0000","last_change":"2024-03-19T01:51:06.941200+0000","last_active":"2024-03-19T02:02:46.552835+0000","last_peered":"2024-03-19T02:02:46.552835+0000","last_clean":"2024-03-19T02:02:46.552835+0000","last_became_active":"2024-03-19T01:51:06.940594+0000","last_became_peered":"2024-03-19T01:51:06.940594+0000","last_unstale":"2024-03-19T02:02:46.552835+0000","last_undegraded":"2024-03-19T02:02:46.552835+0000","last_fullsized":"2024-03-19T02:02:46.552835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:25:05.442423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:23.574385+0000","last_change":"2024-03-19T01:51:16.548804+0000","last_active":"2024-03-19T02:03:23.574385+0000","last_peered":"2024-03-19T02:03:23.574385+0000","last_clean":"2024-03-19T02:03:23.574385+0000","last_became_active":"2024-03-19T01:51:16.547800+0000","last_became_peered":"2024-03-19T01:51:16.547800+0000","last_unstale":"2024-03-19T02:03:23.574385+0000","last_undegraded":"2024-03-19T02:03:23.574385+0000","last_fullsized":"2024-03-19T02:03:23.574385+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:43.172796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.499920+0000","last_change":"2024-03-19T01:51:09.167987+0000","last_active":"2024-03-19T02:02:48.499920+0000","last_peered":"2024-03-19T02:02:48.499920+0000","last_clean":"2024-03-19T02:02:48.499920+0000","last_became_active":"2024-03-19T01:51:09.167670+0000","last_became_peered":"2024-03-19T01:51:09.167670+0000","last_unstale":"2024-03-19T02:02:48.499920+0000","last_undegraded":"2024-03-19T02:02:48.499920+0000","last_fullsized":"2024-03-19T02:02:48.499920+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"29'5","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.499949+0000","last_change":"2024-03-19T01:51:07.048197+0000","last_active":"2024-03-19T02:02:48.499949+0000","last_peered":"2024-03-19T02:02:48.499949+0000","last_clean":"2024-03-19T02:02:48.499949+0000","last_became_active":"2024-03-19T01:51:07.047785+0000","last_became_peered":"2024-03-19T01:51:07.047785+0000","last_unstale":"2024-03-19T02:02:48.499949+0000","last_undegraded":"2024-03-19T02:02:48.499949+0000","last_fullsized":"2024-03-19T02:02:48.499949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:52.323249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:21.630885+0000","last_change":"2024-03-19T01:51:16.583394+0000","last_active":"2024-03-19T02:03:21.630885+0000","last_peered":"2024-03-19T02:03:21.630885+0000","last_clean":"2024-03-19T02:03:21.630885+0000","last_became_active":"2024-03-19T01:51:16.582860+0000","last_became_peered":"2024-03-19T01:51:16.582860+0000","last_unstale":"2024-03-19T02:03:21.630885+0000","last_undegraded":"2024-03-19T02:03:21.630885+0000","last_fullsized":"2024-03-19T02:03:21.630885+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:37.669591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.597290+0000","last_change":"2024-03-19T01:51:09.195008+0000","last_active":"2024-03-19T02:02:48.597290+0000","last_peered":"2024-03-19T02:02:48.597290+0000","last_clean":"2024-03-19T02:02:48.597290+0000","last_became_active":"2024-03-19T01:51:09.194728+0000","last_became_peered":"2024-03-19T01:51:09.194728+0000","last_unstale":"2024-03-19T02:02:48.597290+0000","last_undegraded":"2024-03-19T02:02:48.597290+0000","last_fullsized":"2024-03-19T02:02:48.597290+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:15:10.360817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"33'6","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:46.666678+0000","last_change":"2024-03-19T01:51:06.956905+0000","last_active":"2024-03-19T02:02:46.666678+0000","last_peered":"2024-03-19T02:02:46.666678+0000","last_clean":"2024-03-19T02:02:46.666678+0000","last_became_active":"2024-03-19T01:51:06.955991+0000","last_became_peered":"2024-03-19T01:51:06.955991+0000","last_unstale":"2024-03-19T02:02:46.666678+0000","last_undegraded":"2024-03-19T02:02:46.666678+0000","last_fullsized":"2024-03-19T02:02:46.666678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:02:45.970853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":39,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:56.627291+0000","last_change":"2024-03-19T01:52:14.332240+0000","last_active":"2024-03-19T02:02:56.627291+0000","last_peered":"2024-03-19T02:02:56.627291+0000","last_clean":"2024-03-19T02:02:56.627291+0000","last_became_active":"2024-03-19T01:52:14.331292+0000","last_became_peered":"2024-03-19T01:52:14.331292+0000","last_unstale":"2024-03-19T02:02:56.627291+0000","last_undegraded":"2024-03-19T02:02:56.627291+0000","last_fullsized":"2024-03-19T02:02:56.627291+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:14:19.465379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.570250+0000","last_change":"2024-03-19T01:51:09.188003+0000","last_active":"2024-03-19T02:02:48.570250+0000","last_peered":"2024-03-19T02:02:48.570250+0000","last_clean":"2024-03-19T02:02:48.570250+0000","last_became_active":"2024-03-19T01:51:09.187275+0000","last_became_peered":"2024-03-19T01:51:09.187275+0000","last_unstale":"2024-03-19T02:02:48.570250+0000","last_undegraded":"2024-03-19T02:02:48.570250+0000","last_fullsized":"2024-03-19T02:02:48.570250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:10.624259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"29'9","reported_seq":56,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.570286+0000","last_change":"2024-03-19T01:51:06.964413+0000","last_active":"2024-03-19T02:02:48.570286+0000","last_peered":"2024-03-19T02:02:48.570286+0000","last_clean":"2024-03-19T02:02:48.570286+0000","last_became_active":"2024-03-19T01:51:06.963763+0000","last_became_peered":"2024-03-19T01:51:06.963763+0000","last_unstale":"2024-03-19T02:02:48.570286+0000","last_undegraded":"2024-03-19T02:02:48.570286+0000","last_fullsized":"2024-03-19T02:02:48.570286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:21.671644+0000","last_change":"2024-03-19T01:51:16.574401+0000","last_active":"2024-03-19T02:03:21.671644+0000","last_peered":"2024-03-19T02:03:21.671644+0000","last_clean":"2024-03-19T02:03:21.671644+0000","last_became_active":"2024-03-19T01:51:16.573744+0000","last_became_peered":"2024-03-19T01:51:16.573744+0000","last_unstale":"2024-03-19T02:03:21.671644+0000","last_undegraded":"2024-03-19T02:03:21.671644+0000","last_fullsized":"2024-03-19T02:03:21.671644+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:41.414631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.600865+0000","last_change":"2024-03-19T01:51:09.187207+0000","last_active":"2024-03-19T02:03:13.600865+0000","last_peered":"2024-03-19T02:03:13.600865+0000","last_clean":"2024-03-19T02:03:13.600865+0000","last_became_active":"2024-03-19T01:51:09.175898+0000","last_became_peered":"2024-03-19T01:51:09.175898+0000","last_unstale":"2024-03-19T02:03:13.600865+0000","last_undegraded":"2024-03-19T02:03:13.600865+0000","last_fullsized":"2024-03-19T02:03:13.600865+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:38.387069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'14","reported_seq":61,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.499975+0000","last_change":"2024-03-19T01:51:07.042663+0000","last_active":"2024-03-19T02:02:48.499975+0000","last_peered":"2024-03-19T02:02:48.499975+0000","last_clean":"2024-03-19T02:02:48.499975+0000","last_became_active":"2024-03-19T01:51:07.041262+0000","last_became_peered":"2024-03-19T01:51:07.041262+0000","last_unstale":"2024-03-19T02:02:48.499975+0000","last_undegraded":"2024-03-19T02:02:48.499975+0000","last_fullsized":"2024-03-19T02:02:48.499975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:58.513211+0000","last_change":"2024-03-19T01:51:16.549744+0000","last_active":"2024-03-19T02:02:58.513211+0000","last_peered":"2024-03-19T02:02:58.513211+0000","last_clean":"2024-03-19T02:02:58.513211+0000","last_became_active":"2024-03-19T01:51:16.549347+0000","last_became_peered":"2024-03-19T01:51:16.549347+0000","last_unstale":"2024-03-19T02:02:58.513211+0000","last_undegraded":"2024-03-19T02:02:58.513211+0000","last_fullsized":"2024-03-19T02:02:58.513211+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:19.442287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.612688+0000","last_change":"2024-03-19T01:51:09.194165+0000","last_active":"2024-03-19T02:02:48.612688+0000","last_peered":"2024-03-19T02:02:48.612688+0000","last_clean":"2024-03-19T02:02:48.612688+0000","last_became_active":"2024-03-19T01:51:09.183607+0000","last_became_peered":"2024-03-19T01:51:09.183607+0000","last_unstale":"2024-03-19T02:02:48.612688+0000","last_undegraded":"2024-03-19T02:02:48.612688+0000","last_fullsized":"2024-03-19T02:02:48.612688+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:09.211794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'5","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.514982+0000","last_change":"2024-03-19T01:51:06.952515+0000","last_active":"2024-03-19T02:03:13.514982+0000","last_peered":"2024-03-19T02:03:13.514982+0000","last_clean":"2024-03-19T02:03:13.514982+0000","last_became_active":"2024-03-19T01:51:06.951700+0000","last_became_peered":"2024-03-19T01:51:06.951700+0000","last_unstale":"2024-03-19T02:03:13.514982+0000","last_undegraded":"2024-03-19T02:03:13.514982+0000","last_fullsized":"2024-03-19T02:03:13.514982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:13.137673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:58.613767+0000","last_change":"2024-03-19T01:51:16.587027+0000","last_active":"2024-03-19T02:02:58.613767+0000","last_peered":"2024-03-19T02:02:58.613767+0000","last_clean":"2024-03-19T02:02:58.613767+0000","last_became_active":"2024-03-19T01:51:16.586514+0000","last_became_peered":"2024-03-19T01:51:16.586514+0000","last_unstale":"2024-03-19T02:02:58.613767+0000","last_undegraded":"2024-03-19T02:02:58.613767+0000","last_fullsized":"2024-03-19T02:02:58.613767+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:06.637617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.573086+0000","last_change":"2024-03-19T01:51:09.172554+0000","last_active":"2024-03-19T02:03:13.573086+0000","last_peered":"2024-03-19T02:03:13.573086+0000","last_clean":"2024-03-19T02:03:13.573086+0000","last_became_active":"2024-03-19T01:51:09.171814+0000","last_became_peered":"2024-03-19T01:51:09.171814+0000","last_unstale":"2024-03-19T02:03:13.573086+0000","last_undegraded":"2024-03-19T02:03:13.573086+0000","last_fullsized":"2024-03-19T02:03:13.573086+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'4","reported_seq":51,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:11.670336+0000","last_change":"2024-03-19T01:51:07.043404+0000","last_active":"2024-03-19T02:03:11.670336+0000","last_peered":"2024-03-19T02:03:11.670336+0000","last_clean":"2024-03-19T02:03:11.670336+0000","last_became_active":"2024-03-19T01:51:07.043069+0000","last_became_peered":"2024-03-19T01:51:07.043069+0000","last_unstale":"2024-03-19T02:03:11.670336+0000","last_undegraded":"2024-03-19T02:03:11.670336+0000","last_fullsized":"2024-03-19T02:03:11.670336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:13:14.960132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:56.627225+0000","last_change":"2024-03-19T01:51:16.581797+0000","last_active":"2024-03-19T02:02:56.627225+0000","last_peered":"2024-03-19T02:02:56.627225+0000","last_clean":"2024-03-19T02:02:56.627225+0000","last_became_active":"2024-03-19T01:51:16.581174+0000","last_became_peered":"2024-03-19T01:51:16.581174+0000","last_unstale":"2024-03-19T02:02:56.627225+0000","last_undegraded":"2024-03-19T02:02:56.627225+0000","last_fullsized":"2024-03-19T02:02:56.627225+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:29:34.911347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:16.671090+0000","last_change":"2024-03-19T01:51:09.194372+0000","last_active":"2024-03-19T02:03:16.671090+0000","last_peered":"2024-03-19T02:03:16.671090+0000","last_clean":"2024-03-19T02:03:16.671090+0000","last_became_active":"2024-03-19T01:51:09.193656+0000","last_became_peered":"2024-03-19T01:51:09.193656+0000","last_unstale":"2024-03-19T02:03:16.671090+0000","last_undegraded":"2024-03-19T02:03:16.671090+0000","last_fullsized":"2024-03-19T02:03:16.671090+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:23.464803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'4","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.502733+0000","last_change":"2024-03-19T01:51:07.046089+0000","last_active":"2024-03-19T02:03:13.502733+0000","last_peered":"2024-03-19T02:03:13.502733+0000","last_clean":"2024-03-19T02:03:13.502733+0000","last_became_active":"2024-03-19T01:51:07.045342+0000","last_became_peered":"2024-03-19T01:51:07.045342+0000","last_unstale":"2024-03-19T02:03:13.502733+0000","last_undegraded":"2024-03-19T02:03:13.502733+0000","last_fullsized":"2024-03-19T02:03:13.502733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:26.105915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:56.627262+0000","last_change":"2024-03-19T01:51:16.572559+0000","last_active":"2024-03-19T02:02:56.627262+0000","last_peered":"2024-03-19T02:02:56.627262+0000","last_clean":"2024-03-19T02:02:56.627262+0000","last_became_active":"2024-03-19T01:51:16.572283+0000","last_became_peered":"2024-03-19T01:51:16.572283+0000","last_unstale":"2024-03-19T02:02:56.627262+0000","last_undegraded":"2024-03-19T02:02:56.627262+0000","last_fullsized":"2024-03-19T02:02:56.627262+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:42.703415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:18.573781+0000","last_change":"2024-03-19T01:51:16.534088+0000","last_active":"2024-03-19T02:03:18.573781+0000","last_peered":"2024-03-19T02:03:18.573781+0000","last_clean":"2024-03-19T02:03:18.573781+0000","last_became_active":"2024-03-19T01:51:16.533665+0000","last_became_peered":"2024-03-19T01:51:16.533665+0000","last_unstale":"2024-03-19T02:03:18.573781+0000","last_undegraded":"2024-03-19T02:03:18.573781+0000","last_fullsized":"2024-03-19T02:03:18.573781+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:35.409504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.514931+0000","last_change":"2024-03-19T01:51:09.168372+0000","last_active":"2024-03-19T02:03:13.514931+0000","last_peered":"2024-03-19T02:03:13.514931+0000","last_clean":"2024-03-19T02:03:13.514931+0000","last_became_active":"2024-03-19T01:51:09.167626+0000","last_became_peered":"2024-03-19T01:51:09.167626+0000","last_unstale":"2024-03-19T02:03:13.514931+0000","last_undegraded":"2024-03-19T02:03:13.514931+0000","last_fullsized":"2024-03-19T02:03:13.514931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"33'4","reported_seq":51,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:11.670233+0000","last_change":"2024-03-19T01:51:06.951225+0000","last_active":"2024-03-19T02:03:11.670233+0000","last_peered":"2024-03-19T02:03:11.670233+0000","last_clean":"2024-03-19T02:03:11.670233+0000","last_became_active":"2024-03-19T01:51:06.950826+0000","last_became_peered":"2024-03-19T01:51:06.950826+0000","last_unstale":"2024-03-19T02:03:11.670233+0000","last_undegraded":"2024-03-19T02:03:11.670233+0000","last_fullsized":"2024-03-19T02:03:11.670233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:24.949925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.507160+0000","last_change":"2024-03-19T01:51:16.581997+0000","last_active":"2024-03-19T02:03:48.507160+0000","last_peered":"2024-03-19T02:03:48.507160+0000","last_clean":"2024-03-19T02:03:48.507160+0000","last_became_active":"2024-03-19T01:51:16.581412+0000","last_became_peered":"2024-03-19T01:51:16.581412+0000","last_unstale":"2024-03-19T02:03:48.507160+0000","last_undegraded":"2024-03-19T02:03:48.507160+0000","last_fullsized":"2024-03-19T02:03:48.507160+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:45.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:16.630092+0000","last_change":"2024-03-19T01:51:09.161686+0000","last_active":"2024-03-19T02:03:16.630092+0000","last_peered":"2024-03-19T02:03:16.630092+0000","last_clean":"2024-03-19T02:03:16.630092+0000","last_became_active":"2024-03-19T01:51:09.160978+0000","last_became_peered":"2024-03-19T01:51:09.160978+0000","last_unstale":"2024-03-19T02:03:16.630092+0000","last_undegraded":"2024-03-19T02:03:16.630092+0000","last_fullsized":"2024-03-19T02:03:16.630092+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:27.266807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'3","reported_seq":50,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:11.629313+0000","last_change":"2024-03-19T01:51:07.047363+0000","last_active":"2024-03-19T02:03:11.629313+0000","last_peered":"2024-03-19T02:03:11.629313+0000","last_clean":"2024-03-19T02:03:11.629313+0000","last_became_active":"2024-03-19T01:51:07.046886+0000","last_became_peered":"2024-03-19T01:51:07.046886+0000","last_unstale":"2024-03-19T02:03:11.629313+0000","last_undegraded":"2024-03-19T02:03:11.629313+0000","last_fullsized":"2024-03-19T02:03:11.629313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:19.418603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"29'7","reported_seq":54,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.505874+0000","last_change":"2024-03-19T01:51:07.045496+0000","last_active":"2024-03-19T02:03:38.505874+0000","last_peered":"2024-03-19T02:03:38.505874+0000","last_clean":"2024-03-19T02:03:38.505874+0000","last_became_active":"2024-03-19T01:51:07.044554+0000","last_became_peered":"2024-03-19T01:51:07.044554+0000","last_unstale":"2024-03-19T02:03:38.505874+0000","last_undegraded":"2024-03-19T02:03:38.505874+0000","last_fullsized":"2024-03-19T02:03:38.505874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:14.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.576001+0000","last_change":"2024-03-19T01:51:09.184934+0000","last_active":"2024-03-19T02:03:38.576001+0000","last_peered":"2024-03-19T02:03:38.576001+0000","last_clean":"2024-03-19T02:03:38.576001+0000","last_became_active":"2024-03-19T01:51:09.184666+0000","last_became_peered":"2024-03-19T01:51:09.184666+0000","last_unstale":"2024-03-19T02:03:38.576001+0000","last_undegraded":"2024-03-19T02:03:38.576001+0000","last_fullsized":"2024-03-19T02:03:38.576001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:42.786323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":38,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:16.540961+0000","last_change":"2024-03-19T01:52:14.298105+0000","last_active":"2024-03-19T02:01:16.540961+0000","last_peered":"2024-03-19T02:01:16.540961+0000","last_clean":"2024-03-19T02:01:16.540961+0000","last_became_active":"2024-03-19T01:52:14.297612+0000","last_became_peered":"2024-03-19T01:52:14.297612+0000","last_unstale":"2024-03-19T02:01:16.540961+0000","last_undegraded":"2024-03-19T02:01:16.540961+0000","last_fullsized":"2024-03-19T02:01:16.540961+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:38.753646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,5],"acting":[2,0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"33'7","reported_seq":57,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:36.632647+0000","last_change":"2024-03-19T01:51:06.982280+0000","last_active":"2024-03-19T02:03:36.632647+0000","last_peered":"2024-03-19T02:03:36.632647+0000","last_clean":"2024-03-19T02:03:36.632647+0000","last_became_active":"2024-03-19T01:51:06.981704+0000","last_became_peered":"2024-03-19T01:51:06.981704+0000","last_unstale":"2024-03-19T02:03:36.632647+0000","last_undegraded":"2024-03-19T02:03:36.632647+0000","last_fullsized":"2024-03-19T02:03:36.632647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:22:12.657985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.618444+0000","last_change":"2024-03-19T01:51:09.184624+0000","last_active":"2024-03-19T02:03:38.618444+0000","last_peered":"2024-03-19T02:03:38.618444+0000","last_clean":"2024-03-19T02:03:38.618444+0000","last_became_active":"2024-03-19T01:51:09.183976+0000","last_became_peered":"2024-03-19T01:51:09.183976+0000","last_unstale":"2024-03-19T02:03:38.618444+0000","last_undegraded":"2024-03-19T02:03:38.618444+0000","last_fullsized":"2024-03-19T02:03:38.618444+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:24:29.313729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":38,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:16.614098+0000","last_change":"2024-03-19T01:52:14.299182+0000","last_active":"2024-03-19T02:01:16.614098+0000","last_peered":"2024-03-19T02:01:16.614098+0000","last_clean":"2024-03-19T02:01:16.614098+0000","last_became_active":"2024-03-19T01:52:14.297976+0000","last_became_peered":"2024-03-19T01:52:14.297976+0000","last_unstale":"2024-03-19T02:01:16.614098+0000","last_undegraded":"2024-03-19T02:01:16.614098+0000","last_fullsized":"2024-03-19T02:01:16.614098+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:31.057362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4,2],"acting":[3,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.505794+0000","last_change":"2024-03-19T01:51:09.169493+0000","last_active":"2024-03-19T02:03:38.505794+0000","last_peered":"2024-03-19T02:03:38.505794+0000","last_clean":"2024-03-19T02:03:38.505794+0000","last_became_active":"2024-03-19T01:51:09.168231+0000","last_became_peered":"2024-03-19T01:51:09.168231+0000","last_unstale":"2024-03-19T02:03:38.505794+0000","last_undegraded":"2024-03-19T02:03:38.505794+0000","last_fullsized":"2024-03-19T02:03:38.505794+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:10.624259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:41.559618+0000","last_change":"2024-03-19T01:51:09.155483+0000","last_active":"2024-03-19T02:03:41.559618+0000","last_peered":"2024-03-19T02:03:41.559618+0000","last_clean":"2024-03-19T02:03:41.559618+0000","last_became_active":"2024-03-19T01:51:09.154545+0000","last_became_peered":"2024-03-19T01:51:09.154545+0000","last_unstale":"2024-03-19T02:03:41.559618+0000","last_undegraded":"2024-03-19T02:03:41.559618+0000","last_fullsized":"2024-03-19T02:03:41.559618+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:25:09.517323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:41.673722+0000","last_change":"2024-03-19T01:51:09.180498+0000","last_active":"2024-03-19T02:03:41.673722+0000","last_peered":"2024-03-19T02:03:41.673722+0000","last_clean":"2024-03-19T02:03:41.673722+0000","last_became_active":"2024-03-19T01:51:09.179840+0000","last_became_peered":"2024-03-19T01:51:09.179840+0000","last_unstale":"2024-03-19T02:03:41.673722+0000","last_undegraded":"2024-03-19T02:03:41.673722+0000","last_fullsized":"2024-03-19T02:03:41.673722+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:01:21.716571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.575976+0000","last_change":"2024-03-19T01:51:09.170389+0000","last_active":"2024-03-19T02:03:38.575976+0000","last_peered":"2024-03-19T02:03:38.575976+0000","last_clean":"2024-03-19T02:03:38.575976+0000","last_became_active":"2024-03-19T01:51:09.163974+0000","last_became_peered":"2024-03-19T01:51:09.163974+0000","last_unstale":"2024-03-19T02:03:38.575976+0000","last_undegraded":"2024-03-19T02:03:38.575976+0000","last_fullsized":"2024-03-19T02:03:38.575976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:01.620620+0000","last_change":"2024-03-19T01:51:09.177060+0000","last_active":"2024-03-19T02:02:01.620620+0000","last_peered":"2024-03-19T02:02:01.620620+0000","last_clean":"2024-03-19T02:02:01.620620+0000","last_became_active":"2024-03-19T01:51:09.176474+0000","last_became_peered":"2024-03-19T01:51:09.176474+0000","last_unstale":"2024-03-19T02:02:01.620620+0000","last_undegraded":"2024-03-19T02:02:01.620620+0000","last_fullsized":"2024-03-19T02:02:01.620620+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:56.318137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.504665+0000","last_change":"2024-03-19T01:51:09.168741+0000","last_active":"2024-03-19T02:01:58.504665+0000","last_peered":"2024-03-19T02:01:58.504665+0000","last_clean":"2024-03-19T02:01:58.504665+0000","last_became_active":"2024-03-19T01:51:09.167982+0000","last_became_peered":"2024-03-19T01:51:09.167982+0000","last_unstale":"2024-03-19T02:01:58.504665+0000","last_undegraded":"2024-03-19T02:01:58.504665+0000","last_fullsized":"2024-03-19T02:01:58.504665+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.494217+0000","last_change":"2024-03-19T01:51:09.189814+0000","last_active":"2024-03-19T02:01:58.494217+0000","last_peered":"2024-03-19T02:01:58.494217+0000","last_clean":"2024-03-19T02:01:58.494217+0000","last_became_active":"2024-03-19T01:51:09.189128+0000","last_became_peered":"2024-03-19T01:51:09.189128+0000","last_unstale":"2024-03-19T02:01:58.494217+0000","last_undegraded":"2024-03-19T02:01:58.494217+0000","last_fullsized":"2024-03-19T02:01:58.494217+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:01.620637+0000","last_change":"2024-03-19T01:51:09.183989+0000","last_active":"2024-03-19T02:02:01.620637+0000","last_peered":"2024-03-19T02:02:01.620637+0000","last_clean":"2024-03-19T02:02:01.620637+0000","last_became_active":"2024-03-19T01:51:09.183417+0000","last_became_peered":"2024-03-19T01:51:09.183417+0000","last_unstale":"2024-03-19T02:02:01.620637+0000","last_undegraded":"2024-03-19T02:02:01.620637+0000","last_fullsized":"2024-03-19T02:02:01.620637+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:31.405618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:01.660628+0000","last_change":"2024-03-19T01:51:09.172717+0000","last_active":"2024-03-19T02:02:01.660628+0000","last_peered":"2024-03-19T02:02:01.660628+0000","last_clean":"2024-03-19T02:02:01.660628+0000","last_became_active":"2024-03-19T01:51:09.165872+0000","last_became_peered":"2024-03-19T01:51:09.165872+0000","last_unstale":"2024-03-19T02:02:01.660628+0000","last_undegraded":"2024-03-19T02:02:01.660628+0000","last_fullsized":"2024-03-19T02:02:01.660628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:10.995492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:26.557659+0000","last_change":"2024-03-19T01:52:12.916894+0000","last_active":"2024-03-19T02:03:26.557659+0000","last_peered":"2024-03-19T02:03:26.557659+0000","last_clean":"2024-03-19T02:03:26.557659+0000","last_became_active":"2024-03-19T01:52:12.916608+0000","last_became_peered":"2024-03-19T01:52:12.916608+0000","last_unstale":"2024-03-19T02:03:26.557659+0000","last_undegraded":"2024-03-19T02:03:26.557659+0000","last_fullsized":"2024-03-19T02:03:26.557659+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:31.405618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:26.664130+0000","last_change":"2024-03-19T01:51:09.195141+0000","last_active":"2024-03-19T02:02:26.664130+0000","last_peered":"2024-03-19T02:02:26.664130+0000","last_clean":"2024-03-19T02:02:26.664130+0000","last_became_active":"2024-03-19T01:51:09.194536+0000","last_became_peered":"2024-03-19T01:51:09.194536+0000","last_unstale":"2024-03-19T02:02:26.664130+0000","last_undegraded":"2024-03-19T02:02:26.664130+0000","last_fullsized":"2024-03-19T02:02:26.664130+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:31.395833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.567024+0000","last_change":"2024-03-19T01:51:09.189197+0000","last_active":"2024-03-19T02:02:23.567024+0000","last_peered":"2024-03-19T02:02:23.567024+0000","last_clean":"2024-03-19T02:02:23.567024+0000","last_became_active":"2024-03-19T01:51:09.188565+0000","last_became_peered":"2024-03-19T01:51:09.188565+0000","last_unstale":"2024-03-19T02:02:23.567024+0000","last_undegraded":"2024-03-19T02:02:23.567024+0000","last_fullsized":"2024-03-19T02:02:23.567024+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.497100+0000","last_change":"2024-03-19T01:51:09.168581+0000","last_active":"2024-03-19T02:02:23.497100+0000","last_peered":"2024-03-19T02:02:23.497100+0000","last_clean":"2024-03-19T02:02:23.497100+0000","last_became_active":"2024-03-19T01:51:09.167812+0000","last_became_peered":"2024-03-19T01:51:09.167812+0000","last_unstale":"2024-03-19T02:02:23.497100+0000","last_undegraded":"2024-03-19T02:02:23.497100+0000","last_fullsized":"2024-03-19T02:02:23.497100+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:28.517089+0000","last_change":"2024-03-19T01:52:12.915401+0000","last_active":"2024-03-19T02:03:28.517089+0000","last_peered":"2024-03-19T02:03:28.517089+0000","last_clean":"2024-03-19T02:03:28.517089+0000","last_became_active":"2024-03-19T01:52:12.915089+0000","last_became_peered":"2024-03-19T01:52:12.915089+0000","last_unstale":"2024-03-19T02:03:28.517089+0000","last_undegraded":"2024-03-19T02:03:28.517089+0000","last_fullsized":"2024-03-19T02:03:28.517089+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:54:01.116865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:51.553572+0000","last_change":"2024-03-19T01:51:09.168451+0000","last_active":"2024-03-19T02:02:51.553572+0000","last_peered":"2024-03-19T02:02:51.553572+0000","last_clean":"2024-03-19T02:02:51.553572+0000","last_became_active":"2024-03-19T01:51:09.167744+0000","last_became_peered":"2024-03-19T01:51:09.167744+0000","last_unstale":"2024-03-19T02:02:51.553572+0000","last_undegraded":"2024-03-19T02:02:51.553572+0000","last_fullsized":"2024-03-19T02:02:51.553572+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:21.735485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:51.667789+0000","last_change":"2024-03-19T01:51:09.196067+0000","last_active":"2024-03-19T02:02:51.667789+0000","last_peered":"2024-03-19T02:02:51.667789+0000","last_clean":"2024-03-19T02:02:51.667789+0000","last_became_active":"2024-03-19T01:51:09.195488+0000","last_became_peered":"2024-03-19T01:51:09.195488+0000","last_unstale":"2024-03-19T02:02:51.667789+0000","last_undegraded":"2024-03-19T02:02:51.667789+0000","last_fullsized":"2024-03-19T02:02:51.667789+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:07:44.120161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:51.667650+0000","last_change":"2024-03-19T01:51:09.193831+0000","last_active":"2024-03-19T02:02:51.667650+0000","last_peered":"2024-03-19T02:02:51.667650+0000","last_clean":"2024-03-19T02:02:51.667650+0000","last_became_active":"2024-03-19T01:51:09.192777+0000","last_became_peered":"2024-03-19T01:51:09.192777+0000","last_unstale":"2024-03-19T02:02:51.667650+0000","last_undegraded":"2024-03-19T02:02:51.667650+0000","last_fullsized":"2024-03-19T02:02:51.667650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:56.318137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.511885+0000","last_change":"2024-03-19T01:51:09.193581+0000","last_active":"2024-03-19T02:02:48.511885+0000","last_peered":"2024-03-19T02:02:48.511885+0000","last_clean":"2024-03-19T02:02:48.511885+0000","last_became_active":"2024-03-19T01:51:09.192677+0000","last_became_peered":"2024-03-19T01:51:09.192677+0000","last_unstale":"2024-03-19T02:02:48.511885+0000","last_undegraded":"2024-03-19T02:02:48.511885+0000","last_fullsized":"2024-03-19T02:02:48.511885+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.570213+0000","last_change":"2024-03-19T01:51:09.165568+0000","last_active":"2024-03-19T02:02:48.570213+0000","last_peered":"2024-03-19T02:02:48.570213+0000","last_clean":"2024-03-19T02:02:48.570213+0000","last_became_active":"2024-03-19T01:51:09.164753+0000","last_became_peered":"2024-03-19T01:51:09.164753+0000","last_unstale":"2024-03-19T02:02:48.570213+0000","last_undegraded":"2024-03-19T02:02:48.570213+0000","last_fullsized":"2024-03-19T02:02:48.570213+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:28.422798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:16.630124+0000","last_change":"2024-03-19T01:51:09.170898+0000","last_active":"2024-03-19T02:03:16.630124+0000","last_peered":"2024-03-19T02:03:16.630124+0000","last_clean":"2024-03-19T02:03:16.630124+0000","last_became_active":"2024-03-19T01:51:09.170153+0000","last_became_peered":"2024-03-19T01:51:09.170153+0000","last_unstale":"2024-03-19T02:03:16.630124+0000","last_undegraded":"2024-03-19T02:03:16.630124+0000","last_fullsized":"2024-03-19T02:03:16.630124+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:41.125800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.615856+0000","last_change":"2024-03-19T01:51:09.185953+0000","last_active":"2024-03-19T02:03:13.615856+0000","last_peered":"2024-03-19T02:03:13.615856+0000","last_clean":"2024-03-19T02:03:13.615856+0000","last_became_active":"2024-03-19T01:51:09.177996+0000","last_became_peered":"2024-03-19T01:51:09.177996+0000","last_unstale":"2024-03-19T02:03:13.615856+0000","last_undegraded":"2024-03-19T02:03:13.615856+0000","last_fullsized":"2024-03-19T02:03:13.615856+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:39:06.045413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.502685+0000","last_change":"2024-03-19T01:51:09.183841+0000","last_active":"2024-03-19T02:03:13.502685+0000","last_peered":"2024-03-19T02:03:13.502685+0000","last_clean":"2024-03-19T02:03:13.502685+0000","last_became_active":"2024-03-19T01:51:09.183045+0000","last_became_peered":"2024-03-19T01:51:09.183045+0000","last_unstale":"2024-03-19T02:03:13.502685+0000","last_undegraded":"2024-03-19T02:03:13.502685+0000","last_fullsized":"2024-03-19T02:03:13.502685+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.502713+0000","last_change":"2024-03-19T01:51:09.184022+0000","last_active":"2024-03-19T02:03:13.502713+0000","last_peered":"2024-03-19T02:03:13.502713+0000","last_clean":"2024-03-19T02:03:13.502713+0000","last_became_active":"2024-03-19T01:51:09.183351+0000","last_became_peered":"2024-03-19T01:51:09.183351+0000","last_unstale":"2024-03-19T02:03:13.502713+0000","last_undegraded":"2024-03-19T02:03:13.502713+0000","last_fullsized":"2024-03-19T02:03:13.502713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.600945+0000","last_change":"2024-03-19T01:52:12.922818+0000","last_active":"2024-03-19T02:03:13.600945+0000","last_peered":"2024-03-19T02:03:13.600945+0000","last_clean":"2024-03-19T02:03:13.600945+0000","last_became_active":"2024-03-19T01:52:12.922473+0000","last_became_peered":"2024-03-19T01:52:12.922473+0000","last_unstale":"2024-03-19T02:03:13.600945+0000","last_undegraded":"2024-03-19T02:03:13.600945+0000","last_fullsized":"2024-03-19T02:03:13.600945+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:35.798061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:11.613507+0000","last_change":"2024-03-19T01:51:09.180013+0000","last_active":"2024-03-19T02:01:11.613507+0000","last_peered":"2024-03-19T02:01:11.613507+0000","last_clean":"2024-03-19T02:01:11.613507+0000","last_became_active":"2024-03-19T01:51:09.179541+0000","last_became_peered":"2024-03-19T01:51:09.179541+0000","last_unstale":"2024-03-19T02:01:11.613507+0000","last_undegraded":"2024-03-19T02:01:11.613507+0000","last_fullsized":"2024-03-19T02:01:11.613507+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:19:25.933028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603886+0000","last_change":"2024-03-19T01:51:09.188671+0000","last_active":"2024-03-19T02:03:38.603886+0000","last_peered":"2024-03-19T02:03:38.603886+0000","last_clean":"2024-03-19T02:03:38.603886+0000","last_became_active":"2024-03-19T01:51:09.177627+0000","last_became_peered":"2024-03-19T01:51:09.177627+0000","last_unstale":"2024-03-19T02:03:38.603886+0000","last_undegraded":"2024-03-19T02:03:38.603886+0000","last_fullsized":"2024-03-19T02:03:38.603886+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:53:50.539837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:11.654275+0000","last_change":"2024-03-19T01:51:09.180735+0000","last_active":"2024-03-19T02:01:11.654275+0000","last_peered":"2024-03-19T02:01:11.654275+0000","last_clean":"2024-03-19T02:01:11.654275+0000","last_became_active":"2024-03-19T01:51:09.173235+0000","last_became_peered":"2024-03-19T01:51:09.173235+0000","last_unstale":"2024-03-19T02:01:11.654275+0000","last_undegraded":"2024-03-19T02:01:11.654275+0000","last_fullsized":"2024-03-19T02:01:11.654275+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:31.405618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603918+0000","last_change":"2024-03-19T01:51:09.188346+0000","last_active":"2024-03-19T02:03:38.603918+0000","last_peered":"2024-03-19T02:03:38.603918+0000","last_clean":"2024-03-19T02:03:38.603918+0000","last_became_active":"2024-03-19T01:51:09.187703+0000","last_became_peered":"2024-03-19T01:51:09.187703+0000","last_unstale":"2024-03-19T02:03:38.603918+0000","last_undegraded":"2024-03-19T02:03:38.603918+0000","last_fullsized":"2024-03-19T02:03:38.603918+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:28.051691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:11.548313+0000","last_change":"2024-03-19T01:52:12.919846+0000","last_active":"2024-03-19T02:02:11.548313+0000","last_peered":"2024-03-19T02:02:11.548313+0000","last_clean":"2024-03-19T02:02:11.548313+0000","last_became_active":"2024-03-19T01:52:12.918835+0000","last_became_peered":"2024-03-19T01:52:12.918835+0000","last_unstale":"2024-03-19T02:02:11.548313+0000","last_undegraded":"2024-03-19T02:02:11.548313+0000","last_fullsized":"2024-03-19T02:02:11.548313+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:03:04.997711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.576027+0000","last_change":"2024-03-19T01:51:09.161476+0000","last_active":"2024-03-19T02:03:38.576027+0000","last_peered":"2024-03-19T02:03:38.576027+0000","last_clean":"2024-03-19T02:03:38.576027+0000","last_became_active":"2024-03-19T01:51:09.158609+0000","last_became_peered":"2024-03-19T01:51:09.158609+0000","last_unstale":"2024-03-19T02:03:38.576027+0000","last_undegraded":"2024-03-19T02:03:38.576027+0000","last_fullsized":"2024-03-19T02:03:38.576027+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.518711+0000","last_change":"2024-03-19T01:51:09.165938+0000","last_active":"2024-03-19T02:03:38.518711+0000","last_peered":"2024-03-19T02:03:38.518711+0000","last_clean":"2024-03-19T02:03:38.518711+0000","last_became_active":"2024-03-19T01:51:09.165207+0000","last_became_peered":"2024-03-19T01:51:09.165207+0000","last_unstale":"2024-03-19T02:03:38.518711+0000","last_undegraded":"2024-03-19T02:03:38.518711+0000","last_fullsized":"2024-03-19T02:03:38.518711+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603999+0000","last_change":"2024-03-19T01:51:09.176023+0000","last_active":"2024-03-19T02:03:38.603999+0000","last_peered":"2024-03-19T02:03:38.603999+0000","last_clean":"2024-03-19T02:03:38.603999+0000","last_became_active":"2024-03-19T01:51:09.175682+0000","last_became_peered":"2024-03-19T01:51:09.175682+0000","last_unstale":"2024-03-19T02:03:38.603999+0000","last_undegraded":"2024-03-19T02:03:38.603999+0000","last_fullsized":"2024-03-19T02:03:38.603999+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:35.814983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":209,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":189,"ondisk_log_size":189,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509633,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343828,"statfs":{"total":96636764160,"available":96608079872,"internally_reserved":0,"allocated":1011712,"data_stored":759262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509635,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":171426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509634,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":168124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509635,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31580,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":409600,"data_stored":168326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509634,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":1003520,"data_stored":758036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542337,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":430080,"data_stored":170726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542338,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31588,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340252,"statfs":{"total":96636764160,"available":96604418048,"internally_reserved":0,"allocated":413696,"data_stored":171460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542337,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31588,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340252,"statfs":{"total":96636764160,"available":96604418048,"internally_reserved":0,"allocated":413696,"data_stored":168256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:06:11.667 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-19T02:06:11.668 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:06:11.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.813+0000 7f991a29f640 1 Processor -- start 2024-03-19T02:06:11.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.813+0000 7f991a29f640 1 -- start start 2024-03-19T02:06:11.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.813+0000 7f991a29f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f9914154460 0x7f991406b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:11.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.813+0000 7f991a29f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f99140c2760 0x7f99140c2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:11.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.813+0000 7f991a29f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99140c22b0 con 0x7f9914073540 2024-03-19T02:06:11.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.813+0000 7f991a29f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f99140c3150 con 0x7f9914154460 2024-03-19T02:06:11.816 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.813+0000 7f991a29f640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f991415a230 con 0x7f99140c2760 2024-03-19T02:06:11.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.817+0000 7f9913fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f9914154460 0x7f991406b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:11.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.817+0000 7f991883c640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f99140c2760 0x7f99140c2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:11.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.817+0000 7f9913fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f9914154460 0x7f991406b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57156/0 (socket says 172.21.15.55:57156) 2024-03-19T02:06:11.817 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.817+0000 7f9913fff640 1 -- 172.21.15.55:0/2516767886 learned_addr learned my addr 172.21.15.55:0/2516767886 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:11.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.817+0000 7f9913fff640 1 -- 172.21.15.55:0/2516767886 >> v2:172.21.15.84:3300/0 conn(0x7f99140c2760 msgr2=0x7f99140c2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:11.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.817+0000 7f9913fff640 1 --2- 172.21.15.55:0/2516767886 >> v2:172.21.15.84:3300/0 conn(0x7f99140c2760 0x7f99140c2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:11.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.817+0000 7f9913fff640 1 -- 172.21.15.55:0/2516767886 >> v1:172.21.15.84:6789/0 conn(0x7f9914073540 legacy=0x7f99140cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:11.820 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.817+0000 7f9913fff640 1 -- 172.21.15.55:0/2516767886 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f991415a490 con 0x7f9914154460 2024-03-19T02:06:11.823 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.821+0000 7f9913fff640 1 --2- 172.21.15.55:0/2516767886 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9914154460 0x7f991406b2a0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f991406dfb0 tx=0x7f990401ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69bf1f48de431ad1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:11.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.825+0000 7f9912ffd640 1 -- 172.21.15.55:0/2516767886 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99040b5070 con 0x7f9914154460 2024-03-19T02:06:11.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.825+0000 7f9912ffd640 1 -- 172.21.15.55:0/2516767886 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99040a89d0 con 0x7f9914154460 2024-03-19T02:06:11.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.825+0000 7f9912ffd640 1 -- 172.21.15.55:0/2516767886 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99040a8da0 con 0x7f9914154460 2024-03-19T02:06:11.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.825+0000 7f991a29f640 1 -- 172.21.15.55:0/2516767886 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9914154460 msgr2=0x7f991406b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:11.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.825+0000 7f991a29f640 1 --2- 172.21.15.55:0/2516767886 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9914154460 0x7f991406b2a0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f991406dfb0 tx=0x7f990401ecd0 comp rx=0 tx=0).stop 2024-03-19T02:06:11.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.829+0000 7f991a29f640 1 -- 172.21.15.55:0/2516767886 shutdown_connections 2024-03-19T02:06:11.828 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.829+0000 7f991a29f640 1 --2- 172.21.15.55:0/2516767886 >> v2:172.21.15.84:3300/0 conn(0x7f99140c2760 0x7f99140c2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:11.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.829+0000 7f991a29f640 1 --2- 172.21.15.55:0/2516767886 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9914154460 0x7f991406b2a0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:11.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.829+0000 7f991a29f640 1 -- 172.21.15.55:0/2516767886 >> 172.21.15.55:0/2516767886 conn(0x7f991406e670 msgr2=0x7f991406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:11.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.829+0000 7f991a29f640 1 -- 172.21.15.55:0/2516767886 shutdown_connections 2024-03-19T02:06:11.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.829+0000 7f991a29f640 1 -- 172.21.15.55:0/2516767886 wait complete. 2024-03-19T02:06:11.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.829+0000 7f991a29f640 1 Processor -- start 2024-03-19T02:06:11.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.829+0000 7f991a29f640 1 -- start start 2024-03-19T02:06:11.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.829+0000 7f991a29f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9914073540 0x7f99140c9360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:11.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.829+0000 7f991a29f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f99140c2760 0x7f99141f09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:11.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.829+0000 7f991a29f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9914154460 0x7f99140c3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:11.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.829+0000 7f991a29f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f991415ad30 con 0x7f9914073540 2024-03-19T02:06:11.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.829+0000 7f991a29f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f991415aa50 con 0x7f9914154460 2024-03-19T02:06:11.831 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.829+0000 7f991a29f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f991415b010 con 0x7f99140c2760 2024-03-19T02:06:11.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.829+0000 7f9913fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9914073540 0x7f99140c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:11.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.829+0000 7f99137fe640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f99140c2760 0x7f99141f09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:11.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.829+0000 7f9913fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9914073540 0x7f99140c9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57164/0 (socket says 172.21.15.55:57164) 2024-03-19T02:06:11.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.829+0000 7f9913fff640 1 -- 172.21.15.55:0/2414473183 learned_addr learned my addr 172.21.15.55:0/2414473183 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:11.832 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.829+0000 7f991883c640 1 --2- 172.21.15.55:0/2414473183 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9914154460 0x7f99140c3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:11.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.833+0000 7f9913fff640 1 -- 172.21.15.55:0/2414473183 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f99140c2760 msgr2=0x7f99141f09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:11.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.833+0000 7f9913fff640 1 --2- 172.21.15.55:0/2414473183 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f99140c2760 0x7f99141f09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:11.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.833+0000 7f9913fff640 1 -- 172.21.15.55:0/2414473183 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9914154460 msgr2=0x7f99140c3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:11.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.833+0000 7f9913fff640 1 --2- 172.21.15.55:0/2414473183 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9914154460 0x7f99140c3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:11.835 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.833+0000 7f9913fff640 1 -- 172.21.15.55:0/2414473183 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99140c99d0 con 0x7f9914073540 2024-03-19T02:06:11.839 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.837+0000 7f9913fff640 1 --2- 172.21.15.55:0/2414473183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9914073540 0x7f99140c9360 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f990401b8e0 tx=0x7f990401ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:11.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.841+0000 7f9910ff9640 1 -- 172.21.15.55:0/2414473183 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99040b5050 con 0x7f9914073540 2024-03-19T02:06:11.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.841+0000 7f9910ff9640 1 -- 172.21.15.55:0/2414473183 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99040bf070 con 0x7f9914073540 2024-03-19T02:06:11.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.841+0000 7f991a29f640 1 -- 172.21.15.55:0/2414473183 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99140c9cb0 con 0x7f9914073540 2024-03-19T02:06:11.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.841+0000 7f9910ff9640 1 -- 172.21.15.55:0/2414473183 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99040b0b70 con 0x7f9914073540 2024-03-19T02:06:11.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.841+0000 7f991a29f640 1 -- 172.21.15.55:0/2414473183 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f991415b850 con 0x7f9914073540 2024-03-19T02:06:11.845 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.845+0000 7f98fa7fc640 1 -- 172.21.15.55:0/2414473183 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98ec0052a0 con 0x7f9914073540 2024-03-19T02:06:11.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.861+0000 7f9910ff9640 1 -- 172.21.15.55:0/2414473183 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f99040b0d10 con 0x7f9914073540 2024-03-19T02:06:11.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.861+0000 7f9910ff9640 1 --2- 172.21.15.55:0/2414473183 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f99000676b0 0x7f9900069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:11.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.861+0000 7f99137fe640 1 --2- 172.21.15.55:0/2414473183 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f99000676b0 0x7f9900069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:11.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:11.861+0000 7f99137fe640 1 --2- 172.21.15.55:0/2414473183 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f99000676b0 0x7f9900069b70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9914150440 tx=0x7f9908007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:12.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.149+0000 7f9910ff9640 1 -- 172.21.15.55:0/2414473183 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f99040a8950 con 0x7f9914073540 2024-03-19T02:06:12.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.149+0000 7f9910ff9640 1 -- 172.21.15.55:0/2414473183 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f99040c9020 con 0x7f9914073540 2024-03-19T02:06:12.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.321+0000 7f98fa7fc640 1 -- 172.21.15.55:0/2414473183 --> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f98ec000bd0 con 0x7f99000676b0 2024-03-19T02:06:12.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.337+0000 7f9910ff9640 1 -- 172.21.15.55:0/2414473183 <== mgr.4105 v2:172.21.15.84:6800/998533783 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332628 (secure 0 0 0) 0x7f98ec000bd0 con 0x7f99000676b0 2024-03-19T02:06:12.339 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:06:12.361 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-19T02:06:12.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.341+0000 7f98fa7fc640 1 -- 172.21.15.55:0/2414473183 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f99000676b0 msgr2=0x7f9900069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:12.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.341+0000 7f98fa7fc640 1 --2- 172.21.15.55:0/2414473183 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f99000676b0 0x7f9900069b70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9914150440 tx=0x7f9908007e40 comp rx=0 tx=0).stop 2024-03-19T02:06:12.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.341+0000 7f98fa7fc640 1 -- 172.21.15.55:0/2414473183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9914073540 msgr2=0x7f99140c9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:12.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.341+0000 7f98fa7fc640 1 --2- 172.21.15.55:0/2414473183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9914073540 0x7f99140c9360 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f990401b8e0 tx=0x7f990401ba70 comp rx=0 tx=0).stop 2024-03-19T02:06:12.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.341+0000 7f98fa7fc640 1 -- 172.21.15.55:0/2414473183 shutdown_connections 2024-03-19T02:06:12.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.341+0000 7f98fa7fc640 1 --2- 172.21.15.55:0/2414473183 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f99000676b0 0x7f9900069b70 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:12.363 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.341+0000 7f98fa7fc640 1 --2- 172.21.15.55:0/2414473183 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9914154460 0x7f99140c3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:12.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.341+0000 7f98fa7fc640 1 --2- 172.21.15.55:0/2414473183 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f99140c2760 0x7f99141f09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:12.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.341+0000 7f98fa7fc640 1 --2- 172.21.15.55:0/2414473183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9914073540 0x7f99140c9360 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:12.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.341+0000 7f98fa7fc640 1 -- 172.21.15.55:0/2414473183 >> 172.21.15.55:0/2414473183 conn(0x7f991406e670 msgr2=0x7f99140cb890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:12.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.341+0000 7f98fa7fc640 1 -- 172.21.15.55:0/2414473183 shutdown_connections 2024-03-19T02:06:12.364 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.341+0000 7f98fa7fc640 1 -- 172.21.15.55:0/2414473183 wait complete. 2024-03-19T02:06:12.365 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-03-19T02:06:10.603886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":368,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":306,"ondisk_log_size":306,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237212,"kb_used_data":4392,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737508,"statfs":{"total":773094113280,"available":772851208192,"internally_reserved":0,"allocated":4497408,"data_stored":2535616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004437"},"pg_stats":[{"pgid":"2.1f","version":"29'13","reported_seq":61,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.576125+0000","last_change":"2024-03-19T01:51:06.982533+0000","last_active":"2024-03-19T02:03:38.576125+0000","last_peered":"2024-03-19T02:03:38.576125+0000","last_clean":"2024-03-19T02:03:38.576125+0000","last_became_active":"2024-03-19T01:51:06.981802+0000","last_became_peered":"2024-03-19T01:51:06.981802+0000","last_unstale":"2024-03-19T02:03:38.576125+0000","last_undegraded":"2024-03-19T02:03:38.576125+0000","last_fullsized":"2024-03-19T02:03:38.576125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:10:01.431003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:11.629408+0000","last_change":"2024-03-19T01:51:09.172057+0000","last_active":"2024-03-19T02:03:11.629408+0000","last_peered":"2024-03-19T02:03:11.629408+0000","last_clean":"2024-03-19T02:03:11.629408+0000","last_became_active":"2024-03-19T01:51:09.171753+0000","last_became_peered":"2024-03-19T01:51:09.171753+0000","last_unstale":"2024-03-19T02:03:11.629408+0000","last_undegraded":"2024-03-19T02:03:11.629408+0000","last_fullsized":"2024-03-19T02:03:11.629408+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:50.575804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:21.630796+0000","last_change":"2024-03-19T01:51:16.537753+0000","last_active":"2024-03-19T02:03:21.630796+0000","last_peered":"2024-03-19T02:03:21.630796+0000","last_clean":"2024-03-19T02:03:21.630796+0000","last_became_active":"2024-03-19T01:51:16.534200+0000","last_became_peered":"2024-03-19T01:51:16.534200+0000","last_unstale":"2024-03-19T02:03:21.630796+0000","last_undegraded":"2024-03-19T02:03:21.630796+0000","last_fullsized":"2024-03-19T02:03:21.630796+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:18:34.208854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":50,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:06.628579+0000","last_change":"2024-03-19T01:51:06.962968+0000","last_active":"2024-03-19T02:03:06.628579+0000","last_peered":"2024-03-19T02:03:06.628579+0000","last_clean":"2024-03-19T02:03:06.628579+0000","last_became_active":"2024-03-19T01:51:06.961856+0000","last_became_peered":"2024-03-19T01:51:06.961856+0000","last_unstale":"2024-03-19T02:03:06.628579+0000","last_undegraded":"2024-03-19T02:03:06.628579+0000","last_fullsized":"2024-03-19T02:03:06.628579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:49.676651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.576098+0000","last_change":"2024-03-19T01:51:09.160338+0000","last_active":"2024-03-19T02:03:38.576098+0000","last_peered":"2024-03-19T02:03:38.576098+0000","last_clean":"2024-03-19T02:03:38.576098+0000","last_became_active":"2024-03-19T01:51:09.158682+0000","last_became_peered":"2024-03-19T01:51:09.158682+0000","last_unstale":"2024-03-19T02:03:38.576098+0000","last_undegraded":"2024-03-19T02:03:38.576098+0000","last_fullsized":"2024-03-19T02:03:38.576098+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":39,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:23.616915+0000","last_change":"2024-03-19T01:52:14.299568+0000","last_active":"2024-03-19T02:03:23.616915+0000","last_peered":"2024-03-19T02:03:23.616915+0000","last_clean":"2024-03-19T02:03:23.616915+0000","last_became_active":"2024-03-19T01:52:14.299067+0000","last_became_peered":"2024-03-19T01:52:14.299067+0000","last_unstale":"2024-03-19T02:03:23.616915+0000","last_undegraded":"2024-03-19T02:03:23.616915+0000","last_fullsized":"2024-03-19T02:03:23.616915+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:15:10.967874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"35'12","reported_seq":59,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603963+0000","last_change":"2024-03-19T01:51:07.044049+0000","last_active":"2024-03-19T02:03:38.603963+0000","last_peered":"2024-03-19T02:03:38.603963+0000","last_clean":"2024-03-19T02:03:38.603963+0000","last_became_active":"2024-03-19T01:51:07.043209+0000","last_became_peered":"2024-03-19T01:51:07.043209+0000","last_unstale":"2024-03-19T02:03:38.603963+0000","last_undegraded":"2024-03-19T02:03:38.603963+0000","last_fullsized":"2024-03-19T02:03:38.603963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:42.992419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:11.654315+0000","last_change":"2024-03-19T01:51:09.179615+0000","last_active":"2024-03-19T02:01:11.654315+0000","last_peered":"2024-03-19T02:01:11.654315+0000","last_clean":"2024-03-19T02:01:11.654315+0000","last_became_active":"2024-03-19T01:51:09.173962+0000","last_became_peered":"2024-03-19T01:51:09.173962+0000","last_unstale":"2024-03-19T02:01:11.654315+0000","last_undegraded":"2024-03-19T02:01:11.654315+0000","last_fullsized":"2024-03-19T02:01:11.654315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:06:12.676868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":38,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:18.601453+0000","last_change":"2024-03-19T01:53:15.447584+0000","last_active":"2024-03-19T02:01:18.601453+0000","last_peered":"2024-03-19T02:01:18.601453+0000","last_clean":"2024-03-19T02:01:18.601453+0000","last_became_active":"2024-03-19T01:53:15.447002+0000","last_became_peered":"2024-03-19T01:53:15.447002+0000","last_unstale":"2024-03-19T02:01:18.601453+0000","last_undegraded":"2024-03-19T02:01:18.601453+0000","last_fullsized":"2024-03-19T02:01:18.601453+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:02:57.393473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6,1],"acting":[4,2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"29'5","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603940+0000","last_change":"2024-03-19T01:51:06.983291+0000","last_active":"2024-03-19T02:03:38.603940+0000","last_peered":"2024-03-19T02:03:38.603940+0000","last_clean":"2024-03-19T02:03:38.603940+0000","last_became_active":"2024-03-19T01:51:06.982309+0000","last_became_peered":"2024-03-19T01:51:06.982309+0000","last_unstale":"2024-03-19T02:03:38.603940+0000","last_undegraded":"2024-03-19T02:03:38.603940+0000","last_fullsized":"2024-03-19T02:03:38.603940+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:27.305286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:11.654335+0000","last_change":"2024-03-19T01:51:09.160738+0000","last_active":"2024-03-19T02:01:11.654335+0000","last_peered":"2024-03-19T02:01:11.654335+0000","last_clean":"2024-03-19T02:01:11.654335+0000","last_became_active":"2024-03-19T01:51:09.159951+0000","last_became_peered":"2024-03-19T01:51:09.159951+0000","last_unstale":"2024-03-19T02:01:11.654335+0000","last_undegraded":"2024-03-19T02:01:11.654335+0000","last_fullsized":"2024-03-19T02:01:11.654335+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:30.620041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":31,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:18.601421+0000","last_change":"2024-03-19T01:51:16.577378+0000","last_active":"2024-03-19T02:01:18.601421+0000","last_peered":"2024-03-19T02:01:18.601421+0000","last_clean":"2024-03-19T02:01:18.601421+0000","last_became_active":"2024-03-19T01:51:16.576256+0000","last_became_peered":"2024-03-19T01:51:16.576256+0000","last_unstale":"2024-03-19T02:01:18.601421+0000","last_undegraded":"2024-03-19T02:01:18.601421+0000","last_fullsized":"2024-03-19T02:01:18.601421+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:16.096547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.494171+0000","last_change":"2024-03-19T01:51:09.185969+0000","last_active":"2024-03-19T02:01:58.494171+0000","last_peered":"2024-03-19T02:01:58.494171+0000","last_clean":"2024-03-19T02:01:58.494171+0000","last_became_active":"2024-03-19T01:51:09.185378+0000","last_became_peered":"2024-03-19T01:51:09.185378+0000","last_unstale":"2024-03-19T02:01:58.494171+0000","last_undegraded":"2024-03-19T02:01:58.494171+0000","last_fullsized":"2024-03-19T02:01:58.494171+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":51,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.594329+0000","last_change":"2024-03-19T01:51:06.988067+0000","last_active":"2024-03-19T02:02:23.594329+0000","last_peered":"2024-03-19T02:02:23.594329+0000","last_clean":"2024-03-19T02:02:23.594329+0000","last_became_active":"2024-03-19T01:51:06.987450+0000","last_became_peered":"2024-03-19T01:51:06.987450+0000","last_unstale":"2024-03-19T02:02:23.594329+0000","last_undegraded":"2024-03-19T02:02:23.594329+0000","last_fullsized":"2024-03-19T02:02:23.594329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:56.962922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:33.610729+0000","last_change":"2024-03-19T01:51:16.583190+0000","last_active":"2024-03-19T02:02:33.610729+0000","last_peered":"2024-03-19T02:02:33.610729+0000","last_clean":"2024-03-19T02:02:33.610729+0000","last_became_active":"2024-03-19T01:51:16.582532+0000","last_became_peered":"2024-03-19T01:51:16.582532+0000","last_unstale":"2024-03-19T02:02:33.610729+0000","last_undegraded":"2024-03-19T02:02:33.610729+0000","last_fullsized":"2024-03-19T02:02:33.610729+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:16:49.817940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:26.624005+0000","last_change":"2024-03-19T01:51:09.174687+0000","last_active":"2024-03-19T02:02:26.624005+0000","last_peered":"2024-03-19T02:02:26.624005+0000","last_clean":"2024-03-19T02:02:26.624005+0000","last_became_active":"2024-03-19T01:51:09.174409+0000","last_became_peered":"2024-03-19T01:51:09.174409+0000","last_unstale":"2024-03-19T02:02:26.624005+0000","last_undegraded":"2024-03-19T02:02:26.624005+0000","last_fullsized":"2024-03-19T02:02:26.624005+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:33:07.160350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"35'3","reported_seq":51,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.504638+0000","last_change":"2024-03-19T01:51:06.955434+0000","last_active":"2024-03-19T02:01:58.504638+0000","last_peered":"2024-03-19T02:01:58.504638+0000","last_clean":"2024-03-19T02:01:58.504638+0000","last_became_active":"2024-03-19T01:51:06.954635+0000","last_became_peered":"2024-03-19T01:51:06.954635+0000","last_unstale":"2024-03-19T02:01:58.504638+0000","last_undegraded":"2024-03-19T02:01:58.504638+0000","last_fullsized":"2024-03-19T02:01:58.504638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:41:39.176905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:33.610667+0000","last_change":"2024-03-19T01:51:16.588604+0000","last_active":"2024-03-19T02:02:33.610667+0000","last_peered":"2024-03-19T02:02:33.610667+0000","last_clean":"2024-03-19T02:02:33.610667+0000","last_became_active":"2024-03-19T01:51:16.587483+0000","last_became_peered":"2024-03-19T01:51:16.587483+0000","last_unstale":"2024-03-19T02:02:33.610667+0000","last_undegraded":"2024-03-19T02:02:33.610667+0000","last_fullsized":"2024-03-19T02:02:33.610667+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:51.658547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:01.620586+0000","last_change":"2024-03-19T01:51:09.173987+0000","last_active":"2024-03-19T02:02:01.620586+0000","last_peered":"2024-03-19T02:02:01.620586+0000","last_clean":"2024-03-19T02:02:01.620586+0000","last_became_active":"2024-03-19T01:51:09.173648+0000","last_became_peered":"2024-03-19T01:51:09.173648+0000","last_unstale":"2024-03-19T02:02:01.620586+0000","last_undegraded":"2024-03-19T02:02:01.620586+0000","last_fullsized":"2024-03-19T02:02:01.620586+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:13:17.277014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"29'7","reported_seq":55,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.504602+0000","last_change":"2024-03-19T01:51:06.981820+0000","last_active":"2024-03-19T02:01:58.504602+0000","last_peered":"2024-03-19T02:01:58.504602+0000","last_clean":"2024-03-19T02:01:58.504602+0000","last_became_active":"2024-03-19T01:51:06.981087+0000","last_became_peered":"2024-03-19T01:51:06.981087+0000","last_unstale":"2024-03-19T02:01:58.504602+0000","last_undegraded":"2024-03-19T02:01:58.504602+0000","last_fullsized":"2024-03-19T02:01:58.504602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:25:59.116138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:33.509725+0000","last_change":"2024-03-19T01:51:16.579017+0000","last_active":"2024-03-19T02:02:33.509725+0000","last_peered":"2024-03-19T02:02:33.509725+0000","last_clean":"2024-03-19T02:02:33.509725+0000","last_became_active":"2024-03-19T01:51:16.573845+0000","last_became_peered":"2024-03-19T01:51:16.573845+0000","last_unstale":"2024-03-19T02:02:33.509725+0000","last_undegraded":"2024-03-19T02:02:33.509725+0000","last_fullsized":"2024-03-19T02:02:33.509725+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:42:30.142766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.607044+0000","last_change":"2024-03-19T01:51:09.192886+0000","last_active":"2024-03-19T02:01:58.607044+0000","last_peered":"2024-03-19T02:01:58.607044+0000","last_clean":"2024-03-19T02:01:58.607044+0000","last_became_active":"2024-03-19T01:51:09.185970+0000","last_became_peered":"2024-03-19T01:51:09.185970+0000","last_unstale":"2024-03-19T02:01:58.607044+0000","last_undegraded":"2024-03-19T02:01:58.607044+0000","last_fullsized":"2024-03-19T02:01:58.607044+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:05.196985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":49,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.591524+0000","last_change":"2024-03-19T01:51:06.984875+0000","last_active":"2024-03-19T02:01:58.591524+0000","last_peered":"2024-03-19T02:01:58.591524+0000","last_clean":"2024-03-19T02:01:58.591524+0000","last_became_active":"2024-03-19T01:51:06.983858+0000","last_became_peered":"2024-03-19T01:51:06.983858+0000","last_unstale":"2024-03-19T02:01:58.591524+0000","last_undegraded":"2024-03-19T02:01:58.591524+0000","last_fullsized":"2024-03-19T02:01:58.591524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:14.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:33.610746+0000","last_change":"2024-03-19T01:51:16.591614+0000","last_active":"2024-03-19T02:02:33.610746+0000","last_peered":"2024-03-19T02:02:33.610746+0000","last_clean":"2024-03-19T02:02:33.610746+0000","last_became_active":"2024-03-19T01:51:16.590968+0000","last_became_peered":"2024-03-19T01:51:16.590968+0000","last_unstale":"2024-03-19T02:02:33.610746+0000","last_undegraded":"2024-03-19T02:02:33.610746+0000","last_fullsized":"2024-03-19T02:02:33.610746+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:50.144136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.564107+0000","last_change":"2024-03-19T01:51:09.168941+0000","last_active":"2024-03-19T02:01:58.564107+0000","last_peered":"2024-03-19T02:01:58.564107+0000","last_clean":"2024-03-19T02:01:58.564107+0000","last_became_active":"2024-03-19T01:51:09.168243+0000","last_became_peered":"2024-03-19T01:51:09.168243+0000","last_unstale":"2024-03-19T02:01:58.564107+0000","last_undegraded":"2024-03-19T02:01:58.564107+0000","last_fullsized":"2024-03-19T02:01:58.564107+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'7","reported_seq":54,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.494241+0000","last_change":"2024-03-19T01:51:07.042870+0000","last_active":"2024-03-19T02:01:58.494241+0000","last_peered":"2024-03-19T02:01:58.494241+0000","last_clean":"2024-03-19T02:01:58.494241+0000","last_became_active":"2024-03-19T01:51:07.042327+0000","last_became_peered":"2024-03-19T01:51:07.042327+0000","last_unstale":"2024-03-19T02:01:58.494241+0000","last_undegraded":"2024-03-19T02:01:58.494241+0000","last_fullsized":"2024-03-19T02:01:58.494241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:13.137673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.619433+0000","last_change":"2024-03-19T01:51:16.587730+0000","last_active":"2024-03-19T02:03:48.619433+0000","last_peered":"2024-03-19T02:03:48.619433+0000","last_clean":"2024-03-19T02:03:48.619433+0000","last_became_active":"2024-03-19T01:51:16.586614+0000","last_became_peered":"2024-03-19T01:51:16.586614+0000","last_unstale":"2024-03-19T02:03:48.619433+0000","last_undegraded":"2024-03-19T02:03:48.619433+0000","last_fullsized":"2024-03-19T02:03:48.619433+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:44.830637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:01.620647+0000","last_change":"2024-03-19T01:51:09.186397+0000","last_active":"2024-03-19T02:02:01.620647+0000","last_peered":"2024-03-19T02:02:01.620647+0000","last_clean":"2024-03-19T02:02:01.620647+0000","last_became_active":"2024-03-19T01:51:09.186033+0000","last_became_peered":"2024-03-19T01:51:09.186033+0000","last_unstale":"2024-03-19T02:02:01.620647+0000","last_undegraded":"2024-03-19T02:02:01.620647+0000","last_fullsized":"2024-03-19T02:02:01.620647+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:03:04.997711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"29'7","reported_seq":54,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.504687+0000","last_change":"2024-03-19T01:51:07.043697+0000","last_active":"2024-03-19T02:01:58.504687+0000","last_peered":"2024-03-19T02:01:58.504687+0000","last_clean":"2024-03-19T02:01:58.504687+0000","last_became_active":"2024-03-19T01:51:07.043185+0000","last_became_peered":"2024-03-19T01:51:07.043185+0000","last_unstale":"2024-03-19T02:01:58.504687+0000","last_undegraded":"2024-03-19T02:01:58.504687+0000","last_fullsized":"2024-03-19T02:01:58.504687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:19.016579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:16.630056+0000","last_change":"2024-03-19T01:51:16.535771+0000","last_active":"2024-03-19T02:03:16.630056+0000","last_peered":"2024-03-19T02:03:16.630056+0000","last_clean":"2024-03-19T02:03:16.630056+0000","last_became_active":"2024-03-19T01:51:16.533661+0000","last_became_peered":"2024-03-19T01:51:16.533661+0000","last_unstale":"2024-03-19T02:03:16.630056+0000","last_undegraded":"2024-03-19T02:03:16.630056+0000","last_fullsized":"2024-03-19T02:03:16.630056+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:55.283782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.518547+0000","last_change":"2024-03-19T01:51:09.157526+0000","last_active":"2024-03-19T02:03:38.518547+0000","last_peered":"2024-03-19T02:03:38.518547+0000","last_clean":"2024-03-19T02:03:38.518547+0000","last_became_active":"2024-03-19T01:51:09.156336+0000","last_became_peered":"2024-03-19T01:51:09.156336+0000","last_unstale":"2024-03-19T02:03:38.518547+0000","last_undegraded":"2024-03-19T02:03:38.518547+0000","last_fullsized":"2024-03-19T02:03:38.518547+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'4","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603980+0000","last_change":"2024-03-19T01:51:06.982994+0000","last_active":"2024-03-19T02:03:38.603980+0000","last_peered":"2024-03-19T02:03:38.603980+0000","last_clean":"2024-03-19T02:03:38.603980+0000","last_became_active":"2024-03-19T01:51:06.982133+0000","last_became_peered":"2024-03-19T01:51:06.982133+0000","last_unstale":"2024-03-19T02:03:38.603980+0000","last_undegraded":"2024-03-19T02:03:38.603980+0000","last_fullsized":"2024-03-19T02:03:38.603980+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:03.574098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.577343+0000","last_change":"2024-03-19T01:51:16.578493+0000","last_active":"2024-03-19T02:03:48.577343+0000","last_peered":"2024-03-19T02:03:48.577343+0000","last_clean":"2024-03-19T02:03:48.577343+0000","last_became_active":"2024-03-19T01:51:16.578040+0000","last_became_peered":"2024-03-19T01:51:16.578040+0000","last_unstale":"2024-03-19T02:03:48.577343+0000","last_undegraded":"2024-03-19T02:03:48.577343+0000","last_fullsized":"2024-03-19T02:03:48.577343+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:45.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:41.673779+0000","last_change":"2024-03-19T01:51:09.180758+0000","last_active":"2024-03-19T02:03:41.673779+0000","last_peered":"2024-03-19T02:03:41.673779+0000","last_clean":"2024-03-19T02:03:41.673779+0000","last_became_active":"2024-03-19T01:51:09.180326+0000","last_became_peered":"2024-03-19T01:51:09.180326+0000","last_unstale":"2024-03-19T02:03:41.673779+0000","last_undegraded":"2024-03-19T02:03:41.673779+0000","last_fullsized":"2024-03-19T02:03:41.673779+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:56:07.046285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"29'1","reported_seq":49,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.518603+0000","last_change":"2024-03-19T01:51:06.962643+0000","last_active":"2024-03-19T02:03:38.518603+0000","last_peered":"2024-03-19T02:03:38.518603+0000","last_clean":"2024-03-19T02:03:38.518603+0000","last_became_active":"2024-03-19T01:51:06.961862+0000","last_became_peered":"2024-03-19T01:51:06.961862+0000","last_unstale":"2024-03-19T02:03:38.518603+0000","last_undegraded":"2024-03-19T02:03:38.518603+0000","last_fullsized":"2024-03-19T02:03:38.518603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.520313+0000","last_change":"2024-03-19T01:51:16.577730+0000","last_active":"2024-03-19T02:03:48.520313+0000","last_peered":"2024-03-19T02:03:48.520313+0000","last_clean":"2024-03-19T02:03:48.520313+0000","last_became_active":"2024-03-19T01:51:16.576955+0000","last_became_peered":"2024-03-19T01:51:16.576955+0000","last_unstale":"2024-03-19T02:03:48.520313+0000","last_undegraded":"2024-03-19T02:03:48.520313+0000","last_fullsized":"2024-03-19T02:03:48.520313+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:35.409504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.618478+0000","last_change":"2024-03-19T01:51:09.183678+0000","last_active":"2024-03-19T02:03:38.618478+0000","last_peered":"2024-03-19T02:03:38.618478+0000","last_clean":"2024-03-19T02:03:38.618478+0000","last_became_active":"2024-03-19T01:51:09.176780+0000","last_became_peered":"2024-03-19T01:51:09.176780+0000","last_unstale":"2024-03-19T02:03:38.618478+0000","last_undegraded":"2024-03-19T02:03:38.618478+0000","last_fullsized":"2024-03-19T02:03:38.618478+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:11:02.502333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'5","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:36.673161+0000","last_change":"2024-03-19T01:51:06.956307+0000","last_active":"2024-03-19T02:03:36.673161+0000","last_peered":"2024-03-19T02:03:36.673161+0000","last_clean":"2024-03-19T02:03:36.673161+0000","last_became_active":"2024-03-19T01:51:06.955529+0000","last_became_peered":"2024-03-19T01:51:06.955529+0000","last_unstale":"2024-03-19T02:03:36.673161+0000","last_undegraded":"2024-03-19T02:03:36.673161+0000","last_fullsized":"2024-03-19T02:03:36.673161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:10.486012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.520349+0000","last_change":"2024-03-19T01:51:16.550718+0000","last_active":"2024-03-19T02:03:48.520349+0000","last_peered":"2024-03-19T02:03:48.520349+0000","last_clean":"2024-03-19T02:03:48.520349+0000","last_became_active":"2024-03-19T01:51:16.550115+0000","last_became_peered":"2024-03-19T01:51:16.550115+0000","last_unstale":"2024-03-19T02:03:48.520349+0000","last_undegraded":"2024-03-19T02:03:48.520349+0000","last_fullsized":"2024-03-19T02:03:48.520349+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:43.172796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:08.514405+0000","last_change":"2024-03-19T01:51:09.155718+0000","last_active":"2024-03-19T02:03:08.514405+0000","last_peered":"2024-03-19T02:03:08.514405+0000","last_clean":"2024-03-19T02:03:08.514405+0000","last_became_active":"2024-03-19T01:51:09.155014+0000","last_became_peered":"2024-03-19T01:51:09.155014+0000","last_unstale":"2024-03-19T02:03:08.514405+0000","last_undegraded":"2024-03-19T02:03:08.514405+0000","last_fullsized":"2024-03-19T02:03:08.514405+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"33'6","reported_seq":54,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:36.559085+0000","last_change":"2024-03-19T01:51:06.955372+0000","last_active":"2024-03-19T02:03:36.559085+0000","last_peered":"2024-03-19T02:03:36.559085+0000","last_clean":"2024-03-19T02:03:36.559085+0000","last_became_active":"2024-03-19T01:51:06.954637+0000","last_became_peered":"2024-03-19T01:51:06.954637+0000","last_unstale":"2024-03-19T02:03:36.559085+0000","last_undegraded":"2024-03-19T02:03:36.559085+0000","last_fullsized":"2024-03-19T02:03:36.559085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:38:20.129564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:08.505884+0000","last_change":"2024-03-19T01:51:16.575157+0000","last_active":"2024-03-19T02:02:08.505884+0000","last_peered":"2024-03-19T02:02:08.505884+0000","last_clean":"2024-03-19T02:02:08.505884+0000","last_became_active":"2024-03-19T01:51:16.574575+0000","last_became_peered":"2024-03-19T01:51:16.574575+0000","last_unstale":"2024-03-19T02:02:08.505884+0000","last_undegraded":"2024-03-19T02:02:08.505884+0000","last_fullsized":"2024-03-19T02:02:08.505884+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:15:17.718630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.575915+0000","last_change":"2024-03-19T01:51:09.192610+0000","last_active":"2024-03-19T02:03:38.575915+0000","last_peered":"2024-03-19T02:03:38.575915+0000","last_clean":"2024-03-19T02:03:38.575915+0000","last_became_active":"2024-03-19T01:51:09.192280+0000","last_became_peered":"2024-03-19T01:51:09.192280+0000","last_unstale":"2024-03-19T02:03:38.575915+0000","last_undegraded":"2024-03-19T02:03:38.575915+0000","last_fullsized":"2024-03-19T02:03:38.575915+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:15.454555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"33'5","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:08.600010+0000","last_change":"2024-03-19T01:51:06.986895+0000","last_active":"2024-03-19T02:03:08.600010+0000","last_peered":"2024-03-19T02:03:08.600010+0000","last_clean":"2024-03-19T02:03:08.600010+0000","last_became_active":"2024-03-19T01:51:06.985965+0000","last_became_peered":"2024-03-19T01:51:06.985965+0000","last_unstale":"2024-03-19T02:03:08.600010+0000","last_undegraded":"2024-03-19T02:03:08.600010+0000","last_fullsized":"2024-03-19T02:03:08.600010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:08.565447+0000","last_change":"2024-03-19T01:51:16.505278+0000","last_active":"2024-03-19T02:02:08.565447+0000","last_peered":"2024-03-19T02:02:08.565447+0000","last_clean":"2024-03-19T02:02:08.565447+0000","last_became_active":"2024-03-19T01:51:16.504908+0000","last_became_peered":"2024-03-19T01:51:16.504908+0000","last_unstale":"2024-03-19T02:02:08.565447+0000","last_undegraded":"2024-03-19T02:02:08.565447+0000","last_fullsized":"2024-03-19T02:02:08.565447+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:42:30.142766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:41.633422+0000","last_change":"2024-03-19T01:51:09.179725+0000","last_active":"2024-03-19T02:03:41.633422+0000","last_peered":"2024-03-19T02:03:41.633422+0000","last_clean":"2024-03-19T02:03:41.633422+0000","last_became_active":"2024-03-19T01:51:09.179235+0000","last_became_peered":"2024-03-19T01:51:09.179235+0000","last_unstale":"2024-03-19T02:03:41.633422+0000","last_undegraded":"2024-03-19T02:03:41.633422+0000","last_fullsized":"2024-03-19T02:03:41.633422+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:31.395833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'6","reported_seq":55,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:36.673191+0000","last_change":"2024-03-19T01:51:06.953474+0000","last_active":"2024-03-19T02:03:36.673191+0000","last_peered":"2024-03-19T02:03:36.673191+0000","last_clean":"2024-03-19T02:03:36.673191+0000","last_became_active":"2024-03-19T01:51:06.952898+0000","last_became_peered":"2024-03-19T01:51:06.952898+0000","last_unstale":"2024-03-19T02:03:36.673191+0000","last_undegraded":"2024-03-19T02:03:36.673191+0000","last_fullsized":"2024-03-19T02:03:36.673191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:38.808918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":203,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:28.594947+0000","last_change":"2024-03-19T01:50:26.097484+0000","last_active":"2024-03-19T02:02:28.594947+0000","last_peered":"2024-03-19T02:02:28.594947+0000","last_clean":"2024-03-19T02:02:28.594947+0000","last_became_active":"2024-03-19T01:50:26.095496+0000","last_became_peered":"2024-03-19T01:50:26.095496+0000","last_unstale":"2024-03-19T02:02:28.594947+0000","last_undegraded":"2024-03-19T02:02:28.594947+0000","last_fullsized":"2024-03-19T02:02:28.594947+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:50:24.112070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:50:24.112070+0000","last_clean_scrub_stamp":"2024-03-19T01:50:24.112070+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:05:00.843775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":40,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.507121+0000","last_change":"2024-03-19T01:52:14.333085+0000","last_active":"2024-03-19T02:03:48.507121+0000","last_peered":"2024-03-19T02:03:48.507121+0000","last_clean":"2024-03-19T02:03:48.507121+0000","last_became_active":"2024-03-19T01:52:14.332547+0000","last_became_peered":"2024-03-19T01:52:14.332547+0000","last_unstale":"2024-03-19T02:03:48.507121+0000","last_undegraded":"2024-03-19T02:03:48.507121+0000","last_fullsized":"2024-03-19T02:03:48.507121+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:36.377719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4,7],"acting":[6,0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.594377+0000","last_change":"2024-03-19T01:51:09.196648+0000","last_active":"2024-03-19T02:02:23.594377+0000","last_peered":"2024-03-19T02:02:23.594377+0000","last_clean":"2024-03-19T02:02:23.594377+0000","last_became_active":"2024-03-19T01:51:09.196093+0000","last_became_peered":"2024-03-19T01:51:09.196093+0000","last_unstale":"2024-03-19T02:02:23.594377+0000","last_undegraded":"2024-03-19T02:02:23.594377+0000","last_fullsized":"2024-03-19T02:02:23.594377+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:54:09.655994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"29'5","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:21.549717+0000","last_change":"2024-03-19T01:51:06.950023+0000","last_active":"2024-03-19T02:02:21.549717+0000","last_peered":"2024-03-19T02:02:21.549717+0000","last_clean":"2024-03-19T02:02:21.549717+0000","last_became_active":"2024-03-19T01:51:06.949234+0000","last_became_peered":"2024-03-19T01:51:06.949234+0000","last_unstale":"2024-03-19T02:02:21.549717+0000","last_undegraded":"2024-03-19T02:02:21.549717+0000","last_fullsized":"2024-03-19T02:02:21.549717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:54:13.131325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:08.495157+0000","last_change":"2024-03-19T01:51:16.554046+0000","last_active":"2024-03-19T02:02:08.495157+0000","last_peered":"2024-03-19T02:02:08.495157+0000","last_clean":"2024-03-19T02:02:08.495157+0000","last_became_active":"2024-03-19T01:51:16.553595+0000","last_became_peered":"2024-03-19T01:51:16.553595+0000","last_unstale":"2024-03-19T02:02:08.495157+0000","last_undegraded":"2024-03-19T02:02:08.495157+0000","last_fullsized":"2024-03-19T02:02:08.495157+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:25.684226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:26.557693+0000","last_change":"2024-03-19T01:52:12.918926+0000","last_active":"2024-03-19T02:03:26.557693+0000","last_peered":"2024-03-19T02:03:26.557693+0000","last_clean":"2024-03-19T02:03:26.557693+0000","last_became_active":"2024-03-19T01:52:12.918458+0000","last_became_peered":"2024-03-19T01:52:12.918458+0000","last_unstale":"2024-03-19T02:03:26.557693+0000","last_undegraded":"2024-03-19T02:03:26.557693+0000","last_fullsized":"2024-03-19T02:03:26.557693+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:56.318137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"29'3","reported_seq":50,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.508282+0000","last_change":"2024-03-19T01:51:06.938396+0000","last_active":"2024-03-19T02:02:23.508282+0000","last_peered":"2024-03-19T02:02:23.508282+0000","last_clean":"2024-03-19T02:02:23.508282+0000","last_became_active":"2024-03-19T01:51:06.937580+0000","last_became_peered":"2024-03-19T01:51:06.937580+0000","last_unstale":"2024-03-19T02:02:23.508282+0000","last_undegraded":"2024-03-19T02:02:23.508282+0000","last_fullsized":"2024-03-19T02:02:23.508282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:08.307376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:33.568500+0000","last_change":"2024-03-19T01:51:16.582111+0000","last_active":"2024-03-19T02:02:33.568500+0000","last_peered":"2024-03-19T02:02:33.568500+0000","last_clean":"2024-03-19T02:02:33.568500+0000","last_became_active":"2024-03-19T01:51:16.581746+0000","last_became_peered":"2024-03-19T01:51:16.581746+0000","last_unstale":"2024-03-19T02:02:33.568500+0000","last_undegraded":"2024-03-19T02:02:33.568500+0000","last_fullsized":"2024-03-19T02:02:33.568500+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:19.442287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.594430+0000","last_change":"2024-03-19T01:51:09.195225+0000","last_active":"2024-03-19T02:02:23.594430+0000","last_peered":"2024-03-19T02:02:23.594430+0000","last_clean":"2024-03-19T02:02:23.594430+0000","last_became_active":"2024-03-19T01:51:09.194645+0000","last_became_peered":"2024-03-19T01:51:09.194645+0000","last_unstale":"2024-03-19T02:02:23.594430+0000","last_undegraded":"2024-03-19T02:02:23.594430+0000","last_fullsized":"2024-03-19T02:02:23.594430+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:50:04.024767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"29'5","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:21.549762+0000","last_change":"2024-03-19T01:51:06.955976+0000","last_active":"2024-03-19T02:02:21.549762+0000","last_peered":"2024-03-19T02:02:21.549762+0000","last_clean":"2024-03-19T02:02:21.549762+0000","last_became_active":"2024-03-19T01:51:06.954951+0000","last_became_peered":"2024-03-19T01:51:06.954951+0000","last_unstale":"2024-03-19T02:02:21.549762+0000","last_undegraded":"2024-03-19T02:02:21.549762+0000","last_fullsized":"2024-03-19T02:02:21.549762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:47.536586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:06.661555+0000","last_change":"2024-03-19T01:51:16.586904+0000","last_active":"2024-03-19T02:02:06.661555+0000","last_peered":"2024-03-19T02:02:06.661555+0000","last_clean":"2024-03-19T02:02:06.661555+0000","last_became_active":"2024-03-19T01:51:16.586298+0000","last_became_peered":"2024-03-19T01:51:16.586298+0000","last_unstale":"2024-03-19T02:02:06.661555+0000","last_undegraded":"2024-03-19T02:02:06.661555+0000","last_fullsized":"2024-03-19T02:02:06.661555+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:42.703415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.594405+0000","last_change":"2024-03-19T01:51:09.195400+0000","last_active":"2024-03-19T02:02:23.594405+0000","last_peered":"2024-03-19T02:02:23.594405+0000","last_clean":"2024-03-19T02:02:23.594405+0000","last_became_active":"2024-03-19T01:51:09.195111+0000","last_became_peered":"2024-03-19T01:51:09.195111+0000","last_unstale":"2024-03-19T02:02:23.594405+0000","last_undegraded":"2024-03-19T02:02:23.594405+0000","last_fullsized":"2024-03-19T02:02:23.594405+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:12.084475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"35'2","reported_seq":49,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:40.608950+0000","last_change":"2024-03-19T01:51:06.963650+0000","last_active":"2024-03-19T02:03:40.608950+0000","last_peered":"2024-03-19T02:03:40.608950+0000","last_clean":"2024-03-19T02:03:40.608950+0000","last_became_active":"2024-03-19T01:51:06.963028+0000","last_became_peered":"2024-03-19T01:51:06.963028+0000","last_unstale":"2024-03-19T02:03:40.608950+0000","last_undegraded":"2024-03-19T02:03:40.608950+0000","last_fullsized":"2024-03-19T02:03:40.608950+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:59:26.369856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:08.608117+0000","last_change":"2024-03-19T01:51:16.588077+0000","last_active":"2024-03-19T02:02:08.608117+0000","last_peered":"2024-03-19T02:02:08.608117+0000","last_clean":"2024-03-19T02:02:08.608117+0000","last_became_active":"2024-03-19T01:51:16.587459+0000","last_became_peered":"2024-03-19T01:51:16.587459+0000","last_unstale":"2024-03-19T02:02:08.608117+0000","last_undegraded":"2024-03-19T02:02:08.608117+0000","last_fullsized":"2024-03-19T02:02:08.608117+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:52.667114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.597258+0000","last_change":"2024-03-19T01:51:09.195758+0000","last_active":"2024-03-19T02:02:48.597258+0000","last_peered":"2024-03-19T02:02:48.597258+0000","last_clean":"2024-03-19T02:02:48.597258+0000","last_became_active":"2024-03-19T01:51:09.195139+0000","last_became_peered":"2024-03-19T01:51:09.195139+0000","last_unstale":"2024-03-19T02:02:48.597258+0000","last_undegraded":"2024-03-19T02:02:48.597258+0000","last_fullsized":"2024-03-19T02:02:48.597258+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:18.326414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"29'17","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:46.552835+0000","last_change":"2024-03-19T01:51:06.941200+0000","last_active":"2024-03-19T02:02:46.552835+0000","last_peered":"2024-03-19T02:02:46.552835+0000","last_clean":"2024-03-19T02:02:46.552835+0000","last_became_active":"2024-03-19T01:51:06.940594+0000","last_became_peered":"2024-03-19T01:51:06.940594+0000","last_unstale":"2024-03-19T02:02:46.552835+0000","last_undegraded":"2024-03-19T02:02:46.552835+0000","last_fullsized":"2024-03-19T02:02:46.552835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:25:05.442423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:23.574385+0000","last_change":"2024-03-19T01:51:16.548804+0000","last_active":"2024-03-19T02:03:23.574385+0000","last_peered":"2024-03-19T02:03:23.574385+0000","last_clean":"2024-03-19T02:03:23.574385+0000","last_became_active":"2024-03-19T01:51:16.547800+0000","last_became_peered":"2024-03-19T01:51:16.547800+0000","last_unstale":"2024-03-19T02:03:23.574385+0000","last_undegraded":"2024-03-19T02:03:23.574385+0000","last_fullsized":"2024-03-19T02:03:23.574385+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:43.172796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.499920+0000","last_change":"2024-03-19T01:51:09.167987+0000","last_active":"2024-03-19T02:02:48.499920+0000","last_peered":"2024-03-19T02:02:48.499920+0000","last_clean":"2024-03-19T02:02:48.499920+0000","last_became_active":"2024-03-19T01:51:09.167670+0000","last_became_peered":"2024-03-19T01:51:09.167670+0000","last_unstale":"2024-03-19T02:02:48.499920+0000","last_undegraded":"2024-03-19T02:02:48.499920+0000","last_fullsized":"2024-03-19T02:02:48.499920+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"29'5","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.499949+0000","last_change":"2024-03-19T01:51:07.048197+0000","last_active":"2024-03-19T02:02:48.499949+0000","last_peered":"2024-03-19T02:02:48.499949+0000","last_clean":"2024-03-19T02:02:48.499949+0000","last_became_active":"2024-03-19T01:51:07.047785+0000","last_became_peered":"2024-03-19T01:51:07.047785+0000","last_unstale":"2024-03-19T02:02:48.499949+0000","last_undegraded":"2024-03-19T02:02:48.499949+0000","last_fullsized":"2024-03-19T02:02:48.499949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:52.323249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:21.630885+0000","last_change":"2024-03-19T01:51:16.583394+0000","last_active":"2024-03-19T02:03:21.630885+0000","last_peered":"2024-03-19T02:03:21.630885+0000","last_clean":"2024-03-19T02:03:21.630885+0000","last_became_active":"2024-03-19T01:51:16.582860+0000","last_became_peered":"2024-03-19T01:51:16.582860+0000","last_unstale":"2024-03-19T02:03:21.630885+0000","last_undegraded":"2024-03-19T02:03:21.630885+0000","last_fullsized":"2024-03-19T02:03:21.630885+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:37.669591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.597290+0000","last_change":"2024-03-19T01:51:09.195008+0000","last_active":"2024-03-19T02:02:48.597290+0000","last_peered":"2024-03-19T02:02:48.597290+0000","last_clean":"2024-03-19T02:02:48.597290+0000","last_became_active":"2024-03-19T01:51:09.194728+0000","last_became_peered":"2024-03-19T01:51:09.194728+0000","last_unstale":"2024-03-19T02:02:48.597290+0000","last_undegraded":"2024-03-19T02:02:48.597290+0000","last_fullsized":"2024-03-19T02:02:48.597290+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:15:10.360817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"33'6","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:46.666678+0000","last_change":"2024-03-19T01:51:06.956905+0000","last_active":"2024-03-19T02:02:46.666678+0000","last_peered":"2024-03-19T02:02:46.666678+0000","last_clean":"2024-03-19T02:02:46.666678+0000","last_became_active":"2024-03-19T01:51:06.955991+0000","last_became_peered":"2024-03-19T01:51:06.955991+0000","last_unstale":"2024-03-19T02:02:46.666678+0000","last_undegraded":"2024-03-19T02:02:46.666678+0000","last_fullsized":"2024-03-19T02:02:46.666678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:02:45.970853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":39,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:56.627291+0000","last_change":"2024-03-19T01:52:14.332240+0000","last_active":"2024-03-19T02:02:56.627291+0000","last_peered":"2024-03-19T02:02:56.627291+0000","last_clean":"2024-03-19T02:02:56.627291+0000","last_became_active":"2024-03-19T01:52:14.331292+0000","last_became_peered":"2024-03-19T01:52:14.331292+0000","last_unstale":"2024-03-19T02:02:56.627291+0000","last_undegraded":"2024-03-19T02:02:56.627291+0000","last_fullsized":"2024-03-19T02:02:56.627291+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:14:19.465379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.570250+0000","last_change":"2024-03-19T01:51:09.188003+0000","last_active":"2024-03-19T02:02:48.570250+0000","last_peered":"2024-03-19T02:02:48.570250+0000","last_clean":"2024-03-19T02:02:48.570250+0000","last_became_active":"2024-03-19T01:51:09.187275+0000","last_became_peered":"2024-03-19T01:51:09.187275+0000","last_unstale":"2024-03-19T02:02:48.570250+0000","last_undegraded":"2024-03-19T02:02:48.570250+0000","last_fullsized":"2024-03-19T02:02:48.570250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:10.624259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"29'9","reported_seq":56,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.570286+0000","last_change":"2024-03-19T01:51:06.964413+0000","last_active":"2024-03-19T02:02:48.570286+0000","last_peered":"2024-03-19T02:02:48.570286+0000","last_clean":"2024-03-19T02:02:48.570286+0000","last_became_active":"2024-03-19T01:51:06.963763+0000","last_became_peered":"2024-03-19T01:51:06.963763+0000","last_unstale":"2024-03-19T02:02:48.570286+0000","last_undegraded":"2024-03-19T02:02:48.570286+0000","last_fullsized":"2024-03-19T02:02:48.570286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:21.671644+0000","last_change":"2024-03-19T01:51:16.574401+0000","last_active":"2024-03-19T02:03:21.671644+0000","last_peered":"2024-03-19T02:03:21.671644+0000","last_clean":"2024-03-19T02:03:21.671644+0000","last_became_active":"2024-03-19T01:51:16.573744+0000","last_became_peered":"2024-03-19T01:51:16.573744+0000","last_unstale":"2024-03-19T02:03:21.671644+0000","last_undegraded":"2024-03-19T02:03:21.671644+0000","last_fullsized":"2024-03-19T02:03:21.671644+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:41.414631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.600865+0000","last_change":"2024-03-19T01:51:09.187207+0000","last_active":"2024-03-19T02:03:13.600865+0000","last_peered":"2024-03-19T02:03:13.600865+0000","last_clean":"2024-03-19T02:03:13.600865+0000","last_became_active":"2024-03-19T01:51:09.175898+0000","last_became_peered":"2024-03-19T01:51:09.175898+0000","last_unstale":"2024-03-19T02:03:13.600865+0000","last_undegraded":"2024-03-19T02:03:13.600865+0000","last_fullsized":"2024-03-19T02:03:13.600865+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:38.387069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'14","reported_seq":61,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.499975+0000","last_change":"2024-03-19T01:51:07.042663+0000","last_active":"2024-03-19T02:02:48.499975+0000","last_peered":"2024-03-19T02:02:48.499975+0000","last_clean":"2024-03-19T02:02:48.499975+0000","last_became_active":"2024-03-19T01:51:07.041262+0000","last_became_peered":"2024-03-19T01:51:07.041262+0000","last_unstale":"2024-03-19T02:02:48.499975+0000","last_undegraded":"2024-03-19T02:02:48.499975+0000","last_fullsized":"2024-03-19T02:02:48.499975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:58.513211+0000","last_change":"2024-03-19T01:51:16.549744+0000","last_active":"2024-03-19T02:02:58.513211+0000","last_peered":"2024-03-19T02:02:58.513211+0000","last_clean":"2024-03-19T02:02:58.513211+0000","last_became_active":"2024-03-19T01:51:16.549347+0000","last_became_peered":"2024-03-19T01:51:16.549347+0000","last_unstale":"2024-03-19T02:02:58.513211+0000","last_undegraded":"2024-03-19T02:02:58.513211+0000","last_fullsized":"2024-03-19T02:02:58.513211+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:19.442287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.612688+0000","last_change":"2024-03-19T01:51:09.194165+0000","last_active":"2024-03-19T02:02:48.612688+0000","last_peered":"2024-03-19T02:02:48.612688+0000","last_clean":"2024-03-19T02:02:48.612688+0000","last_became_active":"2024-03-19T01:51:09.183607+0000","last_became_peered":"2024-03-19T01:51:09.183607+0000","last_unstale":"2024-03-19T02:02:48.612688+0000","last_undegraded":"2024-03-19T02:02:48.612688+0000","last_fullsized":"2024-03-19T02:02:48.612688+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:09.211794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'5","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.514982+0000","last_change":"2024-03-19T01:51:06.952515+0000","last_active":"2024-03-19T02:03:13.514982+0000","last_peered":"2024-03-19T02:03:13.514982+0000","last_clean":"2024-03-19T02:03:13.514982+0000","last_became_active":"2024-03-19T01:51:06.951700+0000","last_became_peered":"2024-03-19T01:51:06.951700+0000","last_unstale":"2024-03-19T02:03:13.514982+0000","last_undegraded":"2024-03-19T02:03:13.514982+0000","last_fullsized":"2024-03-19T02:03:13.514982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:13.137673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:58.613767+0000","last_change":"2024-03-19T01:51:16.587027+0000","last_active":"2024-03-19T02:02:58.613767+0000","last_peered":"2024-03-19T02:02:58.613767+0000","last_clean":"2024-03-19T02:02:58.613767+0000","last_became_active":"2024-03-19T01:51:16.586514+0000","last_became_peered":"2024-03-19T01:51:16.586514+0000","last_unstale":"2024-03-19T02:02:58.613767+0000","last_undegraded":"2024-03-19T02:02:58.613767+0000","last_fullsized":"2024-03-19T02:02:58.613767+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:06.637617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.573086+0000","last_change":"2024-03-19T01:51:09.172554+0000","last_active":"2024-03-19T02:03:13.573086+0000","last_peered":"2024-03-19T02:03:13.573086+0000","last_clean":"2024-03-19T02:03:13.573086+0000","last_became_active":"2024-03-19T01:51:09.171814+0000","last_became_peered":"2024-03-19T01:51:09.171814+0000","last_unstale":"2024-03-19T02:03:13.573086+0000","last_undegraded":"2024-03-19T02:03:13.573086+0000","last_fullsized":"2024-03-19T02:03:13.573086+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'4","reported_seq":51,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:11.670336+0000","last_change":"2024-03-19T01:51:07.043404+0000","last_active":"2024-03-19T02:03:11.670336+0000","last_peered":"2024-03-19T02:03:11.670336+0000","last_clean":"2024-03-19T02:03:11.670336+0000","last_became_active":"2024-03-19T01:51:07.043069+0000","last_became_peered":"2024-03-19T01:51:07.043069+0000","last_unstale":"2024-03-19T02:03:11.670336+0000","last_undegraded":"2024-03-19T02:03:11.670336+0000","last_fullsized":"2024-03-19T02:03:11.670336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:13:14.960132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:56.627225+0000","last_change":"2024-03-19T01:51:16.581797+0000","last_active":"2024-03-19T02:02:56.627225+0000","last_peered":"2024-03-19T02:02:56.627225+0000","last_clean":"2024-03-19T02:02:56.627225+0000","last_became_active":"2024-03-19T01:51:16.581174+0000","last_became_peered":"2024-03-19T01:51:16.581174+0000","last_unstale":"2024-03-19T02:02:56.627225+0000","last_undegraded":"2024-03-19T02:02:56.627225+0000","last_fullsized":"2024-03-19T02:02:56.627225+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:29:34.911347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:16.671090+0000","last_change":"2024-03-19T01:51:09.194372+0000","last_active":"2024-03-19T02:03:16.671090+0000","last_peered":"2024-03-19T02:03:16.671090+0000","last_clean":"2024-03-19T02:03:16.671090+0000","last_became_active":"2024-03-19T01:51:09.193656+0000","last_became_peered":"2024-03-19T01:51:09.193656+0000","last_unstale":"2024-03-19T02:03:16.671090+0000","last_undegraded":"2024-03-19T02:03:16.671090+0000","last_fullsized":"2024-03-19T02:03:16.671090+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:23.464803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'4","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.502733+0000","last_change":"2024-03-19T01:51:07.046089+0000","last_active":"2024-03-19T02:03:13.502733+0000","last_peered":"2024-03-19T02:03:13.502733+0000","last_clean":"2024-03-19T02:03:13.502733+0000","last_became_active":"2024-03-19T01:51:07.045342+0000","last_became_peered":"2024-03-19T01:51:07.045342+0000","last_unstale":"2024-03-19T02:03:13.502733+0000","last_undegraded":"2024-03-19T02:03:13.502733+0000","last_fullsized":"2024-03-19T02:03:13.502733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:26.105915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:56.627262+0000","last_change":"2024-03-19T01:51:16.572559+0000","last_active":"2024-03-19T02:02:56.627262+0000","last_peered":"2024-03-19T02:02:56.627262+0000","last_clean":"2024-03-19T02:02:56.627262+0000","last_became_active":"2024-03-19T01:51:16.572283+0000","last_became_peered":"2024-03-19T01:51:16.572283+0000","last_unstale":"2024-03-19T02:02:56.627262+0000","last_undegraded":"2024-03-19T02:02:56.627262+0000","last_fullsized":"2024-03-19T02:02:56.627262+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:42.703415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:18.573781+0000","last_change":"2024-03-19T01:51:16.534088+0000","last_active":"2024-03-19T02:03:18.573781+0000","last_peered":"2024-03-19T02:03:18.573781+0000","last_clean":"2024-03-19T02:03:18.573781+0000","last_became_active":"2024-03-19T01:51:16.533665+0000","last_became_peered":"2024-03-19T01:51:16.533665+0000","last_unstale":"2024-03-19T02:03:18.573781+0000","last_undegraded":"2024-03-19T02:03:18.573781+0000","last_fullsized":"2024-03-19T02:03:18.573781+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:35.409504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.514931+0000","last_change":"2024-03-19T01:51:09.168372+0000","last_active":"2024-03-19T02:03:13.514931+0000","last_peered":"2024-03-19T02:03:13.514931+0000","last_clean":"2024-03-19T02:03:13.514931+0000","last_became_active":"2024-03-19T01:51:09.167626+0000","last_became_peered":"2024-03-19T01:51:09.167626+0000","last_unstale":"2024-03-19T02:03:13.514931+0000","last_undegraded":"2024-03-19T02:03:13.514931+0000","last_fullsized":"2024-03-19T02:03:13.514931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"33'4","reported_seq":51,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:11.670233+0000","last_change":"2024-03-19T01:51:06.951225+0000","last_active":"2024-03-19T02:03:11.670233+0000","last_peered":"2024-03-19T02:03:11.670233+0000","last_clean":"2024-03-19T02:03:11.670233+0000","last_became_active":"2024-03-19T01:51:06.950826+0000","last_became_peered":"2024-03-19T01:51:06.950826+0000","last_unstale":"2024-03-19T02:03:11.670233+0000","last_undegraded":"2024-03-19T02:03:11.670233+0000","last_fullsized":"2024-03-19T02:03:11.670233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:24.949925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.507160+0000","last_change":"2024-03-19T01:51:16.581997+0000","last_active":"2024-03-19T02:03:48.507160+0000","last_peered":"2024-03-19T02:03:48.507160+0000","last_clean":"2024-03-19T02:03:48.507160+0000","last_became_active":"2024-03-19T01:51:16.581412+0000","last_became_peered":"2024-03-19T01:51:16.581412+0000","last_unstale":"2024-03-19T02:03:48.507160+0000","last_undegraded":"2024-03-19T02:03:48.507160+0000","last_fullsized":"2024-03-19T02:03:48.507160+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:45.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:16.630092+0000","last_change":"2024-03-19T01:51:09.161686+0000","last_active":"2024-03-19T02:03:16.630092+0000","last_peered":"2024-03-19T02:03:16.630092+0000","last_clean":"2024-03-19T02:03:16.630092+0000","last_became_active":"2024-03-19T01:51:09.160978+0000","last_became_peered":"2024-03-19T01:51:09.160978+0000","last_unstale":"2024-03-19T02:03:16.630092+0000","last_undegraded":"2024-03-19T02:03:16.630092+0000","last_fullsized":"2024-03-19T02:03:16.630092+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:27.266807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'3","reported_seq":50,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:11.629313+0000","last_change":"2024-03-19T01:51:07.047363+0000","last_active":"2024-03-19T02:03:11.629313+0000","last_peered":"2024-03-19T02:03:11.629313+0000","last_clean":"2024-03-19T02:03:11.629313+0000","last_became_active":"2024-03-19T01:51:07.046886+0000","last_became_peered":"2024-03-19T01:51:07.046886+0000","last_unstale":"2024-03-19T02:03:11.629313+0000","last_undegraded":"2024-03-19T02:03:11.629313+0000","last_fullsized":"2024-03-19T02:03:11.629313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:19.418603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"29'7","reported_seq":54,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.505874+0000","last_change":"2024-03-19T01:51:07.045496+0000","last_active":"2024-03-19T02:03:38.505874+0000","last_peered":"2024-03-19T02:03:38.505874+0000","last_clean":"2024-03-19T02:03:38.505874+0000","last_became_active":"2024-03-19T01:51:07.044554+0000","last_became_peered":"2024-03-19T01:51:07.044554+0000","last_unstale":"2024-03-19T02:03:38.505874+0000","last_undegraded":"2024-03-19T02:03:38.505874+0000","last_fullsized":"2024-03-19T02:03:38.505874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:14.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.576001+0000","last_change":"2024-03-19T01:51:09.184934+0000","last_active":"2024-03-19T02:03:38.576001+0000","last_peered":"2024-03-19T02:03:38.576001+0000","last_clean":"2024-03-19T02:03:38.576001+0000","last_became_active":"2024-03-19T01:51:09.184666+0000","last_became_peered":"2024-03-19T01:51:09.184666+0000","last_unstale":"2024-03-19T02:03:38.576001+0000","last_undegraded":"2024-03-19T02:03:38.576001+0000","last_fullsized":"2024-03-19T02:03:38.576001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:42.786323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":38,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:16.540961+0000","last_change":"2024-03-19T01:52:14.298105+0000","last_active":"2024-03-19T02:01:16.540961+0000","last_peered":"2024-03-19T02:01:16.540961+0000","last_clean":"2024-03-19T02:01:16.540961+0000","last_became_active":"2024-03-19T01:52:14.297612+0000","last_became_peered":"2024-03-19T01:52:14.297612+0000","last_unstale":"2024-03-19T02:01:16.540961+0000","last_undegraded":"2024-03-19T02:01:16.540961+0000","last_fullsized":"2024-03-19T02:01:16.540961+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:38.753646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,5],"acting":[2,0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"33'7","reported_seq":57,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:36.632647+0000","last_change":"2024-03-19T01:51:06.982280+0000","last_active":"2024-03-19T02:03:36.632647+0000","last_peered":"2024-03-19T02:03:36.632647+0000","last_clean":"2024-03-19T02:03:36.632647+0000","last_became_active":"2024-03-19T01:51:06.981704+0000","last_became_peered":"2024-03-19T01:51:06.981704+0000","last_unstale":"2024-03-19T02:03:36.632647+0000","last_undegraded":"2024-03-19T02:03:36.632647+0000","last_fullsized":"2024-03-19T02:03:36.632647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:22:12.657985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.618444+0000","last_change":"2024-03-19T01:51:09.184624+0000","last_active":"2024-03-19T02:03:38.618444+0000","last_peered":"2024-03-19T02:03:38.618444+0000","last_clean":"2024-03-19T02:03:38.618444+0000","last_became_active":"2024-03-19T01:51:09.183976+0000","last_became_peered":"2024-03-19T01:51:09.183976+0000","last_unstale":"2024-03-19T02:03:38.618444+0000","last_undegraded":"2024-03-19T02:03:38.618444+0000","last_fullsized":"2024-03-19T02:03:38.618444+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:24:29.313729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":38,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:16.614098+0000","last_change":"2024-03-19T01:52:14.299182+0000","last_active":"2024-03-19T02:01:16.614098+0000","last_peered":"2024-03-19T02:01:16.614098+0000","last_clean":"2024-03-19T02:01:16.614098+0000","last_became_active":"2024-03-19T01:52:14.297976+0000","last_became_peered":"2024-03-19T01:52:14.297976+0000","last_unstale":"2024-03-19T02:01:16.614098+0000","last_undegraded":"2024-03-19T02:01:16.614098+0000","last_fullsized":"2024-03-19T02:01:16.614098+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:31.057362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4,2],"acting":[3,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.505794+0000","last_change":"2024-03-19T01:51:09.169493+0000","last_active":"2024-03-19T02:03:38.505794+0000","last_peered":"2024-03-19T02:03:38.505794+0000","last_clean":"2024-03-19T02:03:38.505794+0000","last_became_active":"2024-03-19T01:51:09.168231+0000","last_became_peered":"2024-03-19T01:51:09.168231+0000","last_unstale":"2024-03-19T02:03:38.505794+0000","last_undegraded":"2024-03-19T02:03:38.505794+0000","last_fullsized":"2024-03-19T02:03:38.505794+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:10.624259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:41.559618+0000","last_change":"2024-03-19T01:51:09.155483+0000","last_active":"2024-03-19T02:03:41.559618+0000","last_peered":"2024-03-19T02:03:41.559618+0000","last_clean":"2024-03-19T02:03:41.559618+0000","last_became_active":"2024-03-19T01:51:09.154545+0000","last_became_peered":"2024-03-19T01:51:09.154545+0000","last_unstale":"2024-03-19T02:03:41.559618+0000","last_undegraded":"2024-03-19T02:03:41.559618+0000","last_fullsized":"2024-03-19T02:03:41.559618+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:25:09.517323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:41.673722+0000","last_change":"2024-03-19T01:51:09.180498+0000","last_active":"2024-03-19T02:03:41.673722+0000","last_peered":"2024-03-19T02:03:41.673722+0000","last_clean":"2024-03-19T02:03:41.673722+0000","last_became_active":"2024-03-19T01:51:09.179840+0000","last_became_peered":"2024-03-19T01:51:09.179840+0000","last_unstale":"2024-03-19T02:03:41.673722+0000","last_undegraded":"2024-03-19T02:03:41.673722+0000","last_fullsized":"2024-03-19T02:03:41.673722+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:01:21.716571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.575976+0000","last_change":"2024-03-19T01:51:09.170389+0000","last_active":"2024-03-19T02:03:38.575976+0000","last_peered":"2024-03-19T02:03:38.575976+0000","last_clean":"2024-03-19T02:03:38.575976+0000","last_became_active":"2024-03-19T01:51:09.163974+0000","last_became_peered":"2024-03-19T01:51:09.163974+0000","last_unstale":"2024-03-19T02:03:38.575976+0000","last_undegraded":"2024-03-19T02:03:38.575976+0000","last_fullsized":"2024-03-19T02:03:38.575976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:01.620620+0000","last_change":"2024-03-19T01:51:09.177060+0000","last_active":"2024-03-19T02:02:01.620620+0000","last_peered":"2024-03-19T02:02:01.620620+0000","last_clean":"2024-03-19T02:02:01.620620+0000","last_became_active":"2024-03-19T01:51:09.176474+0000","last_became_peered":"2024-03-19T01:51:09.176474+0000","last_unstale":"2024-03-19T02:02:01.620620+0000","last_undegraded":"2024-03-19T02:02:01.620620+0000","last_fullsized":"2024-03-19T02:02:01.620620+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:56.318137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.504665+0000","last_change":"2024-03-19T01:51:09.168741+0000","last_active":"2024-03-19T02:01:58.504665+0000","last_peered":"2024-03-19T02:01:58.504665+0000","last_clean":"2024-03-19T02:01:58.504665+0000","last_became_active":"2024-03-19T01:51:09.167982+0000","last_became_peered":"2024-03-19T01:51:09.167982+0000","last_unstale":"2024-03-19T02:01:58.504665+0000","last_undegraded":"2024-03-19T02:01:58.504665+0000","last_fullsized":"2024-03-19T02:01:58.504665+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.494217+0000","last_change":"2024-03-19T01:51:09.189814+0000","last_active":"2024-03-19T02:01:58.494217+0000","last_peered":"2024-03-19T02:01:58.494217+0000","last_clean":"2024-03-19T02:01:58.494217+0000","last_became_active":"2024-03-19T01:51:09.189128+0000","last_became_peered":"2024-03-19T01:51:09.189128+0000","last_unstale":"2024-03-19T02:01:58.494217+0000","last_undegraded":"2024-03-19T02:01:58.494217+0000","last_fullsized":"2024-03-19T02:01:58.494217+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:01.620637+0000","last_change":"2024-03-19T01:51:09.183989+0000","last_active":"2024-03-19T02:02:01.620637+0000","last_peered":"2024-03-19T02:02:01.620637+0000","last_clean":"2024-03-19T02:02:01.620637+0000","last_became_active":"2024-03-19T01:51:09.183417+0000","last_became_peered":"2024-03-19T01:51:09.183417+0000","last_unstale":"2024-03-19T02:02:01.620637+0000","last_undegraded":"2024-03-19T02:02:01.620637+0000","last_fullsized":"2024-03-19T02:02:01.620637+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:31.405618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:01.660628+0000","last_change":"2024-03-19T01:51:09.172717+0000","last_active":"2024-03-19T02:02:01.660628+0000","last_peered":"2024-03-19T02:02:01.660628+0000","last_clean":"2024-03-19T02:02:01.660628+0000","last_became_active":"2024-03-19T01:51:09.165872+0000","last_became_peered":"2024-03-19T01:51:09.165872+0000","last_unstale":"2024-03-19T02:02:01.660628+0000","last_undegraded":"2024-03-19T02:02:01.660628+0000","last_fullsized":"2024-03-19T02:02:01.660628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:10.995492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:26.557659+0000","last_change":"2024-03-19T01:52:12.916894+0000","last_active":"2024-03-19T02:03:26.557659+0000","last_peered":"2024-03-19T02:03:26.557659+0000","last_clean":"2024-03-19T02:03:26.557659+0000","last_became_active":"2024-03-19T01:52:12.916608+0000","last_became_peered":"2024-03-19T01:52:12.916608+0000","last_unstale":"2024-03-19T02:03:26.557659+0000","last_undegraded":"2024-03-19T02:03:26.557659+0000","last_fullsized":"2024-03-19T02:03:26.557659+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:31.405618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:26.664130+0000","last_change":"2024-03-19T01:51:09.195141+0000","last_active":"2024-03-19T02:02:26.664130+0000","last_peered":"2024-03-19T02:02:26.664130+0000","last_clean":"2024-03-19T02:02:26.664130+0000","last_became_active":"2024-03-19T01:51:09.194536+0000","last_became_peered":"2024-03-19T01:51:09.194536+0000","last_unstale":"2024-03-19T02:02:26.664130+0000","last_undegraded":"2024-03-19T02:02:26.664130+0000","last_fullsized":"2024-03-19T02:02:26.664130+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:31.395833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.567024+0000","last_change":"2024-03-19T01:51:09.189197+0000","last_active":"2024-03-19T02:02:23.567024+0000","last_peered":"2024-03-19T02:02:23.567024+0000","last_clean":"2024-03-19T02:02:23.567024+0000","last_became_active":"2024-03-19T01:51:09.188565+0000","last_became_peered":"2024-03-19T01:51:09.188565+0000","last_unstale":"2024-03-19T02:02:23.567024+0000","last_undegraded":"2024-03-19T02:02:23.567024+0000","last_fullsized":"2024-03-19T02:02:23.567024+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.497100+0000","last_change":"2024-03-19T01:51:09.168581+0000","last_active":"2024-03-19T02:02:23.497100+0000","last_peered":"2024-03-19T02:02:23.497100+0000","last_clean":"2024-03-19T02:02:23.497100+0000","last_became_active":"2024-03-19T01:51:09.167812+0000","last_became_peered":"2024-03-19T01:51:09.167812+0000","last_unstale":"2024-03-19T02:02:23.497100+0000","last_undegraded":"2024-03-19T02:02:23.497100+0000","last_fullsized":"2024-03-19T02:02:23.497100+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:28.517089+0000","last_change":"2024-03-19T01:52:12.915401+0000","last_active":"2024-03-19T02:03:28.517089+0000","last_peered":"2024-03-19T02:03:28.517089+0000","last_clean":"2024-03-19T02:03:28.517089+0000","last_became_active":"2024-03-19T01:52:12.915089+0000","last_became_peered":"2024-03-19T01:52:12.915089+0000","last_unstale":"2024-03-19T02:03:28.517089+0000","last_undegraded":"2024-03-19T02:03:28.517089+0000","last_fullsized":"2024-03-19T02:03:28.517089+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:54:01.116865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:51.553572+0000","last_change":"2024-03-19T01:51:09.168451+0000","last_active":"2024-03-19T02:02:51.553572+0000","last_peered":"2024-03-19T02:02:51.553572+0000","last_clean":"2024-03-19T02:02:51.553572+0000","last_became_active":"2024-03-19T01:51:09.167744+0000","last_became_peered":"2024-03-19T01:51:09.167744+0000","last_unstale":"2024-03-19T02:02:51.553572+0000","last_undegraded":"2024-03-19T02:02:51.553572+0000","last_fullsized":"2024-03-19T02:02:51.553572+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:21.735485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:51.667789+0000","last_change":"2024-03-19T01:51:09.196067+0000","last_active":"2024-03-19T02:02:51.667789+0000","last_peered":"2024-03-19T02:02:51.667789+0000","last_clean":"2024-03-19T02:02:51.667789+0000","last_became_active":"2024-03-19T01:51:09.195488+0000","last_became_peered":"2024-03-19T01:51:09.195488+0000","last_unstale":"2024-03-19T02:02:51.667789+0000","last_undegraded":"2024-03-19T02:02:51.667789+0000","last_fullsized":"2024-03-19T02:02:51.667789+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:07:44.120161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:51.667650+0000","last_change":"2024-03-19T01:51:09.193831+0000","last_active":"2024-03-19T02:02:51.667650+0000","last_peered":"2024-03-19T02:02:51.667650+0000","last_clean":"2024-03-19T02:02:51.667650+0000","last_became_active":"2024-03-19T01:51:09.192777+0000","last_became_peered":"2024-03-19T01:51:09.192777+0000","last_unstale":"2024-03-19T02:02:51.667650+0000","last_undegraded":"2024-03-19T02:02:51.667650+0000","last_fullsized":"2024-03-19T02:02:51.667650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:56.318137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.511885+0000","last_change":"2024-03-19T01:51:09.193581+0000","last_active":"2024-03-19T02:02:48.511885+0000","last_peered":"2024-03-19T02:02:48.511885+0000","last_clean":"2024-03-19T02:02:48.511885+0000","last_became_active":"2024-03-19T01:51:09.192677+0000","last_became_peered":"2024-03-19T01:51:09.192677+0000","last_unstale":"2024-03-19T02:02:48.511885+0000","last_undegraded":"2024-03-19T02:02:48.511885+0000","last_fullsized":"2024-03-19T02:02:48.511885+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.570213+0000","last_change":"2024-03-19T01:51:09.165568+0000","last_active":"2024-03-19T02:02:48.570213+0000","last_peered":"2024-03-19T02:02:48.570213+0000","last_clean":"2024-03-19T02:02:48.570213+0000","last_became_active":"2024-03-19T01:51:09.164753+0000","last_became_peered":"2024-03-19T01:51:09.164753+0000","last_unstale":"2024-03-19T02:02:48.570213+0000","last_undegraded":"2024-03-19T02:02:48.570213+0000","last_fullsized":"2024-03-19T02:02:48.570213+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:28.422798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:16.630124+0000","last_change":"2024-03-19T01:51:09.170898+0000","last_active":"2024-03-19T02:03:16.630124+0000","last_peered":"2024-03-19T02:03:16.630124+0000","last_clean":"2024-03-19T02:03:16.630124+0000","last_became_active":"2024-03-19T01:51:09.170153+0000","last_became_peered":"2024-03-19T01:51:09.170153+0000","last_unstale":"2024-03-19T02:03:16.630124+0000","last_undegraded":"2024-03-19T02:03:16.630124+0000","last_fullsized":"2024-03-19T02:03:16.630124+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:41.125800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.615856+0000","last_change":"2024-03-19T01:51:09.185953+0000","last_active":"2024-03-19T02:03:13.615856+0000","last_peered":"2024-03-19T02:03:13.615856+0000","last_clean":"2024-03-19T02:03:13.615856+0000","last_became_active":"2024-03-19T01:51:09.177996+0000","last_became_peered":"2024-03-19T01:51:09.177996+0000","last_unstale":"2024-03-19T02:03:13.615856+0000","last_undegraded":"2024-03-19T02:03:13.615856+0000","last_fullsized":"2024-03-19T02:03:13.615856+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:39:06.045413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.502685+0000","last_change":"2024-03-19T01:51:09.183841+0000","last_active":"2024-03-19T02:03:13.502685+0000","last_peered":"2024-03-19T02:03:13.502685+0000","last_clean":"2024-03-19T02:03:13.502685+0000","last_became_active":"2024-03-19T01:51:09.183045+0000","last_became_peered":"2024-03-19T01:51:09.183045+0000","last_unstale":"2024-03-19T02:03:13.502685+0000","last_undegraded":"2024-03-19T02:03:13.502685+0000","last_fullsized":"2024-03-19T02:03:13.502685+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.502713+0000","last_change":"2024-03-19T01:51:09.184022+0000","last_active":"2024-03-19T02:03:13.502713+0000","last_peered":"2024-03-19T02:03:13.502713+0000","last_clean":"2024-03-19T02:03:13.502713+0000","last_became_active":"2024-03-19T01:51:09.183351+0000","last_became_peered":"2024-03-19T01:51:09.183351+0000","last_unstale":"2024-03-19T02:03:13.502713+0000","last_undegraded":"2024-03-19T02:03:13.502713+0000","last_fullsized":"2024-03-19T02:03:13.502713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.600945+0000","last_change":"2024-03-19T01:52:12.922818+0000","last_active":"2024-03-19T02:03:13.600945+0000","last_peered":"2024-03-19T02:03:13.600945+0000","last_clean":"2024-03-19T02:03:13.600945+0000","last_became_active":"2024-03-19T01:52:12.922473+0000","last_became_peered":"2024-03-19T01:52:12.922473+0000","last_unstale":"2024-03-19T02:03:13.600945+0000","last_undegraded":"2024-03-19T02:03:13.600945+0000","last_fullsized":"2024-03-19T02:03:13.600945+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:35.798061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:11.613507+0000","last_change":"2024-03-19T01:51:09.180013+0000","last_active":"2024-03-19T02:01:11.613507+0000","last_peered":"2024-03-19T02:01:11.613507+0000","last_clean":"2024-03-19T02:01:11.613507+0000","last_became_active":"2024-03-19T01:51:09.179541+0000","last_became_peered":"2024-03-19T01:51:09.179541+0000","last_unstale":"2024-03-19T02:01:11.613507+0000","last_undegraded":"2024-03-19T02:01:11.613507+0000","last_fullsized":"2024-03-19T02:01:11.613507+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:19:25.933028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603886+0000","last_change":"2024-03-19T01:51:09.188671+0000","last_active":"2024-03-19T02:03:38.603886+0000","last_peered":"2024-03-19T02:03:38.603886+0000","last_clean":"2024-03-19T02:03:38.603886+0000","last_became_active":"2024-03-19T01:51:09.177627+0000","last_became_peered":"2024-03-19T01:51:09.177627+0000","last_unstale":"2024-03-19T02:03:38.603886+0000","last_undegraded":"2024-03-19T02:03:38.603886+0000","last_fullsized":"2024-03-19T02:03:38.603886+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:53:50.539837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:11.654275+0000","last_change":"2024-03-19T01:51:09.180735+0000","last_active":"2024-03-19T02:01:11.654275+0000","last_peered":"2024-03-19T02:01:11.654275+0000","last_clean":"2024-03-19T02:01:11.654275+0000","last_became_active":"2024-03-19T01:51:09.173235+0000","last_became_peered":"2024-03-19T01:51:09.173235+0000","last_unstale":"2024-03-19T02:01:11.654275+0000","last_undegraded":"2024-03-19T02:01:11.654275+0000","last_fullsized":"2024-03-19T02:01:11.654275+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:31.405618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603918+0000","last_change":"2024-03-19T01:51:09.188346+0000","last_active":"2024-03-19T02:03:38.603918+0000","last_peered":"2024-03-19T02:03:38.603918+0000","last_clean":"2024-03-19T02:03:38.603918+0000","last_became_active":"2024-03-19T01:51:09.187703+0000","last_became_peered":"2024-03-19T01:51:09.187703+0000","last_unstale":"2024-03-19T02:03:38.603918+0000","last_undegraded":"2024-03-19T02:03:38.603918+0000","last_fullsized":"2024-03-19T02:03:38.603918+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:28.051691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:11.548313+0000","last_change":"2024-03-19T01:52:12.919846+0000","last_active":"2024-03-19T02:02:11.548313+0000","last_peered":"2024-03-19T02:02:11.548313+0000","last_clean":"2024-03-19T02:02:11.548313+0000","last_became_active":"2024-03-19T01:52:12.918835+0000","last_became_peered":"2024-03-19T01:52:12.918835+0000","last_unstale":"2024-03-19T02:02:11.548313+0000","last_undegraded":"2024-03-19T02:02:11.548313+0000","last_fullsized":"2024-03-19T02:02:11.548313+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:03:04.997711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.576027+0000","last_change":"2024-03-19T01:51:09.161476+0000","last_active":"2024-03-19T02:03:38.576027+0000","last_peered":"2024-03-19T02:03:38.576027+0000","last_clean":"2024-03-19T02:03:38.576027+0000","last_became_active":"2024-03-19T01:51:09.158609+0000","last_became_peered":"2024-03-19T01:51:09.158609+0000","last_unstale":"2024-03-19T02:03:38.576027+0000","last_undegraded":"2024-03-19T02:03:38.576027+0000","last_fullsized":"2024-03-19T02:03:38.576027+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.518711+0000","last_change":"2024-03-19T01:51:09.165938+0000","last_active":"2024-03-19T02:03:38.518711+0000","last_peered":"2024-03-19T02:03:38.518711+0000","last_clean":"2024-03-19T02:03:38.518711+0000","last_became_active":"2024-03-19T01:51:09.165207+0000","last_became_peered":"2024-03-19T01:51:09.165207+0000","last_unstale":"2024-03-19T02:03:38.518711+0000","last_undegraded":"2024-03-19T02:03:38.518711+0000","last_fullsized":"2024-03-19T02:03:38.518711+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603999+0000","last_change":"2024-03-19T01:51:09.176023+0000","last_active":"2024-03-19T02:03:38.603999+0000","last_peered":"2024-03-19T02:03:38.603999+0000","last_clean":"2024-03-19T02:03:38.603999+0000","last_became_active":"2024-03-19T01:51:09.175682+0000","last_became_peered":"2024-03-19T01:51:09.175682+0000","last_unstale":"2024-03-19T02:03:38.603999+0000","last_undegraded":"2024-03-19T02:03:38.603999+0000","last_fullsized":"2024-03-19T02:03:38.603999+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:35.814983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":209,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":189,"ondisk_log_size":189,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509633,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343828,"statfs":{"total":96636764160,"available":96608079872,"internally_reserved":0,"allocated":1011712,"data_stored":759262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509635,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":171426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509634,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":168124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509635,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31580,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":409600,"data_stored":168326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509634,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":1003520,"data_stored":758036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542337,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":430080,"data_stored":170726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542338,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31588,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340252,"statfs":{"total":96636764160,"available":96604418048,"internally_reserved":0,"allocated":413696,"data_stored":171460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542337,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31588,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340252,"statfs":{"total":96636764160,"available":96604418048,"internally_reserved":0,"allocated":413696,"data_stored":168256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:06:12.371 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:06:12.523 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.517+0000 7f68c5200640 1 Processor -- start 2024-03-19T02:06:12.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.517+0000 7f68c5200640 1 -- start start 2024-03-19T02:06:12.524 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.521+0000 7f68c5200640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68c01545f0 0x7f68c00ca210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:12.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.521+0000 7f68c5200640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f68c0071050 0x7f68c00c7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:12.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.521+0000 7f68c5200640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f68c015ae00 0x7f68c006b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:12.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.521+0000 7f68c5200640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f68c0072c90 con 0x7f68c0071050 2024-03-19T02:06:12.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.521+0000 7f68c5200640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f68c0073210 con 0x7f68c015ae00 2024-03-19T02:06:12.525 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.521+0000 7f68c5200640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f68c0072f30 con 0x7f68c01545f0 2024-03-19T02:06:12.548 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.545+0000 7f68b7fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f68c0071050 0x7f68c00c7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:12.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.549+0000 7f68b7fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f68c0071050 0x7f68c00c7d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57166/0 (socket says 172.21.15.55:57166) 2024-03-19T02:06:12.549 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.549+0000 7f68b7fff640 1 -- 172.21.15.55:0/2077420729 learned_addr learned my addr 172.21.15.55:0/2077420729 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:12.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.549+0000 7f68b7fff640 1 -- 172.21.15.55:0/2077420729 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68c01545f0 msgr2=0x7f68c00ca210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:12.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.549+0000 7f68b7fff640 1 --2- 172.21.15.55:0/2077420729 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68c01545f0 0x7f68c00ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:12.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.549+0000 7f68b7fff640 1 -- 172.21.15.55:0/2077420729 >> v2:172.21.15.84:3300/0 conn(0x7f68c015ae00 msgr2=0x7f68c006b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:12.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.549+0000 7f68b7fff640 1 --2- 172.21.15.55:0/2077420729 >> v2:172.21.15.84:3300/0 conn(0x7f68c015ae00 0x7f68c006b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:12.552 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.549+0000 7f68b7fff640 1 -- 172.21.15.55:0/2077420729 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68c00c9730 con 0x7f68c0071050 2024-03-19T02:06:12.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.553+0000 7f68b7fff640 1 --2- 172.21.15.55:0/2077420729 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68c0071050 0x7f68c00c7d60 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f68c006de70 tx=0x7f68a801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a5f45ee205ec6e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:12.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.561+0000 7f68be59c640 1 -- 172.21.15.55:0/2077420729 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a80b0070 con 0x7f68c0071050 2024-03-19T02:06:12.567 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.561+0000 7f68be59c640 1 -- 172.21.15.55:0/2077420729 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68a80ab600 con 0x7f68c0071050 2024-03-19T02:06:12.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.561+0000 7f68be59c640 1 -- 172.21.15.55:0/2077420729 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a80ab9d0 con 0x7f68c0071050 2024-03-19T02:06:12.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.561+0000 7f68c5200640 1 -- 172.21.15.55:0/2077420729 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68c0071050 msgr2=0x7f68c00c7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:12.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.561+0000 7f68c5200640 1 --2- 172.21.15.55:0/2077420729 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68c0071050 0x7f68c00c7d60 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f68c006de70 tx=0x7f68a801c520 comp rx=0 tx=0).stop 2024-03-19T02:06:12.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.561+0000 7f68c5200640 1 -- 172.21.15.55:0/2077420729 shutdown_connections 2024-03-19T02:06:12.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.561+0000 7f68c5200640 1 --2- 172.21.15.55:0/2077420729 >> v2:172.21.15.84:3300/0 conn(0x7f68c015ae00 0x7f68c006b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:12.568 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.561+0000 7f68c5200640 1 --2- 172.21.15.55:0/2077420729 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68c0071050 0x7f68c00c7d60 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:12.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.561+0000 7f68c5200640 1 --2- 172.21.15.55:0/2077420729 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68c01545f0 0x7f68c00ca210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:12.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.561+0000 7f68c5200640 1 -- 172.21.15.55:0/2077420729 >> 172.21.15.55:0/2077420729 conn(0x7f68c0070410 msgr2=0x7f68c0070820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:12.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.561+0000 7f68c5200640 1 -- 172.21.15.55:0/2077420729 shutdown_connections 2024-03-19T02:06:12.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.561+0000 7f68c5200640 1 -- 172.21.15.55:0/2077420729 wait complete. 2024-03-19T02:06:12.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.561+0000 7f68c5200640 1 Processor -- start 2024-03-19T02:06:12.569 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.561+0000 7f68c5200640 1 -- start start 2024-03-19T02:06:12.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.561+0000 7f68c5200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68c0071050 0x7f68c0143db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:12.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.561+0000 7f68c5200640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68c01545f0 0x7f68c015cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:12.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.561+0000 7f68c5200640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68c015ae00 0x7f68c015d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:12.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.561+0000 7f68c5200640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f68c015da70 con 0x7f68c015ae00 2024-03-19T02:06:12.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.561+0000 7f68c5200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f68c015d8f0 con 0x7f68c0071050 2024-03-19T02:06:12.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.561+0000 7f68c5200640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f68c015dbf0 con 0x7f68c01545f0 2024-03-19T02:06:12.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.565+0000 7f68bf59e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68c015ae00 0x7f68c015d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:12.570 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.565+0000 7f68bf59e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68c015ae00 0x7f68c015d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57172/0 (socket says 172.21.15.55:57172) 2024-03-19T02:06:12.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.565+0000 7f68bf59e640 1 -- 172.21.15.55:0/2155794073 learned_addr learned my addr 172.21.15.55:0/2155794073 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:12.571 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.565+0000 7f68bf59e640 1 -- 172.21.15.55:0/2155794073 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68c01545f0 msgr2=0x7f68c015cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:12.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.565+0000 7f68bf59e640 1 --2- 172.21.15.55:0/2155794073 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68c01545f0 0x7f68c015cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:12.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.565+0000 7f68bf59e640 1 -- 172.21.15.55:0/2155794073 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68c0071050 msgr2=0x7f68c0143db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:12.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.565+0000 7f68bf59e640 1 --2- 172.21.15.55:0/2155794073 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68c0071050 0x7f68c0143db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:12.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.565+0000 7f68bf59e640 1 -- 172.21.15.55:0/2155794073 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68c0145bd0 con 0x7f68c015ae00 2024-03-19T02:06:12.572 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.569+0000 7f68bf59e640 1 --2- 172.21.15.55:0/2155794073 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68c015ae00 0x7f68c015d090 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f68b800b970 tx=0x7f68b800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:12.576 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.573+0000 7f68b77fe640 1 -- 172.21.15.55:0/2155794073 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b8019070 con 0x7f68c015ae00 2024-03-19T02:06:12.576 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.573+0000 7f68b77fe640 1 -- 172.21.15.55:0/2155794073 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68b800cb60 con 0x7f68c015ae00 2024-03-19T02:06:12.576 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.573+0000 7f68c5200640 1 -- 172.21.15.55:0/2155794073 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68c0157030 con 0x7f68c015ae00 2024-03-19T02:06:12.576 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.573+0000 7f68b77fe640 1 -- 172.21.15.55:0/2155794073 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b8014440 con 0x7f68c015ae00 2024-03-19T02:06:12.576 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.573+0000 7f68c5200640 1 -- 172.21.15.55:0/2155794073 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68c0151f80 con 0x7f68c015ae00 2024-03-19T02:06:12.577 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.577+0000 7f68c5200640 1 -- 172.21.15.55:0/2155794073 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68c0068a40 con 0x7f68c015ae00 2024-03-19T02:06:12.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.601+0000 7f68b77fe640 1 -- 172.21.15.55:0/2155794073 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f68b801c070 con 0x7f68c015ae00 2024-03-19T02:06:12.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.601+0000 7f68b77fe640 1 --2- 172.21.15.55:0/2155794073 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68a00676b0 0x7f68a0069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:12.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.601+0000 7f68bed9d640 1 --2- 172.21.15.55:0/2155794073 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68a00676b0 0x7f68a0069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:12.603 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.601+0000 7f68bed9d640 1 --2- 172.21.15.55:0/2155794073 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68a00676b0 0x7f68a0069b70 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f68b0009730 tx=0x7f68b0006ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:12.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.893+0000 7f68b77fe640 1 -- 172.21.15.55:0/2155794073 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f68b8054050 con 0x7f68c015ae00 2024-03-19T02:06:12.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:12.893+0000 7f68b77fe640 1 -- 172.21.15.55:0/2155794073 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f68b8023020 con 0x7f68c015ae00 2024-03-19T02:06:13.070 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:13.069+0000 7f68c5200640 1 -- 172.21.15.55:0/2155794073 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f68c0164a40 con 0x7f68c015ae00 2024-03-19T02:06:13.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:13.149+0000 7f68b77fe640 1 -- 172.21.15.55:0/2155794073 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) ==== 74+0+19078 (secure 0 0 0) 0x7f68b80566b0 con 0x7f68c015ae00 2024-03-19T02:06:13.152 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:06:13.152 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":35,"fsid":"def9d4e5-0bc7-449d-b893-2490cbd10e25","created":"2024-03-19T01:50:02.482954+0000","modified":"2024-03-19T01:53:15.014351+0000","last_up_change":"2024-03-19T01:50:21.735508+0000","last_in_change":"2024-03-19T01:50:12.168944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:50:23.456904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:51:04.578583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:51:06.871659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T01:51:14.274442+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"35d5e3c5-d089-4f38-8961-98d064188f80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6825","nonce":2757675043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6827","nonce":2757675043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6831","nonce":2757675043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":2757675043},{"type":"v1","addr":"172.21.15.55:6829","nonce":2757675043}]},"public_addr":"172.21.15.55:6825/2757675043","cluster_addr":"172.21.15.55:6827/2757675043","heartbeat_back_addr":"172.21.15.55:6831/2757675043","heartbeat_front_addr":"172.21.15.55:6829/2757675043","state":["exists","up"]},{"osd":1,"uuid":"8e58f791-001a-44d0-a464-ca68a479f70c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6817","nonce":543460007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6819","nonce":543460007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6823","nonce":543460007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":543460007},{"type":"v1","addr":"172.21.15.55:6821","nonce":543460007}]},"public_addr":"172.21.15.55:6817/543460007","cluster_addr":"172.21.15.55:6819/543460007","heartbeat_back_addr":"172.21.15.55:6823/543460007","heartbeat_front_addr":"172.21.15.55:6821/543460007","state":["exists","up"]},{"osd":2,"uuid":"d44f38b8-67f4-43de-98a6-f3c520acaf10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6801","nonce":1760253550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6803","nonce":1760253550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6807","nonce":1760253550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1760253550},{"type":"v1","addr":"172.21.15.55:6805","nonce":1760253550}]},"public_addr":"172.21.15.55:6801/1760253550","cluster_addr":"172.21.15.55:6803/1760253550","heartbeat_back_addr":"172.21.15.55:6807/1760253550","heartbeat_front_addr":"172.21.15.55:6805/1760253550","state":["exists","up"]},{"osd":3,"uuid":"850a7316-2d74-43f0-8649-b5f8d8f6692e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6809","nonce":2797616448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6811","nonce":2797616448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6815","nonce":2797616448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":2797616448},{"type":"v1","addr":"172.21.15.55:6813","nonce":2797616448}]},"public_addr":"172.21.15.55:6809/2797616448","cluster_addr":"172.21.15.55:6811/2797616448","heartbeat_back_addr":"172.21.15.55:6815/2797616448","heartbeat_front_addr":"172.21.15.55:6813/2797616448","state":["exists","up"]},{"osd":4,"uuid":"4e6152ae-a77d-4bb6-a855-5782ad5411a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6819","nonce":2938436826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6821","nonce":2938436826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6825","nonce":2938436826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":2938436826},{"type":"v1","addr":"172.21.15.84:6823","nonce":2938436826}]},"public_addr":"172.21.15.84:6819/2938436826","cluster_addr":"172.21.15.84:6821/2938436826","heartbeat_back_addr":"172.21.15.84:6825/2938436826","heartbeat_front_addr":"172.21.15.84:6823/2938436826","state":["exists","up"]},{"osd":5,"uuid":"1217ceba-dd31-4615-96f1-dfaf57160f3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6803","nonce":460717295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6805","nonce":460717295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6809","nonce":460717295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":460717295},{"type":"v1","addr":"172.21.15.84:6807","nonce":460717295}]},"public_addr":"172.21.15.84:6803/460717295","cluster_addr":"172.21.15.84:6805/460717295","heartbeat_back_addr":"172.21.15.84:6809/460717295","heartbeat_front_addr":"172.21.15.84:6807/460717295","state":["exists","up"]},{"osd":6,"uuid":"3d9ed78a-2284-4c28-a24a-70eb812c7836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6827","nonce":416123135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6829","nonce":416123135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6833","nonce":416123135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":416123135},{"type":"v1","addr":"172.21.15.84:6831","nonce":416123135}]},"public_addr":"172.21.15.84:6827/416123135","cluster_addr":"172.21.15.84:6829/416123135","heartbeat_back_addr":"172.21.15.84:6833/416123135","heartbeat_front_addr":"172.21.15.84:6831/416123135","state":["exists","up"]},{"osd":7,"uuid":"c80a744f-1d3e-4c86-8dd1-8b34ddfed5dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6811","nonce":460083528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6813","nonce":460083528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6817","nonce":460083528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":460083528},{"type":"v1","addr":"172.21.15.84:6815","nonce":460083528}]},"public_addr":"172.21.15.84:6811/460083528","cluster_addr":"172.21.15.84:6813/460083528","heartbeat_back_addr":"172.21.15.84:6817/460083528","heartbeat_front_addr":"172.21.15.84:6815/460083528","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.269126+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.121001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.075714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.092461+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.194676+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.062833+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.316515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:50:18.237111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.5","mappings":[{"from":3,"to":0}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":3,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:6839/354880008":"2024-03-20T01:52:56.125297+0000","172.21.15.84:6838/354880008":"2024-03-20T01:52:56.125297+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:06:13.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:13.153+0000 7f68c5200640 1 -- 172.21.15.55:0/2155794073 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68a00676b0 msgr2=0x7f68a0069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:13.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:13.153+0000 7f68c5200640 1 --2- 172.21.15.55:0/2155794073 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68a00676b0 0x7f68a0069b70 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f68b0009730 tx=0x7f68b0006ca0 comp rx=0 tx=0).stop 2024-03-19T02:06:13.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:13.153+0000 7f68c5200640 1 -- 172.21.15.55:0/2155794073 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68c015ae00 msgr2=0x7f68c015d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:13.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:13.153+0000 7f68c5200640 1 --2- 172.21.15.55:0/2155794073 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68c015ae00 0x7f68c015d090 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f68b800b970 tx=0x7f68b800be30 comp rx=0 tx=0).stop 2024-03-19T02:06:13.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:13.153+0000 7f68c5200640 1 -- 172.21.15.55:0/2155794073 shutdown_connections 2024-03-19T02:06:13.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:13.153+0000 7f68c5200640 1 --2- 172.21.15.55:0/2155794073 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f68a00676b0 0x7f68a0069b70 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:13.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:13.153+0000 7f68c5200640 1 --2- 172.21.15.55:0/2155794073 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68c015ae00 0x7f68c015d090 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:13.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:13.153+0000 7f68c5200640 1 --2- 172.21.15.55:0/2155794073 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f68c01545f0 0x7f68c015cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:13.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:13.153+0000 7f68c5200640 1 --2- 172.21.15.55:0/2155794073 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68c0071050 0x7f68c0143db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:13.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:13.153+0000 7f68c5200640 1 -- 172.21.15.55:0/2155794073 >> 172.21.15.55:0/2155794073 conn(0x7f68c0070410 msgr2=0x7f68c014a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:13.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:13.153+0000 7f68c5200640 1 -- 172.21.15.55:0/2155794073 shutdown_connections 2024-03-19T02:06:13.156 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:13.153+0000 7f68c5200640 1 -- 172.21.15.55:0/2155794073 wait complete. 2024-03-19T02:06:14.176 INFO:tasks.ceph:Scrubbing osd.0 2024-03-19T02:06:14.176 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-19T02:06:14.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.309+0000 7f9468d4f640 1 Processor -- start 2024-03-19T02:06:14.312 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.309+0000 7f9468d4f640 1 -- start start 2024-03-19T02:06:14.313 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.313+0000 7f9468d4f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f94640c39c0 0x7f9464158a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:14.313 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.313+0000 7f9468d4f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9464072c60 0x7f946415b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:14.313 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.313+0000 7f9468d4f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f946406b9f0 0x7f946406be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:14.313 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.313+0000 7f9468d4f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f9464152660 con 0x7f94640c39c0 2024-03-19T02:06:14.313 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.313+0000 7f9468d4f640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f9464073130 con 0x7f946406b9f0 2024-03-19T02:06:14.313 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.313+0000 7f9468d4f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f94640733f0 con 0x7f9464072c60 2024-03-19T02:06:14.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.313+0000 7f946259c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f94640c39c0 0x7f9464158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:14.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.313+0000 7f9462d9d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f946406b9f0 0x7f946406be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:14.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.313+0000 7f946259c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f94640c39c0 0x7f9464158a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57186/0 (socket says 172.21.15.55:57186) 2024-03-19T02:06:14.314 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.313+0000 7f946259c640 1 -- 172.21.15.55:0/3392938230 learned_addr learned my addr 172.21.15.55:0/3392938230 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:14.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.313+0000 7f9461d9b640 1 --2- 172.21.15.55:0/3392938230 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9464072c60 0x7f946415b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:14.316 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.317+0000 7f946259c640 1 -- 172.21.15.55:0/3392938230 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9464072c60 msgr2=0x7f946415b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:14.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.317+0000 7f946259c640 1 --2- 172.21.15.55:0/3392938230 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9464072c60 0x7f946415b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:14.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.317+0000 7f946259c640 1 -- 172.21.15.55:0/3392938230 >> v2:172.21.15.84:3300/0 conn(0x7f946406b9f0 msgr2=0x7f946406be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:14.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.317+0000 7f946259c640 1 --2- 172.21.15.55:0/3392938230 >> v2:172.21.15.84:3300/0 conn(0x7f946406b9f0 0x7f946406be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:14.317 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.317+0000 7f946259c640 1 -- 172.21.15.55:0/3392938230 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94640736d0 con 0x7f94640c39c0 2024-03-19T02:06:14.319 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.317+0000 7f946259c640 1 --2- 172.21.15.55:0/3392938230 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94640c39c0 0x7f9464158a80 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f94640ca870 tx=0x7f945401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c644b0c87667c7b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:14.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.321+0000 7f946159a640 1 -- 172.21.15.55:0/3392938230 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94540b5070 con 0x7f94640c39c0 2024-03-19T02:06:14.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.321+0000 7f946159a640 1 -- 172.21.15.55:0/3392938230 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94540a89d0 con 0x7f94640c39c0 2024-03-19T02:06:14.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.321+0000 7f946159a640 1 -- 172.21.15.55:0/3392938230 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94540a8da0 con 0x7f94640c39c0 2024-03-19T02:06:14.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.325+0000 7f9468d4f640 1 -- 172.21.15.55:0/3392938230 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94640c39c0 msgr2=0x7f9464158a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:14.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.325+0000 7f9468d4f640 1 --2- 172.21.15.55:0/3392938230 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94640c39c0 0x7f9464158a80 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f94640ca870 tx=0x7f945401ed10 comp rx=0 tx=0).stop 2024-03-19T02:06:14.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.325+0000 7f9468d4f640 1 -- 172.21.15.55:0/3392938230 shutdown_connections 2024-03-19T02:06:14.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.325+0000 7f9468d4f640 1 --2- 172.21.15.55:0/3392938230 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9464072c60 0x7f946415b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:14.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.325+0000 7f9468d4f640 1 --2- 172.21.15.55:0/3392938230 >> v2:172.21.15.84:3300/0 conn(0x7f946406b9f0 0x7f946406be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:14.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.325+0000 7f9468d4f640 1 --2- 172.21.15.55:0/3392938230 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94640c39c0 0x7f9464158a80 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:14.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.325+0000 7f9468d4f640 1 -- 172.21.15.55:0/3392938230 >> 172.21.15.55:0/3392938230 conn(0x7f94640cc9c0 msgr2=0x7f94640ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:14.325 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.325+0000 7f9468d4f640 1 -- 172.21.15.55:0/3392938230 shutdown_connections 2024-03-19T02:06:14.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.325+0000 7f9468d4f640 1 -- 172.21.15.55:0/3392938230 wait complete. 2024-03-19T02:06:14.326 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.325+0000 7f9468d4f640 1 Processor -- start 2024-03-19T02:06:14.327 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.325+0000 7f9468d4f640 1 -- start start 2024-03-19T02:06:14.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.325+0000 7f9468d4f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f946406b9f0 0x7f9464070360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:14.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.325+0000 7f9468d4f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9464072c60 0x7f94641edce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:14.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.325+0000 7f9468d4f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f94640c39c0 0x7f9464139460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:14.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.325+0000 7f9468d4f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9464157f50 con 0x7f946406b9f0 2024-03-19T02:06:14.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.325+0000 7f9468d4f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f9464158560 con 0x7f94640c39c0 2024-03-19T02:06:14.328 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.325+0000 7f9468d4f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f94640c60d0 con 0x7f9464072c60 2024-03-19T02:06:14.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.329+0000 7f946259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f946406b9f0 0x7f9464070360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:14.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.329+0000 7f946259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f946406b9f0 0x7f9464070360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57202/0 (socket says 172.21.15.55:57202) 2024-03-19T02:06:14.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.329+0000 7f946259c640 1 -- 172.21.15.55:0/2577366478 learned_addr learned my addr 172.21.15.55:0/2577366478 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:14.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.329+0000 7f9461d9b640 1 --2- 172.21.15.55:0/2577366478 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9464072c60 0x7f94641edce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:14.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.329+0000 7f9462d9d640 1 --2- 172.21.15.55:0/2577366478 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f94640c39c0 0x7f9464139460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:14.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.333+0000 7f946259c640 1 -- 172.21.15.55:0/2577366478 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9464072c60 msgr2=0x7f94641edce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:14.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.333+0000 7f946259c640 1 --2- 172.21.15.55:0/2577366478 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f9464072c60 0x7f94641edce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:14.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.333+0000 7f946259c640 1 -- 172.21.15.55:0/2577366478 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f94640c39c0 msgr2=0x7f9464139460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:14.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.333+0000 7f946259c640 1 --2- 172.21.15.55:0/2577366478 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f94640c39c0 0x7f9464139460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:14.332 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.333+0000 7f946259c640 1 -- 172.21.15.55:0/2577366478 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94641f2150 con 0x7f946406b9f0 2024-03-19T02:06:14.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.333+0000 7f946259c640 1 --2- 172.21.15.55:0/2577366478 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f946406b9f0 0x7f9464070360 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f94540b3040 tx=0x7f945401bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:14.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.337+0000 7f9452ffd640 1 -- 172.21.15.55:0/2577366478 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94540b5070 con 0x7f946406b9f0 2024-03-19T02:06:14.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.337+0000 7f9452ffd640 1 -- 172.21.15.55:0/2577366478 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94540b0a30 con 0x7f946406b9f0 2024-03-19T02:06:14.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.337+0000 7f9468d4f640 1 -- 172.21.15.55:0/2577366478 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94640cbbb0 con 0x7f946406b9f0 2024-03-19T02:06:14.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.337+0000 7f9452ffd640 1 -- 172.21.15.55:0/2577366478 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94540b0e20 con 0x7f946406b9f0 2024-03-19T02:06:14.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.337+0000 7f9468d4f640 1 -- 172.21.15.55:0/2577366478 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94640708a0 con 0x7f946406b9f0 2024-03-19T02:06:14.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.337+0000 7f9468d4f640 1 -- 172.21.15.55:0/2577366478 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f94641361e0 con 0x7f946406b9f0 2024-03-19T02:06:14.359 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.357+0000 7f9452ffd640 1 -- 172.21.15.55:0/2577366478 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f94540c2060 con 0x7f946406b9f0 2024-03-19T02:06:14.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.357+0000 7f9452ffd640 1 --2- 172.21.15.55:0/2577366478 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f94440676b0 0x7f9444069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:14.360 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.361+0000 7f9461d9b640 1 --2- 172.21.15.55:0/2577366478 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f94440676b0 0x7f9444069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:14.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.361+0000 7f9461d9b640 1 --2- 172.21.15.55:0/2577366478 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f94440676b0 0x7f9444069b70 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f94640cb8f0 tx=0x7f9458008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:14.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.365+0000 7f9452ffd640 1 -- 172.21.15.55:0/2577366478 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f94540f11a0 con 0x7f946406b9f0 2024-03-19T02:06:14.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.365+0000 7f9452ffd640 1 --2- 172.21.15.55:0/2577366478 >> [v2:172.21.15.55:6824/2757675043,v1:172.21.15.55:6825/2757675043] conn(0x7f9444071360 0x7f9444073800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:14.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.365+0000 7f9452ffd640 1 -- 172.21.15.55:0/2577366478 --> [v2:172.21.15.55:6824/2757675043,v1:172.21.15.55:6825/2757675043] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f94540b5e50 con 0x7f9444071360 2024-03-19T02:06:14.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.365+0000 7f9452ffd640 1 -- 172.21.15.55:0/2577366478 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f94540ef3a0 con 0x7f946406b9f0 2024-03-19T02:06:14.366 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.365+0000 7f9462d9d640 1 --2- 172.21.15.55:0/2577366478 >> [v2:172.21.15.55:6824/2757675043,v1:172.21.15.55:6825/2757675043] conn(0x7f9444071360 0x7f9444073800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:14.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.365+0000 7f9462d9d640 1 --2- 172.21.15.55:0/2577366478 >> [v2:172.21.15.55:6824/2757675043,v1:172.21.15.55:6825/2757675043] conn(0x7f9444071360 0x7f9444073800 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:14.367 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.365+0000 7f9452ffd640 1 -- 172.21.15.55:0/2577366478 <== osd.0 v2:172.21.15.55:6824/2757675043 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f94540b5e50 con 0x7f9444071360 2024-03-19T02:06:14.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.381+0000 7f9468d4f640 1 -- 172.21.15.55:0/2577366478 --> [v2:172.21.15.55:6824/2757675043,v1:172.21.15.55:6825/2757675043] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f94640cb960 con 0x7f9444071360 2024-03-19T02:06:14.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.381+0000 7f9452ffd640 1 -- 172.21.15.55:0/2577366478 <== osd.0 v2:172.21.15.55:6824/2757675043 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f94640cb960 con 0x7f9444071360 2024-03-19T02:06:14.381 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-19T02:06:14.381 INFO:teuthology.orchestra.run.smithi055.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T02:06:14.382 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-19T02:06:14.382 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.381+0000 7f9468d4f640 1 -- 172.21.15.55:0/2577366478 >> [v2:172.21.15.55:6824/2757675043,v1:172.21.15.55:6825/2757675043] conn(0x7f9444071360 msgr2=0x7f9444073800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:14.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.381+0000 7f9468d4f640 1 --2- 172.21.15.55:0/2577366478 >> [v2:172.21.15.55:6824/2757675043,v1:172.21.15.55:6825/2757675043] conn(0x7f9444071360 0x7f9444073800 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:14.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.381+0000 7f9468d4f640 1 -- 172.21.15.55:0/2577366478 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f94440676b0 msgr2=0x7f9444069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:14.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.381+0000 7f9468d4f640 1 --2- 172.21.15.55:0/2577366478 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f94440676b0 0x7f9444069b70 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f94640cb8f0 tx=0x7f9458008040 comp rx=0 tx=0).stop 2024-03-19T02:06:14.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.381+0000 7f9468d4f640 1 -- 172.21.15.55:0/2577366478 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f946406b9f0 msgr2=0x7f9464070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:14.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.381+0000 7f9468d4f640 1 --2- 172.21.15.55:0/2577366478 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f946406b9f0 0x7f9464070360 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f94540b3040 tx=0x7f945401bab0 comp rx=0 tx=0).stop 2024-03-19T02:06:14.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.381+0000 7f9462d9d640 1 -- 172.21.15.55:0/2577366478 reap_dead start 2024-03-19T02:06:14.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.381+0000 7f9468d4f640 1 -- 172.21.15.55:0/2577366478 shutdown_connections 2024-03-19T02:06:14.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.381+0000 7f9468d4f640 1 -- 172.21.15.55:0/2577366478 >> 172.21.15.55:0/2577366478 conn(0x7f94640cc9c0 msgr2=0x7f9464151f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:14.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.381+0000 7f9468d4f640 1 -- 172.21.15.55:0/2577366478 shutdown_connections 2024-03-19T02:06:14.383 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.381+0000 7f9468d4f640 1 -- 172.21.15.55:0/2577366478 wait complete. 2024-03-19T02:06:14.402 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-19T02:06:14.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.513+0000 7f4359d3f640 1 Processor -- start 2024-03-19T02:06:14.513 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.513+0000 7f4359d3f640 1 -- start start 2024-03-19T02:06:14.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.513+0000 7f4359d3f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f435414eaa0 0x7f435406b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:14.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.513+0000 7f4359d3f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f435406bec0 0x7f43540c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:14.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.513+0000 7f4359d3f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f43540c6610 0x7f43541585f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:14.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.513+0000 7f4359d3f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f435414ee80 con 0x7f435414eaa0 2024-03-19T02:06:14.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.513+0000 7f4359d3f640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f43540c7540 con 0x7f43540c6610 2024-03-19T02:06:14.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.513+0000 7f4359d3f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f43540c77c0 con 0x7f435406bec0 2024-03-19T02:06:14.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.513+0000 7f43537fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f435414eaa0 0x7f435406b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:14.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.513+0000 7f4353fff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f43540c6610 0x7f43541585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:14.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.513+0000 7f43537fe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f435414eaa0 0x7f435406b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57218/0 (socket says 172.21.15.55:57218) 2024-03-19T02:06:14.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.513+0000 7f43537fe640 1 -- 172.21.15.55:0/1472137566 learned_addr learned my addr 172.21.15.55:0/1472137566 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:14.516 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.513+0000 7f4352ffd640 1 --2- 172.21.15.55:0/1472137566 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f435406bec0 0x7f43540c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:14.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.517+0000 7f43537fe640 1 -- 172.21.15.55:0/1472137566 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f435406bec0 msgr2=0x7f43540c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:14.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.517+0000 7f43537fe640 1 --2- 172.21.15.55:0/1472137566 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f435406bec0 0x7f43540c60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:14.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.517+0000 7f43537fe640 1 -- 172.21.15.55:0/1472137566 >> v2:172.21.15.84:3300/0 conn(0x7f43540c6610 msgr2=0x7f43541585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:14.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.517+0000 7f43537fe640 1 --2- 172.21.15.55:0/1472137566 >> v2:172.21.15.84:3300/0 conn(0x7f43540c6610 0x7f43541585f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:14.518 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.517+0000 7f43537fe640 1 -- 172.21.15.55:0/1472137566 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43540c7aa0 con 0x7f435414eaa0 2024-03-19T02:06:14.521 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.521+0000 7f43537fe640 1 --2- 172.21.15.55:0/1472137566 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f435414eaa0 0x7f435406b980 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f4354155b00 tx=0x7f434401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e73222a0fc3a24d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:14.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.525+0000 7f43527fc640 1 -- 172.21.15.55:0/1472137566 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43440b2070 con 0x7f435414eaa0 2024-03-19T02:06:14.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.525+0000 7f43527fc640 1 -- 172.21.15.55:0/1472137566 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43440a9070 con 0x7f435414eaa0 2024-03-19T02:06:14.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.525+0000 7f43527fc640 1 -- 172.21.15.55:0/1472137566 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43440ad880 con 0x7f435414eaa0 2024-03-19T02:06:14.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.525+0000 7f4359d3f640 1 -- 172.21.15.55:0/1472137566 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f435414eaa0 msgr2=0x7f435406b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:14.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.525+0000 7f4359d3f640 1 --2- 172.21.15.55:0/1472137566 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f435414eaa0 0x7f435406b980 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f4354155b00 tx=0x7f434401c490 comp rx=0 tx=0).stop 2024-03-19T02:06:14.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.525+0000 7f4359d3f640 1 -- 172.21.15.55:0/1472137566 shutdown_connections 2024-03-19T02:06:14.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.525+0000 7f4359d3f640 1 --2- 172.21.15.55:0/1472137566 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f435406bec0 0x7f43540c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:14.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.525+0000 7f4359d3f640 1 --2- 172.21.15.55:0/1472137566 >> v2:172.21.15.84:3300/0 conn(0x7f43540c6610 0x7f43541585f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:14.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.525+0000 7f4359d3f640 1 --2- 172.21.15.55:0/1472137566 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f435414eaa0 0x7f435406b980 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:14.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.525+0000 7f4359d3f640 1 -- 172.21.15.55:0/1472137566 >> 172.21.15.55:0/1472137566 conn(0x7f43540cc9c0 msgr2=0x7f43540ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:14.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.525+0000 7f4359d3f640 1 -- 172.21.15.55:0/1472137566 shutdown_connections 2024-03-19T02:06:14.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.525+0000 7f4359d3f640 1 -- 172.21.15.55:0/1472137566 wait complete. 2024-03-19T02:06:14.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.529+0000 7f4359d3f640 1 Processor -- start 2024-03-19T02:06:14.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.529+0000 7f4359d3f640 1 -- start start 2024-03-19T02:06:14.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.529+0000 7f4359d3f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f435406bec0 0x7f43541f6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:14.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.529+0000 7f4359d3f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43540c6610 0x7f43541f6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:14.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.529+0000 7f4359d3f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f435414eaa0 0x7f43541574a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:14.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.529+0000 7f4359d3f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f43540c6bf0 con 0x7f43540c6610 2024-03-19T02:06:14.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.529+0000 7f4359d3f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4354158f40 con 0x7f435406bec0 2024-03-19T02:06:14.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.529+0000 7f4359d3f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f43540c6d70 con 0x7f435414eaa0 2024-03-19T02:06:14.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.529+0000 7f4352ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43540c6610 0x7f43541f6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:14.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.529+0000 7f4353fff640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f435414eaa0 0x7f43541574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:14.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.529+0000 7f4352ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43540c6610 0x7f43541f6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57232/0 (socket says 172.21.15.55:57232) 2024-03-19T02:06:14.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.529+0000 7f4352ffd640 1 -- 172.21.15.55:0/1253281639 learned_addr learned my addr 172.21.15.55:0/1253281639 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:14.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.529+0000 7f43537fe640 1 --2- 172.21.15.55:0/1253281639 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f435406bec0 0x7f43541f6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:14.533 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.533+0000 7f4352ffd640 1 -- 172.21.15.55:0/1253281639 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f435414eaa0 msgr2=0x7f43541574a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:14.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.533+0000 7f4352ffd640 1 --2- 172.21.15.55:0/1253281639 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f435414eaa0 0x7f43541574a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:14.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.533+0000 7f4352ffd640 1 -- 172.21.15.55:0/1253281639 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f435406bec0 msgr2=0x7f43541f6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:14.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.533+0000 7f4352ffd640 1 --2- 172.21.15.55:0/1253281639 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f435406bec0 0x7f43541f6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:14.534 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.533+0000 7f4352ffd640 1 -- 172.21.15.55:0/1253281639 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43540c4a70 con 0x7f43540c6610 2024-03-19T02:06:14.536 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.537+0000 7f4352ffd640 1 --2- 172.21.15.55:0/1253281639 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43540c6610 0x7f43541f6ea0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f434800e8a0 tx=0x7f434800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:14.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.537+0000 7f433bfff640 1 -- 172.21.15.55:0/1253281639 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4348017070 con 0x7f43540c6610 2024-03-19T02:06:14.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.541+0000 7f433bfff640 1 -- 172.21.15.55:0/1253281639 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f434800ee90 con 0x7f43540c6610 2024-03-19T02:06:14.540 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.541+0000 7f4359d3f640 1 -- 172.21.15.55:0/1253281639 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43540c4d50 con 0x7f43540c6610 2024-03-19T02:06:14.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.541+0000 7f433bfff640 1 -- 172.21.15.55:0/1253281639 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434800cb80 con 0x7f43540c6610 2024-03-19T02:06:14.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.541+0000 7f4359d3f640 1 -- 172.21.15.55:0/1253281639 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43540ca860 con 0x7f43540c6610 2024-03-19T02:06:14.541 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.541+0000 7f4359d3f640 1 -- 172.21.15.55:0/1253281639 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4354068a40 con 0x7f43540c6610 2024-03-19T02:06:14.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.817+0000 7f433bfff640 1 -- 172.21.15.55:0/1253281639 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f434801a070 con 0x7f43540c6610 2024-03-19T02:06:14.818 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.817+0000 7f433bfff640 1 --2- 172.21.15.55:0/1253281639 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4340067660 0x7f4340069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:14.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.817+0000 7f43537fe640 1 --2- 172.21.15.55:0/1253281639 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4340067660 0x7f4340069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:14.819 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.817+0000 7f43537fe640 1 --2- 172.21.15.55:0/1253281639 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4340067660 0x7f4340069b20 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f43440af000 tx=0x7f43440a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:14.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.829+0000 7f433bfff640 1 -- 172.21.15.55:0/1253281639 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f4348054560 con 0x7f43540c6610 2024-03-19T02:06:14.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:14.829+0000 7f433bfff640 1 -- 172.21.15.55:0/1253281639 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f43480564c0 con 0x7f43540c6610 2024-03-19T02:06:15.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.065+0000 7f4359d3f640 1 -- 172.21.15.55:0/1253281639 --> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f435414b670 con 0x7f4340067660 2024-03-19T02:06:15.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.069+0000 7f433bfff640 1 -- 172.21.15.55:0/1253281639 <== mgr.4105 v2:172.21.15.84:6800/998533783 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f435414b670 con 0x7f4340067660 2024-03-19T02:06:15.068 INFO:teuthology.orchestra.run.smithi055.stderr:instructed osd(s) 0 to deep-scrub 2024-03-19T02:06:15.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.069+0000 7f4359d3f640 1 -- 172.21.15.55:0/1253281639 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4340067660 msgr2=0x7f4340069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:15.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.069+0000 7f4359d3f640 1 --2- 172.21.15.55:0/1253281639 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4340067660 0x7f4340069b20 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f43440af000 tx=0x7f43440a7000 comp rx=0 tx=0).stop 2024-03-19T02:06:15.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.069+0000 7f4359d3f640 1 -- 172.21.15.55:0/1253281639 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43540c6610 msgr2=0x7f43541f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:15.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.069+0000 7f4359d3f640 1 --2- 172.21.15.55:0/1253281639 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43540c6610 0x7f43541f6ea0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f434800e8a0 tx=0x7f434800ed60 comp rx=0 tx=0).stop 2024-03-19T02:06:15.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.069+0000 7f4359d3f640 1 -- 172.21.15.55:0/1253281639 shutdown_connections 2024-03-19T02:06:15.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.069+0000 7f4359d3f640 1 --2- 172.21.15.55:0/1253281639 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f4340067660 0x7f4340069b20 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:15.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.069+0000 7f4359d3f640 1 --2- 172.21.15.55:0/1253281639 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f435414eaa0 0x7f43541574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:15.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.069+0000 7f4359d3f640 1 --2- 172.21.15.55:0/1253281639 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43540c6610 0x7f43541f6ea0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:15.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.069+0000 7f4359d3f640 1 --2- 172.21.15.55:0/1253281639 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f435406bec0 0x7f43541f6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:15.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.069+0000 7f4359d3f640 1 -- 172.21.15.55:0/1253281639 >> 172.21.15.55:0/1253281639 conn(0x7f43540cc9c0 msgr2=0x7f43540c32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:15.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.069+0000 7f4359d3f640 1 -- 172.21.15.55:0/1253281639 shutdown_connections 2024-03-19T02:06:15.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.069+0000 7f4359d3f640 1 -- 172.21.15.55:0/1253281639 wait complete. 2024-03-19T02:06:15.088 INFO:tasks.ceph:Scrubbing osd.1 2024-03-19T02:06:15.088 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-19T02:06:15.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.197+0000 7f744ed4f640 1 Processor -- start 2024-03-19T02:06:15.200 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.197+0000 7f744ed4f640 1 -- start start 2024-03-19T02:06:15.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.201+0000 7f744ed4f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f74480ca350 0x7f744815b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:15.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.201+0000 7f744ed4f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f74480c3320 0x7f74480728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:15.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.201+0000 7f744ed4f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7448151330 con 0x7f7448150f50 2024-03-19T02:06:15.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.201+0000 7f744ed4f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f74480c3e00 con 0x7f74480ca350 2024-03-19T02:06:15.201 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.201+0000 7f744ed4f640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f744815ac10 con 0x7f74480c3320 2024-03-19T02:06:15.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.201+0000 7f7447fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f74480ca350 0x7f744815b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:15.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.201+0000 7f7447fff640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f74480ca350 0x7f744815b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57238/0 (socket says 172.21.15.55:57238) 2024-03-19T02:06:15.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.201+0000 7f7447fff640 1 -- 172.21.15.55:0/3272825407 learned_addr learned my addr 172.21.15.55:0/3272825407 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:15.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.205+0000 7f7447fff640 1 -- 172.21.15.55:0/3272825407 >> v2:172.21.15.84:3300/0 conn(0x7f74480c3320 msgr2=0x7f74480728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:15.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.205+0000 7f7447fff640 1 --2- 172.21.15.55:0/3272825407 >> v2:172.21.15.84:3300/0 conn(0x7f74480c3320 0x7f74480728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:15.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.205+0000 7f7447fff640 1 -- 172.21.15.55:0/3272825407 >> v1:172.21.15.84:6789/0 conn(0x7f7448150f50 legacy=0x7f7448157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:15.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.205+0000 7f7447fff640 1 -- 172.21.15.55:0/3272825407 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74480c3840 con 0x7f74480ca350 2024-03-19T02:06:15.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.205+0000 7f7447fff640 1 --2- 172.21.15.55:0/3272825407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74480ca350 0x7f744815b0a0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f744815ab80 tx=0x7f743401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cbc490407cc830f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:15.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.209+0000 7f74477fe640 1 -- 172.21.15.55:0/3272825407 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74340a6040 con 0x7f74480ca350 2024-03-19T02:06:15.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.209+0000 7f74477fe640 1 -- 172.21.15.55:0/3272825407 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74340af070 con 0x7f74480ca350 2024-03-19T02:06:15.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.209+0000 7f74477fe640 1 -- 172.21.15.55:0/3272825407 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74340aa860 con 0x7f74480ca350 2024-03-19T02:06:15.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.213+0000 7f744ed4f640 1 -- 172.21.15.55:0/3272825407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74480ca350 msgr2=0x7f744815b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:15.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.213+0000 7f744ed4f640 1 --2- 172.21.15.55:0/3272825407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74480ca350 0x7f744815b0a0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f744815ab80 tx=0x7f743401c4e0 comp rx=0 tx=0).stop 2024-03-19T02:06:15.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.213+0000 7f744ed4f640 1 -- 172.21.15.55:0/3272825407 shutdown_connections 2024-03-19T02:06:15.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.213+0000 7f744ed4f640 1 --2- 172.21.15.55:0/3272825407 >> v2:172.21.15.84:3300/0 conn(0x7f74480c3320 0x7f74480728b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:15.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.213+0000 7f744ed4f640 1 --2- 172.21.15.55:0/3272825407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74480ca350 0x7f744815b0a0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:15.213 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.213+0000 7f744ed4f640 1 -- 172.21.15.55:0/3272825407 >> 172.21.15.55:0/3272825407 conn(0x7f744806e670 msgr2=0x7f744806ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:15.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.213+0000 7f744ed4f640 1 -- 172.21.15.55:0/3272825407 shutdown_connections 2024-03-19T02:06:15.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.213+0000 7f744ed4f640 1 -- 172.21.15.55:0/3272825407 wait complete. 2024-03-19T02:06:15.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.213+0000 7f744ed4f640 1 Processor -- start 2024-03-19T02:06:15.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.213+0000 7f744ed4f640 1 -- start start 2024-03-19T02:06:15.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.213+0000 7f744ed4f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74480c3320 0x7f7448148a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:15.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.213+0000 7f744ed4f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74480ca350 0x7f74481432f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:15.215 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.213+0000 7f744ed4f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7448150f50 0x7f744815f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:15.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.213+0000 7f744ed4f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f74481639b0 con 0x7f74480c3320 2024-03-19T02:06:15.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.213+0000 7f744ed4f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f7448163830 con 0x7f74480ca350 2024-03-19T02:06:15.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.213+0000 7f744ed4f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f7448163b30 con 0x7f7448150f50 2024-03-19T02:06:15.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.213+0000 7f744caeb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74480c3320 0x7f7448148a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:15.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.217+0000 7f744caeb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74480c3320 0x7f7448148a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57252/0 (socket says 172.21.15.55:57252) 2024-03-19T02:06:15.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.217+0000 7f744caeb640 1 -- 172.21.15.55:0/1739989878 learned_addr learned my addr 172.21.15.55:0/1739989878 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:15.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.217+0000 7f744caeb640 1 -- 172.21.15.55:0/1739989878 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7448150f50 msgr2=0x7f744815f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:15.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.217+0000 7f744caeb640 1 --2- 172.21.15.55:0/1739989878 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f7448150f50 0x7f744815f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:15.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.217+0000 7f744caeb640 1 -- 172.21.15.55:0/1739989878 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74480ca350 msgr2=0x7f74481432f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:15.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.217+0000 7f744caeb640 1 --2- 172.21.15.55:0/1739989878 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74480ca350 0x7f74481432f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:15.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.217+0000 7f744caeb640 1 -- 172.21.15.55:0/1739989878 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74481f79a0 con 0x7f74480c3320 2024-03-19T02:06:15.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.221+0000 7f744caeb640 1 --2- 172.21.15.55:0/1739989878 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74480c3320 0x7f7448148a90 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f743c00b3e0 tx=0x7f743c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:15.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.225+0000 7f74457fa640 1 -- 172.21.15.55:0/1739989878 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c015020 con 0x7f74480c3320 2024-03-19T02:06:15.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.225+0000 7f74457fa640 1 -- 172.21.15.55:0/1739989878 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f743c011850 con 0x7f74480c3320 2024-03-19T02:06:15.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.225+0000 7f744ed4f640 1 -- 172.21.15.55:0/1739989878 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7448149980 con 0x7f74480c3320 2024-03-19T02:06:15.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.225+0000 7f74457fa640 1 -- 172.21.15.55:0/1739989878 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c011c40 con 0x7f74480c3320 2024-03-19T02:06:15.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.225+0000 7f744ed4f640 1 -- 172.21.15.55:0/1739989878 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7448149f40 con 0x7f74480c3320 2024-03-19T02:06:15.227 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.225+0000 7f744ed4f640 1 -- 172.21.15.55:0/1739989878 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7448068a40 con 0x7f74480c3320 2024-03-19T02:06:15.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.257+0000 7f74457fa640 1 -- 172.21.15.55:0/1739989878 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f743c018020 con 0x7f74480c3320 2024-03-19T02:06:15.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.257+0000 7f74457fa640 1 --2- 172.21.15.55:0/1739989878 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f742c0676b0 0x7f742c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:15.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.257+0000 7f7447fff640 1 --2- 172.21.15.55:0/1739989878 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f742c0676b0 0x7f742c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:15.259 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.257+0000 7f7447fff640 1 --2- 172.21.15.55:0/1739989878 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f742c0676b0 0x7f742c069b70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f74340ae000 tx=0x7f7434008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:15.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.261+0000 7f74457fa640 1 -- 172.21.15.55:0/1739989878 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f743c011de0 con 0x7f74480c3320 2024-03-19T02:06:15.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.261+0000 7f74457fa640 1 --2- 172.21.15.55:0/1739989878 >> [v2:172.21.15.55:6816/543460007,v1:172.21.15.55:6817/543460007] conn(0x7f742c0713e0 0x7f742c073880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:15.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.261+0000 7f74457fa640 1 -- 172.21.15.55:0/1739989878 --> [v2:172.21.15.55:6816/543460007,v1:172.21.15.55:6817/543460007] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f743c00de50 con 0x7f742c0713e0 2024-03-19T02:06:15.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.261+0000 7f74457fa640 1 -- 172.21.15.55:0/1739989878 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f743c052680 con 0x7f74480c3320 2024-03-19T02:06:15.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.261+0000 7f744d2ec640 1 --2- 172.21.15.55:0/1739989878 >> [v2:172.21.15.55:6816/543460007,v1:172.21.15.55:6817/543460007] conn(0x7f742c0713e0 0x7f742c073880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:15.263 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.261+0000 7f744d2ec640 1 --2- 172.21.15.55:0/1739989878 >> [v2:172.21.15.55:6816/543460007,v1:172.21.15.55:6817/543460007] conn(0x7f742c0713e0 0x7f742c073880 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:15.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.265+0000 7f74457fa640 1 -- 172.21.15.55:0/1739989878 <== osd.1 v2:172.21.15.55:6816/543460007 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f743c00de50 con 0x7f742c0713e0 2024-03-19T02:06:15.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.277+0000 7f744ed4f640 1 -- 172.21.15.55:0/1739989878 --> [v2:172.21.15.55:6816/543460007,v1:172.21.15.55:6817/543460007] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f74480c20c0 con 0x7f742c0713e0 2024-03-19T02:06:15.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.277+0000 7f74457fa640 1 -- 172.21.15.55:0/1739989878 <== osd.1 v2:172.21.15.55:6816/543460007 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f74480c20c0 con 0x7f742c0713e0 2024-03-19T02:06:15.279 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-19T02:06:15.279 INFO:teuthology.orchestra.run.smithi055.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T02:06:15.279 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-19T02:06:15.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.277+0000 7f744ed4f640 1 -- 172.21.15.55:0/1739989878 >> [v2:172.21.15.55:6816/543460007,v1:172.21.15.55:6817/543460007] conn(0x7f742c0713e0 msgr2=0x7f742c073880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:15.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.277+0000 7f744ed4f640 1 --2- 172.21.15.55:0/1739989878 >> [v2:172.21.15.55:6816/543460007,v1:172.21.15.55:6817/543460007] conn(0x7f742c0713e0 0x7f742c073880 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:15.280 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.277+0000 7f744ed4f640 1 -- 172.21.15.55:0/1739989878 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f742c0676b0 msgr2=0x7f742c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:15.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.277+0000 7f744ed4f640 1 --2- 172.21.15.55:0/1739989878 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f742c0676b0 0x7f742c069b70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f74340ae000 tx=0x7f7434008040 comp rx=0 tx=0).stop 2024-03-19T02:06:15.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.277+0000 7f744ed4f640 1 -- 172.21.15.55:0/1739989878 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74480c3320 msgr2=0x7f7448148a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:15.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.277+0000 7f744ed4f640 1 --2- 172.21.15.55:0/1739989878 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74480c3320 0x7f7448148a90 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f743c00b3e0 tx=0x7f743c00b8a0 comp rx=0 tx=0).stop 2024-03-19T02:06:15.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.277+0000 7f744d2ec640 1 -- 172.21.15.55:0/1739989878 reap_dead start 2024-03-19T02:06:15.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.277+0000 7f744ed4f640 1 -- 172.21.15.55:0/1739989878 shutdown_connections 2024-03-19T02:06:15.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.277+0000 7f744ed4f640 1 -- 172.21.15.55:0/1739989878 >> 172.21.15.55:0/1739989878 conn(0x7f744806e670 msgr2=0x7f7448144160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:15.282 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.277+0000 7f744ed4f640 1 -- 172.21.15.55:0/1739989878 shutdown_connections 2024-03-19T02:06:15.282 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.277+0000 7f744ed4f640 1 -- 172.21.15.55:0/1739989878 wait complete. 2024-03-19T02:06:15.295 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-19T02:06:15.401 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.401+0000 7f18af95b640 1 Processor -- start 2024-03-19T02:06:15.402 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.401+0000 7f18af95b640 1 -- start start 2024-03-19T02:06:15.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.401+0000 7f18af95b640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f18a81570e0 0x7f18a8157510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:15.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.401+0000 7f18af95b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f18a80c3ec0 0x7f18a80c42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:15.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.401+0000 7f18af95b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18a80c6e80 con 0x7f18a814d6a0 2024-03-19T02:06:15.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.401+0000 7f18af95b640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f18a80c4ca0 con 0x7f18a81570e0 2024-03-19T02:06:15.403 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.401+0000 7f18af95b640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f18a8157a50 con 0x7f18a80c3ec0 2024-03-19T02:06:15.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.401+0000 7f18ad6f7640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f18a814d6a0 0x7f18a80c73c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:40418/0 (socket says 172.21.15.55:40418) 2024-03-19T02:06:15.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.401+0000 7f18ad6f7640 1 -- 172.21.15.55:0/1776078286 learned_addr learned my addr 172.21.15.55:0/1776078286 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:06:15.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.401+0000 7f18acef6640 1 --2- 172.21.15.55:0/1776078286 >> v2:172.21.15.84:3300/0 conn(0x7f18a81570e0 0x7f18a8157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:15.404 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.401+0000 7f18adef8640 1 --2- 172.21.15.55:0/1776078286 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f18a80c3ec0 0x7f18a80c42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:15.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.405+0000 7f189ffff640 1 -- 172.21.15.55:0/1776078286 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3393519816 0 0) 0x7f18a80c6e80 con 0x7f18a814d6a0 2024-03-19T02:06:15.407 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.405+0000 7f189ffff640 1 -- 172.21.15.55:0/1776078286 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1894003530 con 0x7f18a814d6a0 2024-03-19T02:06:15.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.409+0000 7f18adef8640 1 -- 172.21.15.55:0/1776078286 >> v2:172.21.15.84:3300/0 conn(0x7f18a81570e0 msgr2=0x7f18a8157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:15.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.409+0000 7f18adef8640 1 --2- 172.21.15.55:0/1776078286 >> v2:172.21.15.84:3300/0 conn(0x7f18a81570e0 0x7f18a8157510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:15.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.409+0000 7f18adef8640 1 -- 172.21.15.55:0/1776078286 >> v1:172.21.15.55:6789/0 conn(0x7f18a814d6a0 legacy=0x7f18a80c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:15.408 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.409+0000 7f18adef8640 1 -- 172.21.15.55:0/1776078286 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a80c6300 con 0x7f18a80c3ec0 2024-03-19T02:06:15.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.409+0000 7f18adef8640 1 --2- 172.21.15.55:0/1776078286 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f18a80c3ec0 0x7f18a80c42f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009d90 tx=0x7f18a001c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e58492f72f8a9df6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:15.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.417+0000 7f189ffff640 1 -- 172.21.15.55:0/1776078286 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a00b2070 con 0x7f18a80c3ec0 2024-03-19T02:06:15.417 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.417+0000 7f189ffff640 1 -- 172.21.15.55:0/1776078286 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18a00a5a00 con 0x7f18a80c3ec0 2024-03-19T02:06:15.418 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.417+0000 7f189ffff640 1 -- 172.21.15.55:0/1776078286 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a00a5dd0 con 0x7f18a80c3ec0 2024-03-19T02:06:15.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.417+0000 7f18af95b640 1 -- 172.21.15.55:0/1776078286 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f18a80c3ec0 msgr2=0x7f18a80c42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:15.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.417+0000 7f18af95b640 1 --2- 172.21.15.55:0/1776078286 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f18a80c3ec0 0x7f18a80c42f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009d90 tx=0x7f18a001c940 comp rx=0 tx=0).stop 2024-03-19T02:06:15.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.417+0000 7f18af95b640 1 -- 172.21.15.55:0/1776078286 shutdown_connections 2024-03-19T02:06:15.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.417+0000 7f18af95b640 1 --2- 172.21.15.55:0/1776078286 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f18a80c3ec0 0x7f18a80c42f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:15.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.417+0000 7f18af95b640 1 --2- 172.21.15.55:0/1776078286 >> v2:172.21.15.84:3300/0 conn(0x7f18a81570e0 0x7f18a8157510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:15.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.417+0000 7f18af95b640 1 -- 172.21.15.55:0/1776078286 >> 172.21.15.55:0/1776078286 conn(0x7f18a806c3b0 msgr2=0x7f18a806c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:15.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.417+0000 7f18af95b640 1 -- 172.21.15.55:0/1776078286 shutdown_connections 2024-03-19T02:06:15.419 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.417+0000 7f18af95b640 1 -- 172.21.15.55:0/1776078286 wait complete. 2024-03-19T02:06:15.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.421+0000 7f18af95b640 1 Processor -- start 2024-03-19T02:06:15.420 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.421+0000 7f18af95b640 1 -- start start 2024-03-19T02:06:15.422 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.421+0000 7f18af95b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18a80c3ec0 0x7f18a813c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:15.422 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.421+0000 7f18af95b640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f18a814d6a0 0x7f18a8153b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:15.422 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.421+0000 7f18af95b640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f18a81570e0 0x7f18a8156010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:15.422 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.421+0000 7f18af95b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f18a80c3ba0 con 0x7f18a80c3ec0 2024-03-19T02:06:15.422 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.421+0000 7f18af95b640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f18a80c3a20 con 0x7f18a814d6a0 2024-03-19T02:06:15.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.421+0000 7f18af95b640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f18a80c7ad0 con 0x7f18a81570e0 2024-03-19T02:06:15.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.421+0000 7f18ad6f7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18a80c3ec0 0x7f18a813c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:15.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.421+0000 7f18acef6640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f18a814d6a0 0x7f18a8153b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:15.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.421+0000 7f18adef8640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f18a81570e0 0x7f18a8156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:15.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.421+0000 7f18ad6f7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18a80c3ec0 0x7f18a813c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57262/0 (socket says 172.21.15.55:57262) 2024-03-19T02:06:15.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.421+0000 7f18ad6f7640 1 -- 172.21.15.55:0/531276974 learned_addr learned my addr 172.21.15.55:0/531276974 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:15.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.425+0000 7f18acef6640 1 -- 172.21.15.55:0/531276974 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f18a81570e0 msgr2=0x7f18a8156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:15.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.425+0000 7f18acef6640 1 --2- 172.21.15.55:0/531276974 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f18a81570e0 0x7f18a8156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:15.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.425+0000 7f18acef6640 1 -- 172.21.15.55:0/531276974 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18a80c3ec0 msgr2=0x7f18a813c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:15.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.425+0000 7f18acef6640 1 --2- 172.21.15.55:0/531276974 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18a80c3ec0 0x7f18a813c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:15.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.425+0000 7f18acef6640 1 -- 172.21.15.55:0/531276974 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a81f3ba0 con 0x7f18a814d6a0 2024-03-19T02:06:15.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.433+0000 7f18acef6640 1 --2- 172.21.15.55:0/531276974 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f18a814d6a0 0x7f18a8153b60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f189800e870 tx=0x7f189800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:15.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.433+0000 7f189dffb640 1 -- 172.21.15.55:0/531276974 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1898017070 con 0x7f18a814d6a0 2024-03-19T02:06:15.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.433+0000 7f18af95b640 1 -- 172.21.15.55:0/531276974 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18a81f3e80 con 0x7f18a814d6a0 2024-03-19T02:06:15.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.433+0000 7f18af95b640 1 -- 172.21.15.55:0/531276974 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18a81f4440 con 0x7f18a814d6a0 2024-03-19T02:06:15.436 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.437+0000 7f18af95b640 1 -- 172.21.15.55:0/531276974 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18a8068a40 con 0x7f18a814d6a0 2024-03-19T02:06:15.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.809+0000 7f189dffb640 1 -- 172.21.15.55:0/531276974 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f189800ee60 con 0x7f18a814d6a0 2024-03-19T02:06:15.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.809+0000 7f189dffb640 1 -- 172.21.15.55:0/531276974 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1898021060 con 0x7f18a814d6a0 2024-03-19T02:06:15.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.813+0000 7f189dffb640 1 -- 172.21.15.55:0/531276974 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f189801a070 con 0x7f18a814d6a0 2024-03-19T02:06:15.814 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.813+0000 7f189dffb640 1 --2- 172.21.15.55:0/531276974 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f188c067660 0x7f188c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:15.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.813+0000 7f18ad6f7640 1 --2- 172.21.15.55:0/531276974 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f188c067660 0x7f188c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:15.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.813+0000 7f189dffb640 1 -- 172.21.15.55:0/531276974 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f1898054490 con 0x7f18a814d6a0 2024-03-19T02:06:15.815 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.813+0000 7f18ad6f7640 1 --2- 172.21.15.55:0/531276974 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f188c067660 0x7f188c069b20 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f18a4007fd0 tx=0x7f18a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:15.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.825+0000 7f189dffb640 1 -- 172.21.15.55:0/531276974 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f18980563f0 con 0x7f18a814d6a0 2024-03-19T02:06:15.995 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.993+0000 7f18af95b640 1 -- 172.21.15.55:0/531276974 --> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f18a814a030 con 0x7f188c067660 2024-03-19T02:06:15.996 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.997+0000 7f189dffb640 1 -- 172.21.15.55:0/531276974 <== mgr.4105 v2:172.21.15.84:6800/998533783 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f18a814a030 con 0x7f188c067660 2024-03-19T02:06:15.996 INFO:teuthology.orchestra.run.smithi055.stderr:instructed osd(s) 1 to deep-scrub 2024-03-19T02:06:15.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.997+0000 7f18af95b640 1 -- 172.21.15.55:0/531276974 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f188c067660 msgr2=0x7f188c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:15.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.997+0000 7f18af95b640 1 --2- 172.21.15.55:0/531276974 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f188c067660 0x7f188c069b20 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f18a4007fd0 tx=0x7f18a4009040 comp rx=0 tx=0).stop 2024-03-19T02:06:15.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.997+0000 7f18af95b640 1 -- 172.21.15.55:0/531276974 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f18a814d6a0 msgr2=0x7f18a8153b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:15.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.997+0000 7f18af95b640 1 --2- 172.21.15.55:0/531276974 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f18a814d6a0 0x7f18a8153b60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f189800e870 tx=0x7f189800ed30 comp rx=0 tx=0).stop 2024-03-19T02:06:15.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.997+0000 7f18af95b640 1 -- 172.21.15.55:0/531276974 shutdown_connections 2024-03-19T02:06:15.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.997+0000 7f18af95b640 1 --2- 172.21.15.55:0/531276974 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f188c067660 0x7f188c069b20 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:15.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.997+0000 7f18af95b640 1 --2- 172.21.15.55:0/531276974 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f18a81570e0 0x7f18a8156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:15.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.997+0000 7f18af95b640 1 --2- 172.21.15.55:0/531276974 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f18a814d6a0 0x7f18a8153b60 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:15.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.997+0000 7f18af95b640 1 --2- 172.21.15.55:0/531276974 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f18a80c3ec0 0x7f18a813c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:15.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.997+0000 7f18af95b640 1 -- 172.21.15.55:0/531276974 >> 172.21.15.55:0/531276974 conn(0x7f18a806c3b0 msgr2=0x7f18a814a550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:15.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.997+0000 7f18af95b640 1 -- 172.21.15.55:0/531276974 shutdown_connections 2024-03-19T02:06:16.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:15.997+0000 7f18af95b640 1 -- 172.21.15.55:0/531276974 wait complete. 2024-03-19T02:06:16.014 INFO:tasks.ceph:Scrubbing osd.2 2024-03-19T02:06:16.015 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-19T02:06:16.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.129+0000 7f6187a63640 1 Processor -- start 2024-03-19T02:06:16.129 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.129+0000 7f6187a63640 1 -- start start 2024-03-19T02:06:16.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.129+0000 7f6187a63640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f61800c60d0 0x7f61800c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:16.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.129+0000 7f6187a63640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f61801585f0 0x7f6180158a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:16.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.129+0000 7f6187a63640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f618014ee80 con 0x7f618014eaa0 2024-03-19T02:06:16.130 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.129+0000 7f6187a63640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f61800c7540 con 0x7f61800c60d0 2024-03-19T02:06:16.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.129+0000 7f6187a63640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f61800c77c0 con 0x7f61801585f0 2024-03-19T02:06:16.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.129+0000 7f6184ffe640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f61800c60d0 0x7f61800c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:16.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.129+0000 7f61857ff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f618014eaa0 0x7f618006b980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:55614/0 (socket says 172.21.15.55:55614) 2024-03-19T02:06:16.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.129+0000 7f61857ff640 1 -- 172.21.15.55:0/3761693236 learned_addr learned my addr 172.21.15.55:0/3761693236 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:06:16.132 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.133+0000 7f6186000640 1 --2- 172.21.15.55:0/3761693236 >> v2:172.21.15.84:3300/0 conn(0x7f61801585f0 0x7f6180158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:16.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.133+0000 7f6184ffe640 1 -- 172.21.15.55:0/3761693236 >> v2:172.21.15.84:3300/0 conn(0x7f61801585f0 msgr2=0x7f6180158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:16.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.133+0000 7f6184ffe640 1 --2- 172.21.15.55:0/3761693236 >> v2:172.21.15.84:3300/0 conn(0x7f61801585f0 0x7f6180158a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:16.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.133+0000 7f6184ffe640 1 -- 172.21.15.55:0/3761693236 >> v1:172.21.15.84:6789/0 conn(0x7f618014eaa0 legacy=0x7f618006b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:16.136 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.133+0000 7f6184ffe640 1 -- 172.21.15.55:0/3761693236 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61800c7aa0 con 0x7f61800c60d0 2024-03-19T02:06:16.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.141+0000 7f6184ffe640 1 --2- 172.21.15.55:0/3761693236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61800c60d0 0x7f61800c6540 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f6180152580 tx=0x7f617001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd1338d7fda7c592 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:16.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.145+0000 7f6177fff640 1 -- 172.21.15.55:0/3761693236 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61700b0070 con 0x7f61800c60d0 2024-03-19T02:06:16.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.145+0000 7f6177fff640 1 -- 172.21.15.55:0/3761693236 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61700ab600 con 0x7f61800c60d0 2024-03-19T02:06:16.145 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.145+0000 7f6177fff640 1 -- 172.21.15.55:0/3761693236 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61700ab9d0 con 0x7f61800c60d0 2024-03-19T02:06:16.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.145+0000 7f6187a63640 1 -- 172.21.15.55:0/3761693236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61800c60d0 msgr2=0x7f61800c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:16.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.145+0000 7f6187a63640 1 --2- 172.21.15.55:0/3761693236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61800c60d0 0x7f61800c6540 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f6180152580 tx=0x7f617001c3f0 comp rx=0 tx=0).stop 2024-03-19T02:06:16.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.145+0000 7f6187a63640 1 -- 172.21.15.55:0/3761693236 shutdown_connections 2024-03-19T02:06:16.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.145+0000 7f6187a63640 1 --2- 172.21.15.55:0/3761693236 >> v2:172.21.15.84:3300/0 conn(0x7f61801585f0 0x7f6180158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:16.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.145+0000 7f6187a63640 1 --2- 172.21.15.55:0/3761693236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61800c60d0 0x7f61800c6540 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:16.146 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.145+0000 7f6187a63640 1 -- 172.21.15.55:0/3761693236 >> 172.21.15.55:0/3761693236 conn(0x7f61800cc9c0 msgr2=0x7f61800ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:16.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.145+0000 7f6187a63640 1 -- 172.21.15.55:0/3761693236 shutdown_connections 2024-03-19T02:06:16.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.145+0000 7f6187a63640 1 -- 172.21.15.55:0/3761693236 wait complete. 2024-03-19T02:06:16.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.145+0000 7f6187a63640 1 Processor -- start 2024-03-19T02:06:16.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.145+0000 7f6187a63640 1 -- start start 2024-03-19T02:06:16.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.145+0000 7f6187a63640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61800c60d0 0x7f61800c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:16.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.145+0000 7f6187a63640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f618014eaa0 0x7f6180147c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:16.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.145+0000 7f6187a63640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f61801585f0 0x7f6180140230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:16.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.149+0000 7f6187a63640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f618006c210 con 0x7f618014eaa0 2024-03-19T02:06:16.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.149+0000 7f6187a63640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f618006c090 con 0x7f61800c60d0 2024-03-19T02:06:16.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.149+0000 7f6187a63640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f61800c6bf0 con 0x7f61801585f0 2024-03-19T02:06:16.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.149+0000 7f6186000640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f61801585f0 0x7f6180140230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:16.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.149+0000 7f61857ff640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61800c60d0 0x7f61800c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:16.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.149+0000 7f6186000640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f61801585f0 0x7f6180140230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:38728/0 (socket says 172.21.15.55:38728) 2024-03-19T02:06:16.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.149+0000 7f6186000640 1 -- 172.21.15.55:0/542405688 learned_addr learned my addr 172.21.15.55:0/542405688 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:16.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.149+0000 7f6184ffe640 1 --2- 172.21.15.55:0/542405688 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f618014eaa0 0x7f6180147c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:16.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.153+0000 7f6186000640 1 -- 172.21.15.55:0/542405688 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61800c60d0 msgr2=0x7f61800c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:16.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.153+0000 7f6186000640 1 --2- 172.21.15.55:0/542405688 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61800c60d0 0x7f61800c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:16.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.153+0000 7f6186000640 1 -- 172.21.15.55:0/542405688 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f618014eaa0 msgr2=0x7f6180147c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:16.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.153+0000 7f6186000640 1 --2- 172.21.15.55:0/542405688 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f618014eaa0 0x7f6180147c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:16.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.153+0000 7f6186000640 1 -- 172.21.15.55:0/542405688 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618014e260 con 0x7f61801585f0 2024-03-19T02:06:16.158 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.157+0000 7f6186000640 1 --2- 172.21.15.55:0/542405688 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f61801585f0 0x7f6180140230 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f617800efc0 tx=0x7f617800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:16.170 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.169+0000 7f6175ffb640 1 -- 172.21.15.55:0/542405688 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6178018020 con 0x7f61801585f0 2024-03-19T02:06:16.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.169+0000 7f6175ffb640 1 -- 172.21.15.55:0/542405688 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6178014480 con 0x7f61801585f0 2024-03-19T02:06:16.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.169+0000 7f6175ffb640 1 -- 172.21.15.55:0/542405688 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6178014870 con 0x7f61801585f0 2024-03-19T02:06:16.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.169+0000 7f6187a63640 1 -- 172.21.15.55:0/542405688 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61800c59f0 con 0x7f61801585f0 2024-03-19T02:06:16.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.169+0000 7f6187a63640 1 -- 172.21.15.55:0/542405688 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f61801414f0 con 0x7f61801585f0 2024-03-19T02:06:16.171 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.169+0000 7f6187a63640 1 -- 172.21.15.55:0/542405688 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6180141bb0 con 0x7f61801585f0 2024-03-19T02:06:16.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.265+0000 7f6175ffb640 1 -- 172.21.15.55:0/542405688 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f617801b020 con 0x7f61801585f0 2024-03-19T02:06:16.268 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.265+0000 7f6175ffb640 1 --2- 172.21.15.55:0/542405688 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6164067690 0x7f6164069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:16.268 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.265+0000 7f61857ff640 1 --2- 172.21.15.55:0/542405688 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6164067690 0x7f6164069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:16.268 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.265+0000 7f6175ffb640 1 -- 172.21.15.55:0/542405688 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f6178054f50 con 0x7f61801585f0 2024-03-19T02:06:16.268 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.265+0000 7f6175ffb640 1 --2- 172.21.15.55:0/542405688 >> [v2:172.21.15.55:6800/1760253550,v1:172.21.15.55:6801/1760253550] conn(0x7f6164071330 0x7f61640737d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:16.268 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.265+0000 7f6175ffb640 1 -- 172.21.15.55:0/542405688 --> [v2:172.21.15.55:6800/1760253550,v1:172.21.15.55:6801/1760253550] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f617800be50 con 0x7f6164071330 2024-03-19T02:06:16.268 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.265+0000 7f61857ff640 1 --2- 172.21.15.55:0/542405688 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6164067690 0x7f6164069b50 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f617c000f70 tx=0x7f617c007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:16.269 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.265+0000 7f6184ffe640 1 --2- 172.21.15.55:0/542405688 >> [v2:172.21.15.55:6800/1760253550,v1:172.21.15.55:6801/1760253550] conn(0x7f6164071330 0x7f61640737d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:16.269 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.269+0000 7f6184ffe640 1 --2- 172.21.15.55:0/542405688 >> [v2:172.21.15.55:6800/1760253550,v1:172.21.15.55:6801/1760253550] conn(0x7f6164071330 0x7f61640737d0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:16.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.269+0000 7f6175ffb640 1 -- 172.21.15.55:0/542405688 <== osd.2 v2:172.21.15.55:6800/1760253550 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f617800be50 con 0x7f6164071330 2024-03-19T02:06:16.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.273+0000 7f6175ffb640 1 -- 172.21.15.55:0/542405688 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6178053150 con 0x7f61801585f0 2024-03-19T02:06:16.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.285+0000 7f6187a63640 1 -- 172.21.15.55:0/542405688 --> [v2:172.21.15.55:6800/1760253550,v1:172.21.15.55:6801/1760253550] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6180068a40 con 0x7f6164071330 2024-03-19T02:06:16.288 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.285+0000 7f6175ffb640 1 -- 172.21.15.55:0/542405688 <== osd.2 v2:172.21.15.55:6800/1760253550 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f6180068a40 con 0x7f6164071330 2024-03-19T02:06:16.288 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-19T02:06:16.288 INFO:teuthology.orchestra.run.smithi055.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T02:06:16.288 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-19T02:06:16.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.285+0000 7f6187a63640 1 -- 172.21.15.55:0/542405688 >> [v2:172.21.15.55:6800/1760253550,v1:172.21.15.55:6801/1760253550] conn(0x7f6164071330 msgr2=0x7f61640737d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:16.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.285+0000 7f6187a63640 1 --2- 172.21.15.55:0/542405688 >> [v2:172.21.15.55:6800/1760253550,v1:172.21.15.55:6801/1760253550] conn(0x7f6164071330 0x7f61640737d0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:16.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.289+0000 7f6187a63640 1 -- 172.21.15.55:0/542405688 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6164067690 msgr2=0x7f6164069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:16.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.289+0000 7f6187a63640 1 --2- 172.21.15.55:0/542405688 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f6164067690 0x7f6164069b50 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f617c000f70 tx=0x7f617c007e90 comp rx=0 tx=0).stop 2024-03-19T02:06:16.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.289+0000 7f6187a63640 1 -- 172.21.15.55:0/542405688 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f61801585f0 msgr2=0x7f6180140230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:16.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.289+0000 7f6187a63640 1 --2- 172.21.15.55:0/542405688 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f61801585f0 0x7f6180140230 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f617800efc0 tx=0x7f617800c490 comp rx=0 tx=0).stop 2024-03-19T02:06:16.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.289+0000 7f6186000640 1 -- 172.21.15.55:0/542405688 reap_dead start 2024-03-19T02:06:16.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.289+0000 7f6187a63640 1 -- 172.21.15.55:0/542405688 shutdown_connections 2024-03-19T02:06:16.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.289+0000 7f6187a63640 1 -- 172.21.15.55:0/542405688 >> 172.21.15.55:0/542405688 conn(0x7f61800cc9c0 msgr2=0x7f61800c31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:16.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.289+0000 7f6187a63640 1 -- 172.21.15.55:0/542405688 shutdown_connections 2024-03-19T02:06:16.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.289+0000 7f6187a63640 1 -- 172.21.15.55:0/542405688 wait complete. 2024-03-19T02:06:16.309 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-19T02:06:16.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.425+0000 7f82799bd640 1 Processor -- start 2024-03-19T02:06:16.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.425+0000 7f82799bd640 1 -- start start 2024-03-19T02:06:16.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.425+0000 7f82799bd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f82741544b0 0x7f827406b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:16.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.425+0000 7f82799bd640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f82740c9050 0x7f82740c9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:16.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.425+0000 7f82799bd640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f827415a6b0 con 0x7f827415a0e0 2024-03-19T02:06:16.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.425+0000 7f82799bd640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f82740c99a0 con 0x7f82741544b0 2024-03-19T02:06:16.428 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.425+0000 7f82799bd640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f827415a990 con 0x7f82740c9050 2024-03-19T02:06:16.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.429+0000 7f8272ffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f82741544b0 0x7f827406b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:16.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.429+0000 7f82737fe640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f82740c9050 0x7f82740c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:16.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.429+0000 7f8272ffd640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f82741544b0 0x7f827406b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57294/0 (socket says 172.21.15.55:57294) 2024-03-19T02:06:16.429 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.429+0000 7f8272ffd640 1 -- 172.21.15.55:0/3707268520 learned_addr learned my addr 172.21.15.55:0/3707268520 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:16.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.429+0000 7f8272ffd640 1 -- 172.21.15.55:0/3707268520 >> v2:172.21.15.84:3300/0 conn(0x7f82740c9050 msgr2=0x7f82740c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:16.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.429+0000 7f8272ffd640 1 --2- 172.21.15.55:0/3707268520 >> v2:172.21.15.84:3300/0 conn(0x7f82740c9050 0x7f82740c9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:16.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.429+0000 7f8272ffd640 1 -- 172.21.15.55:0/3707268520 >> v1:172.21.15.84:6789/0 conn(0x7f827415a0e0 legacy=0x7f82740cbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:16.432 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.429+0000 7f8272ffd640 1 -- 172.21.15.55:0/3707268520 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f827415ac70 con 0x7f82741544b0 2024-03-19T02:06:16.435 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.433+0000 7f8272ffd640 1 --2- 172.21.15.55:0/3707268520 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82741544b0 0x7f827406b2a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f82740c9ed0 tx=0x7f826401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=326c88199d051ae4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:16.439 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.437+0000 7f8271ffb640 1 -- 172.21.15.55:0/3707268520 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82640b5070 con 0x7f82741544b0 2024-03-19T02:06:16.439 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.437+0000 7f8271ffb640 1 -- 172.21.15.55:0/3707268520 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82640a8a10 con 0x7f82741544b0 2024-03-19T02:06:16.440 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.437+0000 7f8271ffb640 1 -- 172.21.15.55:0/3707268520 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82640a8de0 con 0x7f82741544b0 2024-03-19T02:06:16.440 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.441+0000 7f82799bd640 1 -- 172.21.15.55:0/3707268520 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82741544b0 msgr2=0x7f827406b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:16.441 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.441+0000 7f82799bd640 1 --2- 172.21.15.55:0/3707268520 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82741544b0 0x7f827406b2a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f82740c9ed0 tx=0x7f826401ed10 comp rx=0 tx=0).stop 2024-03-19T02:06:16.441 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.441+0000 7f82799bd640 1 -- 172.21.15.55:0/3707268520 shutdown_connections 2024-03-19T02:06:16.441 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.441+0000 7f82799bd640 1 --2- 172.21.15.55:0/3707268520 >> v2:172.21.15.84:3300/0 conn(0x7f82740c9050 0x7f82740c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:16.441 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.441+0000 7f82799bd640 1 --2- 172.21.15.55:0/3707268520 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82741544b0 0x7f827406b2a0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:16.441 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.441+0000 7f82799bd640 1 -- 172.21.15.55:0/3707268520 >> 172.21.15.55:0/3707268520 conn(0x7f827406e5a0 msgr2=0x7f8274072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:16.441 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.441+0000 7f82799bd640 1 -- 172.21.15.55:0/3707268520 shutdown_connections 2024-03-19T02:06:16.441 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.441+0000 7f82799bd640 1 -- 172.21.15.55:0/3707268520 wait complete. 2024-03-19T02:06:16.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.441+0000 7f82799bd640 1 Processor -- start 2024-03-19T02:06:16.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.441+0000 7f82799bd640 1 -- start start 2024-03-19T02:06:16.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.441+0000 7f82799bd640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82740c9050 0x7f82740c49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:16.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.441+0000 7f82799bd640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f82741544b0 0x7f82740c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:16.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.441+0000 7f82799bd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f827415a0e0 0x7f82741408a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:16.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.441+0000 7f82799bd640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f82740c6be0 con 0x7f827415a0e0 2024-03-19T02:06:16.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.441+0000 7f82799bd640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f82740c8b30 con 0x7f82740c9050 2024-03-19T02:06:16.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.441+0000 7f82799bd640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f827406b7e0 con 0x7f82741544b0 2024-03-19T02:06:16.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.441+0000 7f82737fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f827415a0e0 0x7f82741408a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:16.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.441+0000 7f82737fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f827415a0e0 0x7f82741408a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57298/0 (socket says 172.21.15.55:57298) 2024-03-19T02:06:16.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.441+0000 7f82737fe640 1 -- 172.21.15.55:0/1970897131 learned_addr learned my addr 172.21.15.55:0/1970897131 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:16.444 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.441+0000 7f82727fc640 1 --2- 172.21.15.55:0/1970897131 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f82741544b0 0x7f82740c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:16.445 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.445+0000 7f8272ffd640 1 --2- 172.21.15.55:0/1970897131 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82740c9050 0x7f82740c49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:16.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.445+0000 7f82737fe640 1 -- 172.21.15.55:0/1970897131 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f82741544b0 msgr2=0x7f82740c4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:16.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.445+0000 7f82737fe640 1 --2- 172.21.15.55:0/1970897131 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f82741544b0 0x7f82740c4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:16.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.445+0000 7f82737fe640 1 -- 172.21.15.55:0/1970897131 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82740c9050 msgr2=0x7f82740c49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:16.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.445+0000 7f82737fe640 1 --2- 172.21.15.55:0/1970897131 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82740c9050 0x7f82740c49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:16.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.445+0000 7f82737fe640 1 -- 172.21.15.55:0/1970897131 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82740ca9b0 con 0x7f827415a0e0 2024-03-19T02:06:16.449 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.449+0000 7f82737fe640 1 --2- 172.21.15.55:0/1970897131 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f827415a0e0 0x7f82741408a0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f826c009d30 tx=0x7f826c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:16.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.453+0000 7f825bfff640 1 -- 172.21.15.55:0/1970897131 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c018020 con 0x7f827415a0e0 2024-03-19T02:06:16.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.453+0000 7f825bfff640 1 -- 172.21.15.55:0/1970897131 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f826c014480 con 0x7f827415a0e0 2024-03-19T02:06:16.453 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.453+0000 7f825bfff640 1 -- 172.21.15.55:0/1970897131 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c014870 con 0x7f827415a0e0 2024-03-19T02:06:16.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.453+0000 7f82799bd640 1 -- 172.21.15.55:0/1970897131 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82740cac90 con 0x7f827415a0e0 2024-03-19T02:06:16.454 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.453+0000 7f82799bd640 1 -- 172.21.15.55:0/1970897131 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82740cb250 con 0x7f827415a0e0 2024-03-19T02:06:16.455 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.453+0000 7f82799bd640 1 -- 172.21.15.55:0/1970897131 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8274068a40 con 0x7f827415a0e0 2024-03-19T02:06:16.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.729+0000 7f825bfff640 1 -- 172.21.15.55:0/1970897131 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f826c01b020 con 0x7f827415a0e0 2024-03-19T02:06:16.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.729+0000 7f825bfff640 1 --2- 172.21.15.55:0/1970897131 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f82600676b0 0x7f8260069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:16.731 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.729+0000 7f8272ffd640 1 --2- 172.21.15.55:0/1970897131 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f82600676b0 0x7f8260069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:16.732 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.733+0000 7f8272ffd640 1 --2- 172.21.15.55:0/1970897131 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f82600676b0 0x7f8260069b70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f82640b2000 tx=0x7f82640aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:16.753 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.753+0000 7f825bfff640 1 -- 172.21.15.55:0/1970897131 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f826c055070 con 0x7f827415a0e0 2024-03-19T02:06:16.754 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.753+0000 7f825bfff640 1 -- 172.21.15.55:0/1970897131 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f826c022070 con 0x7f827415a0e0 2024-03-19T02:06:16.946 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.945+0000 7f82799bd640 1 -- 172.21.15.55:0/1970897131 --> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f827406bb10 con 0x7f82600676b0 2024-03-19T02:06:16.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.945+0000 7f825bfff640 1 -- 172.21.15.55:0/1970897131 <== mgr.4105 v2:172.21.15.84:6800/998533783 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f827406bb10 con 0x7f82600676b0 2024-03-19T02:06:16.947 INFO:teuthology.orchestra.run.smithi055.stderr:instructed osd(s) 2 to deep-scrub 2024-03-19T02:06:16.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.949+0000 7f82799bd640 1 -- 172.21.15.55:0/1970897131 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f82600676b0 msgr2=0x7f8260069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:16.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.949+0000 7f82799bd640 1 --2- 172.21.15.55:0/1970897131 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f82600676b0 0x7f8260069b70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f82640b2000 tx=0x7f82640aa070 comp rx=0 tx=0).stop 2024-03-19T02:06:16.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.949+0000 7f82799bd640 1 -- 172.21.15.55:0/1970897131 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f827415a0e0 msgr2=0x7f82741408a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:16.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.949+0000 7f82799bd640 1 --2- 172.21.15.55:0/1970897131 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f827415a0e0 0x7f82741408a0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f826c009d30 tx=0x7f826c00c490 comp rx=0 tx=0).stop 2024-03-19T02:06:16.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.949+0000 7f82799bd640 1 -- 172.21.15.55:0/1970897131 shutdown_connections 2024-03-19T02:06:16.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.949+0000 7f82799bd640 1 --2- 172.21.15.55:0/1970897131 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f82600676b0 0x7f8260069b70 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:16.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.949+0000 7f82799bd640 1 --2- 172.21.15.55:0/1970897131 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f827415a0e0 0x7f82741408a0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:16.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.949+0000 7f82799bd640 1 --2- 172.21.15.55:0/1970897131 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f82741544b0 0x7f82740c4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:16.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.949+0000 7f82799bd640 1 --2- 172.21.15.55:0/1970897131 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82740c9050 0x7f82740c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:16.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.949+0000 7f82799bd640 1 -- 172.21.15.55:0/1970897131 >> 172.21.15.55:0/1970897131 conn(0x7f827406e5a0 msgr2=0x7f827406bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:16.949 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.949+0000 7f82799bd640 1 -- 172.21.15.55:0/1970897131 shutdown_connections 2024-03-19T02:06:16.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:16.949+0000 7f82799bd640 1 -- 172.21.15.55:0/1970897131 wait complete. 2024-03-19T02:06:16.964 INFO:tasks.ceph:Scrubbing osd.3 2024-03-19T02:06:16.964 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-19T02:06:17.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.089+0000 7f2cb6a29640 1 Processor -- start 2024-03-19T02:06:17.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.089+0000 7f2cb6a29640 1 -- start start 2024-03-19T02:06:17.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.089+0000 7f2cb6a29640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f2cb00c96b0 0x7f2cb0070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:17.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.089+0000 7f2cb6a29640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2cb01540c0 0x7f2cb0154570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:17.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.089+0000 7f2cb6a29640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cb015e3a0 con 0x7f2cb00c9a90 2024-03-19T02:06:17.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.089+0000 7f2cb6a29640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f2cb00c7ad0 con 0x7f2cb00c96b0 2024-03-19T02:06:17.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.089+0000 7f2cb6a29640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2cb015e680 con 0x7f2cb01540c0 2024-03-19T02:06:17.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.089+0000 7f2cb4fc6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2cb01540c0 0x7f2cb0154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:17.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.089+0000 7f2caffff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f2cb00c96b0 0x7f2cb0070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:17.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.089+0000 7f2cb4fc6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2cb01540c0 0x7f2cb0154570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:38744/0 (socket says 172.21.15.55:38744) 2024-03-19T02:06:17.091 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.089+0000 7f2cb4fc6640 1 -- 172.21.15.55:0/1437231564 learned_addr learned my addr 172.21.15.55:0/1437231564 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:17.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.093+0000 7f2cb4fc6640 1 -- 172.21.15.55:0/1437231564 >> v2:172.21.15.84:3300/0 conn(0x7f2cb00c96b0 msgr2=0x7f2cb0070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:17.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.093+0000 7f2cb4fc6640 1 --2- 172.21.15.55:0/1437231564 >> v2:172.21.15.84:3300/0 conn(0x7f2cb00c96b0 0x7f2cb0070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:17.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.093+0000 7f2cb4fc6640 1 -- 172.21.15.55:0/1437231564 >> v1:172.21.15.84:6789/0 conn(0x7f2cb00c9a90 legacy=0x7f2cb00c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:17.094 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.093+0000 7f2cb4fc6640 1 -- 172.21.15.55:0/1437231564 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cb015a330 con 0x7f2cb01540c0 2024-03-19T02:06:17.098 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.097+0000 7f2cb4fc6640 1 --2- 172.21.15.55:0/1437231564 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2cb01540c0 0x7f2cb0154570 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8009e50 tx=0x7f2ca801c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38a52e4de67209f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:17.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.105+0000 7f2caeffd640 1 -- 172.21.15.55:0/1437231564 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca80a5750 con 0x7f2cb01540c0 2024-03-19T02:06:17.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.105+0000 7f2caeffd640 1 -- 172.21.15.55:0/1437231564 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ca80a58f0 con 0x7f2cb01540c0 2024-03-19T02:06:17.104 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.105+0000 7f2caeffd640 1 -- 172.21.15.55:0/1437231564 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca80a5cc0 con 0x7f2cb01540c0 2024-03-19T02:06:17.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.105+0000 7f2cb6a29640 1 -- 172.21.15.55:0/1437231564 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2cb01540c0 msgr2=0x7f2cb0154570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:17.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.105+0000 7f2cb6a29640 1 --2- 172.21.15.55:0/1437231564 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2cb01540c0 0x7f2cb0154570 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8009e50 tx=0x7f2ca801c9a0 comp rx=0 tx=0).stop 2024-03-19T02:06:17.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.105+0000 7f2cb6a29640 1 -- 172.21.15.55:0/1437231564 shutdown_connections 2024-03-19T02:06:17.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.105+0000 7f2cb6a29640 1 --2- 172.21.15.55:0/1437231564 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2cb01540c0 0x7f2cb0154570 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:17.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.105+0000 7f2cb6a29640 1 --2- 172.21.15.55:0/1437231564 >> v2:172.21.15.84:3300/0 conn(0x7f2cb00c96b0 0x7f2cb0070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:17.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.105+0000 7f2cb6a29640 1 -- 172.21.15.55:0/1437231564 >> 172.21.15.55:0/1437231564 conn(0x7f2cb00cc9c0 msgr2=0x7f2cb00ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:17.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.105+0000 7f2cb6a29640 1 -- 172.21.15.55:0/1437231564 shutdown_connections 2024-03-19T02:06:17.105 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.105+0000 7f2cb6a29640 1 -- 172.21.15.55:0/1437231564 wait complete. 2024-03-19T02:06:17.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.105+0000 7f2cb6a29640 1 Processor -- start 2024-03-19T02:06:17.106 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.105+0000 7f2cb6a29640 1 -- start start 2024-03-19T02:06:17.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.105+0000 7f2cb6a29640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2cb00c96b0 0x7f2cb0152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:17.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.105+0000 7f2cb6a29640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2cb00c9a90 0x7f2cb0145af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:17.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.105+0000 7f2cb6a29640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2cb01540c0 0x7f2cb0144070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:17.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.105+0000 7f2cb6a29640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb0166b90 con 0x7f2cb01540c0 2024-03-19T02:06:17.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.105+0000 7f2cb6a29640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb015b0f0 con 0x7f2cb00c9a90 2024-03-19T02:06:17.107 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.105+0000 7f2cb6a29640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2cb0166d10 con 0x7f2cb00c96b0 2024-03-19T02:06:17.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.105+0000 7f2cb4fc6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2cb01540c0 0x7f2cb0144070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:17.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.105+0000 7f2caf7fe640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2cb00c9a90 0x7f2cb0145af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:17.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.109+0000 7f2cb4fc6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2cb01540c0 0x7f2cb0144070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57308/0 (socket says 172.21.15.55:57308) 2024-03-19T02:06:17.108 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.109+0000 7f2cb4fc6640 1 -- 172.21.15.55:0/1542485007 learned_addr learned my addr 172.21.15.55:0/1542485007 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:17.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.109+0000 7f2caffff640 1 --2- 172.21.15.55:0/1542485007 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2cb00c96b0 0x7f2cb0152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:17.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.109+0000 7f2cb4fc6640 1 -- 172.21.15.55:0/1542485007 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2cb00c96b0 msgr2=0x7f2cb0152ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:17.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.109+0000 7f2cb4fc6640 1 --2- 172.21.15.55:0/1542485007 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2cb00c96b0 0x7f2cb0152ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:17.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.109+0000 7f2cb4fc6640 1 -- 172.21.15.55:0/1542485007 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2cb00c9a90 msgr2=0x7f2cb0145af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:17.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.109+0000 7f2cb4fc6640 1 --2- 172.21.15.55:0/1542485007 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2cb00c9a90 0x7f2cb0145af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:17.112 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.109+0000 7f2cb4fc6640 1 -- 172.21.15.55:0/1542485007 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cb01f6390 con 0x7f2cb01540c0 2024-03-19T02:06:17.114 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.113+0000 7f2cb4fc6640 1 --2- 172.21.15.55:0/1542485007 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2cb01540c0 0x7f2cb0144070 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f2ca80af040 tx=0x7f2ca8019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:17.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.117+0000 7f2cacff9640 1 -- 172.21.15.55:0/1542485007 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca80bc070 con 0x7f2cb01540c0 2024-03-19T02:06:17.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.117+0000 7f2cacff9640 1 -- 172.21.15.55:0/1542485007 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ca80b8640 con 0x7f2cb01540c0 2024-03-19T02:06:17.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.117+0000 7f2cb6a29640 1 -- 172.21.15.55:0/1542485007 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cb0148a10 con 0x7f2cb01540c0 2024-03-19T02:06:17.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.117+0000 7f2cacff9640 1 -- 172.21.15.55:0/1542485007 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca80b8a10 con 0x7f2cb01540c0 2024-03-19T02:06:17.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.117+0000 7f2cb6a29640 1 -- 172.21.15.55:0/1542485007 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cb01f3ce0 con 0x7f2cb01540c0 2024-03-19T02:06:17.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.121+0000 7f2cb6a29640 1 -- 172.21.15.55:0/1542485007 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2cb01f6600 con 0x7f2cb01540c0 2024-03-19T02:06:17.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.145+0000 7f2cacff9640 1 -- 172.21.15.55:0/1542485007 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f2ca80b8bb0 con 0x7f2cb01540c0 2024-03-19T02:06:17.147 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.145+0000 7f2cacff9640 1 --2- 172.21.15.55:0/1542485007 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2c9c0676e0 0x7f2c9c069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:17.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.149+0000 7f2caffff640 1 --2- 172.21.15.55:0/1542485007 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2c9c0676e0 0x7f2c9c069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:17.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.149+0000 7f2caffff640 1 --2- 172.21.15.55:0/1542485007 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2c9c0676e0 0x7f2c9c069ba0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0009ac0 tx=0x7f2ca0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:17.151 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.149+0000 7f2cacff9640 1 -- 172.21.15.55:0/1542485007 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f2ca80eefb0 con 0x7f2cb01540c0 2024-03-19T02:06:17.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.149+0000 7f2cacff9640 1 --2- 172.21.15.55:0/1542485007 >> [v2:172.21.15.55:6808/2797616448,v1:172.21.15.55:6809/2797616448] conn(0x7f2c9c071400 0x7f2c9c0738a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:17.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.149+0000 7f2cacff9640 1 -- 172.21.15.55:0/1542485007 --> [v2:172.21.15.55:6808/2797616448,v1:172.21.15.55:6809/2797616448] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2c9c073f70 con 0x7f2c9c071400 2024-03-19T02:06:17.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.149+0000 7f2cacff9640 1 -- 172.21.15.55:0/1542485007 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2ca80ed1b0 con 0x7f2cb01540c0 2024-03-19T02:06:17.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.149+0000 7f2caf7fe640 1 --2- 172.21.15.55:0/1542485007 >> [v2:172.21.15.55:6808/2797616448,v1:172.21.15.55:6809/2797616448] conn(0x7f2c9c071400 0x7f2c9c0738a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:17.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.153+0000 7f2caf7fe640 1 --2- 172.21.15.55:0/1542485007 >> [v2:172.21.15.55:6808/2797616448,v1:172.21.15.55:6809/2797616448] conn(0x7f2c9c071400 0x7f2c9c0738a0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:17.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.153+0000 7f2cacff9640 1 -- 172.21.15.55:0/1542485007 <== osd.3 v2:172.21.15.55:6808/2797616448 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2c9c073f70 con 0x7f2c9c071400 2024-03-19T02:06:17.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.169+0000 7f2cb6a29640 1 -- 172.21.15.55:0/1542485007 --> [v2:172.21.15.55:6808/2797616448,v1:172.21.15.55:6809/2797616448] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f2cb0068a40 con 0x7f2c9c071400 2024-03-19T02:06:17.172 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.169+0000 7f2cacff9640 1 -- 172.21.15.55:0/1542485007 <== osd.3 v2:172.21.15.55:6808/2797616448 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f2cb0068a40 con 0x7f2c9c071400 2024-03-19T02:06:17.172 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-19T02:06:17.172 INFO:teuthology.orchestra.run.smithi055.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T02:06:17.172 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-19T02:06:17.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.173+0000 7f2cb6a29640 1 -- 172.21.15.55:0/1542485007 >> [v2:172.21.15.55:6808/2797616448,v1:172.21.15.55:6809/2797616448] conn(0x7f2c9c071400 msgr2=0x7f2c9c0738a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:17.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.173+0000 7f2cb6a29640 1 --2- 172.21.15.55:0/1542485007 >> [v2:172.21.15.55:6808/2797616448,v1:172.21.15.55:6809/2797616448] conn(0x7f2c9c071400 0x7f2c9c0738a0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:17.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.173+0000 7f2cb6a29640 1 -- 172.21.15.55:0/1542485007 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2c9c0676e0 msgr2=0x7f2c9c069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:17.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.173+0000 7f2cb6a29640 1 --2- 172.21.15.55:0/1542485007 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f2c9c0676e0 0x7f2c9c069ba0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0009ac0 tx=0x7f2ca0009190 comp rx=0 tx=0).stop 2024-03-19T02:06:17.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.173+0000 7f2cb6a29640 1 -- 172.21.15.55:0/1542485007 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2cb01540c0 msgr2=0x7f2cb0144070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:17.173 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.173+0000 7f2cb6a29640 1 --2- 172.21.15.55:0/1542485007 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2cb01540c0 0x7f2cb0144070 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f2ca80af040 tx=0x7f2ca8019740 comp rx=0 tx=0).stop 2024-03-19T02:06:17.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.173+0000 7f2cb4fc6640 1 -- 172.21.15.55:0/1542485007 reap_dead start 2024-03-19T02:06:17.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.173+0000 7f2cb6a29640 1 -- 172.21.15.55:0/1542485007 shutdown_connections 2024-03-19T02:06:17.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.173+0000 7f2cb6a29640 1 -- 172.21.15.55:0/1542485007 >> 172.21.15.55:0/1542485007 conn(0x7f2cb00cc9c0 msgr2=0x7f2cb00c6170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:17.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.173+0000 7f2cb6a29640 1 -- 172.21.15.55:0/1542485007 shutdown_connections 2024-03-19T02:06:17.174 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.173+0000 7f2cb6a29640 1 -- 172.21.15.55:0/1542485007 wait complete. 2024-03-19T02:06:17.194 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-19T02:06:17.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.305+0000 7f5b9b764640 1 Processor -- start 2024-03-19T02:06:17.305 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.305+0000 7f5b9b764640 1 -- start start 2024-03-19T02:06:17.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.305+0000 7f5b9b764640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5b941510f0 0x7f5b940c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:17.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.305+0000 7f5b9b764640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f5b9415a410 0x7f5b940708a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:17.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.305+0000 7f5b9b764640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f5b940c73c0 0x7f5b940c7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:17.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.305+0000 7f5b9b764640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f5b940c3f60 con 0x7f5b940c73c0 2024-03-19T02:06:17.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.305+0000 7f5b9b764640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f5b941514d0 con 0x7f5b9415a410 2024-03-19T02:06:17.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.305+0000 7f5b9b764640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f5b940c6870 con 0x7f5b941510f0 2024-03-19T02:06:17.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.305+0000 7f5b99500640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5b941510f0 0x7f5b940c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:17.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.305+0000 7f5b99500640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5b941510f0 0x7f5b940c61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:38766/0 (socket says 172.21.15.55:38766) 2024-03-19T02:06:17.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.305+0000 7f5b99500640 1 -- 172.21.15.55:0/1865230848 learned_addr learned my addr 172.21.15.55:0/1865230848 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:17.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.309+0000 7f5b99500640 1 -- 172.21.15.55:0/1865230848 >> v2:172.21.15.84:3300/0 conn(0x7f5b9415a410 msgr2=0x7f5b940708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:17.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.309+0000 7f5b99500640 1 --2- 172.21.15.55:0/1865230848 >> v2:172.21.15.84:3300/0 conn(0x7f5b9415a410 0x7f5b940708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:17.309 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.309+0000 7f5b99500640 1 -- 172.21.15.55:0/1865230848 >> v2:172.21.15.55:3300/0 conn(0x7f5b940c73c0 msgr2=0x7f5b940c7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:17.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.309+0000 7f5b99500640 1 --2- 172.21.15.55:0/1865230848 >> v2:172.21.15.55:3300/0 conn(0x7f5b940c73c0 0x7f5b940c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:17.310 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.309+0000 7f5b99500640 1 -- 172.21.15.55:0/1865230848 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b940c6b50 con 0x7f5b941510f0 2024-03-19T02:06:17.313 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.313+0000 7f5b99500640 1 --2- 172.21.15.55:0/1865230848 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5b941510f0 0x7f5b940c61c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5b940ca580 tx=0x7f5b9001ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4bfc4eb8136c8e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:17.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.317+0000 7f5b8bfff640 1 -- 172.21.15.55:0/1865230848 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b900b5070 con 0x7f5b941510f0 2024-03-19T02:06:17.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.317+0000 7f5b8bfff640 1 -- 172.21.15.55:0/1865230848 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b900a89d0 con 0x7f5b941510f0 2024-03-19T02:06:17.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.317+0000 7f5b8bfff640 1 -- 172.21.15.55:0/1865230848 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b900a8da0 con 0x7f5b941510f0 2024-03-19T02:06:17.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.317+0000 7f5b9b764640 1 -- 172.21.15.55:0/1865230848 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5b941510f0 msgr2=0x7f5b940c61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:17.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.317+0000 7f5b9b764640 1 --2- 172.21.15.55:0/1865230848 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5b941510f0 0x7f5b940c61c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5b940ca580 tx=0x7f5b9001ed10 comp rx=0 tx=0).stop 2024-03-19T02:06:17.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.317+0000 7f5b9b764640 1 -- 172.21.15.55:0/1865230848 shutdown_connections 2024-03-19T02:06:17.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.317+0000 7f5b9b764640 1 --2- 172.21.15.55:0/1865230848 >> v2:172.21.15.55:3300/0 conn(0x7f5b940c73c0 0x7f5b940c7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:17.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.317+0000 7f5b9b764640 1 --2- 172.21.15.55:0/1865230848 >> v2:172.21.15.84:3300/0 conn(0x7f5b9415a410 0x7f5b940708a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:17.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.317+0000 7f5b9b764640 1 --2- 172.21.15.55:0/1865230848 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5b941510f0 0x7f5b940c61c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:17.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.317+0000 7f5b9b764640 1 -- 172.21.15.55:0/1865230848 >> 172.21.15.55:0/1865230848 conn(0x7f5b940cc9c0 msgr2=0x7f5b940ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:17.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.317+0000 7f5b9b764640 1 -- 172.21.15.55:0/1865230848 shutdown_connections 2024-03-19T02:06:17.320 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.317+0000 7f5b9b764640 1 -- 172.21.15.55:0/1865230848 wait complete. 2024-03-19T02:06:17.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.321+0000 7f5b9b764640 1 Processor -- start 2024-03-19T02:06:17.321 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.321+0000 7f5b9b764640 1 -- start start 2024-03-19T02:06:17.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.321+0000 7f5b9b764640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5b940c73c0 0x7f5b9414f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:17.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.321+0000 7f5b9b764640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b941510f0 0x7f5b9406b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:17.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.321+0000 7f5b9b764640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5b9415a410 0x7f5b94149250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:17.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.321+0000 7f5b9b764640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5b940c6e30 con 0x7f5b9415a410 2024-03-19T02:06:17.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.321+0000 7f5b9b764640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f5b940c44e0 con 0x7f5b941510f0 2024-03-19T02:06:17.323 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.321+0000 7f5b9b764640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f5b940c4a00 con 0x7f5b940c73c0 2024-03-19T02:06:17.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.321+0000 7f5b99500640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5b940c73c0 0x7f5b9414f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:17.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.325+0000 7f5b99500640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5b940c73c0 0x7f5b9414f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:38768/0 (socket says 172.21.15.55:38768) 2024-03-19T02:06:17.324 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.325+0000 7f5b99500640 1 -- 172.21.15.55:0/772974723 learned_addr learned my addr 172.21.15.55:0/772974723 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:17.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.329+0000 7f5b99500640 1 -- 172.21.15.55:0/772974723 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b941510f0 msgr2=0x7f5b9406b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:17.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.329+0000 7f5b99500640 1 --2- 172.21.15.55:0/772974723 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b941510f0 0x7f5b9406b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:17.329 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.329+0000 7f5b99500640 1 -- 172.21.15.55:0/772974723 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5b9415a410 msgr2=0x7f5b94149250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:17.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.329+0000 7f5b99500640 1 --2- 172.21.15.55:0/772974723 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5b9415a410 0x7f5b94149250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:17.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.329+0000 7f5b99500640 1 -- 172.21.15.55:0/772974723 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b9406cfa0 con 0x7f5b940c73c0 2024-03-19T02:06:17.333 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.333+0000 7f5b99500640 1 --2- 172.21.15.55:0/772974723 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5b940c73c0 0x7f5b9414f070 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5b900b3040 tx=0x7f5b9001bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:17.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.337+0000 7f5b89ffb640 1 -- 172.21.15.55:0/772974723 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b900c0070 con 0x7f5b940c73c0 2024-03-19T02:06:17.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.337+0000 7f5b89ffb640 1 -- 172.21.15.55:0/772974723 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b900b08d0 con 0x7f5b940c73c0 2024-03-19T02:06:17.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.337+0000 7f5b9b764640 1 -- 172.21.15.55:0/772974723 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b941eff60 con 0x7f5b940c73c0 2024-03-19T02:06:17.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.337+0000 7f5b89ffb640 1 -- 172.21.15.55:0/772974723 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b900b5040 con 0x7f5b940c73c0 2024-03-19T02:06:17.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.337+0000 7f5b9b764640 1 -- 172.21.15.55:0/772974723 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b941f0520 con 0x7f5b940c73c0 2024-03-19T02:06:17.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.341+0000 7f5b9b764640 1 -- 172.21.15.55:0/772974723 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b94068a40 con 0x7f5b940c73c0 2024-03-19T02:06:17.641 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.641+0000 7f5b89ffb640 1 -- 172.21.15.55:0/772974723 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f5b900c3070 con 0x7f5b940c73c0 2024-03-19T02:06:17.642 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.641+0000 7f5b89ffb640 1 --2- 172.21.15.55:0/772974723 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5b780676b0 0x7f5b78069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:17.642 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.641+0000 7f5b98cff640 1 --2- 172.21.15.55:0/772974723 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5b780676b0 0x7f5b78069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:17.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.641+0000 7f5b98cff640 1 --2- 172.21.15.55:0/772974723 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5b780676b0 0x7f5b78069b70 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5b940cc8d0 tx=0x7f5b84006ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:17.643 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.641+0000 7f5b89ffb640 1 -- 172.21.15.55:0/772974723 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f5b900f12f0 con 0x7f5b940c73c0 2024-03-19T02:06:17.656 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.653+0000 7f5b89ffb640 1 -- 172.21.15.55:0/772974723 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f5b900ca080 con 0x7f5b940c73c0 2024-03-19T02:06:17.873 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.873+0000 7f5b9b764640 1 -- 172.21.15.55:0/772974723 --> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f5b9414d740 con 0x7f5b780676b0 2024-03-19T02:06:17.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.873+0000 7f5b89ffb640 1 -- 172.21.15.55:0/772974723 <== mgr.4105 v2:172.21.15.84:6800/998533783 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f5b9414d740 con 0x7f5b780676b0 2024-03-19T02:06:17.883 INFO:teuthology.orchestra.run.smithi055.stderr:instructed osd(s) 3 to deep-scrub 2024-03-19T02:06:17.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.873+0000 7f5b9b764640 1 -- 172.21.15.55:0/772974723 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5b780676b0 msgr2=0x7f5b78069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:17.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.873+0000 7f5b9b764640 1 --2- 172.21.15.55:0/772974723 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5b780676b0 0x7f5b78069b70 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5b940cc8d0 tx=0x7f5b84006ca0 comp rx=0 tx=0).stop 2024-03-19T02:06:17.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.873+0000 7f5b9b764640 1 -- 172.21.15.55:0/772974723 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5b940c73c0 msgr2=0x7f5b9414f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:17.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.873+0000 7f5b9b764640 1 --2- 172.21.15.55:0/772974723 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5b940c73c0 0x7f5b9414f070 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5b900b3040 tx=0x7f5b9001bab0 comp rx=0 tx=0).stop 2024-03-19T02:06:17.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.873+0000 7f5b9b764640 1 -- 172.21.15.55:0/772974723 shutdown_connections 2024-03-19T02:06:17.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.873+0000 7f5b9b764640 1 --2- 172.21.15.55:0/772974723 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f5b780676b0 0x7f5b78069b70 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:17.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.873+0000 7f5b9b764640 1 --2- 172.21.15.55:0/772974723 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5b9415a410 0x7f5b94149250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:17.884 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.873+0000 7f5b9b764640 1 --2- 172.21.15.55:0/772974723 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b941510f0 0x7f5b9406b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:17.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.873+0000 7f5b9b764640 1 --2- 172.21.15.55:0/772974723 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f5b940c73c0 0x7f5b9414f070 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:17.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.873+0000 7f5b9b764640 1 -- 172.21.15.55:0/772974723 >> 172.21.15.55:0/772974723 conn(0x7f5b940cc9c0 msgr2=0x7f5b940ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:17.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.877+0000 7f5b9b764640 1 -- 172.21.15.55:0/772974723 shutdown_connections 2024-03-19T02:06:17.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:17.877+0000 7f5b9b764640 1 -- 172.21.15.55:0/772974723 wait complete. 2024-03-19T02:06:17.893 INFO:tasks.ceph:Scrubbing osd.4 2024-03-19T02:06:17.893 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-19T02:06:18.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.009+0000 7fb48fdd9640 1 Processor -- start 2024-03-19T02:06:18.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.009+0000 7fb48fdd9640 1 -- start start 2024-03-19T02:06:18.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.009+0000 7fb48fdd9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb4880c73c0 0x7fb4880c7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:18.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.009+0000 7fb48fdd9640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4880c3f60 con 0x7fb4880c61c0 2024-03-19T02:06:18.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.009+0000 7fb48fdd9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb488070fb0 con 0x7fb4881510f0 2024-03-19T02:06:18.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.009+0000 7fb48fdd9640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fb4880c6630 con 0x7fb4880c73c0 2024-03-19T02:06:18.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.013+0000 7fb48d374640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fb4880c61c0 0x7fb4880708a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:40430/0 (socket says 172.21.15.55:40430) 2024-03-19T02:06:18.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.013+0000 7fb48d374640 1 -- 172.21.15.55:0/3900319503 learned_addr learned my addr 172.21.15.55:0/3900319503 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:06:18.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.013+0000 7fb48e376640 1 --2- 172.21.15.55:0/3900319503 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb4880c73c0 0x7fb4880c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:18.015 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.013+0000 7fb48cb73640 1 -- 172.21.15.55:0/3900319503 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2396111072 0 0) 0x7fb4880c3f60 con 0x7fb4880c61c0 2024-03-19T02:06:18.016 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.013+0000 7fb48cb73640 1 -- 172.21.15.55:0/3900319503 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb474005740 con 0x7fb4880c61c0 2024-03-19T02:06:18.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.017+0000 7fb48e376640 1 -- 172.21.15.55:0/3900319503 >> v1:172.21.15.84:6789/0 conn(0x7fb4881510f0 legacy=0x7fb48815a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:18.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.017+0000 7fb48e376640 1 -- 172.21.15.55:0/3900319503 >> v1:172.21.15.55:6789/0 conn(0x7fb4880c61c0 legacy=0x7fb4880708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:18.017 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.017+0000 7fb48e376640 1 -- 172.21.15.55:0/3900319503 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4881514d0 con 0x7fb4880c73c0 2024-03-19T02:06:18.021 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.021+0000 7fb48e376640 1 --2- 172.21.15.55:0/3900319503 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb4880c73c0 0x7fb4880c7830 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb480009dc0 tx=0x7fb48001f170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f093bebda553e42f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:18.025 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.025+0000 7fb48cb73640 1 -- 172.21.15.55:0/3900319503 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4800b4070 con 0x7fb4880c73c0 2024-03-19T02:06:18.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.029+0000 7fb48cb73640 1 -- 172.21.15.55:0/3900319503 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4800af600 con 0x7fb4880c73c0 2024-03-19T02:06:18.029 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.029+0000 7fb48cb73640 1 -- 172.21.15.55:0/3900319503 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4800af9b0 con 0x7fb4880c73c0 2024-03-19T02:06:18.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.029+0000 7fb48fdd9640 1 -- 172.21.15.55:0/3900319503 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb4880c73c0 msgr2=0x7fb4880c7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:18.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.029+0000 7fb48fdd9640 1 --2- 172.21.15.55:0/3900319503 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb4880c73c0 0x7fb4880c7830 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb480009dc0 tx=0x7fb48001f170 comp rx=0 tx=0).stop 2024-03-19T02:06:18.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.029+0000 7fb48fdd9640 1 -- 172.21.15.55:0/3900319503 shutdown_connections 2024-03-19T02:06:18.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.029+0000 7fb48fdd9640 1 --2- 172.21.15.55:0/3900319503 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb4880c73c0 0x7fb4880c7830 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:18.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.029+0000 7fb48fdd9640 1 -- 172.21.15.55:0/3900319503 >> 172.21.15.55:0/3900319503 conn(0x7fb4880cc9c0 msgr2=0x7fb4880ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:18.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.029+0000 7fb48fdd9640 1 -- 172.21.15.55:0/3900319503 shutdown_connections 2024-03-19T02:06:18.030 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.029+0000 7fb48fdd9640 1 -- 172.21.15.55:0/3900319503 wait complete. 2024-03-19T02:06:18.031 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.029+0000 7fb48fdd9640 1 Processor -- start 2024-03-19T02:06:18.031 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.029+0000 7fb48fdd9640 1 -- start start 2024-03-19T02:06:18.031 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.029+0000 7fb48fdd9640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb4880c61c0 0x7fb4881497d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:18.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.029+0000 7fb48fdd9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb4880c73c0 0x7fb48806c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:18.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.029+0000 7fb48fdd9640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb4881510f0 0x7fb4880ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:18.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.029+0000 7fb48fdd9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb4880c4200 con 0x7fb4880c73c0 2024-03-19T02:06:18.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.029+0000 7fb48fdd9640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb4880c6b50 con 0x7fb4881510f0 2024-03-19T02:06:18.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.029+0000 7fb48fdd9640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fb4880c44e0 con 0x7fb4880c61c0 2024-03-19T02:06:18.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.033+0000 7fb48d374640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb4880c73c0 0x7fb48806c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:18.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.033+0000 7fb48e376640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb4881510f0 0x7fb4880ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:18.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.033+0000 7fb48d374640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb4880c73c0 0x7fb48806c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57334/0 (socket says 172.21.15.55:57334) 2024-03-19T02:06:18.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.033+0000 7fb48db75640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb4880c61c0 0x7fb4881497d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:18.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.033+0000 7fb48d374640 1 -- 172.21.15.55:0/3137625871 learned_addr learned my addr 172.21.15.55:0/3137625871 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:18.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.037+0000 7fb48d374640 1 -- 172.21.15.55:0/3137625871 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb4880c61c0 msgr2=0x7fb4881497d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:18.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.037+0000 7fb48d374640 1 --2- 172.21.15.55:0/3137625871 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fb4880c61c0 0x7fb4881497d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:18.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.037+0000 7fb48d374640 1 -- 172.21.15.55:0/3137625871 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb4881510f0 msgr2=0x7fb4880ca7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:18.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.037+0000 7fb48d374640 1 --2- 172.21.15.55:0/3137625871 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb4881510f0 0x7fb4880ca7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:18.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.037+0000 7fb48d374640 1 -- 172.21.15.55:0/3137625871 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb48806b3b0 con 0x7fb4880c73c0 2024-03-19T02:06:18.041 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.041+0000 7fb48d374640 1 --2- 172.21.15.55:0/3137625871 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb4880c73c0 0x7fb48806c290 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fb47800c450 tx=0x7fb47800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:18.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.045+0000 7fb47e7fc640 1 -- 172.21.15.55:0/3137625871 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb478017070 con 0x7fb4880c73c0 2024-03-19T02:06:18.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.045+0000 7fb47e7fc640 1 -- 172.21.15.55:0/3137625871 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb478005cf0 con 0x7fb4880c73c0 2024-03-19T02:06:18.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.045+0000 7fb48fdd9640 1 -- 172.21.15.55:0/3137625871 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb48806b650 con 0x7fb4880c73c0 2024-03-19T02:06:18.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.045+0000 7fb47e7fc640 1 -- 172.21.15.55:0/3137625871 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb478007270 con 0x7fb4880c73c0 2024-03-19T02:06:18.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.045+0000 7fb48fdd9640 1 -- 172.21.15.55:0/3137625871 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb48806d080 con 0x7fb4880c73c0 2024-03-19T02:06:18.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.045+0000 7fb48fdd9640 1 -- 172.21.15.55:0/3137625871 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb4881493b0 con 0x7fb4880c73c0 2024-03-19T02:06:18.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.065+0000 7fb47e7fc640 1 -- 172.21.15.55:0/3137625871 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fb47801a070 con 0x7fb4880c73c0 2024-03-19T02:06:18.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.065+0000 7fb47e7fc640 1 --2- 172.21.15.55:0/3137625871 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb46c0676e0 0x7fb46c069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:18.067 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.065+0000 7fb48db75640 1 --2- 172.21.15.55:0/3137625871 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb46c0676e0 0x7fb46c069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:18.068 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.069+0000 7fb48db75640 1 --2- 172.21.15.55:0/3137625871 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb46c0676e0 0x7fb46c069ba0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb484007fd0 tx=0x7fb484009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:18.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.069+0000 7fb47e7fc640 1 -- 172.21.15.55:0/3137625871 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fb4780539c0 con 0x7fb4880c73c0 2024-03-19T02:06:18.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.069+0000 7fb47e7fc640 1 --2- 172.21.15.55:0/3137625871 >> [v2:172.21.15.84:6818/2938436826,v1:172.21.15.84:6819/2938436826] conn(0x7fb46c071390 0x7fb46c073830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:18.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.069+0000 7fb47e7fc640 1 -- 172.21.15.55:0/3137625871 --> [v2:172.21.15.84:6818/2938436826,v1:172.21.15.84:6819/2938436826] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb46c073f00 con 0x7fb46c071390 2024-03-19T02:06:18.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.069+0000 7fb47e7fc640 1 -- 172.21.15.55:0/3137625871 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb478051bc0 con 0x7fb4880c73c0 2024-03-19T02:06:18.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.069+0000 7fb48e376640 1 --2- 172.21.15.55:0/3137625871 >> [v2:172.21.15.84:6818/2938436826,v1:172.21.15.84:6819/2938436826] conn(0x7fb46c071390 0x7fb46c073830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:18.072 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.073+0000 7fb48e376640 1 --2- 172.21.15.55:0/3137625871 >> [v2:172.21.15.84:6818/2938436826,v1:172.21.15.84:6819/2938436826] conn(0x7fb46c071390 0x7fb46c073830 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:18.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.073+0000 7fb47e7fc640 1 -- 172.21.15.55:0/3137625871 <== osd.4 v2:172.21.15.84:6818/2938436826 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb46c073f00 con 0x7fb46c071390 2024-03-19T02:06:18.087 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.085+0000 7fb48fdd9640 1 -- 172.21.15.55:0/3137625871 --> [v2:172.21.15.84:6818/2938436826,v1:172.21.15.84:6819/2938436826] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb488068a40 con 0x7fb46c071390 2024-03-19T02:06:18.088 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.085+0000 7fb47e7fc640 1 -- 172.21.15.55:0/3137625871 <== osd.4 v2:172.21.15.84:6818/2938436826 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb488068a40 con 0x7fb46c071390 2024-03-19T02:06:18.088 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-19T02:06:18.088 INFO:teuthology.orchestra.run.smithi055.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T02:06:18.088 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-19T02:06:18.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.089+0000 7fb48fdd9640 1 -- 172.21.15.55:0/3137625871 >> [v2:172.21.15.84:6818/2938436826,v1:172.21.15.84:6819/2938436826] conn(0x7fb46c071390 msgr2=0x7fb46c073830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:18.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.089+0000 7fb48fdd9640 1 --2- 172.21.15.55:0/3137625871 >> [v2:172.21.15.84:6818/2938436826,v1:172.21.15.84:6819/2938436826] conn(0x7fb46c071390 0x7fb46c073830 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:18.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.089+0000 7fb48fdd9640 1 -- 172.21.15.55:0/3137625871 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb46c0676e0 msgr2=0x7fb46c069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:18.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.089+0000 7fb48fdd9640 1 --2- 172.21.15.55:0/3137625871 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fb46c0676e0 0x7fb46c069ba0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb484007fd0 tx=0x7fb484009040 comp rx=0 tx=0).stop 2024-03-19T02:06:18.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.089+0000 7fb48fdd9640 1 -- 172.21.15.55:0/3137625871 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb4880c73c0 msgr2=0x7fb48806c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:18.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.089+0000 7fb48fdd9640 1 --2- 172.21.15.55:0/3137625871 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb4880c73c0 0x7fb48806c290 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fb47800c450 tx=0x7fb47800c910 comp rx=0 tx=0).stop 2024-03-19T02:06:18.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.089+0000 7fb48e376640 1 -- 172.21.15.55:0/3137625871 reap_dead start 2024-03-19T02:06:18.089 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.089+0000 7fb48fdd9640 1 -- 172.21.15.55:0/3137625871 shutdown_connections 2024-03-19T02:06:18.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.089+0000 7fb48fdd9640 1 -- 172.21.15.55:0/3137625871 >> 172.21.15.55:0/3137625871 conn(0x7fb4880cc9c0 msgr2=0x7fb4880c7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:18.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.089+0000 7fb48fdd9640 1 -- 172.21.15.55:0/3137625871 shutdown_connections 2024-03-19T02:06:18.090 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.089+0000 7fb48fdd9640 1 -- 172.21.15.55:0/3137625871 wait complete. 2024-03-19T02:06:18.103 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-19T02:06:18.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.225+0000 7f643cd08640 1 Processor -- start 2024-03-19T02:06:18.224 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.225+0000 7f643cd08640 1 -- start start 2024-03-19T02:06:18.225 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.225+0000 7f643cd08640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f64380c3320 0x7f643815b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:18.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.225+0000 7f643cd08640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6438151330 con 0x7f64380ca350 2024-03-19T02:06:18.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.225+0000 7f643cd08640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64380c3840 con 0x7f6438150f50 2024-03-19T02:06:18.226 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.225+0000 7f643cd08640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f64380c3e00 con 0x7f64380c3320 2024-03-19T02:06:18.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.261+0000 7f6436d9d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f64380c3320 0x7f643815b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:18.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.265+0000 7f6436d9d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f64380c3320 0x7f643815b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57336/0 (socket says 172.21.15.55:57336) 2024-03-19T02:06:18.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.265+0000 7f6436d9d640 1 -- 172.21.15.55:0/4147786589 learned_addr learned my addr 172.21.15.55:0/4147786589 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:18.267 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.265+0000 7f6436d9d640 1 -- 172.21.15.55:0/4147786589 >> v1:172.21.15.84:6789/0 conn(0x7f6438150f50 legacy=0x7f6438157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:18.267 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.265+0000 7f6436d9d640 1 -- 172.21.15.55:0/4147786589 >> v1:172.21.15.55:6789/0 conn(0x7f64380ca350 legacy=0x7f64380ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:18.267 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.265+0000 7f6436d9d640 1 -- 172.21.15.55:0/4147786589 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64380c3b20 con 0x7f64380c3320 2024-03-19T02:06:18.271 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.269+0000 7f6436d9d640 1 --2- 172.21.15.55:0/4147786589 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64380c3320 0x7f643815b0a0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f6430009e50 tx=0x7f643001c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df29d73aeb3edd74 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:18.276 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.277+0000 7f643559a640 1 -- 172.21.15.55:0/4147786589 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64300a5660 con 0x7f64380c3320 2024-03-19T02:06:18.276 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.277+0000 7f643559a640 1 -- 172.21.15.55:0/4147786589 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64300a5800 con 0x7f64380c3320 2024-03-19T02:06:18.276 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.277+0000 7f643559a640 1 -- 172.21.15.55:0/4147786589 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64300a5bd0 con 0x7f64380c3320 2024-03-19T02:06:18.277 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.277+0000 7f643cd08640 1 -- 172.21.15.55:0/4147786589 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64380c3320 msgr2=0x7f643815b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:18.278 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.277+0000 7f643cd08640 1 --2- 172.21.15.55:0/4147786589 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64380c3320 0x7f643815b0a0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f6430009e50 tx=0x7f643001c9b0 comp rx=0 tx=0).stop 2024-03-19T02:06:18.278 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.277+0000 7f643cd08640 1 -- 172.21.15.55:0/4147786589 shutdown_connections 2024-03-19T02:06:18.278 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.277+0000 7f643cd08640 1 --2- 172.21.15.55:0/4147786589 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64380c3320 0x7f643815b0a0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:18.278 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.277+0000 7f643cd08640 1 -- 172.21.15.55:0/4147786589 >> 172.21.15.55:0/4147786589 conn(0x7f643806e670 msgr2=0x7f643806ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:18.278 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.277+0000 7f643cd08640 1 -- 172.21.15.55:0/4147786589 shutdown_connections 2024-03-19T02:06:18.278 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.277+0000 7f643cd08640 1 -- 172.21.15.55:0/4147786589 wait complete. 2024-03-19T02:06:18.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.277+0000 7f643cd08640 1 Processor -- start 2024-03-19T02:06:18.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.277+0000 7f643cd08640 1 -- start start 2024-03-19T02:06:18.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.281+0000 7f643cd08640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64380c3320 0x7f643814d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:18.281 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.281+0000 7f643cd08640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64380ca350 0x7f6438159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:18.282 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.281+0000 7f643cd08640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6438150f50 0x7f64380c47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:18.282 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.281+0000 7f643cd08640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f64380cd7f0 con 0x7f64380c3320 2024-03-19T02:06:18.282 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.281+0000 7f643cd08640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f64380cd670 con 0x7f64380ca350 2024-03-19T02:06:18.282 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.281+0000 7f643cd08640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f643806bab0 con 0x7f6438150f50 2024-03-19T02:06:18.283 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.281+0000 7f643659c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64380c3320 0x7f643814d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:18.283 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.281+0000 7f643659c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64380c3320 0x7f643814d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:57344/0 (socket says 172.21.15.55:57344) 2024-03-19T02:06:18.283 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.281+0000 7f643659c640 1 -- 172.21.15.55:0/2314909317 learned_addr learned my addr 172.21.15.55:0/2314909317 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:18.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.285+0000 7f6436d9d640 1 --2- 172.21.15.55:0/2314909317 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6438150f50 0x7f64380c47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:18.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.285+0000 7f643659c640 1 -- 172.21.15.55:0/2314909317 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6438150f50 msgr2=0x7f64380c47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:18.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.285+0000 7f643659c640 1 --2- 172.21.15.55:0/2314909317 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6438150f50 0x7f64380c47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:18.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.285+0000 7f643659c640 1 -- 172.21.15.55:0/2314909317 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64380ca350 msgr2=0x7f6438159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:18.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.285+0000 7f643659c640 1 --2- 172.21.15.55:0/2314909317 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64380ca350 0x7f6438159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:18.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.285+0000 7f643659c640 1 -- 172.21.15.55:0/2314909317 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64381f7ec0 con 0x7f64380c3320 2024-03-19T02:06:18.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.289+0000 7f643659c640 1 --2- 172.21.15.55:0/2314909317 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64380c3320 0x7f643814d410 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f642800b410 tx=0x7f642800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:18.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.297+0000 7f6426ffd640 1 -- 172.21.15.55:0/2314909317 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6428015020 con 0x7f64380c3320 2024-03-19T02:06:18.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.297+0000 7f6426ffd640 1 -- 172.21.15.55:0/2314909317 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6428011830 con 0x7f64380c3320 2024-03-19T02:06:18.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.297+0000 7f643cd08640 1 -- 172.21.15.55:0/2314909317 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64381f8180 con 0x7f64380c3320 2024-03-19T02:06:18.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.297+0000 7f6426ffd640 1 -- 172.21.15.55:0/2314909317 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6428011c20 con 0x7f64380c3320 2024-03-19T02:06:18.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.297+0000 7f643cd08640 1 -- 172.21.15.55:0/2314909317 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64381f8740 con 0x7f64380c3320 2024-03-19T02:06:18.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.297+0000 7f643cd08640 1 -- 172.21.15.55:0/2314909317 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6438068a40 con 0x7f64380c3320 2024-03-19T02:06:18.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.605+0000 7f6426ffd640 1 -- 172.21.15.55:0/2314909317 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f6428018020 con 0x7f64380c3320 2024-03-19T02:06:18.608 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.609+0000 7f6426ffd640 1 --2- 172.21.15.55:0/2314909317 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f64180676e0 0x7f6418069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:18.609 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.609+0000 7f6435d9b640 1 --2- 172.21.15.55:0/2314909317 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f64180676e0 0x7f6418069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:18.610 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.609+0000 7f6435d9b640 1 --2- 172.21.15.55:0/2314909317 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f64180676e0 0x7f6418069ba0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f642c007fd0 tx=0x7f642c0079a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:18.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.641+0000 7f6426ffd640 1 -- 172.21.15.55:0/2314909317 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f6428022e70 con 0x7f64380c3320 2024-03-19T02:06:18.644 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.645+0000 7f6426ffd640 1 -- 172.21.15.55:0/2314909317 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f642801f070 con 0x7f64380c3320 2024-03-19T02:06:18.826 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.825+0000 7f643cd08640 1 -- 172.21.15.55:0/2314909317 --> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f64380c7600 con 0x7f64180676e0 2024-03-19T02:06:18.827 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.825+0000 7f6426ffd640 1 -- 172.21.15.55:0/2314909317 <== mgr.4105 v2:172.21.15.84:6800/998533783 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f64380c7600 con 0x7f64180676e0 2024-03-19T02:06:18.827 INFO:teuthology.orchestra.run.smithi055.stderr:instructed osd(s) 4 to deep-scrub 2024-03-19T02:06:18.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.829+0000 7f643cd08640 1 -- 172.21.15.55:0/2314909317 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f64180676e0 msgr2=0x7f6418069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:18.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.829+0000 7f643cd08640 1 --2- 172.21.15.55:0/2314909317 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f64180676e0 0x7f6418069ba0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f642c007fd0 tx=0x7f642c0079a0 comp rx=0 tx=0).stop 2024-03-19T02:06:18.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.829+0000 7f643cd08640 1 -- 172.21.15.55:0/2314909317 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64380c3320 msgr2=0x7f643814d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:18.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.829+0000 7f643cd08640 1 --2- 172.21.15.55:0/2314909317 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64380c3320 0x7f643814d410 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f642800b410 tx=0x7f642800b8d0 comp rx=0 tx=0).stop 2024-03-19T02:06:18.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.829+0000 7f643cd08640 1 -- 172.21.15.55:0/2314909317 shutdown_connections 2024-03-19T02:06:18.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.829+0000 7f643cd08640 1 --2- 172.21.15.55:0/2314909317 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f64180676e0 0x7f6418069ba0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:18.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.829+0000 7f643cd08640 1 --2- 172.21.15.55:0/2314909317 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f6438150f50 0x7f64380c47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:18.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.829+0000 7f643cd08640 1 --2- 172.21.15.55:0/2314909317 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64380ca350 0x7f6438159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:18.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.829+0000 7f643cd08640 1 --2- 172.21.15.55:0/2314909317 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f64380c3320 0x7f643814d410 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:18.829 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.829+0000 7f643cd08640 1 -- 172.21.15.55:0/2314909317 >> 172.21.15.55:0/2314909317 conn(0x7f643806e670 msgr2=0x7f64380c8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:18.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.829+0000 7f643cd08640 1 -- 172.21.15.55:0/2314909317 shutdown_connections 2024-03-19T02:06:18.830 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.829+0000 7f643cd08640 1 -- 172.21.15.55:0/2314909317 wait complete. 2024-03-19T02:06:18.846 INFO:tasks.ceph:Scrubbing osd.5 2024-03-19T02:06:18.846 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-19T02:06:18.983 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.981+0000 7f226bc47640 1 Processor -- start 2024-03-19T02:06:18.983 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.981+0000 7f226bc47640 1 -- start start 2024-03-19T02:06:18.984 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.985+0000 7f226bc47640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2264152120 0x7f22640c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:18.984 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.985+0000 7f226bc47640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f2264070780 0x7f22641586e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:18.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.985+0000 7f226bc47640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f226406fff0 con 0x7f22640c9610 2024-03-19T02:06:18.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.985+0000 7f226bc47640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f2264152500 con 0x7f2264152120 2024-03-19T02:06:18.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.985+0000 7f226bc47640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f2264070590 con 0x7f2264070780 2024-03-19T02:06:18.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.985+0000 7f22699e3640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2264152120 0x7f22640c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:18.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.985+0000 7f226a1e4640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f2264070780 0x7f22641586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:18.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.985+0000 7f22699e3640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f2264152120 0x7f22640c90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60774/0 (socket says 172.21.15.55:60774) 2024-03-19T02:06:18.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.985+0000 7f22699e3640 1 -- 172.21.15.55:0/279233335 learned_addr learned my addr 172.21.15.55:0/279233335 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:18.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.989+0000 7f22699e3640 1 -- 172.21.15.55:0/279233335 >> v2:172.21.15.84:3300/0 conn(0x7f2264070780 msgr2=0x7f22641586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:18.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.989+0000 7f22699e3640 1 --2- 172.21.15.55:0/279233335 >> v2:172.21.15.84:3300/0 conn(0x7f2264070780 0x7f22641586e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:18.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.989+0000 7f22699e3640 1 -- 172.21.15.55:0/279233335 >> v1:172.21.15.55:6789/0 conn(0x7f22640c9610 legacy=0x7f2264157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:18.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.989+0000 7f22699e3640 1 -- 172.21.15.55:0/279233335 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22640c39c0 con 0x7f2264152120 2024-03-19T02:06:18.993 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.993+0000 7f22699e3640 1 --2- 172.21.15.55:0/279233335 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2264152120 0x7f22640c90d0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f22640cb940 tx=0x7f226001c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ab5f458b4a33e50 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:18.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.997+0000 7f22689e1640 1 -- 172.21.15.55:0/279233335 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22600b0070 con 0x7f2264152120 2024-03-19T02:06:18.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.997+0000 7f22689e1640 1 -- 172.21.15.55:0/279233335 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22600ab600 con 0x7f2264152120 2024-03-19T02:06:18.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.997+0000 7f22689e1640 1 -- 172.21.15.55:0/279233335 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22600ab9d0 con 0x7f2264152120 2024-03-19T02:06:18.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.997+0000 7f226bc47640 1 -- 172.21.15.55:0/279233335 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2264152120 msgr2=0x7f22640c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:18.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.997+0000 7f226bc47640 1 --2- 172.21.15.55:0/279233335 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2264152120 0x7f22640c90d0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f22640cb940 tx=0x7f226001c4e0 comp rx=0 tx=0).stop 2024-03-19T02:06:18.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.997+0000 7f226bc47640 1 -- 172.21.15.55:0/279233335 shutdown_connections 2024-03-19T02:06:18.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.997+0000 7f226bc47640 1 --2- 172.21.15.55:0/279233335 >> v2:172.21.15.84:3300/0 conn(0x7f2264070780 0x7f22641586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:18.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.997+0000 7f226bc47640 1 --2- 172.21.15.55:0/279233335 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2264152120 0x7f22640c90d0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:18.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.997+0000 7f226bc47640 1 -- 172.21.15.55:0/279233335 >> 172.21.15.55:0/279233335 conn(0x7f2264067600 msgr2=0x7f2264071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:18.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.997+0000 7f226bc47640 1 -- 172.21.15.55:0/279233335 shutdown_connections 2024-03-19T02:06:18.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:18.997+0000 7f226bc47640 1 -- 172.21.15.55:0/279233335 wait complete. 2024-03-19T02:06:19.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.001+0000 7f226bc47640 1 Processor -- start 2024-03-19T02:06:19.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.001+0000 7f226bc47640 1 -- start start 2024-03-19T02:06:19.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.001+0000 7f226bc47640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2264070780 0x7f22640c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:19.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.001+0000 7f226bc47640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f22640c9610 0x7f22641f1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:19.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.001+0000 7f226bc47640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2264152120 0x7f22640b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:19.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.001+0000 7f226bc47640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f22640c86d0 con 0x7f2264070780 2024-03-19T02:06:19.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.001+0000 7f226bc47640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f22640c31c0 con 0x7f22640c9610 2024-03-19T02:06:19.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.001+0000 7f226bc47640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f2264165170 con 0x7f2264152120 2024-03-19T02:06:19.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.001+0000 7f22699e3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2264070780 0x7f22640c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:19.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.001+0000 7f22699e3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2264070780 0x7f22640c6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60784/0 (socket says 172.21.15.55:60784) 2024-03-19T02:06:19.003 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.001+0000 7f22699e3640 1 -- 172.21.15.55:0/3230278034 learned_addr learned my addr 172.21.15.55:0/3230278034 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:19.003 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.001+0000 7f226a1e4640 1 --2- 172.21.15.55:0/3230278034 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2264152120 0x7f22640b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:19.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.005+0000 7f22699e3640 1 -- 172.21.15.55:0/3230278034 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2264152120 msgr2=0x7f22640b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:19.005 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.005+0000 7f22699e3640 1 --2- 172.21.15.55:0/3230278034 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2264152120 0x7f22640b2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:19.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.005+0000 7f22699e3640 1 -- 172.21.15.55:0/3230278034 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f22640c9610 msgr2=0x7f22641f1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:19.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.005+0000 7f22699e3640 1 --2- 172.21.15.55:0/3230278034 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f22640c9610 0x7f22641f1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:19.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.005+0000 7f22699e3640 1 -- 172.21.15.55:0/3230278034 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22640b1170 con 0x7f2264070780 2024-03-19T02:06:19.008 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.005+0000 7f22699e3640 1 --2- 172.21.15.55:0/3230278034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2264070780 0x7f22640c6180 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f226001a110 tx=0x7f2260019280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:19.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.009+0000 7f225a7fc640 1 -- 172.21.15.55:0/3230278034 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22600b0050 con 0x7f2264070780 2024-03-19T02:06:19.011 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.009+0000 7f225a7fc640 1 -- 172.21.15.55:0/3230278034 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22600ba070 con 0x7f2264070780 2024-03-19T02:06:19.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.009+0000 7f226bc47640 1 -- 172.21.15.55:0/3230278034 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22640b1450 con 0x7f2264070780 2024-03-19T02:06:19.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.009+0000 7f225a7fc640 1 -- 172.21.15.55:0/3230278034 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22600b5680 con 0x7f2264070780 2024-03-19T02:06:19.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.009+0000 7f226bc47640 1 -- 172.21.15.55:0/3230278034 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22640b22c0 con 0x7f2264070780 2024-03-19T02:06:19.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.009+0000 7f226bc47640 1 -- 172.21.15.55:0/3230278034 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f22640b3000 con 0x7f2264070780 2024-03-19T02:06:19.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.033+0000 7f225a7fc640 1 -- 172.21.15.55:0/3230278034 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f22600b5820 con 0x7f2264070780 2024-03-19T02:06:19.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.033+0000 7f225a7fc640 1 --2- 172.21.15.55:0/3230278034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f22480676b0 0x7f2248069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:19.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.033+0000 7f22691e2640 1 --2- 172.21.15.55:0/3230278034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f22480676b0 0x7f2248069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:19.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.033+0000 7f22691e2640 1 --2- 172.21.15.55:0/3230278034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f22480676b0 0x7f2248069b70 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2254007fd0 tx=0x7f22540079a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:19.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.037+0000 7f225a7fc640 1 -- 172.21.15.55:0/3230278034 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f22600eeb40 con 0x7f2264070780 2024-03-19T02:06:19.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.037+0000 7f225a7fc640 1 --2- 172.21.15.55:0/3230278034 >> [v2:172.21.15.84:6802/460717295,v1:172.21.15.84:6803/460717295] conn(0x7f2248071360 0x7f2248073800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:19.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.037+0000 7f225a7fc640 1 -- 172.21.15.55:0/3230278034 --> [v2:172.21.15.84:6802/460717295,v1:172.21.15.84:6803/460717295] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f226000af30 con 0x7f2248071360 2024-03-19T02:06:19.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.037+0000 7f225a7fc640 1 -- 172.21.15.55:0/3230278034 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f22600ecd40 con 0x7f2264070780 2024-03-19T02:06:19.037 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.037+0000 7f226a1e4640 1 --2- 172.21.15.55:0/3230278034 >> [v2:172.21.15.84:6802/460717295,v1:172.21.15.84:6803/460717295] conn(0x7f2248071360 0x7f2248073800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:19.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.037+0000 7f226a1e4640 1 --2- 172.21.15.55:0/3230278034 >> [v2:172.21.15.84:6802/460717295,v1:172.21.15.84:6803/460717295] conn(0x7f2248071360 0x7f2248073800 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:19.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.037+0000 7f225a7fc640 1 -- 172.21.15.55:0/3230278034 <== osd.5 v2:172.21.15.84:6802/460717295 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f226000af30 con 0x7f2248071360 2024-03-19T02:06:19.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.061+0000 7f226bc47640 1 -- 172.21.15.55:0/3230278034 --> [v2:172.21.15.84:6802/460717295,v1:172.21.15.84:6803/460717295] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f2264068a40 con 0x7f2248071360 2024-03-19T02:06:19.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.065+0000 7f225a7fc640 1 -- 172.21.15.55:0/3230278034 <== osd.5 v2:172.21.15.84:6802/460717295 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f2264068a40 con 0x7f2248071360 2024-03-19T02:06:19.064 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-19T02:06:19.065 INFO:teuthology.orchestra.run.smithi055.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T02:06:19.065 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-19T02:06:19.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.065+0000 7f226bc47640 1 -- 172.21.15.55:0/3230278034 >> [v2:172.21.15.84:6802/460717295,v1:172.21.15.84:6803/460717295] conn(0x7f2248071360 msgr2=0x7f2248073800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:19.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.065+0000 7f226bc47640 1 --2- 172.21.15.55:0/3230278034 >> [v2:172.21.15.84:6802/460717295,v1:172.21.15.84:6803/460717295] conn(0x7f2248071360 0x7f2248073800 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:19.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.065+0000 7f226bc47640 1 -- 172.21.15.55:0/3230278034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f22480676b0 msgr2=0x7f2248069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:19.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.065+0000 7f226bc47640 1 --2- 172.21.15.55:0/3230278034 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f22480676b0 0x7f2248069b70 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2254007fd0 tx=0x7f22540079a0 comp rx=0 tx=0).stop 2024-03-19T02:06:19.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.065+0000 7f226bc47640 1 -- 172.21.15.55:0/3230278034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2264070780 msgr2=0x7f22640c6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:19.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.065+0000 7f226bc47640 1 --2- 172.21.15.55:0/3230278034 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2264070780 0x7f22640c6180 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f226001a110 tx=0x7f2260019280 comp rx=0 tx=0).stop 2024-03-19T02:06:19.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.065+0000 7f226a1e4640 1 -- 172.21.15.55:0/3230278034 reap_dead start 2024-03-19T02:06:19.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.065+0000 7f226bc47640 1 -- 172.21.15.55:0/3230278034 shutdown_connections 2024-03-19T02:06:19.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.065+0000 7f226bc47640 1 -- 172.21.15.55:0/3230278034 >> 172.21.15.55:0/3230278034 conn(0x7f2264067600 msgr2=0x7f226414e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:19.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.065+0000 7f226bc47640 1 -- 172.21.15.55:0/3230278034 shutdown_connections 2024-03-19T02:06:19.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.065+0000 7f226bc47640 1 -- 172.21.15.55:0/3230278034 wait complete. 2024-03-19T02:06:19.082 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-19T02:06:19.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.193+0000 7f0b522b2640 1 Processor -- start 2024-03-19T02:06:19.195 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.193+0000 7f0b522b2640 1 -- start start 2024-03-19T02:06:19.196 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.197+0000 7f0b522b2640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f0b4c1510f0 0x7f0b4c0c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:19.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.197+0000 7f0b522b2640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0b4c0c73c0 0x7f0b4c0c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:19.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.197+0000 7f0b522b2640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b4c0c3f60 con 0x7f0b4c15a410 2024-03-19T02:06:19.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.197+0000 7f0b522b2640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f0b4c070fb0 con 0x7f0b4c0c73c0 2024-03-19T02:06:19.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.197+0000 7f0b522b2640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f0b4c1514d0 con 0x7f0b4c1510f0 2024-03-19T02:06:19.197 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.197+0000 7f0b4bfff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f0b4c1510f0 0x7f0b4c0c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:19.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.197+0000 7f0b5084f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f0b4c0c73c0 0x7f0b4c0c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:19.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.197+0000 7f0b4bfff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f0b4c1510f0 0x7f0b4c0c61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:58738/0 (socket says 172.21.15.55:58738) 2024-03-19T02:06:19.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.197+0000 7f0b4bfff640 1 -- 172.21.15.55:0/3086813788 learned_addr learned my addr 172.21.15.55:0/3086813788 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:19.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.201+0000 7f0b4bfff640 1 -- 172.21.15.55:0/3086813788 >> v2:172.21.15.55:3300/0 conn(0x7f0b4c0c73c0 msgr2=0x7f0b4c0c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:19.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.201+0000 7f0b4bfff640 1 --2- 172.21.15.55:0/3086813788 >> v2:172.21.15.55:3300/0 conn(0x7f0b4c0c73c0 0x7f0b4c0c77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:19.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.201+0000 7f0b4bfff640 1 -- 172.21.15.55:0/3086813788 >> v1:172.21.15.55:6789/0 conn(0x7f0b4c15a410 legacy=0x7f0b4c0708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:19.202 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.201+0000 7f0b4bfff640 1 -- 172.21.15.55:0/3086813788 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4c0c6870 con 0x7f0b4c1510f0 2024-03-19T02:06:19.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.205+0000 7f0b4bfff640 1 --2- 172.21.15.55:0/3086813788 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0b4c1510f0 0x7f0b4c0c61c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c15b510 tx=0x7f0b3c01ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94af2dc19f0fa27d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:19.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.217+0000 7f0b4affd640 1 -- 172.21.15.55:0/3086813788 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c0b5070 con 0x7f0b4c1510f0 2024-03-19T02:06:19.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.217+0000 7f0b4affd640 1 -- 172.21.15.55:0/3086813788 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b3c0a89d0 con 0x7f0b4c1510f0 2024-03-19T02:06:19.216 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.217+0000 7f0b4affd640 1 -- 172.21.15.55:0/3086813788 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c0a8da0 con 0x7f0b4c1510f0 2024-03-19T02:06:19.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.217+0000 7f0b522b2640 1 -- 172.21.15.55:0/3086813788 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0b4c1510f0 msgr2=0x7f0b4c0c61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:19.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.217+0000 7f0b522b2640 1 --2- 172.21.15.55:0/3086813788 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0b4c1510f0 0x7f0b4c0c61c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c15b510 tx=0x7f0b3c01ecd0 comp rx=0 tx=0).stop 2024-03-19T02:06:19.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.217+0000 7f0b522b2640 1 -- 172.21.15.55:0/3086813788 shutdown_connections 2024-03-19T02:06:19.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.217+0000 7f0b522b2640 1 --2- 172.21.15.55:0/3086813788 >> v2:172.21.15.55:3300/0 conn(0x7f0b4c0c73c0 0x7f0b4c0c77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:19.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.217+0000 7f0b522b2640 1 --2- 172.21.15.55:0/3086813788 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0b4c1510f0 0x7f0b4c0c61c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:19.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.217+0000 7f0b522b2640 1 -- 172.21.15.55:0/3086813788 >> 172.21.15.55:0/3086813788 conn(0x7f0b4c0cc9c0 msgr2=0x7f0b4c0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:19.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.217+0000 7f0b522b2640 1 -- 172.21.15.55:0/3086813788 shutdown_connections 2024-03-19T02:06:19.218 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.217+0000 7f0b522b2640 1 -- 172.21.15.55:0/3086813788 wait complete. 2024-03-19T02:06:19.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.217+0000 7f0b522b2640 1 Processor -- start 2024-03-19T02:06:19.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.217+0000 7f0b522b2640 1 -- start start 2024-03-19T02:06:19.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.217+0000 7f0b522b2640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0b4c0c73c0 0x7f0b4c072cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:19.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.217+0000 7f0b522b2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b4c1510f0 0x7f0b4c147740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:19.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.217+0000 7f0b522b2640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0b4c15a410 0x7f0b4c158c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:19.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.217+0000 7f0b522b2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0b4c0c44e0 con 0x7f0b4c1510f0 2024-03-19T02:06:19.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.217+0000 7f0b522b2640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f0b4c0c4200 con 0x7f0b4c0c73c0 2024-03-19T02:06:19.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.217+0000 7f0b522b2640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f0b4c0c6e30 con 0x7f0b4c15a410 2024-03-19T02:06:19.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.221+0000 7f0b4b7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b4c1510f0 0x7f0b4c147740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:19.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.221+0000 7f0b5084f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0b4c15a410 0x7f0b4c158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:19.221 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.221+0000 7f0b5084f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0b4c15a410 0x7f0b4c158c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3301/0 says I am v2:172.21.15.55:48732/0 (socket says 172.21.15.55:48732) 2024-03-19T02:06:19.222 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.221+0000 7f0b5084f640 1 -- 172.21.15.55:0/556920855 learned_addr learned my addr 172.21.15.55:0/556920855 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:19.223 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.221+0000 7f0b4bfff640 1 --2- 172.21.15.55:0/556920855 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0b4c0c73c0 0x7f0b4c072cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:19.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.225+0000 7f0b5084f640 1 -- 172.21.15.55:0/556920855 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0b4c0c73c0 msgr2=0x7f0b4c072cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:19.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.225+0000 7f0b5084f640 1 --2- 172.21.15.55:0/556920855 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0b4c0c73c0 0x7f0b4c072cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:19.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.225+0000 7f0b5084f640 1 -- 172.21.15.55:0/556920855 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b4c1510f0 msgr2=0x7f0b4c147740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:19.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.225+0000 7f0b5084f640 1 --2- 172.21.15.55:0/556920855 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b4c1510f0 0x7f0b4c147740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:19.228 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.225+0000 7f0b5084f640 1 -- 172.21.15.55:0/556920855 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4c15a240 con 0x7f0b4c15a410 2024-03-19T02:06:19.231 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.229+0000 7f0b5084f640 1 --2- 172.21.15.55:0/556920855 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0b4c15a410 0x7f0b4c158c40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0b44009d30 tx=0x7f0b4400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:19.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.249+0000 7f0b48ff9640 1 -- 172.21.15.55:0/556920855 <== mon.2 v2:172.21.15.84:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b44018020 con 0x7f0b4c15a410 2024-03-19T02:06:19.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.249+0000 7f0b48ff9640 1 -- 172.21.15.55:0/556920855 <== mon.2 v2:172.21.15.84:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b44014480 con 0x7f0b4c15a410 2024-03-19T02:06:19.251 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.249+0000 7f0b48ff9640 1 -- 172.21.15.55:0/556920855 <== mon.2 v2:172.21.15.84:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b44014870 con 0x7f0b4c15a410 2024-03-19T02:06:19.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.249+0000 7f0b522b2640 1 -- 172.21.15.55:0/556920855 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b4c1597e0 con 0x7f0b4c15a410 2024-03-19T02:06:19.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.249+0000 7f0b522b2640 1 -- 172.21.15.55:0/556920855 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b4c159ca0 con 0x7f0b4c15a410 2024-03-19T02:06:19.252 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.249+0000 7f0b522b2640 1 -- 172.21.15.55:0/556920855 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b4c068a40 con 0x7f0b4c15a410 2024-03-19T02:06:19.650 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.649+0000 7f0b48ff9640 1 -- 172.21.15.55:0/556920855 <== mon.2 v2:172.21.15.84:3301/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f0b4401b020 con 0x7f0b4c15a410 2024-03-19T02:06:19.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.649+0000 7f0b48ff9640 1 --2- 172.21.15.55:0/556920855 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0b380676b0 0x7f0b38069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:19.651 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.649+0000 7f0b4bfff640 1 --2- 172.21.15.55:0/556920855 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0b380676b0 0x7f0b38069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:19.652 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.649+0000 7f0b4bfff640 1 --2- 172.21.15.55:0/556920855 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0b380676b0 0x7f0b38069b70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c0b2000 tx=0x7f0b3c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:19.657 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.657+0000 7f0b48ff9640 1 -- 172.21.15.55:0/556920855 <== mon.2 v2:172.21.15.84:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f0b44055010 con 0x7f0b4c15a410 2024-03-19T02:06:19.678 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.677+0000 7f0b48ff9640 1 -- 172.21.15.55:0/556920855 <== mon.2 v2:172.21.15.84:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f0b44022070 con 0x7f0b4c15a410 2024-03-19T02:06:19.848 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.849+0000 7f0b522b2640 1 -- 172.21.15.55:0/556920855 --> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f0b4c14a870 con 0x7f0b380676b0 2024-03-19T02:06:19.849 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.849+0000 7f0b48ff9640 1 -- 172.21.15.55:0/556920855 <== mgr.4105 v2:172.21.15.84:6800/998533783 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f0b4c14a870 con 0x7f0b380676b0 2024-03-19T02:06:19.849 INFO:teuthology.orchestra.run.smithi055.stderr:instructed osd(s) 5 to deep-scrub 2024-03-19T02:06:19.851 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.849+0000 7f0b522b2640 1 -- 172.21.15.55:0/556920855 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0b380676b0 msgr2=0x7f0b38069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:19.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.849+0000 7f0b522b2640 1 --2- 172.21.15.55:0/556920855 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0b380676b0 0x7f0b38069b70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c0b2000 tx=0x7f0b3c009040 comp rx=0 tx=0).stop 2024-03-19T02:06:19.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.849+0000 7f0b522b2640 1 -- 172.21.15.55:0/556920855 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0b4c15a410 msgr2=0x7f0b4c158c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:19.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.849+0000 7f0b522b2640 1 --2- 172.21.15.55:0/556920855 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0b4c15a410 0x7f0b4c158c40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0b44009d30 tx=0x7f0b4400c490 comp rx=0 tx=0).stop 2024-03-19T02:06:19.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.849+0000 7f0b522b2640 1 -- 172.21.15.55:0/556920855 shutdown_connections 2024-03-19T02:06:19.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.849+0000 7f0b522b2640 1 --2- 172.21.15.55:0/556920855 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f0b380676b0 0x7f0b38069b70 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:19.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.849+0000 7f0b522b2640 1 --2- 172.21.15.55:0/556920855 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f0b4c15a410 0x7f0b4c158c40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:19.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.849+0000 7f0b522b2640 1 --2- 172.21.15.55:0/556920855 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b4c1510f0 0x7f0b4c147740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:19.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.849+0000 7f0b522b2640 1 --2- 172.21.15.55:0/556920855 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0b4c0c73c0 0x7f0b4c072cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:19.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.849+0000 7f0b522b2640 1 -- 172.21.15.55:0/556920855 >> 172.21.15.55:0/556920855 conn(0x7f0b4c0cc9c0 msgr2=0x7f0b4c0c9770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:19.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.849+0000 7f0b522b2640 1 -- 172.21.15.55:0/556920855 shutdown_connections 2024-03-19T02:06:19.852 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.849+0000 7f0b522b2640 1 -- 172.21.15.55:0/556920855 wait complete. 2024-03-19T02:06:19.868 INFO:tasks.ceph:Scrubbing osd.6 2024-03-19T02:06:19.869 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-19T02:06:19.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.985+0000 7feddf30a640 1 Processor -- start 2024-03-19T02:06:19.985 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.985+0000 7feddf30a640 1 -- start start 2024-03-19T02:06:19.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.985+0000 7feddf30a640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fedd80ca350 0x7fedd815b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:19.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.985+0000 7feddf30a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedd8151330 con 0x7fedd8150f50 2024-03-19T02:06:19.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.985+0000 7feddf30a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedd80c3840 con 0x7fedd80c3320 2024-03-19T02:06:19.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.985+0000 7feddf30a640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fedd80c3e00 con 0x7fedd80ca350 2024-03-19T02:06:19.986 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.985+0000 7feddc8a5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fedd80ca350 0x7fedd815b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:19.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.985+0000 7feddd8a7640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fedd80c3320 0x7fedd80ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:50248/0 (socket says 172.21.15.55:50248) 2024-03-19T02:06:19.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.985+0000 7feddc8a5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fedd80ca350 0x7fedd815b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60816/0 (socket says 172.21.15.55:60816) 2024-03-19T02:06:19.987 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.985+0000 7feddd8a7640 1 -- 172.21.15.55:0/2100533132 learned_addr learned my addr 172.21.15.55:0/2100533132 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:06:19.989 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.989+0000 7feddc8a5640 1 -- 172.21.15.55:0/2100533132 >> v1:172.21.15.84:6789/0 conn(0x7fedd80c3320 legacy=0x7fedd80ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:19.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.989+0000 7feddc8a5640 1 -- 172.21.15.55:0/2100533132 >> v1:172.21.15.55:6789/0 conn(0x7fedd8150f50 legacy=0x7fedd8157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:19.990 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.989+0000 7feddc8a5640 1 -- 172.21.15.55:0/2100533132 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd80c3b20 con 0x7fedd80ca350 2024-03-19T02:06:19.994 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.993+0000 7feddc8a5640 1 --2- 172.21.15.55:0/2100533132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedd80ca350 0x7fedd815b0a0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fedcc0097c0 tx=0x7fedcc01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92869b3dbc5bfbc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:19.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.997+0000 7fedd7fff640 1 -- 172.21.15.55:0/2100533132 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedcc0b0070 con 0x7fedd80ca350 2024-03-19T02:06:19.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.997+0000 7fedd7fff640 1 -- 172.21.15.55:0/2100533132 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedcc004230 con 0x7fedd80ca350 2024-03-19T02:06:19.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.997+0000 7fedd7fff640 1 -- 172.21.15.55:0/2100533132 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedcc003620 con 0x7fedd80ca350 2024-03-19T02:06:19.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.997+0000 7feddf30a640 1 -- 172.21.15.55:0/2100533132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedd80ca350 msgr2=0x7fedd815b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:19.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.997+0000 7feddf30a640 1 --2- 172.21.15.55:0/2100533132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedd80ca350 0x7fedd815b0a0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fedcc0097c0 tx=0x7fedcc01c460 comp rx=0 tx=0).stop 2024-03-19T02:06:19.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.997+0000 7feddf30a640 1 -- 172.21.15.55:0/2100533132 shutdown_connections 2024-03-19T02:06:19.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.997+0000 7feddf30a640 1 --2- 172.21.15.55:0/2100533132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedd80ca350 0x7fedd815b0a0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:19.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.997+0000 7feddf30a640 1 -- 172.21.15.55:0/2100533132 >> 172.21.15.55:0/2100533132 conn(0x7fedd806e670 msgr2=0x7fedd806ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:19.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.997+0000 7feddf30a640 1 -- 172.21.15.55:0/2100533132 shutdown_connections 2024-03-19T02:06:20.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:19.997+0000 7feddf30a640 1 -- 172.21.15.55:0/2100533132 wait complete. 2024-03-19T02:06:20.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.001+0000 7feddf30a640 1 Processor -- start 2024-03-19T02:06:20.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.001+0000 7feddf30a640 1 -- start start 2024-03-19T02:06:20.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.001+0000 7feddf30a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedd80c3320 0x7fedd8145860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:20.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.001+0000 7feddf30a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fedd80ca350 0x7fedd814e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:20.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.001+0000 7feddf30a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedd8150f50 0x7fedd8142b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:20.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.001+0000 7feddf30a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fedd80cd7f0 con 0x7fedd80c3320 2024-03-19T02:06:20.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.001+0000 7feddf30a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fedd80cd670 con 0x7fedd8150f50 2024-03-19T02:06:20.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.001+0000 7feddf30a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fedd8165460 con 0x7fedd80ca350 2024-03-19T02:06:20.002 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.001+0000 7feddd0a6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedd80c3320 0x7fedd8145860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:20.003 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.001+0000 7feddc8a5640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fedd80ca350 0x7fedd814e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:20.003 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.001+0000 7feddd8a7640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedd8150f50 0x7fedd8142b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:20.003 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.001+0000 7feddd0a6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedd80c3320 0x7fedd8145860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60824/0 (socket says 172.21.15.55:60824) 2024-03-19T02:06:20.003 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.001+0000 7feddd0a6640 1 -- 172.21.15.55:0/1544870305 learned_addr learned my addr 172.21.15.55:0/1544870305 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:20.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.005+0000 7feddd0a6640 1 -- 172.21.15.55:0/1544870305 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fedd80ca350 msgr2=0x7fedd814e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:20.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.005+0000 7feddd0a6640 1 --2- 172.21.15.55:0/1544870305 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fedd80ca350 0x7fedd814e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:20.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.005+0000 7feddd0a6640 1 -- 172.21.15.55:0/1544870305 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedd8150f50 msgr2=0x7fedd8142b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:20.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.005+0000 7feddd0a6640 1 --2- 172.21.15.55:0/1544870305 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedd8150f50 0x7fedd8142b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:20.006 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.005+0000 7feddd0a6640 1 -- 172.21.15.55:0/1544870305 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd81543b0 con 0x7fedd80c3320 2024-03-19T02:06:20.008 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.005+0000 7feddd0a6640 1 --2- 172.21.15.55:0/1544870305 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedd80c3320 0x7fedd8145860 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fedc800c7a0 tx=0x7fedc800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:20.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.009+0000 7fedd5ffb640 1 -- 172.21.15.55:0/1544870305 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc8017070 con 0x7fedd80c3320 2024-03-19T02:06:20.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.009+0000 7fedd5ffb640 1 -- 172.21.15.55:0/1544870305 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedc8003db0 con 0x7fedd80c3320 2024-03-19T02:06:20.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.009+0000 7feddf30a640 1 -- 172.21.15.55:0/1544870305 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedd8144c20 con 0x7fedd80c3320 2024-03-19T02:06:20.012 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.009+0000 7fedd5ffb640 1 -- 172.21.15.55:0/1544870305 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc80052b0 con 0x7fedd80c3320 2024-03-19T02:06:20.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.009+0000 7feddf30a640 1 -- 172.21.15.55:0/1544870305 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedd81525d0 con 0x7fedd80c3320 2024-03-19T02:06:20.013 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.013+0000 7feddf30a640 1 -- 172.21.15.55:0/1544870305 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fedd8068a40 con 0x7fedd80c3320 2024-03-19T02:06:20.032 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.033+0000 7fedd5ffb640 1 -- 172.21.15.55:0/1544870305 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fedc801a070 con 0x7fedd80c3320 2024-03-19T02:06:20.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.033+0000 7fedd5ffb640 1 --2- 172.21.15.55:0/1544870305 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fedc4067660 0x7fedc4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:20.033 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.033+0000 7feddc8a5640 1 --2- 172.21.15.55:0/1544870305 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fedc4067660 0x7fedc4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:20.034 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.033+0000 7feddc8a5640 1 --2- 172.21.15.55:0/1544870305 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fedc4067660 0x7fedc4069b20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fedcc0ad000 tx=0x7fedcc0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:20.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.037+0000 7fedd5ffb640 1 -- 172.21.15.55:0/1544870305 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fedc8053d10 con 0x7fedd80c3320 2024-03-19T02:06:20.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.037+0000 7fedd5ffb640 1 --2- 172.21.15.55:0/1544870305 >> [v2:172.21.15.84:6826/416123135,v1:172.21.15.84:6827/416123135] conn(0x7fedc4071310 0x7fedc40737b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:20.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.037+0000 7fedd5ffb640 1 -- 172.21.15.55:0/1544870305 --> [v2:172.21.15.84:6826/416123135,v1:172.21.15.84:6827/416123135] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fedc800ae50 con 0x7fedc4071310 2024-03-19T02:06:20.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.037+0000 7fedd5ffb640 1 -- 172.21.15.55:0/1544870305 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fedc8051f10 con 0x7fedd80c3320 2024-03-19T02:06:20.038 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.037+0000 7feddd8a7640 1 --2- 172.21.15.55:0/1544870305 >> [v2:172.21.15.84:6826/416123135,v1:172.21.15.84:6827/416123135] conn(0x7fedc4071310 0x7fedc40737b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:20.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.037+0000 7feddd8a7640 1 --2- 172.21.15.55:0/1544870305 >> [v2:172.21.15.84:6826/416123135,v1:172.21.15.84:6827/416123135] conn(0x7fedc4071310 0x7fedc40737b0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:20.039 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.037+0000 7fedd5ffb640 1 -- 172.21.15.55:0/1544870305 <== osd.6 v2:172.21.15.84:6826/416123135 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fedc800ae50 con 0x7fedc4071310 2024-03-19T02:06:20.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.053+0000 7feddf30a640 1 -- 172.21.15.55:0/1544870305 --> [v2:172.21.15.84:6826/416123135,v1:172.21.15.84:6827/416123135] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fedd80c7600 con 0x7fedc4071310 2024-03-19T02:06:20.054 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.053+0000 7fedd5ffb640 1 -- 172.21.15.55:0/1544870305 <== osd.6 v2:172.21.15.84:6826/416123135 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fedd80c7600 con 0x7fedc4071310 2024-03-19T02:06:20.055 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-19T02:06:20.055 INFO:teuthology.orchestra.run.smithi055.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T02:06:20.055 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-19T02:06:20.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.053+0000 7feddf30a640 1 -- 172.21.15.55:0/1544870305 >> [v2:172.21.15.84:6826/416123135,v1:172.21.15.84:6827/416123135] conn(0x7fedc4071310 msgr2=0x7fedc40737b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:20.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.053+0000 7feddf30a640 1 --2- 172.21.15.55:0/1544870305 >> [v2:172.21.15.84:6826/416123135,v1:172.21.15.84:6827/416123135] conn(0x7fedc4071310 0x7fedc40737b0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:20.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.053+0000 7feddf30a640 1 -- 172.21.15.55:0/1544870305 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fedc4067660 msgr2=0x7fedc4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:20.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.053+0000 7feddf30a640 1 --2- 172.21.15.55:0/1544870305 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fedc4067660 0x7fedc4069b20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fedcc0ad000 tx=0x7fedcc0a6000 comp rx=0 tx=0).stop 2024-03-19T02:06:20.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.053+0000 7feddf30a640 1 -- 172.21.15.55:0/1544870305 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedd80c3320 msgr2=0x7fedd8145860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:20.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.053+0000 7feddf30a640 1 --2- 172.21.15.55:0/1544870305 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedd80c3320 0x7fedd8145860 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fedc800c7a0 tx=0x7fedc800cc60 comp rx=0 tx=0).stop 2024-03-19T02:06:20.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.053+0000 7feddd8a7640 1 -- 172.21.15.55:0/1544870305 reap_dead start 2024-03-19T02:06:20.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.053+0000 7feddf30a640 1 -- 172.21.15.55:0/1544870305 shutdown_connections 2024-03-19T02:06:20.056 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.053+0000 7feddf30a640 1 -- 172.21.15.55:0/1544870305 >> 172.21.15.55:0/1544870305 conn(0x7fedd806e670 msgr2=0x7fedd814f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:20.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.053+0000 7feddf30a640 1 -- 172.21.15.55:0/1544870305 shutdown_connections 2024-03-19T02:06:20.057 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.053+0000 7feddf30a640 1 -- 172.21.15.55:0/1544870305 wait complete. 2024-03-19T02:06:20.075 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-19T02:06:20.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.185+0000 7f286f290640 1 Processor -- start 2024-03-19T02:06:20.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.185+0000 7f286f290640 1 -- start start 2024-03-19T02:06:20.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.189+0000 7f286f290640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f28680c52b0 0x7f2868151d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:20.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.189+0000 7f286f290640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28680724c0 con 0x7f28680c89f0 2024-03-19T02:06:20.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.189+0000 7f286f290640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f286813d290 con 0x7f2868158780 2024-03-19T02:06:20.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.189+0000 7f286f290640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f286813d570 con 0x7f28680c52b0 2024-03-19T02:06:20.189 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.189+0000 7f286d02c640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f28680c52b0 0x7f2868151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:20.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.189+0000 7f286d82d640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f2868158780 0x7f2868071810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:50254/0 (socket says 172.21.15.55:50254) 2024-03-19T02:06:20.190 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.189+0000 7f286d82d640 1 -- 172.21.15.55:0/718950591 learned_addr learned my addr 172.21.15.55:0/718950591 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:06:20.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.193+0000 7f285ffff640 1 -- 172.21.15.55:0/718950591 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1143674464 0 0) 0x7f286813d290 con 0x7f2868158780 2024-03-19T02:06:20.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.193+0000 7f285ffff640 1 -- 172.21.15.55:0/718950591 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2854003530 con 0x7f2868158780 2024-03-19T02:06:20.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.193+0000 7f286d02c640 1 -- 172.21.15.55:0/718950591 >> v1:172.21.15.84:6789/0 conn(0x7f2868158780 legacy=0x7f2868071810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:20.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.193+0000 7f286d02c640 1 -- 172.21.15.55:0/718950591 >> v1:172.21.15.55:6789/0 conn(0x7f28680c89f0 legacy=0x7f2868158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:20.194 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.193+0000 7f286d02c640 1 -- 172.21.15.55:0/718950591 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f286813d850 con 0x7f28680c52b0 2024-03-19T02:06:20.198 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.197+0000 7f286d02c640 1 --2- 172.21.15.55:0/718950591 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f28680c52b0 0x7f2868151d20 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f28640097c0 tx=0x7f286401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=523f9dded42e1a6d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:20.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.205+0000 7f285ffff640 1 -- 172.21.15.55:0/718950591 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28640b5070 con 0x7f28680c52b0 2024-03-19T02:06:20.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.205+0000 7f285ffff640 1 -- 172.21.15.55:0/718950591 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28640a8950 con 0x7f28680c52b0 2024-03-19T02:06:20.204 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.205+0000 7f285ffff640 1 -- 172.21.15.55:0/718950591 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28640a8d20 con 0x7f28680c52b0 2024-03-19T02:06:20.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.205+0000 7f286f290640 1 -- 172.21.15.55:0/718950591 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f28680c52b0 msgr2=0x7f2868151d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:20.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.205+0000 7f286f290640 1 --2- 172.21.15.55:0/718950591 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f28680c52b0 0x7f2868151d20 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f28640097c0 tx=0x7f286401ec50 comp rx=0 tx=0).stop 2024-03-19T02:06:20.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.205+0000 7f286f290640 1 -- 172.21.15.55:0/718950591 shutdown_connections 2024-03-19T02:06:20.205 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.205+0000 7f286f290640 1 --2- 172.21.15.55:0/718950591 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f28680c52b0 0x7f2868151d20 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:20.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.205+0000 7f286f290640 1 -- 172.21.15.55:0/718950591 >> 172.21.15.55:0/718950591 conn(0x7f286806f3b0 msgr2=0x7f286806f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:20.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.205+0000 7f286f290640 1 -- 172.21.15.55:0/718950591 shutdown_connections 2024-03-19T02:06:20.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.205+0000 7f286f290640 1 -- 172.21.15.55:0/718950591 wait complete. 2024-03-19T02:06:20.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.205+0000 7f286f290640 1 Processor -- start 2024-03-19T02:06:20.206 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.205+0000 7f286f290640 1 -- start start 2024-03-19T02:06:20.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.205+0000 7f286f290640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28680c52b0 0x7f286815a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:20.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.205+0000 7f286f290640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f28680c89f0 0x7f2868139e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:20.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.205+0000 7f286f290640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2868158780 0x7f286813a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:20.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.205+0000 7f286f290640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f28680c5690 con 0x7f28680c89f0 2024-03-19T02:06:20.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.205+0000 7f286f290640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f286813de10 con 0x7f28680c52b0 2024-03-19T02:06:20.207 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.205+0000 7f286f290640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f28680c5810 con 0x7f2868158780 2024-03-19T02:06:20.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.205+0000 7f286c82b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f28680c89f0 0x7f2868139e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:20.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.205+0000 7f286d02c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28680c52b0 0x7f286815a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:20.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.209+0000 7f286d02c640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28680c52b0 0x7f286815a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:58774/0 (socket says 172.21.15.55:58774) 2024-03-19T02:06:20.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.209+0000 7f286d02c640 1 -- 172.21.15.55:0/283457894 learned_addr learned my addr 172.21.15.55:0/283457894 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:20.208 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.209+0000 7f286d82d640 1 --2- 172.21.15.55:0/283457894 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2868158780 0x7f286813a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:20.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.209+0000 7f286c82b640 1 -- 172.21.15.55:0/283457894 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2868158780 msgr2=0x7f286813a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:20.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.209+0000 7f286c82b640 1 --2- 172.21.15.55:0/283457894 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2868158780 0x7f286813a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:20.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.209+0000 7f286c82b640 1 -- 172.21.15.55:0/283457894 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28680c52b0 msgr2=0x7f286815a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:20.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.209+0000 7f286c82b640 1 --2- 172.21.15.55:0/283457894 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28680c52b0 0x7f286815a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:20.212 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.209+0000 7f286c82b640 1 -- 172.21.15.55:0/283457894 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f286813aa40 con 0x7f28680c89f0 2024-03-19T02:06:20.214 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.213+0000 7f286c82b640 1 --2- 172.21.15.55:0/283457894 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f28680c89f0 0x7f2868139e40 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f285800c390 tx=0x7f285800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:20.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.217+0000 7f285dffb640 1 -- 172.21.15.55:0/283457894 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2858017070 con 0x7f28680c89f0 2024-03-19T02:06:20.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.217+0000 7f285dffb640 1 -- 172.21.15.55:0/283457894 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2858003600 con 0x7f28680c89f0 2024-03-19T02:06:20.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.217+0000 7f285dffb640 1 -- 172.21.15.55:0/283457894 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28580039f0 con 0x7f28680c89f0 2024-03-19T02:06:20.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.217+0000 7f286f290640 1 -- 172.21.15.55:0/283457894 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28681edf10 con 0x7f28680c89f0 2024-03-19T02:06:20.219 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.217+0000 7f286f290640 1 -- 172.21.15.55:0/283457894 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28681ee4d0 con 0x7f28680c89f0 2024-03-19T02:06:20.220 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.217+0000 7f284b7fe640 1 -- 172.21.15.55:0/283457894 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28440052a0 con 0x7f28680c89f0 2024-03-19T02:06:20.555 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.553+0000 7f285dffb640 1 -- 172.21.15.55:0/283457894 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f285801a070 con 0x7f28680c89f0 2024-03-19T02:06:20.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.553+0000 7f285dffb640 1 --2- 172.21.15.55:0/283457894 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f284c067690 0x7f284c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:20.556 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.557+0000 7f286d02c640 1 --2- 172.21.15.55:0/283457894 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f284c067690 0x7f284c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:20.557 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.557+0000 7f286d02c640 1 --2- 172.21.15.55:0/283457894 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f284c067690 0x7f284c069b50 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f286401c880 tx=0x7f2864008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:20.594 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.593+0000 7f285dffb640 1 -- 172.21.15.55:0/283457894 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f28580533a0 con 0x7f28680c89f0 2024-03-19T02:06:20.594 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.593+0000 7f285dffb640 1 -- 172.21.15.55:0/283457894 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2858055300 con 0x7f28680c89f0 2024-03-19T02:06:20.771 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.769+0000 7f284b7fe640 1 -- 172.21.15.55:0/283457894 --> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f2844000bd0 con 0x7f284c067690 2024-03-19T02:06:20.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.769+0000 7f285dffb640 1 -- 172.21.15.55:0/283457894 <== mgr.4105 v2:172.21.15.84:6800/998533783 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2844000bd0 con 0x7f284c067690 2024-03-19T02:06:20.772 INFO:teuthology.orchestra.run.smithi055.stderr:instructed osd(s) 6 to deep-scrub 2024-03-19T02:06:20.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.773+0000 7f284b7fe640 1 -- 172.21.15.55:0/283457894 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f284c067690 msgr2=0x7f284c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:20.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.773+0000 7f284b7fe640 1 --2- 172.21.15.55:0/283457894 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f284c067690 0x7f284c069b50 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f286401c880 tx=0x7f2864008040 comp rx=0 tx=0).stop 2024-03-19T02:06:20.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.773+0000 7f284b7fe640 1 -- 172.21.15.55:0/283457894 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f28680c89f0 msgr2=0x7f2868139e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:20.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.773+0000 7f284b7fe640 1 --2- 172.21.15.55:0/283457894 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f28680c89f0 0x7f2868139e40 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f285800c390 tx=0x7f285800c850 comp rx=0 tx=0).stop 2024-03-19T02:06:20.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.773+0000 7f284b7fe640 1 -- 172.21.15.55:0/283457894 shutdown_connections 2024-03-19T02:06:20.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.773+0000 7f284b7fe640 1 --2- 172.21.15.55:0/283457894 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f284c067690 0x7f284c069b50 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:20.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.773+0000 7f284b7fe640 1 --2- 172.21.15.55:0/283457894 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f2868158780 0x7f286813a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:20.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.773+0000 7f284b7fe640 1 --2- 172.21.15.55:0/283457894 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f28680c89f0 0x7f2868139e40 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:20.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.773+0000 7f284b7fe640 1 --2- 172.21.15.55:0/283457894 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28680c52b0 0x7f286815a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:20.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.773+0000 7f284b7fe640 1 -- 172.21.15.55:0/283457894 >> 172.21.15.55:0/283457894 conn(0x7f286806f3b0 msgr2=0x7f28681402d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:20.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.773+0000 7f284b7fe640 1 -- 172.21.15.55:0/283457894 shutdown_connections 2024-03-19T02:06:20.775 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.773+0000 7f284b7fe640 1 -- 172.21.15.55:0/283457894 wait complete. 2024-03-19T02:06:20.789 INFO:tasks.ceph:Scrubbing osd.7 2024-03-19T02:06:20.789 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-19T02:06:20.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.909+0000 7f3fa0d5f640 1 Processor -- start 2024-03-19T02:06:20.908 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.909+0000 7f3fa0d5f640 1 -- start start 2024-03-19T02:06:20.910 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.909+0000 7f3fa0d5f640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f3f9c1544f0 0x7f3f9c073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:20.910 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.909+0000 7f3fa0d5f640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f3f9c06b2a0 0x7f3f9c15e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:20.910 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.909+0000 7f3fa0d5f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f9c0c7fa0 con 0x7f3f9c15acb0 2024-03-19T02:06:20.910 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.909+0000 7f3fa0d5f640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f3f9c15b210 con 0x7f3f9c1544f0 2024-03-19T02:06:20.910 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.909+0000 7f3fa0d5f640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f3f9c0c8240 con 0x7f3f9c06b2a0 2024-03-19T02:06:20.910 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.909+0000 7f3f93fff640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f3f9c06b2a0 0x7f3f9c15e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:20.911 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.909+0000 7f3f9ad9d640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f3f9c15acb0 0x7f3f9c0cce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.55:50262/0 (socket says 172.21.15.55:50262) 2024-03-19T02:06:20.911 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.909+0000 7f3f9ad9d640 1 -- 172.21.15.55:0/1346561087 learned_addr learned my addr 172.21.15.55:0/1346561087 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:06:20.917 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.917+0000 7f3f99d9b640 1 -- 172.21.15.55:0/1346561087 <== mon.1 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4056963441 0 0) 0x7f3f9c0c7fa0 con 0x7f3f9c15acb0 2024-03-19T02:06:20.917 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.917+0000 7f3f99d9b640 1 -- 172.21.15.55:0/1346561087 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f88003530 con 0x7f3f9c15acb0 2024-03-19T02:06:20.922 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.921+0000 7f3f9a59c640 1 --2- 172.21.15.55:0/1346561087 >> v2:172.21.15.55:3300/0 conn(0x7f3f9c1544f0 0x7f3f9c073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:20.924 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.925+0000 7f3f93fff640 1 -- 172.21.15.55:0/1346561087 >> v2:172.21.15.55:3300/0 conn(0x7f3f9c1544f0 msgr2=0x7f3f9c073540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:20.925 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.925+0000 7f3f93fff640 1 --2- 172.21.15.55:0/1346561087 >> v2:172.21.15.55:3300/0 conn(0x7f3f9c1544f0 0x7f3f9c073540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:20.925 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.925+0000 7f3f93fff640 1 -- 172.21.15.55:0/1346561087 >> v1:172.21.15.84:6789/0 conn(0x7f3f9c15acb0 legacy=0x7f3f9c0cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:20.925 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.925+0000 7f3f93fff640 1 -- 172.21.15.55:0/1346561087 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f9c15d920 con 0x7f3f9c06b2a0 2024-03-19T02:06:20.932 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.933+0000 7f3f93fff640 1 --2- 172.21.15.55:0/1346561087 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3f9c06b2a0 0x7f3f9c15e120 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c0ca790 tx=0x7f3f8401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac7bd25d387323ff server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:20.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.937+0000 7f3f99d9b640 1 -- 172.21.15.55:0/1346561087 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f840b5070 con 0x7f3f9c06b2a0 2024-03-19T02:06:20.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.937+0000 7f3f99d9b640 1 -- 172.21.15.55:0/1346561087 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f840a8970 con 0x7f3f9c06b2a0 2024-03-19T02:06:20.940 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.937+0000 7f3f99d9b640 1 -- 172.21.15.55:0/1346561087 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f840a8d40 con 0x7f3f9c06b2a0 2024-03-19T02:06:20.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.941+0000 7f3fa0d5f640 1 -- 172.21.15.55:0/1346561087 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3f9c06b2a0 msgr2=0x7f3f9c15e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:20.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.941+0000 7f3fa0d5f640 1 --2- 172.21.15.55:0/1346561087 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3f9c06b2a0 0x7f3f9c15e120 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c0ca790 tx=0x7f3f8401ec70 comp rx=0 tx=0).stop 2024-03-19T02:06:20.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.941+0000 7f3fa0d5f640 1 -- 172.21.15.55:0/1346561087 shutdown_connections 2024-03-19T02:06:20.941 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.941+0000 7f3fa0d5f640 1 --2- 172.21.15.55:0/1346561087 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3f9c06b2a0 0x7f3f9c15e120 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:20.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.941+0000 7f3fa0d5f640 1 --2- 172.21.15.55:0/1346561087 >> v2:172.21.15.55:3300/0 conn(0x7f3f9c1544f0 0x7f3f9c073540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:20.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.941+0000 7f3fa0d5f640 1 -- 172.21.15.55:0/1346561087 >> 172.21.15.55:0/1346561087 conn(0x7f3f9c06e670 msgr2=0x7f3f9c06ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:20.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.941+0000 7f3fa0d5f640 1 -- 172.21.15.55:0/1346561087 shutdown_connections 2024-03-19T02:06:20.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.941+0000 7f3fa0d5f640 1 -- 172.21.15.55:0/1346561087 wait complete. 2024-03-19T02:06:20.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.941+0000 7f3fa0d5f640 1 Processor -- start 2024-03-19T02:06:20.942 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.941+0000 7f3fa0d5f640 1 -- start start 2024-03-19T02:06:20.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.941+0000 7f3fa0d5f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f9c06b2a0 0x7f3f9c1efe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:20.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.941+0000 7f3fa0d5f640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3f9c1544f0 0x7f3f9c1e6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:20.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.941+0000 7f3fa0d5f640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3f9c15acb0 0x7f3f9c1e3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:20.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.941+0000 7f3fa0d5f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3f9c1548d0 con 0x7f3f9c06b2a0 2024-03-19T02:06:20.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.941+0000 7f3fa0d5f640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3f9c15e660 con 0x7f3f9c1544f0 2024-03-19T02:06:20.943 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.941+0000 7f3fa0d5f640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f3f9c0cd540 con 0x7f3f9c15acb0 2024-03-19T02:06:20.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.941+0000 7f3f9a59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f9c06b2a0 0x7f3f9c1efe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:20.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.941+0000 7f3f9a59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f9c06b2a0 0x7f3f9c1efe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60868/0 (socket says 172.21.15.55:60868) 2024-03-19T02:06:20.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.941+0000 7f3f9a59c640 1 -- 172.21.15.55:0/3144554377 learned_addr learned my addr 172.21.15.55:0/3144554377 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:20.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.941+0000 7f3f93fff640 1 --2- 172.21.15.55:0/3144554377 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3f9c1544f0 0x7f3f9c1e6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:20.944 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.945+0000 7f3f9ad9d640 1 --2- 172.21.15.55:0/3144554377 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3f9c15acb0 0x7f3f9c1e3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:20.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.945+0000 7f3f9a59c640 1 -- 172.21.15.55:0/3144554377 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3f9c15acb0 msgr2=0x7f3f9c1e3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:20.947 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.945+0000 7f3f9a59c640 1 --2- 172.21.15.55:0/3144554377 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f3f9c15acb0 0x7f3f9c1e3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:20.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.945+0000 7f3f9a59c640 1 -- 172.21.15.55:0/3144554377 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3f9c1544f0 msgr2=0x7f3f9c1e6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:20.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.945+0000 7f3f9a59c640 1 --2- 172.21.15.55:0/3144554377 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3f9c1544f0 0x7f3f9c1e6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:20.948 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.945+0000 7f3f9a59c640 1 -- 172.21.15.55:0/3144554377 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f9c14e2f0 con 0x7f3f9c06b2a0 2024-03-19T02:06:20.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.949+0000 7f3f9a59c640 1 --2- 172.21.15.55:0/3144554377 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f9c06b2a0 0x7f3f9c1efe30 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c00e930 tx=0x7f3f8c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:20.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.953+0000 7f3f92ffd640 1 -- 172.21.15.55:0/3144554377 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c017070 con 0x7f3f9c06b2a0 2024-03-19T02:06:20.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.953+0000 7f3f92ffd640 1 -- 172.21.15.55:0/3144554377 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f8c00ca00 con 0x7f3f9c06b2a0 2024-03-19T02:06:20.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.953+0000 7f3f92ffd640 1 -- 172.21.15.55:0/3144554377 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c00cdb0 con 0x7f3f9c06b2a0 2024-03-19T02:06:20.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.953+0000 7f3fa0d5f640 1 -- 172.21.15.55:0/3144554377 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f9c1508a0 con 0x7f3f9c06b2a0 2024-03-19T02:06:20.955 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.953+0000 7f3fa0d5f640 1 -- 172.21.15.55:0/3144554377 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f9c150e40 con 0x7f3f9c06b2a0 2024-03-19T02:06:20.956 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.957+0000 7f3fa0d5f640 1 -- 172.21.15.55:0/3144554377 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3f9c068a40 con 0x7f3f9c06b2a0 2024-03-19T02:06:20.977 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.977+0000 7f3f92ffd640 1 -- 172.21.15.55:0/3144554377 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f3f8c01a070 con 0x7f3f9c06b2a0 2024-03-19T02:06:20.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.977+0000 7f3f92ffd640 1 --2- 172.21.15.55:0/3144554377 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3f7c067660 0x7f3f7c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:20.978 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.977+0000 7f3f93fff640 1 --2- 172.21.15.55:0/3144554377 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3f7c067660 0x7f3f7c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:20.979 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.977+0000 7f3f93fff640 1 --2- 172.21.15.55:0/3144554377 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3f7c067660 0x7f3f7c069b20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3f840b2000 tx=0x7f3f84008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:20.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.981+0000 7f3f92ffd640 1 -- 172.21.15.55:0/3144554377 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f3f8c0543e0 con 0x7f3f9c06b2a0 2024-03-19T02:06:20.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.981+0000 7f3f92ffd640 1 --2- 172.21.15.55:0/3144554377 >> [v2:172.21.15.84:6810/460083528,v1:172.21.15.84:6811/460083528] conn(0x7f3f7c071290 0x7f3f7c073730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:20.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.981+0000 7f3f92ffd640 1 -- 172.21.15.55:0/3144554377 --> [v2:172.21.15.84:6810/460083528,v1:172.21.15.84:6811/460083528] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3f7c073e00 con 0x7f3f7c071290 2024-03-19T02:06:20.982 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.981+0000 7f3f92ffd640 1 -- 172.21.15.55:0/3144554377 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3f8c0525e0 con 0x7f3f9c06b2a0 2024-03-19T02:06:20.983 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.981+0000 7f3f9ad9d640 1 --2- 172.21.15.55:0/3144554377 >> [v2:172.21.15.84:6810/460083528,v1:172.21.15.84:6811/460083528] conn(0x7f3f7c071290 0x7f3f7c073730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:20.983 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.981+0000 7f3f9ad9d640 1 --2- 172.21.15.55:0/3144554377 >> [v2:172.21.15.84:6810/460083528,v1:172.21.15.84:6811/460083528] conn(0x7f3f7c071290 0x7f3f7c073730 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:20.984 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.981+0000 7f3f92ffd640 1 -- 172.21.15.55:0/3144554377 <== osd.7 v2:172.21.15.84:6810/460083528 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3f7c073e00 con 0x7f3f7c071290 2024-03-19T02:06:20.998 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.997+0000 7f3fa0d5f640 1 -- 172.21.15.55:0/3144554377 --> [v2:172.21.15.84:6810/460083528,v1:172.21.15.84:6811/460083528] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3f9c1e5860 con 0x7f3f7c071290 2024-03-19T02:06:20.999 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.997+0000 7f3f92ffd640 1 -- 172.21.15.55:0/3144554377 <== osd.7 v2:172.21.15.84:6810/460083528 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f3f94001f80 con 0x7f3f7c071290 2024-03-19T02:06:20.999 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-19T02:06:20.999 INFO:teuthology.orchestra.run.smithi055.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-19T02:06:20.999 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-19T02:06:21.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.997+0000 7f3fa0d5f640 1 -- 172.21.15.55:0/3144554377 >> [v2:172.21.15.84:6810/460083528,v1:172.21.15.84:6811/460083528] conn(0x7f3f7c071290 msgr2=0x7f3f7c073730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:21.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.997+0000 7f3fa0d5f640 1 --2- 172.21.15.55:0/3144554377 >> [v2:172.21.15.84:6810/460083528,v1:172.21.15.84:6811/460083528] conn(0x7f3f7c071290 0x7f3f7c073730 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:21.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.997+0000 7f3fa0d5f640 1 -- 172.21.15.55:0/3144554377 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3f7c067660 msgr2=0x7f3f7c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:21.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.997+0000 7f3fa0d5f640 1 --2- 172.21.15.55:0/3144554377 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f3f7c067660 0x7f3f7c069b20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3f840b2000 tx=0x7f3f84008040 comp rx=0 tx=0).stop 2024-03-19T02:06:21.000 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.997+0000 7f3fa0d5f640 1 -- 172.21.15.55:0/3144554377 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f9c06b2a0 msgr2=0x7f3f9c1efe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:21.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.997+0000 7f3fa0d5f640 1 --2- 172.21.15.55:0/3144554377 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f9c06b2a0 0x7f3f9c1efe30 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c00e930 tx=0x7f3f8c00edf0 comp rx=0 tx=0).stop 2024-03-19T02:06:21.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.997+0000 7f3f9ad9d640 1 -- 172.21.15.55:0/3144554377 reap_dead start 2024-03-19T02:06:21.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.997+0000 7f3fa0d5f640 1 -- 172.21.15.55:0/3144554377 shutdown_connections 2024-03-19T02:06:21.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.997+0000 7f3fa0d5f640 1 -- 172.21.15.55:0/3144554377 >> 172.21.15.55:0/3144554377 conn(0x7f3f9c06e670 msgr2=0x7f3f9c0cc0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:21.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.997+0000 7f3fa0d5f640 1 -- 172.21.15.55:0/3144554377 shutdown_connections 2024-03-19T02:06:21.001 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:20.997+0000 7f3fa0d5f640 1 -- 172.21.15.55:0/3144554377 wait complete. 2024-03-19T02:06:21.015 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-19T02:06:21.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.125+0000 7fc1f0520640 1 Processor -- start 2024-03-19T02:06:21.126 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.125+0000 7fc1f0520640 1 -- start start 2024-03-19T02:06:21.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.125+0000 7fc1f0520640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc1e8150e50 0x7fc1e8072900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:21.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.125+0000 7fc1f0520640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1e80738b0 con 0x7fc1e815a490 2024-03-19T02:06:21.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.125+0000 7fc1f0520640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1e8072f40 con 0x7fc1e80ca350 2024-03-19T02:06:21.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.125+0000 7fc1f0520640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7fc1e80732f0 con 0x7fc1e8150e50 2024-03-19T02:06:21.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.125+0000 7fc1ee2bc640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc1e8150e50 0x7fc1e8072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:21.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.129+0000 7fc1ee2bc640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7fc1e8150e50 0x7fc1e8072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60884/0 (socket says 172.21.15.55:60884) 2024-03-19T02:06:21.128 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.129+0000 7fc1ee2bc640 1 -- 172.21.15.55:0/1015272173 learned_addr learned my addr 172.21.15.55:0/1015272173 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:21.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.129+0000 7fc1ee2bc640 1 -- 172.21.15.55:0/1015272173 >> v1:172.21.15.84:6789/0 conn(0x7fc1e80ca350 legacy=0x7fc1e8070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:21.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.129+0000 7fc1ee2bc640 1 -- 172.21.15.55:0/1015272173 >> v1:172.21.15.55:6789/0 conn(0x7fc1e815a490 legacy=0x7fc1e81578b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:21.131 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.129+0000 7fc1ee2bc640 1 -- 172.21.15.55:0/1015272173 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1e80735d0 con 0x7fc1e8150e50 2024-03-19T02:06:21.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.133+0000 7fc1ee2bc640 1 --2- 172.21.15.55:0/1015272173 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1e8150e50 0x7fc1e8072900 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4009870 tx=0x7fc1e401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c8d2aaa7da4fa51 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:21.139 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.137+0000 7fc1ed2ba640 1 -- 172.21.15.55:0/1015272173 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e40ab610 con 0x7fc1e8150e50 2024-03-19T02:06:21.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.137+0000 7fc1ed2ba640 1 -- 172.21.15.55:0/1015272173 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1e40b0070 con 0x7fc1e8150e50 2024-03-19T02:06:21.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.137+0000 7fc1ed2ba640 1 -- 172.21.15.55:0/1015272173 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e40ab9d0 con 0x7fc1e8150e50 2024-03-19T02:06:21.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.141+0000 7fc1f0520640 1 -- 172.21.15.55:0/1015272173 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1e8150e50 msgr2=0x7fc1e8072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:21.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.141+0000 7fc1f0520640 1 --2- 172.21.15.55:0/1015272173 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1e8150e50 0x7fc1e8072900 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4009870 tx=0x7fc1e401c520 comp rx=0 tx=0).stop 2024-03-19T02:06:21.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.141+0000 7fc1f0520640 1 -- 172.21.15.55:0/1015272173 shutdown_connections 2024-03-19T02:06:21.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.141+0000 7fc1f0520640 1 --2- 172.21.15.55:0/1015272173 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1e8150e50 0x7fc1e8072900 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:21.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.141+0000 7fc1f0520640 1 -- 172.21.15.55:0/1015272173 >> 172.21.15.55:0/1015272173 conn(0x7fc1e806c3b0 msgr2=0x7fc1e806c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:21.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.141+0000 7fc1f0520640 1 -- 172.21.15.55:0/1015272173 shutdown_connections 2024-03-19T02:06:21.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.141+0000 7fc1f0520640 1 -- 172.21.15.55:0/1015272173 wait complete. 2024-03-19T02:06:21.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.141+0000 7fc1f0520640 1 Processor -- start 2024-03-19T02:06:21.141 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.141+0000 7fc1f0520640 1 -- start start 2024-03-19T02:06:21.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.141+0000 7fc1f0520640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc1e80ca350 0x7fc1e81f4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:21.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.141+0000 7fc1f0520640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1e8150e50 0x7fc1e8143d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:21.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.141+0000 7fc1f0520640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1e815a490 0x7fc1e813bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:21.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.141+0000 7fc1f0520640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e81513b0 con 0x7fc1e815a490 2024-03-19T02:06:21.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.141+0000 7fc1f0520640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e8151230 con 0x7fc1e8150e50 2024-03-19T02:06:21.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.141+0000 7fc1f0520640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7fc1e8070c10 con 0x7fc1e80ca350 2024-03-19T02:06:21.143 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.141+0000 7fc1eeabd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1e815a490 0x7fc1e813bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:21.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.141+0000 7fc1edabb640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1e8150e50 0x7fc1e8143d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:21.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.141+0000 7fc1eeabd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1e815a490 0x7fc1e813bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60900/0 (socket says 172.21.15.55:60900) 2024-03-19T02:06:21.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.141+0000 7fc1eeabd640 1 -- 172.21.15.55:0/358251294 learned_addr learned my addr 172.21.15.55:0/358251294 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:21.144 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.141+0000 7fc1ee2bc640 1 --2- 172.21.15.55:0/358251294 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc1e80ca350 0x7fc1e81f4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:21.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.149+0000 7fc1eeabd640 1 -- 172.21.15.55:0/358251294 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc1e80ca350 msgr2=0x7fc1e81f4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:21.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.149+0000 7fc1eeabd640 1 --2- 172.21.15.55:0/358251294 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc1e80ca350 0x7fc1e81f4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:21.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.149+0000 7fc1eeabd640 1 -- 172.21.15.55:0/358251294 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1e8150e50 msgr2=0x7fc1e8143d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:21.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.149+0000 7fc1eeabd640 1 --2- 172.21.15.55:0/358251294 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1e8150e50 0x7fc1e8143d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:21.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.149+0000 7fc1eeabd640 1 -- 172.21.15.55:0/358251294 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1e814e820 con 0x7fc1e815a490 2024-03-19T02:06:21.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.149+0000 7fc1eeabd640 1 --2- 172.21.15.55:0/358251294 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1e815a490 0x7fc1e813bd50 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000c730 tx=0x7fc1e000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:21.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.161+0000 7fc1deffd640 1 -- 172.21.15.55:0/358251294 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0003b80 con 0x7fc1e815a490 2024-03-19T02:06:21.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.161+0000 7fc1deffd640 1 -- 172.21.15.55:0/358251294 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1e0003d20 con 0x7fc1e815a490 2024-03-19T02:06:21.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.161+0000 7fc1deffd640 1 -- 172.21.15.55:0/358251294 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e000aa30 con 0x7fc1e815a490 2024-03-19T02:06:21.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.161+0000 7fc1f0520640 1 -- 172.21.15.55:0/358251294 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1e814eb00 con 0x7fc1e815a490 2024-03-19T02:06:21.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.161+0000 7fc1f0520640 1 -- 172.21.15.55:0/358251294 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1e814f0c0 con 0x7fc1e815a490 2024-03-19T02:06:21.161 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.161+0000 7fc1f0520640 1 -- 172.21.15.55:0/358251294 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1e8068a40 con 0x7fc1e815a490 2024-03-19T02:06:21.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.441+0000 7fc1deffd640 1 -- 172.21.15.55:0/358251294 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7fc1e000abd0 con 0x7fc1e815a490 2024-03-19T02:06:21.442 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.441+0000 7fc1deffd640 1 --2- 172.21.15.55:0/358251294 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc1cc067660 0x7fc1cc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:21.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.441+0000 7fc1ee2bc640 1 --2- 172.21.15.55:0/358251294 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc1cc067660 0x7fc1cc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:21.443 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.441+0000 7fc1ee2bc640 1 --2- 172.21.15.55:0/358251294 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc1cc067660 0x7fc1cc069b20 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40af000 tx=0x7fc1e40a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:21.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.465+0000 7fc1deffd640 1 -- 172.21.15.55:0/358251294 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7fc1e00547b0 con 0x7fc1e815a490 2024-03-19T02:06:21.465 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.465+0000 7fc1deffd640 1 -- 172.21.15.55:0/358251294 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7fc1e0056710 con 0x7fc1e815a490 2024-03-19T02:06:21.641 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.641+0000 7fc1f0520640 1 -- 172.21.15.55:0/358251294 --> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fc1e8154850 con 0x7fc1cc067660 2024-03-19T02:06:21.642 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.641+0000 7fc1deffd640 1 -- 172.21.15.55:0/358251294 <== mgr.4105 v2:172.21.15.84:6800/998533783 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc1e8154850 con 0x7fc1cc067660 2024-03-19T02:06:21.642 INFO:teuthology.orchestra.run.smithi055.stderr:instructed osd(s) 7 to deep-scrub 2024-03-19T02:06:21.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.645+0000 7fc1f0520640 1 -- 172.21.15.55:0/358251294 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc1cc067660 msgr2=0x7fc1cc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:21.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.645+0000 7fc1f0520640 1 --2- 172.21.15.55:0/358251294 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc1cc067660 0x7fc1cc069b20 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40af000 tx=0x7fc1e40a7000 comp rx=0 tx=0).stop 2024-03-19T02:06:21.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.645+0000 7fc1f0520640 1 -- 172.21.15.55:0/358251294 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1e815a490 msgr2=0x7fc1e813bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:21.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.645+0000 7fc1f0520640 1 --2- 172.21.15.55:0/358251294 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1e815a490 0x7fc1e813bd50 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000c730 tx=0x7fc1e000cbf0 comp rx=0 tx=0).stop 2024-03-19T02:06:21.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.645+0000 7fc1f0520640 1 -- 172.21.15.55:0/358251294 shutdown_connections 2024-03-19T02:06:21.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.645+0000 7fc1f0520640 1 --2- 172.21.15.55:0/358251294 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7fc1cc067660 0x7fc1cc069b20 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:21.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.645+0000 7fc1f0520640 1 --2- 172.21.15.55:0/358251294 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc1e815a490 0x7fc1e813bd50 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:21.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.645+0000 7fc1f0520640 1 --2- 172.21.15.55:0/358251294 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1e8150e50 0x7fc1e8143d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:21.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.645+0000 7fc1f0520640 1 --2- 172.21.15.55:0/358251294 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7fc1e80ca350 0x7fc1e81f4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:21.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.645+0000 7fc1f0520640 1 -- 172.21.15.55:0/358251294 >> 172.21.15.55:0/358251294 conn(0x7fc1e806c3b0 msgr2=0x7fc1e80c8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:21.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.645+0000 7fc1f0520640 1 -- 172.21.15.55:0/358251294 shutdown_connections 2024-03-19T02:06:21.645 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.645+0000 7fc1f0520640 1 -- 172.21.15.55:0/358251294 wait complete. 2024-03-19T02:06:21.663 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:06:21.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.773+0000 7ff72458a640 1 Processor -- start 2024-03-19T02:06:21.772 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.773+0000 7ff72458a640 1 -- start start 2024-03-19T02:06:21.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.773+0000 7ff72458a640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7ff71c154450 0x7ff71c0706d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:21.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.773+0000 7ff72458a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff71c0c9840 con 0x7ff71c15a2c0 2024-03-19T02:06:21.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.773+0000 7ff72458a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff71c15e410 con 0x7ff71c154070 2024-03-19T02:06:21.773 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.773+0000 7ff72458a640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7ff71c06f660 con 0x7ff71c154450 2024-03-19T02:06:21.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.773+0000 7ff722b27640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7ff71c15a2c0 0x7ff71c15d760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:59892/0 (socket says 172.21.15.55:59892) 2024-03-19T02:06:21.774 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.773+0000 7ff722b27640 1 -- 172.21.15.55:0/2879743108 learned_addr learned my addr 172.21.15.55:0/2879743108 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:06:21.776 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.777+0000 7ff721b25640 1 --2- 172.21.15.55:0/2879743108 >> v2:172.21.15.84:3300/0 conn(0x7ff71c154450 0x7ff71c0706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:21.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.777+0000 7ff721324640 1 -- 172.21.15.55:0/2879743108 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 547798153 0 0) 0x7ff71c0c9840 con 0x7ff71c15a2c0 2024-03-19T02:06:21.777 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.777+0000 7ff721324640 1 -- 172.21.15.55:0/2879743108 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff708003530 con 0x7ff71c15a2c0 2024-03-19T02:06:21.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.781+0000 7ff721324640 1 -- 172.21.15.55:0/2879743108 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2585254705 0 0) 0x7ff708003530 con 0x7ff71c15a2c0 2024-03-19T02:06:21.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.781+0000 7ff721324640 1 -- 172.21.15.55:0/2879743108 >> v2:172.21.15.84:3300/0 conn(0x7ff71c154450 msgr2=0x7ff71c0706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:21.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.781+0000 7ff721324640 1 --2- 172.21.15.55:0/2879743108 >> v2:172.21.15.84:3300/0 conn(0x7ff71c154450 0x7ff71c0706d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:21.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.781+0000 7ff721324640 1 -- 172.21.15.55:0/2879743108 >> v1:172.21.15.84:6789/0 conn(0x7ff71c154070 legacy=0x7ff71c0c9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:21.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.781+0000 7ff721324640 1 -- 172.21.15.55:0/2879743108 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff71c06f940 con 0x7ff71c15a2c0 2024-03-19T02:06:21.782 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.781+0000 7ff721324640 1 -- 172.21.15.55:0/2879743108 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1076293021 0 0) 0x7ff714001a00 con 0x7ff71c15a2c0 2024-03-19T02:06:21.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.781+0000 7ff721324640 1 -- 172.21.15.55:0/2879743108 >> v1:172.21.15.55:6789/0 conn(0x7ff71c15a2c0 legacy=0x7ff71c15d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:21.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.781+0000 7ff721324640 1 --2- 172.21.15.55:0/2879743108 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff708003a10 0x7ff708003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:21.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.781+0000 7ff721324640 1 --2- 172.21.15.55:0/2879743108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff708004810 0x7ff708004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:21.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.781+0000 7ff721324640 1 --2- 172.21.15.55:0/2879743108 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7080051d0 0x7ff70800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:21.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.781+0000 7ff721324640 1 -- 172.21.15.55:0/2879743108 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c06f660 con 0x7ff708004810 2024-03-19T02:06:21.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.781+0000 7ff721324640 1 -- 172.21.15.55:0/2879743108 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff70800c0d0 con 0x7ff7080051d0 2024-03-19T02:06:21.783 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.781+0000 7ff721324640 1 -- 172.21.15.55:0/2879743108 --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff70800c2d0 con 0x7ff708003a10 2024-03-19T02:06:21.786 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.785+0000 7ff721b25640 1 --2- 172.21.15.55:0/2879743108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff708004810 0x7ff708004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:21.788 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.785+0000 7ff722b27640 1 --2- 172.21.15.55:0/2879743108 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7080051d0 0x7ff70800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:21.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.789+0000 7ff721b25640 1 -- 172.21.15.55:0/2879743108 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff708003a10 msgr2=0x7ff708003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:21.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.789+0000 7ff721b25640 1 --2- 172.21.15.55:0/2879743108 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff708003a10 0x7ff708003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:21.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.789+0000 7ff721b25640 1 -- 172.21.15.55:0/2879743108 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7080051d0 msgr2=0x7ff70800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:21.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.789+0000 7ff721b25640 1 --2- 172.21.15.55:0/2879743108 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7080051d0 0x7ff70800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:21.789 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.789+0000 7ff722b27640 1 -- 172.21.15.55:0/2879743108 reap_dead start 2024-03-19T02:06:21.790 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.789+0000 7ff721b25640 1 -- 172.21.15.55:0/2879743108 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff708003530 con 0x7ff708004810 2024-03-19T02:06:21.793 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.793+0000 7ff721b25640 1 --2- 172.21.15.55:0/2879743108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff708004810 0x7ff708004c90 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7ff70c009ae0 tx=0x7ff70c01ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed9bce8fe9c12b7d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:21.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.797+0000 7ff721324640 1 -- 172.21.15.55:0/2879743108 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70c0b2070 con 0x7ff708004810 2024-03-19T02:06:21.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.797+0000 7ff721324640 1 -- 172.21.15.55:0/2879743108 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff70c0a5630 con 0x7ff708004810 2024-03-19T02:06:21.798 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.797+0000 7ff721324640 1 -- 172.21.15.55:0/2879743108 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70c0096b0 con 0x7ff708004810 2024-03-19T02:06:21.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.797+0000 7ff72458a640 1 -- 172.21.15.55:0/2879743108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff708004810 msgr2=0x7ff708004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:21.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.797+0000 7ff72458a640 1 --2- 172.21.15.55:0/2879743108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff708004810 0x7ff708004c90 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7ff70c009ae0 tx=0x7ff70c01ca40 comp rx=0 tx=0).stop 2024-03-19T02:06:21.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.797+0000 7ff72458a640 1 -- 172.21.15.55:0/2879743108 shutdown_connections 2024-03-19T02:06:21.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.797+0000 7ff72458a640 1 --2- 172.21.15.55:0/2879743108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff708004810 0x7ff708004c90 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:21.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.797+0000 7ff72458a640 1 -- 172.21.15.55:0/2879743108 >> 172.21.15.55:0/2879743108 conn(0x7ff71c067600 msgr2=0x7ff71c0714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:21.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.797+0000 7ff72458a640 1 -- 172.21.15.55:0/2879743108 shutdown_connections 2024-03-19T02:06:21.799 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.797+0000 7ff72458a640 1 -- 172.21.15.55:0/2879743108 wait complete. 2024-03-19T02:06:21.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.797+0000 7ff72458a640 1 Processor -- start 2024-03-19T02:06:21.800 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.801+0000 7ff72458a640 1 -- start start 2024-03-19T02:06:21.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.801+0000 7ff72458a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff708004810 0x7ff71c1f0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:21.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.801+0000 7ff72458a640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff71c0ccd70 0x7ff71c1f46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:21.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.801+0000 7ff72458a640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff71c157850 0x7ff71c0c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:21.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.801+0000 7ff72458a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c070420 con 0x7ff708004810 2024-03-19T02:06:21.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.801+0000 7ff72458a640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c06ff00 con 0x7ff71c157850 2024-03-19T02:06:21.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.801+0000 7ff72458a640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7ff71c0c9130 con 0x7ff71c0ccd70 2024-03-19T02:06:21.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.801+0000 7ff722326640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff708004810 0x7ff71c1f0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:21.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.801+0000 7ff722326640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff708004810 0x7ff71c1f0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60918/0 (socket says 172.21.15.55:60918) 2024-03-19T02:06:21.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.801+0000 7ff722326640 1 -- 172.21.15.55:0/1963710142 learned_addr learned my addr 172.21.15.55:0/1963710142 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:21.802 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.801+0000 7ff722b27640 1 --2- 172.21.15.55:0/1963710142 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff71c157850 0x7ff71c0c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:21.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.805+0000 7ff722326640 1 -- 172.21.15.55:0/1963710142 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff71c0ccd70 msgr2=0x7ff71c1f46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:21.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.805+0000 7ff722326640 1 --2- 172.21.15.55:0/1963710142 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff71c0ccd70 0x7ff71c1f46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:21.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.805+0000 7ff722326640 1 -- 172.21.15.55:0/1963710142 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff71c157850 msgr2=0x7ff71c0c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:21.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.805+0000 7ff722326640 1 --2- 172.21.15.55:0/1963710142 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff71c157850 0x7ff71c0c9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:21.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.805+0000 7ff722326640 1 -- 172.21.15.55:0/1963710142 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff71c0ce550 con 0x7ff708004810 2024-03-19T02:06:21.807 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.805+0000 7ff722326640 1 --2- 172.21.15.55:0/1963710142 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff708004810 0x7ff71c1f0700 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7ff71800c500 tx=0x7ff71800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:21.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.809+0000 7ff712ffd640 1 -- 172.21.15.55:0/1963710142 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff718017070 con 0x7ff708004810 2024-03-19T02:06:21.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.813+0000 7ff712ffd640 1 -- 172.21.15.55:0/1963710142 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7180052f0 con 0x7ff708004810 2024-03-19T02:06:21.812 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.813+0000 7ff72458a640 1 -- 172.21.15.55:0/1963710142 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff71c159030 con 0x7ff708004810 2024-03-19T02:06:21.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.813+0000 7ff712ffd640 1 -- 172.21.15.55:0/1963710142 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff718003840 con 0x7ff708004810 2024-03-19T02:06:21.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.813+0000 7ff72458a640 1 -- 172.21.15.55:0/1963710142 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff71c1f3ec0 con 0x7ff708004810 2024-03-19T02:06:21.813 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.813+0000 7ff72458a640 1 -- 172.21.15.55:0/1963710142 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff71c068a40 con 0x7ff708004810 2024-03-19T02:06:21.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.837+0000 7ff712ffd640 1 -- 172.21.15.55:0/1963710142 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7ff71801a070 con 0x7ff708004810 2024-03-19T02:06:21.837 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.837+0000 7ff712ffd640 1 --2- 172.21.15.55:0/1963710142 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff7000676b0 0x7ff700069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:21.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.837+0000 7ff721b25640 1 --2- 172.21.15.55:0/1963710142 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff7000676b0 0x7ff700069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:21.838 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.837+0000 7ff721b25640 1 --2- 172.21.15.55:0/1963710142 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff7000676b0 0x7ff700069b70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff70c0af000 tx=0x7ff70c004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:21.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:21.841+0000 7ff712ffd640 1 -- 172.21.15.55:0/1963710142 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7ff718053470 con 0x7ff708004810 2024-03-19T02:06:22.140 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:22.137+0000 7ff712ffd640 1 -- 172.21.15.55:0/1963710142 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7ff718021080 con 0x7ff708004810 2024-03-19T02:06:22.318 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:22.317+0000 7ff72458a640 1 -- 172.21.15.55:0/1963710142 --> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff71c0b2c10 con 0x7ff7000676b0 2024-03-19T02:06:22.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:22.333+0000 7ff712ffd640 1 -- 172.21.15.55:0/1963710142 <== mgr.4105 v2:172.21.15.84:6800/998533783 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+331633 (secure 0 0 0) 0x7ff71c0b2c10 con 0x7ff7000676b0 2024-03-19T02:06:22.340 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:06:22.342 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-19T02:06:22.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:22.337+0000 7ff72458a640 1 -- 172.21.15.55:0/1963710142 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff7000676b0 msgr2=0x7ff700069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:22.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:22.337+0000 7ff72458a640 1 --2- 172.21.15.55:0/1963710142 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff7000676b0 0x7ff700069b70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff70c0af000 tx=0x7ff70c004740 comp rx=0 tx=0).stop 2024-03-19T02:06:22.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:22.337+0000 7ff72458a640 1 -- 172.21.15.55:0/1963710142 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff708004810 msgr2=0x7ff71c1f0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:22.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:22.337+0000 7ff72458a640 1 --2- 172.21.15.55:0/1963710142 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff708004810 0x7ff71c1f0700 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7ff71800c500 tx=0x7ff71800c900 comp rx=0 tx=0).stop 2024-03-19T02:06:22.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:22.337+0000 7ff72458a640 1 -- 172.21.15.55:0/1963710142 shutdown_connections 2024-03-19T02:06:22.343 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:22.337+0000 7ff72458a640 1 --2- 172.21.15.55:0/1963710142 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7ff7000676b0 0x7ff700069b70 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:22.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:22.337+0000 7ff72458a640 1 --2- 172.21.15.55:0/1963710142 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff71c157850 0x7ff71c0c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:22.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:22.337+0000 7ff72458a640 1 --2- 172.21.15.55:0/1963710142 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7ff71c0ccd70 0x7ff71c1f46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:22.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:22.337+0000 7ff72458a640 1 --2- 172.21.15.55:0/1963710142 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff708004810 0x7ff71c1f0700 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:22.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:22.341+0000 7ff72458a640 1 -- 172.21.15.55:0/1963710142 >> 172.21.15.55:0/1963710142 conn(0x7ff71c067600 msgr2=0x7ff71c150f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:22.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:22.341+0000 7ff72458a640 1 -- 172.21.15.55:0/1963710142 shutdown_connections 2024-03-19T02:06:22.344 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:22.341+0000 7ff72458a640 1 -- 172.21.15.55:0/1963710142 wait complete. 2024-03-19T02:06:22.357 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2024-03-19T02:06:20.607712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":368,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":319,"ondisk_log_size":319,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237188,"kb_used_data":4392,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737532,"statfs":{"total":773094113280,"available":772851232768,"internally_reserved":0,"allocated":4497408,"data_stored":2535616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004852"},"pg_stats":[{"pgid":"2.1f","version":"35'19","reported_seq":76,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:17.916642+0000","last_change":"2024-03-19T02:06:17.916673+0000","last_active":"2024-03-19T02:06:17.916642+0000","last_peered":"2024-03-19T02:06:17.916642+0000","last_clean":"2024-03-19T02:06:17.916642+0000","last_became_active":"2024-03-19T01:51:06.981802+0000","last_became_peered":"2024-03-19T01:51:06.981802+0000","last_unstale":"2024-03-19T02:06:17.916642+0000","last_undegraded":"2024-03-19T02:06:17.916642+0000","last_fullsized":"2024-03-19T02:06:17.916642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'19","last_scrub_stamp":"2024-03-19T02:06:17.916580+0000","last_deep_scrub":"35'19","last_deep_scrub_stamp":"2024-03-19T02:06:17.916580+0000","last_clean_scrub_stamp":"2024-03-19T02:06:17.916580+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:26:00.196368+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:11.629408+0000","last_change":"2024-03-19T01:51:09.172057+0000","last_active":"2024-03-19T02:03:11.629408+0000","last_peered":"2024-03-19T02:03:11.629408+0000","last_clean":"2024-03-19T02:03:11.629408+0000","last_became_active":"2024-03-19T01:51:09.171753+0000","last_became_peered":"2024-03-19T01:51:09.171753+0000","last_unstale":"2024-03-19T02:03:11.629408+0000","last_undegraded":"2024-03-19T02:03:11.629408+0000","last_fullsized":"2024-03-19T02:03:11.629408+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:50.575804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:21.630796+0000","last_change":"2024-03-19T01:51:16.537753+0000","last_active":"2024-03-19T02:03:21.630796+0000","last_peered":"2024-03-19T02:03:21.630796+0000","last_clean":"2024-03-19T02:03:21.630796+0000","last_became_active":"2024-03-19T01:51:16.534200+0000","last_became_peered":"2024-03-19T01:51:16.534200+0000","last_unstale":"2024-03-19T02:03:21.630796+0000","last_undegraded":"2024-03-19T02:03:21.630796+0000","last_fullsized":"2024-03-19T02:03:21.630796+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:18:34.208854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":50,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:06.628579+0000","last_change":"2024-03-19T01:51:06.962968+0000","last_active":"2024-03-19T02:03:06.628579+0000","last_peered":"2024-03-19T02:03:06.628579+0000","last_clean":"2024-03-19T02:03:06.628579+0000","last_became_active":"2024-03-19T01:51:06.961856+0000","last_became_peered":"2024-03-19T01:51:06.961856+0000","last_unstale":"2024-03-19T02:03:06.628579+0000","last_undegraded":"2024-03-19T02:03:06.628579+0000","last_fullsized":"2024-03-19T02:03:06.628579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:49.676651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":45,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:05:43.481930+0000","last_change":"2024-03-19T01:51:09.160338+0000","last_active":"2024-03-19T02:05:43.481930+0000","last_peered":"2024-03-19T02:05:43.481930+0000","last_clean":"2024-03-19T02:05:43.481930+0000","last_became_active":"2024-03-19T01:51:09.158682+0000","last_became_peered":"2024-03-19T01:51:09.158682+0000","last_unstale":"2024-03-19T02:05:43.481930+0000","last_undegraded":"2024-03-19T02:05:43.481930+0000","last_fullsized":"2024-03-19T02:05:43.481930+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":39,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:23.616915+0000","last_change":"2024-03-19T01:52:14.299568+0000","last_active":"2024-03-19T02:03:23.616915+0000","last_peered":"2024-03-19T02:03:23.616915+0000","last_clean":"2024-03-19T02:03:23.616915+0000","last_became_active":"2024-03-19T01:52:14.299067+0000","last_became_peered":"2024-03-19T01:52:14.299067+0000","last_unstale":"2024-03-19T02:03:23.616915+0000","last_undegraded":"2024-03-19T02:03:23.616915+0000","last_fullsized":"2024-03-19T02:03:23.616915+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:15:10.967874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"35'12","reported_seq":59,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603963+0000","last_change":"2024-03-19T01:51:07.044049+0000","last_active":"2024-03-19T02:03:38.603963+0000","last_peered":"2024-03-19T02:03:38.603963+0000","last_clean":"2024-03-19T02:03:38.603963+0000","last_became_active":"2024-03-19T01:51:07.043209+0000","last_became_peered":"2024-03-19T01:51:07.043209+0000","last_unstale":"2024-03-19T02:03:38.603963+0000","last_undegraded":"2024-03-19T02:03:38.603963+0000","last_fullsized":"2024-03-19T02:03:38.603963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:42.992419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:41.673745+0000","last_change":"2024-03-19T01:51:09.179615+0000","last_active":"2024-03-19T02:03:41.673745+0000","last_peered":"2024-03-19T02:03:41.673745+0000","last_clean":"2024-03-19T02:03:41.673745+0000","last_became_active":"2024-03-19T01:51:09.173962+0000","last_became_peered":"2024-03-19T01:51:09.173962+0000","last_unstale":"2024-03-19T02:03:41.673745+0000","last_undegraded":"2024-03-19T02:03:41.673745+0000","last_fullsized":"2024-03-19T02:03:41.673745+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:06:12.676868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":39,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.619503+0000","last_change":"2024-03-19T01:53:15.447584+0000","last_active":"2024-03-19T02:03:48.619503+0000","last_peered":"2024-03-19T02:03:48.619503+0000","last_clean":"2024-03-19T02:03:48.619503+0000","last_became_active":"2024-03-19T01:53:15.447002+0000","last_became_peered":"2024-03-19T01:53:15.447002+0000","last_unstale":"2024-03-19T02:03:48.619503+0000","last_undegraded":"2024-03-19T02:03:48.619503+0000","last_fullsized":"2024-03-19T02:03:48.619503+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:02:57.393473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6,1],"acting":[4,2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"29'5","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603940+0000","last_change":"2024-03-19T01:51:06.983291+0000","last_active":"2024-03-19T02:03:38.603940+0000","last_peered":"2024-03-19T02:03:38.603940+0000","last_clean":"2024-03-19T02:03:38.603940+0000","last_became_active":"2024-03-19T01:51:06.982309+0000","last_became_peered":"2024-03-19T01:51:06.982309+0000","last_unstale":"2024-03-19T02:03:38.603940+0000","last_undegraded":"2024-03-19T02:03:38.603940+0000","last_fullsized":"2024-03-19T02:03:38.603940+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:27.305286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:41.673765+0000","last_change":"2024-03-19T01:51:09.160738+0000","last_active":"2024-03-19T02:03:41.673765+0000","last_peered":"2024-03-19T02:03:41.673765+0000","last_clean":"2024-03-19T02:03:41.673765+0000","last_became_active":"2024-03-19T01:51:09.159951+0000","last_became_peered":"2024-03-19T01:51:09.159951+0000","last_unstale":"2024-03-19T02:03:41.673765+0000","last_undegraded":"2024-03-19T02:03:41.673765+0000","last_fullsized":"2024-03-19T02:03:41.673765+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:30.620041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.619478+0000","last_change":"2024-03-19T01:51:16.577378+0000","last_active":"2024-03-19T02:03:48.619478+0000","last_peered":"2024-03-19T02:03:48.619478+0000","last_clean":"2024-03-19T02:03:48.619478+0000","last_became_active":"2024-03-19T01:51:16.576256+0000","last_became_peered":"2024-03-19T01:51:16.576256+0000","last_unstale":"2024-03-19T02:03:48.619478+0000","last_undegraded":"2024-03-19T02:03:48.619478+0000","last_fullsized":"2024-03-19T02:03:48.619478+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:16.096547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:01:58.494171+0000","last_change":"2024-03-19T01:51:09.185969+0000","last_active":"2024-03-19T02:01:58.494171+0000","last_peered":"2024-03-19T02:01:58.494171+0000","last_clean":"2024-03-19T02:01:58.494171+0000","last_became_active":"2024-03-19T01:51:09.185378+0000","last_became_peered":"2024-03-19T01:51:09.185378+0000","last_unstale":"2024-03-19T02:01:58.494171+0000","last_undegraded":"2024-03-19T02:01:58.494171+0000","last_fullsized":"2024-03-19T02:01:58.494171+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:45.309301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":51,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.594329+0000","last_change":"2024-03-19T01:51:06.988067+0000","last_active":"2024-03-19T02:02:23.594329+0000","last_peered":"2024-03-19T02:02:23.594329+0000","last_clean":"2024-03-19T02:02:23.594329+0000","last_became_active":"2024-03-19T01:51:06.987450+0000","last_became_peered":"2024-03-19T01:51:06.987450+0000","last_unstale":"2024-03-19T02:02:23.594329+0000","last_undegraded":"2024-03-19T02:02:23.594329+0000","last_fullsized":"2024-03-19T02:02:23.594329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:56.962922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:33.610729+0000","last_change":"2024-03-19T01:51:16.583190+0000","last_active":"2024-03-19T02:02:33.610729+0000","last_peered":"2024-03-19T02:02:33.610729+0000","last_clean":"2024-03-19T02:02:33.610729+0000","last_became_active":"2024-03-19T01:51:16.582532+0000","last_became_peered":"2024-03-19T01:51:16.582532+0000","last_unstale":"2024-03-19T02:02:33.610729+0000","last_undegraded":"2024-03-19T02:02:33.610729+0000","last_fullsized":"2024-03-19T02:02:33.610729+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:16:49.817940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:26.624005+0000","last_change":"2024-03-19T01:51:09.174687+0000","last_active":"2024-03-19T02:02:26.624005+0000","last_peered":"2024-03-19T02:02:26.624005+0000","last_clean":"2024-03-19T02:02:26.624005+0000","last_became_active":"2024-03-19T01:51:09.174409+0000","last_became_peered":"2024-03-19T01:51:09.174409+0000","last_unstale":"2024-03-19T02:02:26.624005+0000","last_undegraded":"2024-03-19T02:02:26.624005+0000","last_fullsized":"2024-03-19T02:02:26.624005+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:33:07.160350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"35'3","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:04:08.522579+0000","last_change":"2024-03-19T01:51:06.955434+0000","last_active":"2024-03-19T02:04:08.522579+0000","last_peered":"2024-03-19T02:04:08.522579+0000","last_clean":"2024-03-19T02:04:08.522579+0000","last_became_active":"2024-03-19T01:51:06.954635+0000","last_became_peered":"2024-03-19T01:51:06.954635+0000","last_unstale":"2024-03-19T02:04:08.522579+0000","last_undegraded":"2024-03-19T02:04:08.522579+0000","last_fullsized":"2024-03-19T02:04:08.522579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:33.610667+0000","last_change":"2024-03-19T01:51:16.588604+0000","last_active":"2024-03-19T02:02:33.610667+0000","last_peered":"2024-03-19T02:02:33.610667+0000","last_clean":"2024-03-19T02:02:33.610667+0000","last_became_active":"2024-03-19T01:51:16.587483+0000","last_became_peered":"2024-03-19T01:51:16.587483+0000","last_unstale":"2024-03-19T02:02:33.610667+0000","last_undegraded":"2024-03-19T02:02:33.610667+0000","last_fullsized":"2024-03-19T02:02:33.610667+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:51.658547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:01.620586+0000","last_change":"2024-03-19T01:51:09.173987+0000","last_active":"2024-03-19T02:02:01.620586+0000","last_peered":"2024-03-19T02:02:01.620586+0000","last_clean":"2024-03-19T02:02:01.620586+0000","last_became_active":"2024-03-19T01:51:09.173648+0000","last_became_peered":"2024-03-19T01:51:09.173648+0000","last_unstale":"2024-03-19T02:02:01.620586+0000","last_undegraded":"2024-03-19T02:02:01.620586+0000","last_fullsized":"2024-03-19T02:02:01.620586+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:13:17.277014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"29'7","reported_seq":57,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:16.000651+0000","last_change":"2024-03-19T01:51:06.981820+0000","last_active":"2024-03-19T02:06:16.000651+0000","last_peered":"2024-03-19T02:06:16.000651+0000","last_clean":"2024-03-19T02:06:16.000651+0000","last_became_active":"2024-03-19T01:51:06.981087+0000","last_became_peered":"2024-03-19T01:51:06.981087+0000","last_unstale":"2024-03-19T02:06:16.000651+0000","last_undegraded":"2024-03-19T02:06:16.000651+0000","last_fullsized":"2024-03-19T02:06:16.000651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:04:48.527847+0000","last_change":"2024-03-19T01:51:16.579017+0000","last_active":"2024-03-19T02:04:48.527847+0000","last_peered":"2024-03-19T02:04:48.527847+0000","last_clean":"2024-03-19T02:04:48.527847+0000","last_became_active":"2024-03-19T01:51:16.573845+0000","last_became_peered":"2024-03-19T01:51:16.573845+0000","last_unstale":"2024-03-19T02:04:48.527847+0000","last_undegraded":"2024-03-19T02:04:48.527847+0000","last_fullsized":"2024-03-19T02:04:48.527847+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:04:08.621845+0000","last_change":"2024-03-19T01:51:09.192886+0000","last_active":"2024-03-19T02:04:08.621845+0000","last_peered":"2024-03-19T02:04:08.621845+0000","last_clean":"2024-03-19T02:04:08.621845+0000","last_became_active":"2024-03-19T01:51:09.185970+0000","last_became_peered":"2024-03-19T01:51:09.185970+0000","last_unstale":"2024-03-19T02:04:08.621845+0000","last_undegraded":"2024-03-19T02:04:08.621845+0000","last_fullsized":"2024-03-19T02:04:08.621845+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:05.196985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":50,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:04:08.607086+0000","last_change":"2024-03-19T01:51:06.984875+0000","last_active":"2024-03-19T02:04:08.607086+0000","last_peered":"2024-03-19T02:04:08.607086+0000","last_clean":"2024-03-19T02:04:08.607086+0000","last_became_active":"2024-03-19T01:51:06.983858+0000","last_became_peered":"2024-03-19T01:51:06.983858+0000","last_unstale":"2024-03-19T02:04:08.607086+0000","last_undegraded":"2024-03-19T02:04:08.607086+0000","last_fullsized":"2024-03-19T02:04:08.607086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:14.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:33.610746+0000","last_change":"2024-03-19T01:51:16.591614+0000","last_active":"2024-03-19T02:02:33.610746+0000","last_peered":"2024-03-19T02:02:33.610746+0000","last_clean":"2024-03-19T02:02:33.610746+0000","last_became_active":"2024-03-19T01:51:16.590968+0000","last_became_peered":"2024-03-19T01:51:16.590968+0000","last_unstale":"2024-03-19T02:02:33.610746+0000","last_undegraded":"2024-03-19T02:02:33.610746+0000","last_fullsized":"2024-03-19T02:02:33.610746+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:50.144136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":45,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:15.072796+0000","last_change":"2024-03-19T01:51:09.168941+0000","last_active":"2024-03-19T02:06:15.072796+0000","last_peered":"2024-03-19T02:06:15.072796+0000","last_clean":"2024-03-19T02:06:15.072796+0000","last_became_active":"2024-03-19T01:51:09.168243+0000","last_became_peered":"2024-03-19T01:51:09.168243+0000","last_unstale":"2024-03-19T02:06:15.072796+0000","last_undegraded":"2024-03-19T02:06:15.072796+0000","last_fullsized":"2024-03-19T02:06:15.072796+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'7","reported_seq":55,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:04:08.509375+0000","last_change":"2024-03-19T01:51:07.042870+0000","last_active":"2024-03-19T02:04:08.509375+0000","last_peered":"2024-03-19T02:04:08.509375+0000","last_clean":"2024-03-19T02:04:08.509375+0000","last_became_active":"2024-03-19T01:51:07.042327+0000","last_became_peered":"2024-03-19T01:51:07.042327+0000","last_unstale":"2024-03-19T02:04:08.509375+0000","last_undegraded":"2024-03-19T02:04:08.509375+0000","last_fullsized":"2024-03-19T02:04:08.509375+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:13.137673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.619433+0000","last_change":"2024-03-19T01:51:16.587730+0000","last_active":"2024-03-19T02:03:48.619433+0000","last_peered":"2024-03-19T02:03:48.619433+0000","last_clean":"2024-03-19T02:03:48.619433+0000","last_became_active":"2024-03-19T01:51:16.586614+0000","last_became_peered":"2024-03-19T01:51:16.586614+0000","last_unstale":"2024-03-19T02:03:48.619433+0000","last_undegraded":"2024-03-19T02:03:48.619433+0000","last_fullsized":"2024-03-19T02:03:48.619433+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:44.830637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:01.620647+0000","last_change":"2024-03-19T01:51:09.186397+0000","last_active":"2024-03-19T02:02:01.620647+0000","last_peered":"2024-03-19T02:02:01.620647+0000","last_clean":"2024-03-19T02:02:01.620647+0000","last_became_active":"2024-03-19T01:51:09.186033+0000","last_became_peered":"2024-03-19T01:51:09.186033+0000","last_unstale":"2024-03-19T02:02:01.620647+0000","last_undegraded":"2024-03-19T02:02:01.620647+0000","last_fullsized":"2024-03-19T02:02:01.620647+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:03:04.997711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"29'7","reported_seq":56,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:16.000414+0000","last_change":"2024-03-19T01:51:07.043697+0000","last_active":"2024-03-19T02:06:16.000414+0000","last_peered":"2024-03-19T02:06:16.000414+0000","last_clean":"2024-03-19T02:06:16.000414+0000","last_became_active":"2024-03-19T01:51:07.043185+0000","last_became_peered":"2024-03-19T01:51:07.043185+0000","last_unstale":"2024-03-19T02:06:16.000414+0000","last_undegraded":"2024-03-19T02:06:16.000414+0000","last_fullsized":"2024-03-19T02:06:16.000414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:16.630056+0000","last_change":"2024-03-19T01:51:16.535771+0000","last_active":"2024-03-19T02:03:16.630056+0000","last_peered":"2024-03-19T02:03:16.630056+0000","last_clean":"2024-03-19T02:03:16.630056+0000","last_became_active":"2024-03-19T01:51:16.533661+0000","last_became_peered":"2024-03-19T01:51:16.533661+0000","last_unstale":"2024-03-19T02:03:16.630056+0000","last_undegraded":"2024-03-19T02:03:16.630056+0000","last_fullsized":"2024-03-19T02:03:16.630056+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:55.283782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":45,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:05:43.421653+0000","last_change":"2024-03-19T01:51:09.157526+0000","last_active":"2024-03-19T02:05:43.421653+0000","last_peered":"2024-03-19T02:05:43.421653+0000","last_clean":"2024-03-19T02:05:43.421653+0000","last_became_active":"2024-03-19T01:51:09.156336+0000","last_became_peered":"2024-03-19T01:51:09.156336+0000","last_unstale":"2024-03-19T02:05:43.421653+0000","last_undegraded":"2024-03-19T02:05:43.421653+0000","last_fullsized":"2024-03-19T02:05:43.421653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'4","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603980+0000","last_change":"2024-03-19T01:51:06.982994+0000","last_active":"2024-03-19T02:03:38.603980+0000","last_peered":"2024-03-19T02:03:38.603980+0000","last_clean":"2024-03-19T02:03:38.603980+0000","last_became_active":"2024-03-19T01:51:06.982133+0000","last_became_peered":"2024-03-19T01:51:06.982133+0000","last_unstale":"2024-03-19T02:03:38.603980+0000","last_undegraded":"2024-03-19T02:03:38.603980+0000","last_fullsized":"2024-03-19T02:03:38.603980+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:03.574098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":34,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:05:53.483214+0000","last_change":"2024-03-19T01:51:16.578493+0000","last_active":"2024-03-19T02:05:53.483214+0000","last_peered":"2024-03-19T02:05:53.483214+0000","last_clean":"2024-03-19T02:05:53.483214+0000","last_became_active":"2024-03-19T01:51:16.578040+0000","last_became_peered":"2024-03-19T01:51:16.578040+0000","last_unstale":"2024-03-19T02:05:53.483214+0000","last_undegraded":"2024-03-19T02:05:53.483214+0000","last_fullsized":"2024-03-19T02:05:53.483214+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:41.673779+0000","last_change":"2024-03-19T01:51:09.180758+0000","last_active":"2024-03-19T02:03:41.673779+0000","last_peered":"2024-03-19T02:03:41.673779+0000","last_clean":"2024-03-19T02:03:41.673779+0000","last_became_active":"2024-03-19T01:51:09.180326+0000","last_became_peered":"2024-03-19T01:51:09.180326+0000","last_unstale":"2024-03-19T02:03:41.673779+0000","last_undegraded":"2024-03-19T02:03:41.673779+0000","last_fullsized":"2024-03-19T02:03:41.673779+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:56:07.046285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"29'1","reported_seq":50,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:05:43.421674+0000","last_change":"2024-03-19T01:51:06.962643+0000","last_active":"2024-03-19T02:05:43.421674+0000","last_peered":"2024-03-19T02:05:43.421674+0000","last_clean":"2024-03-19T02:05:43.421674+0000","last_became_active":"2024-03-19T01:51:06.961862+0000","last_became_peered":"2024-03-19T01:51:06.961862+0000","last_unstale":"2024-03-19T02:05:43.421674+0000","last_undegraded":"2024-03-19T02:05:43.421674+0000","last_fullsized":"2024-03-19T02:05:43.421674+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":34,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:05:53.422811+0000","last_change":"2024-03-19T01:51:16.577730+0000","last_active":"2024-03-19T02:05:53.422811+0000","last_peered":"2024-03-19T02:05:53.422811+0000","last_clean":"2024-03-19T02:05:53.422811+0000","last_became_active":"2024-03-19T01:51:16.576955+0000","last_became_peered":"2024-03-19T01:51:16.576955+0000","last_unstale":"2024-03-19T02:05:53.422811+0000","last_undegraded":"2024-03-19T02:05:53.422811+0000","last_fullsized":"2024-03-19T02:05:53.422811+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.618478+0000","last_change":"2024-03-19T01:51:09.183678+0000","last_active":"2024-03-19T02:03:38.618478+0000","last_peered":"2024-03-19T02:03:38.618478+0000","last_clean":"2024-03-19T02:03:38.618478+0000","last_became_active":"2024-03-19T01:51:09.176780+0000","last_became_peered":"2024-03-19T01:51:09.176780+0000","last_unstale":"2024-03-19T02:03:38.618478+0000","last_undegraded":"2024-03-19T02:03:38.618478+0000","last_fullsized":"2024-03-19T02:03:38.618478+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:11:02.502333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'5","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:36.673161+0000","last_change":"2024-03-19T01:51:06.956307+0000","last_active":"2024-03-19T02:03:36.673161+0000","last_peered":"2024-03-19T02:03:36.673161+0000","last_clean":"2024-03-19T02:03:36.673161+0000","last_became_active":"2024-03-19T01:51:06.955529+0000","last_became_peered":"2024-03-19T01:51:06.955529+0000","last_unstale":"2024-03-19T02:03:36.673161+0000","last_undegraded":"2024-03-19T02:03:36.673161+0000","last_fullsized":"2024-03-19T02:03:36.673161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:10.486012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":34,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:05:53.422856+0000","last_change":"2024-03-19T01:51:16.550718+0000","last_active":"2024-03-19T02:05:53.422856+0000","last_peered":"2024-03-19T02:05:53.422856+0000","last_clean":"2024-03-19T02:05:53.422856+0000","last_became_active":"2024-03-19T01:51:16.550115+0000","last_became_peered":"2024-03-19T01:51:16.550115+0000","last_unstale":"2024-03-19T02:05:53.422856+0000","last_undegraded":"2024-03-19T02:05:53.422856+0000","last_fullsized":"2024-03-19T02:05:53.422856+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":45,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:05:08.530741+0000","last_change":"2024-03-19T01:51:09.155718+0000","last_active":"2024-03-19T02:05:08.530741+0000","last_peered":"2024-03-19T02:05:08.530741+0000","last_clean":"2024-03-19T02:05:08.530741+0000","last_became_active":"2024-03-19T01:51:09.155014+0000","last_became_peered":"2024-03-19T01:51:09.155014+0000","last_unstale":"2024-03-19T02:05:08.530741+0000","last_undegraded":"2024-03-19T02:05:08.530741+0000","last_fullsized":"2024-03-19T02:05:08.530741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"33'6","reported_seq":54,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:36.559085+0000","last_change":"2024-03-19T01:51:06.955372+0000","last_active":"2024-03-19T02:03:36.559085+0000","last_peered":"2024-03-19T02:03:36.559085+0000","last_clean":"2024-03-19T02:03:36.559085+0000","last_became_active":"2024-03-19T01:51:06.954637+0000","last_became_peered":"2024-03-19T01:51:06.954637+0000","last_unstale":"2024-03-19T02:03:36.559085+0000","last_undegraded":"2024-03-19T02:03:36.559085+0000","last_fullsized":"2024-03-19T02:03:36.559085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:38:20.129564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:04:18.524070+0000","last_change":"2024-03-19T01:51:16.575157+0000","last_active":"2024-03-19T02:04:18.524070+0000","last_peered":"2024-03-19T02:04:18.524070+0000","last_clean":"2024-03-19T02:04:18.524070+0000","last_became_active":"2024-03-19T01:51:16.574575+0000","last_became_peered":"2024-03-19T01:51:16.574575+0000","last_unstale":"2024-03-19T02:04:18.524070+0000","last_undegraded":"2024-03-19T02:04:18.524070+0000","last_fullsized":"2024-03-19T02:04:18.524070+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":45,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:05:43.481854+0000","last_change":"2024-03-19T01:51:09.192610+0000","last_active":"2024-03-19T02:05:43.481854+0000","last_peered":"2024-03-19T02:05:43.481854+0000","last_clean":"2024-03-19T02:05:43.481854+0000","last_became_active":"2024-03-19T01:51:09.192280+0000","last_became_peered":"2024-03-19T01:51:09.192280+0000","last_unstale":"2024-03-19T02:05:43.481854+0000","last_undegraded":"2024-03-19T02:05:43.481854+0000","last_fullsized":"2024-03-19T02:05:43.481854+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"33'5","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:08.600010+0000","last_change":"2024-03-19T01:51:06.986895+0000","last_active":"2024-03-19T02:03:08.600010+0000","last_peered":"2024-03-19T02:03:08.600010+0000","last_clean":"2024-03-19T02:03:08.600010+0000","last_became_active":"2024-03-19T01:51:06.985965+0000","last_became_peered":"2024-03-19T01:51:06.985965+0000","last_unstale":"2024-03-19T02:03:08.600010+0000","last_undegraded":"2024-03-19T02:03:08.600010+0000","last_fullsized":"2024-03-19T02:03:08.600010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:04:18.580812+0000","last_change":"2024-03-19T01:51:16.505278+0000","last_active":"2024-03-19T02:04:18.580812+0000","last_peered":"2024-03-19T02:04:18.580812+0000","last_clean":"2024-03-19T02:04:18.580812+0000","last_became_active":"2024-03-19T01:51:16.504908+0000","last_became_peered":"2024-03-19T01:51:16.504908+0000","last_unstale":"2024-03-19T02:04:18.580812+0000","last_undegraded":"2024-03-19T02:04:18.580812+0000","last_fullsized":"2024-03-19T02:04:18.580812+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:41.633422+0000","last_change":"2024-03-19T01:51:09.179725+0000","last_active":"2024-03-19T02:03:41.633422+0000","last_peered":"2024-03-19T02:03:41.633422+0000","last_clean":"2024-03-19T02:03:41.633422+0000","last_became_active":"2024-03-19T01:51:09.179235+0000","last_became_peered":"2024-03-19T01:51:09.179235+0000","last_unstale":"2024-03-19T02:03:41.633422+0000","last_undegraded":"2024-03-19T02:03:41.633422+0000","last_fullsized":"2024-03-19T02:03:41.633422+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:31.395833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'6","reported_seq":55,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:36.673191+0000","last_change":"2024-03-19T01:51:06.953474+0000","last_active":"2024-03-19T02:03:36.673191+0000","last_peered":"2024-03-19T02:03:36.673191+0000","last_clean":"2024-03-19T02:03:36.673191+0000","last_became_active":"2024-03-19T01:51:06.952898+0000","last_became_peered":"2024-03-19T01:51:06.952898+0000","last_unstale":"2024-03-19T02:03:36.673191+0000","last_undegraded":"2024-03-19T02:03:36.673191+0000","last_fullsized":"2024-03-19T02:03:36.673191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:38.808918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":203,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:28.594947+0000","last_change":"2024-03-19T01:50:26.097484+0000","last_active":"2024-03-19T02:02:28.594947+0000","last_peered":"2024-03-19T02:02:28.594947+0000","last_clean":"2024-03-19T02:02:28.594947+0000","last_became_active":"2024-03-19T01:50:26.095496+0000","last_became_peered":"2024-03-19T01:50:26.095496+0000","last_unstale":"2024-03-19T02:02:28.594947+0000","last_undegraded":"2024-03-19T02:02:28.594947+0000","last_fullsized":"2024-03-19T02:02:28.594947+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:50:24.112070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:50:24.112070+0000","last_clean_scrub_stamp":"2024-03-19T01:50:24.112070+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:05:00.843775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":40,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.507121+0000","last_change":"2024-03-19T01:52:14.333085+0000","last_active":"2024-03-19T02:03:48.507121+0000","last_peered":"2024-03-19T02:03:48.507121+0000","last_clean":"2024-03-19T02:03:48.507121+0000","last_became_active":"2024-03-19T01:52:14.332547+0000","last_became_peered":"2024-03-19T01:52:14.332547+0000","last_unstale":"2024-03-19T02:03:48.507121+0000","last_undegraded":"2024-03-19T02:03:48.507121+0000","last_fullsized":"2024-03-19T02:03:48.507121+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:36.377719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4,7],"acting":[6,0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.594377+0000","last_change":"2024-03-19T01:51:09.196648+0000","last_active":"2024-03-19T02:02:23.594377+0000","last_peered":"2024-03-19T02:02:23.594377+0000","last_clean":"2024-03-19T02:02:23.594377+0000","last_became_active":"2024-03-19T01:51:09.196093+0000","last_became_peered":"2024-03-19T01:51:09.196093+0000","last_unstale":"2024-03-19T02:02:23.594377+0000","last_undegraded":"2024-03-19T02:02:23.594377+0000","last_fullsized":"2024-03-19T02:02:23.594377+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:54:09.655994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"29'5","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:21.549717+0000","last_change":"2024-03-19T01:51:06.950023+0000","last_active":"2024-03-19T02:02:21.549717+0000","last_peered":"2024-03-19T02:02:21.549717+0000","last_clean":"2024-03-19T02:02:21.549717+0000","last_became_active":"2024-03-19T01:51:06.949234+0000","last_became_peered":"2024-03-19T01:51:06.949234+0000","last_unstale":"2024-03-19T02:02:21.549717+0000","last_undegraded":"2024-03-19T02:02:21.549717+0000","last_fullsized":"2024-03-19T02:02:21.549717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:54:13.131325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:08.495157+0000","last_change":"2024-03-19T01:51:16.554046+0000","last_active":"2024-03-19T02:02:08.495157+0000","last_peered":"2024-03-19T02:02:08.495157+0000","last_clean":"2024-03-19T02:02:08.495157+0000","last_became_active":"2024-03-19T01:51:16.553595+0000","last_became_peered":"2024-03-19T01:51:16.553595+0000","last_unstale":"2024-03-19T02:02:08.495157+0000","last_undegraded":"2024-03-19T02:02:08.495157+0000","last_fullsized":"2024-03-19T02:02:08.495157+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:25.684226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:26.557693+0000","last_change":"2024-03-19T01:52:12.918926+0000","last_active":"2024-03-19T02:03:26.557693+0000","last_peered":"2024-03-19T02:03:26.557693+0000","last_clean":"2024-03-19T02:03:26.557693+0000","last_became_active":"2024-03-19T01:52:12.918458+0000","last_became_peered":"2024-03-19T01:52:12.918458+0000","last_unstale":"2024-03-19T02:03:26.557693+0000","last_undegraded":"2024-03-19T02:03:26.557693+0000","last_fullsized":"2024-03-19T02:03:26.557693+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:56.318137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'4","reported_seq":60,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:18.401067+0000","last_change":"2024-03-19T02:06:18.401112+0000","last_active":"2024-03-19T02:06:18.401067+0000","last_peered":"2024-03-19T02:06:18.401067+0000","last_clean":"2024-03-19T02:06:18.401067+0000","last_became_active":"2024-03-19T01:51:06.937580+0000","last_became_peered":"2024-03-19T01:51:06.937580+0000","last_unstale":"2024-03-19T02:06:18.401067+0000","last_undegraded":"2024-03-19T02:06:18.401067+0000","last_fullsized":"2024-03-19T02:06:18.401067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4","last_scrub_stamp":"2024-03-19T02:06:18.400964+0000","last_deep_scrub":"35'4","last_deep_scrub_stamp":"2024-03-19T02:06:18.400964+0000","last_clean_scrub_stamp":"2024-03-19T02:06:18.400964+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:16:43.147574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:04:48.585512+0000","last_change":"2024-03-19T01:51:16.582111+0000","last_active":"2024-03-19T02:04:48.585512+0000","last_peered":"2024-03-19T02:04:48.585512+0000","last_clean":"2024-03-19T02:04:48.585512+0000","last_became_active":"2024-03-19T01:51:16.581746+0000","last_became_peered":"2024-03-19T01:51:16.581746+0000","last_unstale":"2024-03-19T02:04:48.585512+0000","last_undegraded":"2024-03-19T02:04:48.585512+0000","last_fullsized":"2024-03-19T02:04:48.585512+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.594430+0000","last_change":"2024-03-19T01:51:09.195225+0000","last_active":"2024-03-19T02:02:23.594430+0000","last_peered":"2024-03-19T02:02:23.594430+0000","last_clean":"2024-03-19T02:02:23.594430+0000","last_became_active":"2024-03-19T01:51:09.194645+0000","last_became_peered":"2024-03-19T01:51:09.194645+0000","last_unstale":"2024-03-19T02:02:23.594430+0000","last_undegraded":"2024-03-19T02:02:23.594430+0000","last_fullsized":"2024-03-19T02:02:23.594430+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:50:04.024767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"29'5","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:21.549762+0000","last_change":"2024-03-19T01:51:06.955976+0000","last_active":"2024-03-19T02:02:21.549762+0000","last_peered":"2024-03-19T02:02:21.549762+0000","last_clean":"2024-03-19T02:02:21.549762+0000","last_became_active":"2024-03-19T01:51:06.954951+0000","last_became_peered":"2024-03-19T01:51:06.954951+0000","last_unstale":"2024-03-19T02:02:21.549762+0000","last_undegraded":"2024-03-19T02:02:21.549762+0000","last_fullsized":"2024-03-19T02:02:21.549762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:47.536586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:06.661555+0000","last_change":"2024-03-19T01:51:16.586904+0000","last_active":"2024-03-19T02:02:06.661555+0000","last_peered":"2024-03-19T02:02:06.661555+0000","last_clean":"2024-03-19T02:02:06.661555+0000","last_became_active":"2024-03-19T01:51:16.586298+0000","last_became_peered":"2024-03-19T01:51:16.586298+0000","last_unstale":"2024-03-19T02:02:06.661555+0000","last_undegraded":"2024-03-19T02:02:06.661555+0000","last_fullsized":"2024-03-19T02:02:06.661555+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:42.703415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.594405+0000","last_change":"2024-03-19T01:51:09.195400+0000","last_active":"2024-03-19T02:02:23.594405+0000","last_peered":"2024-03-19T02:02:23.594405+0000","last_clean":"2024-03-19T02:02:23.594405+0000","last_became_active":"2024-03-19T01:51:09.195111+0000","last_became_peered":"2024-03-19T01:51:09.195111+0000","last_unstale":"2024-03-19T02:02:23.594405+0000","last_undegraded":"2024-03-19T02:02:23.594405+0000","last_fullsized":"2024-03-19T02:02:23.594405+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:12.084475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"35'2","reported_seq":49,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:40.608950+0000","last_change":"2024-03-19T01:51:06.963650+0000","last_active":"2024-03-19T02:03:40.608950+0000","last_peered":"2024-03-19T02:03:40.608950+0000","last_clean":"2024-03-19T02:03:40.608950+0000","last_became_active":"2024-03-19T01:51:06.963028+0000","last_became_peered":"2024-03-19T01:51:06.963028+0000","last_unstale":"2024-03-19T02:03:40.608950+0000","last_undegraded":"2024-03-19T02:03:40.608950+0000","last_fullsized":"2024-03-19T02:03:40.608950+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:59:26.369856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:08.608117+0000","last_change":"2024-03-19T01:51:16.588077+0000","last_active":"2024-03-19T02:02:08.608117+0000","last_peered":"2024-03-19T02:02:08.608117+0000","last_clean":"2024-03-19T02:02:08.608117+0000","last_became_active":"2024-03-19T01:51:16.587459+0000","last_became_peered":"2024-03-19T01:51:16.587459+0000","last_unstale":"2024-03-19T02:02:08.608117+0000","last_undegraded":"2024-03-19T02:02:08.608117+0000","last_fullsized":"2024-03-19T02:02:08.608117+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:17:52.667114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.597258+0000","last_change":"2024-03-19T01:51:09.195758+0000","last_active":"2024-03-19T02:02:48.597258+0000","last_peered":"2024-03-19T02:02:48.597258+0000","last_clean":"2024-03-19T02:02:48.597258+0000","last_became_active":"2024-03-19T01:51:09.195139+0000","last_became_peered":"2024-03-19T01:51:09.195139+0000","last_unstale":"2024-03-19T02:02:48.597258+0000","last_undegraded":"2024-03-19T02:02:48.597258+0000","last_fullsized":"2024-03-19T02:02:48.597258+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:18.326414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"29'17","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:46.552835+0000","last_change":"2024-03-19T01:51:06.941200+0000","last_active":"2024-03-19T02:02:46.552835+0000","last_peered":"2024-03-19T02:02:46.552835+0000","last_clean":"2024-03-19T02:02:46.552835+0000","last_became_active":"2024-03-19T01:51:06.940594+0000","last_became_peered":"2024-03-19T01:51:06.940594+0000","last_unstale":"2024-03-19T02:02:46.552835+0000","last_undegraded":"2024-03-19T02:02:46.552835+0000","last_fullsized":"2024-03-19T02:02:46.552835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:25:05.442423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:05:48.482706+0000","last_change":"2024-03-19T01:51:16.548804+0000","last_active":"2024-03-19T02:05:48.482706+0000","last_peered":"2024-03-19T02:05:48.482706+0000","last_clean":"2024-03-19T02:05:48.482706+0000","last_became_active":"2024-03-19T01:51:16.547800+0000","last_became_peered":"2024-03-19T01:51:16.547800+0000","last_unstale":"2024-03-19T02:05:48.482706+0000","last_undegraded":"2024-03-19T02:05:48.482706+0000","last_fullsized":"2024-03-19T02:05:48.482706+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.499920+0000","last_change":"2024-03-19T01:51:09.167987+0000","last_active":"2024-03-19T02:02:48.499920+0000","last_peered":"2024-03-19T02:02:48.499920+0000","last_clean":"2024-03-19T02:02:48.499920+0000","last_became_active":"2024-03-19T01:51:09.167670+0000","last_became_peered":"2024-03-19T01:51:09.167670+0000","last_unstale":"2024-03-19T02:02:48.499920+0000","last_undegraded":"2024-03-19T02:02:48.499920+0000","last_fullsized":"2024-03-19T02:02:48.499920+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:37.942649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"29'5","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.499949+0000","last_change":"2024-03-19T01:51:07.048197+0000","last_active":"2024-03-19T02:02:48.499949+0000","last_peered":"2024-03-19T02:02:48.499949+0000","last_clean":"2024-03-19T02:02:48.499949+0000","last_became_active":"2024-03-19T01:51:07.047785+0000","last_became_peered":"2024-03-19T01:51:07.047785+0000","last_unstale":"2024-03-19T02:02:48.499949+0000","last_undegraded":"2024-03-19T02:02:48.499949+0000","last_fullsized":"2024-03-19T02:02:48.499949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:52.323249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:21.630885+0000","last_change":"2024-03-19T01:51:16.583394+0000","last_active":"2024-03-19T02:03:21.630885+0000","last_peered":"2024-03-19T02:03:21.630885+0000","last_clean":"2024-03-19T02:03:21.630885+0000","last_became_active":"2024-03-19T01:51:16.582860+0000","last_became_peered":"2024-03-19T01:51:16.582860+0000","last_unstale":"2024-03-19T02:03:21.630885+0000","last_undegraded":"2024-03-19T02:03:21.630885+0000","last_fullsized":"2024-03-19T02:03:21.630885+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:37.669591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.597290+0000","last_change":"2024-03-19T01:51:09.195008+0000","last_active":"2024-03-19T02:02:48.597290+0000","last_peered":"2024-03-19T02:02:48.597290+0000","last_clean":"2024-03-19T02:02:48.597290+0000","last_became_active":"2024-03-19T01:51:09.194728+0000","last_became_peered":"2024-03-19T01:51:09.194728+0000","last_unstale":"2024-03-19T02:02:48.597290+0000","last_undegraded":"2024-03-19T02:02:48.597290+0000","last_fullsized":"2024-03-19T02:02:48.597290+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:15:10.360817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"33'6","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:46.666678+0000","last_change":"2024-03-19T01:51:06.956905+0000","last_active":"2024-03-19T02:02:46.666678+0000","last_peered":"2024-03-19T02:02:46.666678+0000","last_clean":"2024-03-19T02:02:46.666678+0000","last_became_active":"2024-03-19T01:51:06.955991+0000","last_became_peered":"2024-03-19T01:51:06.955991+0000","last_unstale":"2024-03-19T02:02:46.666678+0000","last_undegraded":"2024-03-19T02:02:46.666678+0000","last_fullsized":"2024-03-19T02:02:46.666678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:02:45.970853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":39,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:56.627291+0000","last_change":"2024-03-19T01:52:14.332240+0000","last_active":"2024-03-19T02:02:56.627291+0000","last_peered":"2024-03-19T02:02:56.627291+0000","last_clean":"2024-03-19T02:02:56.627291+0000","last_became_active":"2024-03-19T01:52:14.331292+0000","last_became_peered":"2024-03-19T01:52:14.331292+0000","last_unstale":"2024-03-19T02:02:56.627291+0000","last_undegraded":"2024-03-19T02:02:56.627291+0000","last_fullsized":"2024-03-19T02:02:56.627291+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:14:19.465379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:05:08.587694+0000","last_change":"2024-03-19T01:51:09.188003+0000","last_active":"2024-03-19T02:05:08.587694+0000","last_peered":"2024-03-19T02:05:08.587694+0000","last_clean":"2024-03-19T02:05:08.587694+0000","last_became_active":"2024-03-19T01:51:09.187275+0000","last_became_peered":"2024-03-19T01:51:09.187275+0000","last_unstale":"2024-03-19T02:05:08.587694+0000","last_undegraded":"2024-03-19T02:05:08.587694+0000","last_fullsized":"2024-03-19T02:05:08.587694+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"35'15","reported_seq":71,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:16.884717+0000","last_change":"2024-03-19T02:06:16.884792+0000","last_active":"2024-03-19T02:06:16.884717+0000","last_peered":"2024-03-19T02:06:16.884717+0000","last_clean":"2024-03-19T02:06:16.884717+0000","last_became_active":"2024-03-19T01:51:06.963763+0000","last_became_peered":"2024-03-19T01:51:06.963763+0000","last_unstale":"2024-03-19T02:06:16.884717+0000","last_undegraded":"2024-03-19T02:06:16.884717+0000","last_fullsized":"2024-03-19T02:06:16.884717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'15","last_scrub_stamp":"2024-03-19T02:06:16.884516+0000","last_deep_scrub":"35'15","last_deep_scrub_stamp":"2024-03-19T02:06:16.884516+0000","last_clean_scrub_stamp":"2024-03-19T02:06:16.884516+0000","objects_scrubbed":6,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:17.027904+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:21.671644+0000","last_change":"2024-03-19T01:51:16.574401+0000","last_active":"2024-03-19T02:03:21.671644+0000","last_peered":"2024-03-19T02:03:21.671644+0000","last_clean":"2024-03-19T02:03:21.671644+0000","last_became_active":"2024-03-19T01:51:16.573744+0000","last_became_peered":"2024-03-19T01:51:16.573744+0000","last_unstale":"2024-03-19T02:03:21.671644+0000","last_undegraded":"2024-03-19T02:03:21.671644+0000","last_fullsized":"2024-03-19T02:03:21.671644+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:41.414631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.600865+0000","last_change":"2024-03-19T01:51:09.187207+0000","last_active":"2024-03-19T02:03:13.600865+0000","last_peered":"2024-03-19T02:03:13.600865+0000","last_clean":"2024-03-19T02:03:13.600865+0000","last_became_active":"2024-03-19T01:51:09.175898+0000","last_became_peered":"2024-03-19T01:51:09.175898+0000","last_unstale":"2024-03-19T02:03:13.600865+0000","last_undegraded":"2024-03-19T02:03:13.600865+0000","last_fullsized":"2024-03-19T02:03:13.600865+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:38.387069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'14","reported_seq":61,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.499975+0000","last_change":"2024-03-19T01:51:07.042663+0000","last_active":"2024-03-19T02:02:48.499975+0000","last_peered":"2024-03-19T02:02:48.499975+0000","last_clean":"2024-03-19T02:02:48.499975+0000","last_became_active":"2024-03-19T01:51:07.041262+0000","last_became_peered":"2024-03-19T01:51:07.041262+0000","last_unstale":"2024-03-19T02:02:48.499975+0000","last_undegraded":"2024-03-19T02:02:48.499975+0000","last_fullsized":"2024-03-19T02:02:48.499975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:45.093380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:05:18.531938+0000","last_change":"2024-03-19T01:51:16.549744+0000","last_active":"2024-03-19T02:05:18.531938+0000","last_peered":"2024-03-19T02:05:18.531938+0000","last_clean":"2024-03-19T02:05:18.531938+0000","last_became_active":"2024-03-19T01:51:16.549347+0000","last_became_peered":"2024-03-19T01:51:16.549347+0000","last_unstale":"2024-03-19T02:05:18.531938+0000","last_undegraded":"2024-03-19T02:05:18.531938+0000","last_fullsized":"2024-03-19T02:05:18.531938+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:48.612688+0000","last_change":"2024-03-19T01:51:09.194165+0000","last_active":"2024-03-19T02:02:48.612688+0000","last_peered":"2024-03-19T02:02:48.612688+0000","last_clean":"2024-03-19T02:02:48.612688+0000","last_became_active":"2024-03-19T01:51:09.183607+0000","last_became_peered":"2024-03-19T01:51:09.183607+0000","last_unstale":"2024-03-19T02:02:48.612688+0000","last_undegraded":"2024-03-19T02:02:48.612688+0000","last_fullsized":"2024-03-19T02:02:48.612688+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:09.211794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'5","reported_seq":54,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:05:38.420914+0000","last_change":"2024-03-19T01:51:06.952515+0000","last_active":"2024-03-19T02:05:38.420914+0000","last_peered":"2024-03-19T02:05:38.420914+0000","last_clean":"2024-03-19T02:05:38.420914+0000","last_became_active":"2024-03-19T01:51:06.951700+0000","last_became_peered":"2024-03-19T01:51:06.951700+0000","last_unstale":"2024-03-19T02:05:38.420914+0000","last_undegraded":"2024-03-19T02:05:38.420914+0000","last_fullsized":"2024-03-19T02:05:38.420914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:58.613767+0000","last_change":"2024-03-19T01:51:16.587027+0000","last_active":"2024-03-19T02:02:58.613767+0000","last_peered":"2024-03-19T02:02:58.613767+0000","last_clean":"2024-03-19T02:02:58.613767+0000","last_became_active":"2024-03-19T01:51:16.586514+0000","last_became_peered":"2024-03-19T01:51:16.586514+0000","last_unstale":"2024-03-19T02:02:58.613767+0000","last_undegraded":"2024-03-19T02:02:58.613767+0000","last_fullsized":"2024-03-19T02:02:58.613767+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:06.637617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:05:38.481314+0000","last_change":"2024-03-19T01:51:09.172554+0000","last_active":"2024-03-19T02:05:38.481314+0000","last_peered":"2024-03-19T02:05:38.481314+0000","last_clean":"2024-03-19T02:05:38.481314+0000","last_became_active":"2024-03-19T01:51:09.171814+0000","last_became_peered":"2024-03-19T01:51:09.171814+0000","last_unstale":"2024-03-19T02:05:38.481314+0000","last_undegraded":"2024-03-19T02:05:38.481314+0000","last_fullsized":"2024-03-19T02:05:38.481314+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'4","reported_seq":51,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:11.670336+0000","last_change":"2024-03-19T01:51:07.043404+0000","last_active":"2024-03-19T02:03:11.670336+0000","last_peered":"2024-03-19T02:03:11.670336+0000","last_clean":"2024-03-19T02:03:11.670336+0000","last_became_active":"2024-03-19T01:51:07.043069+0000","last_became_peered":"2024-03-19T01:51:07.043069+0000","last_unstale":"2024-03-19T02:03:11.670336+0000","last_undegraded":"2024-03-19T02:03:11.670336+0000","last_fullsized":"2024-03-19T02:03:11.670336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:13:14.960132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:56.627225+0000","last_change":"2024-03-19T01:51:16.581797+0000","last_active":"2024-03-19T02:02:56.627225+0000","last_peered":"2024-03-19T02:02:56.627225+0000","last_clean":"2024-03-19T02:02:56.627225+0000","last_became_active":"2024-03-19T01:51:16.581174+0000","last_became_peered":"2024-03-19T01:51:16.581174+0000","last_unstale":"2024-03-19T02:02:56.627225+0000","last_undegraded":"2024-03-19T02:02:56.627225+0000","last_fullsized":"2024-03-19T02:02:56.627225+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:29:34.911347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:16.671090+0000","last_change":"2024-03-19T01:51:09.194372+0000","last_active":"2024-03-19T02:03:16.671090+0000","last_peered":"2024-03-19T02:03:16.671090+0000","last_clean":"2024-03-19T02:03:16.671090+0000","last_became_active":"2024-03-19T01:51:09.193656+0000","last_became_peered":"2024-03-19T01:51:09.193656+0000","last_unstale":"2024-03-19T02:03:16.671090+0000","last_undegraded":"2024-03-19T02:03:16.671090+0000","last_fullsized":"2024-03-19T02:03:16.671090+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:23.464803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'4","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.502733+0000","last_change":"2024-03-19T01:51:07.046089+0000","last_active":"2024-03-19T02:03:13.502733+0000","last_peered":"2024-03-19T02:03:13.502733+0000","last_clean":"2024-03-19T02:03:13.502733+0000","last_became_active":"2024-03-19T01:51:07.045342+0000","last_became_peered":"2024-03-19T01:51:07.045342+0000","last_unstale":"2024-03-19T02:03:13.502733+0000","last_undegraded":"2024-03-19T02:03:13.502733+0000","last_fullsized":"2024-03-19T02:03:13.502733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:26.105915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:56.627262+0000","last_change":"2024-03-19T01:51:16.572559+0000","last_active":"2024-03-19T02:02:56.627262+0000","last_peered":"2024-03-19T02:02:56.627262+0000","last_clean":"2024-03-19T02:02:56.627262+0000","last_became_active":"2024-03-19T01:51:16.572283+0000","last_became_peered":"2024-03-19T01:51:16.572283+0000","last_unstale":"2024-03-19T02:02:56.627262+0000","last_undegraded":"2024-03-19T02:02:56.627262+0000","last_fullsized":"2024-03-19T02:02:56.627262+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:42.703415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":34,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:05:18.588778+0000","last_change":"2024-03-19T01:51:16.534088+0000","last_active":"2024-03-19T02:05:18.588778+0000","last_peered":"2024-03-19T02:05:18.588778+0000","last_clean":"2024-03-19T02:05:18.588778+0000","last_became_active":"2024-03-19T01:51:16.533665+0000","last_became_peered":"2024-03-19T01:51:16.533665+0000","last_unstale":"2024-03-19T02:05:18.588778+0000","last_undegraded":"2024-03-19T02:05:18.588778+0000","last_fullsized":"2024-03-19T02:05:18.588778+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:16.354419+0000","last_change":"2024-03-19T02:06:16.354453+0000","last_active":"2024-03-19T02:06:16.354419+0000","last_peered":"2024-03-19T02:06:16.354419+0000","last_clean":"2024-03-19T02:06:16.354419+0000","last_became_active":"2024-03-19T01:51:09.167626+0000","last_became_peered":"2024-03-19T01:51:09.167626+0000","last_unstale":"2024-03-19T02:06:16.354419+0000","last_undegraded":"2024-03-19T02:06:16.354419+0000","last_fullsized":"2024-03-19T02:06:16.354419+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:16.354304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:16.354304+0000","last_clean_scrub_stamp":"2024-03-19T02:06:16.354304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:26:28.867422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"33'4","reported_seq":51,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:11.670233+0000","last_change":"2024-03-19T01:51:06.951225+0000","last_active":"2024-03-19T02:03:11.670233+0000","last_peered":"2024-03-19T02:03:11.670233+0000","last_clean":"2024-03-19T02:03:11.670233+0000","last_became_active":"2024-03-19T01:51:06.950826+0000","last_became_peered":"2024-03-19T01:51:06.950826+0000","last_unstale":"2024-03-19T02:03:11.670233+0000","last_undegraded":"2024-03-19T02:03:11.670233+0000","last_fullsized":"2024-03-19T02:03:11.670233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:24.949925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:48.507160+0000","last_change":"2024-03-19T01:51:16.581997+0000","last_active":"2024-03-19T02:03:48.507160+0000","last_peered":"2024-03-19T02:03:48.507160+0000","last_clean":"2024-03-19T02:03:48.507160+0000","last_became_active":"2024-03-19T01:51:16.581412+0000","last_became_peered":"2024-03-19T01:51:16.581412+0000","last_unstale":"2024-03-19T02:03:48.507160+0000","last_undegraded":"2024-03-19T02:03:48.507160+0000","last_fullsized":"2024-03-19T02:03:48.507160+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:45.744882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:16.630092+0000","last_change":"2024-03-19T01:51:09.161686+0000","last_active":"2024-03-19T02:03:16.630092+0000","last_peered":"2024-03-19T02:03:16.630092+0000","last_clean":"2024-03-19T02:03:16.630092+0000","last_became_active":"2024-03-19T01:51:09.160978+0000","last_became_peered":"2024-03-19T01:51:09.160978+0000","last_unstale":"2024-03-19T02:03:16.630092+0000","last_undegraded":"2024-03-19T02:03:16.630092+0000","last_fullsized":"2024-03-19T02:03:16.630092+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:27.266807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'3","reported_seq":50,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:11.629313+0000","last_change":"2024-03-19T01:51:07.047363+0000","last_active":"2024-03-19T02:03:11.629313+0000","last_peered":"2024-03-19T02:03:11.629313+0000","last_clean":"2024-03-19T02:03:11.629313+0000","last_became_active":"2024-03-19T01:51:07.046886+0000","last_became_peered":"2024-03-19T01:51:07.046886+0000","last_unstale":"2024-03-19T02:03:11.629313+0000","last_undegraded":"2024-03-19T02:03:11.629313+0000","last_fullsized":"2024-03-19T02:03:11.629313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:19.418603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"29'7","reported_seq":54,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.505874+0000","last_change":"2024-03-19T01:51:07.045496+0000","last_active":"2024-03-19T02:03:38.505874+0000","last_peered":"2024-03-19T02:03:38.505874+0000","last_clean":"2024-03-19T02:03:38.505874+0000","last_became_active":"2024-03-19T01:51:07.044554+0000","last_became_peered":"2024-03-19T01:51:07.044554+0000","last_unstale":"2024-03-19T02:03:38.505874+0000","last_undegraded":"2024-03-19T02:03:38.505874+0000","last_fullsized":"2024-03-19T02:03:38.505874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:14.027809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:15.879720+0000","last_change":"2024-03-19T02:06:15.879756+0000","last_active":"2024-03-19T02:06:15.879720+0000","last_peered":"2024-03-19T02:06:15.879720+0000","last_clean":"2024-03-19T02:06:15.879720+0000","last_became_active":"2024-03-19T01:51:09.184666+0000","last_became_peered":"2024-03-19T01:51:09.184666+0000","last_unstale":"2024-03-19T02:06:15.879720+0000","last_undegraded":"2024-03-19T02:06:15.879720+0000","last_fullsized":"2024-03-19T02:06:15.879720+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:15.879603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:15.879603+0000","last_clean_scrub_stamp":"2024-03-19T02:06:15.879603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:23.923689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":39,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:46.560321+0000","last_change":"2024-03-19T01:52:14.298105+0000","last_active":"2024-03-19T02:03:46.560321+0000","last_peered":"2024-03-19T02:03:46.560321+0000","last_clean":"2024-03-19T02:03:46.560321+0000","last_became_active":"2024-03-19T01:52:14.297612+0000","last_became_peered":"2024-03-19T01:52:14.297612+0000","last_unstale":"2024-03-19T02:03:46.560321+0000","last_undegraded":"2024-03-19T02:03:46.560321+0000","last_fullsized":"2024-03-19T02:03:46.560321+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:38.753646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,5],"acting":[2,0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"33'7","reported_seq":57,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:36.632647+0000","last_change":"2024-03-19T01:51:06.982280+0000","last_active":"2024-03-19T02:03:36.632647+0000","last_peered":"2024-03-19T02:03:36.632647+0000","last_clean":"2024-03-19T02:03:36.632647+0000","last_became_active":"2024-03-19T01:51:06.981704+0000","last_became_peered":"2024-03-19T01:51:06.981704+0000","last_unstale":"2024-03-19T02:03:36.632647+0000","last_undegraded":"2024-03-19T02:03:36.632647+0000","last_fullsized":"2024-03-19T02:03:36.632647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:05.452188+0000","last_clean_scrub_stamp":"2024-03-19T01:51:05.452188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:22:12.657985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.618444+0000","last_change":"2024-03-19T01:51:09.184624+0000","last_active":"2024-03-19T02:03:38.618444+0000","last_peered":"2024-03-19T02:03:38.618444+0000","last_clean":"2024-03-19T02:03:38.618444+0000","last_became_active":"2024-03-19T01:51:09.183976+0000","last_became_peered":"2024-03-19T01:51:09.183976+0000","last_unstale":"2024-03-19T02:03:38.618444+0000","last_undegraded":"2024-03-19T02:03:38.618444+0000","last_fullsized":"2024-03-19T02:03:38.618444+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:24:29.313729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":39,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:46.633984+0000","last_change":"2024-03-19T01:52:14.299182+0000","last_active":"2024-03-19T02:03:46.633984+0000","last_peered":"2024-03-19T02:03:46.633984+0000","last_clean":"2024-03-19T02:03:46.633984+0000","last_became_active":"2024-03-19T01:52:14.297976+0000","last_became_peered":"2024-03-19T01:52:14.297976+0000","last_unstale":"2024-03-19T02:03:46.633984+0000","last_undegraded":"2024-03-19T02:03:46.633984+0000","last_fullsized":"2024-03-19T02:03:46.633984+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:15.126883+0000","last_clean_scrub_stamp":"2024-03-19T01:51:15.126883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:31.057362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4,2],"acting":[3,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.505794+0000","last_change":"2024-03-19T01:51:09.169493+0000","last_active":"2024-03-19T02:03:38.505794+0000","last_peered":"2024-03-19T02:03:38.505794+0000","last_clean":"2024-03-19T02:03:38.505794+0000","last_became_active":"2024-03-19T01:51:09.168231+0000","last_became_peered":"2024-03-19T01:51:09.168231+0000","last_unstale":"2024-03-19T02:03:38.505794+0000","last_undegraded":"2024-03-19T02:03:38.505794+0000","last_fullsized":"2024-03-19T02:03:38.505794+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:16:10.624259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:41.559618+0000","last_change":"2024-03-19T01:51:09.155483+0000","last_active":"2024-03-19T02:03:41.559618+0000","last_peered":"2024-03-19T02:03:41.559618+0000","last_clean":"2024-03-19T02:03:41.559618+0000","last_became_active":"2024-03-19T01:51:09.154545+0000","last_became_peered":"2024-03-19T01:51:09.154545+0000","last_unstale":"2024-03-19T02:03:41.559618+0000","last_undegraded":"2024-03-19T02:03:41.559618+0000","last_fullsized":"2024-03-19T02:03:41.559618+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:25:09.517323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:41.673722+0000","last_change":"2024-03-19T01:51:09.180498+0000","last_active":"2024-03-19T02:03:41.673722+0000","last_peered":"2024-03-19T02:03:41.673722+0000","last_clean":"2024-03-19T02:03:41.673722+0000","last_became_active":"2024-03-19T01:51:09.179840+0000","last_became_peered":"2024-03-19T01:51:09.179840+0000","last_unstale":"2024-03-19T02:03:41.673722+0000","last_undegraded":"2024-03-19T02:03:41.673722+0000","last_fullsized":"2024-03-19T02:03:41.673722+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:01:21.716571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":45,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:05:48.482656+0000","last_change":"2024-03-19T01:51:09.170389+0000","last_active":"2024-03-19T02:05:48.482656+0000","last_peered":"2024-03-19T02:05:48.482656+0000","last_clean":"2024-03-19T02:05:48.482656+0000","last_became_active":"2024-03-19T01:51:09.163974+0000","last_became_peered":"2024-03-19T01:51:09.163974+0000","last_unstale":"2024-03-19T02:05:48.482656+0000","last_undegraded":"2024-03-19T02:05:48.482656+0000","last_fullsized":"2024-03-19T02:05:48.482656+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:01.620620+0000","last_change":"2024-03-19T01:51:09.177060+0000","last_active":"2024-03-19T02:02:01.620620+0000","last_peered":"2024-03-19T02:02:01.620620+0000","last_clean":"2024-03-19T02:02:01.620620+0000","last_became_active":"2024-03-19T01:51:09.176474+0000","last_became_peered":"2024-03-19T01:51:09.176474+0000","last_unstale":"2024-03-19T02:02:01.620620+0000","last_undegraded":"2024-03-19T02:02:01.620620+0000","last_fullsized":"2024-03-19T02:02:01.620620+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:56.318137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":45,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:16.000591+0000","last_change":"2024-03-19T01:51:09.168741+0000","last_active":"2024-03-19T02:06:16.000591+0000","last_peered":"2024-03-19T02:06:16.000591+0000","last_clean":"2024-03-19T02:06:16.000591+0000","last_became_active":"2024-03-19T01:51:09.167982+0000","last_became_peered":"2024-03-19T01:51:09.167982+0000","last_unstale":"2024-03-19T02:06:16.000591+0000","last_undegraded":"2024-03-19T02:06:16.000591+0000","last_fullsized":"2024-03-19T02:06:16.000591+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:04:08.509354+0000","last_change":"2024-03-19T01:51:09.189814+0000","last_active":"2024-03-19T02:04:08.509354+0000","last_peered":"2024-03-19T02:04:08.509354+0000","last_clean":"2024-03-19T02:04:08.509354+0000","last_became_active":"2024-03-19T01:51:09.189128+0000","last_became_peered":"2024-03-19T01:51:09.189128+0000","last_unstale":"2024-03-19T02:04:08.509354+0000","last_undegraded":"2024-03-19T02:04:08.509354+0000","last_fullsized":"2024-03-19T02:04:08.509354+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:31:59.279804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:01.620637+0000","last_change":"2024-03-19T01:51:09.183989+0000","last_active":"2024-03-19T02:02:01.620637+0000","last_peered":"2024-03-19T02:02:01.620637+0000","last_clean":"2024-03-19T02:02:01.620637+0000","last_became_active":"2024-03-19T01:51:09.183417+0000","last_became_peered":"2024-03-19T01:51:09.183417+0000","last_unstale":"2024-03-19T02:02:01.620637+0000","last_undegraded":"2024-03-19T02:02:01.620637+0000","last_fullsized":"2024-03-19T02:02:01.620637+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:31.405618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:01.660628+0000","last_change":"2024-03-19T01:51:09.172717+0000","last_active":"2024-03-19T02:02:01.660628+0000","last_peered":"2024-03-19T02:02:01.660628+0000","last_clean":"2024-03-19T02:02:01.660628+0000","last_became_active":"2024-03-19T01:51:09.165872+0000","last_became_peered":"2024-03-19T01:51:09.165872+0000","last_unstale":"2024-03-19T02:02:01.660628+0000","last_undegraded":"2024-03-19T02:02:01.660628+0000","last_fullsized":"2024-03-19T02:02:01.660628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:10.995492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:26.557659+0000","last_change":"2024-03-19T01:52:12.916894+0000","last_active":"2024-03-19T02:03:26.557659+0000","last_peered":"2024-03-19T02:03:26.557659+0000","last_clean":"2024-03-19T02:03:26.557659+0000","last_became_active":"2024-03-19T01:52:12.916608+0000","last_became_peered":"2024-03-19T01:52:12.916608+0000","last_unstale":"2024-03-19T02:03:26.557659+0000","last_undegraded":"2024-03-19T02:03:26.557659+0000","last_fullsized":"2024-03-19T02:03:26.557659+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:31.405618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:26.664130+0000","last_change":"2024-03-19T01:51:09.195141+0000","last_active":"2024-03-19T02:02:26.664130+0000","last_peered":"2024-03-19T02:02:26.664130+0000","last_clean":"2024-03-19T02:02:26.664130+0000","last_became_active":"2024-03-19T01:51:09.194536+0000","last_became_peered":"2024-03-19T01:51:09.194536+0000","last_unstale":"2024-03-19T02:02:26.664130+0000","last_undegraded":"2024-03-19T02:02:26.664130+0000","last_fullsized":"2024-03-19T02:02:26.664130+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:31.395833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:04:38.583954+0000","last_change":"2024-03-19T01:51:09.189197+0000","last_active":"2024-03-19T02:04:38.583954+0000","last_peered":"2024-03-19T02:04:38.583954+0000","last_clean":"2024-03-19T02:04:38.583954+0000","last_became_active":"2024-03-19T01:51:09.188565+0000","last_became_peered":"2024-03-19T01:51:09.188565+0000","last_unstale":"2024-03-19T02:04:38.583954+0000","last_undegraded":"2024-03-19T02:04:38.583954+0000","last_fullsized":"2024-03-19T02:04:38.583954+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:23.497100+0000","last_change":"2024-03-19T01:51:09.168581+0000","last_active":"2024-03-19T02:02:23.497100+0000","last_peered":"2024-03-19T02:02:23.497100+0000","last_clean":"2024-03-19T02:02:23.497100+0000","last_became_active":"2024-03-19T01:51:09.167812+0000","last_became_peered":"2024-03-19T01:51:09.167812+0000","last_unstale":"2024-03-19T02:02:23.497100+0000","last_undegraded":"2024-03-19T02:02:23.497100+0000","last_fullsized":"2024-03-19T02:02:23.497100+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:08.738735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:05:48.422251+0000","last_change":"2024-03-19T01:52:12.915401+0000","last_active":"2024-03-19T02:05:48.422251+0000","last_peered":"2024-03-19T02:05:48.422251+0000","last_clean":"2024-03-19T02:05:48.422251+0000","last_became_active":"2024-03-19T01:52:12.915089+0000","last_became_peered":"2024-03-19T01:52:12.915089+0000","last_unstale":"2024-03-19T02:05:48.422251+0000","last_undegraded":"2024-03-19T02:05:48.422251+0000","last_fullsized":"2024-03-19T02:05:48.422251+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:51.553572+0000","last_change":"2024-03-19T01:51:09.168451+0000","last_active":"2024-03-19T02:02:51.553572+0000","last_peered":"2024-03-19T02:02:51.553572+0000","last_clean":"2024-03-19T02:02:51.553572+0000","last_became_active":"2024-03-19T01:51:09.167744+0000","last_became_peered":"2024-03-19T01:51:09.167744+0000","last_unstale":"2024-03-19T02:02:51.553572+0000","last_undegraded":"2024-03-19T02:02:51.553572+0000","last_fullsized":"2024-03-19T02:02:51.553572+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:21.735485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:51.667789+0000","last_change":"2024-03-19T01:51:09.196067+0000","last_active":"2024-03-19T02:02:51.667789+0000","last_peered":"2024-03-19T02:02:51.667789+0000","last_clean":"2024-03-19T02:02:51.667789+0000","last_became_active":"2024-03-19T01:51:09.195488+0000","last_became_peered":"2024-03-19T01:51:09.195488+0000","last_unstale":"2024-03-19T02:02:51.667789+0000","last_undegraded":"2024-03-19T02:02:51.667789+0000","last_fullsized":"2024-03-19T02:02:51.667789+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:07:44.120161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:02:51.667650+0000","last_change":"2024-03-19T01:51:09.193831+0000","last_active":"2024-03-19T02:02:51.667650+0000","last_peered":"2024-03-19T02:02:51.667650+0000","last_clean":"2024-03-19T02:02:51.667650+0000","last_became_active":"2024-03-19T01:51:09.192777+0000","last_became_peered":"2024-03-19T01:51:09.192777+0000","last_unstale":"2024-03-19T02:02:51.667650+0000","last_undegraded":"2024-03-19T02:02:51.667650+0000","last_fullsized":"2024-03-19T02:02:51.667650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:56.318137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:05:08.530808+0000","last_change":"2024-03-19T01:51:09.193581+0000","last_active":"2024-03-19T02:05:08.530808+0000","last_peered":"2024-03-19T02:05:08.530808+0000","last_clean":"2024-03-19T02:05:08.530808+0000","last_became_active":"2024-03-19T01:51:09.192677+0000","last_became_peered":"2024-03-19T01:51:09.192677+0000","last_unstale":"2024-03-19T02:05:08.530808+0000","last_undegraded":"2024-03-19T02:05:08.530808+0000","last_fullsized":"2024-03-19T02:05:08.530808+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:05:08.587665+0000","last_change":"2024-03-19T01:51:09.165568+0000","last_active":"2024-03-19T02:05:08.587665+0000","last_peered":"2024-03-19T02:05:08.587665+0000","last_clean":"2024-03-19T02:05:08.587665+0000","last_became_active":"2024-03-19T01:51:09.164753+0000","last_became_peered":"2024-03-19T01:51:09.164753+0000","last_unstale":"2024-03-19T02:05:08.587665+0000","last_undegraded":"2024-03-19T02:05:08.587665+0000","last_fullsized":"2024-03-19T02:05:08.587665+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:16.630124+0000","last_change":"2024-03-19T01:51:09.170898+0000","last_active":"2024-03-19T02:03:16.630124+0000","last_peered":"2024-03-19T02:03:16.630124+0000","last_clean":"2024-03-19T02:03:16.630124+0000","last_became_active":"2024-03-19T01:51:09.170153+0000","last_became_peered":"2024-03-19T01:51:09.170153+0000","last_unstale":"2024-03-19T02:03:16.630124+0000","last_undegraded":"2024-03-19T02:03:16.630124+0000","last_fullsized":"2024-03-19T02:03:16.630124+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:41.125800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.615856+0000","last_change":"2024-03-19T01:51:09.185953+0000","last_active":"2024-03-19T02:03:13.615856+0000","last_peered":"2024-03-19T02:03:13.615856+0000","last_clean":"2024-03-19T02:03:13.615856+0000","last_became_active":"2024-03-19T01:51:09.177996+0000","last_became_peered":"2024-03-19T01:51:09.177996+0000","last_unstale":"2024-03-19T02:03:13.615856+0000","last_undegraded":"2024-03-19T02:03:13.615856+0000","last_fullsized":"2024-03-19T02:03:13.615856+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:39:06.045413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.502685+0000","last_change":"2024-03-19T01:51:09.183841+0000","last_active":"2024-03-19T02:03:13.502685+0000","last_peered":"2024-03-19T02:03:13.502685+0000","last_clean":"2024-03-19T02:03:13.502685+0000","last_became_active":"2024-03-19T01:51:09.183045+0000","last_became_peered":"2024-03-19T01:51:09.183045+0000","last_unstale":"2024-03-19T02:03:13.502685+0000","last_undegraded":"2024-03-19T02:03:13.502685+0000","last_fullsized":"2024-03-19T02:03:13.502685+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:37.472824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.502713+0000","last_change":"2024-03-19T01:51:09.184022+0000","last_active":"2024-03-19T02:03:13.502713+0000","last_peered":"2024-03-19T02:03:13.502713+0000","last_clean":"2024-03-19T02:03:13.502713+0000","last_became_active":"2024-03-19T01:51:09.183351+0000","last_became_peered":"2024-03-19T01:51:09.183351+0000","last_unstale":"2024-03-19T02:03:13.502713+0000","last_undegraded":"2024-03-19T02:03:13.502713+0000","last_fullsized":"2024-03-19T02:03:13.502713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:44.300735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:13.600945+0000","last_change":"2024-03-19T01:52:12.922818+0000","last_active":"2024-03-19T02:03:13.600945+0000","last_peered":"2024-03-19T02:03:13.600945+0000","last_clean":"2024-03-19T02:03:13.600945+0000","last_became_active":"2024-03-19T01:52:12.922473+0000","last_became_peered":"2024-03-19T01:52:12.922473+0000","last_unstale":"2024-03-19T02:03:13.600945+0000","last_undegraded":"2024-03-19T02:03:13.600945+0000","last_fullsized":"2024-03-19T02:03:13.600945+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:35.798061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:41.633368+0000","last_change":"2024-03-19T01:51:09.180013+0000","last_active":"2024-03-19T02:03:41.633368+0000","last_peered":"2024-03-19T02:03:41.633368+0000","last_clean":"2024-03-19T02:03:41.633368+0000","last_became_active":"2024-03-19T01:51:09.179541+0000","last_became_peered":"2024-03-19T01:51:09.179541+0000","last_unstale":"2024-03-19T02:03:41.633368+0000","last_undegraded":"2024-03-19T02:03:41.633368+0000","last_fullsized":"2024-03-19T02:03:41.633368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:19:25.933028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603886+0000","last_change":"2024-03-19T01:51:09.188671+0000","last_active":"2024-03-19T02:03:38.603886+0000","last_peered":"2024-03-19T02:03:38.603886+0000","last_clean":"2024-03-19T02:03:38.603886+0000","last_became_active":"2024-03-19T01:51:09.177627+0000","last_became_peered":"2024-03-19T01:51:09.177627+0000","last_unstale":"2024-03-19T02:03:38.603886+0000","last_undegraded":"2024-03-19T02:03:38.603886+0000","last_fullsized":"2024-03-19T02:03:38.603886+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:53:50.539837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:41.673692+0000","last_change":"2024-03-19T01:51:09.180735+0000","last_active":"2024-03-19T02:03:41.673692+0000","last_peered":"2024-03-19T02:03:41.673692+0000","last_clean":"2024-03-19T02:03:41.673692+0000","last_became_active":"2024-03-19T01:51:09.173235+0000","last_became_peered":"2024-03-19T01:51:09.173235+0000","last_unstale":"2024-03-19T02:03:41.673692+0000","last_undegraded":"2024-03-19T02:03:41.673692+0000","last_fullsized":"2024-03-19T02:03:41.673692+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:31.405618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":43,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603918+0000","last_change":"2024-03-19T01:51:09.188346+0000","last_active":"2024-03-19T02:03:38.603918+0000","last_peered":"2024-03-19T02:03:38.603918+0000","last_clean":"2024-03-19T02:03:38.603918+0000","last_became_active":"2024-03-19T01:51:09.187703+0000","last_became_peered":"2024-03-19T01:51:09.187703+0000","last_unstale":"2024-03-19T02:03:38.603918+0000","last_undegraded":"2024-03-19T02:03:38.603918+0000","last_fullsized":"2024-03-19T02:03:38.603918+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:28.051691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:04:11.563457+0000","last_change":"2024-03-19T01:52:12.919846+0000","last_active":"2024-03-19T02:04:11.563457+0000","last_peered":"2024-03-19T02:04:11.563457+0000","last_clean":"2024-03-19T02:04:11.563457+0000","last_became_active":"2024-03-19T01:52:12.918835+0000","last_became_peered":"2024-03-19T01:52:12.918835+0000","last_unstale":"2024-03-19T02:04:11.563457+0000","last_undegraded":"2024-03-19T02:04:11.563457+0000","last_fullsized":"2024-03-19T02:04:11.563457+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:03:04.997711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":45,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:05:43.481904+0000","last_change":"2024-03-19T01:51:09.161476+0000","last_active":"2024-03-19T02:05:43.481904+0000","last_peered":"2024-03-19T02:05:43.481904+0000","last_clean":"2024-03-19T02:05:43.481904+0000","last_became_active":"2024-03-19T01:51:09.158609+0000","last_became_peered":"2024-03-19T01:51:09.158609+0000","last_unstale":"2024-03-19T02:05:43.481904+0000","last_undegraded":"2024-03-19T02:05:43.481904+0000","last_fullsized":"2024-03-19T02:05:43.481904+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:17.378760+0000","last_change":"2024-03-19T02:06:17.378828+0000","last_active":"2024-03-19T02:06:17.378760+0000","last_peered":"2024-03-19T02:06:17.378760+0000","last_clean":"2024-03-19T02:06:17.378760+0000","last_became_active":"2024-03-19T01:51:09.165207+0000","last_became_peered":"2024-03-19T01:51:09.165207+0000","last_unstale":"2024-03-19T02:06:17.378760+0000","last_undegraded":"2024-03-19T02:06:17.378760+0000","last_fullsized":"2024-03-19T02:06:17.378760+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:17.378630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:17.378630+0000","last_clean_scrub_stamp":"2024-03-19T02:06:17.378630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:00:02.131538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:03:38.603999+0000","last_change":"2024-03-19T01:51:09.176023+0000","last_active":"2024-03-19T02:03:38.603999+0000","last_peered":"2024-03-19T02:03:38.603999+0000","last_clean":"2024-03-19T02:03:38.603999+0000","last_became_active":"2024-03-19T01:51:09.175682+0000","last_became_peered":"2024-03-19T01:51:09.175682+0000","last_unstale":"2024-03-19T02:03:38.603999+0000","last_undegraded":"2024-03-19T02:03:38.603999+0000","last_fullsized":"2024-03-19T02:03:38.603999+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:22:35.814983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":209,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":202,"ondisk_log_size":202,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509635,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343828,"statfs":{"total":96636764160,"available":96608079872,"internally_reserved":0,"allocated":1011712,"data_stored":759262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509637,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":171426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509636,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":405504,"data_stored":168124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509637,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31576,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340264,"statfs":{"total":96636764160,"available":96604430336,"internally_reserved":0,"allocated":409600,"data_stored":168326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509636,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":1003520,"data_stored":758036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542339,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":430080,"data_stored":170726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542340,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31580,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":413696,"data_stored":171460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542339,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31588,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340252,"statfs":{"total":96636764160,"available":96604418048,"internally_reserved":0,"allocated":413696,"data_stored":168256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:06:22.365 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.366 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.366 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.366 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.366 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.366 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.366 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.366 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.366 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.367 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.367 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.367 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.367 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.367 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.367 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.367 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.367 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.367 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.368 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.368 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.368 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.368 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.368 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.368 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.368 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.368 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.369 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.369 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.369 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.369 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.369 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.369 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.369 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.369 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.369 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.370 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.370 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.370 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.370 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.370 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.370 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.370 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.370 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.371 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.371 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.371 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.371 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-19T01:50:24.112070+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=50, tm_sec=24, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.371 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.371 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.371 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.372 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.372 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.372 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.372 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.372 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.372 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.372 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.372 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.373 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.373 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.373 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.373 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.373 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.373 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.373 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.373 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.373 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.374 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.374 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.374 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.374 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.374 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.374 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.374 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.374 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.374 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.375 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.375 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.375 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.375 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.375 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.375 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.375 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.375 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.375 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.376 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.376 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.376 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.376 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.376 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-03-19T01:51:05.452188+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=5, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.376 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.376 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-03-19T01:51:15.126883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=15, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.376 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.376 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.376 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.377 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.377 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.377 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.377 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.377 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.377 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.377 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.377 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.377 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.378 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.378 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.378 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.378 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.378 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.378 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.378 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.378 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.379 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.379 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.379 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.379 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.379 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.379 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.379 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.379 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.379 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.379 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.380 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:22.380 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-19T02:06:41.418 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:06:41.421 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:06:42.380 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:06:42.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.508+0000 7f22839f6640 1 Processor -- start 2024-03-19T02:06:42.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.508+0000 7f22839f6640 1 -- start start 2024-03-19T02:06:42.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.508+0000 7f22839f6640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f227c1544f0 0x7f227c073540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:42.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.508+0000 7f22839f6640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f227c06b2a0 0x7f227c15e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:42.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.508+0000 7f22839f6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f227c0c7fa0 con 0x7f227c15acb0 2024-03-19T02:06:42.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.508+0000 7f22839f6640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f227c15b210 con 0x7f227c1544f0 2024-03-19T02:06:42.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.508+0000 7f22839f6640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f227c0c8240 con 0x7f227c06b2a0 2024-03-19T02:06:42.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.508+0000 7f2281792640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f227c1544f0 0x7f227c073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:42.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.508+0000 7f2280f91640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f227c06b2a0 0x7f227c15e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:42.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.508+0000 7f2281f93640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f227c15acb0 0x7f227c0cce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:40282/0 (socket says 172.21.15.55:40282) 2024-03-19T02:06:42.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.508+0000 7f2281f93640 1 -- 172.21.15.55:0/3744100299 learned_addr learned my addr 172.21.15.55:0/3744100299 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-19T02:06:42.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.512+0000 7f2280f91640 1 -- 172.21.15.55:0/3744100299 >> v2:172.21.15.55:3300/0 conn(0x7f227c1544f0 msgr2=0x7f227c073540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:06:42.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.512+0000 7f2280f91640 1 --2- 172.21.15.55:0/3744100299 >> v2:172.21.15.55:3300/0 conn(0x7f227c1544f0 0x7f227c073540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:42.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.512+0000 7f2280f91640 1 -- 172.21.15.55:0/3744100299 >> v1:172.21.15.55:6789/0 conn(0x7f227c15acb0 legacy=0x7f227c0cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:42.515 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.512+0000 7f2280f91640 1 -- 172.21.15.55:0/3744100299 --> v2:172.21.15.84:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f227c15d920 con 0x7f227c06b2a0 2024-03-19T02:06:42.519 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.516+0000 7f2280f91640 1 --2- 172.21.15.55:0/3744100299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f227c06b2a0 0x7f227c15e120 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f227c0ca790 tx=0x7f226c01ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0c85588fdb18d4f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:42.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.524+0000 7f2273fff640 1 -- 172.21.15.55:0/3744100299 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c0b5070 con 0x7f227c06b2a0 2024-03-19T02:06:42.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.524+0000 7f2273fff640 1 -- 172.21.15.55:0/3744100299 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f226c0a8970 con 0x7f227c06b2a0 2024-03-19T02:06:42.526 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.524+0000 7f2273fff640 1 -- 172.21.15.55:0/3744100299 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c0a8d40 con 0x7f227c06b2a0 2024-03-19T02:06:42.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.524+0000 7f22839f6640 1 -- 172.21.15.55:0/3744100299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f227c06b2a0 msgr2=0x7f227c15e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:42.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.524+0000 7f22839f6640 1 --2- 172.21.15.55:0/3744100299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f227c06b2a0 0x7f227c15e120 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f227c0ca790 tx=0x7f226c01ec70 comp rx=0 tx=0).stop 2024-03-19T02:06:42.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.524+0000 7f22839f6640 1 -- 172.21.15.55:0/3744100299 shutdown_connections 2024-03-19T02:06:42.527 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.524+0000 7f22839f6640 1 --2- 172.21.15.55:0/3744100299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f227c06b2a0 0x7f227c15e120 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:42.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.524+0000 7f22839f6640 1 --2- 172.21.15.55:0/3744100299 >> v2:172.21.15.55:3300/0 conn(0x7f227c1544f0 0x7f227c073540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:42.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.524+0000 7f22839f6640 1 -- 172.21.15.55:0/3744100299 >> 172.21.15.55:0/3744100299 conn(0x7f227c06e670 msgr2=0x7f227c06ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:42.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.524+0000 7f22839f6640 1 -- 172.21.15.55:0/3744100299 shutdown_connections 2024-03-19T02:06:42.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.524+0000 7f22839f6640 1 -- 172.21.15.55:0/3744100299 wait complete. 2024-03-19T02:06:42.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.528+0000 7f22839f6640 1 Processor -- start 2024-03-19T02:06:42.528 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.528+0000 7f22839f6640 1 -- start start 2024-03-19T02:06:42.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.528+0000 7f22839f6640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f227c06b2a0 0x7f227c151030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:42.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.528+0000 7f22839f6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f227c1544f0 0x7f227c0caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:42.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.528+0000 7f22839f6640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f227c15acb0 0x7f227c0cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:42.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.528+0000 7f22839f6640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f227c1548d0 con 0x7f227c1544f0 2024-03-19T02:06:42.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.528+0000 7f22839f6640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f227c15e660 con 0x7f227c15acb0 2024-03-19T02:06:42.529 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.528+0000 7f22839f6640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f227c0cd540 con 0x7f227c06b2a0 2024-03-19T02:06:42.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.528+0000 7f2281f93640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f227c15acb0 0x7f227c0cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:42.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.528+0000 7f2280f91640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f227c1544f0 0x7f227c0caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:42.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.528+0000 7f2281f93640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f227c15acb0 0x7f227c0cafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.55:45038/0 (socket says 172.21.15.55:45038) 2024-03-19T02:06:42.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.528+0000 7f2281f93640 1 -- 172.21.15.55:0/3102591391 learned_addr learned my addr 172.21.15.55:0/3102591391 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:06:42.530 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.528+0000 7f2281792640 1 --2- 172.21.15.55:0/3102591391 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f227c06b2a0 0x7f227c151030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:42.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.532+0000 7f2281f93640 1 -- 172.21.15.55:0/3102591391 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f227c06b2a0 msgr2=0x7f227c151030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:42.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.532+0000 7f2281f93640 1 --2- 172.21.15.55:0/3102591391 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f227c06b2a0 0x7f227c151030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:42.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.532+0000 7f2281f93640 1 -- 172.21.15.55:0/3102591391 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f227c1544f0 msgr2=0x7f227c0caa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:42.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.532+0000 7f2281f93640 1 --2- 172.21.15.55:0/3102591391 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f227c1544f0 0x7f227c0caa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:42.535 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.532+0000 7f2281f93640 1 -- 172.21.15.55:0/3102591391 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f227c1f3fa0 con 0x7f227c15acb0 2024-03-19T02:06:42.539 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.536+0000 7f2281f93640 1 --2- 172.21.15.55:0/3102591391 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f227c15acb0 0x7f227c0cafa0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f227400ca70 tx=0x7f227400cf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:42.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.544+0000 7f2271ffb640 1 -- 172.21.15.55:0/3102591391 <== mon.1 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2274017070 con 0x7f227c15acb0 2024-03-19T02:06:42.544 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.544+0000 7f2271ffb640 1 -- 172.21.15.55:0/3102591391 <== mon.1 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f227400abd0 con 0x7f227c15acb0 2024-03-19T02:06:42.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.544+0000 7f2271ffb640 1 -- 172.21.15.55:0/3102591391 <== mon.1 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2274012400 con 0x7f227c15acb0 2024-03-19T02:06:42.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.544+0000 7f22839f6640 1 -- 172.21.15.55:0/3102591391 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f227c06bdc0 con 0x7f227c15acb0 2024-03-19T02:06:42.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.544+0000 7f22839f6640 1 -- 172.21.15.55:0/3102591391 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f227c1524a0 con 0x7f227c15acb0 2024-03-19T02:06:42.545 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.544+0000 7f22839f6640 1 -- 172.21.15.55:0/3102591391 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f227c068a40 con 0x7f227c15acb0 2024-03-19T02:06:42.579 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.576+0000 7f2271ffb640 1 -- 172.21.15.55:0/3102591391 <== mon.1 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f227401a070 con 0x7f227c15acb0 2024-03-19T02:06:42.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.576+0000 7f2271ffb640 1 --2- 172.21.15.55:0/3102591391 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f22600676b0 0x7f2260069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:06:42.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.580+0000 7f2281792640 1 --2- 172.21.15.55:0/3102591391 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f22600676b0 0x7f2260069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:06:42.581 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.580+0000 7f2281792640 1 --2- 172.21.15.55:0/3102591391 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f22600676b0 0x7f2260069b70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2278009b60 tx=0x7f2278009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:06:42.586 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.584+0000 7f2271ffb640 1 -- 172.21.15.55:0/3102591391 <== mon.1 v2:172.21.15.84:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f2274054010 con 0x7f227c15acb0 2024-03-19T02:06:42.927 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:42.924+0000 7f2271ffb640 1 -- 172.21.15.55:0/3102591391 <== mon.1 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f2274021020 con 0x7f227c15acb0 2024-03-19T02:06:43.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:43.108+0000 7f22839f6640 1 -- 172.21.15.55:0/3102591391 --> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f227c0c7be0 con 0x7f22600676b0 2024-03-19T02:06:43.127 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:43.124+0000 7f2271ffb640 1 -- 172.21.15.55:0/3102591391 <== mgr.4105 v2:172.21.15.84:6800/998533783 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332763 (secure 0 0 0) 0x7f227c0c7be0 con 0x7f22600676b0 2024-03-19T02:06:43.131 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:06:43.133 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-19T02:06:43.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:43.128+0000 7f22839f6640 1 -- 172.21.15.55:0/3102591391 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f22600676b0 msgr2=0x7f2260069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:43.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:43.128+0000 7f22839f6640 1 --2- 172.21.15.55:0/3102591391 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f22600676b0 0x7f2260069b70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2278009b60 tx=0x7f2278009190 comp rx=0 tx=0).stop 2024-03-19T02:06:43.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:43.128+0000 7f22839f6640 1 -- 172.21.15.55:0/3102591391 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f227c15acb0 msgr2=0x7f227c0cafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:06:43.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:43.128+0000 7f22839f6640 1 --2- 172.21.15.55:0/3102591391 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f227c15acb0 0x7f227c0cafa0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f227400ca70 tx=0x7f227400cf30 comp rx=0 tx=0).stop 2024-03-19T02:06:43.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:43.128+0000 7f22839f6640 1 -- 172.21.15.55:0/3102591391 shutdown_connections 2024-03-19T02:06:43.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:43.128+0000 7f22839f6640 1 --2- 172.21.15.55:0/3102591391 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f22600676b0 0x7f2260069b70 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:43.134 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:43.128+0000 7f22839f6640 1 --2- 172.21.15.55:0/3102591391 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f227c15acb0 0x7f227c0cafa0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:43.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:43.128+0000 7f22839f6640 1 --2- 172.21.15.55:0/3102591391 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f227c1544f0 0x7f227c0caa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:43.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:43.128+0000 7f22839f6640 1 --2- 172.21.15.55:0/3102591391 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f227c06b2a0 0x7f227c151030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:06:43.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:43.128+0000 7f22839f6640 1 -- 172.21.15.55:0/3102591391 >> 172.21.15.55:0/3102591391 conn(0x7f227c06e670 msgr2=0x7f227c15cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:06:43.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:43.128+0000 7f22839f6640 1 -- 172.21.15.55:0/3102591391 shutdown_connections 2024-03-19T02:06:43.135 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:06:43.128+0000 7f22839f6640 1 -- 172.21.15.55:0/3102591391 wait complete. 2024-03-19T02:06:43.147 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":521,"stamp":"2024-03-19T02:06:42.614156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":368,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":406,"ondisk_log_size":406,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":241320,"kb_used_data":4392,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754733400,"statfs":{"total":773094113280,"available":772847001600,"internally_reserved":0,"allocated":4497408,"data_stored":2535616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":242535986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003303"},"pg_stats":[{"pgid":"2.1f","version":"35'19","reported_seq":76,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:17.916642+0000","last_change":"2024-03-19T02:06:17.916673+0000","last_active":"2024-03-19T02:06:17.916642+0000","last_peered":"2024-03-19T02:06:17.916642+0000","last_clean":"2024-03-19T02:06:17.916642+0000","last_became_active":"2024-03-19T01:51:06.981802+0000","last_became_peered":"2024-03-19T01:51:06.981802+0000","last_unstale":"2024-03-19T02:06:17.916642+0000","last_undegraded":"2024-03-19T02:06:17.916642+0000","last_fullsized":"2024-03-19T02:06:17.916642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'19","last_scrub_stamp":"2024-03-19T02:06:17.916580+0000","last_deep_scrub":"35'19","last_deep_scrub_stamp":"2024-03-19T02:06:17.916580+0000","last_clean_scrub_stamp":"2024-03-19T02:06:17.916580+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:26:00.196368+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:27.349806+0000","last_change":"2024-03-19T02:06:27.349841+0000","last_active":"2024-03-19T02:06:27.349806+0000","last_peered":"2024-03-19T02:06:27.349806+0000","last_clean":"2024-03-19T02:06:27.349806+0000","last_became_active":"2024-03-19T01:51:09.171753+0000","last_became_peered":"2024-03-19T01:51:09.171753+0000","last_unstale":"2024-03-19T02:06:27.349806+0000","last_undegraded":"2024-03-19T02:06:27.349806+0000","last_fullsized":"2024-03-19T02:06:27.349806+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:27.349699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:27.349699+0000","last_clean_scrub_stamp":"2024-03-19T02:06:27.349699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:28:10.213189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:32.371974+0000","last_change":"2024-03-19T02:06:32.372012+0000","last_active":"2024-03-19T02:06:32.371974+0000","last_peered":"2024-03-19T02:06:32.371974+0000","last_clean":"2024-03-19T02:06:32.371974+0000","last_became_active":"2024-03-19T01:51:16.534200+0000","last_became_peered":"2024-03-19T01:51:16.534200+0000","last_unstale":"2024-03-19T02:06:32.371974+0000","last_undegraded":"2024-03-19T02:06:32.371974+0000","last_fullsized":"2024-03-19T02:06:32.371974+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:32.371880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:32.371880+0000","last_clean_scrub_stamp":"2024-03-19T02:06:32.371880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:38:35.816038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":59,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:22.387238+0000","last_change":"2024-03-19T02:06:22.387260+0000","last_active":"2024-03-19T02:06:22.387238+0000","last_peered":"2024-03-19T02:06:22.387238+0000","last_clean":"2024-03-19T02:06:22.387238+0000","last_became_active":"2024-03-19T01:51:06.961856+0000","last_became_peered":"2024-03-19T01:51:06.961856+0000","last_unstale":"2024-03-19T02:06:22.387238+0000","last_undegraded":"2024-03-19T02:06:22.387238+0000","last_fullsized":"2024-03-19T02:06:22.387238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-03-19T02:06:22.387160+0000","last_deep_scrub":"29'2","last_deep_scrub_stamp":"2024-03-19T02:06:22.387160+0000","last_clean_scrub_stamp":"2024-03-19T02:06:22.387160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:43:21.444784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:18.949547+0000","last_change":"2024-03-19T02:06:18.949583+0000","last_active":"2024-03-19T02:06:18.949547+0000","last_peered":"2024-03-19T02:06:18.949547+0000","last_clean":"2024-03-19T02:06:18.949547+0000","last_became_active":"2024-03-19T01:51:09.158682+0000","last_became_peered":"2024-03-19T01:51:09.158682+0000","last_unstale":"2024-03-19T02:06:18.949547+0000","last_undegraded":"2024-03-19T02:06:18.949547+0000","last_fullsized":"2024-03-19T02:06:18.949547+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:18.949433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:18.949433+0000","last_clean_scrub_stamp":"2024-03-19T02:06:18.949433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:48:07.131245+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:32.371391+0000","last_change":"2024-03-19T02:06:32.371429+0000","last_active":"2024-03-19T02:06:32.371391+0000","last_peered":"2024-03-19T02:06:32.371391+0000","last_clean":"2024-03-19T02:06:32.371391+0000","last_became_active":"2024-03-19T01:52:14.299067+0000","last_became_peered":"2024-03-19T01:52:14.299067+0000","last_unstale":"2024-03-19T02:06:32.371391+0000","last_undegraded":"2024-03-19T02:06:32.371391+0000","last_fullsized":"2024-03-19T02:06:32.371391+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:32.371292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:32.371292+0000","last_clean_scrub_stamp":"2024-03-19T02:06:32.371292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:04:03.357841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"35'19","reported_seq":75,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:29.540168+0000","last_change":"2024-03-19T02:06:29.540228+0000","last_active":"2024-03-19T02:06:29.540168+0000","last_peered":"2024-03-19T02:06:29.540168+0000","last_clean":"2024-03-19T02:06:29.540168+0000","last_became_active":"2024-03-19T01:51:07.043209+0000","last_became_peered":"2024-03-19T01:51:07.043209+0000","last_unstale":"2024-03-19T02:06:29.540168+0000","last_undegraded":"2024-03-19T02:06:29.540168+0000","last_fullsized":"2024-03-19T02:06:29.540168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'19","last_scrub_stamp":"2024-03-19T02:06:29.540063+0000","last_deep_scrub":"35'19","last_deep_scrub_stamp":"2024-03-19T02:06:29.540063+0000","last_clean_scrub_stamp":"2024-03-19T02:06:29.540063+0000","objects_scrubbed":7,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:08.570677+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:28.034752+0000","last_change":"2024-03-19T02:06:28.034791+0000","last_active":"2024-03-19T02:06:28.034752+0000","last_peered":"2024-03-19T02:06:28.034752+0000","last_clean":"2024-03-19T02:06:28.034752+0000","last_became_active":"2024-03-19T01:51:09.173962+0000","last_became_peered":"2024-03-19T01:51:09.173962+0000","last_unstale":"2024-03-19T02:06:28.034752+0000","last_undegraded":"2024-03-19T02:06:28.034752+0000","last_fullsized":"2024-03-19T02:06:28.034752+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:28.034641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:28.034641+0000","last_clean_scrub_stamp":"2024-03-19T02:06:28.034641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:27:01.730560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:33.384750+0000","last_change":"2024-03-19T02:06:33.384790+0000","last_active":"2024-03-19T02:06:33.384750+0000","last_peered":"2024-03-19T02:06:33.384750+0000","last_clean":"2024-03-19T02:06:33.384750+0000","last_became_active":"2024-03-19T01:53:15.447002+0000","last_became_peered":"2024-03-19T01:53:15.447002+0000","last_unstale":"2024-03-19T02:06:33.384750+0000","last_undegraded":"2024-03-19T02:06:33.384750+0000","last_fullsized":"2024-03-19T02:06:33.384750+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:33.384644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:33.384644+0000","last_clean_scrub_stamp":"2024-03-19T02:06:33.384644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:12.415258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6,1],"acting":[4,2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"35'8","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:25.598633+0000","last_change":"2024-03-19T02:06:25.598684+0000","last_active":"2024-03-19T02:06:25.598633+0000","last_peered":"2024-03-19T02:06:25.598633+0000","last_clean":"2024-03-19T02:06:25.598633+0000","last_became_active":"2024-03-19T01:51:06.982309+0000","last_became_peered":"2024-03-19T01:51:06.982309+0000","last_unstale":"2024-03-19T02:06:25.598633+0000","last_undegraded":"2024-03-19T02:06:25.598633+0000","last_fullsized":"2024-03-19T02:06:25.598633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'8","last_scrub_stamp":"2024-03-19T02:06:25.598534+0000","last_deep_scrub":"35'8","last_deep_scrub_stamp":"2024-03-19T02:06:25.598534+0000","last_clean_scrub_stamp":"2024-03-19T02:06:25.598534+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:04:01.844620+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:26.033500+0000","last_change":"2024-03-19T02:06:26.033537+0000","last_active":"2024-03-19T02:06:26.033500+0000","last_peered":"2024-03-19T02:06:26.033500+0000","last_clean":"2024-03-19T02:06:26.033500+0000","last_became_active":"2024-03-19T01:51:09.159951+0000","last_became_peered":"2024-03-19T01:51:09.159951+0000","last_unstale":"2024-03-19T02:06:26.033500+0000","last_undegraded":"2024-03-19T02:06:26.033500+0000","last_fullsized":"2024-03-19T02:06:26.033500+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:26.033394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:26.033394+0000","last_clean_scrub_stamp":"2024-03-19T02:06:26.033394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:39:41.332644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:25.340674+0000","last_change":"2024-03-19T02:06:25.340715+0000","last_active":"2024-03-19T02:06:25.340674+0000","last_peered":"2024-03-19T02:06:25.340674+0000","last_clean":"2024-03-19T02:06:25.340674+0000","last_became_active":"2024-03-19T01:51:16.576256+0000","last_became_peered":"2024-03-19T01:51:16.576256+0000","last_unstale":"2024-03-19T02:06:25.340674+0000","last_undegraded":"2024-03-19T02:06:25.340674+0000","last_fullsized":"2024-03-19T02:06:25.340674+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:25.340551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:25.340551+0000","last_clean_scrub_stamp":"2024-03-19T02:06:25.340551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:29.399979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:31.291156+0000","last_change":"2024-03-19T02:06:31.291193+0000","last_active":"2024-03-19T02:06:31.291156+0000","last_peered":"2024-03-19T02:06:31.291156+0000","last_clean":"2024-03-19T02:06:31.291156+0000","last_became_active":"2024-03-19T01:51:09.185378+0000","last_became_peered":"2024-03-19T01:51:09.185378+0000","last_unstale":"2024-03-19T02:06:31.291156+0000","last_undegraded":"2024-03-19T02:06:31.291156+0000","last_fullsized":"2024-03-19T02:06:31.291156+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:31.291046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:31.291046+0000","last_clean_scrub_stamp":"2024-03-19T02:06:31.291046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:04:07.537132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"35'6","reported_seq":62,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:28.558870+0000","last_change":"2024-03-19T02:06:28.558898+0000","last_active":"2024-03-19T02:06:28.558870+0000","last_peered":"2024-03-19T02:06:28.558870+0000","last_clean":"2024-03-19T02:06:28.558870+0000","last_became_active":"2024-03-19T01:51:06.987450+0000","last_became_peered":"2024-03-19T01:51:06.987450+0000","last_unstale":"2024-03-19T02:06:28.558870+0000","last_undegraded":"2024-03-19T02:06:28.558870+0000","last_fullsized":"2024-03-19T02:06:28.558870+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'6","last_scrub_stamp":"2024-03-19T02:06:28.558804+0000","last_deep_scrub":"35'6","last_deep_scrub_stamp":"2024-03-19T02:06:28.558804+0000","last_clean_scrub_stamp":"2024-03-19T02:06:28.558804+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:03:59.545353+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:26.324338+0000","last_change":"2024-03-19T02:06:26.324377+0000","last_active":"2024-03-19T02:06:26.324338+0000","last_peered":"2024-03-19T02:06:26.324338+0000","last_clean":"2024-03-19T02:06:26.324338+0000","last_became_active":"2024-03-19T01:51:16.582532+0000","last_became_peered":"2024-03-19T01:51:16.582532+0000","last_unstale":"2024-03-19T02:06:26.324338+0000","last_undegraded":"2024-03-19T02:06:26.324338+0000","last_fullsized":"2024-03-19T02:06:26.324338+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:26.324234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:26.324234+0000","last_clean_scrub_stamp":"2024-03-19T02:06:26.324234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:18:15.761468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:29.339109+0000","last_change":"2024-03-19T02:06:29.339147+0000","last_active":"2024-03-19T02:06:29.339109+0000","last_peered":"2024-03-19T02:06:29.339109+0000","last_clean":"2024-03-19T02:06:29.339109+0000","last_became_active":"2024-03-19T01:51:09.174409+0000","last_became_peered":"2024-03-19T01:51:09.174409+0000","last_unstale":"2024-03-19T02:06:29.339109+0000","last_undegraded":"2024-03-19T02:06:29.339109+0000","last_fullsized":"2024-03-19T02:06:29.339109+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:29.339008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:29.339008+0000","last_clean_scrub_stamp":"2024-03-19T02:06:29.339008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:14:10.906353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"35'5","reported_seq":63,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:20.478235+0000","last_change":"2024-03-19T02:06:20.478235+0000","last_active":"2024-03-19T02:06:20.478235+0000","last_peered":"2024-03-19T02:06:20.478235+0000","last_clean":"2024-03-19T02:06:20.478235+0000","last_became_active":"2024-03-19T01:51:06.954635+0000","last_became_peered":"2024-03-19T01:51:06.954635+0000","last_unstale":"2024-03-19T02:06:20.478235+0000","last_undegraded":"2024-03-19T02:06:20.478235+0000","last_fullsized":"2024-03-19T02:06:20.478235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5","last_scrub_stamp":"2024-03-19T02:06:20.478085+0000","last_deep_scrub":"35'5","last_deep_scrub_stamp":"2024-03-19T02:06:20.478085+0000","last_clean_scrub_stamp":"2024-03-19T02:06:20.478085+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:59:50.831717+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:30.381814+0000","last_change":"2024-03-19T02:06:30.381853+0000","last_active":"2024-03-19T02:06:30.381814+0000","last_peered":"2024-03-19T02:06:30.381814+0000","last_clean":"2024-03-19T02:06:30.381814+0000","last_became_active":"2024-03-19T01:51:16.587483+0000","last_became_peered":"2024-03-19T01:51:16.587483+0000","last_unstale":"2024-03-19T02:06:30.381814+0000","last_undegraded":"2024-03-19T02:06:30.381814+0000","last_fullsized":"2024-03-19T02:06:30.381814+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:30.381697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:30.381697+0000","last_clean_scrub_stamp":"2024-03-19T02:06:30.381697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:14:36.256253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:28.341166+0000","last_change":"2024-03-19T02:06:28.341203+0000","last_active":"2024-03-19T02:06:28.341166+0000","last_peered":"2024-03-19T02:06:28.341166+0000","last_clean":"2024-03-19T02:06:28.341166+0000","last_became_active":"2024-03-19T01:51:09.173648+0000","last_became_peered":"2024-03-19T01:51:09.173648+0000","last_unstale":"2024-03-19T02:06:28.341166+0000","last_undegraded":"2024-03-19T02:06:28.341166+0000","last_fullsized":"2024-03-19T02:06:28.341166+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:28.341061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:28.341061+0000","last_clean_scrub_stamp":"2024-03-19T02:06:28.341061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:07.793500+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"35'11","reported_seq":69,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:22.415658+0000","last_change":"2024-03-19T02:06:22.415809+0000","last_active":"2024-03-19T02:06:22.415658+0000","last_peered":"2024-03-19T02:06:22.415658+0000","last_clean":"2024-03-19T02:06:22.415658+0000","last_became_active":"2024-03-19T01:51:06.981087+0000","last_became_peered":"2024-03-19T01:51:06.981087+0000","last_unstale":"2024-03-19T02:06:22.415658+0000","last_undegraded":"2024-03-19T02:06:22.415658+0000","last_fullsized":"2024-03-19T02:06:22.415658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'11","last_scrub_stamp":"2024-03-19T02:06:22.415422+0000","last_deep_scrub":"35'11","last_deep_scrub_stamp":"2024-03-19T02:06:22.415422+0000","last_clean_scrub_stamp":"2024-03-19T02:06:22.415422+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:56:14.292172+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:32.369802+0000","last_change":"2024-03-19T02:06:32.369842+0000","last_active":"2024-03-19T02:06:32.369802+0000","last_peered":"2024-03-19T02:06:32.369802+0000","last_clean":"2024-03-19T02:06:32.369802+0000","last_became_active":"2024-03-19T01:51:16.573845+0000","last_became_peered":"2024-03-19T01:51:16.573845+0000","last_unstale":"2024-03-19T02:06:32.369802+0000","last_undegraded":"2024-03-19T02:06:32.369802+0000","last_fullsized":"2024-03-19T02:06:32.369802+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:32.369699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:32.369699+0000","last_clean_scrub_stamp":"2024-03-19T02:06:32.369699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:07.743577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:23.275620+0000","last_change":"2024-03-19T02:06:23.275657+0000","last_active":"2024-03-19T02:06:23.275620+0000","last_peered":"2024-03-19T02:06:23.275620+0000","last_clean":"2024-03-19T02:06:23.275620+0000","last_became_active":"2024-03-19T01:51:09.185970+0000","last_became_peered":"2024-03-19T01:51:09.185970+0000","last_unstale":"2024-03-19T02:06:23.275620+0000","last_undegraded":"2024-03-19T02:06:23.275620+0000","last_fullsized":"2024-03-19T02:06:23.275620+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:23.275500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:23.275500+0000","last_clean_scrub_stamp":"2024-03-19T02:06:23.275500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:48:11.457312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"35'3","reported_seq":60,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:26.576554+0000","last_change":"2024-03-19T02:06:26.576601+0000","last_active":"2024-03-19T02:06:26.576554+0000","last_peered":"2024-03-19T02:06:26.576554+0000","last_clean":"2024-03-19T02:06:26.576554+0000","last_became_active":"2024-03-19T01:51:06.983858+0000","last_became_peered":"2024-03-19T01:51:06.983858+0000","last_unstale":"2024-03-19T02:06:26.576554+0000","last_undegraded":"2024-03-19T02:06:26.576554+0000","last_fullsized":"2024-03-19T02:06:26.576554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3","last_scrub_stamp":"2024-03-19T02:06:26.576460+0000","last_deep_scrub":"35'3","last_deep_scrub_stamp":"2024-03-19T02:06:26.576460+0000","last_clean_scrub_stamp":"2024-03-19T02:06:26.576460+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:14:32.451016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:31.344299+0000","last_change":"2024-03-19T02:06:31.344340+0000","last_active":"2024-03-19T02:06:31.344299+0000","last_peered":"2024-03-19T02:06:31.344299+0000","last_clean":"2024-03-19T02:06:31.344299+0000","last_became_active":"2024-03-19T01:51:16.590968+0000","last_became_peered":"2024-03-19T01:51:16.590968+0000","last_unstale":"2024-03-19T02:06:31.344299+0000","last_undegraded":"2024-03-19T02:06:31.344299+0000","last_fullsized":"2024-03-19T02:06:31.344299+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:31.344187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:31.344187+0000","last_clean_scrub_stamp":"2024-03-19T02:06:31.344187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:21.842807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:22.035764+0000","last_change":"2024-03-19T02:06:22.035802+0000","last_active":"2024-03-19T02:06:22.035764+0000","last_peered":"2024-03-19T02:06:22.035764+0000","last_clean":"2024-03-19T02:06:22.035764+0000","last_became_active":"2024-03-19T01:51:09.168243+0000","last_became_peered":"2024-03-19T01:51:09.168243+0000","last_unstale":"2024-03-19T02:06:22.035764+0000","last_undegraded":"2024-03-19T02:06:22.035764+0000","last_fullsized":"2024-03-19T02:06:22.035764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:22.035658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:22.035658+0000","last_clean_scrub_stamp":"2024-03-19T02:06:22.035658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:31:09.025690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"35'11","reported_seq":68,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:22.264551+0000","last_change":"2024-03-19T02:06:22.264602+0000","last_active":"2024-03-19T02:06:22.264551+0000","last_peered":"2024-03-19T02:06:22.264551+0000","last_clean":"2024-03-19T02:06:22.264551+0000","last_became_active":"2024-03-19T01:51:07.042327+0000","last_became_peered":"2024-03-19T01:51:07.042327+0000","last_unstale":"2024-03-19T02:06:22.264551+0000","last_undegraded":"2024-03-19T02:06:22.264551+0000","last_fullsized":"2024-03-19T02:06:22.264551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'11","last_scrub_stamp":"2024-03-19T02:06:22.264451+0000","last_deep_scrub":"35'11","last_deep_scrub_stamp":"2024-03-19T02:06:22.264451+0000","last_clean_scrub_stamp":"2024-03-19T02:06:22.264451+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:33:46.680769+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:28.370487+0000","last_change":"2024-03-19T02:06:28.370527+0000","last_active":"2024-03-19T02:06:28.370487+0000","last_peered":"2024-03-19T02:06:28.370487+0000","last_clean":"2024-03-19T02:06:28.370487+0000","last_became_active":"2024-03-19T01:51:16.586614+0000","last_became_peered":"2024-03-19T01:51:16.586614+0000","last_unstale":"2024-03-19T02:06:28.370487+0000","last_undegraded":"2024-03-19T02:06:28.370487+0000","last_fullsized":"2024-03-19T02:06:28.370487+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:28.370364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:28.370364+0000","last_clean_scrub_stamp":"2024-03-19T02:06:28.370364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:33:20.645912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:30.346070+0000","last_change":"2024-03-19T02:06:30.346106+0000","last_active":"2024-03-19T02:06:30.346070+0000","last_peered":"2024-03-19T02:06:30.346070+0000","last_clean":"2024-03-19T02:06:30.346070+0000","last_became_active":"2024-03-19T01:51:09.186033+0000","last_became_peered":"2024-03-19T01:51:09.186033+0000","last_unstale":"2024-03-19T02:06:30.346070+0000","last_undegraded":"2024-03-19T02:06:30.346070+0000","last_fullsized":"2024-03-19T02:06:30.346070+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:30.345964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:30.345964+0000","last_clean_scrub_stamp":"2024-03-19T02:06:30.345964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:08:19.486082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"35'11","reported_seq":68,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:23.431605+0000","last_change":"2024-03-19T02:06:23.431657+0000","last_active":"2024-03-19T02:06:23.431605+0000","last_peered":"2024-03-19T02:06:23.431605+0000","last_clean":"2024-03-19T02:06:23.431605+0000","last_became_active":"2024-03-19T01:51:07.043185+0000","last_became_peered":"2024-03-19T01:51:07.043185+0000","last_unstale":"2024-03-19T02:06:23.431605+0000","last_undegraded":"2024-03-19T02:06:23.431605+0000","last_fullsized":"2024-03-19T02:06:23.431605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'11","last_scrub_stamp":"2024-03-19T02:06:23.431506+0000","last_deep_scrub":"35'11","last_deep_scrub_stamp":"2024-03-19T02:06:23.431506+0000","last_clean_scrub_stamp":"2024-03-19T02:06:23.431506+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:32:56.869679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:31.332790+0000","last_change":"2024-03-19T02:06:31.332845+0000","last_active":"2024-03-19T02:06:31.332790+0000","last_peered":"2024-03-19T02:06:31.332790+0000","last_clean":"2024-03-19T02:06:31.332790+0000","last_became_active":"2024-03-19T01:51:16.533661+0000","last_became_peered":"2024-03-19T01:51:16.533661+0000","last_unstale":"2024-03-19T02:06:31.332790+0000","last_undegraded":"2024-03-19T02:06:31.332790+0000","last_fullsized":"2024-03-19T02:06:31.332790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:31.332614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:31.332614+0000","last_clean_scrub_stamp":"2024-03-19T02:06:31.332614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:53:49.821983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:25.404579+0000","last_change":"2024-03-19T02:06:25.404618+0000","last_active":"2024-03-19T02:06:25.404579+0000","last_peered":"2024-03-19T02:06:25.404579+0000","last_clean":"2024-03-19T02:06:25.404579+0000","last_became_active":"2024-03-19T01:51:09.156336+0000","last_became_peered":"2024-03-19T01:51:09.156336+0000","last_unstale":"2024-03-19T02:06:25.404579+0000","last_undegraded":"2024-03-19T02:06:25.404579+0000","last_fullsized":"2024-03-19T02:06:25.404579+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:25.404464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:25.404464+0000","last_clean_scrub_stamp":"2024-03-19T02:06:25.404464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:14:31.279020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"35'7","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:24.608993+0000","last_change":"2024-03-19T02:06:24.609039+0000","last_active":"2024-03-19T02:06:24.608993+0000","last_peered":"2024-03-19T02:06:24.608993+0000","last_clean":"2024-03-19T02:06:24.608993+0000","last_became_active":"2024-03-19T01:51:06.982133+0000","last_became_peered":"2024-03-19T01:51:06.982133+0000","last_unstale":"2024-03-19T02:06:24.608993+0000","last_undegraded":"2024-03-19T02:06:24.608993+0000","last_fullsized":"2024-03-19T02:06:24.608993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'7","last_scrub_stamp":"2024-03-19T02:06:24.608898+0000","last_deep_scrub":"35'7","last_deep_scrub_stamp":"2024-03-19T02:06:24.608898+0000","last_clean_scrub_stamp":"2024-03-19T02:06:24.608898+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:33:16.884446+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:31.127607+0000","last_change":"2024-03-19T02:06:31.127644+0000","last_active":"2024-03-19T02:06:31.127607+0000","last_peered":"2024-03-19T02:06:31.127607+0000","last_clean":"2024-03-19T02:06:31.127607+0000","last_became_active":"2024-03-19T01:51:16.578040+0000","last_became_peered":"2024-03-19T01:51:16.578040+0000","last_unstale":"2024-03-19T02:06:31.127607+0000","last_undegraded":"2024-03-19T02:06:31.127607+0000","last_fullsized":"2024-03-19T02:06:31.127607+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:31.127506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:31.127506+0000","last_clean_scrub_stamp":"2024-03-19T02:06:31.127506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:29.520168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:29.065598+0000","last_change":"2024-03-19T02:06:29.065640+0000","last_active":"2024-03-19T02:06:29.065598+0000","last_peered":"2024-03-19T02:06:29.065598+0000","last_clean":"2024-03-19T02:06:29.065598+0000","last_became_active":"2024-03-19T01:51:09.180326+0000","last_became_peered":"2024-03-19T01:51:09.180326+0000","last_unstale":"2024-03-19T02:06:29.065598+0000","last_undegraded":"2024-03-19T02:06:29.065598+0000","last_fullsized":"2024-03-19T02:06:29.065598+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:29.065481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:29.065481+0000","last_clean_scrub_stamp":"2024-03-19T02:06:29.065481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:08.114344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"29'1","reported_seq":58,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:21.432269+0000","last_change":"2024-03-19T02:06:21.432380+0000","last_active":"2024-03-19T02:06:21.432269+0000","last_peered":"2024-03-19T02:06:21.432269+0000","last_clean":"2024-03-19T02:06:21.432269+0000","last_became_active":"2024-03-19T01:51:06.961862+0000","last_became_peered":"2024-03-19T01:51:06.961862+0000","last_unstale":"2024-03-19T02:06:21.432269+0000","last_undegraded":"2024-03-19T02:06:21.432269+0000","last_fullsized":"2024-03-19T02:06:21.432269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'1","last_scrub_stamp":"2024-03-19T02:06:21.431961+0000","last_deep_scrub":"29'1","last_deep_scrub_stamp":"2024-03-19T02:06:21.431961+0000","last_clean_scrub_stamp":"2024-03-19T02:06:21.431961+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:31:08.421993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:31.374290+0000","last_change":"2024-03-19T02:06:31.374357+0000","last_active":"2024-03-19T02:06:31.374290+0000","last_peered":"2024-03-19T02:06:31.374290+0000","last_clean":"2024-03-19T02:06:31.374290+0000","last_became_active":"2024-03-19T01:51:16.576955+0000","last_became_peered":"2024-03-19T01:51:16.576955+0000","last_unstale":"2024-03-19T02:06:31.374290+0000","last_undegraded":"2024-03-19T02:06:31.374290+0000","last_fullsized":"2024-03-19T02:06:31.374290+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:31.374107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:31.374107+0000","last_clean_scrub_stamp":"2024-03-19T02:06:31.374107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:29.766769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:21.271255+0000","last_change":"2024-03-19T02:06:21.271307+0000","last_active":"2024-03-19T02:06:21.271255+0000","last_peered":"2024-03-19T02:06:21.271255+0000","last_clean":"2024-03-19T02:06:21.271255+0000","last_became_active":"2024-03-19T01:51:09.176780+0000","last_became_peered":"2024-03-19T01:51:09.176780+0000","last_unstale":"2024-03-19T02:06:21.271255+0000","last_undegraded":"2024-03-19T02:06:21.271255+0000","last_fullsized":"2024-03-19T02:06:21.271255+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:21.271153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:21.271153+0000","last_clean_scrub_stamp":"2024-03-19T02:06:21.271153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:21.414541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"35'7","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:22.974179+0000","last_change":"2024-03-19T02:06:22.974223+0000","last_active":"2024-03-19T02:06:22.974179+0000","last_peered":"2024-03-19T02:06:22.974179+0000","last_clean":"2024-03-19T02:06:22.974179+0000","last_became_active":"2024-03-19T01:51:06.955529+0000","last_became_peered":"2024-03-19T01:51:06.955529+0000","last_unstale":"2024-03-19T02:06:22.974179+0000","last_undegraded":"2024-03-19T02:06:22.974179+0000","last_fullsized":"2024-03-19T02:06:22.974179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'7","last_scrub_stamp":"2024-03-19T02:06:22.974085+0000","last_deep_scrub":"35'7","last_deep_scrub_stamp":"2024-03-19T02:06:22.974085+0000","last_clean_scrub_stamp":"2024-03-19T02:06:22.974085+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:06:29.753855+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:29.422596+0000","last_change":"2024-03-19T02:06:29.422716+0000","last_active":"2024-03-19T02:06:29.422596+0000","last_peered":"2024-03-19T02:06:29.422596+0000","last_clean":"2024-03-19T02:06:29.422596+0000","last_became_active":"2024-03-19T01:51:16.550115+0000","last_became_peered":"2024-03-19T01:51:16.550115+0000","last_unstale":"2024-03-19T02:06:29.422596+0000","last_undegraded":"2024-03-19T02:06:29.422596+0000","last_fullsized":"2024-03-19T02:06:29.422596+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:29.422361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:29.422361+0000","last_clean_scrub_stamp":"2024-03-19T02:06:29.422361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:35:38.891729+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:24.443618+0000","last_change":"2024-03-19T02:06:24.443653+0000","last_active":"2024-03-19T02:06:24.443618+0000","last_peered":"2024-03-19T02:06:24.443618+0000","last_clean":"2024-03-19T02:06:24.443618+0000","last_became_active":"2024-03-19T01:51:09.155014+0000","last_became_peered":"2024-03-19T01:51:09.155014+0000","last_unstale":"2024-03-19T02:06:24.443618+0000","last_undegraded":"2024-03-19T02:06:24.443618+0000","last_fullsized":"2024-03-19T02:06:24.443618+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:24.443511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:24.443511+0000","last_clean_scrub_stamp":"2024-03-19T02:06:24.443511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:57:43.214959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"35'9","reported_seq":66,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:19.874066+0000","last_change":"2024-03-19T02:06:19.874106+0000","last_active":"2024-03-19T02:06:19.874066+0000","last_peered":"2024-03-19T02:06:19.874066+0000","last_clean":"2024-03-19T02:06:19.874066+0000","last_became_active":"2024-03-19T01:51:06.954637+0000","last_became_peered":"2024-03-19T01:51:06.954637+0000","last_unstale":"2024-03-19T02:06:19.874066+0000","last_undegraded":"2024-03-19T02:06:19.874066+0000","last_fullsized":"2024-03-19T02:06:19.874066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9","last_scrub_stamp":"2024-03-19T02:06:19.873972+0000","last_deep_scrub":"35'9","last_deep_scrub_stamp":"2024-03-19T02:06:19.873972+0000","last_clean_scrub_stamp":"2024-03-19T02:06:19.873972+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:11.223479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:30.409910+0000","last_change":"2024-03-19T02:06:30.409955+0000","last_active":"2024-03-19T02:06:30.409910+0000","last_peered":"2024-03-19T02:06:30.409910+0000","last_clean":"2024-03-19T02:06:30.409910+0000","last_became_active":"2024-03-19T01:51:16.574575+0000","last_became_peered":"2024-03-19T01:51:16.574575+0000","last_unstale":"2024-03-19T02:06:30.409910+0000","last_undegraded":"2024-03-19T02:06:30.409910+0000","last_fullsized":"2024-03-19T02:06:30.409910+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:30.409799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:30.409799+0000","last_clean_scrub_stamp":"2024-03-19T02:06:30.409799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:55:22.158974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:27.038404+0000","last_change":"2024-03-19T02:06:27.038549+0000","last_active":"2024-03-19T02:06:27.038404+0000","last_peered":"2024-03-19T02:06:27.038404+0000","last_clean":"2024-03-19T02:06:27.038404+0000","last_became_active":"2024-03-19T01:51:09.192280+0000","last_became_peered":"2024-03-19T01:51:09.192280+0000","last_unstale":"2024-03-19T02:06:27.038404+0000","last_undegraded":"2024-03-19T02:06:27.038404+0000","last_fullsized":"2024-03-19T02:06:27.038404+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:27.038199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:27.038199+0000","last_clean_scrub_stamp":"2024-03-19T02:06:27.038199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:43:00.319633+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"35'6","reported_seq":63,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:27.564208+0000","last_change":"2024-03-19T02:06:27.564251+0000","last_active":"2024-03-19T02:06:27.564208+0000","last_peered":"2024-03-19T02:06:27.564208+0000","last_clean":"2024-03-19T02:06:27.564208+0000","last_became_active":"2024-03-19T01:51:06.985965+0000","last_became_peered":"2024-03-19T01:51:06.985965+0000","last_unstale":"2024-03-19T02:06:27.564208+0000","last_undegraded":"2024-03-19T02:06:27.564208+0000","last_fullsized":"2024-03-19T02:06:27.564208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'6","last_scrub_stamp":"2024-03-19T02:06:27.564139+0000","last_deep_scrub":"35'6","last_deep_scrub_stamp":"2024-03-19T02:06:27.564139+0000","last_clean_scrub_stamp":"2024-03-19T02:06:27.564139+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:18.062759+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:28.062420+0000","last_change":"2024-03-19T02:06:28.062420+0000","last_active":"2024-03-19T02:06:28.062420+0000","last_peered":"2024-03-19T02:06:28.062420+0000","last_clean":"2024-03-19T02:06:28.062420+0000","last_became_active":"2024-03-19T01:51:16.504908+0000","last_became_peered":"2024-03-19T01:51:16.504908+0000","last_unstale":"2024-03-19T02:06:28.062420+0000","last_undegraded":"2024-03-19T02:06:28.062420+0000","last_fullsized":"2024-03-19T02:06:28.062420+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:28.062276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:28.062276+0000","last_clean_scrub_stamp":"2024-03-19T02:06:28.062276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:07.092891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:19.423893+0000","last_change":"2024-03-19T02:06:19.423943+0000","last_active":"2024-03-19T02:06:19.423893+0000","last_peered":"2024-03-19T02:06:19.423893+0000","last_clean":"2024-03-19T02:06:19.423893+0000","last_became_active":"2024-03-19T01:51:09.179235+0000","last_became_peered":"2024-03-19T01:51:09.179235+0000","last_unstale":"2024-03-19T02:06:19.423893+0000","last_undegraded":"2024-03-19T02:06:19.423893+0000","last_fullsized":"2024-03-19T02:06:19.423893+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:19.423733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:19.423733+0000","last_clean_scrub_stamp":"2024-03-19T02:06:19.423733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:36:34.853974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"35'11","reported_seq":69,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:21.983610+0000","last_change":"2024-03-19T02:06:21.983659+0000","last_active":"2024-03-19T02:06:21.983610+0000","last_peered":"2024-03-19T02:06:21.983610+0000","last_clean":"2024-03-19T02:06:21.983610+0000","last_became_active":"2024-03-19T01:51:06.952898+0000","last_became_peered":"2024-03-19T01:51:06.952898+0000","last_unstale":"2024-03-19T02:06:21.983610+0000","last_undegraded":"2024-03-19T02:06:21.983610+0000","last_fullsized":"2024-03-19T02:06:21.983610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'11","last_scrub_stamp":"2024-03-19T02:06:21.983515+0000","last_deep_scrub":"35'11","last_deep_scrub_stamp":"2024-03-19T02:06:21.983515+0000","last_clean_scrub_stamp":"2024-03-19T02:06:21.983515+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:17:09.673112+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"35'119","reported_seq":215,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:23.569961+0000","last_change":"2024-03-19T02:06:23.569961+0000","last_active":"2024-03-19T02:06:23.569961+0000","last_peered":"2024-03-19T02:06:23.569961+0000","last_clean":"2024-03-19T02:06:23.569961+0000","last_became_active":"2024-03-19T01:50:26.095496+0000","last_became_peered":"2024-03-19T01:50:26.095496+0000","last_unstale":"2024-03-19T02:06:23.569961+0000","last_undegraded":"2024-03-19T02:06:23.569961+0000","last_fullsized":"2024-03-19T02:06:23.569961+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'119","last_scrub_stamp":"2024-03-19T02:06:23.569806+0000","last_deep_scrub":"35'119","last_deep_scrub_stamp":"2024-03-19T02:06:23.569806+0000","last_clean_scrub_stamp":"2024-03-19T02:06:23.569806+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:31:10.559838+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":49,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:35.281242+0000","last_change":"2024-03-19T02:06:35.281284+0000","last_active":"2024-03-19T02:06:35.281242+0000","last_peered":"2024-03-19T02:06:35.281242+0000","last_clean":"2024-03-19T02:06:35.281242+0000","last_became_active":"2024-03-19T01:52:14.332547+0000","last_became_peered":"2024-03-19T01:52:14.332547+0000","last_unstale":"2024-03-19T02:06:35.281242+0000","last_undegraded":"2024-03-19T02:06:35.281242+0000","last_fullsized":"2024-03-19T02:06:35.281242+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:35.281114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:35.281114+0000","last_clean_scrub_stamp":"2024-03-19T02:06:35.281114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:14.311729+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4,7],"acting":[6,0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:37.524649+0000","last_change":"2024-03-19T02:06:37.524685+0000","last_active":"2024-03-19T02:06:37.524649+0000","last_peered":"2024-03-19T02:06:37.524649+0000","last_clean":"2024-03-19T02:06:37.524649+0000","last_became_active":"2024-03-19T01:51:09.196093+0000","last_became_peered":"2024-03-19T01:51:09.196093+0000","last_unstale":"2024-03-19T02:06:37.524649+0000","last_undegraded":"2024-03-19T02:06:37.524649+0000","last_fullsized":"2024-03-19T02:06:37.524649+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:37.524550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:37.524550+0000","last_clean_scrub_stamp":"2024-03-19T02:06:37.524550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:09:34.396521+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"35'8","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:18.842474+0000","last_change":"2024-03-19T02:06:18.842513+0000","last_active":"2024-03-19T02:06:18.842474+0000","last_peered":"2024-03-19T02:06:18.842474+0000","last_clean":"2024-03-19T02:06:18.842474+0000","last_became_active":"2024-03-19T01:51:06.949234+0000","last_became_peered":"2024-03-19T01:51:06.949234+0000","last_unstale":"2024-03-19T02:06:18.842474+0000","last_undegraded":"2024-03-19T02:06:18.842474+0000","last_fullsized":"2024-03-19T02:06:18.842474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'8","last_scrub_stamp":"2024-03-19T02:06:18.842379+0000","last_deep_scrub":"35'8","last_deep_scrub_stamp":"2024-03-19T02:06:18.842379+0000","last_clean_scrub_stamp":"2024-03-19T02:06:18.842379+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:40:26.699462+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:33.285479+0000","last_change":"2024-03-19T02:06:33.285479+0000","last_active":"2024-03-19T02:06:33.285479+0000","last_peered":"2024-03-19T02:06:33.285479+0000","last_clean":"2024-03-19T02:06:33.285479+0000","last_became_active":"2024-03-19T01:51:16.553595+0000","last_became_peered":"2024-03-19T01:51:16.553595+0000","last_unstale":"2024-03-19T02:06:33.285479+0000","last_undegraded":"2024-03-19T02:06:33.285479+0000","last_fullsized":"2024-03-19T02:06:33.285479+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:33.285316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:33.285316+0000","last_clean_scrub_stamp":"2024-03-19T02:06:33.285316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:23.783936+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:24.888086+0000","last_change":"2024-03-19T02:06:24.888122+0000","last_active":"2024-03-19T02:06:24.888086+0000","last_peered":"2024-03-19T02:06:24.888086+0000","last_clean":"2024-03-19T02:06:24.888086+0000","last_became_active":"2024-03-19T01:52:12.918458+0000","last_became_peered":"2024-03-19T01:52:12.918458+0000","last_unstale":"2024-03-19T02:06:24.888086+0000","last_undegraded":"2024-03-19T02:06:24.888086+0000","last_fullsized":"2024-03-19T02:06:24.888086+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:24.887985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:24.887985+0000","last_clean_scrub_stamp":"2024-03-19T02:06:24.887985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:44.778135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'4","reported_seq":60,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:18.401067+0000","last_change":"2024-03-19T02:06:18.401112+0000","last_active":"2024-03-19T02:06:18.401067+0000","last_peered":"2024-03-19T02:06:18.401067+0000","last_clean":"2024-03-19T02:06:18.401067+0000","last_became_active":"2024-03-19T01:51:06.937580+0000","last_became_peered":"2024-03-19T01:51:06.937580+0000","last_unstale":"2024-03-19T02:06:18.401067+0000","last_undegraded":"2024-03-19T02:06:18.401067+0000","last_fullsized":"2024-03-19T02:06:18.401067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4","last_scrub_stamp":"2024-03-19T02:06:18.400964+0000","last_deep_scrub":"35'4","last_deep_scrub_stamp":"2024-03-19T02:06:18.400964+0000","last_clean_scrub_stamp":"2024-03-19T02:06:18.400964+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:16:43.147574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:32.151923+0000","last_change":"2024-03-19T02:06:32.151985+0000","last_active":"2024-03-19T02:06:32.151923+0000","last_peered":"2024-03-19T02:06:32.151923+0000","last_clean":"2024-03-19T02:06:32.151923+0000","last_became_active":"2024-03-19T01:51:16.581746+0000","last_became_peered":"2024-03-19T01:51:16.581746+0000","last_unstale":"2024-03-19T02:06:32.151923+0000","last_undegraded":"2024-03-19T02:06:32.151923+0000","last_fullsized":"2024-03-19T02:06:32.151923+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:32.151724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:32.151724+0000","last_clean_scrub_stamp":"2024-03-19T02:06:32.151724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:07.525601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:34.543962+0000","last_change":"2024-03-19T02:06:34.543999+0000","last_active":"2024-03-19T02:06:34.543962+0000","last_peered":"2024-03-19T02:06:34.543962+0000","last_clean":"2024-03-19T02:06:34.543962+0000","last_became_active":"2024-03-19T01:51:09.194645+0000","last_became_peered":"2024-03-19T01:51:09.194645+0000","last_unstale":"2024-03-19T02:06:34.543962+0000","last_undegraded":"2024-03-19T02:06:34.543962+0000","last_fullsized":"2024-03-19T02:06:34.543962+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:34.543860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:34.543860+0000","last_clean_scrub_stamp":"2024-03-19T02:06:34.543860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:55.099052+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"35'7","reported_seq":63,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:20.900585+0000","last_change":"2024-03-19T02:06:20.900634+0000","last_active":"2024-03-19T02:06:20.900585+0000","last_peered":"2024-03-19T02:06:20.900585+0000","last_clean":"2024-03-19T02:06:20.900585+0000","last_became_active":"2024-03-19T01:51:06.954951+0000","last_became_peered":"2024-03-19T01:51:06.954951+0000","last_unstale":"2024-03-19T02:06:20.900585+0000","last_undegraded":"2024-03-19T02:06:20.900585+0000","last_fullsized":"2024-03-19T02:06:20.900585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'7","last_scrub_stamp":"2024-03-19T02:06:20.900479+0000","last_deep_scrub":"35'7","last_deep_scrub_stamp":"2024-03-19T02:06:20.900479+0000","last_clean_scrub_stamp":"2024-03-19T02:06:20.900479+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:11:33.806639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:36.131519+0000","last_change":"2024-03-19T02:06:36.131557+0000","last_active":"2024-03-19T02:06:36.131519+0000","last_peered":"2024-03-19T02:06:36.131519+0000","last_clean":"2024-03-19T02:06:36.131519+0000","last_became_active":"2024-03-19T01:51:16.586298+0000","last_became_peered":"2024-03-19T01:51:16.586298+0000","last_unstale":"2024-03-19T02:06:36.131519+0000","last_undegraded":"2024-03-19T02:06:36.131519+0000","last_fullsized":"2024-03-19T02:06:36.131519+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:36.131411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:36.131411+0000","last_clean_scrub_stamp":"2024-03-19T02:06:36.131411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:38:39.575569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:35.532002+0000","last_change":"2024-03-19T02:06:35.532037+0000","last_active":"2024-03-19T02:06:35.532002+0000","last_peered":"2024-03-19T02:06:35.532002+0000","last_clean":"2024-03-19T02:06:35.532002+0000","last_became_active":"2024-03-19T01:51:09.195111+0000","last_became_peered":"2024-03-19T01:51:09.195111+0000","last_unstale":"2024-03-19T02:06:35.532002+0000","last_undegraded":"2024-03-19T02:06:35.532002+0000","last_fullsized":"2024-03-19T02:06:35.532002+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:35.531891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:35.531891+0000","last_clean_scrub_stamp":"2024-03-19T02:06:35.531891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:45.524317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"35'4","reported_seq":60,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:19.269164+0000","last_change":"2024-03-19T02:06:19.269208+0000","last_active":"2024-03-19T02:06:19.269164+0000","last_peered":"2024-03-19T02:06:19.269164+0000","last_clean":"2024-03-19T02:06:19.269164+0000","last_became_active":"2024-03-19T01:51:06.963028+0000","last_became_peered":"2024-03-19T01:51:06.963028+0000","last_unstale":"2024-03-19T02:06:19.269164+0000","last_undegraded":"2024-03-19T02:06:19.269164+0000","last_fullsized":"2024-03-19T02:06:19.269164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4","last_scrub_stamp":"2024-03-19T02:06:19.269068+0000","last_deep_scrub":"35'4","last_deep_scrub_stamp":"2024-03-19T02:06:19.269068+0000","last_clean_scrub_stamp":"2024-03-19T02:06:19.269068+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:34.129979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:29.378449+0000","last_change":"2024-03-19T02:06:29.378488+0000","last_active":"2024-03-19T02:06:29.378449+0000","last_peered":"2024-03-19T02:06:29.378449+0000","last_clean":"2024-03-19T02:06:29.378449+0000","last_became_active":"2024-03-19T01:51:16.587459+0000","last_became_peered":"2024-03-19T01:51:16.587459+0000","last_unstale":"2024-03-19T02:06:29.378449+0000","last_undegraded":"2024-03-19T02:06:29.378449+0000","last_fullsized":"2024-03-19T02:06:29.378449+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:29.378342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:29.378342+0000","last_clean_scrub_stamp":"2024-03-19T02:06:29.378342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:33:02.816515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:36.506650+0000","last_change":"2024-03-19T02:06:36.506685+0000","last_active":"2024-03-19T02:06:36.506650+0000","last_peered":"2024-03-19T02:06:36.506650+0000","last_clean":"2024-03-19T02:06:36.506650+0000","last_became_active":"2024-03-19T01:51:09.195139+0000","last_became_peered":"2024-03-19T01:51:09.195139+0000","last_unstale":"2024-03-19T02:06:36.506650+0000","last_undegraded":"2024-03-19T02:06:36.506650+0000","last_fullsized":"2024-03-19T02:06:36.506650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:36.506539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:36.506539+0000","last_clean_scrub_stamp":"2024-03-19T02:06:36.506539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:45:38.375714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"35'25","reported_seq":81,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:17.851277+0000","last_change":"2024-03-19T02:06:17.851348+0000","last_active":"2024-03-19T02:06:17.851277+0000","last_peered":"2024-03-19T02:06:17.851277+0000","last_clean":"2024-03-19T02:06:17.851277+0000","last_became_active":"2024-03-19T01:51:06.940594+0000","last_became_peered":"2024-03-19T01:51:06.940594+0000","last_unstale":"2024-03-19T02:06:17.851277+0000","last_undegraded":"2024-03-19T02:06:17.851277+0000","last_fullsized":"2024-03-19T02:06:17.851277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'25","last_scrub_stamp":"2024-03-19T02:06:17.851166+0000","last_deep_scrub":"35'25","last_deep_scrub_stamp":"2024-03-19T02:06:17.851166+0000","last_clean_scrub_stamp":"2024-03-19T02:06:17.851166+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:28:11.172347+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:30.107005+0000","last_change":"2024-03-19T02:06:30.107148+0000","last_active":"2024-03-19T02:06:30.107005+0000","last_peered":"2024-03-19T02:06:30.107005+0000","last_clean":"2024-03-19T02:06:30.107005+0000","last_became_active":"2024-03-19T01:51:16.547800+0000","last_became_peered":"2024-03-19T01:51:16.547800+0000","last_unstale":"2024-03-19T02:06:30.107005+0000","last_undegraded":"2024-03-19T02:06:30.107005+0000","last_fullsized":"2024-03-19T02:06:30.107005+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:30.106760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:30.106760+0000","last_clean_scrub_stamp":"2024-03-19T02:06:30.106760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:30:40.317630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:26.234708+0000","last_change":"2024-03-19T02:06:26.234757+0000","last_active":"2024-03-19T02:06:26.234708+0000","last_peered":"2024-03-19T02:06:26.234708+0000","last_clean":"2024-03-19T02:06:26.234708+0000","last_became_active":"2024-03-19T01:51:09.167670+0000","last_became_peered":"2024-03-19T01:51:09.167670+0000","last_unstale":"2024-03-19T02:06:26.234708+0000","last_undegraded":"2024-03-19T02:06:26.234708+0000","last_fullsized":"2024-03-19T02:06:26.234708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:26.234591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:26.234591+0000","last_clean_scrub_stamp":"2024-03-19T02:06:26.234591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:04:11.958629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"35'8","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:25.277657+0000","last_change":"2024-03-19T02:06:25.277717+0000","last_active":"2024-03-19T02:06:25.277657+0000","last_peered":"2024-03-19T02:06:25.277657+0000","last_clean":"2024-03-19T02:06:25.277657+0000","last_became_active":"2024-03-19T01:51:07.047785+0000","last_became_peered":"2024-03-19T01:51:07.047785+0000","last_unstale":"2024-03-19T02:06:25.277657+0000","last_undegraded":"2024-03-19T02:06:25.277657+0000","last_fullsized":"2024-03-19T02:06:25.277657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'8","last_scrub_stamp":"2024-03-19T02:06:25.277535+0000","last_deep_scrub":"35'8","last_deep_scrub_stamp":"2024-03-19T02:06:25.277535+0000","last_clean_scrub_stamp":"2024-03-19T02:06:25.277535+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:16:50.024145+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:35.382512+0000","last_change":"2024-03-19T02:06:35.382550+0000","last_active":"2024-03-19T02:06:35.382512+0000","last_peered":"2024-03-19T02:06:35.382512+0000","last_clean":"2024-03-19T02:06:35.382512+0000","last_became_active":"2024-03-19T01:51:16.582860+0000","last_became_peered":"2024-03-19T01:51:16.582860+0000","last_unstale":"2024-03-19T02:06:35.382512+0000","last_undegraded":"2024-03-19T02:06:35.382512+0000","last_fullsized":"2024-03-19T02:06:35.382512+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:35.382407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:35.382407+0000","last_clean_scrub_stamp":"2024-03-19T02:06:35.382407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:45:30.482728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:33.550448+0000","last_change":"2024-03-19T02:06:33.550483+0000","last_active":"2024-03-19T02:06:33.550448+0000","last_peered":"2024-03-19T02:06:33.550448+0000","last_clean":"2024-03-19T02:06:33.550448+0000","last_became_active":"2024-03-19T01:51:09.194728+0000","last_became_peered":"2024-03-19T01:51:09.194728+0000","last_unstale":"2024-03-19T02:06:33.550448+0000","last_undegraded":"2024-03-19T02:06:33.550448+0000","last_fullsized":"2024-03-19T02:06:33.550448+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:33.550341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:33.550341+0000","last_clean_scrub_stamp":"2024-03-19T02:06:33.550341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:08.924219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"35'9","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:23.987918+0000","last_change":"2024-03-19T02:06:23.987971+0000","last_active":"2024-03-19T02:06:23.987918+0000","last_peered":"2024-03-19T02:06:23.987918+0000","last_clean":"2024-03-19T02:06:23.987918+0000","last_became_active":"2024-03-19T01:51:06.955991+0000","last_became_peered":"2024-03-19T01:51:06.955991+0000","last_unstale":"2024-03-19T02:06:23.987918+0000","last_undegraded":"2024-03-19T02:06:23.987918+0000","last_fullsized":"2024-03-19T02:06:23.987918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9","last_scrub_stamp":"2024-03-19T02:06:23.987816+0000","last_deep_scrub":"35'9","last_deep_scrub_stamp":"2024-03-19T02:06:23.987816+0000","last_clean_scrub_stamp":"2024-03-19T02:06:23.987816+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:01:50.622324+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:37.346629+0000","last_change":"2024-03-19T02:06:37.346749+0000","last_active":"2024-03-19T02:06:37.346629+0000","last_peered":"2024-03-19T02:06:37.346629+0000","last_clean":"2024-03-19T02:06:37.346629+0000","last_became_active":"2024-03-19T01:52:14.331292+0000","last_became_peered":"2024-03-19T01:52:14.331292+0000","last_unstale":"2024-03-19T02:06:37.346629+0000","last_undegraded":"2024-03-19T02:06:37.346629+0000","last_fullsized":"2024-03-19T02:06:37.346629+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:37.346424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:37.346424+0000","last_clean_scrub_stamp":"2024-03-19T02:06:37.346424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:39.966045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:25.011560+0000","last_change":"2024-03-19T02:06:25.011612+0000","last_active":"2024-03-19T02:06:25.011560+0000","last_peered":"2024-03-19T02:06:25.011560+0000","last_clean":"2024-03-19T02:06:25.011560+0000","last_became_active":"2024-03-19T01:51:09.187275+0000","last_became_peered":"2024-03-19T01:51:09.187275+0000","last_unstale":"2024-03-19T02:06:25.011560+0000","last_undegraded":"2024-03-19T02:06:25.011560+0000","last_fullsized":"2024-03-19T02:06:25.011560+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:25.011380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:25.011380+0000","last_clean_scrub_stamp":"2024-03-19T02:06:25.011380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:17:12.000361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"35'15","reported_seq":71,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:16.884717+0000","last_change":"2024-03-19T02:06:16.884792+0000","last_active":"2024-03-19T02:06:16.884717+0000","last_peered":"2024-03-19T02:06:16.884717+0000","last_clean":"2024-03-19T02:06:16.884717+0000","last_became_active":"2024-03-19T01:51:06.963763+0000","last_became_peered":"2024-03-19T01:51:06.963763+0000","last_unstale":"2024-03-19T02:06:16.884717+0000","last_undegraded":"2024-03-19T02:06:16.884717+0000","last_fullsized":"2024-03-19T02:06:16.884717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'15","last_scrub_stamp":"2024-03-19T02:06:16.884516+0000","last_deep_scrub":"35'15","last_deep_scrub_stamp":"2024-03-19T02:06:16.884516+0000","last_clean_scrub_stamp":"2024-03-19T02:06:16.884516+0000","objects_scrubbed":6,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:17.027904+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:35.119679+0000","last_change":"2024-03-19T02:06:35.119716+0000","last_active":"2024-03-19T02:06:35.119679+0000","last_peered":"2024-03-19T02:06:35.119679+0000","last_clean":"2024-03-19T02:06:35.119679+0000","last_became_active":"2024-03-19T01:51:16.573744+0000","last_became_peered":"2024-03-19T01:51:16.573744+0000","last_unstale":"2024-03-19T02:06:35.119679+0000","last_undegraded":"2024-03-19T02:06:35.119679+0000","last_fullsized":"2024-03-19T02:06:35.119679+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:35.119566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:35.119566+0000","last_clean_scrub_stamp":"2024-03-19T02:06:35.119566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:38:39.689905+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:22.562311+0000","last_change":"2024-03-19T02:06:22.562349+0000","last_active":"2024-03-19T02:06:22.562311+0000","last_peered":"2024-03-19T02:06:22.562311+0000","last_clean":"2024-03-19T02:06:22.562311+0000","last_became_active":"2024-03-19T01:51:09.175898+0000","last_became_peered":"2024-03-19T01:51:09.175898+0000","last_unstale":"2024-03-19T02:06:22.562311+0000","last_undegraded":"2024-03-19T02:06:22.562311+0000","last_fullsized":"2024-03-19T02:06:22.562311+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:22.562197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:22.562197+0000","last_clean_scrub_stamp":"2024-03-19T02:06:22.562197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:31.131402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"35'20","reported_seq":76,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:21.236538+0000","last_change":"2024-03-19T02:06:21.236590+0000","last_active":"2024-03-19T02:06:21.236538+0000","last_peered":"2024-03-19T02:06:21.236538+0000","last_clean":"2024-03-19T02:06:21.236538+0000","last_became_active":"2024-03-19T01:51:07.041262+0000","last_became_peered":"2024-03-19T01:51:07.041262+0000","last_unstale":"2024-03-19T02:06:21.236538+0000","last_undegraded":"2024-03-19T02:06:21.236538+0000","last_fullsized":"2024-03-19T02:06:21.236538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'20","last_scrub_stamp":"2024-03-19T02:06:21.236435+0000","last_deep_scrub":"35'20","last_deep_scrub_stamp":"2024-03-19T02:06:21.236435+0000","last_clean_scrub_stamp":"2024-03-19T02:06:21.236435+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:00:47.854583+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:28.398093+0000","last_change":"2024-03-19T02:06:28.398138+0000","last_active":"2024-03-19T02:06:28.398093+0000","last_peered":"2024-03-19T02:06:28.398093+0000","last_clean":"2024-03-19T02:06:28.398093+0000","last_became_active":"2024-03-19T01:51:16.549347+0000","last_became_peered":"2024-03-19T01:51:16.549347+0000","last_unstale":"2024-03-19T02:06:28.398093+0000","last_undegraded":"2024-03-19T02:06:28.398093+0000","last_fullsized":"2024-03-19T02:06:28.398093+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:28.397971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:28.397971+0000","last_clean_scrub_stamp":"2024-03-19T02:06:28.397971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:07.428585+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:24.320175+0000","last_change":"2024-03-19T02:06:24.320210+0000","last_active":"2024-03-19T02:06:24.320175+0000","last_peered":"2024-03-19T02:06:24.320175+0000","last_clean":"2024-03-19T02:06:24.320175+0000","last_became_active":"2024-03-19T01:51:09.183607+0000","last_became_peered":"2024-03-19T01:51:09.183607+0000","last_unstale":"2024-03-19T02:06:24.320175+0000","last_undegraded":"2024-03-19T02:06:24.320175+0000","last_fullsized":"2024-03-19T02:06:24.320175+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:24.320061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:24.320061+0000","last_clean_scrub_stamp":"2024-03-19T02:06:24.320061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:32:21.563530+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"35'9","reported_seq":66,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:19.460598+0000","last_change":"2024-03-19T02:06:19.460645+0000","last_active":"2024-03-19T02:06:19.460598+0000","last_peered":"2024-03-19T02:06:19.460598+0000","last_clean":"2024-03-19T02:06:19.460598+0000","last_became_active":"2024-03-19T01:51:06.951700+0000","last_became_peered":"2024-03-19T01:51:06.951700+0000","last_unstale":"2024-03-19T02:06:19.460598+0000","last_undegraded":"2024-03-19T02:06:19.460598+0000","last_fullsized":"2024-03-19T02:06:19.460598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9","last_scrub_stamp":"2024-03-19T02:06:19.460499+0000","last_deep_scrub":"35'9","last_deep_scrub_stamp":"2024-03-19T02:06:19.460499+0000","last_clean_scrub_stamp":"2024-03-19T02:06:19.460499+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:23.519926+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:27.354502+0000","last_change":"2024-03-19T02:06:27.354543+0000","last_active":"2024-03-19T02:06:27.354502+0000","last_peered":"2024-03-19T02:06:27.354502+0000","last_clean":"2024-03-19T02:06:27.354502+0000","last_became_active":"2024-03-19T01:51:16.586514+0000","last_became_peered":"2024-03-19T01:51:16.586514+0000","last_unstale":"2024-03-19T02:06:27.354502+0000","last_undegraded":"2024-03-19T02:06:27.354502+0000","last_fullsized":"2024-03-19T02:06:27.354502+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:27.354385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:27.354385+0000","last_clean_scrub_stamp":"2024-03-19T02:06:27.354385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:35:49.920881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:24.018946+0000","last_change":"2024-03-19T02:06:24.018982+0000","last_active":"2024-03-19T02:06:24.018946+0000","last_peered":"2024-03-19T02:06:24.018946+0000","last_clean":"2024-03-19T02:06:24.018946+0000","last_became_active":"2024-03-19T01:51:09.171814+0000","last_became_peered":"2024-03-19T01:51:09.171814+0000","last_unstale":"2024-03-19T02:06:24.018946+0000","last_undegraded":"2024-03-19T02:06:24.018946+0000","last_fullsized":"2024-03-19T02:06:24.018946+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:24.018844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:24.018844+0000","last_clean_scrub_stamp":"2024-03-19T02:06:24.018844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:04:00.264930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"35'6","reported_seq":62,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:25.028492+0000","last_change":"2024-03-19T02:06:25.028542+0000","last_active":"2024-03-19T02:06:25.028492+0000","last_peered":"2024-03-19T02:06:25.028492+0000","last_clean":"2024-03-19T02:06:25.028492+0000","last_became_active":"2024-03-19T01:51:07.043069+0000","last_became_peered":"2024-03-19T01:51:07.043069+0000","last_unstale":"2024-03-19T02:06:25.028492+0000","last_undegraded":"2024-03-19T02:06:25.028492+0000","last_fullsized":"2024-03-19T02:06:25.028492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'6","last_scrub_stamp":"2024-03-19T02:06:25.028393+0000","last_deep_scrub":"35'6","last_deep_scrub_stamp":"2024-03-19T02:06:25.028393+0000","last_clean_scrub_stamp":"2024-03-19T02:06:25.028393+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:04:00.723861+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:34.393491+0000","last_change":"2024-03-19T02:06:34.393545+0000","last_active":"2024-03-19T02:06:34.393491+0000","last_peered":"2024-03-19T02:06:34.393491+0000","last_clean":"2024-03-19T02:06:34.393491+0000","last_became_active":"2024-03-19T01:51:16.581174+0000","last_became_peered":"2024-03-19T01:51:16.581174+0000","last_unstale":"2024-03-19T02:06:34.393491+0000","last_undegraded":"2024-03-19T02:06:34.393491+0000","last_fullsized":"2024-03-19T02:06:34.393491+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:34.393323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:34.393323+0000","last_clean_scrub_stamp":"2024-03-19T02:06:34.393323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:11.393858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:32.082636+0000","last_change":"2024-03-19T02:06:32.082671+0000","last_active":"2024-03-19T02:06:32.082636+0000","last_peered":"2024-03-19T02:06:32.082636+0000","last_clean":"2024-03-19T02:06:32.082636+0000","last_became_active":"2024-03-19T01:51:09.193656+0000","last_became_peered":"2024-03-19T01:51:09.193656+0000","last_unstale":"2024-03-19T02:06:32.082636+0000","last_undegraded":"2024-03-19T02:06:32.082636+0000","last_fullsized":"2024-03-19T02:06:32.082636+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:32.082535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:32.082535+0000","last_clean_scrub_stamp":"2024-03-19T02:06:32.082535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:11.534974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"35'7","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:24.274968+0000","last_change":"2024-03-19T02:06:24.275021+0000","last_active":"2024-03-19T02:06:24.274968+0000","last_peered":"2024-03-19T02:06:24.274968+0000","last_clean":"2024-03-19T02:06:24.274968+0000","last_became_active":"2024-03-19T01:51:07.045342+0000","last_became_peered":"2024-03-19T01:51:07.045342+0000","last_unstale":"2024-03-19T02:06:24.274968+0000","last_undegraded":"2024-03-19T02:06:24.274968+0000","last_fullsized":"2024-03-19T02:06:24.274968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'7","last_scrub_stamp":"2024-03-19T02:06:24.274855+0000","last_deep_scrub":"35'7","last_deep_scrub_stamp":"2024-03-19T02:06:24.274855+0000","last_clean_scrub_stamp":"2024-03-19T02:06:24.274855+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:00:09.027764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:33.357129+0000","last_change":"2024-03-19T02:06:33.357166+0000","last_active":"2024-03-19T02:06:33.357129+0000","last_peered":"2024-03-19T02:06:33.357129+0000","last_clean":"2024-03-19T02:06:33.357129+0000","last_became_active":"2024-03-19T01:51:16.572283+0000","last_became_peered":"2024-03-19T01:51:16.572283+0000","last_unstale":"2024-03-19T02:06:33.357129+0000","last_undegraded":"2024-03-19T02:06:33.357129+0000","last_fullsized":"2024-03-19T02:06:33.357129+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:33.357026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:33.357026+0000","last_clean_scrub_stamp":"2024-03-19T02:06:33.357026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:34.662092+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:29.113095+0000","last_change":"2024-03-19T02:06:29.113136+0000","last_active":"2024-03-19T02:06:29.113095+0000","last_peered":"2024-03-19T02:06:29.113095+0000","last_clean":"2024-03-19T02:06:29.113095+0000","last_became_active":"2024-03-19T01:51:16.533665+0000","last_became_peered":"2024-03-19T01:51:16.533665+0000","last_unstale":"2024-03-19T02:06:29.113095+0000","last_undegraded":"2024-03-19T02:06:29.113095+0000","last_fullsized":"2024-03-19T02:06:29.113095+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:29.112989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:29.112989+0000","last_clean_scrub_stamp":"2024-03-19T02:06:29.112989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:35:38.582357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:16.354419+0000","last_change":"2024-03-19T02:06:16.354453+0000","last_active":"2024-03-19T02:06:16.354419+0000","last_peered":"2024-03-19T02:06:16.354419+0000","last_clean":"2024-03-19T02:06:16.354419+0000","last_became_active":"2024-03-19T01:51:09.167626+0000","last_became_peered":"2024-03-19T01:51:09.167626+0000","last_unstale":"2024-03-19T02:06:16.354419+0000","last_undegraded":"2024-03-19T02:06:16.354419+0000","last_fullsized":"2024-03-19T02:06:16.354419+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:16.354304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:16.354304+0000","last_clean_scrub_stamp":"2024-03-19T02:06:16.354304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:26:28.867422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"35'5","reported_seq":61,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:21.022465+0000","last_change":"2024-03-19T02:06:21.022508+0000","last_active":"2024-03-19T02:06:21.022465+0000","last_peered":"2024-03-19T02:06:21.022465+0000","last_clean":"2024-03-19T02:06:21.022465+0000","last_became_active":"2024-03-19T01:51:06.950826+0000","last_became_peered":"2024-03-19T01:51:06.950826+0000","last_unstale":"2024-03-19T02:06:21.022465+0000","last_undegraded":"2024-03-19T02:06:21.022465+0000","last_fullsized":"2024-03-19T02:06:21.022465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5","last_scrub_stamp":"2024-03-19T02:06:21.022368+0000","last_deep_scrub":"35'5","last_deep_scrub_stamp":"2024-03-19T02:06:21.022368+0000","last_clean_scrub_stamp":"2024-03-19T02:06:21.022368+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:07.201424+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:34.319937+0000","last_change":"2024-03-19T02:06:34.319983+0000","last_active":"2024-03-19T02:06:34.319937+0000","last_peered":"2024-03-19T02:06:34.319937+0000","last_clean":"2024-03-19T02:06:34.319937+0000","last_became_active":"2024-03-19T01:51:16.581412+0000","last_became_peered":"2024-03-19T01:51:16.581412+0000","last_unstale":"2024-03-19T02:06:34.319937+0000","last_undegraded":"2024-03-19T02:06:34.319937+0000","last_fullsized":"2024-03-19T02:06:34.319937+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:34.319823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:34.319823+0000","last_clean_scrub_stamp":"2024-03-19T02:06:34.319823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:43:07.601257+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:25.348284+0000","last_change":"2024-03-19T02:06:25.348319+0000","last_active":"2024-03-19T02:06:25.348284+0000","last_peered":"2024-03-19T02:06:25.348284+0000","last_clean":"2024-03-19T02:06:25.348284+0000","last_became_active":"2024-03-19T01:51:09.160978+0000","last_became_peered":"2024-03-19T01:51:09.160978+0000","last_unstale":"2024-03-19T02:06:25.348284+0000","last_undegraded":"2024-03-19T02:06:25.348284+0000","last_fullsized":"2024-03-19T02:06:25.348284+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:25.348182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:25.348182+0000","last_clean_scrub_stamp":"2024-03-19T02:06:25.348182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:04:09.991288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"35'5","reported_seq":61,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:24.385489+0000","last_change":"2024-03-19T02:06:24.385534+0000","last_active":"2024-03-19T02:06:24.385489+0000","last_peered":"2024-03-19T02:06:24.385489+0000","last_clean":"2024-03-19T02:06:24.385489+0000","last_became_active":"2024-03-19T01:51:07.046886+0000","last_became_peered":"2024-03-19T01:51:07.046886+0000","last_unstale":"2024-03-19T02:06:24.385489+0000","last_undegraded":"2024-03-19T02:06:24.385489+0000","last_fullsized":"2024-03-19T02:06:24.385489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5","last_scrub_stamp":"2024-03-19T02:06:24.385386+0000","last_deep_scrub":"35'5","last_deep_scrub_stamp":"2024-03-19T02:06:24.385386+0000","last_clean_scrub_stamp":"2024-03-19T02:06:24.385386+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:43:49.506626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"35'10","reported_seq":66,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:23.235868+0000","last_change":"2024-03-19T02:06:23.235935+0000","last_active":"2024-03-19T02:06:23.235868+0000","last_peered":"2024-03-19T02:06:23.235868+0000","last_clean":"2024-03-19T02:06:23.235868+0000","last_became_active":"2024-03-19T01:51:07.044554+0000","last_became_peered":"2024-03-19T01:51:07.044554+0000","last_unstale":"2024-03-19T02:06:23.235868+0000","last_undegraded":"2024-03-19T02:06:23.235868+0000","last_fullsized":"2024-03-19T02:06:23.235868+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'10","last_scrub_stamp":"2024-03-19T02:06:23.235758+0000","last_deep_scrub":"35'10","last_deep_scrub_stamp":"2024-03-19T02:06:23.235758+0000","last_clean_scrub_stamp":"2024-03-19T02:06:23.235758+0000","objects_scrubbed":3,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:26:35.748876+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:15.879720+0000","last_change":"2024-03-19T02:06:15.879756+0000","last_active":"2024-03-19T02:06:15.879720+0000","last_peered":"2024-03-19T02:06:15.879720+0000","last_clean":"2024-03-19T02:06:15.879720+0000","last_became_active":"2024-03-19T01:51:09.184666+0000","last_became_peered":"2024-03-19T01:51:09.184666+0000","last_unstale":"2024-03-19T02:06:15.879720+0000","last_undegraded":"2024-03-19T02:06:15.879720+0000","last_fullsized":"2024-03-19T02:06:15.879720+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:15.879603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:15.879603+0000","last_clean_scrub_stamp":"2024-03-19T02:06:15.879603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:23.923689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:26.913891+0000","last_change":"2024-03-19T02:06:26.913946+0000","last_active":"2024-03-19T02:06:26.913891+0000","last_peered":"2024-03-19T02:06:26.913891+0000","last_clean":"2024-03-19T02:06:26.913891+0000","last_became_active":"2024-03-19T01:52:14.297612+0000","last_became_peered":"2024-03-19T01:52:14.297612+0000","last_unstale":"2024-03-19T02:06:26.913891+0000","last_undegraded":"2024-03-19T02:06:26.913891+0000","last_fullsized":"2024-03-19T02:06:26.913891+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:26.913710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:26.913710+0000","last_clean_scrub_stamp":"2024-03-19T02:06:26.913710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:08:32.266655+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,5],"acting":[2,0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"35'9","reported_seq":68,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:23.378780+0000","last_change":"2024-03-19T02:06:23.378829+0000","last_active":"2024-03-19T02:06:23.378780+0000","last_peered":"2024-03-19T02:06:23.378780+0000","last_clean":"2024-03-19T02:06:23.378780+0000","last_became_active":"2024-03-19T01:51:06.981704+0000","last_became_peered":"2024-03-19T01:51:06.981704+0000","last_unstale":"2024-03-19T02:06:23.378780+0000","last_undegraded":"2024-03-19T02:06:23.378780+0000","last_fullsized":"2024-03-19T02:06:23.378780+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9","last_scrub_stamp":"2024-03-19T02:06:23.378684+0000","last_deep_scrub":"35'9","last_deep_scrub_stamp":"2024-03-19T02:06:23.378684+0000","last_clean_scrub_stamp":"2024-03-19T02:06:23.378684+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:36:15.260186+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:20.246970+0000","last_change":"2024-03-19T02:06:20.247006+0000","last_active":"2024-03-19T02:06:20.246970+0000","last_peered":"2024-03-19T02:06:20.246970+0000","last_clean":"2024-03-19T02:06:20.246970+0000","last_became_active":"2024-03-19T01:51:09.183976+0000","last_became_peered":"2024-03-19T01:51:09.183976+0000","last_unstale":"2024-03-19T02:06:20.246970+0000","last_undegraded":"2024-03-19T02:06:20.246970+0000","last_fullsized":"2024-03-19T02:06:20.246970+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:20.246864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:20.246864+0000","last_clean_scrub_stamp":"2024-03-19T02:06:20.246864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:00:46.865011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:36.333558+0000","last_change":"2024-03-19T02:06:36.333595+0000","last_active":"2024-03-19T02:06:36.333558+0000","last_peered":"2024-03-19T02:06:36.333558+0000","last_clean":"2024-03-19T02:06:36.333558+0000","last_became_active":"2024-03-19T01:52:14.297976+0000","last_became_peered":"2024-03-19T01:52:14.297976+0000","last_unstale":"2024-03-19T02:06:36.333558+0000","last_undegraded":"2024-03-19T02:06:36.333558+0000","last_fullsized":"2024-03-19T02:06:36.333558+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:36.333450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:36.333450+0000","last_clean_scrub_stamp":"2024-03-19T02:06:36.333450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:05.257132+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4,2],"acting":[3,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:28.257966+0000","last_change":"2024-03-19T02:06:28.258005+0000","last_active":"2024-03-19T02:06:28.257966+0000","last_peered":"2024-03-19T02:06:28.257966+0000","last_clean":"2024-03-19T02:06:28.257966+0000","last_became_active":"2024-03-19T01:51:09.168231+0000","last_became_peered":"2024-03-19T01:51:09.168231+0000","last_unstale":"2024-03-19T02:06:28.257966+0000","last_undegraded":"2024-03-19T02:06:28.257966+0000","last_fullsized":"2024-03-19T02:06:28.257966+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:28.257856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:28.257856+0000","last_clean_scrub_stamp":"2024-03-19T02:06:28.257856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:18:17.695090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:21.903900+0000","last_change":"2024-03-19T02:06:21.904002+0000","last_active":"2024-03-19T02:06:21.903900+0000","last_peered":"2024-03-19T02:06:21.903900+0000","last_clean":"2024-03-19T02:06:21.903900+0000","last_became_active":"2024-03-19T01:51:09.154545+0000","last_became_peered":"2024-03-19T01:51:09.154545+0000","last_unstale":"2024-03-19T02:06:21.903900+0000","last_undegraded":"2024-03-19T02:06:21.903900+0000","last_fullsized":"2024-03-19T02:06:21.903900+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:21.903728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:21.903728+0000","last_clean_scrub_stamp":"2024-03-19T02:06:21.903728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:15.348802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:20.061988+0000","last_change":"2024-03-19T02:06:20.062028+0000","last_active":"2024-03-19T02:06:20.061988+0000","last_peered":"2024-03-19T02:06:20.061988+0000","last_clean":"2024-03-19T02:06:20.061988+0000","last_became_active":"2024-03-19T01:51:09.179840+0000","last_became_peered":"2024-03-19T01:51:09.179840+0000","last_unstale":"2024-03-19T02:06:20.061988+0000","last_undegraded":"2024-03-19T02:06:20.061988+0000","last_fullsized":"2024-03-19T02:06:20.061988+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:20.061877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:20.061877+0000","last_clean_scrub_stamp":"2024-03-19T02:06:20.061877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:49:51.541579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:23.056496+0000","last_change":"2024-03-19T02:06:23.056546+0000","last_active":"2024-03-19T02:06:23.056496+0000","last_peered":"2024-03-19T02:06:23.056496+0000","last_clean":"2024-03-19T02:06:23.056496+0000","last_became_active":"2024-03-19T01:51:09.163974+0000","last_became_peered":"2024-03-19T01:51:09.163974+0000","last_unstale":"2024-03-19T02:06:23.056496+0000","last_undegraded":"2024-03-19T02:06:23.056496+0000","last_fullsized":"2024-03-19T02:06:23.056496+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:23.056302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:23.056302+0000","last_clean_scrub_stamp":"2024-03-19T02:06:23.056302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:33:15.331850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:21.368625+0000","last_change":"2024-03-19T02:06:21.368625+0000","last_active":"2024-03-19T02:06:21.368625+0000","last_peered":"2024-03-19T02:06:21.368625+0000","last_clean":"2024-03-19T02:06:21.368625+0000","last_became_active":"2024-03-19T01:51:09.176474+0000","last_became_peered":"2024-03-19T01:51:09.176474+0000","last_unstale":"2024-03-19T02:06:21.368625+0000","last_undegraded":"2024-03-19T02:06:21.368625+0000","last_fullsized":"2024-03-19T02:06:21.368625+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:21.368487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:21.368487+0000","last_clean_scrub_stamp":"2024-03-19T02:06:21.368487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:19.229395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:26.381504+0000","last_change":"2024-03-19T02:06:26.381541+0000","last_active":"2024-03-19T02:06:26.381504+0000","last_peered":"2024-03-19T02:06:26.381504+0000","last_clean":"2024-03-19T02:06:26.381504+0000","last_became_active":"2024-03-19T01:51:09.167982+0000","last_became_peered":"2024-03-19T01:51:09.167982+0000","last_unstale":"2024-03-19T02:06:26.381504+0000","last_undegraded":"2024-03-19T02:06:26.381504+0000","last_fullsized":"2024-03-19T02:06:26.381504+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:26.381399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:26.381399+0000","last_clean_scrub_stamp":"2024-03-19T02:06:26.381399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:16.880019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:32.296134+0000","last_change":"2024-03-19T02:06:32.296171+0000","last_active":"2024-03-19T02:06:32.296134+0000","last_peered":"2024-03-19T02:06:32.296134+0000","last_clean":"2024-03-19T02:06:32.296134+0000","last_became_active":"2024-03-19T01:51:09.189128+0000","last_became_peered":"2024-03-19T01:51:09.189128+0000","last_unstale":"2024-03-19T02:06:32.296134+0000","last_undegraded":"2024-03-19T02:06:32.296134+0000","last_fullsized":"2024-03-19T02:06:32.296134+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:32.296025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:32.296025+0000","last_clean_scrub_stamp":"2024-03-19T02:06:32.296025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:17:19.285005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:20.415920+0000","last_change":"2024-03-19T02:06:20.415920+0000","last_active":"2024-03-19T02:06:20.415920+0000","last_peered":"2024-03-19T02:06:20.415920+0000","last_clean":"2024-03-19T02:06:20.415920+0000","last_became_active":"2024-03-19T01:51:09.183417+0000","last_became_peered":"2024-03-19T01:51:09.183417+0000","last_unstale":"2024-03-19T02:06:20.415920+0000","last_undegraded":"2024-03-19T02:06:20.415920+0000","last_fullsized":"2024-03-19T02:06:20.415920+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:20.415782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:20.415782+0000","last_clean_scrub_stamp":"2024-03-19T02:06:20.415782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:51:05.370550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:27.044189+0000","last_change":"2024-03-19T02:06:27.044230+0000","last_active":"2024-03-19T02:06:27.044189+0000","last_peered":"2024-03-19T02:06:27.044189+0000","last_clean":"2024-03-19T02:06:27.044189+0000","last_became_active":"2024-03-19T01:51:09.165872+0000","last_became_peered":"2024-03-19T01:51:09.165872+0000","last_unstale":"2024-03-19T02:06:27.044189+0000","last_undegraded":"2024-03-19T02:06:27.044189+0000","last_fullsized":"2024-03-19T02:06:27.044189+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:27.044082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:27.044082+0000","last_clean_scrub_stamp":"2024-03-19T02:06:27.044082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:03.468351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:23.875122+0000","last_change":"2024-03-19T02:06:23.875179+0000","last_active":"2024-03-19T02:06:23.875122+0000","last_peered":"2024-03-19T02:06:23.875122+0000","last_clean":"2024-03-19T02:06:23.875122+0000","last_became_active":"2024-03-19T01:52:12.916608+0000","last_became_peered":"2024-03-19T01:52:12.916608+0000","last_unstale":"2024-03-19T02:06:23.875122+0000","last_undegraded":"2024-03-19T02:06:23.875122+0000","last_fullsized":"2024-03-19T02:06:23.875122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:23.874947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:23.874947+0000","last_clean_scrub_stamp":"2024-03-19T02:06:23.874947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:59:16.698826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:33.064312+0000","last_change":"2024-03-19T02:06:33.064403+0000","last_active":"2024-03-19T02:06:33.064312+0000","last_peered":"2024-03-19T02:06:33.064312+0000","last_clean":"2024-03-19T02:06:33.064312+0000","last_became_active":"2024-03-19T01:51:09.194536+0000","last_became_peered":"2024-03-19T01:51:09.194536+0000","last_unstale":"2024-03-19T02:06:33.064312+0000","last_undegraded":"2024-03-19T02:06:33.064312+0000","last_fullsized":"2024-03-19T02:06:33.064312+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:33.064112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:33.064112+0000","last_clean_scrub_stamp":"2024-03-19T02:06:33.064112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:14:14.631457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:25.988344+0000","last_change":"2024-03-19T02:06:25.988382+0000","last_active":"2024-03-19T02:06:25.988344+0000","last_peered":"2024-03-19T02:06:25.988344+0000","last_clean":"2024-03-19T02:06:25.988344+0000","last_became_active":"2024-03-19T01:51:09.188565+0000","last_became_peered":"2024-03-19T01:51:09.188565+0000","last_unstale":"2024-03-19T02:06:25.988344+0000","last_undegraded":"2024-03-19T02:06:25.988344+0000","last_fullsized":"2024-03-19T02:06:25.988344+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:25.988238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:25.988238+0000","last_clean_scrub_stamp":"2024-03-19T02:06:25.988238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:16.486859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:27.265109+0000","last_change":"2024-03-19T02:06:27.265146+0000","last_active":"2024-03-19T02:06:27.265109+0000","last_peered":"2024-03-19T02:06:27.265109+0000","last_clean":"2024-03-19T02:06:27.265109+0000","last_became_active":"2024-03-19T01:51:09.167812+0000","last_became_peered":"2024-03-19T01:51:09.167812+0000","last_unstale":"2024-03-19T02:06:27.265109+0000","last_undegraded":"2024-03-19T02:06:27.265109+0000","last_fullsized":"2024-03-19T02:06:27.265109+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:27.265006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:27.265006+0000","last_clean_scrub_stamp":"2024-03-19T02:06:27.265006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:35.834211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:33.366731+0000","last_change":"2024-03-19T02:06:33.366770+0000","last_active":"2024-03-19T02:06:33.366731+0000","last_peered":"2024-03-19T02:06:33.366731+0000","last_clean":"2024-03-19T02:06:33.366731+0000","last_became_active":"2024-03-19T01:52:12.915089+0000","last_became_peered":"2024-03-19T01:52:12.915089+0000","last_unstale":"2024-03-19T02:06:33.366731+0000","last_undegraded":"2024-03-19T02:06:33.366731+0000","last_fullsized":"2024-03-19T02:06:33.366731+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:33.366634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:33.366634+0000","last_clean_scrub_stamp":"2024-03-19T02:06:33.366634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:53.921826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:22.911264+0000","last_change":"2024-03-19T02:06:22.911299+0000","last_active":"2024-03-19T02:06:22.911264+0000","last_peered":"2024-03-19T02:06:22.911264+0000","last_clean":"2024-03-19T02:06:22.911264+0000","last_became_active":"2024-03-19T01:51:09.167744+0000","last_became_peered":"2024-03-19T01:51:09.167744+0000","last_unstale":"2024-03-19T02:06:22.911264+0000","last_undegraded":"2024-03-19T02:06:22.911264+0000","last_fullsized":"2024-03-19T02:06:22.911264+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:22.911160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:22.911160+0000","last_clean_scrub_stamp":"2024-03-19T02:06:22.911160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:21:26.132056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:34.104437+0000","last_change":"2024-03-19T02:06:34.104473+0000","last_active":"2024-03-19T02:06:34.104437+0000","last_peered":"2024-03-19T02:06:34.104437+0000","last_clean":"2024-03-19T02:06:34.104437+0000","last_became_active":"2024-03-19T01:51:09.195488+0000","last_became_peered":"2024-03-19T01:51:09.195488+0000","last_unstale":"2024-03-19T02:06:34.104437+0000","last_undegraded":"2024-03-19T02:06:34.104437+0000","last_fullsized":"2024-03-19T02:06:34.104437+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:34.104330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:34.104330+0000","last_clean_scrub_stamp":"2024-03-19T02:06:34.104330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:50:55.504189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:31.124219+0000","last_change":"2024-03-19T02:06:31.124257+0000","last_active":"2024-03-19T02:06:31.124219+0000","last_peered":"2024-03-19T02:06:31.124219+0000","last_clean":"2024-03-19T02:06:31.124219+0000","last_became_active":"2024-03-19T01:51:09.192777+0000","last_became_peered":"2024-03-19T01:51:09.192777+0000","last_unstale":"2024-03-19T02:06:31.124219+0000","last_undegraded":"2024-03-19T02:06:31.124219+0000","last_fullsized":"2024-03-19T02:06:31.124219+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:31.124115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:31.124115+0000","last_clean_scrub_stamp":"2024-03-19T02:06:31.124115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:07:23.620783+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:27.376951+0000","last_change":"2024-03-19T02:06:27.376985+0000","last_active":"2024-03-19T02:06:27.376951+0000","last_peered":"2024-03-19T02:06:27.376951+0000","last_clean":"2024-03-19T02:06:27.376951+0000","last_became_active":"2024-03-19T01:51:09.192677+0000","last_became_peered":"2024-03-19T01:51:09.192677+0000","last_unstale":"2024-03-19T02:06:27.376951+0000","last_undegraded":"2024-03-19T02:06:27.376951+0000","last_fullsized":"2024-03-19T02:06:27.376951+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:27.376850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:27.376850+0000","last_clean_scrub_stamp":"2024-03-19T02:06:27.376850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:03:58.363399+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:21.006265+0000","last_change":"2024-03-19T02:06:21.006305+0000","last_active":"2024-03-19T02:06:21.006265+0000","last_peered":"2024-03-19T02:06:21.006265+0000","last_clean":"2024-03-19T02:06:21.006265+0000","last_became_active":"2024-03-19T01:51:09.164753+0000","last_became_peered":"2024-03-19T01:51:09.164753+0000","last_unstale":"2024-03-19T02:06:21.006265+0000","last_undegraded":"2024-03-19T02:06:21.006265+0000","last_fullsized":"2024-03-19T02:06:21.006265+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:21.006153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:21.006153+0000","last_clean_scrub_stamp":"2024-03-19T02:06:21.006153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:29.575358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:26.383760+0000","last_change":"2024-03-19T02:06:26.383794+0000","last_active":"2024-03-19T02:06:26.383760+0000","last_peered":"2024-03-19T02:06:26.383760+0000","last_clean":"2024-03-19T02:06:26.383760+0000","last_became_active":"2024-03-19T01:51:09.170153+0000","last_became_peered":"2024-03-19T01:51:09.170153+0000","last_unstale":"2024-03-19T02:06:26.383760+0000","last_undegraded":"2024-03-19T02:06:26.383760+0000","last_fullsized":"2024-03-19T02:06:26.383760+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:26.383666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:26.383666+0000","last_clean_scrub_stamp":"2024-03-19T02:06:26.383666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:05.432530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:22.281933+0000","last_change":"2024-03-19T02:06:22.281968+0000","last_active":"2024-03-19T02:06:22.281933+0000","last_peered":"2024-03-19T02:06:22.281933+0000","last_clean":"2024-03-19T02:06:22.281933+0000","last_became_active":"2024-03-19T01:51:09.177996+0000","last_became_peered":"2024-03-19T01:51:09.177996+0000","last_unstale":"2024-03-19T02:06:22.281933+0000","last_undegraded":"2024-03-19T02:06:22.281933+0000","last_fullsized":"2024-03-19T02:06:22.281933+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:22.281831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:22.281831+0000","last_clean_scrub_stamp":"2024-03-19T02:06:22.281831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:26:04.561619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:29.259559+0000","last_change":"2024-03-19T02:06:29.259595+0000","last_active":"2024-03-19T02:06:29.259559+0000","last_peered":"2024-03-19T02:06:29.259559+0000","last_clean":"2024-03-19T02:06:29.259559+0000","last_became_active":"2024-03-19T01:51:09.183045+0000","last_became_peered":"2024-03-19T01:51:09.183045+0000","last_unstale":"2024-03-19T02:06:29.259559+0000","last_undegraded":"2024-03-19T02:06:29.259559+0000","last_fullsized":"2024-03-19T02:06:29.259559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:29.259450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:29.259450+0000","last_clean_scrub_stamp":"2024-03-19T02:06:29.259450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:56:21.136199+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:30.302658+0000","last_change":"2024-03-19T02:06:30.302695+0000","last_active":"2024-03-19T02:06:30.302658+0000","last_peered":"2024-03-19T02:06:30.302658+0000","last_clean":"2024-03-19T02:06:30.302658+0000","last_became_active":"2024-03-19T01:51:09.183351+0000","last_became_peered":"2024-03-19T01:51:09.183351+0000","last_unstale":"2024-03-19T02:06:30.302658+0000","last_undegraded":"2024-03-19T02:06:30.302658+0000","last_fullsized":"2024-03-19T02:06:30.302658+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:30.302550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:30.302550+0000","last_clean_scrub_stamp":"2024-03-19T02:06:30.302550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:42:50.137731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":49,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:05:43.506519+0000","last_change":"2024-03-19T01:52:12.922818+0000","last_active":"2024-03-19T02:05:43.506519+0000","last_peered":"2024-03-19T02:05:43.506519+0000","last_clean":"2024-03-19T02:05:43.506519+0000","last_became_active":"2024-03-19T01:52:12.922473+0000","last_became_peered":"2024-03-19T01:52:12.922473+0000","last_unstale":"2024-03-19T02:05:43.506519+0000","last_undegraded":"2024-03-19T02:05:43.506519+0000","last_fullsized":"2024-03-19T02:05:43.506519+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:51:07.769071+0000","last_clean_scrub_stamp":"2024-03-19T01:51:07.769071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:18.464703+0000","last_change":"2024-03-19T02:06:18.464745+0000","last_active":"2024-03-19T02:06:18.464703+0000","last_peered":"2024-03-19T02:06:18.464703+0000","last_clean":"2024-03-19T02:06:18.464703+0000","last_became_active":"2024-03-19T01:51:09.179541+0000","last_became_peered":"2024-03-19T01:51:09.179541+0000","last_unstale":"2024-03-19T02:06:18.464703+0000","last_undegraded":"2024-03-19T02:06:18.464703+0000","last_fullsized":"2024-03-19T02:06:18.464703+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:18.464592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:18.464592+0000","last_clean_scrub_stamp":"2024-03-19T02:06:18.464592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:49:49.944294+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:32.561207+0000","last_change":"2024-03-19T02:06:32.561248+0000","last_active":"2024-03-19T02:06:32.561207+0000","last_peered":"2024-03-19T02:06:32.561207+0000","last_clean":"2024-03-19T02:06:32.561207+0000","last_became_active":"2024-03-19T01:51:09.177627+0000","last_became_peered":"2024-03-19T01:51:09.177627+0000","last_unstale":"2024-03-19T02:06:32.561207+0000","last_undegraded":"2024-03-19T02:06:32.561207+0000","last_fullsized":"2024-03-19T02:06:32.561207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:32.561101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:32.561101+0000","last_clean_scrub_stamp":"2024-03-19T02:06:32.561101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:30.953763+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:30.094426+0000","last_change":"2024-03-19T02:06:30.094462+0000","last_active":"2024-03-19T02:06:30.094426+0000","last_peered":"2024-03-19T02:06:30.094426+0000","last_clean":"2024-03-19T02:06:30.094426+0000","last_became_active":"2024-03-19T01:51:09.173235+0000","last_became_peered":"2024-03-19T01:51:09.173235+0000","last_unstale":"2024-03-19T02:06:30.094426+0000","last_undegraded":"2024-03-19T02:06:30.094426+0000","last_fullsized":"2024-03-19T02:06:30.094426+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:30.094321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:30.094321+0000","last_clean_scrub_stamp":"2024-03-19T02:06:30.094321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:55:48.376679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:31.517729+0000","last_change":"2024-03-19T02:06:31.517766+0000","last_active":"2024-03-19T02:06:31.517729+0000","last_peered":"2024-03-19T02:06:31.517729+0000","last_clean":"2024-03-19T02:06:31.517729+0000","last_became_active":"2024-03-19T01:51:09.187703+0000","last_became_peered":"2024-03-19T01:51:09.187703+0000","last_unstale":"2024-03-19T02:06:31.517729+0000","last_undegraded":"2024-03-19T02:06:31.517729+0000","last_fullsized":"2024-03-19T02:06:31.517729+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:31.517625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:31.517625+0000","last_clean_scrub_stamp":"2024-03-19T02:06:31.517625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:55:23.266800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":34,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:25.888076+0000","last_change":"2024-03-19T02:06:25.888127+0000","last_active":"2024-03-19T02:06:25.888076+0000","last_peered":"2024-03-19T02:06:25.888076+0000","last_clean":"2024-03-19T02:06:25.888076+0000","last_became_active":"2024-03-19T01:52:12.918835+0000","last_became_peered":"2024-03-19T01:52:12.918835+0000","last_unstale":"2024-03-19T02:06:25.888076+0000","last_undegraded":"2024-03-19T02:06:25.888076+0000","last_fullsized":"2024-03-19T02:06:25.888076+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:25.887920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:25.887920+0000","last_clean_scrub_stamp":"2024-03-19T02:06:25.887920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:34.106758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:19.958444+0000","last_change":"2024-03-19T02:06:19.958491+0000","last_active":"2024-03-19T02:06:19.958444+0000","last_peered":"2024-03-19T02:06:19.958444+0000","last_clean":"2024-03-19T02:06:19.958444+0000","last_became_active":"2024-03-19T01:51:09.158609+0000","last_became_peered":"2024-03-19T01:51:09.158609+0000","last_unstale":"2024-03-19T02:06:19.958444+0000","last_undegraded":"2024-03-19T02:06:19.958444+0000","last_fullsized":"2024-03-19T02:06:19.958444+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:19.958320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:19.958320+0000","last_clean_scrub_stamp":"2024-03-19T02:06:19.958320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:32:17.201789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:17.378760+0000","last_change":"2024-03-19T02:06:17.378828+0000","last_active":"2024-03-19T02:06:17.378760+0000","last_peered":"2024-03-19T02:06:17.378760+0000","last_clean":"2024-03-19T02:06:17.378760+0000","last_became_active":"2024-03-19T01:51:09.165207+0000","last_became_peered":"2024-03-19T01:51:09.165207+0000","last_unstale":"2024-03-19T02:06:17.378760+0000","last_undegraded":"2024-03-19T02:06:17.378760+0000","last_fullsized":"2024-03-19T02:06:17.378760+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:17.378630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:17.378630+0000","last_clean_scrub_stamp":"2024-03-19T02:06:17.378630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:00:02.131538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:30.514667+0000","last_change":"2024-03-19T02:06:30.514702+0000","last_active":"2024-03-19T02:06:30.514667+0000","last_peered":"2024-03-19T02:06:30.514667+0000","last_clean":"2024-03-19T02:06:30.514667+0000","last_became_active":"2024-03-19T01:51:09.175682+0000","last_became_peered":"2024-03-19T01:51:09.175682+0000","last_unstale":"2024-03-19T02:06:30.514667+0000","last_undegraded":"2024-03-19T02:06:30.514667+0000","last_fullsized":"2024-03-19T02:06:30.514667+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:30.514562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:30.514562+0000","last_clean_scrub_stamp":"2024-03-19T02:06:30.514562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:35:39.983930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":209,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509639,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":1011712,"data_stored":759262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509641,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":409600,"data_stored":171426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509640,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":405504,"data_stored":168124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509641,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31576,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340264,"statfs":{"total":96636764160,"available":96604430336,"internally_reserved":0,"allocated":409600,"data_stored":168326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509640,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":1003520,"data_stored":758036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542344,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":430080,"data_stored":170726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542345,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31580,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":413696,"data_stored":171460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542344,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31580,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":413696,"data_stored":168256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:06:43.154 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-03-19T01:51:07.769071+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=1, tm_min=51, tm_sec=7, tm_wday=1, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=19, tm_hour=2, tm_min=6, tm_sec=13, tm_wday=1, tm_yday=79, tm_isdst=0) 2024-03-19T02:06:43.155 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-19T02:06:44.074 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:16:56.233 27555== Warning: unimplemented fcntl command: 1036 2024-03-19T02:06:44.077 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:16:56.236 27555== Warning: unimplemented fcntl command: 1036 2024-03-19T02:06:44.142 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:16:56.301 27555== Warning: unimplemented fcntl command: 1036 2024-03-19T02:06:44.174 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:16:56.269 27627== Warning: unimplemented fcntl command: 1036 2024-03-19T02:06:44.185 INFO:tasks.ceph.mon.b.smithi084.stderr:==00:00:16:56.280 27626== Warning: unimplemented fcntl command: 1036 2024-03-19T02:06:44.185 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:16:56.280 27627== Warning: unimplemented fcntl command: 1036 2024-03-19T02:06:44.197 INFO:tasks.ceph.mon.b.smithi084.stderr:==00:00:16:56.292 27626== Warning: unimplemented fcntl command: 1036 2024-03-19T02:06:44.248 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:16:56.343 27627== Warning: unimplemented fcntl command: 1036 2024-03-19T02:06:44.273 INFO:tasks.ceph.mon.b.smithi084.stderr:==00:00:16:56.368 27626== Warning: unimplemented fcntl command: 1036 2024-03-19T02:07:03.156 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:07:03.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.272+0000 7f964abf5640 1 Processor -- start 2024-03-19T02:07:03.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.272+0000 7f964abf5640 1 -- start start 2024-03-19T02:07:03.274 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.272+0000 7f964abf5640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f96441510f0 0x7f96440c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:07:03.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.272+0000 7f964abf5640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f96440c73c0 0x7f96440c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:07:03.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.272+0000 7f964abf5640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96440c3f60 con 0x7f964415a410 2024-03-19T02:07:03.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.272+0000 7f964abf5640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f9644070fb0 con 0x7f96441510f0 2024-03-19T02:07:03.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.272+0000 7f964abf5640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f96441514d0 con 0x7f96440c73c0 2024-03-19T02:07:03.275 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.272+0000 7f9648991640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f96441510f0 0x7f96440c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:07:03.276 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.272+0000 7f9648991640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f96441510f0 0x7f96440c61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:44758/0 (socket says 172.21.15.55:44758) 2024-03-19T02:07:03.276 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.272+0000 7f9648991640 1 -- 172.21.15.55:0/350257804 learned_addr learned my addr 172.21.15.55:0/350257804 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:07:03.276 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.276+0000 7f9649192640 1 --2- 172.21.15.55:0/350257804 >> v2:172.21.15.84:3300/0 conn(0x7f96440c73c0 0x7f96440c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:07:03.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.276+0000 7f9648991640 1 -- 172.21.15.55:0/350257804 >> v2:172.21.15.84:3300/0 conn(0x7f96440c73c0 msgr2=0x7f96440c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:07:03.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.276+0000 7f9648991640 1 --2- 172.21.15.55:0/350257804 >> v2:172.21.15.84:3300/0 conn(0x7f96440c73c0 0x7f96440c77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:07:03.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.276+0000 7f9648991640 1 -- 172.21.15.55:0/350257804 >> v1:172.21.15.84:6789/0 conn(0x7f964415a410 legacy=0x7f96440708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:07:03.279 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.276+0000 7f9648991640 1 -- 172.21.15.55:0/350257804 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96440c6870 con 0x7f96441510f0 2024-03-19T02:07:03.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.280+0000 7f9648991640 1 --2- 172.21.15.55:0/350257804 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96441510f0 0x7f96440c61c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f964415b510 tx=0x7f963801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c3ade6b68d704d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:07:03.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.284+0000 7f96437fe640 1 -- 172.21.15.55:0/350257804 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96380b5070 con 0x7f96441510f0 2024-03-19T02:07:03.288 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.284+0000 7f96437fe640 1 -- 172.21.15.55:0/350257804 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96380a8a10 con 0x7f96441510f0 2024-03-19T02:07:03.288 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.284+0000 7f96437fe640 1 -- 172.21.15.55:0/350257804 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96380a8de0 con 0x7f96441510f0 2024-03-19T02:07:03.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.288+0000 7f964abf5640 1 -- 172.21.15.55:0/350257804 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96441510f0 msgr2=0x7f96440c61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:07:03.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.288+0000 7f964abf5640 1 --2- 172.21.15.55:0/350257804 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96441510f0 0x7f96440c61c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f964415b510 tx=0x7f963801ed10 comp rx=0 tx=0).stop 2024-03-19T02:07:03.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.288+0000 7f964abf5640 1 -- 172.21.15.55:0/350257804 shutdown_connections 2024-03-19T02:07:03.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.288+0000 7f964abf5640 1 --2- 172.21.15.55:0/350257804 >> v2:172.21.15.84:3300/0 conn(0x7f96440c73c0 0x7f96440c77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:07:03.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.288+0000 7f964abf5640 1 --2- 172.21.15.55:0/350257804 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96441510f0 0x7f96440c61c0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:07:03.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.288+0000 7f964abf5640 1 -- 172.21.15.55:0/350257804 >> 172.21.15.55:0/350257804 conn(0x7f96440cc9c0 msgr2=0x7f96440ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:07:03.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.288+0000 7f964abf5640 1 -- 172.21.15.55:0/350257804 shutdown_connections 2024-03-19T02:07:03.289 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.288+0000 7f964abf5640 1 -- 172.21.15.55:0/350257804 wait complete. 2024-03-19T02:07:03.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.288+0000 7f964abf5640 1 Processor -- start 2024-03-19T02:07:03.290 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.288+0000 7f964abf5640 1 -- start start 2024-03-19T02:07:03.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.288+0000 7f964abf5640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96440c73c0 0x7f96440cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:07:03.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.288+0000 7f964abf5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96441510f0 0x7f96440cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:07:03.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.288+0000 7f964abf5640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f964415a410 0x7f96441407e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:07:03.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.288+0000 7f964abf5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f96440c44e0 con 0x7f96441510f0 2024-03-19T02:07:03.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.288+0000 7f964abf5640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f96440c4200 con 0x7f96440c73c0 2024-03-19T02:07:03.291 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.288+0000 7f964abf5640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f96440c6e30 con 0x7f964415a410 2024-03-19T02:07:03.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.288+0000 7f9643fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96441510f0 0x7f96440cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:07:03.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.288+0000 7f9648991640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96440c73c0 0x7f96440cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:07:03.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.288+0000 7f9643fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96441510f0 0x7f96440cb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:44774/0 (socket says 172.21.15.55:44774) 2024-03-19T02:07:03.292 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.288+0000 7f9643fff640 1 -- 172.21.15.55:0/1092349700 learned_addr learned my addr 172.21.15.55:0/1092349700 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:07:03.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.296+0000 7f9643fff640 1 -- 172.21.15.55:0/1092349700 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f964415a410 msgr2=0x7f96441407e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:07:03.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.296+0000 7f9643fff640 1 --2- 172.21.15.55:0/1092349700 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f964415a410 0x7f96441407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:07:03.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.296+0000 7f9643fff640 1 -- 172.21.15.55:0/1092349700 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96440c73c0 msgr2=0x7f96440cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:07:03.297 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.296+0000 7f9643fff640 1 --2- 172.21.15.55:0/1092349700 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96440c73c0 0x7f96440cbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:07:03.298 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.296+0000 7f9643fff640 1 -- 172.21.15.55:0/1092349700 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96441f16b0 con 0x7f96441510f0 2024-03-19T02:07:03.300 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.300+0000 7f9643fff640 1 --2- 172.21.15.55:0/1092349700 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96441510f0 0x7f96440cb4f0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f963000c400 tx=0x7f963000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:07:03.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.304+0000 7f96417fa640 1 -- 172.21.15.55:0/1092349700 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9630017070 con 0x7f96441510f0 2024-03-19T02:07:03.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.304+0000 7f96417fa640 1 -- 172.21.15.55:0/1092349700 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96300052d0 con 0x7f96441510f0 2024-03-19T02:07:03.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.304+0000 7f964abf5640 1 -- 172.21.15.55:0/1092349700 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96441f1990 con 0x7f96441510f0 2024-03-19T02:07:03.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.304+0000 7f96417fa640 1 -- 172.21.15.55:0/1092349700 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9630003840 con 0x7f96441510f0 2024-03-19T02:07:03.306 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.304+0000 7f964abf5640 1 -- 172.21.15.55:0/1092349700 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96441f1f50 con 0x7f96441510f0 2024-03-19T02:07:03.307 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.304+0000 7f964abf5640 1 -- 172.21.15.55:0/1092349700 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9644068a40 con 0x7f96441510f0 2024-03-19T02:07:03.338 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.336+0000 7f96417fa640 1 -- 172.21.15.55:0/1092349700 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f963001a070 con 0x7f96441510f0 2024-03-19T02:07:03.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.336+0000 7f96417fa640 1 --2- 172.21.15.55:0/1092349700 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f96280676e0 0x7f9628069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:07:03.339 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.336+0000 7f9648991640 1 --2- 172.21.15.55:0/1092349700 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f96280676e0 0x7f9628069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:07:03.340 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.336+0000 7f9648991640 1 --2- 172.21.15.55:0/1092349700 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f96280676e0 0x7f9628069ba0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f96380b2000 tx=0x7f96380aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:07:03.361 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.360+0000 7f96417fa640 1 -- 172.21.15.55:0/1092349700 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f9630053460 con 0x7f96441510f0 2024-03-19T02:07:03.677 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.676+0000 7f96417fa640 1 -- 172.21.15.55:0/1092349700 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f9630021080 con 0x7f96441510f0 2024-03-19T02:07:03.853 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.852+0000 7f964abf5640 1 -- 172.21.15.55:0/1092349700 --> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f96440cc6e0 con 0x7f96280676e0 2024-03-19T02:07:03.875 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.872+0000 7f96417fa640 1 -- 172.21.15.55:0/1092349700 <== mgr.4105 v2:172.21.15.84:6800/998533783 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332772 (secure 0 0 0) 0x7f9638111400 con 0x7f96280676e0 2024-03-19T02:07:03.879 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-19T02:07:03.881 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-19T02:07:03.882 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.876+0000 7f964abf5640 1 -- 172.21.15.55:0/1092349700 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f96280676e0 msgr2=0x7f9628069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:07:03.882 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.876+0000 7f964abf5640 1 --2- 172.21.15.55:0/1092349700 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f96280676e0 0x7f9628069ba0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f96380b2000 tx=0x7f96380aa070 comp rx=0 tx=0).stop 2024-03-19T02:07:03.882 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.876+0000 7f964abf5640 1 -- 172.21.15.55:0/1092349700 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96441510f0 msgr2=0x7f96440cb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:07:03.882 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.876+0000 7f964abf5640 1 --2- 172.21.15.55:0/1092349700 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96441510f0 0x7f96440cb4f0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f963000c400 tx=0x7f963000c8c0 comp rx=0 tx=0).stop 2024-03-19T02:07:03.882 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.876+0000 7f964abf5640 1 -- 172.21.15.55:0/1092349700 shutdown_connections 2024-03-19T02:07:03.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.876+0000 7f964abf5640 1 --2- 172.21.15.55:0/1092349700 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f96280676e0 0x7f9628069ba0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:07:03.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.876+0000 7f964abf5640 1 --2- 172.21.15.55:0/1092349700 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f964415a410 0x7f96441407e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:07:03.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.876+0000 7f964abf5640 1 --2- 172.21.15.55:0/1092349700 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96441510f0 0x7f96440cb4f0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:07:03.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.876+0000 7f964abf5640 1 --2- 172.21.15.55:0/1092349700 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96440c73c0 0x7f96440cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:07:03.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.876+0000 7f964abf5640 1 -- 172.21.15.55:0/1092349700 >> 172.21.15.55:0/1092349700 conn(0x7f96440cc9c0 msgr2=0x7f96440c9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:07:03.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.876+0000 7f964abf5640 1 -- 172.21.15.55:0/1092349700 shutdown_connections 2024-03-19T02:07:03.883 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:03.876+0000 7f964abf5640 1 -- 172.21.15.55:0/1092349700 wait complete. 2024-03-19T02:07:03.895 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2024-03-19T02:07:02.620540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":368,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":406,"ondisk_log_size":406,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":241320,"kb_used_data":4392,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754733400,"statfs":{"total":773094113280,"available":772847001600,"internally_reserved":0,"allocated":4497408,"data_stored":2535616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":242535986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003526"},"pg_stats":[{"pgid":"2.1f","version":"35'19","reported_seq":76,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:17.916642+0000","last_change":"2024-03-19T02:06:17.916673+0000","last_active":"2024-03-19T02:06:17.916642+0000","last_peered":"2024-03-19T02:06:17.916642+0000","last_clean":"2024-03-19T02:06:17.916642+0000","last_became_active":"2024-03-19T01:51:06.981802+0000","last_became_peered":"2024-03-19T01:51:06.981802+0000","last_unstale":"2024-03-19T02:06:17.916642+0000","last_undegraded":"2024-03-19T02:06:17.916642+0000","last_fullsized":"2024-03-19T02:06:17.916642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'19","last_scrub_stamp":"2024-03-19T02:06:17.916580+0000","last_deep_scrub":"35'19","last_deep_scrub_stamp":"2024-03-19T02:06:17.916580+0000","last_clean_scrub_stamp":"2024-03-19T02:06:17.916580+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:26:00.196368+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:27.349806+0000","last_change":"2024-03-19T02:06:27.349841+0000","last_active":"2024-03-19T02:06:27.349806+0000","last_peered":"2024-03-19T02:06:27.349806+0000","last_clean":"2024-03-19T02:06:27.349806+0000","last_became_active":"2024-03-19T01:51:09.171753+0000","last_became_peered":"2024-03-19T01:51:09.171753+0000","last_unstale":"2024-03-19T02:06:27.349806+0000","last_undegraded":"2024-03-19T02:06:27.349806+0000","last_fullsized":"2024-03-19T02:06:27.349806+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:27.349699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:27.349699+0000","last_clean_scrub_stamp":"2024-03-19T02:06:27.349699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:28:10.213189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:32.371974+0000","last_change":"2024-03-19T02:06:32.372012+0000","last_active":"2024-03-19T02:06:32.371974+0000","last_peered":"2024-03-19T02:06:32.371974+0000","last_clean":"2024-03-19T02:06:32.371974+0000","last_became_active":"2024-03-19T01:51:16.534200+0000","last_became_peered":"2024-03-19T01:51:16.534200+0000","last_unstale":"2024-03-19T02:06:32.371974+0000","last_undegraded":"2024-03-19T02:06:32.371974+0000","last_fullsized":"2024-03-19T02:06:32.371974+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:32.371880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:32.371880+0000","last_clean_scrub_stamp":"2024-03-19T02:06:32.371880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:38:35.816038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":59,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:22.387238+0000","last_change":"2024-03-19T02:06:22.387260+0000","last_active":"2024-03-19T02:06:22.387238+0000","last_peered":"2024-03-19T02:06:22.387238+0000","last_clean":"2024-03-19T02:06:22.387238+0000","last_became_active":"2024-03-19T01:51:06.961856+0000","last_became_peered":"2024-03-19T01:51:06.961856+0000","last_unstale":"2024-03-19T02:06:22.387238+0000","last_undegraded":"2024-03-19T02:06:22.387238+0000","last_fullsized":"2024-03-19T02:06:22.387238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-03-19T02:06:22.387160+0000","last_deep_scrub":"29'2","last_deep_scrub_stamp":"2024-03-19T02:06:22.387160+0000","last_clean_scrub_stamp":"2024-03-19T02:06:22.387160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:43:21.444784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:18.949547+0000","last_change":"2024-03-19T02:06:18.949583+0000","last_active":"2024-03-19T02:06:18.949547+0000","last_peered":"2024-03-19T02:06:18.949547+0000","last_clean":"2024-03-19T02:06:18.949547+0000","last_became_active":"2024-03-19T01:51:09.158682+0000","last_became_peered":"2024-03-19T01:51:09.158682+0000","last_unstale":"2024-03-19T02:06:18.949547+0000","last_undegraded":"2024-03-19T02:06:18.949547+0000","last_fullsized":"2024-03-19T02:06:18.949547+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:18.949433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:18.949433+0000","last_clean_scrub_stamp":"2024-03-19T02:06:18.949433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:48:07.131245+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:32.371391+0000","last_change":"2024-03-19T02:06:32.371429+0000","last_active":"2024-03-19T02:06:32.371391+0000","last_peered":"2024-03-19T02:06:32.371391+0000","last_clean":"2024-03-19T02:06:32.371391+0000","last_became_active":"2024-03-19T01:52:14.299067+0000","last_became_peered":"2024-03-19T01:52:14.299067+0000","last_unstale":"2024-03-19T02:06:32.371391+0000","last_undegraded":"2024-03-19T02:06:32.371391+0000","last_fullsized":"2024-03-19T02:06:32.371391+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:32.371292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:32.371292+0000","last_clean_scrub_stamp":"2024-03-19T02:06:32.371292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:04:03.357841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"35'19","reported_seq":75,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:29.540168+0000","last_change":"2024-03-19T02:06:29.540228+0000","last_active":"2024-03-19T02:06:29.540168+0000","last_peered":"2024-03-19T02:06:29.540168+0000","last_clean":"2024-03-19T02:06:29.540168+0000","last_became_active":"2024-03-19T01:51:07.043209+0000","last_became_peered":"2024-03-19T01:51:07.043209+0000","last_unstale":"2024-03-19T02:06:29.540168+0000","last_undegraded":"2024-03-19T02:06:29.540168+0000","last_fullsized":"2024-03-19T02:06:29.540168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'19","last_scrub_stamp":"2024-03-19T02:06:29.540063+0000","last_deep_scrub":"35'19","last_deep_scrub_stamp":"2024-03-19T02:06:29.540063+0000","last_clean_scrub_stamp":"2024-03-19T02:06:29.540063+0000","objects_scrubbed":7,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:08.570677+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:28.034752+0000","last_change":"2024-03-19T02:06:28.034791+0000","last_active":"2024-03-19T02:06:28.034752+0000","last_peered":"2024-03-19T02:06:28.034752+0000","last_clean":"2024-03-19T02:06:28.034752+0000","last_became_active":"2024-03-19T01:51:09.173962+0000","last_became_peered":"2024-03-19T01:51:09.173962+0000","last_unstale":"2024-03-19T02:06:28.034752+0000","last_undegraded":"2024-03-19T02:06:28.034752+0000","last_fullsized":"2024-03-19T02:06:28.034752+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:28.034641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:28.034641+0000","last_clean_scrub_stamp":"2024-03-19T02:06:28.034641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:27:01.730560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:33.384750+0000","last_change":"2024-03-19T02:06:33.384790+0000","last_active":"2024-03-19T02:06:33.384750+0000","last_peered":"2024-03-19T02:06:33.384750+0000","last_clean":"2024-03-19T02:06:33.384750+0000","last_became_active":"2024-03-19T01:53:15.447002+0000","last_became_peered":"2024-03-19T01:53:15.447002+0000","last_unstale":"2024-03-19T02:06:33.384750+0000","last_undegraded":"2024-03-19T02:06:33.384750+0000","last_fullsized":"2024-03-19T02:06:33.384750+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:33.384644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:33.384644+0000","last_clean_scrub_stamp":"2024-03-19T02:06:33.384644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:12.415258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6,1],"acting":[4,2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"35'8","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:25.598633+0000","last_change":"2024-03-19T02:06:25.598684+0000","last_active":"2024-03-19T02:06:25.598633+0000","last_peered":"2024-03-19T02:06:25.598633+0000","last_clean":"2024-03-19T02:06:25.598633+0000","last_became_active":"2024-03-19T01:51:06.982309+0000","last_became_peered":"2024-03-19T01:51:06.982309+0000","last_unstale":"2024-03-19T02:06:25.598633+0000","last_undegraded":"2024-03-19T02:06:25.598633+0000","last_fullsized":"2024-03-19T02:06:25.598633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'8","last_scrub_stamp":"2024-03-19T02:06:25.598534+0000","last_deep_scrub":"35'8","last_deep_scrub_stamp":"2024-03-19T02:06:25.598534+0000","last_clean_scrub_stamp":"2024-03-19T02:06:25.598534+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:04:01.844620+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:26.033500+0000","last_change":"2024-03-19T02:06:26.033537+0000","last_active":"2024-03-19T02:06:26.033500+0000","last_peered":"2024-03-19T02:06:26.033500+0000","last_clean":"2024-03-19T02:06:26.033500+0000","last_became_active":"2024-03-19T01:51:09.159951+0000","last_became_peered":"2024-03-19T01:51:09.159951+0000","last_unstale":"2024-03-19T02:06:26.033500+0000","last_undegraded":"2024-03-19T02:06:26.033500+0000","last_fullsized":"2024-03-19T02:06:26.033500+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:26.033394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:26.033394+0000","last_clean_scrub_stamp":"2024-03-19T02:06:26.033394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:39:41.332644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:25.340674+0000","last_change":"2024-03-19T02:06:25.340715+0000","last_active":"2024-03-19T02:06:25.340674+0000","last_peered":"2024-03-19T02:06:25.340674+0000","last_clean":"2024-03-19T02:06:25.340674+0000","last_became_active":"2024-03-19T01:51:16.576256+0000","last_became_peered":"2024-03-19T01:51:16.576256+0000","last_unstale":"2024-03-19T02:06:25.340674+0000","last_undegraded":"2024-03-19T02:06:25.340674+0000","last_fullsized":"2024-03-19T02:06:25.340674+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:25.340551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:25.340551+0000","last_clean_scrub_stamp":"2024-03-19T02:06:25.340551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:29.399979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:31.291156+0000","last_change":"2024-03-19T02:06:31.291193+0000","last_active":"2024-03-19T02:06:31.291156+0000","last_peered":"2024-03-19T02:06:31.291156+0000","last_clean":"2024-03-19T02:06:31.291156+0000","last_became_active":"2024-03-19T01:51:09.185378+0000","last_became_peered":"2024-03-19T01:51:09.185378+0000","last_unstale":"2024-03-19T02:06:31.291156+0000","last_undegraded":"2024-03-19T02:06:31.291156+0000","last_fullsized":"2024-03-19T02:06:31.291156+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:31.291046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:31.291046+0000","last_clean_scrub_stamp":"2024-03-19T02:06:31.291046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:04:07.537132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"35'6","reported_seq":63,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:28.558898+0000","last_change":"2024-03-19T02:06:28.558898+0000","last_active":"2024-03-19T02:06:28.558898+0000","last_peered":"2024-03-19T02:06:28.558898+0000","last_clean":"2024-03-19T02:06:28.558898+0000","last_became_active":"2024-03-19T01:51:06.987450+0000","last_became_peered":"2024-03-19T01:51:06.987450+0000","last_unstale":"2024-03-19T02:06:28.558898+0000","last_undegraded":"2024-03-19T02:06:28.558898+0000","last_fullsized":"2024-03-19T02:06:28.558898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'6","last_scrub_stamp":"2024-03-19T02:06:28.558804+0000","last_deep_scrub":"35'6","last_deep_scrub_stamp":"2024-03-19T02:06:28.558804+0000","last_clean_scrub_stamp":"2024-03-19T02:06:28.558804+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:03:59.545353+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:26.324338+0000","last_change":"2024-03-19T02:06:26.324377+0000","last_active":"2024-03-19T02:06:26.324338+0000","last_peered":"2024-03-19T02:06:26.324338+0000","last_clean":"2024-03-19T02:06:26.324338+0000","last_became_active":"2024-03-19T01:51:16.582532+0000","last_became_peered":"2024-03-19T01:51:16.582532+0000","last_unstale":"2024-03-19T02:06:26.324338+0000","last_undegraded":"2024-03-19T02:06:26.324338+0000","last_fullsized":"2024-03-19T02:06:26.324338+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:26.324234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:26.324234+0000","last_clean_scrub_stamp":"2024-03-19T02:06:26.324234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:18:15.761468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:29.339147+0000","last_change":"2024-03-19T02:06:29.339147+0000","last_active":"2024-03-19T02:06:29.339147+0000","last_peered":"2024-03-19T02:06:29.339147+0000","last_clean":"2024-03-19T02:06:29.339147+0000","last_became_active":"2024-03-19T01:51:09.174409+0000","last_became_peered":"2024-03-19T01:51:09.174409+0000","last_unstale":"2024-03-19T02:06:29.339147+0000","last_undegraded":"2024-03-19T02:06:29.339147+0000","last_fullsized":"2024-03-19T02:06:29.339147+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:29.339008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:29.339008+0000","last_clean_scrub_stamp":"2024-03-19T02:06:29.339008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:14:10.906353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"35'5","reported_seq":63,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:20.478235+0000","last_change":"2024-03-19T02:06:20.478235+0000","last_active":"2024-03-19T02:06:20.478235+0000","last_peered":"2024-03-19T02:06:20.478235+0000","last_clean":"2024-03-19T02:06:20.478235+0000","last_became_active":"2024-03-19T01:51:06.954635+0000","last_became_peered":"2024-03-19T01:51:06.954635+0000","last_unstale":"2024-03-19T02:06:20.478235+0000","last_undegraded":"2024-03-19T02:06:20.478235+0000","last_fullsized":"2024-03-19T02:06:20.478235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5","last_scrub_stamp":"2024-03-19T02:06:20.478085+0000","last_deep_scrub":"35'5","last_deep_scrub_stamp":"2024-03-19T02:06:20.478085+0000","last_clean_scrub_stamp":"2024-03-19T02:06:20.478085+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:59:50.831717+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:30.381814+0000","last_change":"2024-03-19T02:06:30.381853+0000","last_active":"2024-03-19T02:06:30.381814+0000","last_peered":"2024-03-19T02:06:30.381814+0000","last_clean":"2024-03-19T02:06:30.381814+0000","last_became_active":"2024-03-19T01:51:16.587483+0000","last_became_peered":"2024-03-19T01:51:16.587483+0000","last_unstale":"2024-03-19T02:06:30.381814+0000","last_undegraded":"2024-03-19T02:06:30.381814+0000","last_fullsized":"2024-03-19T02:06:30.381814+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:30.381697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:30.381697+0000","last_clean_scrub_stamp":"2024-03-19T02:06:30.381697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:14:36.256253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:28.341166+0000","last_change":"2024-03-19T02:06:28.341203+0000","last_active":"2024-03-19T02:06:28.341166+0000","last_peered":"2024-03-19T02:06:28.341166+0000","last_clean":"2024-03-19T02:06:28.341166+0000","last_became_active":"2024-03-19T01:51:09.173648+0000","last_became_peered":"2024-03-19T01:51:09.173648+0000","last_unstale":"2024-03-19T02:06:28.341166+0000","last_undegraded":"2024-03-19T02:06:28.341166+0000","last_fullsized":"2024-03-19T02:06:28.341166+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:28.341061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:28.341061+0000","last_clean_scrub_stamp":"2024-03-19T02:06:28.341061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:07.793500+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"35'11","reported_seq":69,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:22.415658+0000","last_change":"2024-03-19T02:06:22.415809+0000","last_active":"2024-03-19T02:06:22.415658+0000","last_peered":"2024-03-19T02:06:22.415658+0000","last_clean":"2024-03-19T02:06:22.415658+0000","last_became_active":"2024-03-19T01:51:06.981087+0000","last_became_peered":"2024-03-19T01:51:06.981087+0000","last_unstale":"2024-03-19T02:06:22.415658+0000","last_undegraded":"2024-03-19T02:06:22.415658+0000","last_fullsized":"2024-03-19T02:06:22.415658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'11","last_scrub_stamp":"2024-03-19T02:06:22.415422+0000","last_deep_scrub":"35'11","last_deep_scrub_stamp":"2024-03-19T02:06:22.415422+0000","last_clean_scrub_stamp":"2024-03-19T02:06:22.415422+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:56:14.292172+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:32.369802+0000","last_change":"2024-03-19T02:06:32.369842+0000","last_active":"2024-03-19T02:06:32.369802+0000","last_peered":"2024-03-19T02:06:32.369802+0000","last_clean":"2024-03-19T02:06:32.369802+0000","last_became_active":"2024-03-19T01:51:16.573845+0000","last_became_peered":"2024-03-19T01:51:16.573845+0000","last_unstale":"2024-03-19T02:06:32.369802+0000","last_undegraded":"2024-03-19T02:06:32.369802+0000","last_fullsized":"2024-03-19T02:06:32.369802+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:32.369699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:32.369699+0000","last_clean_scrub_stamp":"2024-03-19T02:06:32.369699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:07.743577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:23.275620+0000","last_change":"2024-03-19T02:06:23.275657+0000","last_active":"2024-03-19T02:06:23.275620+0000","last_peered":"2024-03-19T02:06:23.275620+0000","last_clean":"2024-03-19T02:06:23.275620+0000","last_became_active":"2024-03-19T01:51:09.185970+0000","last_became_peered":"2024-03-19T01:51:09.185970+0000","last_unstale":"2024-03-19T02:06:23.275620+0000","last_undegraded":"2024-03-19T02:06:23.275620+0000","last_fullsized":"2024-03-19T02:06:23.275620+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:23.275500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:23.275500+0000","last_clean_scrub_stamp":"2024-03-19T02:06:23.275500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:48:11.457312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"35'3","reported_seq":60,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:26.576554+0000","last_change":"2024-03-19T02:06:26.576601+0000","last_active":"2024-03-19T02:06:26.576554+0000","last_peered":"2024-03-19T02:06:26.576554+0000","last_clean":"2024-03-19T02:06:26.576554+0000","last_became_active":"2024-03-19T01:51:06.983858+0000","last_became_peered":"2024-03-19T01:51:06.983858+0000","last_unstale":"2024-03-19T02:06:26.576554+0000","last_undegraded":"2024-03-19T02:06:26.576554+0000","last_fullsized":"2024-03-19T02:06:26.576554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3","last_scrub_stamp":"2024-03-19T02:06:26.576460+0000","last_deep_scrub":"35'3","last_deep_scrub_stamp":"2024-03-19T02:06:26.576460+0000","last_clean_scrub_stamp":"2024-03-19T02:06:26.576460+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:14:32.451016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:31.344299+0000","last_change":"2024-03-19T02:06:31.344340+0000","last_active":"2024-03-19T02:06:31.344299+0000","last_peered":"2024-03-19T02:06:31.344299+0000","last_clean":"2024-03-19T02:06:31.344299+0000","last_became_active":"2024-03-19T01:51:16.590968+0000","last_became_peered":"2024-03-19T01:51:16.590968+0000","last_unstale":"2024-03-19T02:06:31.344299+0000","last_undegraded":"2024-03-19T02:06:31.344299+0000","last_fullsized":"2024-03-19T02:06:31.344299+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:31.344187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:31.344187+0000","last_clean_scrub_stamp":"2024-03-19T02:06:31.344187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:21.842807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:22.035764+0000","last_change":"2024-03-19T02:06:22.035802+0000","last_active":"2024-03-19T02:06:22.035764+0000","last_peered":"2024-03-19T02:06:22.035764+0000","last_clean":"2024-03-19T02:06:22.035764+0000","last_became_active":"2024-03-19T01:51:09.168243+0000","last_became_peered":"2024-03-19T01:51:09.168243+0000","last_unstale":"2024-03-19T02:06:22.035764+0000","last_undegraded":"2024-03-19T02:06:22.035764+0000","last_fullsized":"2024-03-19T02:06:22.035764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:22.035658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:22.035658+0000","last_clean_scrub_stamp":"2024-03-19T02:06:22.035658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:31:09.025690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"35'11","reported_seq":68,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:22.264551+0000","last_change":"2024-03-19T02:06:22.264602+0000","last_active":"2024-03-19T02:06:22.264551+0000","last_peered":"2024-03-19T02:06:22.264551+0000","last_clean":"2024-03-19T02:06:22.264551+0000","last_became_active":"2024-03-19T01:51:07.042327+0000","last_became_peered":"2024-03-19T01:51:07.042327+0000","last_unstale":"2024-03-19T02:06:22.264551+0000","last_undegraded":"2024-03-19T02:06:22.264551+0000","last_fullsized":"2024-03-19T02:06:22.264551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'11","last_scrub_stamp":"2024-03-19T02:06:22.264451+0000","last_deep_scrub":"35'11","last_deep_scrub_stamp":"2024-03-19T02:06:22.264451+0000","last_clean_scrub_stamp":"2024-03-19T02:06:22.264451+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:33:46.680769+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:28.370487+0000","last_change":"2024-03-19T02:06:28.370527+0000","last_active":"2024-03-19T02:06:28.370487+0000","last_peered":"2024-03-19T02:06:28.370487+0000","last_clean":"2024-03-19T02:06:28.370487+0000","last_became_active":"2024-03-19T01:51:16.586614+0000","last_became_peered":"2024-03-19T01:51:16.586614+0000","last_unstale":"2024-03-19T02:06:28.370487+0000","last_undegraded":"2024-03-19T02:06:28.370487+0000","last_fullsized":"2024-03-19T02:06:28.370487+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:28.370364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:28.370364+0000","last_clean_scrub_stamp":"2024-03-19T02:06:28.370364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:33:20.645912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:30.346070+0000","last_change":"2024-03-19T02:06:30.346106+0000","last_active":"2024-03-19T02:06:30.346070+0000","last_peered":"2024-03-19T02:06:30.346070+0000","last_clean":"2024-03-19T02:06:30.346070+0000","last_became_active":"2024-03-19T01:51:09.186033+0000","last_became_peered":"2024-03-19T01:51:09.186033+0000","last_unstale":"2024-03-19T02:06:30.346070+0000","last_undegraded":"2024-03-19T02:06:30.346070+0000","last_fullsized":"2024-03-19T02:06:30.346070+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:30.345964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:30.345964+0000","last_clean_scrub_stamp":"2024-03-19T02:06:30.345964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:08:19.486082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"35'11","reported_seq":68,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:23.431605+0000","last_change":"2024-03-19T02:06:23.431657+0000","last_active":"2024-03-19T02:06:23.431605+0000","last_peered":"2024-03-19T02:06:23.431605+0000","last_clean":"2024-03-19T02:06:23.431605+0000","last_became_active":"2024-03-19T01:51:07.043185+0000","last_became_peered":"2024-03-19T01:51:07.043185+0000","last_unstale":"2024-03-19T02:06:23.431605+0000","last_undegraded":"2024-03-19T02:06:23.431605+0000","last_fullsized":"2024-03-19T02:06:23.431605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'11","last_scrub_stamp":"2024-03-19T02:06:23.431506+0000","last_deep_scrub":"35'11","last_deep_scrub_stamp":"2024-03-19T02:06:23.431506+0000","last_clean_scrub_stamp":"2024-03-19T02:06:23.431506+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:32:56.869679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:31.332790+0000","last_change":"2024-03-19T02:06:31.332845+0000","last_active":"2024-03-19T02:06:31.332790+0000","last_peered":"2024-03-19T02:06:31.332790+0000","last_clean":"2024-03-19T02:06:31.332790+0000","last_became_active":"2024-03-19T01:51:16.533661+0000","last_became_peered":"2024-03-19T01:51:16.533661+0000","last_unstale":"2024-03-19T02:06:31.332790+0000","last_undegraded":"2024-03-19T02:06:31.332790+0000","last_fullsized":"2024-03-19T02:06:31.332790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:31.332614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:31.332614+0000","last_clean_scrub_stamp":"2024-03-19T02:06:31.332614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:53:49.821983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:25.404579+0000","last_change":"2024-03-19T02:06:25.404618+0000","last_active":"2024-03-19T02:06:25.404579+0000","last_peered":"2024-03-19T02:06:25.404579+0000","last_clean":"2024-03-19T02:06:25.404579+0000","last_became_active":"2024-03-19T01:51:09.156336+0000","last_became_peered":"2024-03-19T01:51:09.156336+0000","last_unstale":"2024-03-19T02:06:25.404579+0000","last_undegraded":"2024-03-19T02:06:25.404579+0000","last_fullsized":"2024-03-19T02:06:25.404579+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:25.404464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:25.404464+0000","last_clean_scrub_stamp":"2024-03-19T02:06:25.404464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:14:31.279020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"35'7","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:24.608993+0000","last_change":"2024-03-19T02:06:24.609039+0000","last_active":"2024-03-19T02:06:24.608993+0000","last_peered":"2024-03-19T02:06:24.608993+0000","last_clean":"2024-03-19T02:06:24.608993+0000","last_became_active":"2024-03-19T01:51:06.982133+0000","last_became_peered":"2024-03-19T01:51:06.982133+0000","last_unstale":"2024-03-19T02:06:24.608993+0000","last_undegraded":"2024-03-19T02:06:24.608993+0000","last_fullsized":"2024-03-19T02:06:24.608993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'7","last_scrub_stamp":"2024-03-19T02:06:24.608898+0000","last_deep_scrub":"35'7","last_deep_scrub_stamp":"2024-03-19T02:06:24.608898+0000","last_clean_scrub_stamp":"2024-03-19T02:06:24.608898+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:33:16.884446+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:31.127607+0000","last_change":"2024-03-19T02:06:31.127644+0000","last_active":"2024-03-19T02:06:31.127607+0000","last_peered":"2024-03-19T02:06:31.127607+0000","last_clean":"2024-03-19T02:06:31.127607+0000","last_became_active":"2024-03-19T01:51:16.578040+0000","last_became_peered":"2024-03-19T01:51:16.578040+0000","last_unstale":"2024-03-19T02:06:31.127607+0000","last_undegraded":"2024-03-19T02:06:31.127607+0000","last_fullsized":"2024-03-19T02:06:31.127607+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:31.127506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:31.127506+0000","last_clean_scrub_stamp":"2024-03-19T02:06:31.127506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:29.520168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:29.065598+0000","last_change":"2024-03-19T02:06:29.065640+0000","last_active":"2024-03-19T02:06:29.065598+0000","last_peered":"2024-03-19T02:06:29.065598+0000","last_clean":"2024-03-19T02:06:29.065598+0000","last_became_active":"2024-03-19T01:51:09.180326+0000","last_became_peered":"2024-03-19T01:51:09.180326+0000","last_unstale":"2024-03-19T02:06:29.065598+0000","last_undegraded":"2024-03-19T02:06:29.065598+0000","last_fullsized":"2024-03-19T02:06:29.065598+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:29.065481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:29.065481+0000","last_clean_scrub_stamp":"2024-03-19T02:06:29.065481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:08.114344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"29'1","reported_seq":58,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:21.432269+0000","last_change":"2024-03-19T02:06:21.432380+0000","last_active":"2024-03-19T02:06:21.432269+0000","last_peered":"2024-03-19T02:06:21.432269+0000","last_clean":"2024-03-19T02:06:21.432269+0000","last_became_active":"2024-03-19T01:51:06.961862+0000","last_became_peered":"2024-03-19T01:51:06.961862+0000","last_unstale":"2024-03-19T02:06:21.432269+0000","last_undegraded":"2024-03-19T02:06:21.432269+0000","last_fullsized":"2024-03-19T02:06:21.432269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'1","last_scrub_stamp":"2024-03-19T02:06:21.431961+0000","last_deep_scrub":"29'1","last_deep_scrub_stamp":"2024-03-19T02:06:21.431961+0000","last_clean_scrub_stamp":"2024-03-19T02:06:21.431961+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:31:08.421993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:31.374290+0000","last_change":"2024-03-19T02:06:31.374357+0000","last_active":"2024-03-19T02:06:31.374290+0000","last_peered":"2024-03-19T02:06:31.374290+0000","last_clean":"2024-03-19T02:06:31.374290+0000","last_became_active":"2024-03-19T01:51:16.576955+0000","last_became_peered":"2024-03-19T01:51:16.576955+0000","last_unstale":"2024-03-19T02:06:31.374290+0000","last_undegraded":"2024-03-19T02:06:31.374290+0000","last_fullsized":"2024-03-19T02:06:31.374290+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:31.374107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:31.374107+0000","last_clean_scrub_stamp":"2024-03-19T02:06:31.374107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:29.766769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:21.271255+0000","last_change":"2024-03-19T02:06:21.271307+0000","last_active":"2024-03-19T02:06:21.271255+0000","last_peered":"2024-03-19T02:06:21.271255+0000","last_clean":"2024-03-19T02:06:21.271255+0000","last_became_active":"2024-03-19T01:51:09.176780+0000","last_became_peered":"2024-03-19T01:51:09.176780+0000","last_unstale":"2024-03-19T02:06:21.271255+0000","last_undegraded":"2024-03-19T02:06:21.271255+0000","last_fullsized":"2024-03-19T02:06:21.271255+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:21.271153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:21.271153+0000","last_clean_scrub_stamp":"2024-03-19T02:06:21.271153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:21.414541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"35'7","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:22.974179+0000","last_change":"2024-03-19T02:06:22.974223+0000","last_active":"2024-03-19T02:06:22.974179+0000","last_peered":"2024-03-19T02:06:22.974179+0000","last_clean":"2024-03-19T02:06:22.974179+0000","last_became_active":"2024-03-19T01:51:06.955529+0000","last_became_peered":"2024-03-19T01:51:06.955529+0000","last_unstale":"2024-03-19T02:06:22.974179+0000","last_undegraded":"2024-03-19T02:06:22.974179+0000","last_fullsized":"2024-03-19T02:06:22.974179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'7","last_scrub_stamp":"2024-03-19T02:06:22.974085+0000","last_deep_scrub":"35'7","last_deep_scrub_stamp":"2024-03-19T02:06:22.974085+0000","last_clean_scrub_stamp":"2024-03-19T02:06:22.974085+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:06:29.753855+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:29.422596+0000","last_change":"2024-03-19T02:06:29.422716+0000","last_active":"2024-03-19T02:06:29.422596+0000","last_peered":"2024-03-19T02:06:29.422596+0000","last_clean":"2024-03-19T02:06:29.422596+0000","last_became_active":"2024-03-19T01:51:16.550115+0000","last_became_peered":"2024-03-19T01:51:16.550115+0000","last_unstale":"2024-03-19T02:06:29.422596+0000","last_undegraded":"2024-03-19T02:06:29.422596+0000","last_fullsized":"2024-03-19T02:06:29.422596+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:29.422361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:29.422361+0000","last_clean_scrub_stamp":"2024-03-19T02:06:29.422361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:35:38.891729+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:24.443618+0000","last_change":"2024-03-19T02:06:24.443653+0000","last_active":"2024-03-19T02:06:24.443618+0000","last_peered":"2024-03-19T02:06:24.443618+0000","last_clean":"2024-03-19T02:06:24.443618+0000","last_became_active":"2024-03-19T01:51:09.155014+0000","last_became_peered":"2024-03-19T01:51:09.155014+0000","last_unstale":"2024-03-19T02:06:24.443618+0000","last_undegraded":"2024-03-19T02:06:24.443618+0000","last_fullsized":"2024-03-19T02:06:24.443618+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:24.443511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:24.443511+0000","last_clean_scrub_stamp":"2024-03-19T02:06:24.443511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:57:43.214959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"35'9","reported_seq":66,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:19.874066+0000","last_change":"2024-03-19T02:06:19.874106+0000","last_active":"2024-03-19T02:06:19.874066+0000","last_peered":"2024-03-19T02:06:19.874066+0000","last_clean":"2024-03-19T02:06:19.874066+0000","last_became_active":"2024-03-19T01:51:06.954637+0000","last_became_peered":"2024-03-19T01:51:06.954637+0000","last_unstale":"2024-03-19T02:06:19.874066+0000","last_undegraded":"2024-03-19T02:06:19.874066+0000","last_fullsized":"2024-03-19T02:06:19.874066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9","last_scrub_stamp":"2024-03-19T02:06:19.873972+0000","last_deep_scrub":"35'9","last_deep_scrub_stamp":"2024-03-19T02:06:19.873972+0000","last_clean_scrub_stamp":"2024-03-19T02:06:19.873972+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:11.223479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:30.409910+0000","last_change":"2024-03-19T02:06:30.409955+0000","last_active":"2024-03-19T02:06:30.409910+0000","last_peered":"2024-03-19T02:06:30.409910+0000","last_clean":"2024-03-19T02:06:30.409910+0000","last_became_active":"2024-03-19T01:51:16.574575+0000","last_became_peered":"2024-03-19T01:51:16.574575+0000","last_unstale":"2024-03-19T02:06:30.409910+0000","last_undegraded":"2024-03-19T02:06:30.409910+0000","last_fullsized":"2024-03-19T02:06:30.409910+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:30.409799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:30.409799+0000","last_clean_scrub_stamp":"2024-03-19T02:06:30.409799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:55:22.158974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:27.038404+0000","last_change":"2024-03-19T02:06:27.038549+0000","last_active":"2024-03-19T02:06:27.038404+0000","last_peered":"2024-03-19T02:06:27.038404+0000","last_clean":"2024-03-19T02:06:27.038404+0000","last_became_active":"2024-03-19T01:51:09.192280+0000","last_became_peered":"2024-03-19T01:51:09.192280+0000","last_unstale":"2024-03-19T02:06:27.038404+0000","last_undegraded":"2024-03-19T02:06:27.038404+0000","last_fullsized":"2024-03-19T02:06:27.038404+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:27.038199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:27.038199+0000","last_clean_scrub_stamp":"2024-03-19T02:06:27.038199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:43:00.319633+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"35'6","reported_seq":63,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:27.564208+0000","last_change":"2024-03-19T02:06:27.564251+0000","last_active":"2024-03-19T02:06:27.564208+0000","last_peered":"2024-03-19T02:06:27.564208+0000","last_clean":"2024-03-19T02:06:27.564208+0000","last_became_active":"2024-03-19T01:51:06.985965+0000","last_became_peered":"2024-03-19T01:51:06.985965+0000","last_unstale":"2024-03-19T02:06:27.564208+0000","last_undegraded":"2024-03-19T02:06:27.564208+0000","last_fullsized":"2024-03-19T02:06:27.564208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'6","last_scrub_stamp":"2024-03-19T02:06:27.564139+0000","last_deep_scrub":"35'6","last_deep_scrub_stamp":"2024-03-19T02:06:27.564139+0000","last_clean_scrub_stamp":"2024-03-19T02:06:27.564139+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:18.062759+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:28.062420+0000","last_change":"2024-03-19T02:06:28.062420+0000","last_active":"2024-03-19T02:06:28.062420+0000","last_peered":"2024-03-19T02:06:28.062420+0000","last_clean":"2024-03-19T02:06:28.062420+0000","last_became_active":"2024-03-19T01:51:16.504908+0000","last_became_peered":"2024-03-19T01:51:16.504908+0000","last_unstale":"2024-03-19T02:06:28.062420+0000","last_undegraded":"2024-03-19T02:06:28.062420+0000","last_fullsized":"2024-03-19T02:06:28.062420+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:28.062276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:28.062276+0000","last_clean_scrub_stamp":"2024-03-19T02:06:28.062276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:07.092891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:19.423893+0000","last_change":"2024-03-19T02:06:19.423943+0000","last_active":"2024-03-19T02:06:19.423893+0000","last_peered":"2024-03-19T02:06:19.423893+0000","last_clean":"2024-03-19T02:06:19.423893+0000","last_became_active":"2024-03-19T01:51:09.179235+0000","last_became_peered":"2024-03-19T01:51:09.179235+0000","last_unstale":"2024-03-19T02:06:19.423893+0000","last_undegraded":"2024-03-19T02:06:19.423893+0000","last_fullsized":"2024-03-19T02:06:19.423893+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:19.423733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:19.423733+0000","last_clean_scrub_stamp":"2024-03-19T02:06:19.423733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:36:34.853974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"35'11","reported_seq":69,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:21.983610+0000","last_change":"2024-03-19T02:06:21.983659+0000","last_active":"2024-03-19T02:06:21.983610+0000","last_peered":"2024-03-19T02:06:21.983610+0000","last_clean":"2024-03-19T02:06:21.983610+0000","last_became_active":"2024-03-19T01:51:06.952898+0000","last_became_peered":"2024-03-19T01:51:06.952898+0000","last_unstale":"2024-03-19T02:06:21.983610+0000","last_undegraded":"2024-03-19T02:06:21.983610+0000","last_fullsized":"2024-03-19T02:06:21.983610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'11","last_scrub_stamp":"2024-03-19T02:06:21.983515+0000","last_deep_scrub":"35'11","last_deep_scrub_stamp":"2024-03-19T02:06:21.983515+0000","last_clean_scrub_stamp":"2024-03-19T02:06:21.983515+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:17:09.673112+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"35'119","reported_seq":215,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:23.569961+0000","last_change":"2024-03-19T02:06:23.569961+0000","last_active":"2024-03-19T02:06:23.569961+0000","last_peered":"2024-03-19T02:06:23.569961+0000","last_clean":"2024-03-19T02:06:23.569961+0000","last_became_active":"2024-03-19T01:50:26.095496+0000","last_became_peered":"2024-03-19T01:50:26.095496+0000","last_unstale":"2024-03-19T02:06:23.569961+0000","last_undegraded":"2024-03-19T02:06:23.569961+0000","last_fullsized":"2024-03-19T02:06:23.569961+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'119","last_scrub_stamp":"2024-03-19T02:06:23.569806+0000","last_deep_scrub":"35'119","last_deep_scrub_stamp":"2024-03-19T02:06:23.569806+0000","last_clean_scrub_stamp":"2024-03-19T02:06:23.569806+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:31:10.559838+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":49,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:35.281242+0000","last_change":"2024-03-19T02:06:35.281284+0000","last_active":"2024-03-19T02:06:35.281242+0000","last_peered":"2024-03-19T02:06:35.281242+0000","last_clean":"2024-03-19T02:06:35.281242+0000","last_became_active":"2024-03-19T01:52:14.332547+0000","last_became_peered":"2024-03-19T01:52:14.332547+0000","last_unstale":"2024-03-19T02:06:35.281242+0000","last_undegraded":"2024-03-19T02:06:35.281242+0000","last_fullsized":"2024-03-19T02:06:35.281242+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:35.281114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:35.281114+0000","last_clean_scrub_stamp":"2024-03-19T02:06:35.281114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:14.311729+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4,7],"acting":[6,0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:37.524685+0000","last_change":"2024-03-19T02:06:37.524685+0000","last_active":"2024-03-19T02:06:37.524685+0000","last_peered":"2024-03-19T02:06:37.524685+0000","last_clean":"2024-03-19T02:06:37.524685+0000","last_became_active":"2024-03-19T01:51:09.196093+0000","last_became_peered":"2024-03-19T01:51:09.196093+0000","last_unstale":"2024-03-19T02:06:37.524685+0000","last_undegraded":"2024-03-19T02:06:37.524685+0000","last_fullsized":"2024-03-19T02:06:37.524685+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:37.524550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:37.524550+0000","last_clean_scrub_stamp":"2024-03-19T02:06:37.524550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:09:34.396521+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"35'8","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:18.842513+0000","last_change":"2024-03-19T02:06:18.842513+0000","last_active":"2024-03-19T02:06:18.842513+0000","last_peered":"2024-03-19T02:06:18.842513+0000","last_clean":"2024-03-19T02:06:18.842513+0000","last_became_active":"2024-03-19T01:51:06.949234+0000","last_became_peered":"2024-03-19T01:51:06.949234+0000","last_unstale":"2024-03-19T02:06:18.842513+0000","last_undegraded":"2024-03-19T02:06:18.842513+0000","last_fullsized":"2024-03-19T02:06:18.842513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'8","last_scrub_stamp":"2024-03-19T02:06:18.842379+0000","last_deep_scrub":"35'8","last_deep_scrub_stamp":"2024-03-19T02:06:18.842379+0000","last_clean_scrub_stamp":"2024-03-19T02:06:18.842379+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:40:26.699462+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:33.285479+0000","last_change":"2024-03-19T02:06:33.285479+0000","last_active":"2024-03-19T02:06:33.285479+0000","last_peered":"2024-03-19T02:06:33.285479+0000","last_clean":"2024-03-19T02:06:33.285479+0000","last_became_active":"2024-03-19T01:51:16.553595+0000","last_became_peered":"2024-03-19T01:51:16.553595+0000","last_unstale":"2024-03-19T02:06:33.285479+0000","last_undegraded":"2024-03-19T02:06:33.285479+0000","last_fullsized":"2024-03-19T02:06:33.285479+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:33.285316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:33.285316+0000","last_clean_scrub_stamp":"2024-03-19T02:06:33.285316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:23.783936+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:24.888086+0000","last_change":"2024-03-19T02:06:24.888122+0000","last_active":"2024-03-19T02:06:24.888086+0000","last_peered":"2024-03-19T02:06:24.888086+0000","last_clean":"2024-03-19T02:06:24.888086+0000","last_became_active":"2024-03-19T01:52:12.918458+0000","last_became_peered":"2024-03-19T01:52:12.918458+0000","last_unstale":"2024-03-19T02:06:24.888086+0000","last_undegraded":"2024-03-19T02:06:24.888086+0000","last_fullsized":"2024-03-19T02:06:24.888086+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:24.887985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:24.887985+0000","last_clean_scrub_stamp":"2024-03-19T02:06:24.887985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:44.778135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'4","reported_seq":61,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:18.401112+0000","last_change":"2024-03-19T02:06:18.401112+0000","last_active":"2024-03-19T02:06:18.401112+0000","last_peered":"2024-03-19T02:06:18.401112+0000","last_clean":"2024-03-19T02:06:18.401112+0000","last_became_active":"2024-03-19T01:51:06.937580+0000","last_became_peered":"2024-03-19T01:51:06.937580+0000","last_unstale":"2024-03-19T02:06:18.401112+0000","last_undegraded":"2024-03-19T02:06:18.401112+0000","last_fullsized":"2024-03-19T02:06:18.401112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4","last_scrub_stamp":"2024-03-19T02:06:18.400964+0000","last_deep_scrub":"35'4","last_deep_scrub_stamp":"2024-03-19T02:06:18.400964+0000","last_clean_scrub_stamp":"2024-03-19T02:06:18.400964+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:16:43.147574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:32.151923+0000","last_change":"2024-03-19T02:06:32.151985+0000","last_active":"2024-03-19T02:06:32.151923+0000","last_peered":"2024-03-19T02:06:32.151923+0000","last_clean":"2024-03-19T02:06:32.151923+0000","last_became_active":"2024-03-19T01:51:16.581746+0000","last_became_peered":"2024-03-19T01:51:16.581746+0000","last_unstale":"2024-03-19T02:06:32.151923+0000","last_undegraded":"2024-03-19T02:06:32.151923+0000","last_fullsized":"2024-03-19T02:06:32.151923+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:32.151724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:32.151724+0000","last_clean_scrub_stamp":"2024-03-19T02:06:32.151724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:07.525601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:34.543999+0000","last_change":"2024-03-19T02:06:34.543999+0000","last_active":"2024-03-19T02:06:34.543999+0000","last_peered":"2024-03-19T02:06:34.543999+0000","last_clean":"2024-03-19T02:06:34.543999+0000","last_became_active":"2024-03-19T01:51:09.194645+0000","last_became_peered":"2024-03-19T01:51:09.194645+0000","last_unstale":"2024-03-19T02:06:34.543999+0000","last_undegraded":"2024-03-19T02:06:34.543999+0000","last_fullsized":"2024-03-19T02:06:34.543999+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:34.543860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:34.543860+0000","last_clean_scrub_stamp":"2024-03-19T02:06:34.543860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:55.099052+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"35'7","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:20.900634+0000","last_change":"2024-03-19T02:06:20.900634+0000","last_active":"2024-03-19T02:06:20.900634+0000","last_peered":"2024-03-19T02:06:20.900634+0000","last_clean":"2024-03-19T02:06:20.900634+0000","last_became_active":"2024-03-19T01:51:06.954951+0000","last_became_peered":"2024-03-19T01:51:06.954951+0000","last_unstale":"2024-03-19T02:06:20.900634+0000","last_undegraded":"2024-03-19T02:06:20.900634+0000","last_fullsized":"2024-03-19T02:06:20.900634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'7","last_scrub_stamp":"2024-03-19T02:06:20.900479+0000","last_deep_scrub":"35'7","last_deep_scrub_stamp":"2024-03-19T02:06:20.900479+0000","last_clean_scrub_stamp":"2024-03-19T02:06:20.900479+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:11:33.806639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:36.131519+0000","last_change":"2024-03-19T02:06:36.131557+0000","last_active":"2024-03-19T02:06:36.131519+0000","last_peered":"2024-03-19T02:06:36.131519+0000","last_clean":"2024-03-19T02:06:36.131519+0000","last_became_active":"2024-03-19T01:51:16.586298+0000","last_became_peered":"2024-03-19T01:51:16.586298+0000","last_unstale":"2024-03-19T02:06:36.131519+0000","last_undegraded":"2024-03-19T02:06:36.131519+0000","last_fullsized":"2024-03-19T02:06:36.131519+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:36.131411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:36.131411+0000","last_clean_scrub_stamp":"2024-03-19T02:06:36.131411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:38:39.575569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:35.532037+0000","last_change":"2024-03-19T02:06:35.532037+0000","last_active":"2024-03-19T02:06:35.532037+0000","last_peered":"2024-03-19T02:06:35.532037+0000","last_clean":"2024-03-19T02:06:35.532037+0000","last_became_active":"2024-03-19T01:51:09.195111+0000","last_became_peered":"2024-03-19T01:51:09.195111+0000","last_unstale":"2024-03-19T02:06:35.532037+0000","last_undegraded":"2024-03-19T02:06:35.532037+0000","last_fullsized":"2024-03-19T02:06:35.532037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:35.531891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:35.531891+0000","last_clean_scrub_stamp":"2024-03-19T02:06:35.531891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:45.524317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"35'4","reported_seq":61,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:19.269208+0000","last_change":"2024-03-19T02:06:19.269208+0000","last_active":"2024-03-19T02:06:19.269208+0000","last_peered":"2024-03-19T02:06:19.269208+0000","last_clean":"2024-03-19T02:06:19.269208+0000","last_became_active":"2024-03-19T01:51:06.963028+0000","last_became_peered":"2024-03-19T01:51:06.963028+0000","last_unstale":"2024-03-19T02:06:19.269208+0000","last_undegraded":"2024-03-19T02:06:19.269208+0000","last_fullsized":"2024-03-19T02:06:19.269208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4","last_scrub_stamp":"2024-03-19T02:06:19.269068+0000","last_deep_scrub":"35'4","last_deep_scrub_stamp":"2024-03-19T02:06:19.269068+0000","last_clean_scrub_stamp":"2024-03-19T02:06:19.269068+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:34.129979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:29.378449+0000","last_change":"2024-03-19T02:06:29.378488+0000","last_active":"2024-03-19T02:06:29.378449+0000","last_peered":"2024-03-19T02:06:29.378449+0000","last_clean":"2024-03-19T02:06:29.378449+0000","last_became_active":"2024-03-19T01:51:16.587459+0000","last_became_peered":"2024-03-19T01:51:16.587459+0000","last_unstale":"2024-03-19T02:06:29.378449+0000","last_undegraded":"2024-03-19T02:06:29.378449+0000","last_fullsized":"2024-03-19T02:06:29.378449+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:29.378342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:29.378342+0000","last_clean_scrub_stamp":"2024-03-19T02:06:29.378342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:33:02.816515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:36.506650+0000","last_change":"2024-03-19T02:06:36.506685+0000","last_active":"2024-03-19T02:06:36.506650+0000","last_peered":"2024-03-19T02:06:36.506650+0000","last_clean":"2024-03-19T02:06:36.506650+0000","last_became_active":"2024-03-19T01:51:09.195139+0000","last_became_peered":"2024-03-19T01:51:09.195139+0000","last_unstale":"2024-03-19T02:06:36.506650+0000","last_undegraded":"2024-03-19T02:06:36.506650+0000","last_fullsized":"2024-03-19T02:06:36.506650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:36.506539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:36.506539+0000","last_clean_scrub_stamp":"2024-03-19T02:06:36.506539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:45:38.375714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"35'25","reported_seq":81,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:17.851277+0000","last_change":"2024-03-19T02:06:17.851348+0000","last_active":"2024-03-19T02:06:17.851277+0000","last_peered":"2024-03-19T02:06:17.851277+0000","last_clean":"2024-03-19T02:06:17.851277+0000","last_became_active":"2024-03-19T01:51:06.940594+0000","last_became_peered":"2024-03-19T01:51:06.940594+0000","last_unstale":"2024-03-19T02:06:17.851277+0000","last_undegraded":"2024-03-19T02:06:17.851277+0000","last_fullsized":"2024-03-19T02:06:17.851277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'25","last_scrub_stamp":"2024-03-19T02:06:17.851166+0000","last_deep_scrub":"35'25","last_deep_scrub_stamp":"2024-03-19T02:06:17.851166+0000","last_clean_scrub_stamp":"2024-03-19T02:06:17.851166+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:28:11.172347+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:30.107005+0000","last_change":"2024-03-19T02:06:30.107148+0000","last_active":"2024-03-19T02:06:30.107005+0000","last_peered":"2024-03-19T02:06:30.107005+0000","last_clean":"2024-03-19T02:06:30.107005+0000","last_became_active":"2024-03-19T01:51:16.547800+0000","last_became_peered":"2024-03-19T01:51:16.547800+0000","last_unstale":"2024-03-19T02:06:30.107005+0000","last_undegraded":"2024-03-19T02:06:30.107005+0000","last_fullsized":"2024-03-19T02:06:30.107005+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:30.106760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:30.106760+0000","last_clean_scrub_stamp":"2024-03-19T02:06:30.106760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:30:40.317630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:26.234708+0000","last_change":"2024-03-19T02:06:26.234757+0000","last_active":"2024-03-19T02:06:26.234708+0000","last_peered":"2024-03-19T02:06:26.234708+0000","last_clean":"2024-03-19T02:06:26.234708+0000","last_became_active":"2024-03-19T01:51:09.167670+0000","last_became_peered":"2024-03-19T01:51:09.167670+0000","last_unstale":"2024-03-19T02:06:26.234708+0000","last_undegraded":"2024-03-19T02:06:26.234708+0000","last_fullsized":"2024-03-19T02:06:26.234708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:26.234591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:26.234591+0000","last_clean_scrub_stamp":"2024-03-19T02:06:26.234591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:04:11.958629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"35'8","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:25.277657+0000","last_change":"2024-03-19T02:06:25.277717+0000","last_active":"2024-03-19T02:06:25.277657+0000","last_peered":"2024-03-19T02:06:25.277657+0000","last_clean":"2024-03-19T02:06:25.277657+0000","last_became_active":"2024-03-19T01:51:07.047785+0000","last_became_peered":"2024-03-19T01:51:07.047785+0000","last_unstale":"2024-03-19T02:06:25.277657+0000","last_undegraded":"2024-03-19T02:06:25.277657+0000","last_fullsized":"2024-03-19T02:06:25.277657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'8","last_scrub_stamp":"2024-03-19T02:06:25.277535+0000","last_deep_scrub":"35'8","last_deep_scrub_stamp":"2024-03-19T02:06:25.277535+0000","last_clean_scrub_stamp":"2024-03-19T02:06:25.277535+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:16:50.024145+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:35.382512+0000","last_change":"2024-03-19T02:06:35.382550+0000","last_active":"2024-03-19T02:06:35.382512+0000","last_peered":"2024-03-19T02:06:35.382512+0000","last_clean":"2024-03-19T02:06:35.382512+0000","last_became_active":"2024-03-19T01:51:16.582860+0000","last_became_peered":"2024-03-19T01:51:16.582860+0000","last_unstale":"2024-03-19T02:06:35.382512+0000","last_undegraded":"2024-03-19T02:06:35.382512+0000","last_fullsized":"2024-03-19T02:06:35.382512+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:35.382407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:35.382407+0000","last_clean_scrub_stamp":"2024-03-19T02:06:35.382407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:45:30.482728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:33.550448+0000","last_change":"2024-03-19T02:06:33.550483+0000","last_active":"2024-03-19T02:06:33.550448+0000","last_peered":"2024-03-19T02:06:33.550448+0000","last_clean":"2024-03-19T02:06:33.550448+0000","last_became_active":"2024-03-19T01:51:09.194728+0000","last_became_peered":"2024-03-19T01:51:09.194728+0000","last_unstale":"2024-03-19T02:06:33.550448+0000","last_undegraded":"2024-03-19T02:06:33.550448+0000","last_fullsized":"2024-03-19T02:06:33.550448+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:33.550341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:33.550341+0000","last_clean_scrub_stamp":"2024-03-19T02:06:33.550341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:08.924219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"35'9","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:23.987918+0000","last_change":"2024-03-19T02:06:23.987971+0000","last_active":"2024-03-19T02:06:23.987918+0000","last_peered":"2024-03-19T02:06:23.987918+0000","last_clean":"2024-03-19T02:06:23.987918+0000","last_became_active":"2024-03-19T01:51:06.955991+0000","last_became_peered":"2024-03-19T01:51:06.955991+0000","last_unstale":"2024-03-19T02:06:23.987918+0000","last_undegraded":"2024-03-19T02:06:23.987918+0000","last_fullsized":"2024-03-19T02:06:23.987918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9","last_scrub_stamp":"2024-03-19T02:06:23.987816+0000","last_deep_scrub":"35'9","last_deep_scrub_stamp":"2024-03-19T02:06:23.987816+0000","last_clean_scrub_stamp":"2024-03-19T02:06:23.987816+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:01:50.622324+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:37.346629+0000","last_change":"2024-03-19T02:06:37.346749+0000","last_active":"2024-03-19T02:06:37.346629+0000","last_peered":"2024-03-19T02:06:37.346629+0000","last_clean":"2024-03-19T02:06:37.346629+0000","last_became_active":"2024-03-19T01:52:14.331292+0000","last_became_peered":"2024-03-19T01:52:14.331292+0000","last_unstale":"2024-03-19T02:06:37.346629+0000","last_undegraded":"2024-03-19T02:06:37.346629+0000","last_fullsized":"2024-03-19T02:06:37.346629+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:37.346424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:37.346424+0000","last_clean_scrub_stamp":"2024-03-19T02:06:37.346424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:39.966045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:25.011560+0000","last_change":"2024-03-19T02:06:25.011612+0000","last_active":"2024-03-19T02:06:25.011560+0000","last_peered":"2024-03-19T02:06:25.011560+0000","last_clean":"2024-03-19T02:06:25.011560+0000","last_became_active":"2024-03-19T01:51:09.187275+0000","last_became_peered":"2024-03-19T01:51:09.187275+0000","last_unstale":"2024-03-19T02:06:25.011560+0000","last_undegraded":"2024-03-19T02:06:25.011560+0000","last_fullsized":"2024-03-19T02:06:25.011560+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:25.011380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:25.011380+0000","last_clean_scrub_stamp":"2024-03-19T02:06:25.011380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:17:12.000361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"35'15","reported_seq":71,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:16.884717+0000","last_change":"2024-03-19T02:06:16.884792+0000","last_active":"2024-03-19T02:06:16.884717+0000","last_peered":"2024-03-19T02:06:16.884717+0000","last_clean":"2024-03-19T02:06:16.884717+0000","last_became_active":"2024-03-19T01:51:06.963763+0000","last_became_peered":"2024-03-19T01:51:06.963763+0000","last_unstale":"2024-03-19T02:06:16.884717+0000","last_undegraded":"2024-03-19T02:06:16.884717+0000","last_fullsized":"2024-03-19T02:06:16.884717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'15","last_scrub_stamp":"2024-03-19T02:06:16.884516+0000","last_deep_scrub":"35'15","last_deep_scrub_stamp":"2024-03-19T02:06:16.884516+0000","last_clean_scrub_stamp":"2024-03-19T02:06:16.884516+0000","objects_scrubbed":6,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:17.027904+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:35.119679+0000","last_change":"2024-03-19T02:06:35.119716+0000","last_active":"2024-03-19T02:06:35.119679+0000","last_peered":"2024-03-19T02:06:35.119679+0000","last_clean":"2024-03-19T02:06:35.119679+0000","last_became_active":"2024-03-19T01:51:16.573744+0000","last_became_peered":"2024-03-19T01:51:16.573744+0000","last_unstale":"2024-03-19T02:06:35.119679+0000","last_undegraded":"2024-03-19T02:06:35.119679+0000","last_fullsized":"2024-03-19T02:06:35.119679+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:35.119566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:35.119566+0000","last_clean_scrub_stamp":"2024-03-19T02:06:35.119566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:38:39.689905+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:22.562311+0000","last_change":"2024-03-19T02:06:22.562349+0000","last_active":"2024-03-19T02:06:22.562311+0000","last_peered":"2024-03-19T02:06:22.562311+0000","last_clean":"2024-03-19T02:06:22.562311+0000","last_became_active":"2024-03-19T01:51:09.175898+0000","last_became_peered":"2024-03-19T01:51:09.175898+0000","last_unstale":"2024-03-19T02:06:22.562311+0000","last_undegraded":"2024-03-19T02:06:22.562311+0000","last_fullsized":"2024-03-19T02:06:22.562311+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:22.562197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:22.562197+0000","last_clean_scrub_stamp":"2024-03-19T02:06:22.562197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:31.131402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"35'20","reported_seq":76,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:21.236538+0000","last_change":"2024-03-19T02:06:21.236590+0000","last_active":"2024-03-19T02:06:21.236538+0000","last_peered":"2024-03-19T02:06:21.236538+0000","last_clean":"2024-03-19T02:06:21.236538+0000","last_became_active":"2024-03-19T01:51:07.041262+0000","last_became_peered":"2024-03-19T01:51:07.041262+0000","last_unstale":"2024-03-19T02:06:21.236538+0000","last_undegraded":"2024-03-19T02:06:21.236538+0000","last_fullsized":"2024-03-19T02:06:21.236538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'20","last_scrub_stamp":"2024-03-19T02:06:21.236435+0000","last_deep_scrub":"35'20","last_deep_scrub_stamp":"2024-03-19T02:06:21.236435+0000","last_clean_scrub_stamp":"2024-03-19T02:06:21.236435+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:00:47.854583+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:28.398093+0000","last_change":"2024-03-19T02:06:28.398138+0000","last_active":"2024-03-19T02:06:28.398093+0000","last_peered":"2024-03-19T02:06:28.398093+0000","last_clean":"2024-03-19T02:06:28.398093+0000","last_became_active":"2024-03-19T01:51:16.549347+0000","last_became_peered":"2024-03-19T01:51:16.549347+0000","last_unstale":"2024-03-19T02:06:28.398093+0000","last_undegraded":"2024-03-19T02:06:28.398093+0000","last_fullsized":"2024-03-19T02:06:28.398093+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:28.397971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:28.397971+0000","last_clean_scrub_stamp":"2024-03-19T02:06:28.397971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:55:07.428585+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:24.320175+0000","last_change":"2024-03-19T02:06:24.320210+0000","last_active":"2024-03-19T02:06:24.320175+0000","last_peered":"2024-03-19T02:06:24.320175+0000","last_clean":"2024-03-19T02:06:24.320175+0000","last_became_active":"2024-03-19T01:51:09.183607+0000","last_became_peered":"2024-03-19T01:51:09.183607+0000","last_unstale":"2024-03-19T02:06:24.320175+0000","last_undegraded":"2024-03-19T02:06:24.320175+0000","last_fullsized":"2024-03-19T02:06:24.320175+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:24.320061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:24.320061+0000","last_clean_scrub_stamp":"2024-03-19T02:06:24.320061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:32:21.563530+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"35'9","reported_seq":66,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:19.460598+0000","last_change":"2024-03-19T02:06:19.460645+0000","last_active":"2024-03-19T02:06:19.460598+0000","last_peered":"2024-03-19T02:06:19.460598+0000","last_clean":"2024-03-19T02:06:19.460598+0000","last_became_active":"2024-03-19T01:51:06.951700+0000","last_became_peered":"2024-03-19T01:51:06.951700+0000","last_unstale":"2024-03-19T02:06:19.460598+0000","last_undegraded":"2024-03-19T02:06:19.460598+0000","last_fullsized":"2024-03-19T02:06:19.460598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9","last_scrub_stamp":"2024-03-19T02:06:19.460499+0000","last_deep_scrub":"35'9","last_deep_scrub_stamp":"2024-03-19T02:06:19.460499+0000","last_clean_scrub_stamp":"2024-03-19T02:06:19.460499+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:23.519926+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:27.354502+0000","last_change":"2024-03-19T02:06:27.354543+0000","last_active":"2024-03-19T02:06:27.354502+0000","last_peered":"2024-03-19T02:06:27.354502+0000","last_clean":"2024-03-19T02:06:27.354502+0000","last_became_active":"2024-03-19T01:51:16.586514+0000","last_became_peered":"2024-03-19T01:51:16.586514+0000","last_unstale":"2024-03-19T02:06:27.354502+0000","last_undegraded":"2024-03-19T02:06:27.354502+0000","last_fullsized":"2024-03-19T02:06:27.354502+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:27.354385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:27.354385+0000","last_clean_scrub_stamp":"2024-03-19T02:06:27.354385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:35:49.920881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:24.018946+0000","last_change":"2024-03-19T02:06:24.018982+0000","last_active":"2024-03-19T02:06:24.018946+0000","last_peered":"2024-03-19T02:06:24.018946+0000","last_clean":"2024-03-19T02:06:24.018946+0000","last_became_active":"2024-03-19T01:51:09.171814+0000","last_became_peered":"2024-03-19T01:51:09.171814+0000","last_unstale":"2024-03-19T02:06:24.018946+0000","last_undegraded":"2024-03-19T02:06:24.018946+0000","last_fullsized":"2024-03-19T02:06:24.018946+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:24.018844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:24.018844+0000","last_clean_scrub_stamp":"2024-03-19T02:06:24.018844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:04:00.264930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"35'6","reported_seq":62,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:25.028492+0000","last_change":"2024-03-19T02:06:25.028542+0000","last_active":"2024-03-19T02:06:25.028492+0000","last_peered":"2024-03-19T02:06:25.028492+0000","last_clean":"2024-03-19T02:06:25.028492+0000","last_became_active":"2024-03-19T01:51:07.043069+0000","last_became_peered":"2024-03-19T01:51:07.043069+0000","last_unstale":"2024-03-19T02:06:25.028492+0000","last_undegraded":"2024-03-19T02:06:25.028492+0000","last_fullsized":"2024-03-19T02:06:25.028492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'6","last_scrub_stamp":"2024-03-19T02:06:25.028393+0000","last_deep_scrub":"35'6","last_deep_scrub_stamp":"2024-03-19T02:06:25.028393+0000","last_clean_scrub_stamp":"2024-03-19T02:06:25.028393+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:04:00.723861+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:34.393491+0000","last_change":"2024-03-19T02:06:34.393545+0000","last_active":"2024-03-19T02:06:34.393491+0000","last_peered":"2024-03-19T02:06:34.393491+0000","last_clean":"2024-03-19T02:06:34.393491+0000","last_became_active":"2024-03-19T01:51:16.581174+0000","last_became_peered":"2024-03-19T01:51:16.581174+0000","last_unstale":"2024-03-19T02:06:34.393491+0000","last_undegraded":"2024-03-19T02:06:34.393491+0000","last_fullsized":"2024-03-19T02:06:34.393491+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:34.393323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:34.393323+0000","last_clean_scrub_stamp":"2024-03-19T02:06:34.393323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:11.393858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:32.082636+0000","last_change":"2024-03-19T02:06:32.082671+0000","last_active":"2024-03-19T02:06:32.082636+0000","last_peered":"2024-03-19T02:06:32.082636+0000","last_clean":"2024-03-19T02:06:32.082636+0000","last_became_active":"2024-03-19T01:51:09.193656+0000","last_became_peered":"2024-03-19T01:51:09.193656+0000","last_unstale":"2024-03-19T02:06:32.082636+0000","last_undegraded":"2024-03-19T02:06:32.082636+0000","last_fullsized":"2024-03-19T02:06:32.082636+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:32.082535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:32.082535+0000","last_clean_scrub_stamp":"2024-03-19T02:06:32.082535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:11.534974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"35'7","reported_seq":64,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:24.274968+0000","last_change":"2024-03-19T02:06:24.275021+0000","last_active":"2024-03-19T02:06:24.274968+0000","last_peered":"2024-03-19T02:06:24.274968+0000","last_clean":"2024-03-19T02:06:24.274968+0000","last_became_active":"2024-03-19T01:51:07.045342+0000","last_became_peered":"2024-03-19T01:51:07.045342+0000","last_unstale":"2024-03-19T02:06:24.274968+0000","last_undegraded":"2024-03-19T02:06:24.274968+0000","last_fullsized":"2024-03-19T02:06:24.274968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'7","last_scrub_stamp":"2024-03-19T02:06:24.274855+0000","last_deep_scrub":"35'7","last_deep_scrub_stamp":"2024-03-19T02:06:24.274855+0000","last_clean_scrub_stamp":"2024-03-19T02:06:24.274855+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:00:09.027764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":41,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:33.357129+0000","last_change":"2024-03-19T02:06:33.357166+0000","last_active":"2024-03-19T02:06:33.357129+0000","last_peered":"2024-03-19T02:06:33.357129+0000","last_clean":"2024-03-19T02:06:33.357129+0000","last_became_active":"2024-03-19T01:51:16.572283+0000","last_became_peered":"2024-03-19T01:51:16.572283+0000","last_unstale":"2024-03-19T02:06:33.357129+0000","last_undegraded":"2024-03-19T02:06:33.357129+0000","last_fullsized":"2024-03-19T02:06:33.357129+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:33.357026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:33.357026+0000","last_clean_scrub_stamp":"2024-03-19T02:06:33.357026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:34.662092+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:29.113095+0000","last_change":"2024-03-19T02:06:29.113136+0000","last_active":"2024-03-19T02:06:29.113095+0000","last_peered":"2024-03-19T02:06:29.113095+0000","last_clean":"2024-03-19T02:06:29.113095+0000","last_became_active":"2024-03-19T01:51:16.533665+0000","last_became_peered":"2024-03-19T01:51:16.533665+0000","last_unstale":"2024-03-19T02:06:29.113095+0000","last_undegraded":"2024-03-19T02:06:29.113095+0000","last_fullsized":"2024-03-19T02:06:29.113095+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:29.112989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:29.112989+0000","last_clean_scrub_stamp":"2024-03-19T02:06:29.112989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:35:38.582357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:16.354419+0000","last_change":"2024-03-19T02:06:16.354453+0000","last_active":"2024-03-19T02:06:16.354419+0000","last_peered":"2024-03-19T02:06:16.354419+0000","last_clean":"2024-03-19T02:06:16.354419+0000","last_became_active":"2024-03-19T01:51:09.167626+0000","last_became_peered":"2024-03-19T01:51:09.167626+0000","last_unstale":"2024-03-19T02:06:16.354419+0000","last_undegraded":"2024-03-19T02:06:16.354419+0000","last_fullsized":"2024-03-19T02:06:16.354419+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:16.354304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:16.354304+0000","last_clean_scrub_stamp":"2024-03-19T02:06:16.354304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:26:28.867422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"35'5","reported_seq":61,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:21.022465+0000","last_change":"2024-03-19T02:06:21.022508+0000","last_active":"2024-03-19T02:06:21.022465+0000","last_peered":"2024-03-19T02:06:21.022465+0000","last_clean":"2024-03-19T02:06:21.022465+0000","last_became_active":"2024-03-19T01:51:06.950826+0000","last_became_peered":"2024-03-19T01:51:06.950826+0000","last_unstale":"2024-03-19T02:06:21.022465+0000","last_undegraded":"2024-03-19T02:06:21.022465+0000","last_fullsized":"2024-03-19T02:06:21.022465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5","last_scrub_stamp":"2024-03-19T02:06:21.022368+0000","last_deep_scrub":"35'5","last_deep_scrub_stamp":"2024-03-19T02:06:21.022368+0000","last_clean_scrub_stamp":"2024-03-19T02:06:21.022368+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:07.201424+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":42,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:34.319937+0000","last_change":"2024-03-19T02:06:34.319983+0000","last_active":"2024-03-19T02:06:34.319937+0000","last_peered":"2024-03-19T02:06:34.319937+0000","last_clean":"2024-03-19T02:06:34.319937+0000","last_became_active":"2024-03-19T01:51:16.581412+0000","last_became_peered":"2024-03-19T01:51:16.581412+0000","last_unstale":"2024-03-19T02:06:34.319937+0000","last_undegraded":"2024-03-19T02:06:34.319937+0000","last_fullsized":"2024-03-19T02:06:34.319937+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:34.319823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:34.319823+0000","last_clean_scrub_stamp":"2024-03-19T02:06:34.319823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:43:07.601257+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:25.348284+0000","last_change":"2024-03-19T02:06:25.348319+0000","last_active":"2024-03-19T02:06:25.348284+0000","last_peered":"2024-03-19T02:06:25.348284+0000","last_clean":"2024-03-19T02:06:25.348284+0000","last_became_active":"2024-03-19T01:51:09.160978+0000","last_became_peered":"2024-03-19T01:51:09.160978+0000","last_unstale":"2024-03-19T02:06:25.348284+0000","last_undegraded":"2024-03-19T02:06:25.348284+0000","last_fullsized":"2024-03-19T02:06:25.348284+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:25.348182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:25.348182+0000","last_clean_scrub_stamp":"2024-03-19T02:06:25.348182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:04:09.991288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"35'5","reported_seq":61,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:24.385489+0000","last_change":"2024-03-19T02:06:24.385534+0000","last_active":"2024-03-19T02:06:24.385489+0000","last_peered":"2024-03-19T02:06:24.385489+0000","last_clean":"2024-03-19T02:06:24.385489+0000","last_became_active":"2024-03-19T01:51:07.046886+0000","last_became_peered":"2024-03-19T01:51:07.046886+0000","last_unstale":"2024-03-19T02:06:24.385489+0000","last_undegraded":"2024-03-19T02:06:24.385489+0000","last_fullsized":"2024-03-19T02:06:24.385489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'5","last_scrub_stamp":"2024-03-19T02:06:24.385386+0000","last_deep_scrub":"35'5","last_deep_scrub_stamp":"2024-03-19T02:06:24.385386+0000","last_clean_scrub_stamp":"2024-03-19T02:06:24.385386+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:43:49.506626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"35'10","reported_seq":66,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:23.235868+0000","last_change":"2024-03-19T02:06:23.235935+0000","last_active":"2024-03-19T02:06:23.235868+0000","last_peered":"2024-03-19T02:06:23.235868+0000","last_clean":"2024-03-19T02:06:23.235868+0000","last_became_active":"2024-03-19T01:51:07.044554+0000","last_became_peered":"2024-03-19T01:51:07.044554+0000","last_unstale":"2024-03-19T02:06:23.235868+0000","last_undegraded":"2024-03-19T02:06:23.235868+0000","last_fullsized":"2024-03-19T02:06:23.235868+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'10","last_scrub_stamp":"2024-03-19T02:06:23.235758+0000","last_deep_scrub":"35'10","last_deep_scrub_stamp":"2024-03-19T02:06:23.235758+0000","last_clean_scrub_stamp":"2024-03-19T02:06:23.235758+0000","objects_scrubbed":3,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:26:35.748876+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:15.879720+0000","last_change":"2024-03-19T02:06:15.879756+0000","last_active":"2024-03-19T02:06:15.879720+0000","last_peered":"2024-03-19T02:06:15.879720+0000","last_clean":"2024-03-19T02:06:15.879720+0000","last_became_active":"2024-03-19T01:51:09.184666+0000","last_became_peered":"2024-03-19T01:51:09.184666+0000","last_unstale":"2024-03-19T02:06:15.879720+0000","last_undegraded":"2024-03-19T02:06:15.879720+0000","last_fullsized":"2024-03-19T02:06:15.879720+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:15.879603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:15.879603+0000","last_clean_scrub_stamp":"2024-03-19T02:06:15.879603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:23.923689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:26.913891+0000","last_change":"2024-03-19T02:06:26.913946+0000","last_active":"2024-03-19T02:06:26.913891+0000","last_peered":"2024-03-19T02:06:26.913891+0000","last_clean":"2024-03-19T02:06:26.913891+0000","last_became_active":"2024-03-19T01:52:14.297612+0000","last_became_peered":"2024-03-19T01:52:14.297612+0000","last_unstale":"2024-03-19T02:06:26.913891+0000","last_undegraded":"2024-03-19T02:06:26.913891+0000","last_fullsized":"2024-03-19T02:06:26.913891+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:26.913710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:26.913710+0000","last_clean_scrub_stamp":"2024-03-19T02:06:26.913710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:08:32.266655+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,5],"acting":[2,0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"35'9","reported_seq":68,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:23.378780+0000","last_change":"2024-03-19T02:06:23.378829+0000","last_active":"2024-03-19T02:06:23.378780+0000","last_peered":"2024-03-19T02:06:23.378780+0000","last_clean":"2024-03-19T02:06:23.378780+0000","last_became_active":"2024-03-19T01:51:06.981704+0000","last_became_peered":"2024-03-19T01:51:06.981704+0000","last_unstale":"2024-03-19T02:06:23.378780+0000","last_undegraded":"2024-03-19T02:06:23.378780+0000","last_fullsized":"2024-03-19T02:06:23.378780+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9","last_scrub_stamp":"2024-03-19T02:06:23.378684+0000","last_deep_scrub":"35'9","last_deep_scrub_stamp":"2024-03-19T02:06:23.378684+0000","last_clean_scrub_stamp":"2024-03-19T02:06:23.378684+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:36:15.260186+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:20.246970+0000","last_change":"2024-03-19T02:06:20.247006+0000","last_active":"2024-03-19T02:06:20.246970+0000","last_peered":"2024-03-19T02:06:20.246970+0000","last_clean":"2024-03-19T02:06:20.246970+0000","last_became_active":"2024-03-19T01:51:09.183976+0000","last_became_peered":"2024-03-19T01:51:09.183976+0000","last_unstale":"2024-03-19T02:06:20.246970+0000","last_undegraded":"2024-03-19T02:06:20.246970+0000","last_fullsized":"2024-03-19T02:06:20.246970+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:20.246864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:20.246864+0000","last_clean_scrub_stamp":"2024-03-19T02:06:20.246864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:00:46.865011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:36.333558+0000","last_change":"2024-03-19T02:06:36.333595+0000","last_active":"2024-03-19T02:06:36.333558+0000","last_peered":"2024-03-19T02:06:36.333558+0000","last_clean":"2024-03-19T02:06:36.333558+0000","last_became_active":"2024-03-19T01:52:14.297976+0000","last_became_peered":"2024-03-19T01:52:14.297976+0000","last_unstale":"2024-03-19T02:06:36.333558+0000","last_undegraded":"2024-03-19T02:06:36.333558+0000","last_fullsized":"2024-03-19T02:06:36.333558+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:36.333450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:36.333450+0000","last_clean_scrub_stamp":"2024-03-19T02:06:36.333450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:05.257132+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4,2],"acting":[3,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:28.257966+0000","last_change":"2024-03-19T02:06:28.258005+0000","last_active":"2024-03-19T02:06:28.257966+0000","last_peered":"2024-03-19T02:06:28.257966+0000","last_clean":"2024-03-19T02:06:28.257966+0000","last_became_active":"2024-03-19T01:51:09.168231+0000","last_became_peered":"2024-03-19T01:51:09.168231+0000","last_unstale":"2024-03-19T02:06:28.257966+0000","last_undegraded":"2024-03-19T02:06:28.257966+0000","last_fullsized":"2024-03-19T02:06:28.257966+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:28.257856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:28.257856+0000","last_clean_scrub_stamp":"2024-03-19T02:06:28.257856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:18:17.695090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:21.903900+0000","last_change":"2024-03-19T02:06:21.904002+0000","last_active":"2024-03-19T02:06:21.903900+0000","last_peered":"2024-03-19T02:06:21.903900+0000","last_clean":"2024-03-19T02:06:21.903900+0000","last_became_active":"2024-03-19T01:51:09.154545+0000","last_became_peered":"2024-03-19T01:51:09.154545+0000","last_unstale":"2024-03-19T02:06:21.903900+0000","last_undegraded":"2024-03-19T02:06:21.903900+0000","last_fullsized":"2024-03-19T02:06:21.903900+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:21.903728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:21.903728+0000","last_clean_scrub_stamp":"2024-03-19T02:06:21.903728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:15.348802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:20.061988+0000","last_change":"2024-03-19T02:06:20.062028+0000","last_active":"2024-03-19T02:06:20.061988+0000","last_peered":"2024-03-19T02:06:20.061988+0000","last_clean":"2024-03-19T02:06:20.061988+0000","last_became_active":"2024-03-19T01:51:09.179840+0000","last_became_peered":"2024-03-19T01:51:09.179840+0000","last_unstale":"2024-03-19T02:06:20.061988+0000","last_undegraded":"2024-03-19T02:06:20.061988+0000","last_fullsized":"2024-03-19T02:06:20.061988+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:20.061877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:20.061877+0000","last_clean_scrub_stamp":"2024-03-19T02:06:20.061877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:49:51.541579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:23.056496+0000","last_change":"2024-03-19T02:06:23.056546+0000","last_active":"2024-03-19T02:06:23.056496+0000","last_peered":"2024-03-19T02:06:23.056496+0000","last_clean":"2024-03-19T02:06:23.056496+0000","last_became_active":"2024-03-19T01:51:09.163974+0000","last_became_peered":"2024-03-19T01:51:09.163974+0000","last_unstale":"2024-03-19T02:06:23.056496+0000","last_undegraded":"2024-03-19T02:06:23.056496+0000","last_fullsized":"2024-03-19T02:06:23.056496+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:23.056302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:23.056302+0000","last_clean_scrub_stamp":"2024-03-19T02:06:23.056302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:33:15.331850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:21.368625+0000","last_change":"2024-03-19T02:06:21.368625+0000","last_active":"2024-03-19T02:06:21.368625+0000","last_peered":"2024-03-19T02:06:21.368625+0000","last_clean":"2024-03-19T02:06:21.368625+0000","last_became_active":"2024-03-19T01:51:09.176474+0000","last_became_peered":"2024-03-19T01:51:09.176474+0000","last_unstale":"2024-03-19T02:06:21.368625+0000","last_undegraded":"2024-03-19T02:06:21.368625+0000","last_fullsized":"2024-03-19T02:06:21.368625+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:21.368487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:21.368487+0000","last_clean_scrub_stamp":"2024-03-19T02:06:21.368487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:19.229395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:26.381504+0000","last_change":"2024-03-19T02:06:26.381541+0000","last_active":"2024-03-19T02:06:26.381504+0000","last_peered":"2024-03-19T02:06:26.381504+0000","last_clean":"2024-03-19T02:06:26.381504+0000","last_became_active":"2024-03-19T01:51:09.167982+0000","last_became_peered":"2024-03-19T01:51:09.167982+0000","last_unstale":"2024-03-19T02:06:26.381504+0000","last_undegraded":"2024-03-19T02:06:26.381504+0000","last_fullsized":"2024-03-19T02:06:26.381504+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:26.381399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:26.381399+0000","last_clean_scrub_stamp":"2024-03-19T02:06:26.381399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:16.880019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:32.296134+0000","last_change":"2024-03-19T02:06:32.296171+0000","last_active":"2024-03-19T02:06:32.296134+0000","last_peered":"2024-03-19T02:06:32.296134+0000","last_clean":"2024-03-19T02:06:32.296134+0000","last_became_active":"2024-03-19T01:51:09.189128+0000","last_became_peered":"2024-03-19T01:51:09.189128+0000","last_unstale":"2024-03-19T02:06:32.296134+0000","last_undegraded":"2024-03-19T02:06:32.296134+0000","last_fullsized":"2024-03-19T02:06:32.296134+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:32.296025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:32.296025+0000","last_clean_scrub_stamp":"2024-03-19T02:06:32.296025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:17:19.285005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:20.415920+0000","last_change":"2024-03-19T02:06:20.415920+0000","last_active":"2024-03-19T02:06:20.415920+0000","last_peered":"2024-03-19T02:06:20.415920+0000","last_clean":"2024-03-19T02:06:20.415920+0000","last_became_active":"2024-03-19T01:51:09.183417+0000","last_became_peered":"2024-03-19T01:51:09.183417+0000","last_unstale":"2024-03-19T02:06:20.415920+0000","last_undegraded":"2024-03-19T02:06:20.415920+0000","last_fullsized":"2024-03-19T02:06:20.415920+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:20.415782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:20.415782+0000","last_clean_scrub_stamp":"2024-03-19T02:06:20.415782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:51:05.370550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:27.044189+0000","last_change":"2024-03-19T02:06:27.044230+0000","last_active":"2024-03-19T02:06:27.044189+0000","last_peered":"2024-03-19T02:06:27.044189+0000","last_clean":"2024-03-19T02:06:27.044189+0000","last_became_active":"2024-03-19T01:51:09.165872+0000","last_became_peered":"2024-03-19T01:51:09.165872+0000","last_unstale":"2024-03-19T02:06:27.044189+0000","last_undegraded":"2024-03-19T02:06:27.044189+0000","last_fullsized":"2024-03-19T02:06:27.044189+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:27.044082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:27.044082+0000","last_clean_scrub_stamp":"2024-03-19T02:06:27.044082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:03.468351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:23.875122+0000","last_change":"2024-03-19T02:06:23.875179+0000","last_active":"2024-03-19T02:06:23.875122+0000","last_peered":"2024-03-19T02:06:23.875122+0000","last_clean":"2024-03-19T02:06:23.875122+0000","last_became_active":"2024-03-19T01:52:12.916608+0000","last_became_peered":"2024-03-19T01:52:12.916608+0000","last_unstale":"2024-03-19T02:06:23.875122+0000","last_undegraded":"2024-03-19T02:06:23.875122+0000","last_fullsized":"2024-03-19T02:06:23.875122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:23.874947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:23.874947+0000","last_clean_scrub_stamp":"2024-03-19T02:06:23.874947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:59:16.698826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:33.064403+0000","last_change":"2024-03-19T02:06:33.064403+0000","last_active":"2024-03-19T02:06:33.064403+0000","last_peered":"2024-03-19T02:06:33.064403+0000","last_clean":"2024-03-19T02:06:33.064403+0000","last_became_active":"2024-03-19T01:51:09.194536+0000","last_became_peered":"2024-03-19T01:51:09.194536+0000","last_unstale":"2024-03-19T02:06:33.064403+0000","last_undegraded":"2024-03-19T02:06:33.064403+0000","last_fullsized":"2024-03-19T02:06:33.064403+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:33.064112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:33.064112+0000","last_clean_scrub_stamp":"2024-03-19T02:06:33.064112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:14:14.631457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:25.988382+0000","last_change":"2024-03-19T02:06:25.988382+0000","last_active":"2024-03-19T02:06:25.988382+0000","last_peered":"2024-03-19T02:06:25.988382+0000","last_clean":"2024-03-19T02:06:25.988382+0000","last_became_active":"2024-03-19T01:51:09.188565+0000","last_became_peered":"2024-03-19T01:51:09.188565+0000","last_unstale":"2024-03-19T02:06:25.988382+0000","last_undegraded":"2024-03-19T02:06:25.988382+0000","last_fullsized":"2024-03-19T02:06:25.988382+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:25.988238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:25.988238+0000","last_clean_scrub_stamp":"2024-03-19T02:06:25.988238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:16.486859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:27.265146+0000","last_change":"2024-03-19T02:06:27.265146+0000","last_active":"2024-03-19T02:06:27.265146+0000","last_peered":"2024-03-19T02:06:27.265146+0000","last_clean":"2024-03-19T02:06:27.265146+0000","last_became_active":"2024-03-19T01:51:09.167812+0000","last_became_peered":"2024-03-19T01:51:09.167812+0000","last_unstale":"2024-03-19T02:06:27.265146+0000","last_undegraded":"2024-03-19T02:06:27.265146+0000","last_fullsized":"2024-03-19T02:06:27.265146+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:27.265006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:27.265006+0000","last_clean_scrub_stamp":"2024-03-19T02:06:27.265006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:35.834211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:33.366731+0000","last_change":"2024-03-19T02:06:33.366770+0000","last_active":"2024-03-19T02:06:33.366731+0000","last_peered":"2024-03-19T02:06:33.366731+0000","last_clean":"2024-03-19T02:06:33.366731+0000","last_became_active":"2024-03-19T01:52:12.915089+0000","last_became_peered":"2024-03-19T01:52:12.915089+0000","last_unstale":"2024-03-19T02:06:33.366731+0000","last_undegraded":"2024-03-19T02:06:33.366731+0000","last_fullsized":"2024-03-19T02:06:33.366731+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:33.366634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:33.366634+0000","last_clean_scrub_stamp":"2024-03-19T02:06:33.366634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:53.921826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:22.911264+0000","last_change":"2024-03-19T02:06:22.911299+0000","last_active":"2024-03-19T02:06:22.911264+0000","last_peered":"2024-03-19T02:06:22.911264+0000","last_clean":"2024-03-19T02:06:22.911264+0000","last_became_active":"2024-03-19T01:51:09.167744+0000","last_became_peered":"2024-03-19T01:51:09.167744+0000","last_unstale":"2024-03-19T02:06:22.911264+0000","last_undegraded":"2024-03-19T02:06:22.911264+0000","last_fullsized":"2024-03-19T02:06:22.911264+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:22.911160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:22.911160+0000","last_clean_scrub_stamp":"2024-03-19T02:06:22.911160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:21:26.132056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:34.104437+0000","last_change":"2024-03-19T02:06:34.104473+0000","last_active":"2024-03-19T02:06:34.104437+0000","last_peered":"2024-03-19T02:06:34.104437+0000","last_clean":"2024-03-19T02:06:34.104437+0000","last_became_active":"2024-03-19T01:51:09.195488+0000","last_became_peered":"2024-03-19T01:51:09.195488+0000","last_unstale":"2024-03-19T02:06:34.104437+0000","last_undegraded":"2024-03-19T02:06:34.104437+0000","last_fullsized":"2024-03-19T02:06:34.104437+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:34.104330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:34.104330+0000","last_clean_scrub_stamp":"2024-03-19T02:06:34.104330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:50:55.504189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:31.124219+0000","last_change":"2024-03-19T02:06:31.124257+0000","last_active":"2024-03-19T02:06:31.124219+0000","last_peered":"2024-03-19T02:06:31.124219+0000","last_clean":"2024-03-19T02:06:31.124219+0000","last_became_active":"2024-03-19T01:51:09.192777+0000","last_became_peered":"2024-03-19T01:51:09.192777+0000","last_unstale":"2024-03-19T02:06:31.124219+0000","last_undegraded":"2024-03-19T02:06:31.124219+0000","last_fullsized":"2024-03-19T02:06:31.124219+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:31.124115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:31.124115+0000","last_clean_scrub_stamp":"2024-03-19T02:06:31.124115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:07:23.620783+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:27.376951+0000","last_change":"2024-03-19T02:06:27.376985+0000","last_active":"2024-03-19T02:06:27.376951+0000","last_peered":"2024-03-19T02:06:27.376951+0000","last_clean":"2024-03-19T02:06:27.376951+0000","last_became_active":"2024-03-19T01:51:09.192677+0000","last_became_peered":"2024-03-19T01:51:09.192677+0000","last_unstale":"2024-03-19T02:06:27.376951+0000","last_undegraded":"2024-03-19T02:06:27.376951+0000","last_fullsized":"2024-03-19T02:06:27.376951+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:27.376850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:27.376850+0000","last_clean_scrub_stamp":"2024-03-19T02:06:27.376850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:03:58.363399+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:21.006265+0000","last_change":"2024-03-19T02:06:21.006305+0000","last_active":"2024-03-19T02:06:21.006265+0000","last_peered":"2024-03-19T02:06:21.006265+0000","last_clean":"2024-03-19T02:06:21.006265+0000","last_became_active":"2024-03-19T01:51:09.164753+0000","last_became_peered":"2024-03-19T01:51:09.164753+0000","last_unstale":"2024-03-19T02:06:21.006265+0000","last_undegraded":"2024-03-19T02:06:21.006265+0000","last_fullsized":"2024-03-19T02:06:21.006265+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:21.006153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:21.006153+0000","last_clean_scrub_stamp":"2024-03-19T02:06:21.006153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:29.575358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:26.383760+0000","last_change":"2024-03-19T02:06:26.383794+0000","last_active":"2024-03-19T02:06:26.383760+0000","last_peered":"2024-03-19T02:06:26.383760+0000","last_clean":"2024-03-19T02:06:26.383760+0000","last_became_active":"2024-03-19T01:51:09.170153+0000","last_became_peered":"2024-03-19T01:51:09.170153+0000","last_unstale":"2024-03-19T02:06:26.383760+0000","last_undegraded":"2024-03-19T02:06:26.383760+0000","last_fullsized":"2024-03-19T02:06:26.383760+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:26.383666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:26.383666+0000","last_clean_scrub_stamp":"2024-03-19T02:06:26.383666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:05.432530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:22.281933+0000","last_change":"2024-03-19T02:06:22.281968+0000","last_active":"2024-03-19T02:06:22.281933+0000","last_peered":"2024-03-19T02:06:22.281933+0000","last_clean":"2024-03-19T02:06:22.281933+0000","last_became_active":"2024-03-19T01:51:09.177996+0000","last_became_peered":"2024-03-19T01:51:09.177996+0000","last_unstale":"2024-03-19T02:06:22.281933+0000","last_undegraded":"2024-03-19T02:06:22.281933+0000","last_fullsized":"2024-03-19T02:06:22.281933+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:22.281831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:22.281831+0000","last_clean_scrub_stamp":"2024-03-19T02:06:22.281831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:26:04.561619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:29.259559+0000","last_change":"2024-03-19T02:06:29.259595+0000","last_active":"2024-03-19T02:06:29.259559+0000","last_peered":"2024-03-19T02:06:29.259559+0000","last_clean":"2024-03-19T02:06:29.259559+0000","last_became_active":"2024-03-19T01:51:09.183045+0000","last_became_peered":"2024-03-19T01:51:09.183045+0000","last_unstale":"2024-03-19T02:06:29.259559+0000","last_undegraded":"2024-03-19T02:06:29.259559+0000","last_fullsized":"2024-03-19T02:06:29.259559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:29.259450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:29.259450+0000","last_clean_scrub_stamp":"2024-03-19T02:06:29.259450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:56:21.136199+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:30.302658+0000","last_change":"2024-03-19T02:06:30.302695+0000","last_active":"2024-03-19T02:06:30.302658+0000","last_peered":"2024-03-19T02:06:30.302658+0000","last_clean":"2024-03-19T02:06:30.302658+0000","last_became_active":"2024-03-19T01:51:09.183351+0000","last_became_peered":"2024-03-19T01:51:09.183351+0000","last_unstale":"2024-03-19T02:06:30.302658+0000","last_undegraded":"2024-03-19T02:06:30.302658+0000","last_fullsized":"2024-03-19T02:06:30.302658+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:30.302550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:30.302550+0000","last_clean_scrub_stamp":"2024-03-19T02:06:30.302550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:42:50.137731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":57,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:38.551328+0000","last_change":"2024-03-19T02:06:38.551366+0000","last_active":"2024-03-19T02:06:38.551328+0000","last_peered":"2024-03-19T02:06:38.551328+0000","last_clean":"2024-03-19T02:06:38.551328+0000","last_became_active":"2024-03-19T01:52:12.922473+0000","last_became_peered":"2024-03-19T01:52:12.922473+0000","last_unstale":"2024-03-19T02:06:38.551328+0000","last_undegraded":"2024-03-19T02:06:38.551328+0000","last_fullsized":"2024-03-19T02:06:38.551328+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:38.551209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:38.551209+0000","last_clean_scrub_stamp":"2024-03-19T02:06:38.551209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:31:38.005917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:18.464703+0000","last_change":"2024-03-19T02:06:18.464745+0000","last_active":"2024-03-19T02:06:18.464703+0000","last_peered":"2024-03-19T02:06:18.464703+0000","last_clean":"2024-03-19T02:06:18.464703+0000","last_became_active":"2024-03-19T01:51:09.179541+0000","last_became_peered":"2024-03-19T01:51:09.179541+0000","last_unstale":"2024-03-19T02:06:18.464703+0000","last_undegraded":"2024-03-19T02:06:18.464703+0000","last_fullsized":"2024-03-19T02:06:18.464703+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:18.464592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:18.464592+0000","last_clean_scrub_stamp":"2024-03-19T02:06:18.464592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:49:49.944294+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:32.561207+0000","last_change":"2024-03-19T02:06:32.561248+0000","last_active":"2024-03-19T02:06:32.561207+0000","last_peered":"2024-03-19T02:06:32.561207+0000","last_clean":"2024-03-19T02:06:32.561207+0000","last_became_active":"2024-03-19T01:51:09.177627+0000","last_became_peered":"2024-03-19T01:51:09.177627+0000","last_unstale":"2024-03-19T02:06:32.561207+0000","last_undegraded":"2024-03-19T02:06:32.561207+0000","last_fullsized":"2024-03-19T02:06:32.561207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:32.561101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:32.561101+0000","last_clean_scrub_stamp":"2024-03-19T02:06:32.561101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:30.953763+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:30.094426+0000","last_change":"2024-03-19T02:06:30.094462+0000","last_active":"2024-03-19T02:06:30.094426+0000","last_peered":"2024-03-19T02:06:30.094426+0000","last_clean":"2024-03-19T02:06:30.094426+0000","last_became_active":"2024-03-19T01:51:09.173235+0000","last_became_peered":"2024-03-19T01:51:09.173235+0000","last_unstale":"2024-03-19T02:06:30.094426+0000","last_undegraded":"2024-03-19T02:06:30.094426+0000","last_fullsized":"2024-03-19T02:06:30.094426+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:30.094321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:30.094321+0000","last_clean_scrub_stamp":"2024-03-19T02:06:30.094321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:55:48.376679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":52,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:31.517729+0000","last_change":"2024-03-19T02:06:31.517766+0000","last_active":"2024-03-19T02:06:31.517729+0000","last_peered":"2024-03-19T02:06:31.517729+0000","last_clean":"2024-03-19T02:06:31.517729+0000","last_became_active":"2024-03-19T01:51:09.187703+0000","last_became_peered":"2024-03-19T01:51:09.187703+0000","last_unstale":"2024-03-19T02:06:31.517729+0000","last_undegraded":"2024-03-19T02:06:31.517729+0000","last_fullsized":"2024-03-19T02:06:31.517729+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:31.517625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:31.517625+0000","last_clean_scrub_stamp":"2024-03-19T02:06:31.517625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:55:23.266800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":34,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:25.888076+0000","last_change":"2024-03-19T02:06:25.888127+0000","last_active":"2024-03-19T02:06:25.888076+0000","last_peered":"2024-03-19T02:06:25.888076+0000","last_clean":"2024-03-19T02:06:25.888076+0000","last_became_active":"2024-03-19T01:52:12.918835+0000","last_became_peered":"2024-03-19T01:52:12.918835+0000","last_unstale":"2024-03-19T02:06:25.888076+0000","last_undegraded":"2024-03-19T02:06:25.888076+0000","last_fullsized":"2024-03-19T02:06:25.888076+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:25.887920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:25.887920+0000","last_clean_scrub_stamp":"2024-03-19T02:06:25.887920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:34.106758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:19.958444+0000","last_change":"2024-03-19T02:06:19.958491+0000","last_active":"2024-03-19T02:06:19.958444+0000","last_peered":"2024-03-19T02:06:19.958444+0000","last_clean":"2024-03-19T02:06:19.958444+0000","last_became_active":"2024-03-19T01:51:09.158609+0000","last_became_peered":"2024-03-19T01:51:09.158609+0000","last_unstale":"2024-03-19T02:06:19.958444+0000","last_undegraded":"2024-03-19T02:06:19.958444+0000","last_fullsized":"2024-03-19T02:06:19.958444+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:19.958320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:19.958320+0000","last_clean_scrub_stamp":"2024-03-19T02:06:19.958320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:32:17.201789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:17.378760+0000","last_change":"2024-03-19T02:06:17.378828+0000","last_active":"2024-03-19T02:06:17.378760+0000","last_peered":"2024-03-19T02:06:17.378760+0000","last_clean":"2024-03-19T02:06:17.378760+0000","last_became_active":"2024-03-19T01:51:09.165207+0000","last_became_peered":"2024-03-19T01:51:09.165207+0000","last_unstale":"2024-03-19T02:06:17.378760+0000","last_undegraded":"2024-03-19T02:06:17.378760+0000","last_fullsized":"2024-03-19T02:06:17.378760+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:17.378630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:17.378630+0000","last_clean_scrub_stamp":"2024-03-19T02:06:17.378630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:00:02.131538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T02:06:30.514667+0000","last_change":"2024-03-19T02:06:30.514702+0000","last_active":"2024-03-19T02:06:30.514667+0000","last_peered":"2024-03-19T02:06:30.514667+0000","last_clean":"2024-03-19T02:06:30.514667+0000","last_became_active":"2024-03-19T01:51:09.175682+0000","last_became_peered":"2024-03-19T01:51:09.175682+0000","last_unstale":"2024-03-19T02:06:30.514667+0000","last_undegraded":"2024-03-19T02:06:30.514667+0000","last_fullsized":"2024-03-19T02:06:30.514667+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:06:30.514562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:06:30.514562+0000","last_clean_scrub_stamp":"2024-03-19T02:06:30.514562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:35:39.983930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":209,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509643,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":1011712,"data_stored":759262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509645,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":409600,"data_stored":171426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509644,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":405504,"data_stored":168124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509645,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31576,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340264,"statfs":{"total":96636764160,"available":96604430336,"internally_reserved":0,"allocated":409600,"data_stored":168326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509644,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":1003520,"data_stored":758036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542348,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":430080,"data_stored":170726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542349,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31580,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":413696,"data_stored":171460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542348,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31580,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":413696,"data_stored":168256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:07:03.902 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-19T02:07:04.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.044+0000 7f17ab78d640 1 Processor -- start 2024-03-19T02:07:04.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.044+0000 7f17ab78d640 1 -- start start 2024-03-19T02:07:04.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.044+0000 7f17ab78d640 1 --2- >> v2:172.21.15.84:3300/0 conn(0x7f17a40c96b0 0x7f17a4070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:07:04.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.044+0000 7f17ab78d640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f17a40c9a90 0x7f17a40c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:07:04.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.044+0000 7f17ab78d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17a415e3a0 con 0x7f17a41540c0 2024-03-19T02:07:04.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.044+0000 7f17ab78d640 1 -- --> v2:172.21.15.55:3300/0 -- mon_getmap magic: 0 -- 0x7f17a415e680 con 0x7f17a40c9a90 2024-03-19T02:07:04.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.044+0000 7f17ab78d640 1 -- --> v2:172.21.15.84:3300/0 -- mon_getmap magic: 0 -- 0x7f17a415a330 con 0x7f17a40c96b0 2024-03-19T02:07:04.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.044+0000 7f17a8d28640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f17a40c9a90 0x7f17a40c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:07:04.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.048+0000 7f17a8d28640 1 --2- >> v2:172.21.15.55:3300/0 conn(0x7f17a40c9a90 0x7f17a40c73c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:44790/0 (socket says 172.21.15.55:44790) 2024-03-19T02:07:04.048 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.048+0000 7f17a8d28640 1 -- 172.21.15.55:0/1295901644 learned_addr learned my addr 172.21.15.55:0/1295901644 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:07:04.049 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.048+0000 7f17a9529640 1 --2- 172.21.15.55:0/1295901644 >> v2:172.21.15.84:3300/0 conn(0x7f17a40c96b0 0x7f17a4070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:07:04.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.048+0000 7f17a8d28640 1 -- 172.21.15.55:0/1295901644 >> v2:172.21.15.84:3300/0 conn(0x7f17a40c96b0 msgr2=0x7f17a4070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:07:04.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.048+0000 7f17a8d28640 1 --2- 172.21.15.55:0/1295901644 >> v2:172.21.15.84:3300/0 conn(0x7f17a40c96b0 0x7f17a4070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:07:04.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.048+0000 7f17a8d28640 1 -- 172.21.15.55:0/1295901644 >> v1:172.21.15.84:6789/0 conn(0x7f17a41540c0 legacy=0x7f17a415d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:07:04.052 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.048+0000 7f17a8d28640 1 -- 172.21.15.55:0/1295901644 --> v2:172.21.15.55:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a415a610 con 0x7f17a40c9a90 2024-03-19T02:07:04.055 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.052+0000 7f17a8d28640 1 --2- 172.21.15.55:0/1295901644 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17a40c9a90 0x7f17a40c73c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f17a4072b40 tx=0x7f179401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe30e6748014120c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:07:04.059 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.056+0000 7f179bfff640 1 -- 172.21.15.55:0/1295901644 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17940b2070 con 0x7f17a40c9a90 2024-03-19T02:07:04.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.060+0000 7f179bfff640 1 -- 172.21.15.55:0/1295901644 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17940ad680 con 0x7f17a40c9a90 2024-03-19T02:07:04.061 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.060+0000 7f179bfff640 1 -- 172.21.15.55:0/1295901644 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17940ada50 con 0x7f17a40c9a90 2024-03-19T02:07:04.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.060+0000 7f17ab78d640 1 -- 172.21.15.55:0/1295901644 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17a40c9a90 msgr2=0x7f17a40c73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:07:04.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.060+0000 7f17ab78d640 1 --2- 172.21.15.55:0/1295901644 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17a40c9a90 0x7f17a40c73c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f17a4072b40 tx=0x7f179401c490 comp rx=0 tx=0).stop 2024-03-19T02:07:04.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.060+0000 7f17ab78d640 1 -- 172.21.15.55:0/1295901644 shutdown_connections 2024-03-19T02:07:04.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.060+0000 7f17ab78d640 1 --2- 172.21.15.55:0/1295901644 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17a40c9a90 0x7f17a40c73c0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:07:04.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.060+0000 7f17ab78d640 1 --2- 172.21.15.55:0/1295901644 >> v2:172.21.15.84:3300/0 conn(0x7f17a40c96b0 0x7f17a4070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:07:04.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.060+0000 7f17ab78d640 1 -- 172.21.15.55:0/1295901644 >> 172.21.15.55:0/1295901644 conn(0x7f17a40cc9c0 msgr2=0x7f17a40ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:07:04.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.060+0000 7f17ab78d640 1 -- 172.21.15.55:0/1295901644 shutdown_connections 2024-03-19T02:07:04.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.060+0000 7f17ab78d640 1 -- 172.21.15.55:0/1295901644 wait complete. 2024-03-19T02:07:04.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.060+0000 7f17ab78d640 1 Processor -- start 2024-03-19T02:07:04.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.060+0000 7f17ab78d640 1 -- start start 2024-03-19T02:07:04.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.060+0000 7f17ab78d640 1 --2- >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17a40c96b0 0x7f17a4158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:07:04.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.060+0000 7f17ab78d640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f17a40c9a90 0x7f17a41499e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:07:04.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.060+0000 7f17ab78d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17a41540c0 0x7f17a4141690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:07:04.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.060+0000 7f17ab78d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f17a40c7900 con 0x7f17a41540c0 2024-03-19T02:07:04.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.060+0000 7f17ab78d640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f17a41548e0 con 0x7f17a40c9a90 2024-03-19T02:07:04.064 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.060+0000 7f17ab78d640 1 -- --> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] -- mon_getmap magic: 0 -- 0x7f17a40c7a80 con 0x7f17a40c96b0 2024-03-19T02:07:04.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.064+0000 7f17a9d2a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17a41540c0 0x7f17a4141690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:07:04.065 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.064+0000 7f17a8d28640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f17a40c9a90 0x7f17a41499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:07:04.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.064+0000 7f17a9d2a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17a41540c0 0x7f17a4141690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:44800/0 (socket says 172.21.15.55:44800) 2024-03-19T02:07:04.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.064+0000 7f17a9d2a640 1 -- 172.21.15.55:0/1804444267 learned_addr learned my addr 172.21.15.55:0/1804444267 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-03-19T02:07:04.066 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.064+0000 7f17a9529640 1 --2- 172.21.15.55:0/1804444267 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17a40c96b0 0x7f17a4158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:07:04.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.072+0000 7f17a9d2a640 1 -- 172.21.15.55:0/1804444267 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17a40c96b0 msgr2=0x7f17a4158130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:07:04.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.072+0000 7f17a9d2a640 1 --2- 172.21.15.55:0/1804444267 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17a40c96b0 0x7f17a4158130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:07:04.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.072+0000 7f17a9d2a640 1 -- 172.21.15.55:0/1804444267 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f17a40c9a90 msgr2=0x7f17a41499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:07:04.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.072+0000 7f17a9d2a640 1 --2- 172.21.15.55:0/1804444267 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f17a40c9a90 0x7f17a41499e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:07:04.071 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.072+0000 7f17a9d2a640 1 -- 172.21.15.55:0/1804444267 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a4159dc0 con 0x7f17a41540c0 2024-03-19T02:07:04.074 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.076+0000 7f17a9d2a640 1 --2- 172.21.15.55:0/1804444267 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17a41540c0 0x7f17a4141690 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f179c00c730 tx=0x7f179c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:07:04.079 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.080+0000 7f1799ffb640 1 -- 172.21.15.55:0/1804444267 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179c003b80 con 0x7f17a41540c0 2024-03-19T02:07:04.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.080+0000 7f1799ffb640 1 -- 172.21.15.55:0/1804444267 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f179c003d20 con 0x7f17a41540c0 2024-03-19T02:07:04.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.080+0000 7f1799ffb640 1 -- 172.21.15.55:0/1804444267 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179c00aa30 con 0x7f17a41540c0 2024-03-19T02:07:04.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.080+0000 7f17ab78d640 1 -- 172.21.15.55:0/1804444267 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17a40cbc90 con 0x7f17a41540c0 2024-03-19T02:07:04.080 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.084+0000 7f17ab78d640 1 -- 172.21.15.55:0/1804444267 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17a40cb5e0 con 0x7f17a41540c0 2024-03-19T02:07:04.081 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.084+0000 7f17ab78d640 1 -- 172.21.15.55:0/1804444267 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17a4068a40 con 0x7f17a41540c0 2024-03-19T02:07:04.109 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.112+0000 7f1799ffb640 1 -- 172.21.15.55:0/1804444267 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) ==== 87083+0+0 (secure 0 0 0) 0x7f179c00abd0 con 0x7f17a41540c0 2024-03-19T02:07:04.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.112+0000 7f1799ffb640 1 --2- 172.21.15.55:0/1804444267 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f17880676b0 0x7f1788069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:07:04.110 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.112+0000 7f17a9529640 1 --2- 172.21.15.55:0/1804444267 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f17880676b0 0x7f1788069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:07:04.111 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.112+0000 7f17a9529640 1 --2- 172.21.15.55:0/1804444267 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f17880676b0 0x7f1788069b70 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f17a00079c0 tx=0x7f17a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:07:04.412 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.412+0000 7f1799ffb640 1 -- 172.21.15.55:0/1804444267 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6937+0+0 (secure 0 0 0) 0x7f179c054790 con 0x7f17a41540c0 2024-03-19T02:07:04.413 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.416+0000 7f1799ffb640 1 -- 172.21.15.55:0/1804444267 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177849 (secure 0 0 0) 0x7f179c0566f0 con 0x7f17a41540c0 2024-03-19T02:07:04.582 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.584+0000 7f17ab78d640 1 -- 172.21.15.55:0/1804444267 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f17a4151cf0 con 0x7f17a41540c0 2024-03-19T02:07:04.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.728+0000 7f1799ffb640 1 -- 172.21.15.55:0/1804444267 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) ==== 123+0+0 (secure 0 0 0) 0x7f179c05a790 con 0x7f17a41540c0 2024-03-19T02:07:04.727 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.728+0000 7f1799ffb640 1 -- 172.21.15.55:0/1804444267 <== mon.0 v2:172.21.15.55:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f179c00ce40 con 0x7f17a41540c0 2024-03-19T02:07:04.729 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.732+0000 7f17ab78d640 1 -- 172.21.15.55:0/1804444267 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f17880676b0 msgr2=0x7f1788069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:07:04.729 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.732+0000 7f17ab78d640 1 --2- 172.21.15.55:0/1804444267 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f17880676b0 0x7f1788069b70 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f17a00079c0 tx=0x7f17a0008040 comp rx=0 tx=0).stop 2024-03-19T02:07:04.729 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.732+0000 7f17ab78d640 1 -- 172.21.15.55:0/1804444267 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17a41540c0 msgr2=0x7f17a4141690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:07:04.729 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.732+0000 7f17ab78d640 1 --2- 172.21.15.55:0/1804444267 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17a41540c0 0x7f17a4141690 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f179c00c730 tx=0x7f179c00cbf0 comp rx=0 tx=0).stop 2024-03-19T02:07:04.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.732+0000 7f17ab78d640 1 -- 172.21.15.55:0/1804444267 shutdown_connections 2024-03-19T02:07:04.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.732+0000 7f17ab78d640 1 --2- 172.21.15.55:0/1804444267 >> [v2:172.21.15.84:6800/998533783,v1:172.21.15.84:6801/998533783] conn(0x7f17880676b0 0x7f1788069b70 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:07:04.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.732+0000 7f17ab78d640 1 --2- 172.21.15.55:0/1804444267 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17a41540c0 0x7f17a4141690 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:07:04.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.732+0000 7f17ab78d640 1 --2- 172.21.15.55:0/1804444267 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f17a40c9a90 0x7f17a41499e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:07:04.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.732+0000 7f17ab78d640 1 --2- 172.21.15.55:0/1804444267 >> [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] conn(0x7f17a40c96b0 0x7f17a4158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:07:04.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.732+0000 7f17ab78d640 1 -- 172.21.15.55:0/1804444267 >> 172.21.15.55:0/1804444267 conn(0x7f17a40cc9c0 msgr2=0x7f17a40c5e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:07:04.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.732+0000 7f17ab78d640 1 -- 172.21.15.55:0/1804444267 shutdown_connections 2024-03-19T02:07:04.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-19T02:07:04.732+0000 7f17ab78d640 1 -- 172.21.15.55:0/1804444267 wait complete. 2024-03-19T02:07:04.746 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-19T02:07:04.747 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-19T02:07:04.747 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-19T02:07:04.748 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T02:07:11.489 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:07:11.492 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:07:39.877 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:17:52.035 27555== Warning: unimplemented fcntl command: 1036 2024-03-19T02:07:39.884 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:17:52.043 27555== Warning: unimplemented fcntl command: 1036 2024-03-19T02:07:39.933 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:17:52.091 27555== Warning: unimplemented fcntl command: 1036 2024-03-19T02:07:39.981 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:17:52.076 27627== Warning: unimplemented fcntl command: 1036 2024-03-19T02:07:39.992 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:17:52.087 27627== Warning: unimplemented fcntl command: 1036 2024-03-19T02:07:40.002 INFO:tasks.ceph.mon.b.smithi084.stderr:==00:00:17:52.097 27626== Warning: unimplemented fcntl command: 1036 2024-03-19T02:07:40.010 INFO:tasks.ceph.mon.b.smithi084.stderr:==00:00:17:52.105 27626== Warning: unimplemented fcntl command: 1036 2024-03-19T02:07:40.029 INFO:tasks.ceph.mon.c.smithi084.stderr:==00:00:17:52.124 27627== Warning: unimplemented fcntl command: 1036 2024-03-19T02:07:40.062 INFO:tasks.ceph.mon.b.smithi084.stderr:==00:00:17:52.157 27626== Warning: unimplemented fcntl command: 1036 2024-03-19T02:07:41.577 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:07:41.580 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:08:11.629 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:08:11.632 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:08:41.679 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:08:41.683 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:09:11.796 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:09:11.798 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:09:41.842 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:09:41.846 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:10:11.960 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:10:11.963 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:10:42.030 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:10:42.033 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:11:12.079 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:11:12.082 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:11:28.702 INFO:tasks.ceph.mds.a.smithi055.stderr:2024-03-19T02:08:22.578+0000 b9ed640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.a.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-mds -f --cluster ceph -i a (PID: 32306) UID: 0 2024-03-19T02:11:28.703 INFO:tasks.ceph.mds.a.smithi055.stderr:2024-03-19T02:08:22.582+0000 b9ed640 -1 mds.a *** got signal Terminated *** 2024-03-19T02:11:42.196 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:11:42.199 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:12:03.817 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-19T02:12:03.817 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-19T02:12:03.818 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T02:12:03.821 INFO:tasks.ceph.osd.0.smithi055.stderr:2024-03-19T02:12:03.818+0000 7f80fd0c8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term ceph-osd -f --cluster ceph -i 0 (PID: 27670) UID: 0 2024-03-19T02:12:03.821 INFO:tasks.ceph.osd.0.smithi055.stderr:2024-03-19T02:12:03.818+0000 7f80fd0c8640 -1 osd.0 35 *** Got signal Terminated *** 2024-03-19T02:12:07.830 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-03-19T02:12:09.918 INFO:tasks.ceph.osd.0:Stopped 2024-03-19T02:12:09.919 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-19T02:12:09.919 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T02:12:09.920 INFO:tasks.ceph.osd.1.smithi055.stderr:2024-03-19T02:12:09.918+0000 7f7f83fc9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term ceph-osd -f --cluster ceph -i 1 (PID: 27672) UID: 0 2024-03-19T02:12:09.920 INFO:tasks.ceph.osd.1.smithi055.stderr:2024-03-19T02:12:09.918+0000 7f7f83fc9640 -1 osd.1 37 *** Got signal Terminated *** 2024-03-19T02:12:12.241 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:12:12.244 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:12:14.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-03-19T02:12:14.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-03-19T02:12:16.021 INFO:tasks.ceph.osd.1:Stopped 2024-03-19T02:12:16.021 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-19T02:12:16.021 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T02:12:16.022 INFO:tasks.ceph.osd.2.smithi055.stderr:2024-03-19T02:12:16.022+0000 7fe4e8b96640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term ceph-osd -f --cluster ceph -i 2 (PID: 27674) UID: 0 2024-03-19T02:12:16.022 INFO:tasks.ceph.osd.2.smithi055.stderr:2024-03-19T02:12:16.022+0000 7fe4e8b96640 -1 osd.2 39 *** Got signal Terminated *** 2024-03-19T02:12:21.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-03-19T02:12:21.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-03-19T02:12:22.123 INFO:tasks.ceph.osd.2:Stopped 2024-03-19T02:12:22.123 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-19T02:12:22.123 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T02:12:22.124 INFO:tasks.ceph.osd.3.smithi055.stderr:2024-03-19T02:12:22.126+0000 7faace4d0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term ceph-osd -f --cluster ceph -i 3 (PID: 27676) UID: 0 2024-03-19T02:12:22.124 INFO:tasks.ceph.osd.3.smithi055.stderr:2024-03-19T02:12:22.126+0000 7faace4d0640 -1 osd.3 41 *** Got signal Terminated *** 2024-03-19T02:12:27.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-03-19T02:12:27.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-03-19T02:12:28.225 INFO:tasks.ceph.osd.3:Stopped 2024-03-19T02:12:28.225 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-19T02:12:28.225 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T02:12:28.226 INFO:tasks.ceph.osd.4.smithi084.stderr:2024-03-19T02:12:28.226+0000 7ff7113b8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term ceph-osd -f --cluster ceph -i 4 (PID: 28025) UID: 0 2024-03-19T02:12:28.226 INFO:tasks.ceph.osd.4.smithi084.stderr:2024-03-19T02:12:28.226+0000 7ff7113b8640 -1 osd.4 43 *** Got signal Terminated *** 2024-03-19T02:12:34.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-03-19T02:12:34.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-03-19T02:12:34.326 INFO:tasks.ceph.osd.4:Stopped 2024-03-19T02:12:34.326 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-19T02:12:34.326 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T02:12:34.327 INFO:tasks.ceph.osd.5.smithi084.stderr:2024-03-19T02:12:34.326+0000 7fec45384640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term ceph-osd -f --cluster ceph -i 5 (PID: 28023) UID: 0 2024-03-19T02:12:34.327 INFO:tasks.ceph.osd.5.smithi084.stderr:2024-03-19T02:12:34.326+0000 7fec45384640 -1 osd.5 45 *** Got signal Terminated *** 2024-03-19T02:12:40.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-03-19T02:12:40.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-03-19T02:12:40.428 INFO:tasks.ceph.osd.5:Stopped 2024-03-19T02:12:40.428 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-19T02:12:40.428 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T02:12:40.429 INFO:tasks.ceph.osd.6.smithi084.stderr:2024-03-19T02:12:40.426+0000 7f3c22cba640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term ceph-osd -f --cluster ceph -i 6 (PID: 28027) UID: 0 2024-03-19T02:12:40.429 INFO:tasks.ceph.osd.6.smithi084.stderr:2024-03-19T02:12:40.426+0000 7f3c22cba640 -1 osd.6 47 *** Got signal Terminated *** 2024-03-19T02:12:42.361 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:12:42.363 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:12:46.529 INFO:tasks.ceph.osd.6:Stopped 2024-03-19T02:12:46.530 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-19T02:12:46.530 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T02:12:46.531 INFO:tasks.ceph.osd.7.smithi084.stderr:2024-03-19T02:12:46.530+0000 7f1e995ed640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term ceph-osd -f --cluster ceph -i 7 (PID: 28029) UID: 0 2024-03-19T02:12:46.531 INFO:tasks.ceph.osd.7.smithi084.stderr:2024-03-19T02:12:46.530+0000 7f1e995ed640 -1 osd.7 47 *** Got signal Terminated *** 2024-03-19T02:12:46.600 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-03-19T02:12:46.600 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-03-19T02:12:52.631 INFO:tasks.ceph.osd.7:Stopped 2024-03-19T02:12:52.632 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-19T02:12:52.632 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-19T02:12:52.632 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T02:12:52.656 INFO:tasks.ceph.mgr.y:Stopped 2024-03-19T02:12:52.657 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-19T02:12:52.657 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T02:12:52.695 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-03-19T02:12:52.695 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-03-19T02:12:52.695 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-03-19T02:12:52.695 INFO:tasks.ceph.mgr.x:Stopped 2024-03-19T02:12:52.695 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-19T02:12:52.696 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-19T02:12:52.696 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T02:12:52.715 INFO:tasks.ceph.mon.a.smithi055.stderr:2024-03-19T02:12:52.701+0000 a38e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 27552) UID: 0 2024-03-19T02:12:52.715 INFO:tasks.ceph.mon.a.smithi055.stderr:2024-03-19T02:12:52.705+0000 a38e640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-19T02:12:58.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-03-19T02:12:58.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-03-19T02:12:58.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2024-03-19T02:12:58.797 INFO:tasks.ceph.mon.a:Stopped 2024-03-19T02:12:58.797 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-19T02:12:58.797 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T02:12:58.835 INFO:tasks.ceph.mon.b.smithi084.stderr:2024-03-19T02:12:58.805+0000 a38e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 27619) UID: 0 2024-03-19T02:12:58.836 INFO:tasks.ceph.mon.b.smithi084.stderr:2024-03-19T02:12:58.809+0000 a38e640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-03-19T02:13:04.308 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-03-19T02:13:12.448 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:13:12.450 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:13:42.501 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:13:42.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:13:50.659 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-03-19T02:13:53.711 INFO:tasks.ceph.mon.b:Stopped 2024-03-19T02:13:53.712 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-19T02:13:53.712 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T02:13:53.750 INFO:tasks.ceph.mon.c.smithi084.stderr:2024-03-19T02:13:53.720+0000 a38e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 27622) UID: 0 2024-03-19T02:13:53.750 INFO:tasks.ceph.mon.c.smithi084.stderr:2024-03-19T02:13:53.720+0000 a38e640 -1 mon.c@2(probing) e1 *** Got Signal Terminated *** 2024-03-19T02:13:56.264 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-03-19T02:13:56.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-03-19T02:13:59.814 INFO:tasks.ceph.mon.c:Stopped 2024-03-19T02:13:59.814 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-19T02:13:59.814 DEBUG:teuthology.orchestra.run.smithi055:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v MON_DOWN | head -n 1 2024-03-19T02:13:59.832 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-19T02:13:59.832 DEBUG:teuthology.orchestra.run.smithi055:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-19T02:14:00.078 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-19T02:14:00.079 DEBUG:teuthology.orchestra.run.smithi055:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-19T02:14:00.275 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-19T02:14:00.275 DEBUG:teuthology.orchestra.run.smithi055:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-19T02:14:00.584 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-19T02:14:00.584 DEBUG:teuthology.orchestra.run.smithi055:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-19T02:14:00.752 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-19T02:14:00.752 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-19T02:14:00.999 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-19T02:14:00.999 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-19T02:14:01.138 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-19T02:14:01.138 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-19T02:14:01.295 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-19T02:14:01.295 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-19T02:14:01.421 INFO:tasks.ceph:Archiving mon data... 2024-03-19T02:14:01.423 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609330/data/mon.a.tgz 2024-03-19T02:14:01.423 DEBUG:teuthology.orchestra.run.smithi055:> mktemp 2024-03-19T02:14:01.429 INFO:teuthology.orchestra.run.smithi055.stdout:/tmp/tmp.mW4o84H29o 2024-03-19T02:14:01.429 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.mW4o84H29o 2024-03-19T02:14:01.552 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0666 /tmp/tmp.mW4o84H29o 2024-03-19T02:14:01.612 DEBUG:teuthology.orchestra.remote:smithi055:/tmp/tmp.mW4o84H29o is 322KB 2024-03-19T02:14:01.670 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.mW4o84H29o 2024-03-19T02:14:01.675 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609330/data/mon.b.tgz 2024-03-19T02:14:01.675 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2024-03-19T02:14:01.680 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.eARvwvsQhN 2024-03-19T02:14:01.681 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.eARvwvsQhN 2024-03-19T02:14:01.814 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.eARvwvsQhN 2024-03-19T02:14:01.870 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-03-19T02:14:01.872 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.eARvwvsQhN is 344KB 2024-03-19T02:14:01.927 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.eARvwvsQhN 2024-03-19T02:14:01.932 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609330/data/mon.c.tgz 2024-03-19T02:14:01.932 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2024-03-19T02:14:01.978 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.dXz6pX0kpM 2024-03-19T02:14:01.978 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.dXz6pX0kpM 2024-03-19T02:14:02.112 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.dXz6pX0kpM 2024-03-19T02:14:02.173 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.dXz6pX0kpM is 344KB 2024-03-19T02:14:02.231 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.dXz6pX0kpM 2024-03-19T02:14:02.235 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-19T02:14:02.235 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-19T02:14:02.237 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-19T02:14:02.350 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 48, column 1 (, line 48) 2024-03-19T02:14:02.359 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 48, column 1 (, line 48) 2024-03-19T02:14:02.368 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 48, column 1 (, line 48) 2024-03-19T02:14:02.385 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun 2024-03-19T02:14:02.442 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 48, column 1 (, line 48) 2024-03-19T02:14:02.449 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 48, column 1 (, line 48) 2024-03-19T02:14:02.460 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 48, column 1 (, line 48) 2024-03-19T02:14:02.484 INFO:tasks.ceph:Archiving crash dumps... 2024-03-19T02:14:02.486 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609330/remote/smithi055/crash 2024-03-19T02:14:02.487 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-19T02:14:02.500 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609330/remote/smithi084/crash 2024-03-19T02:14:02.500 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-19T02:14:02.510 INFO:tasks.ceph:Shutting down logrotate 2024-03-19T02:14:02.510 DEBUG:teuthology.orchestra.run.smithi055:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:14:02.545 DEBUG:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-19T02:14:02.585 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-19T02:14:02.635 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-19T02:14:02.644 INFO:tasks.ceph:Compressing logs... 2024-03-19T02:14:02.644 DEBUG:teuthology.orchestra.run.smithi055:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-19T02:14:02.678 DEBUG:teuthology.orchestra.run.smithi084:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-19T02:14:02.694 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42880.log 2024-03-19T02:14:02.695 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40020.log 2024-03-19T02:14:02.695 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41645.log 2024-03-19T02:14:02.695 INFO:teuthology.orchestra.run.smithi055.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42880.log.gz 2024-03-19T02:14:02.695 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32344.log 2024-03-19T02:14:02.700 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37422.log 2024-03-19T02:14:02.700 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41645.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40020.log.gz 2024-03-19T02:14:02.700 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.32344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36715.log 2024-03-19T02:14:02.700 INFO:teuthology.orchestra.run.smithi055.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41645.log.gz 2024-03-19T02:14:02.700 INFO:teuthology.orchestra.run.smithi055.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32344.log.gz 2024-03-19T02:14:02.700 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32594.log 2024-03-19T02:14:02.700 INFO:teuthology.orchestra.run.smithi055.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37422.log.gz 2024-03-19T02:14:02.700 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41723.log 2024-03-19T02:14:02.700 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35112.log 2024-03-19T02:14:02.700 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.32594.log: /var/log/ceph/ceph-client.admin.41723.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36715.log.gz 2024-03-19T02:14:02.700 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41723.log.gz 2024-03-19T02:14:02.700 INFO:teuthology.orchestra.run.smithi055.stderr: /var/log/ceph/ceph-client.admin.42671.log 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32594.log.gz 2024-03-19T02:14:02.700 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-19T02:14:02.701 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39706.log 2024-03-19T02:14:02.701 INFO:teuthology.orchestra.run.smithi055.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35112.log.gz 2024-03-19T02:14:02.701 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.42671.log: /var/log/ceph/ceph-client.admin.40281.log 2024-03-19T02:14:02.701 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42063.log 2024-03-19T02:14:02.701 INFO:teuthology.orchestra.run.smithi055.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42671.log.gz 2024-03-19T02:14:02.701 INFO:teuthology.orchestra.run.smithi055.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39706.log.gz 2024-03-19T02:14:02.701 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38286.log 2024-03-19T02:14:02.701 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40281.log: /var/log/ceph/ceph-client.admin.42063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40675.log 2024-03-19T02:14:02.701 INFO:teuthology.orchestra.run.smithi055.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42063.log.gz 2024-03-19T02:14:02.701 INFO:teuthology.orchestra.run.smithi055.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40281.log.gz 2024-03-19T02:14:02.701 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40072.log 2024-03-19T02:14:02.710 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38286.log: /var/log/ceph/ceph-client.admin.40675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37056.log 2024-03-19T02:14:02.710 INFO:teuthology.orchestra.run.smithi055.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40675.log.gz 2024-03-19T02:14:02.710 INFO:teuthology.orchestra.run.smithi055.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38286.log.gz 2024-03-19T02:14:02.710 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36453.log 2024-03-19T02:14:02.710 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40072.log: /var/log/ceph/ceph-client.admin.37056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40831.log 2024-03-19T02:14:02.710 INFO:teuthology.orchestra.run.smithi055.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37056.log.gz 2024-03-19T02:14:02.710 INFO:teuthology.orchestra.run.smithi055.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40072.log.gz 2024-03-19T02:14:02.710 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-19T02:14:02.710 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36453.log: /var/log/ceph/ceph-client.admin.40831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40229.log 2024-03-19T02:14:02.710 INFO:teuthology.orchestra.run.smithi055.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40831.log.gz 2024-03-19T02:14:02.710 INFO:teuthology.orchestra.run.smithi055.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36453.log.gz 2024-03-19T02:14:02.711 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43324.log 2024-03-19T02:14:02.711 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39312.log 2024-03-19T02:14:02.711 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40229.log: /var/log/ceph/ceph-client.admin.43324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33577.log 2024-03-19T02:14:02.711 INFO:teuthology.orchestra.run.smithi055.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40229.log.gz 2024-03-19T02:14:02.711 INFO:teuthology.orchestra.run.smithi055.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43324.log.gz/var/log/ceph/ceph-client.admin.39312.log: 2024-03-19T02:14:02.711 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38682.log 2024-03-19T02:14:02.711 INFO:teuthology.orchestra.run.smithi055.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39312.log.gz 2024-03-19T02:14:02.711 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43246.log 2024-03-19T02:14:02.711 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41302.log 2024-03-19T02:14:02.711 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38682.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33577.log.gz 2024-03-19T02:14:02.711 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.43246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42249.log 2024-03-19T02:14:02.711 INFO:teuthology.orchestra.run.smithi055.stderr: 88.3%/var/log/ceph/ceph-client.admin.41302.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43246.log.gz 2024-03-19T02:14:02.711 INFO:teuthology.orchestra.run.smithi055.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38682.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43508.log 2024-03-19T02:14:02.711 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-19T02:14:02.711 INFO:teuthology.orchestra.run.smithi055.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41302.log.gz 2024-03-19T02:14:02.712 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34496.log 2024-03-19T02:14:02.712 INFO:teuthology.orchestra.run.smithi055.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42249.log.gz 2024-03-19T02:14:02.712 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.43508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36845.log 2024-03-19T02:14:02.712 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34496.log: gzip -5 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43508.log.gz 2024-03-19T02:14:02.712 INFO:teuthology.orchestra.run.smithi055.stderr: --verbose 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34496.log.gz 2024-03-19T02:14:02.712 INFO:teuthology.orchestra.run.smithi055.stderr: -- /var/log/ceph/ceph-client.admin.36767.log 2024-03-19T02:14:02.712 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36845.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36845.log.gz 2024-03-19T02:14:02.712 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42697.log 2024-03-19T02:14:02.712 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40884.log 2024-03-19T02:14:02.712 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36767.log: /var/log/ceph/ceph-client.admin.42697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40753.log 2024-03-19T02:14:02.712 INFO:teuthology.orchestra.run.smithi055.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36767.log.gz 2024-03-19T02:14:02.712 INFO:teuthology.orchestra.run.smithi055.stderr: 88.0%/var/log/ceph/ceph-client.admin.40884.log: -- replaced with /var/log/ceph/ceph-client.admin.42697.log.gz 2024-03-19T02:14:02.712 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35138.log 2024-03-19T02:14:02.712 INFO:teuthology.orchestra.run.smithi055.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40884.log.gz 2024-03-19T02:14:02.712 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33103.log 2024-03-19T02:14:02.713 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose 87.3% -- -- replaced with /var/log/ceph/ceph-client.admin.40753.log.gz /var/log/ceph/ceph-client.admin.34486.log 2024-03-19T02:14:02.713 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-19T02:14:02.713 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35138.log: /var/log/ceph/ceph-client.admin.33103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33053.log 2024-03-19T02:14:02.713 INFO:teuthology.orchestra.run.smithi055.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35138.log.gz 2024-03-19T02:14:02.713 INFO:teuthology.orchestra.run.smithi055.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33103.log.gz 2024-03-19T02:14:02.713 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42645.log 2024-03-19T02:14:02.713 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-19T02:14:02.713 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33053.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34486.log.gz 2024-03-19T02:14:02.713 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33394.log 2024-03-19T02:14:02.713 INFO:teuthology.orchestra.run.smithi055.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42645.log.gz 2024-03-19T02:14:02.713 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36401.log 2024-03-19T02:14:02.713 INFO:teuthology.orchestra.run.smithi055.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.33053.log.gz 2024-03-19T02:14:02.713 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34223.log 2024-03-19T02:14:02.713 INFO:teuthology.orchestra.run.smithi055.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33394.log.gz 2024-03-19T02:14:02.713 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42037.log 2024-03-19T02:14:02.719 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27764.log 2024-03-19T02:14:02.719 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27845.log 2024-03-19T02:14:02.719 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.27764.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-19T02:14:02.719 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.27845.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.27764.log.gz 2024-03-19T02:14:02.719 INFO:teuthology.orchestra.run.smithi084.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-19T02:14:02.719 INFO:teuthology.orchestra.run.smithi084.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.27845.log.gz 2024-03-19T02:14:02.719 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-19T02:14:02.719 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-19T02:14:02.719 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2024-03-19T02:14:02.719 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-19T02:14:02.719 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-mds.f.log: 90.6% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2024-03-19T02:14:02.719 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27795.log 2024-03-19T02:14:02.723 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph.log: 93.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-19T02:14:02.723 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-19T02:14:02.723 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.27795.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.27795.log.gz 2024-03-19T02:14:02.723 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27894.log 2024-03-19T02:14:02.737 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27933.log 2024-03-19T02:14:02.737 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.27894.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.27894.log.gz 2024-03-19T02:14:02.737 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-19T02:14:02.737 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.27933.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27933.log.gz 2024-03-19T02:14:02.739 INFO:teuthology.orchestra.run.smithi055.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36401.log.gz 2024-03-19T02:14:02.739 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35694.log 2024-03-19T02:14:02.739 INFO:teuthology.orchestra.run.smithi055.stderr:gzip 88.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.34223.log.gz -- 2024-03-19T02:14:02.739 INFO:teuthology.orchestra.run.smithi055.stderr: /var/log/ceph/ceph-client.admin.43038.log 2024-03-19T02:14:02.739 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42037.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42037.log.gz 2024-03-19T02:14:02.739 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38524.log 2024-03-19T02:14:02.739 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35530.log 2024-03-19T02:14:02.739 INFO:teuthology.orchestra.run.smithi055.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35694.log.gz 2024-03-19T02:14:02.740 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.43038.log: /var/log/ceph/ceph-client.admin.38524.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43038.log.gz 2024-03-19T02:14:02.740 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39258.log 2024-03-19T02:14:02.740 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39024.log 2024-03-19T02:14:02.740 INFO:teuthology.orchestra.run.smithi055.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38524.log.gz 2024-03-19T02:14:02.740 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35530.log: /var/log/ceph/ceph-client.admin.39258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32315.log 2024-03-19T02:14:02.740 INFO:teuthology.orchestra.run.smithi055.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35530.log.gz 2024-03-19T02:14:02.740 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39024.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39258.log.gz 2024-03-19T02:14:02.740 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42277.log 2024-03-19T02:14:02.740 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.32315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35242.log 2024-03-19T02:14:02.740 INFO:teuthology.orchestra.run.smithi055.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39024.log.gz 2024-03-19T02:14:02.740 INFO:teuthology.orchestra.run.smithi055.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36689.log 2024-03-19T02:14:02.740 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42277.log: -- replaced with /var/log/ceph/ceph-client.admin.32315.log.gz 2024-03-19T02:14:02.740 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35268.log 2024-03-19T02:14:02.740 INFO:teuthology.orchestra.run.smithi055.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42277.log.gz 2024-03-19T02:14:02.740 INFO:teuthology.orchestra.run.smithi055.stderr:gzip/var/log/ceph/ceph-client.admin.36689.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34608.log 2024-03-19T02:14:02.741 INFO:teuthology.orchestra.run.smithi055.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35242.log.gz 2024-03-19T02:14:02.741 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35268.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.36689.log.gz 2024-03-19T02:14:02.741 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34925.log 2024-03-19T02:14:02.741 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37108.log 2024-03-19T02:14:02.741 INFO:teuthology.orchestra.run.smithi055.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35268.log.gz 2024-03-19T02:14:02.741 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34608.log: /var/log/ceph/ceph-client.admin.34925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42591.log 2024-03-19T02:14:02.741 INFO:teuthology.orchestra.run.smithi055.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34608.log.gz 2024-03-19T02:14:02.741 INFO:teuthology.orchestra.run.smithi055.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34925.log.gz 2024-03-19T02:14:02.741 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37108.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37108.log.gz 2024-03-19T02:14:02.741 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39102.log 2024-03-19T02:14:02.741 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33740.log 2024-03-19T02:14:02.741 INFO:teuthology.orchestra.run.smithi055.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42591.log.gz 2024-03-19T02:14:02.741 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33792.log 2024-03-19T02:14:02.741 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39994.log/var/log/ceph/ceph-client.admin.33740.log: 2024-03-19T02:14:02.741 INFO:teuthology.orchestra.run.smithi055.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39102.log.gz 2024-03-19T02:14:02.741 INFO:teuthology.orchestra.run.smithi055.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33740.log.gz 2024-03-19T02:14:02.742 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40727.log 2024-03-19T02:14:02.742 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39836.log 2024-03-19T02:14:02.742 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39994.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33792.log.gz 2024-03-19T02:14:02.742 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40727.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39994.log.gz 2024-03-19T02:14:02.742 INFO:teuthology.orchestra.run.smithi055.stderr: 94.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-19T02:14:02.742 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33766.log 2024-03-19T02:14:02.742 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33447.log 2024-03-19T02:14:02.742 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39836.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40727.log.gz 2024-03-19T02:14:02.742 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33766.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39836.log.gz 2024-03-19T02:14:02.742 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37815.log 2024-03-19T02:14:02.742 INFO:teuthology.orchestra.run.smithi055.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33766.log.gz 2024-03-19T02:14:02.742 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32809.log 2024-03-19T02:14:02.742 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33447.log: /var/log/ceph/ceph-client.admin.37815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41957.log 2024-03-19T02:14:02.742 INFO:teuthology.orchestra.run.smithi055.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33447.log.gz 2024-03-19T02:14:02.743 INFO:teuthology.orchestra.run.smithi055.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37815.log.gz 2024-03-19T02:14:02.743 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.32809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36060.log 2024-03-19T02:14:02.743 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41460.log 2024-03-19T02:14:02.743 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41957.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32809.log.gz 2024-03-19T02:14:02.743 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36060.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41957.log.gz 2024-03-19T02:14:02.743 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39966.log 2024-03-19T02:14:02.743 INFO:teuthology.orchestra.run.smithi055.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36060.log.gz 2024-03-19T02:14:02.743 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43480.log 2024-03-19T02:14:02.743 INFO:teuthology.orchestra.run.smithi055.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41460.log.gz 2024-03-19T02:14:02.743 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37552.log 2024-03-19T02:14:02.743 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.43480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42331.log 2024-03-19T02:14:02.743 INFO:teuthology.orchestra.run.smithi055.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39966.log.gz 2024-03-19T02:14:02.743 INFO:teuthology.orchestra.run.smithi055.stderr: 88.2%/var/log/ceph/ceph-client.admin.37552.log: -- replaced with /var/log/ceph/ceph-client.admin.43480.log.gzgzip 2024-03-19T02:14:02.743 INFO:teuthology.orchestra.run.smithi055.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38864.log 2024-03-19T02:14:02.743 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42331.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37552.log.gz 2024-03-19T02:14:02.744 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38207.log 2024-03-19T02:14:02.744 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38864.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42331.log.gz 2024-03-19T02:14:02.744 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41565.log 2024-03-19T02:14:02.744 INFO:teuthology.orchestra.run.smithi055.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38864.log.gz 2024-03-19T02:14:02.744 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36034.log 2024-03-19T02:14:02.744 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41565.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38207.log.gz 2024-03-19T02:14:02.744 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35190.log 2024-03-19T02:14:02.744 INFO:teuthology.orchestra.run.smithi055.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41565.log.gz 2024-03-19T02:14:02.744 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37238.log 2024-03-19T02:14:02.744 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32841.log 2024-03-19T02:14:02.744 INFO:teuthology.orchestra.run.smithi055.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36034.log.gz 2024-03-19T02:14:02.744 INFO:teuthology.orchestra.run.smithi055.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35190.log.gz 2024-03-19T02:14:02.744 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33922.log 2024-03-19T02:14:02.744 INFO:teuthology.orchestra.run.smithi055.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37238.log.gz 2024-03-19T02:14:02.744 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32566.log 2024-03-19T02:14:02.744 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.32841.log: /var/log/ceph/ceph-client.admin.33922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33610.log 2024-03-19T02:14:02.745 INFO:teuthology.orchestra.run.smithi055.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32841.log.gz 2024-03-19T02:14:02.745 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.32566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37264.log 2024-03-19T02:14:02.745 INFO:teuthology.orchestra.run.smithi055.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33922.log.gz 2024-03-19T02:14:02.745 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43064.log 2024-03-19T02:14:02.745 INFO:teuthology.orchestra.run.smithi055.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.32566.log.gz 2024-03-19T02:14:02.745 INFO:teuthology.orchestra.run.smithi055.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33610.log.gz/var/log/ceph/ceph-client.admin.37264.log: 2024-03-19T02:14:02.745 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36741.log 2024-03-19T02:14:02.745 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.43064.log: gzip 88.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.37264.log.gz 2024-03-19T02:14:02.745 INFO:teuthology.orchestra.run.smithi055.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41985.log 2024-03-19T02:14:02.745 INFO:teuthology.orchestra.run.smithi055.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43064.log.gz 2024-03-19T02:14:02.745 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36950.log 2024-03-19T02:14:02.745 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36819.log 2024-03-19T02:14:02.745 INFO:teuthology.orchestra.run.smithi055.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36741.log.gz 2024-03-19T02:14:02.746 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36950.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41985.log.gz 2024-03-19T02:14:02.746 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37082.log 2024-03-19T02:14:02.746 INFO:teuthology.orchestra.run.smithi055.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36950.log.gz 2024-03-19T02:14:02.746 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33290.log 2024-03-19T02:14:02.746 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43168.log 2024-03-19T02:14:02.746 INFO:teuthology.orchestra.run.smithi055.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36819.log.gz 2024-03-19T02:14:02.746 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37082.log: /var/log/ceph/ceph-client.admin.33290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40387.log 2024-03-19T02:14:02.746 INFO:teuthology.orchestra.run.smithi055.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37082.log.gz 2024-03-19T02:14:02.746 INFO:teuthology.orchestra.run.smithi055.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.33290.log.gz 2024-03-19T02:14:02.746 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.43168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34201.log 2024-03-19T02:14:02.746 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38838.log 2024-03-19T02:14:02.746 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40387.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43168.log.gz 2024-03-19T02:14:02.746 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34201.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40387.log.gz 2024-03-19T02:14:02.746 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42141.log 2024-03-19T02:14:02.747 INFO:teuthology.orchestra.run.smithi055.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34201.log.gz 2024-03-19T02:14:02.747 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37134.log 2024-03-19T02:14:02.747 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42141.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38838.log.gz 2024-03-19T02:14:02.747 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34491.log 2024-03-19T02:14:02.747 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37134.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42141.log.gz 2024-03-19T02:14:02.747 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41931.log 2024-03-19T02:14:02.747 INFO:teuthology.orchestra.run.smithi055.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37134.log.gz 2024-03-19T02:14:02.747 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38446.log 2024-03-19T02:14:02.747 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41931.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34491.log.gz 2024-03-19T02:14:02.747 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36793.log 2024-03-19T02:14:02.747 INFO:teuthology.orchestra.run.smithi055.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41931.log.gz 2024-03-19T02:14:02.747 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38446.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-19T02:14:02.747 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36793.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38446.log.gz 2024-03-19T02:14:02.747 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33027.log 2024-03-19T02:14:02.748 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-osd.2.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36793.log.gz 2024-03-19T02:14:02.748 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35876.log 2024-03-19T02:14:02.748 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41697.log 2024-03-19T02:14:02.748 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35876.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33027.log.gz 2024-03-19T02:14:02.748 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42303.log 2024-03-19T02:14:02.748 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34899.log 2024-03-19T02:14:02.748 INFO:teuthology.orchestra.run.smithi055.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35876.log.gz 2024-03-19T02:14:02.748 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42303.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41697.log.gz 2024-03-19T02:14:02.748 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34441.log 2024-03-19T02:14:02.748 INFO:teuthology.orchestra.run.smithi055.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42303.log.gz 2024-03-19T02:14:02.748 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35164.log 2024-03-19T02:14:02.748 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41250.log 2024-03-19T02:14:02.748 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34441.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34899.log.gz 2024-03-19T02:14:02.748 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38656.log 2024-03-19T02:14:02.748 INFO:teuthology.orchestra.run.smithi055.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34441.log.gz 2024-03-19T02:14:02.749 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41250.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35164.log.gz 2024-03-19T02:14:02.749 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33551.log 2024-03-19T02:14:02.749 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39154.log 2024-03-19T02:14:02.749 INFO:teuthology.orchestra.run.smithi055.stderr: 89.1%/var/log/ceph/ceph-client.admin.38656.log: -- replaced with /var/log/ceph/ceph-client.admin.41250.log.gz 2024-03-19T02:14:02.749 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35772.log 2024-03-19T02:14:02.749 INFO:teuthology.orchestra.run.smithi055.stderr: 87.4%/var/log/ceph/ceph-client.admin.39154.log: -- replaced with /var/log/ceph/ceph-client.admin.38656.log.gz 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33551.log.gzgzip 2024-03-19T02:14:02.749 INFO:teuthology.orchestra.run.smithi055.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40779.log 2024-03-19T02:14:02.749 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-19T02:14:02.749 INFO:teuthology.orchestra.run.smithi055.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39154.log.gz 2024-03-19T02:14:02.749 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35772.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35772.log.gz 2024-03-19T02:14:02.749 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33844.log 2024-03-19T02:14:02.749 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37605.log 2024-03-19T02:14:02.749 INFO:teuthology.orchestra.run.smithi055.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40779.log.gz 2024-03-19T02:14:02.749 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34634.log 2024-03-19T02:14:02.750 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38103.log 2024-03-19T02:14:02.750 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37605.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33844.log.gz 2024-03-19T02:14:02.750 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34634.log: gzip 88.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.37605.log.gz --verbose 2024-03-19T02:14:02.750 INFO:teuthology.orchestra.run.smithi055.stderr: -- /var/log/ceph/ceph-client.admin.35374.log 2024-03-19T02:14:02.750 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38103.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34634.log.gz 2024-03-19T02:14:02.750 INFO:teuthology.orchestra.run.smithi055.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38103.log.gz 2024-03-19T02:14:02.751 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-19T02:14:02.751 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph.audit.log: 94.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-19T02:14:02.768 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-19T02:14:02.768 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-mds.d.log: gzip: /var/log/ceph/ceph-mds.d.log: file size changed while zipping 2024-03-19T02:14:02.769 INFO:teuthology.orchestra.run.smithi084.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-19T02:14:02.769 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mds.f.log 2024-03-19T02:14:02.769 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/valgrind/mon.c.log: 91.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-19T02:14:02.769 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mds.b.log 2024-03-19T02:14:02.769 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/valgrind/mds.f.log: 54.4% -- replaced with /var/log/ceph/valgrind/mds.f.log.gz 2024-03-19T02:14:02.769 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mds.d.log 2024-03-19T02:14:02.769 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/valgrind/mds.b.log: 54.4% -- replaced with /var/log/ceph/valgrind/mds.b.log.gz 2024-03-19T02:14:02.769 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-19T02:14:02.769 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/valgrind/mds.d.log: 54.4% -- replaced with /var/log/ceph/valgrind/mds.d.log.gz 2024-03-19T02:14:02.769 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27870.log 2024-03-19T02:14:02.775 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/valgrind/mon.b.log: 91.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-19T02:14:02.775 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-19T02:14:02.775 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.27870.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.27870.log.gz 2024-03-19T02:14:02.775 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27740.log 2024-03-19T02:14:02.778 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-19T02:14:02.778 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35426.log 2024-03-19T02:14:02.778 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-mon.a.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35374.log.gz 2024-03-19T02:14:02.778 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34136.log 2024-03-19T02:14:02.778 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34222.log 2024-03-19T02:14:02.778 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34136.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35426.log.gz 2024-03-19T02:14:02.779 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34162.log 2024-03-19T02:14:02.779 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36924.log 2024-03-19T02:14:02.779 INFO:teuthology.orchestra.run.smithi055.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34136.log.gz/var/log/ceph/ceph-client.admin.34222.log: 2024-03-19T02:14:02.779 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40361.log 2024-03-19T02:14:02.779 INFO:teuthology.orchestra.run.smithi055.stderr: 88.3% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34222.log.gz 2024-03-19T02:14:02.779 INFO:teuthology.orchestra.run.smithi055.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34162.log.gz 2024-03-19T02:14:02.779 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34767.log 2024-03-19T02:14:02.779 INFO:teuthology.orchestra.run.smithi055.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36924.log.gz 2024-03-19T02:14:02.779 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40361.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40361.log.gz 2024-03-19T02:14:02.779 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38918.log 2024-03-19T02:14:02.779 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34767.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34767.log.gz 2024-03-19T02:14:02.779 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36086.log 2024-03-19T02:14:02.779 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40649.log 2024-03-19T02:14:02.779 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35556.log 2024-03-19T02:14:02.780 INFO:teuthology.orchestra.run.smithi055.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38918.log.gz 2024-03-19T02:14:02.780 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40649.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36086.log.gz 2024-03-19T02:14:02.780 INFO:teuthology.orchestra.run.smithi055.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39521.log 2024-03-19T02:14:02.780 INFO:teuthology.orchestra.run.smithi055.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40649.log.gz 2024-03-19T02:14:02.780 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38708.log 2024-03-19T02:14:02.780 INFO:teuthology.orchestra.run.smithi055.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39810.log 2024-03-19T02:14:02.780 INFO:teuthology.orchestra.run.smithi055.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35556.log.gz 2024-03-19T02:14:02.780 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39521.log: /var/log/ceph/ceph-client.admin.38708.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39521.log.gz 2024-03-19T02:14:02.780 INFO:teuthology.orchestra.run.smithi055.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38708.log.gz 2024-03-19T02:14:02.780 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38576.log 2024-03-19T02:14:02.780 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42749.log 2024-03-19T02:14:02.780 INFO:teuthology.orchestra.run.smithi055.stderr: 88.5%/var/log/ceph/ceph-client.admin.38576.log: -- replaced with /var/log/ceph/ceph-client.admin.39810.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37711.log 2024-03-19T02:14:02.780 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-19T02:14:02.780 INFO:teuthology.orchestra.run.smithi055.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38576.log.gz 2024-03-19T02:14:02.781 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42749.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42749.log.gz 2024-03-19T02:14:02.781 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38550.log 2024-03-19T02:14:02.781 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37711.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.22752.log 2024-03-19T02:14:02.781 INFO:teuthology.orchestra.run.smithi055.stderr: 88.6%/var/log/ceph/ceph-client.admin.38550.log: gzip -- replaced with /var/log/ceph/ceph-client.admin.37711.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.33368.log 2024-03-19T02:14:02.781 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-19T02:14:02.781 INFO:teuthology.orchestra.run.smithi055.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38550.log.gz/var/log/ceph/ceph.tmp-client.admin.22752.log: 2024-03-19T02:14:02.781 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42435.log 2024-03-19T02:14:02.781 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.22752.log.gz 2024-03-19T02:14:02.781 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38998.log 2024-03-19T02:14:02.781 INFO:teuthology.orchestra.run.smithi055.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33368.log.gz 2024-03-19T02:14:02.781 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35294.log 2024-03-19T02:14:02.781 INFO:teuthology.orchestra.run.smithi055.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42435.log.gz 2024-03-19T02:14:02.781 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38998.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38998.log.gz 2024-03-19T02:14:02.781 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40936.log 2024-03-19T02:14:02.781 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37893.log 2024-03-19T02:14:02.782 INFO:teuthology.orchestra.run.smithi055.stderr: 87.5%/var/log/ceph/ceph-client.admin.40936.log: -- replaced with /var/log/ceph/ceph-client.admin.35294.log.gz 2024-03-19T02:14:02.782 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38602.log 2024-03-19T02:14:02.782 INFO:teuthology.orchestra.run.smithi055.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40936.log.gz 2024-03-19T02:14:02.782 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37396.log 2024-03-19T02:14:02.782 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37893.log: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36557.log 2024-03-19T02:14:02.782 INFO:teuthology.orchestra.run.smithi055.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37893.log.gz/var/log/ceph/ceph-client.admin.38602.log: 2024-03-19T02:14:02.782 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37396.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38602.log.gz 2024-03-19T02:14:02.782 INFO:teuthology.orchestra.run.smithi055.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37396.log.gz 2024-03-19T02:14:02.782 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39758.log 2024-03-19T02:14:02.782 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35980.log 2024-03-19T02:14:02.782 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39758.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36557.log.gz 2024-03-19T02:14:02.782 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43194.log 2024-03-19T02:14:02.782 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39680.log 2024-03-19T02:14:02.783 INFO:teuthology.orchestra.run.smithi055.stderr: 88.7%/var/log/ceph/ceph-client.admin.35980.log: -- replaced with /var/log/ceph/ceph-client.admin.39758.log.gz 2024-03-19T02:14:02.783 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.43194.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41408.log 2024-03-19T02:14:02.783 INFO:teuthology.orchestra.run.smithi055.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35980.log.gz 2024-03-19T02:14:02.783 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39680.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43194.log.gz 2024-03-19T02:14:02.783 INFO:teuthology.orchestra.run.smithi055.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39680.log.gz 2024-03-19T02:14:02.783 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41538.log 2024-03-19T02:14:02.783 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35216.log 2024-03-19T02:14:02.783 INFO:teuthology.orchestra.run.smithi055.stderr: 88.4%/var/log/ceph/ceph-client.admin.41538.log: -- replaced with /var/log/ceph/ceph-client.admin.41408.log.gz 2024-03-19T02:14:02.783 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34417.log 2024-03-19T02:14:02.783 INFO:teuthology.orchestra.run.smithi055.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41538.log.gz 2024-03-19T02:14:02.783 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35824.log 2024-03-19T02:14:02.783 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33714.log 2024-03-19T02:14:02.783 INFO:teuthology.orchestra.run.smithi055.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35216.log.gz 2024-03-19T02:14:02.783 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34417.log: /var/log/ceph/ceph-client.admin.35824.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34417.log.gz 2024-03-19T02:14:02.783 INFO:teuthology.orchestra.run.smithi055.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35824.log.gz 2024-03-19T02:14:02.783 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43428.log 2024-03-19T02:14:02.784 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35086.log 2024-03-19T02:14:02.784 INFO:teuthology.orchestra.run.smithi055.stderr: 87.0%/var/log/ceph/ceph-client.admin.43428.log: -- replaced with /var/log/ceph/ceph-client.admin.33714.log.gz 2024-03-19T02:14:02.784 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34977.log 2024-03-19T02:14:02.784 INFO:teuthology.orchestra.run.smithi055.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43428.log.gz 2024-03-19T02:14:02.784 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41801.log 2024-03-19T02:14:02.784 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35086.log: /var/log/ceph/ceph-client.admin.34977.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35086.log.gz 2024-03-19T02:14:02.784 INFO:teuthology.orchestra.run.smithi055.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34977.log.gz 2024-03-19T02:14:02.784 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35954.log 2024-03-19T02:14:02.784 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35400.log 2024-03-19T02:14:02.784 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35003.log 2024-03-19T02:14:02.784 INFO:teuthology.orchestra.run.smithi055.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41801.log.gz 2024-03-19T02:14:02.784 INFO:teuthology.orchestra.run.smithi055.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35954.log.gz 2024-03-19T02:14:02.784 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35400.log: gzip 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35400.log.gz -5 2024-03-19T02:14:02.784 INFO:teuthology.orchestra.run.smithi055.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39548.log 2024-03-19T02:14:02.784 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41853.log 2024-03-19T02:14:02.785 INFO:teuthology.orchestra.run.smithi055.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35003.log.gz/var/log/ceph/ceph-client.admin.39548.log: 2024-03-19T02:14:02.785 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41434.log 2024-03-19T02:14:02.785 INFO:teuthology.orchestra.run.smithi055.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39548.log.gz 2024-03-19T02:14:02.785 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39442.log 2024-03-19T02:14:02.785 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41853.log: 88.5%/var/log/ceph/ceph-client.admin.41434.log: -- replaced with /var/log/ceph/ceph-client.admin.41853.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40910.log 2024-03-19T02:14:02.785 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-19T02:14:02.785 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39442.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41434.log.gz 2024-03-19T02:14:02.785 INFO:teuthology.orchestra.run.smithi055.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39442.log.gz 2024-03-19T02:14:02.785 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34795.log 2024-03-19T02:14:02.785 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32376.log 2024-03-19T02:14:02.785 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34795.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-19T02:14:02.785 INFO:teuthology.orchestra.run.smithi055.stderr: 88.3%/var/log/ceph/ceph-client.admin.32376.log: -- replaced with /var/log/ceph/ceph-client.admin.34795.log.gz 2024-03-19T02:14:02.785 INFO:teuthology.orchestra.run.smithi055.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32376.log.gz 2024-03-19T02:14:02.785 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36138.log 2024-03-19T02:14:02.785 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41094.log 2024-03-19T02:14:02.786 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36138.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36138.log.gz 2024-03-19T02:14:02.786 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32674.log 2024-03-19T02:14:02.786 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41094.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41094.log.gz 2024-03-19T02:14:02.786 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43142.log 2024-03-19T02:14:02.786 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.32674.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32674.log.gz 2024-03-19T02:14:02.786 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39940.log 2024-03-19T02:14:02.786 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.43142.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43142.log.gz 2024-03-19T02:14:02.786 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43560.log 2024-03-19T02:14:02.786 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39940.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39940.log.gz 2024-03-19T02:14:02.786 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43454.log 2024-03-19T02:14:02.786 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40910.log: /var/log/ceph/ceph-client.admin.43560.log: 88.3% 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43560.log.gz 2024-03-19T02:14:02.786 INFO:teuthology.orchestra.run.smithi055.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40910.log.gz 2024-03-19T02:14:02.786 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32778.log 2024-03-19T02:14:02.786 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42565.log 2024-03-19T02:14:02.789 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-mds.b.log: 91.2% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-19T02:14:02.789 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-19T02:14:02.789 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.27740.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27740.log.gz 2024-03-19T02:14:02.789 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-19T02:14:02.795 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.43454.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43454.log.gz 2024-03-19T02:14:02.795 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33688.log 2024-03-19T02:14:02.795 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42565.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42565.log.gz 2024-03-19T02:14:02.795 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.32778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40046.log 2024-03-19T02:14:02.795 INFO:teuthology.orchestra.run.smithi055.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32778.log.gz 2024-03-19T02:14:02.796 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33688.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33688.log.gz 2024-03-19T02:14:02.796 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40176.log 2024-03-19T02:14:02.796 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42409.log 2024-03-19T02:14:02.796 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40046.log: /var/log/ceph/ceph-client.admin.40176.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40046.log.gz 2024-03-19T02:14:02.796 INFO:teuthology.orchestra.run.smithi055.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40176.log.gz 2024-03-19T02:14:02.796 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37919.log 2024-03-19T02:14:02.796 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35640.log 2024-03-19T02:14:02.796 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42409.log: /var/log/ceph/ceph-client.admin.37919.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42409.log.gz 2024-03-19T02:14:02.796 INFO:teuthology.orchestra.run.smithi055.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37919.log.gz 2024-03-19T02:14:02.796 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42461.log 2024-03-19T02:14:02.796 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35029.log 2024-03-19T02:14:02.796 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36427.log 2024-03-19T02:14:02.796 INFO:teuthology.orchestra.run.smithi055.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35640.log.gz 2024-03-19T02:14:02.796 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32726.log 2024-03-19T02:14:02.796 INFO:teuthology.orchestra.run.smithi055.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.35029.log.gz/var/log/ceph/ceph-client.admin.36427.log: 2024-03-19T02:14:02.797 INFO:teuthology.orchestra.run.smithi055.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36427.log.gz 2024-03-19T02:14:02.797 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42958.log 2024-03-19T02:14:02.797 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43298.log 2024-03-19T02:14:02.797 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.32726.log: /var/log/ceph/ceph-client.admin.42958.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32726.log.gz 2024-03-19T02:14:02.797 INFO:teuthology.orchestra.run.smithi055.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42958.log.gz 2024-03-19T02:14:02.797 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42089.log 2024-03-19T02:14:02.797 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.43298.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43298.log.gz 2024-03-19T02:14:02.797 INFO:teuthology.orchestra.run.smithi055.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42461.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32540.log 2024-03-19T02:14:02.797 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-19T02:14:02.797 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42089.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42089.log.gz 2024-03-19T02:14:02.797 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32700.log 2024-03-19T02:14:02.797 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27681.log 2024-03-19T02:14:02.797 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.32540.log: /var/log/ceph/ceph-client.admin.32700.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32540.log.gz 2024-03-19T02:14:02.798 INFO:teuthology.orchestra.run.smithi055.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32700.log.gz 2024-03-19T02:14:02.798 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41486.log 2024-03-19T02:14:02.798 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38155.log 2024-03-19T02:14:02.798 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.27681.log: /var/log/ceph/ceph-client.admin.41486.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27681.log.gz 2024-03-19T02:14:02.798 INFO:teuthology.orchestra.run.smithi055.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41486.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38498.log 2024-03-19T02:14:02.798 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-19T02:14:02.798 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38155.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38155.log.gz 2024-03-19T02:14:02.798 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38260.log 2024-03-19T02:14:02.798 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36190.log 2024-03-19T02:14:02.798 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38498.log: /var/log/ceph/ceph-client.admin.38260.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38498.log.gz 2024-03-19T02:14:02.798 INFO:teuthology.orchestra.run.smithi055.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38260.log.gz 2024-03-19T02:14:02.798 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38366.log 2024-03-19T02:14:02.798 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36190.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36190.log.gz 2024-03-19T02:14:02.799 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27819.log 2024-03-19T02:14:02.800 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39128.log 2024-03-19T02:14:02.800 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-19T02:14:02.800 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38366.log: /var/log/ceph/ceph-client.admin.39128.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38366.log.gz 2024-03-19T02:14:02.800 INFO:teuthology.orchestra.run.smithi055.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39128.log.gz 2024-03-19T02:14:02.800 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37212.log 2024-03-19T02:14:02.812 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34847.log 2024-03-19T02:14:02.813 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37212.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37212.log.gz 2024-03-19T02:14:02.814 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33636.log 2024-03-19T02:14:02.815 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.27819.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27819.log.gz 2024-03-19T02:14:02.815 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34847.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34847.log.gz 2024-03-19T02:14:02.816 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33473.log 2024-03-19T02:14:02.817 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33636.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33636.log.gz 2024-03-19T02:14:02.818 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37579.log 2024-03-19T02:14:02.819 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33473.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33473.log.gz 2024-03-19T02:14:02.819 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41671.log 2024-03-19T02:14:02.820 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37579.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37579.log.gz 2024-03-19T02:14:02.820 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27800.log 2024-03-19T02:14:02.822 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41671.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41671.log.gz 2024-03-19T02:14:02.822 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34873.log 2024-03-19T02:14:02.823 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.27800.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.27800.log.gz 2024-03-19T02:14:02.823 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39050.log 2024-03-19T02:14:02.824 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34873.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34873.log.gz 2024-03-19T02:14:02.824 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33962.log 2024-03-19T02:14:02.825 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39050.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39050.log.gz 2024-03-19T02:14:02.826 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33420.log 2024-03-19T02:14:02.827 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33962.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.33962.log.gz 2024-03-19T02:14:02.827 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39862.log 2024-03-19T02:14:02.828 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33420.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33420.log.gz 2024-03-19T02:14:02.828 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38786.log 2024-03-19T02:14:02.829 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39862.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39862.log.gz 2024-03-19T02:14:02.830 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36375.log 2024-03-19T02:14:02.831 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38786.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38786.log.gz 2024-03-19T02:14:02.831 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41382.log 2024-03-19T02:14:02.832 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36375.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36375.log.gz 2024-03-19T02:14:02.832 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34821.log 2024-03-19T02:14:02.834 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41382.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41382.log.gz 2024-03-19T02:14:02.834 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43090.log 2024-03-19T02:14:02.835 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34821.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34821.log.gz 2024-03-19T02:14:02.836 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41617.log 2024-03-19T02:14:02.837 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.43090.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43090.log.gz 2024-03-19T02:14:02.837 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40465.log 2024-03-19T02:14:02.838 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41617.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41617.log.gz 2024-03-19T02:14:02.838 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43376.log 2024-03-19T02:14:02.839 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40465.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40465.log.gz 2024-03-19T02:14:02.840 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39338.log 2024-03-19T02:14:02.841 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.43376.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43376.log.gz 2024-03-19T02:14:02.841 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38420.log 2024-03-19T02:14:02.842 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39338.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39338.log.gz 2024-03-19T02:14:02.842 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33525.log 2024-03-19T02:14:02.843 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38420.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38420.log.gz 2024-03-19T02:14:02.843 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32484.log 2024-03-19T02:14:02.844 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33525.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33525.log.gz 2024-03-19T02:14:02.845 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36976.log 2024-03-19T02:14:02.846 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.32484.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.32484.log.gz 2024-03-19T02:14:02.846 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39784.log 2024-03-19T02:14:02.847 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36976.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36976.log.gz 2024-03-19T02:14:02.847 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34154.log 2024-03-19T02:14:02.849 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39784.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39784.log.gz 2024-03-19T02:14:02.849 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42487.log 2024-03-19T02:14:02.850 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34154.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34154.log.gz 2024-03-19T02:14:02.850 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42984.log 2024-03-19T02:14:02.851 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42487.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42487.log.gz 2024-03-19T02:14:02.851 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39626.log 2024-03-19T02:14:02.852 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42984.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42984.log.gz 2024-03-19T02:14:02.853 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42827.log 2024-03-19T02:14:02.854 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39626.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39626.log.gz 2024-03-19T02:14:02.854 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40595.log 2024-03-19T02:14:02.855 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42827.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42827.log.gz 2024-03-19T02:14:02.855 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36243.log 2024-03-19T02:14:02.856 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40595.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40595.log.gz 2024-03-19T02:14:02.857 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38812.log 2024-03-19T02:14:02.857 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36243.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36243.log.gz 2024-03-19T02:14:02.858 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33316.log 2024-03-19T02:14:02.859 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38812.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38812.log.gz 2024-03-19T02:14:02.860 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33316.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33316.log.gz 2024-03-19T02:14:02.860 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43012.log 2024-03-19T02:14:02.860 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40569.log 2024-03-19T02:14:02.861 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39180.log 2024-03-19T02:14:02.861 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.43012.log: /var/log/ceph/ceph-client.admin.40569.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43012.log.gz 2024-03-19T02:14:02.862 INFO:teuthology.orchestra.run.smithi055.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40569.log.gz 2024-03-19T02:14:02.868 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34715.log 2024-03-19T02:14:02.868 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39180.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39180.log.gz 2024-03-19T02:14:02.868 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35322.log 2024-03-19T02:14:02.868 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34715.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34715.log.gz 2024-03-19T02:14:02.868 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43402.log 2024-03-19T02:14:02.868 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35322.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35322.log.gz 2024-03-19T02:14:02.869 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32423.log 2024-03-19T02:14:02.877 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.43402.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.43402.log.gz 2024-03-19T02:14:02.877 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42932.log 2024-03-19T02:14:02.877 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.32423.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32423.log.gz 2024-03-19T02:14:02.877 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39416.log 2024-03-19T02:14:02.877 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42932.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42932.log.gz 2024-03-19T02:14:02.877 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37945.log 2024-03-19T02:14:02.877 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39416.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39416.log.gz 2024-03-19T02:14:02.877 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42539.log 2024-03-19T02:14:02.877 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37945.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37945.log.gz 2024-03-19T02:14:02.877 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35850.log 2024-03-19T02:14:02.877 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42539.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42539.log.gz 2024-03-19T02:14:02.878 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37763.log 2024-03-19T02:14:02.878 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35850.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35850.log.gz 2024-03-19T02:14:02.878 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41330.log 2024-03-19T02:14:02.878 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37763.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37763.log.gz 2024-03-19T02:14:02.878 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-19T02:14:02.878 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41330.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41330.log.gz 2024-03-19T02:14:02.878 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33968.log 2024-03-19T02:14:02.881 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph.log: 88.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-19T02:14:02.881 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42801.log 2024-03-19T02:14:02.881 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33968.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33968.log.gz 2024-03-19T02:14:02.881 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36112.log 2024-03-19T02:14:02.882 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42801.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42801.log.gz 2024-03-19T02:14:02.882 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37344.log 2024-03-19T02:14:02.883 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36112.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.36112.log.gz 2024-03-19T02:14:02.883 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41016.log 2024-03-19T02:14:02.884 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37344.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37344.log.gz 2024-03-19T02:14:02.885 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42167.log 2024-03-19T02:14:02.885 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41016.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41016.log.gz 2024-03-19T02:14:02.886 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37004.log 2024-03-19T02:14:02.887 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42167.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42167.log.gz 2024-03-19T02:14:02.887 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40990.log 2024-03-19T02:14:02.888 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37004.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37004.log.gz 2024-03-19T02:14:02.888 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38760.log 2024-03-19T02:14:02.889 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40990.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40990.log.gz 2024-03-19T02:14:02.889 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41146.log 2024-03-19T02:14:02.890 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38760.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38760.log.gz 2024-03-19T02:14:02.891 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41068.log 2024-03-19T02:14:02.891 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41146.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41146.log.gz 2024-03-19T02:14:02.892 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35504.log 2024-03-19T02:14:02.893 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41068.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41068.log.gz 2024-03-19T02:14:02.893 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37316.log 2024-03-19T02:14:02.894 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35504.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35504.log.gz 2024-03-19T02:14:02.894 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39914.log 2024-03-19T02:14:02.895 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37316.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37316.log.gz 2024-03-19T02:14:02.895 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32648.log 2024-03-19T02:14:02.896 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39914.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39914.log.gz 2024-03-19T02:14:02.897 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41775.log 2024-03-19T02:14:02.898 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.32648.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.32648.log.gz 2024-03-19T02:14:02.898 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40255.log 2024-03-19T02:14:02.899 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41775.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41775.log.gz 2024-03-19T02:14:02.899 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33818.log 2024-03-19T02:14:02.900 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40255.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40255.log.gz 2024-03-19T02:14:02.900 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37186.log 2024-03-19T02:14:02.902 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33818.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33818.log.gz 2024-03-19T02:14:02.902 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38392.log 2024-03-19T02:14:02.903 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37186.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37186.log.gz 2024-03-19T02:14:02.903 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37737.log 2024-03-19T02:14:02.904 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38392.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38392.log.gz 2024-03-19T02:14:02.904 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41749.log 2024-03-19T02:14:02.905 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37737.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37737.log.gz 2024-03-19T02:14:02.905 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36479.log 2024-03-19T02:14:02.906 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41749.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41749.log.gz 2024-03-19T02:14:02.907 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41042.log 2024-03-19T02:14:02.907 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36479.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36479.log.gz 2024-03-19T02:14:02.908 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41224.log 2024-03-19T02:14:02.909 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41042.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41042.log.gz 2024-03-19T02:14:02.909 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42115.log 2024-03-19T02:14:02.910 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41224.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41224.log.gz 2024-03-19T02:14:02.910 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36164.log 2024-03-19T02:14:02.911 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42115.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42115.log.gz 2024-03-19T02:14:02.911 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42775.log 2024-03-19T02:14:02.912 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36164.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36164.log.gz 2024-03-19T02:14:02.913 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36216.log 2024-03-19T02:14:02.913 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42775.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42775.log.gz 2024-03-19T02:14:02.914 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38944.log 2024-03-19T02:14:02.915 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36216.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36216.log.gz 2024-03-19T02:14:02.915 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40150.log 2024-03-19T02:14:02.916 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38944.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38944.log.gz 2024-03-19T02:14:02.916 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38025.log 2024-03-19T02:14:02.917 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40150.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40150.log.gz 2024-03-19T02:14:02.917 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42853.log 2024-03-19T02:14:02.918 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38025.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38025.log.gz 2024-03-19T02:14:02.919 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37685.log 2024-03-19T02:14:02.919 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42853.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42853.log.gz 2024-03-19T02:14:02.920 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41512.log 2024-03-19T02:14:02.921 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37685.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37685.log.gz 2024-03-19T02:14:02.921 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36663.log 2024-03-19T02:14:02.922 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41512.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41512.log.gz 2024-03-19T02:14:02.922 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39284.log 2024-03-19T02:14:02.923 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36663.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36663.log.gz 2024-03-19T02:14:02.923 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33870.log 2024-03-19T02:14:02.924 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39284.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39284.log.gz 2024-03-19T02:14:02.925 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34573.log 2024-03-19T02:14:02.926 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33870.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33870.log.gz 2024-03-19T02:14:02.926 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32949.log 2024-03-19T02:14:02.927 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34573.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34573.log.gz 2024-03-19T02:14:02.927 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39076.log 2024-03-19T02:14:02.928 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.32949.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32949.log.gz 2024-03-19T02:14:02.929 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38077.log 2024-03-19T02:14:02.929 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39076.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39076.log.gz 2024-03-19T02:14:02.930 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33958.log 2024-03-19T02:14:02.931 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38077.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38077.log.gz 2024-03-19T02:14:02.931 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42617.log 2024-03-19T02:14:02.932 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33958.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33958.log.gz 2024-03-19T02:14:02.932 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37290.log 2024-03-19T02:14:02.933 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42617.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42617.log.gz 2024-03-19T02:14:02.933 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37370.log 2024-03-19T02:14:02.934 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37290.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37290.log.gz 2024-03-19T02:14:02.935 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37867.log 2024-03-19T02:14:02.935 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37370.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37370.log.gz 2024-03-19T02:14:02.936 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33966.log 2024-03-19T02:14:02.937 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37867.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37867.log.gz 2024-03-19T02:14:02.937 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40439.log 2024-03-19T02:14:02.938 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33966.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33966.log.gz 2024-03-19T02:14:02.938 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39574.log 2024-03-19T02:14:02.939 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40439.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40439.log.gz 2024-03-19T02:14:02.939 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37500.log 2024-03-19T02:14:02.940 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39574.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39574.log.gz 2024-03-19T02:14:02.941 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35746.log 2024-03-19T02:14:02.942 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37500.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37500.log.gz 2024-03-19T02:14:02.942 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42513.log 2024-03-19T02:14:02.943 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35746.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35746.log.gz 2024-03-19T02:14:02.943 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43220.log 2024-03-19T02:14:02.944 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42513.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42513.log.gz 2024-03-19T02:14:02.944 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41172.log 2024-03-19T02:14:02.945 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.43220.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43220.log.gz 2024-03-19T02:14:02.946 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37474.log 2024-03-19T02:14:02.946 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41172.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41172.log.gz 2024-03-19T02:14:02.947 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40098.log 2024-03-19T02:14:02.948 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37474.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37474.log.gz 2024-03-19T02:14:02.948 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32923.log 2024-03-19T02:14:02.949 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40098.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40098.log.gz 2024-03-19T02:14:02.949 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34688.log 2024-03-19T02:14:02.950 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.32923.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32923.log.gz 2024-03-19T02:14:02.950 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34365.log 2024-03-19T02:14:02.951 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34688.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34688.log.gz 2024-03-19T02:14:02.952 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39232.log 2024-03-19T02:14:02.953 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34365.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34365.log.gz 2024-03-19T02:14:02.953 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38051.log 2024-03-19T02:14:02.954 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39232.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39232.log.gz 2024-03-19T02:14:02.954 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35452.log 2024-03-19T02:14:02.955 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38051.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38051.log.gz 2024-03-19T02:14:02.955 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36898.log 2024-03-19T02:14:02.956 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35452.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35452.log.gz 2024-03-19T02:14:02.957 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32975.log 2024-03-19T02:14:02.957 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36898.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.36898.log.gz 2024-03-19T02:14:02.958 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34369.log 2024-03-19T02:14:02.959 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.32975.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32975.log.gz 2024-03-19T02:14:02.959 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39600.log 2024-03-19T02:14:02.960 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34369.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34369.log.gz 2024-03-19T02:14:02.960 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-19T02:14:02.961 INFO:teuthology.orchestra.run.smithi084.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-19T02:14:02.962 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39600.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39600.log.gz 2024-03-19T02:14:02.962 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39495.log 2024-03-19T02:14:02.964 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph.audit.log: 89.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-19T02:14:02.964 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40543.log 2024-03-19T02:14:02.965 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39495.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39495.log.gz 2024-03-19T02:14:02.966 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-19T02:14:02.967 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40543.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40543.log.gz 2024-03-19T02:14:02.967 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37030.log 2024-03-19T02:14:02.974 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-19T02:14:02.977 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37030.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37030.log.gz 2024-03-19T02:14:02.990 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36871.log 2024-03-19T02:14:02.993 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-mds.a.log: gzip: /var/log/ceph/ceph-mds.a.log: file size changed while zipping 2024-03-19T02:14:02.993 INFO:teuthology.orchestra.run.smithi055.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-19T02:14:02.994 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35668.log 2024-03-19T02:14:02.994 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36871.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36871.log.gz 2024-03-19T02:14:03.012 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37659.log 2024-03-19T02:14:03.013 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35668.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35668.log.gz 2024-03-19T02:14:03.020 INFO:teuthology.orchestra.run.smithi055.stderr:gzip: /var/log/ceph/ceph-mds.c.log: file size changed while zipping 2024-03-19T02:14:03.020 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43536.log 2024-03-19T02:14:03.021 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37659.log: 92.6% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-19T02:14:03.021 INFO:teuthology.orchestra.run.smithi055.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37659.log.gz 2024-03-19T02:14:03.024 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36349.log 2024-03-19T02:14:03.025 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.43536.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43536.log.gz 2024-03-19T02:14:03.025 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42357.log 2024-03-19T02:14:03.026 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36349.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36349.log.gz 2024-03-19T02:14:03.027 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35586.log 2024-03-19T02:14:03.028 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42357.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42357.log.gz 2024-03-19T02:14:03.028 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43116.log 2024-03-19T02:14:03.029 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35586.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35586.log.gz 2024-03-19T02:14:03.029 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36269.log 2024-03-19T02:14:03.030 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.43116.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43116.log.gz 2024-03-19T02:14:03.031 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34413.log 2024-03-19T02:14:03.031 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36269.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36269.log.gz 2024-03-19T02:14:03.032 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41276.log 2024-03-19T02:14:03.033 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34413.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34413.log.gz 2024-03-19T02:14:03.033 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33975.log 2024-03-19T02:14:03.034 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41276.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41276.log.gz 2024-03-19T02:14:03.034 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40307.log 2024-03-19T02:14:03.035 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33975.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33975.log.gz 2024-03-19T02:14:03.035 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36295.log 2024-03-19T02:14:03.036 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40307.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40307.log.gz 2024-03-19T02:14:03.037 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32897.log 2024-03-19T02:14:03.038 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36295.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36295.log.gz 2024-03-19T02:14:03.038 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41905.log 2024-03-19T02:14:03.039 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.32897.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32897.log.gz 2024-03-19T02:14:03.039 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37160.log 2024-03-19T02:14:03.040 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41905.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41905.log.gz 2024-03-19T02:14:03.040 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37448.log 2024-03-19T02:14:03.041 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37160.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37160.log.gz 2024-03-19T02:14:03.042 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39654.log 2024-03-19T02:14:03.042 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37448.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37448.log.gz 2024-03-19T02:14:03.043 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mds.a.log 2024-03-19T02:14:03.044 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39654.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39654.log.gz 2024-03-19T02:14:03.044 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-19T02:14:03.045 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/valgrind/mds.a.log: 54.5% -- replaced with /var/log/ceph/valgrind/mds.a.log.gz 2024-03-19T02:14:03.045 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mds.c.log 2024-03-19T02:14:03.046 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/valgrind/mon.a.log: 91.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-19T02:14:03.047 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mds.e.log 2024-03-19T02:14:03.047 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/valgrind/mds.c.log: 54.4% -- replaced with /var/log/ceph/valgrind/mds.c.log.gz 2024-03-19T02:14:03.047 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33970.log 2024-03-19T02:14:03.048 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/valgrind/mds.e.log: 54.5% -- replaced with /var/log/ceph/valgrind/mds.e.log.gz 2024-03-19T02:14:03.049 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34126.log 2024-03-19T02:14:03.050 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33970.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.33970.log.gz 2024-03-19T02:14:03.050 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35720.log 2024-03-19T02:14:03.051 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34126.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34126.log.gz 2024-03-19T02:14:03.051 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41198.log 2024-03-19T02:14:03.052 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35720.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35720.log.gz 2024-03-19T02:14:03.052 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35348.log 2024-03-19T02:14:03.053 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41198.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41198.log.gz 2024-03-19T02:14:03.054 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34660.log 2024-03-19T02:14:03.054 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35348.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35348.log.gz 2024-03-19T02:14:03.055 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37999.log 2024-03-19T02:14:03.056 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34660.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34660.log.gz 2024-03-19T02:14:03.056 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33499.log 2024-03-19T02:14:03.057 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37999.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37999.log.gz 2024-03-19T02:14:03.057 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40491.log 2024-03-19T02:14:03.058 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33499.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33499.log.gz 2024-03-19T02:14:03.058 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42383.log 2024-03-19T02:14:03.059 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40491.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40491.log.gz 2024-03-19T02:14:03.060 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38181.log 2024-03-19T02:14:03.061 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42383.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42383.log.gz 2024-03-19T02:14:03.061 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42906.log 2024-03-19T02:14:03.062 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38181.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38181.log.gz 2024-03-19T02:14:03.062 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35798.log 2024-03-19T02:14:03.063 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42906.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42906.log.gz 2024-03-19T02:14:03.063 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40124.log 2024-03-19T02:14:03.064 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35798.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35798.log.gz 2024-03-19T02:14:03.065 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33232.log 2024-03-19T02:14:03.065 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40124.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40124.log.gz 2024-03-19T02:14:03.066 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37631.log 2024-03-19T02:14:03.069 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33232.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.33232.log.gz 2024-03-19T02:14:03.069 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43272.log 2024-03-19T02:14:03.070 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37631.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37631.log.gz 2024-03-19T02:14:03.070 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35478.log 2024-03-19T02:14:03.071 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.43272.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43272.log.gz 2024-03-19T02:14:03.071 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36008.log 2024-03-19T02:14:03.073 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35478.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35478.log.gz 2024-03-19T02:14:03.073 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40857.log 2024-03-19T02:14:03.074 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36008.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36008.log.gz 2024-03-19T02:14:03.074 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36635.log 2024-03-19T02:14:03.075 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40857.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40857.log.gz 2024-03-19T02:14:03.075 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33342.log 2024-03-19T02:14:03.076 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36635.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36635.log.gz 2024-03-19T02:14:03.076 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42011.log 2024-03-19T02:14:03.077 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33342.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33342.log.gz 2024-03-19T02:14:03.077 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39364.log 2024-03-19T02:14:03.078 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42011.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42011.log.gz 2024-03-19T02:14:03.079 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33896.log 2024-03-19T02:14:03.080 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39364.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39364.log.gz 2024-03-19T02:14:03.080 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35928.log 2024-03-19T02:14:03.081 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33896.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33896.log.gz 2024-03-19T02:14:03.081 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33969.log 2024-03-19T02:14:03.082 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35928.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35928.log.gz 2024-03-19T02:14:03.082 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38340.log 2024-03-19T02:14:03.083 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33969.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33969.log.gz 2024-03-19T02:14:03.084 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34741.log 2024-03-19T02:14:03.085 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38340.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38340.log.gz 2024-03-19T02:14:03.085 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27559.log 2024-03-19T02:14:03.086 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34741.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34741.log.gz 2024-03-19T02:14:03.086 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33976.log 2024-03-19T02:14:03.087 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.27559.log: 93.2% -- replaced with /var/log/ceph/ceph-client.admin.27559.log.gz 2024-03-19T02:14:03.088 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41827.log 2024-03-19T02:14:03.089 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33976.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33976.log.gz 2024-03-19T02:14:03.089 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40413.log 2024-03-19T02:14:03.090 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41827.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41827.log.gz 2024-03-19T02:14:03.090 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38972.log 2024-03-19T02:14:03.091 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40413.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40413.log.gz 2024-03-19T02:14:03.092 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38234.log 2024-03-19T02:14:03.092 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38972.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38972.log.gz 2024-03-19T02:14:03.093 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39732.log 2024-03-19T02:14:03.094 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38234.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38234.log.gz 2024-03-19T02:14:03.094 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38628.log 2024-03-19T02:14:03.095 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39732.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39732.log.gz 2024-03-19T02:14:03.095 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36323.log 2024-03-19T02:14:03.096 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38628.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38628.log.gz 2024-03-19T02:14:03.096 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42723.log 2024-03-19T02:14:03.097 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36323.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36323.log.gz 2024-03-19T02:14:03.097 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37526.log 2024-03-19T02:14:03.099 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42723.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42723.log.gz 2024-03-19T02:14:03.099 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40202.log 2024-03-19T02:14:03.100 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37526.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37526.log.gz 2024-03-19T02:14:03.100 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41591.log 2024-03-19T02:14:03.101 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40202.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40202.log.gz 2024-03-19T02:14:03.101 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38312.log 2024-03-19T02:14:03.102 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41591.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41591.log.gz 2024-03-19T02:14:03.103 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40335.log 2024-03-19T02:14:03.103 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38312.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38312.log.gz 2024-03-19T02:14:03.104 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40805.log 2024-03-19T02:14:03.105 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40335.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40335.log.gz 2024-03-19T02:14:03.105 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39206.log 2024-03-19T02:14:03.106 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40805.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40805.log.gz 2024-03-19T02:14:03.106 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35902.log 2024-03-19T02:14:03.107 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39206.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39206.log.gz 2024-03-19T02:14:03.108 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36505.log 2024-03-19T02:14:03.108 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35902.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35902.log.gz 2024-03-19T02:14:03.109 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36531.log 2024-03-19T02:14:03.110 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36505.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36505.log.gz 2024-03-19T02:14:03.110 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38129.log 2024-03-19T02:14:03.111 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36531.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36531.log.gz 2024-03-19T02:14:03.111 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41120.log 2024-03-19T02:14:03.112 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38129.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38129.log.gz 2024-03-19T02:14:03.112 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36583.log 2024-03-19T02:14:03.113 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41120.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41120.log.gz 2024-03-19T02:14:03.114 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37971.log 2024-03-19T02:14:03.115 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36583.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36583.log.gz 2024-03-19T02:14:03.115 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36609.log 2024-03-19T02:14:03.116 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37971.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37971.log.gz 2024-03-19T02:14:03.116 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39888.log 2024-03-19T02:14:03.117 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36609.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36609.log.gz 2024-03-19T02:14:03.117 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35614.log 2024-03-19T02:14:03.118 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39888.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39888.log.gz 2024-03-19T02:14:03.119 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32752.log 2024-03-19T02:14:03.119 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35614.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35614.log.gz 2024-03-19T02:14:03.120 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32279.log 2024-03-19T02:14:03.121 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.32752.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32752.log.gz 2024-03-19T02:14:03.121 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41879.log 2024-03-19T02:14:03.122 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.32279.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32279.log.gz 2024-03-19T02:14:03.122 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43350.log 2024-03-19T02:14:03.123 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41879.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41879.log.gz 2024-03-19T02:14:03.123 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39468.log 2024-03-19T02:14:03.124 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.43350.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43350.log.gz 2024-03-19T02:14:03.125 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41356.log 2024-03-19T02:14:03.126 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39468.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39468.log.gz 2024-03-19T02:14:03.126 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33001.log 2024-03-19T02:14:03.127 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41356.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41356.log.gz 2024-03-19T02:14:03.127 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38891.log 2024-03-19T02:14:03.128 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33001.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33001.log.gz 2024-03-19T02:14:03.129 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37789.log 2024-03-19T02:14:03.130 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38891.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38891.log.gz 2024-03-19T02:14:03.130 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39390.log 2024-03-19T02:14:03.131 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37789.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37789.log.gz 2024-03-19T02:14:03.131 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32871.log 2024-03-19T02:14:03.132 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39390.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39390.log.gz 2024-03-19T02:14:03.132 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40517.log 2024-03-19T02:14:03.133 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.32871.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32871.log.gz 2024-03-19T02:14:03.133 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33662.log 2024-03-19T02:14:03.134 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40517.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40517.log.gz 2024-03-19T02:14:03.135 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2024-03-19T02:14:03.135 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.33662.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33662.log.gz 2024-03-19T02:14:03.136 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34951.log 2024-03-19T02:14:03.148 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34184.log 2024-03-19T02:14:03.150 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34951.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34951.log.gz 2024-03-19T02:14:03.154 INFO:teuthology.orchestra.run.smithi055.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2024-03-19T02:14:03.154 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40701.log 2024-03-19T02:14:03.155 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34184.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34184.log.gz 2024-03-19T02:14:03.155 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42193.log 2024-03-19T02:14:03.156 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40701.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40701.log.gz 2024-03-19T02:14:03.156 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38472.log 2024-03-19T02:14:03.157 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42193.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42193.log.gz 2024-03-19T02:14:03.158 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40962.log 2024-03-19T02:14:03.158 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38472.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38472.log.gz 2024-03-19T02:14:03.159 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37841.log 2024-03-19T02:14:03.160 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40962.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40962.log.gz 2024-03-19T02:14:03.160 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40621.log 2024-03-19T02:14:03.161 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37841.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37841.log.gz 2024-03-19T02:14:03.161 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32620.log 2024-03-19T02:14:03.162 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40621.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40621.log.gz 2024-03-19T02:14:03.162 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38734.log 2024-03-19T02:14:03.163 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.32620.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32620.log.gz 2024-03-19T02:14:03.164 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42223.log 2024-03-19T02:14:03.165 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38734.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38734.log.gz 2024-03-19T02:14:03.166 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42223.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42223.log.gz 2024-03-19T02:14:03.274 INFO:teuthology.orchestra.run.smithi084.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-19T02:14:03.351 INFO:teuthology.orchestra.run.smithi084.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-19T02:14:03.717 INFO:teuthology.orchestra.run.smithi055.stderr: 89.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-19T02:14:04.092 INFO:teuthology.orchestra.run.smithi055.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-19T02:14:04.102 INFO:teuthology.orchestra.run.smithi084.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-19T02:14:04.112 INFO:teuthology.orchestra.run.smithi055.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-19T02:14:04.138 INFO:teuthology.orchestra.run.smithi055.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-19T02:14:04.165 INFO:teuthology.orchestra.run.smithi084.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-19T02:14:04.177 INFO:teuthology.orchestra.run.smithi055.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-19T02:14:04.179 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-19T02:14:04.179 INFO:teuthology.orchestra.run.smithi055.stderr:real 0m1.499s 2024-03-19T02:14:04.180 INFO:teuthology.orchestra.run.smithi055.stderr:user 0m6.741s 2024-03-19T02:14:04.180 INFO:teuthology.orchestra.run.smithi055.stderr:sys 0m0.305s 2024-03-19T02:14:04.193 INFO:teuthology.orchestra.run.smithi084.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-19T02:14:04.231 INFO:teuthology.orchestra.run.smithi084.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-19T02:14:04.233 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-19T02:14:04.233 INFO:teuthology.orchestra.run.smithi084.stderr:real 0m1.544s 2024-03-19T02:14:04.233 INFO:teuthology.orchestra.run.smithi084.stderr:user 0m6.560s 2024-03-19T02:14:04.233 INFO:teuthology.orchestra.run.smithi084.stderr:sys 0m0.246s 2024-03-19T02:14:04.233 INFO:tasks.ceph:Archiving logs... 2024-03-19T02:14:04.233 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609330/remote/smithi055/log 2024-03-19T02:14:04.234 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-19T02:14:05.182 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609330/remote/smithi084/log 2024-03-19T02:14:05.183 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-19T02:14:05.823 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun 2024-03-19T02:14:06.058 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c7461268b80f4bf8b8cf1c0d17f03453 Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun 2024-03-19T02:14:06.062 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-19T02:14:06.072 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-19T02:14:06.092 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-19T02:14:06.093 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-19T02:14:06.107 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-19T02:14:06.171 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-19T02:14:06.180 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-19T02:14:06.180 DEBUG:teuthology.orchestra.run.smithi055:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T02:14:06.183 DEBUG:teuthology.orchestra.run.smithi084:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T02:14:06.192 INFO:teuthology.orchestra.run.smithi084.stdout: remote refid st t when poll reach delay offset jitter 2024-03-19T02:14:06.192 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================== 2024-03-19T02:14:06.192 INFO:teuthology.orchestra.run.smithi084.stdout:*hv01.front.sepi 44.4.53.2 2 u 21 64 377 0.053 -2.552 0.891 2024-03-19T02:14:06.192 INFO:teuthology.orchestra.run.smithi084.stdout:+hv02.front.sepi 63.231.80.2 3 u 39 64 377 0.102 -1.170 1.265 2024-03-19T02:14:06.192 INFO:teuthology.orchestra.run.smithi084.stdout:+hv03.front.sepi 74.6.168.72 3 u 21 64 377 0.085 -1.131 0.772 2024-03-19T02:14:06.193 INFO:teuthology.orchestra.run.smithi084.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-03-19T02:14:06.194 INFO:teuthology.orchestra.run.smithi055.stdout: remote refid st t when poll reach delay offset jitter 2024-03-19T02:14:06.194 INFO:teuthology.orchestra.run.smithi055.stdout:============================================================================== 2024-03-19T02:14:06.194 INFO:teuthology.orchestra.run.smithi055.stdout:*hv01.front.sepi 44.4.53.2 2 u 112 128 377 0.063 -5.165 0.387 2024-03-19T02:14:06.195 INFO:teuthology.orchestra.run.smithi055.stdout:+hv02.front.sepi 63.231.80.2 3 u 33 128 377 0.093 -1.564 0.246 2024-03-19T02:14:06.195 INFO:teuthology.orchestra.run.smithi055.stdout:+hv03.front.sepi 74.6.168.72 3 u 110 128 377 0.084 -3.541 0.361 2024-03-19T02:14:06.195 INFO:teuthology.orchestra.run.smithi055.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-03-19T02:14:06.195 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-19T02:14:06.204 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-19T02:14:06.205 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-19T02:14:06.213 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-19T02:14:06.221 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-19T02:14:06.229 INFO:teuthology.task.internal:Duration was 2429.854806 seconds 2024-03-19T02:14:06.229 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-19T02:14:06.238 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-19T02:14:06.238 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-19T02:14:06.240 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-19T02:14:06.270 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-19T02:14:06.270 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi055.front.sepia.ceph.com 2024-03-19T02:14:06.270 DEBUG:teuthology.orchestra.run.smithi055:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-19T02:14:06.320 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi084.front.sepia.ceph.com 2024-03-19T02:14:06.321 DEBUG:teuthology.orchestra.run.smithi084:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-19T02:14:06.331 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-19T02:14:06.331 DEBUG:teuthology.orchestra.run.smithi055:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-19T02:14:06.365 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-19T02:14:06.399 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-19T02:14:06.399 DEBUG:teuthology.orchestra.run.smithi055:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T02:14:06.433 DEBUG:teuthology.orchestra.run.smithi084:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T02:14:06.517 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-19T02:14:06.526 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-19T02:14:06.526 DEBUG:teuthology.orchestra.run.smithi055:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-19T02:14:06.536 DEBUG:teuthology.orchestra.run.smithi084:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-19T02:14:06.545 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-19T02:14:06.556 DEBUG:teuthology.orchestra.run.smithi055:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-19T02:14:06.581 DEBUG:teuthology.orchestra.run.smithi084:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-19T02:14:06.589 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = core 2024-03-19T02:14:06.597 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = core 2024-03-19T02:14:06.615 DEBUG:teuthology.orchestra.run.smithi055:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-19T02:14:06.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:14:06.644 DEBUG:teuthology.orchestra.run.smithi084:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-19T02:14:06.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:14:06.663 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-19T02:14:06.674 INFO:teuthology.task.internal:Transferring archived files... 2024-03-19T02:14:06.674 DEBUG:teuthology.misc:Transferring archived files from smithi055:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609330/remote/smithi055 2024-03-19T02:14:06.675 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-19T02:14:06.724 DEBUG:teuthology.misc:Transferring archived files from smithi084:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609330/remote/smithi084 2024-03-19T02:14:06.724 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-19T02:14:06.750 INFO:teuthology.task.internal:Removing archive directory... 2024-03-19T02:14:06.751 DEBUG:teuthology.orchestra.run.smithi055:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-19T02:14:06.753 DEBUG:teuthology.orchestra.run.smithi084:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-19T02:14:06.786 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-19T02:14:06.814 INFO:teuthology.task.internal:Not uploading archives. 2024-03-19T02:14:06.814 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-19T02:14:06.849 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-19T02:14:06.849 DEBUG:teuthology.orchestra.run.smithi055:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-19T02:14:06.852 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-19T02:14:06.856 INFO:teuthology.orchestra.run.smithi055.stdout: 658625 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 02:14 /home/ubuntu/cephtest 2024-03-19T02:14:06.858 INFO:teuthology.orchestra.run.smithi084.stdout: 658641 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 02:14 /home/ubuntu/cephtest 2024-03-19T02:14:06.859 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-19T02:14:06.901 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-19T02:14:07.022 INFO:teuthology.nuke:Checking targets against current locks 2024-03-19T02:14:07.055 DEBUG:teuthology.nuke:shortname: smithi055 2024-03-19T02:14:07.055 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T02:14:07.089 DEBUG:teuthology.nuke:shortname: smithi084 2024-03-19T02:14:07.090 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T02:14:07.094 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609330', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 01:24:59.424129', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKlRIQTGkVhqf+6pMMXumtETSjuIj1Q6n2/pZ7EW7JggKQFzxhOFNzD60SE8WCRzx7rVm3nQcyVq4i5pk+nPOnQ='} 2024-03-19T02:14:07.117 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609330', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 01:24:59.426172', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBlNX356useTeVfcKcLCf9iAQL/T/oUnh9iHG7sjZ48DuT6vicuCSMWgw8zihf0RmaAtR/cuXslFxmaySc5msVc='} 2024-03-19T02:14:07.148 INFO:teuthology.orchestra.console.smithi055:Power off 2024-03-19T02:14:07.149 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T02:14:07.185 INFO:teuthology.orchestra.console.smithi084:Power off 2024-03-19T02:14:07.185 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T02:14:07.207 DEBUG:teuthology.orchestra.console.smithi055:power off output: Chassis Power Control: Down/Off 2024-03-19T02:14:07.208 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T02:14:07.217 DEBUG:teuthology.orchestra.console.smithi084:power off output: Chassis Power Control: Down/Off 2024-03-19T02:14:07.218 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T02:14:07.239 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-03-19T02:14:07.263 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-03-19T02:14:11.241 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T02:14:11.263 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T02:14:11.353 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-03-19T02:14:11.373 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-03-19T02:14:12.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-03-19T02:14:12.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-03-19T02:14:12.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~0s 2024-03-19T02:14:12.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-19T02:14:12.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-03-19T02:14:12.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~0s 2024-03-19T02:14:15.357 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T02:14:15.373 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T02:14:15.468 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-19T02:14:15.483 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is off 2024-03-19T02:14:15.569 INFO:teuthology.orchestra.console.smithi055:Power off completed 2024-03-19T02:14:15.584 INFO:teuthology.orchestra.console.smithi084:Power off completed 2024-03-19T02:14:15.689 INFO:teuthology.run:Summary data: description: fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} duration: 2429.8548061847687 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c7461268b80f4bf8b8cf1c0d17f03453 status: fail success: false 2024-03-19T02:14:15.689 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-19T02:14:15.768 INFO:teuthology.run:FAIL